We documented double read features (displayed by column) for the five clinical trials (displayed by row). The two left-most columns display discrepancy features. Only for patients reported as having measurable diseases, the three right-most columns display the means of each reader's measurements (independent reader [IR]1 and 2), the p-value of the corresponding two-sample test of difference is indicated by asterisks: **, p<0.001; *, p<0.05; no asterisk means no statistically significant difference. The last row is the average overall measurements of both R1 and R2 with corresponding confidence intervals. 60.5% (229/378) Lung: 38.6 LN: 20.6 Bone: 10.8 Pleura: 4.8 LN: 14.8 Bone: 12.4 Lung: 6.9 Pleura: 6.1

Overall (N= 1720) 43.5

1121/1720) Lung: 32.7 LN: 28.3 Bone: 12.0 Pleura: 7.6 LN: 20.1 Bone: 12.2 Lung: 10.1 Pleura: 8.4 LN, Lymph node; TL, Target lesion; NTL, Non target lesion. For the five clinical trials (displayed in rows) we computed: The proportion of patients for which readers targeted all their TLs in totally different locations (DisLocTLAll); the proportion of patients for which readers targeted at least one TL (DisTLLoc) or NTL (DisLocNTL) at a different disease location; the top proportion of discrepancies in TL (TopDisLocTL), NTL (TopDisLocNTL) and disease locations (TopDisLocDisease) (

SOD, Sum of Diamters; TLNum, Number of target lesion. For seven readers (displayed in rows) involved in two or more trials, we reported in column 1) the number of assessed patients at baseline, 2) the average number of TLs selected in patients, 3) the average measured SOD, 4) the proportion of patients for whom nodal, bone, lung, pleura and infrequent disease were evaluated. Confidence intervals are provided in brackets.

FDA, U.S. Department of Health and Human Services, Center for Drug Evaluation and Research and Center for Biologics Evaluations and Research. Guidance for industry developing medical imaging drug and biologic products. part 3: Design, analysis, and interpretation of clinical studies. (2004).

FDA, US Department of Health and Human Services, Center for Drug Evaluation and Research and Center for Biologics Evaluations and Research. Clinical trial imaging endpoints process standards guidance for industry draft. (2015).

US Food and Drug Administration. Clinical trial imaging endpoint process standards: Guidance for industry. (2018). pp. 18571-3.

Beaumont H, Iannessi A, Wang Y, Voyton CM, Cillario J, Liu Y. Blinded independent central review (BICR) in new therapeutic lung cancer trials. Cancers (Basel) (2021) 13:4533. doi: 10.3390/cancers13184533

Schmid AM, Raunig DL, Miller CG, Walovitch RC, Ford RW, O'Connor M, et al. Radiologists and clinical trials: Part 1 the truth about reader disagreements. Ther Innov Regul Sci (2021). doi: 10.1007/s43441-021-00316-6

Raunig DL, McShane LM, Pennello G, Gatsonis C, Carson PL, Voyvodic JT, et al. Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment. Stat Methods Med Res (2015) 24:27-67. doi: 10.1177/0962280214537344

Zhao B, Tan Y, Bell D, Marley S. Intra-and inter-reader variability in uni- dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J (2013) 82:1-22. doi: 10.1016/ j.ejrad.2013.02.018.Exploring

Beaumont H, Evans TL, Klifa C, Guermazi A, Hong SR, Chadjaa M, et al. Discrepancies of assessments in a RECIST 1.1 phase II clinical trial -association between adjudication rate and variability in images and tumors selection. Cancer Imaging (2018) 18:50. doi: 10.1186/s40644-018-0186-0

Fournier L, de Geus-Oei LF, Regge D, Oprea-Lager DE, D'Anastasi M, Bidaut L, et al. Twenty years on: RECIST as a biomarker of response in solid tumours an EORTC imaging group -ESOI joint paper. Front Oncol (2022) 11:800547. doi: 10.3389/fonc.2021.800547

Iannessi A, Beaumont H, Liu Y, Bertrand AS. RECIST 1.1 and lesion selection: How to deal with ambiguity at baseline? Insights Imaging (2021) 12. doi: 10.1186/s13244- 021-00976-w 11. Eisenhauer EA, Therasse P, Bogaerts J, Schwartz LH, Sargent D, Ford R, et al. New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). Eur J Cancer (2009) 45:228-47. doi: 10.1016/j.ejca.2008.10.026

Ford R, O' Neal M, Moskowitz S, Fraunberger J. Adjudication rates between readers in blinded independent central review of oncology studies. J Clin Trials (2016) 06. doi: 10.4172/2167-0870.1000289

Marascuilo LA. Extensions of the significance test for one-parameter signal detection hypotheses. Psychometrika (1970) 35:237-43. doi: 10.1007/BF02291265

Niu FY, Zhou Q, Yang JJ, Zhong WZ, Chen ZH, Deng W, et al. Distribution and prognosis of uncommon metastases from non-small cell lung cancer. BMC Cancer (2016) 16:1. doi: 10.1186/s12885-016-2169-5

Kuhl CK. RECIST needs revision: A wake-up call for radiologists. Radiology (2019) 292:110-1. doi: 10.1148/radiol.2019190785

Sharma M, Singareddy A, Bajpai S, Narang J, O'Connor M, Jarecha R. To determine correlation of inter reader variability in sum of diameters using RECIST 1.1 with end point assessment in lung cancer. J Clin Oncol (2021) 39:e13557-7. doi: 10.1200/ JCO.2021.39.15_suppl.e13557

Yoon SH, Kim KW, Goo JM, Kim D-W, Hahn S. Observer variability in RECIST- based tumour burden measurements: A meta-analysis. Eur J Cancer (2016) 53:5-15. doi: 10.1016/j.ejca.2015.10.014

Darkeh MHSE, Suzuki C, Torkzad MR. The minimum number of target lesions that need to be measured to be representative of the total number of target lesions (according to RECIST). Br J Radiol (2009) 82:681-6. doi: 10.1259/bjr/72829563

Machida N, Yoshino T, Boku N, Hironaka S, Onozawa Y, Fukutomi A, et al. Impact of baseline sum of longest diameter in target lesions by RECIST on survival of patients with metastatic colorectal cancer. Jpn J Clin Oncol (2008) 38:689-94. doi: 10.1093/jjco/hyn086

Morse B, Jeong D, Ihnat G, Silva AC. Pearls and pitfalls of response evaluation criteria in solid tumors (RECIST) v1.1 non-target lesion assessment. Abdom Radiol (2019) 44:766-74. doi: 10.1007/s00261-018-1752-4

Libshitz HI, McKenna RJ. Mediastinal lymph node size in lung cancer. AJR Am J Roentgenol (1984) 143:715-8. doi: 10.2214/ajr.143.4.715

Kuhl CK, Alparslan Y, Schmoee J, Sequeira B, Keulers A, Brümmendorf TH, et al. Validity of RECIST version 1.1 for response assessment in metastatic cancer: A prospective, multireader study. Radiology (2019) 290:349-56. doi: 10.1148/radiol.2018180648

Coy HJ, Douek ML, Ruchalski K, Kim HJ, Gutierrez A, Patel M, et al. Components of radiologic progressive disease defined by RECIST 1.1 in patients with metastatic clear cell renal cell carcinoma. Radiology (2019) 292:103-9. doi: 10.1148/radiol.2019182922

Tozuka T, Kitazono S, Sakamoto H, Yoshida H, Amino Y, Uematsu S, et al. Dissociated responses at initial computed tomography evaluation is a good prognostic factor in non-small cell lung cancer patients treated with anti-programmed cell death-1/ ligand 1 inhibitors. BMC Cancer (2020) 20:1-8. doi: 10.1186/s12885-020-6704-z 25. Chen DT, Chan W, Thompson ZJ, Thapa R, Beg AA, Saltos AN, et al. Utilization of target lesion heterogeneity for treatment efficacy assessment in late stage lung cancer. PloS One (2021) 16:1-15. doi: 10.1371/journal.pone.0252041

Humbert O, Chardin D. Dissociated response in metastatic cancer: An atypical pattern brought into the spotlight with immunotherapy. Front Oncol (2020) 10:566297. doi: 10.3389/fonc.2020.566297

Beaumont H, Faye N, Iannessi A, Chamorey E, Kliffa C, Hsieh C-Y. Differences in sensitivity to new therapies between primary and metastatic breast cancer: A need to stratify the tumor response? (2020), 1-21. doi: 10.21203/rs.3.rs-92580/v1

Carter BW, Bhosale PR, Yang WT. Immunotherapy and the role of imaging. Cancer (2018) 124:2906-22. doi: 10.1002/cncr.31349

Raunig DL, Schmid AM, Miller CG, Walovitch RC, O'Connor M, Noever K, et al. Radiologists and clinical trials: Part 2: Practical statistical methods for understanding and monitoring independent reader performance. Ther Innov Regul Sci (2021) 55:1122-38. doi: 10.1007/s43441-021-00317-5

Geijer H, Geijer M. Added value of double reading in diagnostic radiology, a systematic review. Insights Imaging (2018) 9:287-301. doi: 10.1007/s13244-018-0599-0

Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials (2015) 16:1-14. doi: 10.1186/s13063-015-1023-4

Liu R, Rizzo S, Whipple S, Pal N, Pineda AL, Lu M, et al. Evaluating eligibility criteria of oncology trials using real-world data and AI. Nature (2021) 592:629-33. doi: 10.1038/s41586-021-03430-5

Thorlund K, Dron L, Park JJH, Mills EJ. Synthetic and external controls in clinical trials -a primer for researchers. Clin Epidemiol (2020) 12:457-67. doi: 10.2147/CLEP.S242097

Teslenko I, Belotserkovsky M. Common pitfalls of RECIST 1.1 application in clinical trials. Eur J Cancer (2015) 51:S132. doi: 10.1016/S0959-8049(16)30387-2