Cross-Site Validation of Clinical Prediction Models on Cloud Data Warehouses: A Reproducibility-Centred Evaluation
DOI:
https://doi.org/10.66687/d4dj7516Keywords:
Artificial Intelligence, Machine Learning, Medical Informatics, Decision Support Systems, Clinical, Reproducibility of ResultsAbstract
Cloud-hosted clinical data pipelines can harmonise hospital records while obscuring differences in coding, missingness and data provenance. An evaluation that overlooks these constraints may reach conclusions that cannot be reproduced. The research question is whether it is possible to evaluate the transferability of prediction models across institutions using a traceable data-warehouse workflow. Previous work provides context for sampling, input validation and study boundaries [3–7]. The planned analysis uses a multi-institution protocol with source-hospital development data, held-out partner sites and time-based external validation. The technical pipeline must remain auditable: prior work on storage operations provides system-level context [1], while a second source identifies infrastructure conditions relevant to data continuity [2], rather than patient-level outcomes. Under matched evaluation conditions, the protocol compares local versus pooled models after identical coding rules, feature windows and missing-data checks. Methodological sources are used to specify transparent comparison and reporting practices [8–12]. The reproducibility extension records dataset versions, model settings, exclusion rules and provenance before any held-out evaluation. Performance will be judged using calibration-in-the-large, discrimination, subgroup error and reproducibility of extracted cohorts. Interpretation will account for data lineage, hospital-specific coding shifts, delayed ingestion and governance of cross-site access. An independent replication exercise would repeat the main analysis from the documented artefacts and quantify disagreements. The supplementary technical literature is used to identify plausible failure mechanisms [13–17]. No study result is assumed in advance; the analysis is designed to show both when the method helps and when it fails, with reporting context from the remaining sources [18–23].
References
[1] Mohammed Nazir. (2022). Performance Comparison of AWS EBS, EFS and S3 for Enterprise Cloud-Storage Workloads. International Journal of Business & Computational Sciences, 2(1), 26–33. https://ijbcs.org/index.php/IJBCS/article/view/2022-01-05
[2] Mohammed Nazir. (2022). Automated SnapMirror Migration from On-Premises Storage to Cloud Volumes ONTAP: A Replication Performance Study. Journal of Multidisciplinary Research and Integrated Sciences, 2(1), 61–72. https://jmris.com/index.php/jmris/article/view/2022V1P6
[3] Johnson, A., Pollard, T., & Mark, R. (2016). MIMIC-III Clinical Database, version 1.4. PhysioNet. https://physionet.org/content/mimiciii/1.4/
[4] Singhal, K., Azizi, S., Tu, T., et al. (2023). Large language models encode clinical knowledge. Nature, 620, 172–180. https://www.nature.com/articles/s41586-023-06291-2
[5] McKinney, S. M., Sieniek, M., Godbole, V., et al. (2020). International evaluation of an AI system for breast cancer screening. Nature, 577, 89–94. https://www.nature.com/articles/s41586-019-1799-6
[6] Begum, M. H., Talukder, S. I., Ahmed, M. J., Khan, M. F. I., & Azam, M. A. (2025). XAI-Driven Multimodal Deep Learning for Early Sepsis Prediction in ICU. Journal of Science Technology and Social Transformation, 1(1), 30–41. https://jstst.info/j/article/view/25
[7] Jumper, J., Evans, R., Pritzel, A., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596, 583–589. https://www.nature.com/articles/s41586-021-03819-2
[8] Esteva, A., Kuprel, B., Novoa, R. A., et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115–118. https://www.nature.com/articles/nature21056
[9] Liu, X., Faes, L., Kale, A. U., et al. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nature Medicine, 26, 1364–1374. https://www.nature.com/articles/s41591-020-1034-x
[10] Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1705.07874
[11] Mahdisoltani, F., Stefanovici, I. & Schroeder, B. (2017). Proactive Error Prediction to Improve Storage System Reliability. USENIX ATC '17. https://www.usenix.org/conference/atc17/technical-sessions/presentation/mahdisoltani
[12] Cidon, A. et al. (2013). Copysets: Reducing the Frequency of Data Loss in Cloud Storage. USENIX ATC '13. https://www.usenix.org/conference/atc13/technical-sessions/presentation/cidon
[13] Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
[14] Guo, C. et al. (2017). On Calibration of Modern Neural Networks. Proceedings of Machine Learning Research, 70, 1321–1330. https://proceedings.mlr.press/v70/guo17a.html
[15] Khan, M. F. I., Begum, M. H., Rahman, M. A., Limon, G. Q., Azam, M. A., & Masum, A. K. M. (2025). A comprehensive review of advances in transformer, GAN, and attention mechanisms: Their role in multimodal learning and applications across NLP. International Journal of Science and Research Archive, 15(1), 454–459. https://ijsra.net/content/comprehensive-review-advances-transformer-gan-and-attention-mechanisms-their-role
[16] Vrable, M., Savage, S. & Voelker, G.M. (2012). BlueSky: A Cloud-Backed File System for the Enterprise. USENIX FAST '12. https://www.usenix.org/conference/fast12/bluesky-cloud-backed-file-system-enterprise
[17] Burke, Q. et al. (2025). On Scalable Integrity Checking for Secure Cloud Disks. USENIX FAST '25. https://www.usenix.org/conference/fast25/presentation/burke
[18] Cao, Z., Kuenning, G. & Zadok, E. (2020). Carver: Finding Important Parameters for Storage System Tuning. USENIX FAST '20. https://www.usenix.org/conference/fast20/presentation/cao-zhen
[19] Nachman, A., Yadgar, G. & Sheinvald, S. (2020). GoSeed: Generating an Optimal Seeding Plan for Deduplicated Storage. USENIX FAST '20. https://www.usenix.org/conference/fast20/presentation/nachman
[20] Levi, A. et al. (2024). Physical vs. Logical Indexing with IDEA: Inverted Deduplication-Aware Index. USENIX FAST '24. https://www.usenix.org/conference/fast24/presentation/levi
[21] Alagappan, R. et al. (2019). Protocol-Aware Recovery for Consensus-Based Storage. USENIX ATC '19. https://www.usenix.org/conference/atc19/presentation/alagappan
[22] Patel, T. et al. (2020). Uncovering Access, Reuse, and Sharing Characteristics of I/O-Intensive Files on Large-Scale Production HPC Systems. USENIX FAST '20. https://www.usenix.org/conference/fast20/presentation/patel-hpc-systems
[23] Yang, Z. et al. (2023). λ-IO: A Unified IO Stack for Computational Storage. USENIX FAST '23. https://www.usenix.org/conference/fast23/presentation/yang-zhe
Published
Issue
Section
License
Copyright (c) 2026 Francis Aila
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright in their published work. Unless an individual Version of Record identifies different terms, JMRIS articles are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/. This permits sharing and adaptation, including commercial use, with appropriate attribution, a link to the license, and indication of changes. Third-party material may be subject to separate terms. Existing articles retain the license originally granted to their Version of Record; a policy update does not retrospectively replace that license.
