When Automation Passes but Configuration Fails: Auditing AI-Generated Cloud Runbooks: A Reproducibility-Centred Evaluation

Abstract

Cloud runbooks can report successful execution even when their final storage, identity or rollback state violates operating policy. This gap has consequences for interpreting performance outside carefully curated evaluation settings. The research question is whether it is possible to develop checks that distinguish a successful command sequence from a correct and recoverable infrastructure state. Previous work provides context for sampling, input validation and study boundaries [3–7]. The planned analysis uses a software-assurance experiment using mutated runbooks, controlled API faults and machine-readable postconditions. Prior work on storage operations provides technical context for validation [1], while complementary systems evidence motivates service-continuity checks [2]. Under matched evaluation conditions, the protocol compares human-authored scripts, assistant-generated scripts and scripts with policy verification gates. Methodological sources are used to specify transparent comparison and reporting practices [8–12]. The reproducibility extension records dataset versions, model settings, exclusion rules and provenance before any held-out evaluation. Performance will be judged using configuration violations, undetected partial failures, idempotence and time spent diagnosing recovery. Interpretation will account for privilege boundaries, rollback safety, configuration drift and reviewer-blinded assessment. An independent replication exercise would repeat the main analysis from the documented artefacts and quantify disagreements. The supplementary technical literature is used to identify plausible failure mechanisms [13–17]. No study result is assumed in advance; the analysis is designed to show both when the method helps and when it fails, with reporting context from the remaining sources [18–23].