KSI-CMT-VTDValidating Throughout Deployment
Persistent testing and validation of changes throughout deployment is automated.
Mapped NIST 800-53 controls (4)
- recipe3
- KSI only1
The mark says whether an authored AWS recipe fetches evidence for the control; the tier strip shows which Rev5 baselines also require it — where automated KSI evidence doubles as Rev5 control evidence. = in the class B, C and D baselines, left to right
- recipe— an authored recipe collects evidence for this control
- KSI only— a Key Security Indicator reaches it, but no recipe is authored yet
- orphan— no Key Security Indicator reaches it — a person writes it up instead
Collect evidence (3)
Authored AWS and pipeline recipes whose output is evidence for KSI-CMT-VTD. This mapping is this project’s opinion (AWS overlay v3.0.0, pipeline overlay v0.8.0), versioned separately from the dataset — the upstream FedRAMP rules name none of these tools.
- partialPatch Manager compliance state plus Amazon Inspector scan status and coverage — proving flaws are being found continuously (Inspector enabled and actually covering your resources) and that the fixes landed (per-node missing/failed patch counts and the time of the last scan or install)dailycliawspatch-and-vulnerability-remediationAWS Config · AWS Systems Manager Patch Manager · Amazon Inspector
This proves flaws are detected and shows exactly what is still missing and when patching last ran — it does not prove the remediation clock was met. Whether an open finding sits inside your SI-02 timeframe, or carries an approved deviation or POA&M entry, is a judgement joined against your risk-acceptance record, not an API result. Substitute your real instance ids and account id; describe-instance-patch-states requires --instance-ids (use describe-instance-patch-states-for-patch-group to sweep a patch group). Note also that patch compliance data is a point-in-time snapshot and each successful scan overwrites the previous one, so capture the output at collection time rather than reconstructing history later.
- partialWho changed what, when, and from which state to which — CloudTrail's record of every mutating API call and Config's per-resource configuration history, plus, where Change Manager is in use, the change-request executions that carry the approvalweeklycliawscloudtrail-config-change-historyAWS CloudTrail · AWS Config · AWS Systems Manager Change Manager
CloudTrail and Config prove a change was recorded and is reconstructable — actor, time, parameters, and the before/after configuration state — which is CM-03's record-retention and monitoring half. They do not prove the change was proposed, reviewed and approved before it happened; that decision lives in your CCB or ticketing system and has to be joined by ticket or change id. State the retention limits plainly rather than implying full history: lookup-events reaches back only 90 days and returns management (and Insights) events only, so anything older must be read from the trail's S3 objects or a CloudTrail Lake event data store, and get-resource-config-history honours your Config retention period (30 days minimum, up to 7 years) but each call spans at most 7 days, so a long window needs paging by time. Substitute your real resource type, resource id and time window. Change Manager has also been closed to new customers since 2025-11-07, so treat that command as available only if you were already signed up. Whether the post-change configuration is still compliant is a different question answered by the Config rule recipes.
- partialFor every change that reached the assessed branch, the automated verification that ran against it — which workflows ran, on which commit, and what each concluded — together with the two things that decide whether those runs were a condition of the change or merely adjacent to it: the rule that made the checks required, and the platform's own per-push record of whether that rule held, failed, or was bypassed. The runs alone are activity; the rule and the per-push record are what make them a gate.weeklyapipipelinechange-verification-status-checks-and-run-recordsGitHub Actions · GitHub repository rulesets
CM-04 (02) requires the provider to verify that the IMPACTED CONTROLS are implemented correctly, operating as intended, and producing the desired outcome. This output shows that automated checks ran against each change, that passing them was a condition of the change, and what each concluded. Which controls a change impacts, and whether a passing test verifies one of them, is a mapping from test to control that a human writes; a pipeline that publishes such a mapping is publishing an authored artifact rather than a measurement. The disposition this recipe spends said exactly that, and writing the commands down has not changed the rating. THE DRY-RUN TRAP, AND THE ENDPOINT CHOICE THAT ALREADY CLOSED IT. A ruleset in `evaluate` mode is a complete, well-formed set of rules that enforces nothing, and it is this plane's empty list wearing a different coat: it appears in `/rulesets` intact and generates rule suite records. The first draft of this batch hedged about whether `/rules/branches/{branch}` returns such rules, called it a live risk, and told the reader to make one call against a live repository to settle it. That was a manufactured uncertainty, and the page the recipe already cites settles it in one sentence: the endpoint "Returns all active rules that apply to the specified branch", and "Rules in rulesets with \"evaluate\" or \"disabled\" enforcement statuses are not returned." So reading the rules IN EFFECT rather than the rulesets that contain them is what closes the trap here, and it was closed before any clause was written. The consequence runs both ways and both are worth stating: the first two clauses are stronger than the draft claimed, because a rule they find is enforcing by construction; and the evaluate clause is weaker, because it guards nothing above it. It stays as a finding in its own right — a branch ruleset somebody believes is in force and that blocks nothing — narrowed to `target=='branch'`, since the same response carries tag, push and repository rulesets whose dry-run state says nothing about this change path. A REQUIRED-CHECKS RULE THAT REQUIRES NOTHING. The `required_status_checks` rule takes a `required_status_checks` array of contexts, and an empty one is a rule that is present in every listing, satisfies the first clause, and gates on no check whatsoever. The second clause is spelled `!parameters.required_status_checks` rather than with `length()` deliberately: an empty array and an absent key are both falsy under that spelling and both are genuine offenders, while `length()` raises on the row where `parameters` is missing entirely — which is the row most worth seeing. A rule object that legitimately carries no `parameters` would fail this clause; that is a false red that makes somebody look, which is the direction this repo has chosen every time it has had the choice. WHY THE BYPASS CLAUSE IS ASSERTED HERE AND THE PUSH-PROTECTION ONE WAS NOT. `secret-exposure-detection-and-push-protection` deliberately does not count push-protection bypasses to zero, because a bypass there is a documented decision with a named actor and one of three stated reasons, and a provider with a legitimate test fixture would fail such a clause while behaving correctly. A rule suite bypass has no reason field in anything fetched: the record carries `actor_name`, `result` and the rule evaluations, and nothing that says why. It therefore cannot be read as an authorized exception — all it says is that a change reached the assessed branch without the verification the rule required, which for CM-04 (02) is the finding rather than a footnote. What it CAN be read against, and the first draft of this batch missed this, is `bypass_actors` — returned in the same `rulesets` response this recipe already collects, with a `bypass_mode` of `always`, `pull_request` or `exempt`. That is a standing authorization to bypass, configured and visible, so a provider with a designated break-glass integration fails this clause while operating exactly as configured. The per-event reason is absent; the standing permission is not, and a failure of this clause is read against that list before it is read as a finding. Note one collection caveat that would make any future clause over `bypass_actors` vacuous: GitHub documents that "To prevent leaking sensitive information, the bypass_actors property is only returned if the user making the API request has write access to the ruleset", so a read-only assessment token sees an empty bypass list rather than an error. THE WINDOW, AND WHY THE CADENCE IS WEEKLY. `time_period` accepts `hour`, `day`, `week` and `month`, defaults to `day`, and `month` is the ceiling — and no retention period is documented for rule suite records at all. The bypass clause is therefore a statement about at most a month, and about however much of that month the platform still holds. Collecting monthly against a monthly ceiling leaves no margin: a collection that slips by a day loses records silently rather than erroring, and the records it loses are exactly the ones nobody saw. Weekly is four chances at the same evidence. WHY NO CLAUSE COUNTS THE FAILED RUNS. The sixth command returns a list built to contain offenders, and it carries no assertion. A workflow run that failed on the protected branch is the verification apparatus working — the desired state is that failures happen and are addressed, not that none occur — and a scheduled nightly that failed once says nothing about whether any change was verified. Both run lists are corroboration for a human: that runs exist at all, that they run on the branch the rules protect, and that their names correspond to the contexts the required-checks rule names. Nothing here checks that last correspondence, and it is the join an assessment most wants — a required context is a string, a workflow run has a name, and no response in this recipe says the string was produced by the run. NO CLAUSE READS A POSITION, deliberately. Nothing fetched documents the sort order of `/actions/runs`, so nothing here reads `[0]` as "the most recent". The immediately preceding batch on this plane put a freshness clause on `analyses[0]` and had to be corrected, because position zero was whichever language finished last. The same shape is available here and is not taken; a freshness claim over these runs would need a `created` range in the query rather than an index into the response. KSI-CMT-VTD is the direct claim: persistent testing and validation of changes throughout deployment, automated. KSI-CMT-LMC — modifications to the offering are logged and monitored — is claimed on the LOGGED limb only. The rule suite record is a per-push log of every change measured against the rules, and the workflow runs are a per-change record of what ran against it, which is logging in the most literal sense. Monitored is weaker and is not claimed: nothing in this recipe routes any of it to a person or to an alert, and no threshold in it fires. The evidence platform is itself an external system; see `external_system`. Two sibling recipes in the same batch — `developer-change-control-and-integrity` and `security-representative-change-approval` — read overlapping commands for different controls. They are separate recipes rather than one, because a single recipe carrying all three would put one `automatable` rating and one assertion set over three separate arguments, and the argument is the thing being assessed.
Default evidence owed by every indicator (5)
From info.default_artifacts.KSI — no indicator carries its own artifacts; requirement-specific evidence lives on /evidence.
- Explanation of measures (and their objectives) that demonstrate the Key Security Indicator, or an explanation of the reason and resulting risk to customers for not having measures available for that Key Security Indicator.
- Explanation of the cycle for any measures that are implemented persistently (if applicable).
- Verification that the measures demonstrate the Key Security Indicator, or that the reason for not having them is accepted.
- Verification that the automation in place is accurate and sufficient to demonstrate appropriate measures for the Key Security Indicator, or that automation is not necessary for each measure.
- Validation that the measures are accurately produced and are in place and working as intended, or that the reason for not having them is valid.
Defined terms used (2)
- Persistently
- Occurring in a firm, steady way that is repeated over a long period of time in spite of obstacles or difficulties. Persistent activities may vary between actors, may occur irregularly, and may include interruptions or waiting periods between cycles. These attributes of persistent activities should be intentional, understood, and documented; the status of persistent activities will always be known.
- Validation
- Confirmation through objective evidence that implemented security capabilities and related certification data are suitable for their intended FedRAMP Certification use and support the expected security outcomes for a cloud service offering.
The Change Management run (4)
4/4 have an authored AWS recipe- 3 authored recipesKSI-CMT-LMCLogging Changes
- 6 authored recipesKSI-CMT-RMVRedeploying vs Modifying
- 2 authored recipesKSI-CMT-RVPReviewing Change Procedures
- 3 authored recipesKSI-CMT-VTDValidating Throughout Deploymentyou are here