What a gap actually is
The number your team quotes is a set intersection, a gap is the distance between the evidence you hold and the evidence an assessor can confirm on demand, and it comes in four shapes — only one of which is “the config is wrong”. Each shape has a plane it lives on and a tool class that finds it.
Teams quote one number for “the automated controls” and argue about it for an hour. Here is the number, drawn as what it is: the part of a class’s baseline that at least one Key Security Indicator reaches — sense 2 from the last step, not a promise that a machine can prove any of it.
Class B baseline95 reached · 60 orphans · 155 in baseline
Class C baseline199 reached · 123 orphans · 322 in baseline
Class D baseline199 reached · 210 orphans · 409 in baseline
The step B → C adds104 reached · 167 added · 0 removed
The step C → D adds0 reached · 87 added · 0 removed
Two things to take from the picture. The lit segment on the two upper classes is the same set — climbing a class adds controls, and the step’s own bar shows how many of the added ones any indicator reaches. And the dim remainder is not “manual”: it is scope that 20x’s machinery has no opinion about, which is a different sentence and needs a different plan.
Now the word the number is usually quoted next to. A gap is the distance between the evidence you hold today and the evidence an assessor can confirm — and re-confirm on demand. It takes four shapes, and only the first is what a dashboard means by the word.
- Present statethe evidence you hold today
- The gapone of 4 shapes, below
- Satisfiedconfirmable, and re-confirmable
AFinding
Something is wrong, and you can see exactly what.
The evidence exists and it says non-compliant. The estate is wrong; engineering fixes it. The only shape most dashboards can show.
For example, a query lists every database instance in the account with its storage-encryption setting, and two come back false. SC-28 asks for encryption at rest; those two rows are its answer.
ask can you point at the row that failed? · closed by engineering changes the estate
BUncollected
It may well be fine. You just cannot show that it is.
The control may well be met, but nothing collects proof — or the proof is a screenshot rather than something regenerable on demand. Nobody knows the state.
For example, AU-06 asks that someone reviews the audit records. Your team genuinely does, every Monday, in a meeting. Nothing you could re-run on a Tuesday afternoon produces a record of that review.
ask if an assessor asked for this today, what would you run? · closed by a collector is built, on the plane the evidence lives on
CNot automatable
No machine will ever show it. A human artifact has to.
No machine can produce probative evidence: the control is about people, documents, physical space or a third party. Its gap is document currency and assessor review, and the honest tool output is “narrative”.
For example, PE-03 asks who may physically enter the room holding the hardware. You rent that room. No API call you can make proves it; the provider’s attestation, and the date on it, is the evidence.
ask could any machine, ever, produce this? · closed by a written artifact, tracked for freshness
DScope unverified
The check passed. Nobody knows how much of the estate it looked at.
The check passed over a population the scanner chose for itself. “No non-compliant device” is the output of a well-run fleet and of a fleet with one device enrolled. Until an independent inventory is reconciled, the green is unearned.
For example, a vulnerability scan reports no critical findings across your instances for RA-05. It reads an agent baked into the golden image — and three older instances predate that image and never got the agent. They are not in the pass, not in the fail, and not in the report at all.
ask how many things did that check look at, and who wrote that list? · closed by an inventory the scanner does not control
Three of those are states your evidence is in; one is a property of the control itself. A single control moves from B to A to satisfied as you build the collector and then fix what it finds — and slides to D the day the estate grows something the collector was never pointed at. C never moves. No collector is coming for a background check, and a plan that budgets engineering time to close it is budgeting for nothing.
The two that get mixed up are B and D, and one question separates them. B has no answer at all; D has an answer, it is green, and the list it is green over is a list the scanner wrote for itself. B looks like a hole in the report — which is why it gets found. D looks like a pass, which is why it does not. FedRAMP has a control for exactly this problem: CM-08asks for a system component inventory, so “reconcile against a list the scanner does not control” is an obligation you already owe, not a nicety this site invented.
Every instance carries the agent
the scan prints no critical findings · it read 12 of 12 · 0 never reached
every instance in the account was launched from the golden image, and the image carries the scanner’s agent. The verdict is true of the whole estate.
Every instance launched since the image changed carries the agent
the scan prints no critical findings · it read 9 of 12 · 3 never reached
three instances predate that golden image and never got the agent. They are not in the pass, not in the fail, and not in the report — a check cannot report a machine it never reached.
Nothing in the scan’s own output separates those two, and re-running it will not help: a check cannot count what it never reached, and it does not know that it did not reach it. The difference is visible only against a list the scanner did not write — the account’s own instance inventory, the repository list from the source-control API, the HR headcount behind a laptop fleet — reconciled against the set the check reported on.
Which makes the habit that closes D a small one: never quote a green without its denominator and the name of the list that denominator came from. “No non-compliant devices” is not a reading. “No non-compliant devices, over every name in this morning’s HR headcount” is. If you cannot name the list, you have not measured the estate — you have measured the scanner.
Asked in order rather than read as a menu, only one question has to be answered at a time:
Could any machine ever produce probative evidence — or is the subject matter a person, a document, a building or a third party?
If no machine can, the shape is C Not automatable. Otherwise, next question.
Is something collecting it today, and could you re-run that collection on demand and get the same kind of answer?
If nothing collects it, or the proof is a rendering, the shape is B Uncollected. Otherwise, next question.
What does the collected evidence say?
If non-compliant, the shape is A Finding. Otherwise, next question.
It passed — over what population, and who wrote that list?
If the scanner wrote it, or nobody can say, the shape is D Scope unverified. Otherwise, read on.
Answer past all four — something collects it, you can re-run that collection, it comes back compliant, and it comes back compliant over a population an independent inventory confirms — and there is no gap. That is what “satisfied” meant at the top of this step.
Which tool finds them? No single one. Evidence lives on planes — the cloud control plane, the pipeline that builds it, the identity provider people actually live in, the device fleet, the systems that hold training and screening records — and each plane has its own scanner class and, crucially, its own inventory. A scanner cannot report what it was never pointed at, which is shape D, and it is defeated only by an inventory the scanner does not control.
| Plane | Tool class | Families | Finds | Here |
|---|---|---|---|---|
| Cloud control planeresource configuration, encryption, network boundaries, logging, backup settings, IAM policy | Cloud configuration compliance (CSPM)AWS Config rules and conformance packs, Security Hub, IAM Access Analyzer; Azure Policy; Wiz | AC · AU · CM · CP · IA · SC · SI | A B | AWS plane · 51 recipes |
| Code and pipelinesource, dependencies, secrets, infrastructure templates, build provenance, review and approval records | SAST · SCA · secret scanning · IaC scanning · attestationCodeQL, Dependabot, secret scanning, artifact attestations, repository rulesets; Semgrep, Trivy, Checkov, Sigstore | SA · SI · SR · CM · IA | A B D | pipeline plane · 13 recipes |
| Infrastructure and workloadsOS packages, images, patch state, malware, host integrity | Vulnerability management · endpoint detectionAmazon Inspector, Systems Manager Patch Manager, GuardDuty runtime; Tenable, Qualys | RA · SI · CM | A D | AWS plane · 51 recipes |
| Identity providerhuman accounts, MFA enrolment, session policy, joiners and leavers, privileged elevation | Identity provider APIs · identity governanceIAM Identity Center, Okta, Entra ID | IA · AC · PS | A B D | named, not yet authored |
| Endpoint fleetdevice compliance, disk encryption, wireless configuration | Device management (MDM / UEM)Intune, Jamf, Kandji | AC · CM · RA | A D | named, not yet authored |
| People and process systemstraining completion, screening, signed agreements, plans, exercise records | Learning, HR and ticketing systems · GRCan LMS, an HRIS, a ticket tracker, a GRC platform | AT · PS · PL · CP · IR · CA | C | named, not yet authored |
| Shared responsibilitywhat the cloud provider or platform vendor does on your behalf | Provider authorizations and inheritanceAWS Artifact, the platform's own FedRAMP package | SR · SC · SA · CA · AC | C | named, not yet authored |
One consequence worth saying out loud: every plane you add to close a control is a new external system, and external systems are themselves controls — the plane that closes a supply-chain control raises the supply-chain question about the plane. Honest coverage counts that cost. Now call the shape on eight scenarios — the drill is the step:
Exit check — a stakeholder asks “what percentage of the gaps are closed?”
Four columns, never one: findings (A), uncollected (B), not automatable (C), scope unverified (D) — each with its plane and its denominator stated. A single percentage folds B and D into whichever side flatters it. The register at /automation is this site’s own four-column answer for the surface it reaches.