Regulatory Mapping Completeness (RMC)

GMI-IT-037IT OperationsInfrastructure-Dependent

Definition

Regulatory Mapping Completeness (RMC) measures the percentage of controls actually affected by a regulatory change that were correctly identified by the AI regulatory change tracker: measured retrospectively by comparing AI output to a qualified expert review of the same change.

The Measurement Gap

The NIST Cybersecurity Framework 2.0 was published in February 2024. The GRC platform's AI regulatory tracker processed it within 48 hours, mapped the update to 28 of the organization's 150 IT controls, and generated a work queue for policy reviews. The team worked through the queue over the following six weeks, reviewed each mapped control, and updated the ones that needed changes. The regulatory change was "processed." The dashboard showed complete.

Eight months later, during the annual SOC 2 audit, the auditor flagged four controls that needed updating based on CSF 2.0's new Govern function, a functional area that didn't exist in CSF 1.1 at all. All four controls related to organizational governance structures for AI risk management. The AI regulatory tracker had missed them entirely because the controls were relatively new additions the organization had added specifically for AI governance, and the AI's training data predated both the controls and the Govern function. There was no prior mapping to update. The AI had no frame of reference.

The audit finding was fixable. The controls were updated within 30 days. But the team had operated for eight months believing the CSF 2.0 transition was complete when it wasn't.

This is the failure mode Regulatory Mapping Completeness (RMC) measures: the gap between what the AI regulatory change tracker identifies as affected by a regulatory change and what actually is.

The AI regulatory tracker is doing something genuinely valuable. Monitoring regulatory feeds, parsing legal and standards language, and mapping changes to internal controls is exactly the kind of high-volume, tedious work that AI should absorb. The failure mode isn't that the AI is wrong in some obvious way. It's that the AI's completeness is variable and unmeasured. Straightforward changes (a new version of an established standard with clear correspondence to prior requirements) get high-completeness mapping. Novel regulatory changes (new functional areas, new compliance domains, cross-boundary requirements affecting controls that weren't previously mapped to the framework) get lower-completeness mapping because the AI has less analogous training data to draw on.

The legacy metrics don't see this gap. "Regulatory changes processed" counts how many updates were ingested and acted on, a process compliance metric that says nothing about completeness. Time-to-process measures operational efficiency. It doesn't measure whether the processing found everything. These metrics were designed for a world where humans read the regulatory text and made mapping judgments themselves. Human reviewers have their own accuracy limits, but they bring contextual understanding (particularly for novel regulatory language) that AI systems are still developing.

The measurement gap is this: there is no metric that tells you what percentage of controls actually affected by a regulatory change were correctly identified by the AI tracker. An organization could process every regulatory change quickly and document the processing thoroughly while systematically under-mapping novel or complex regulatory updates, and the gap would only surface at audit, not during the processing workflow.

Formula

Regulatory Mapping Completeness (RMC) measures the percentage of controls actually affected by a regulatory change that were correctly identified by the AI regulatory change tracker: measured retrospectively by comparing AI output to a qualified expert review of the same change.

Formula: Plain Language

After the AI maps a regulatory change to affected controls, have a qualified compliance expert review the same change independently and identify any controls the AI missed. The ratio of AI-identified controls to total truly affected controls (AI-identified plus expert-identified missed controls) is the completeness rate.

Formula: Notation

RMC = C_AI / (C_AI + C_missed)

Where:

  • C_AI = controls the AI correctly identified as affected by the regulatory change
  • C_missed = controls that the expert review identified as affected but the AI failed to flag
  • C_AI + C_missed = total controls truly affected by the change (ground truth as established by expert review)

Units: Percentage. Higher is better. 100% means the AI identified every affected control.

Worked Example

Your AI tracker processes a new GDPR supplemental guidance publication affecting data handling requirements for automated decision-making. The AI maps the change to 12 controls. Your DPO and GRC analyst review the same guidance independently and identify 3 additional controls the AI missed: two involving data retention policies for HR systems that weren't mapped to GDPR in your control library, and one involving logging requirements for automated data subject request handling, a control added recently that the AI's training data didn't include in its GDPR mapping context. C_AI = 12C_missed = 3

RMC = 12 / (12 + 3) = 12 / 15 = 80%

At 80% RMC, the AI caught 4 out of 5 affected controls, which is operationally useful: it substantially reduces the expert review burden. But the 3 missed controls represent 20% of the actual compliance gap from this regulatory change. Without measuring RMC, you would have no signal to go looking for them.

Sign Convention

Higher RMC is better. An RMC of 100% means the AI caught everything identified by expert review. Lower RMC means a higher proportion of affected controls are being missed and will not receive the regulatory update response they need unless expert review is maintained as a systematic step.

Edge Cases

Expert review may also miss controls: Expert review is not perfect ground truth, particularly for indirect control dependencies. When available, use subsequent audit findings related to the same regulatory change as a validation source. If auditors find controls that neither the AI nor the expert review identified, add those to C_missed retroactively and note that the RMC calculation for that change was optimistic. Small sample sizes: If the measurement period contains only 1-2 regulatory changes, individual RMC values are noisy. Roll up RMC across 6-12 months to produce a stable number. A single change where the AI finds 3 of 4 controls yields 75% RMC, not a meaningful basis for conclusions about the AI system's general performance. Novel regulatory domains: The first time the AI processes requirements in a new domain (a brand-new regulation, a new technology category, a new compliance framework), RMC will be lower than for established domains. Track RMC separately by regulatory domain and change type. "The AI is learning this new framework" has a different implication than "the AI is systematically missing a control category in a framework it has processed for three years."C_missed equals zero: If the expert review finds nothing the AI missed, RMC is 100% for that change. This is possible for well-established frameworks with incremental updates. Don't use a streak of 100% RMC values to justify eliminating expert review: the value of expert review is precisely that it catches the changes where RMC would have been low if review hadn't occurred.

Cross-References

Upstream: None within this KPI framework. RMC measures the quality of the AI regulatory intelligence input. The AI tracker reads external regulatory feeds and produces a control list: there is no upstream AI decision in this framework that feeds into the tracker's output quality. Downstream: Baseline Currency Index (BCI, GMI-IT-035). The mechanism linking RMC to BCI: missed regulatory mappings mean affected controls don't enter the baseline review queue. The review trigger ("this regulation changed, go update the baselines that implement it") never arrives for unmapped controls. BCI degrades not because nobody is doing reviews, but because the review queue is missing work items. Low RMC silently degrades BCI.Downstream: Compliance Exposure Duration (CED, GMI-IT-034). Missed regulatory mappings mean configurations are checked against baselines that don't include the new requirements. Systems "pass" compliance scans because the scan standard doesn't reflect the current regulatory expectation. CED measures how long non-compliant states persist, but when the standard itself is missing requirements due to low RMC, CED is optimistic by construction. A low CED against an incomplete standard is not the same as genuine compliance.

Proposed formula, pending validation pilots.

At a Glance

Category
AI-Era IT Operations
← Refresh Decision Payback Accuracy (RDPA)Remediation Masking Rate (RMR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment