Certification Auto-Approval Accuracy (CAAA)

GMI-IT-028IT OperationsIntegration-Required

Definition

Certification Auto-Approval Accuracy (CAAA) measures the percentage of AI-auto-approved access certifications where the access was actually used by the user in the subsequent measurement period: quantifying whether the AI certification tool is correctly identifying necessary access or perpetuating entitlement creep at automation speed. This is a proposed metric. It doesn't appear in IGA platform dashboards, IAM maturity frameworks, or compliance reporting standards. It addresses the gap between "the certification happened" and "the certification was correct."The

The Measurement Gap

Before the IGA platform upgrade, the quarterly access review was a rubber-stamp exercise. Reviewers clicked "approve" on 96% of entitlements because the review tool gave them 847 certifications to complete in four days: no usage context, no risk context, just a list of names and permissions. The organization knew it was a problem. Certifications were completing on time and accomplishing almost nothing. The AI-assisted certification module was supposed to fix it.

The new platform auto-approved entitlements it classified as low-risk: users actively using the access, no anomalous behavior, no segregation-of-duties violations. High-risk and anomalous items surfaced for human review. Now reviewers had 94 items to work through instead of 847. Time-to-completion dropped from four days of frantic clicking to one focused afternoon. The metrics looked excellent.

Six months after the AI-assisted campaign completed, a routine IAM audit cross-referenced active entitlements against actual application access logs. The finding: 31% of the entitlements the AI had auto-approved had not been accessed by the user in the 90 days since certification. Not marginally unused: completely untouched. Production database access for a developer who moved to a management role four months before the certification campaign ran. Full Salesforce admin permissions for a regional sales manager whose actual usage was standard user functions. ServiceNow CMDB write access for a support analyst with no operational reason to modify configuration records.

Each of these access grants was individually low-risk by the AI's scoring criteria: the users weren't showing behavioral anomalies, weren't triggering SoD violations, and had at some point historically used those systems. The AI auto-approved access based on the absence of red flags, not on evidence that the access was currently necessary. The result was automated perpetuation of entitlement creep: each approval cycle locking in excess access that would compound into the next certification.

What made this invisible: access certification completion rate went from 62% on time to 100% on time. Review volume dropped 89%. By every operational metric the IAM team tracked, the AI-assisted certification was a success.

The structural problem is that legacy certification metrics measure process compliance, not decision quality. Did the review happen? Yes. Were all entitlements reviewed? Yes. Did the review actually reduce access risk? That question has no metric in standard IAM reporting.

When AI auto-approves access at scale, this measurement gap becomes a structural risk. The AI makes hundreds of approve-or-revoke recommendations per campaign. If the risk scoring is well-calibrated, those recommendations are correct. If it's not (if the AI is approving based on historical usage patterns no longer relevant, or on the absence of obvious anomalies rather than the presence of legitimate current need) the automation amplifies the rubber-stamping it was supposed to eliminate.

There is no standard metric that tells you whether AI-auto-approved certifications correctly identified necessary access. Certification Auto-Approval Accuracy measures that directly.

Formula

Certification Auto-Approval Accuracy (CAAA) measures the percentage of AI-auto-approved access certifications where the access was actually used by the user in the subsequent measurement period: quantifying whether the AI certification tool is correctly identifying necessary access or perpetuating entitlement creep at automation speed. This is a proposed metric. It doesn't appear in IGA platform dashboards, IAM maturity frameworks, or compliance reporting standards. It addresses the gap between "the certification happened" and "the certification was correct."The FormulaPlain language: Take every entitlement the AI auto-approved in a certification campaign. Wait 90 days. Check how many of those auto-approved entitlements were actually accessed by the user. Entitlements that were auto-approved but never used in 90 days were candidates for revocation that the AI missed.

Notation:CAAA = E_used / E_auto_approved

Where:

  • E_auto_approved = total entitlements the AI auto-approved during the certification campaign
  • E_used = subset of auto-approved entitlements where the user accessed the system or resource at least once in the 90 days following campaign close

Units: Percentage. Higher is better in most operational contexts, but very high CAAA (above 95%) warrants a secondary check. If the AI is only auto-approving entitlements with recent active use, it may be applying a simple recency filter rather than making a more sophisticated judgment about access necessity. An AI that auto-approves only obviously-active access isn't adding much value beyond a basic "did they log in last week?" rule.

Worked Example

Campaign statistics: 2,400 total entitlements. AI auto-approved 1,850. Human reviewers saw the remaining 550.Post-campaign access analysis at 90 days: of the 1,850 auto-approved entitlements, 1,258 were accessed at least once by the user. 592 were never accessed. CAAA = 1,258 / 1,850 = 68%.The interpretation is direct: 32% of the entitlements the AI certified as appropriate access went completely unused in the following 90 days. Some of those are legitimately infrequent-use entitlements: quarterly reporting tools, annual compliance workflows, emergency access. But 32% is high enough to warrant investigation. At minimum, the AI is auto-approving a substantial volume of unused access that should have been surfaced for human review or automatically revoked.

Edge Cases

Higher CAAA is better. The operational target should reflect the organization's access usage patterns. For environments where most entitlements are daily-use access (SaaS productivity tools, ITSM, core business applications) a target of 80%+ is reasonable. For environments with significant episodic access (quarterly financial processes, annual compliance tools, emergency break-glass accounts), 90-day usage windows will understate necessity: use 180-day windows for entitlement types with known seasonal usage patterns.

Edge Cases

Emergency or break-glass access should never be auto-approved. If the AI auto-approved any break-glass entitlements, flag as miscategorization regardless of usage outcome. Shared accounts make usage attribution unreliable: exclude from CAAA. Users who departed within the 90-day observation window: exclude their unused entitlements from the accuracy calculation, since non-usage reflects departure rather than AI decision error.

Cross-References

Upstream: Provisioning Entitlement Utilization (PEU, GMI-IT-030). Low PEU at provisioning means users were over-provisioned from day one. Those excess entitlements propagate forward: unnecessarily provisioned access becomes unnecessarily certified access in the next campaign. CAAA captures the certification stage of the same accumulation problem PEU captures at provisioning. Downstream: Entitlement creep (legacy metric). CAAA measures the mechanism by which AI perpetuates entitlement creep: auto-approved unused access is entitlement creep, produced by automation and compounding with each certification cycle. Lateral: Analyst Trust Calibration Index (existing ATCI, GMI-IT-010). If IAM reviewers lose confidence in the AI's auto-approval decisions because CAAA is low or because they find egregiously wrong auto-approvals, they'll start reviewing everything manually. That eliminates the efficiency gain from the AI certification tool: replacing one problem (human rubber-stamping) with a different one (human reviewers doing all the work anyway, now with added resentment toward the tool).

Proposed formula, pending validation pilots.

At a Glance

Category
AI-Era IT Operations
← Capacity Prediction Accuracy (CPA)Change Risk Score Accuracy (CRSA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment