Definition
Knowledge Retrieval Accuracy Score (KRAS) measures the percentage of AI-retrieved knowledge items that pass three criteria simultaneously: factual accuracy, currency, and contextual appropriateness. A response passes only if it clears all three.
The Measurement Gap
A utility in the Pacific Northwest deployed a RAG-based maintenance knowledge system to give field technicians on-demand access to OEM manuals, procedures, and regulatory requirements. The system had ingested over 4,000 documents accumulated across 20 years: original equipment manuals, revised procedures, superseded specs, and current engineering standards all living in the same index.
A technician working on a chlorine dosing pump queried the system for the torque specification on the packing gland nuts. The system returned a confident answer: 15 ft-lbs. The technician applied it. Three days later, the packing gland leaked chlorine gas, triggering an emergency evacuation of the pump room. When the incident team pulled the OEM documentation, they found that the correct torque for that specific pump model and vintage was 8 ft-lbs. The system had retrieved a spec from a different model in the same product family: accurate for a 2003 vintage, wrong for the 2019 version in that pump room.
The system's usage metrics looked fine. High adoption. Fast response time. The system's internal confidence score for that query was 0.92. Every number the team was tracking said the knowledge system was working. The chlorine leak said it wasn't.
This failure mode is easy to dismiss as an edge case. It isn't. Maintenance knowledge is contextually dense. The same torque spec, procedure, or troubleshooting sequence that's correct for one asset class, vintage, or operating condition can cause a failure when applied to a different one. A RAG system that retrieves technically accurate information for the wrong context isn't less dangerous than a broken search: in some ways it's more dangerous, because the technician had no indication anything was wrong. The system returned an answer in a confident tone, with no caveat that the retrieved document applied to a different product vintage.
This creates a specific accountability problem. When a knowledge system returns a blank result, the technician knows to find another source. When it returns a confident, plausible-sounding answer that happens to be contextually wrong, the technician has no reason to doubt it. The gap between "accurate for a different situation" and "accurate for this situation" is invisible without a metric designed to surface it.
Training completion rate measures how many hours your technicians spent in class. That number has nothing to say about whether the system gave them correct guidance when they queried it at 2 AM during a breakdown. Knowledge base article count tells you how many documents the system has indexed: volume is not accuracy. Search usage metrics confirm the system is getting queried. None of these tell you whether the answers are right, whether the version retrieved is current, or whether the guidance applies to the specific asset the technician is standing in front of.
There is no metric that tells you whether AI-retrieved maintenance knowledge is accurate, current, and contextually appropriate.
Formula
Knowledge Retrieval Accuracy Score (KRAS) measures the percentage of AI-retrieved knowledge items that pass three criteria simultaneously: factual accuracy, currency, and contextual appropriateness. A response passes only if it clears all three. One failure out of three is a fail.
The Formula
The formula is straightforward:
KRAS = Qₚ / Qₜ × 100
Where Qₚ is the number of queries where the AI response passed all three criteria, and Qₜ is the total number of queries in the audit sample.
The three criteria work like this. Factual accuracy: is the information technically correct? Currency: is this the most recent version of the guidance. Has it been superseded by a revised document or updated specification? Contextual appropriateness: does this guidance apply to the specific asset class, vintage, operating condition, and situation described in the query? A response that's accurate and current but retrieved for a pump model the technician isn't working on fails on the third criterion. You need all three.
Worked Example
A refinery audits its AI knowledge system with 80 test queries distributed across four domains: procedures, specifications, troubleshooting, and regulatory. Domain-by-domain results:
Procedures (20 queries): 17 pass all three criteria. 2 fail currency: superseded documents were retrieved. 1 fails contextual appropriateness: the procedure was for a different equipment configuration at a different operating pressure.
Specifications (20 queries): 14 pass all three. 1 fails on factual accuracy: the retrieved spec value was incorrect. 2 fail currency. 3 fail contextual appropriateness: correct spec values for the wrong asset class variant.
Troubleshooting (20 queries): 12 pass all three. 2 fail on factual accuracy. 1 fails currency. 5 fail contextual appropriateness. Troubleshooting is the weakest domain: the guidance is highly dependent on specific operating conditions, failure mode signatures, and asset configuration details that the system consistently fails to match. A context-appropriateness failure rate of 25% in troubleshooting means technicians diagnosing live breakdowns have a one-in-four chance of receiving guidance that doesn't apply to their situation.
Regulatory (20 queries): 18 pass all three. 1 fails currency, an updated regulation was retrieved in its previous version. 1 fails contextual appropriateness, a jurisdiction-specific requirement was applied to a site governed by different regulations.
KRAS = (17 + 14 + 12 + 18) / 80 × 100 = 61 / 80 × 100 = 76.3%
Overall KRAS is below the 90% threshold for unsupervised use. The system needs improvement before technicians can rely on it without independent verification. The breakdown tells you where to focus: troubleshooting guidance needs a better context-matching mechanism, and the document management system needs stronger version control integration to reduce currency failures.
Edge Cases
Three edge cases worth handling correctly. First, when the system returns "I don't know" or "insufficient information," score this as a pass for factual accuracy: providing no information is better than providing wrong information. Flag it as a coverage gap separately. Second, when the system returns a partially correct response (right procedure, wrong torque value) score as fail. Partial accuracy in maintenance guidance is often more dangerous than no guidance, because the technician acts with false confidence on the parts that seem right. Third, when multiple valid procedures exist for the same task, accept any valid procedure as correct, but flag if the system failed to indicate alternatives exist.
Sign convention: higher is better. 90% or above permits unsupervised use. 80–90% is acceptable with "verify critical steps" guidance added to the interface. Below 80%, mandatory source document verification for all safety-critical and regulatory tasks applies regardless of what the system returns.
Cross-References
KRAS has no upstream in this catalog: it measures an independent AI capability. The knowledge retrieval system's quality depends on the retrieval model architecture, the document indexing quality, and how well context is enforced at query time. Those are engineering inputs, not outputs of other KPIs in this framework.
Downstream, KRAS feeds directly into Override Outcome Index (GMI-AR-020). If KRAS is low, technicians who override AI knowledge guidance are likely making rational decisions: the guidance may be wrong for their situation. You can't interpret high override rates as a trust or adoption problem if the guidance being overridden is bad. KRAS also feeds into Operator Trust Calibration Score (GMI-AR-021). Knowledge system accuracy is a foundational driver of appropriate trust. Operators can't calibrate their responses correctly when the system's reliability varies widely across domains and they have no visibility into which domains are weak.
Low KRAS creates a specific organizational damage pattern that's worse than it looks in aggregate. Technicians who've been burned by a contextually wrong answer stop using the system entirely: including for queries where it would have given them correct answers. The trust loss is asymmetric. One bad answer can erase the credibility of dozens of good ones. You end up with a knowledge system that's technically capable but behaviorally abandoned.
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Can I trust what the AI knowledge system just told me, or do I need to verify against the original source document or consult a subject matter expert?