Disposition Recommendation Bias Index (DRBI)

GMI-AVI-031AviationIntegration-Required

Definition

Degree to which the AI engineering disposition model's recommendation distribution diverges from expert engineering judgment distribution: specifically measuring whether the model has developed a bias toward less disruptive dispositions (defer, accept-as-is) relative to what qualified engineers would recommend for the same findings.

The Measurement Gap

The measurement gap DRBI addresses: engineering teams that adopt AI disposition assistance track how often the AI's recommendation is accepted versus overridden, but not whether the AI's recommendation distribution is systematically biased relative to expert judgment. An AI that recommends "defer" 70% of the time may be right if the finding population is 70% deferrable, or it may be biased if the same findings would receive "repair" recommendations from a qualified engineer at a 50% rate. Without a comparison distribution, the acceptance rate alone cannot diagnose bias.

Disposition bias has a directional asymmetry that makes it safety-critical in one direction: bias toward less disruptive actions (defer, accept-as-is) on findings of medium or high severity means serious findings are systematically being underresponded to. An engineer who accepts an AI "defer" recommendation on a finding that warranted "repair immediately" isn't wrong because they ignored the finding: they're wrong because the AI anchored their judgment toward a less disruptive outcome, and the engineer accepted that anchor.

The decision DRBI supports: should the AI disposition model be retrained with severity-weighted loss functions? Should human engineering review be required for all AI "defer" and "accept-as-is" recommendations on findings above minor severity? Is the AI acceptable as a first-pass screening tool, or is it biasing decisions in ways that create airworthiness risk? These decisions require evidence that the bias exists and in which direction it operates, which is what DRBI measures by comparing the AI's recommendation distribution against a matched expert distribution on the same findings.

Formula

Plain language: Sample non-routine findings where the AI proposed a disposition. Have qualified engineers independently assess the appropriate disposition without seeing the AI recommendation. Compare the distribution of AI recommendations to the distribution of expert recommendations across all disposition categories.

Notation:
DRBI = Σ |P_AI(d) − P_expert(d)| / 2 across all disposition categories d

This is the total variation distance between the AI and expert recommendation distributions.

Variables:

  • P_AI(d) = proportion of AI recommendations in disposition category d across the sample
  • P_expert(d) = proportion of expert recommendations in disposition category d across the same sample
  • d = disposition categories: {repair_immediately, replace, defer_with_conditions, accept_as_is, reject_scrap}

Units: 0 to 1. 0 = AI and expert distributions match perfectly. Higher = more divergence. Always compute the direction of bias separately: is the AI more conservative (biased toward defer/accept) or more aggressive (biased toward repair/replace) than experts?

Critical sampling requirements:

Severity stratification: sample must include findings across all severity levels. Minor, moderate, and significant. A sample composed predominantly of minor findings will show high concordance because both AI and humans agree "accept-as-is" on minor findings. The bias appears in medium and significant findings. Require at least 15 samples at each severity tier.

Blind review: experts must assess findings without seeing the AI recommendation. If experts see the AI recommendation before forming their own judgment, their assessment may be anchored by the AI, invalidating the comparison.

Minimum sample: 50 non-routine findings per quarter, stratified by severity and ATA chapter.

Proposed formula, pending validation pilots.

At a Glance

Category
Engineering Disposition & Regulatory Classification
← AD/SB Classification Concordance Rate (ADCR)CV Detection Threshold Alignment Rate (CVDTAR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment