Definition
Degree to which AI refactoring preserves the original code's semantic behavior, including the deliberate edge-case handling that looks like clutter to a model.
The Measurement Gap
Measurement Gap and Decision
The Gap: Traditional code review catches some behavioral changes but not reliably. A reviewer looking at "cleaner" error handling may not remember the original structure existed to handle a specific production edge case. The refactored version looks objectively better. The behavioral change is invisible without deep context. Worse, behavioral regressions from "pure refactoring" are hard to diagnose because nobody looks at the refactoring PR first: they look at recent feature changes.
The Decision:
Can AI-generated refactoring PRs be merged with standard code review, or do they require behavioral equivalence verification beyond what existing tests cover? The answer determines review process: standard review for high-RSFS tools, behavioral assertion checklists for low-RSFS tools.
What Changes When You Have This Number:
Above 0.85, standard code review is sufficient for AI refactoring PRs. Between 0.70–0.85, require targeted review on dimensions where violations are most common (usually exception handling). Below 0.70, require explicit behavioral equivalence verification covering all five dimensions before merge. Per-dimension breakdown reveals exactly which behavioral contracts need review attention.
Formula
Formula:
RSFS = (1/N) x Σ [ (1/5) x (func_equiv + exception_equiv + log_equiv + timing_equiv + sideeffect_equiv) ]
Where each dimension scores 1 (preserved) or 0 (changed): func_equiv = functional output unchanged, exception_equiv = exception types and messages preserved, log_equiv = logging behavior preserved, timing_equiv = latency within 10% of pre-refactoring values, sideeffect_equiv = file I/O, network calls, state mutations unchanged. N = sampled AI refactoring PRs.
Sign Convention:
1.0 = perfect behavioral preservation. Lower scores indicate unintended behavioral changes.
Worked Example:
Assess 12 AI-generated refactoring PRs:
PR #1: func=1, exception=0 (catch blocks broadened), log=1, timing=1, side=1 → 0.80
PR #2: func=1, exception=1, log=0 (log levels changed), timing=1, side=1 → 0.80
PR #3: func=1, exception=0, log=1, timing=0 (async boundary added), side=0 (extra DB call) → 0.40
PR #4–12: All dimensions preserved → 1.0 each
RSFS = [0.80 + 0.80 + 0.40 + (9 x 1.0)] / 12 = 11.0 / 12 = 0.917
Dimension breakdown: exception_equiv failed twice, log_equiv once, timing_equiv once, sideeffect_equiv once. Exception handling is the most common violation: consistent with industry patterns where AI tools consolidate catch blocks, broadening exception types in ways that change error propagation behavior.
Key Distinction:
Only PRs labeled as "pure refactoring" are measured. Intentional behavioral improvements approved by reviewers as part of a refactoring PR are excluded from scoring.
Confidence Grading (required alongside every RSFS value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ observed events in measurement window provide direct tail observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; derived entirely from Monte Carlo simulation | Modeled, not observed; treat as indicative, not validated |
A RSFS of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.