Can the AI be trusted, and is it worth it?
Two questions anyone paying for this will ask. The answers below come from a method the real pilot will repeat on a customer's own records.
Shadow mode: the AI against an expert, on past records
In shadow mode the AI grades records alongside the people who grade them today, and nothing it says is acted on. The two are compared. That comparison is the evidence that decides whether the AI is allowed to clear anything on its own.
Where they agree and disagree
| 1 | 2 | 3 | 4 | |
|---|---|---|---|---|
| 1 | 89 | 4 | 0 | 0 |
| 2 | 0 | 77 | 4 | 0 |
| 3 | 0 | 1 | 43 | 1 |
| 4 | 0 | 0 | 2 | 19 |
The diagonal is agreement. Above it the AI was more cautious than the expert (9 cases). Below it the AI under-called the risk (3 cases), which is the kind that matters most.
Agreement by process stage
- Research & Development100% of 21
- Manufacturing98% of 86
- Packaging90% of 39
- Distribution94% of 35
- Quality System93% of 59
A stage that agrees less is where the pilot should look first and where the rules should stay strictest.
The disagreements, worst first
The platform is judged by its misses, not its average. A safe system keeps the dangerous ones rare, and the rules catch most of the rest.
| Case | Stage | Expert | AI | AI confidence | What happened |
|---|---|---|---|---|---|
| HIST-0106 | Manufacturing | C3 | B2 | 83% | False clear AI under-called the risk and it was cleared without a human: a false clear |
| HIST-0113 | Quality System | D4 | C3 | 65% | Caught by the rules AI under-called the risk, but the rules still sent it to a human |
| HIST-0204 | Packaging | D4 | C3 | 64% | Caught by the rules AI under-called the risk, but the rules still sent it to a human |
| HIST-0036 | Packaging | B2 | C3 | 57% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0095 | Packaging | C3 | D4 | 64% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0158 | Quality System | A1 | B2 | 69% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0185 | Quality System | C2 | D3 | 76% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0187 | Distribution | B2 | C3 | 58% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0207 | Quality System | B1 | C2 | 74% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0216 | Manufacturing | B2 | C3 | 61% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0220 | Distribution | A1 | B2 | 72% | Over-cautious AI more cautious: it asked for more review than the expert did |
| HIST-0232 | Packaging | A1 | B2 | 73% | Over-cautious AI more cautious: it asked for more review than the expert did |
What it is worth
Move the assumptions to match a real team. The one measured input, the share the AI handles alone, starts from this live demo (50% of the 14 records currently in the system).
A model, not a measurement. It multiplies the inputs on the left, nothing more, and every input is an assumption to be replaced with measured values in the pilot.