Consistency is not fidelity

Our own competitive research proposes a pitch we are going to argue against, and we are recording it rather than quietly dropping it. Franchise your methodology without franchisees, it says, with zero-marginal-cost replication and zero fidelity drift, on the reasoning that human operators introduce cognitive bias, emotional fatigue, personal interpretation, and deviation, while a machine applies the method with absolute consistency across unlimited simultaneous deployments. The argument is coherent and it is aimed at a real wound, since fidelity variance is the franchise model’s most expensive problem. We think the conclusion is inverted, and the discipline this corpus keeps is to leave the losing argument on the page with its reasoning attached, because that is the only way anyone later can check which of us was right.

Start with the half of it that holds. Some human variance is decay rather than judgment. An implementer who is tired, who glosses over dissent because the meeting is running long, who has quietly swapped the method’s harder move for one they find easier, is not exercising discrimination. They are drifting, and a licensor who wants that gone is right to want it gone. Any honest account of this has to keep the two apart.

Here is where they separate. Fidelity in an adaptive method is agreement with what the method would have decided here, while consistency is agreement with what it decided last time, and only the second of those can be maximized by a system that is not deciding anything. The four dimensions where judgment lives are all conditional on the person present. Timing responds to the state of the room, interpretation responds to what this particular behavior means for this particular history, the branch you take depends on which of a dozen resistances you are looking at. A delivery that is identical every time has not achieved high fidelity on those dimensions. It has scored a perfect zero on all four and been graded on a different exam.

The evaluation literature worked this out for complex interventions two decades ago. Hawe, Shiell and Riley argued in the BMJ in 2004 that requiring an intervention to look the same everywhere may be part of why some community interventions produce weak effects, and proposed that what is standardized should be the steps in the change process and the key functions the components are meant to serve rather than the components themselves. Integrity defined functionally, rather than compositionally, is the key, in their phrase. Carroll and colleagues built the same understanding into measurement three years later, assessing fidelity as adherence to content and dose moderated by quality of delivery and participant responsiveness, which is a way of saying that the field already knows a protocol can be followed exactly and delivered badly.

The educators reached it from the other direction and further. The Reggio Emilia approach is described by its own alliance as not a method and not a pre-established curriculum, and no certifications in it are approved, precisely because the practice is supposed to be constructed around the children who are actually in the room. A version of it that could be shipped identically to a thousand classrooms would not be a scaled version of the approach. It would be a different thing wearing the name, which is the same failure label drift produces, arriving this time through the front door with a fidelity guarantee attached.

None of which means machine delivery cannot be faithful. It means fidelity is unclaimable until somebody states what the method is for, at the level of function rather than form, because that statement is the only thing a deviation can be judged against. Do that and the interesting question becomes answerable. Did this departure from the script serve the function the script exists to serve? A system that never departs cannot pass that test, and it cannot fail it either, which is the more damning result. Ask any method’s owner to write down its functions and you will learn a great deal in the first ten minutes, most of it about how much of their method they have never had to say out loud, and every one of those sentences is a piece of the remainder coming into the light.