Success looks like disuse

Read a retention curve for a wellbeing app and you are reading an accusation. Median fifteen-day retention across mental health apps was 3.9 percent when Amit Baumel and colleagues measured real usage panels for a 2019 paper in the Journal of Medical Internet Research, and thirty-day retention was 3.3 percent. The field received those figures the way any field receives retention that low, as evidence of products that do not work. Grant the reading its due, because for most software it is correct and the alternative sounds like an excuse. Then notice the one thing the number physically cannot do. A person who deleted the app because it was useless and a person who stopped opening it because the practice had become theirs are the same row in the panel.

That confusion is not a limitation of the measurement. It is written into the definition of the thing being measured. When David Wood, Jerome Bruner and Gail Ross introduced scaffolding in the Journal of Child Psychology and Psychiatry in 1976, they defined it as a process enabling a novice to solve a problem beyond their unassisted efforts, and among the six tutoring functions they listed was frustration control, qualified on the spot with a warning that relieving frustration must not create dependence on the tutor. Removal was on the first page. A generation later, reviewing a decade of empirical work in Educational Psychology Review, Janneke van de Pol, Monique Volman and Jos Beishuizen distilled the construct to three characteristics, contingency, fading, and transfer of responsibility. Two of the three describe the support going away. Scaffolding, built on Vygotsky’s zone of proximal development, is not a helping relationship that happens to end. It is a helping relationship defined by its ending, so a product built on it succeeds by becoming unnecessary, and every usage number it reports moves down at exactly the moment it works.

Which puts the sign of the metric at the center of the design rather than at the edge of it. An engagement number has one direction of goodness and it is up, and no amount of care in choosing which engagement to count repairs that, because the repair required is a minus sign. Our economics writing has already argued the strategic half, that restraint of this kind is uncopyable by any competitor whose reporting structure runs on the same numbers. The measurement half is narrower and harder to dodge. If the tool’s job is to build capacity in a person, graduation is the positive outcome, and a metric set with no field for graduation is a metric set in which the best thing the product can do to someone is unrecordable.

Building the field is ordinary work once the sign question is settled. Count the people who reached a defined competence and left, separately from the people who simply left. Ask the departed which they were, and accept that the response rate will be poor and the number noisy, because a noisy measurement of the right quantity beats a precise measurement of its opposite. Follow the unassisted case the way the deskilling research grades a student after the model is taken away, and let the follow-up rather than the session be the thing the product is priced on.

None of this makes the retention curve dishonest. It makes it a measurement of dependency, reported in the slot where capability was supposed to go. A company willing to publish its graduation number beside its retention number has said, in the only language its own board reads, which of the two it is actually building, and that is a sentence a customer can hold it to for as long as the product exists.