The evidence trap

The comfortable version of this argument says that evidence eventually wins, that a product which submits to measurement outlasts the ones that decline. We would like to carry that version. The record does not support it, and the clearest counter-case is recent. On June 30, 2025, Woebot Health shut down its direct-to-consumer application. The company had been founded in 2017 by a clinical research psychologist, had raised roughly $123 million, and in 2021 had earned an FDA Breakthrough Device Designation for WB001, its therapeutic for postpartum depression. Its founder told STAT that the shutdown was largely attributable to the cost and challenge of meeting the agency’s requirements for marketing authorization. One of the few products in the category to have earned a federal breakthrough designation left the consumer market, and the products that had never gathered evidence at all stayed.

The trap is built into where the bar sits. Medical-device regulation attaches to a product’s intended use, which means the evidence bar is triggered by what a product holds itself out to do rather than by the mechanism it uses. Two applications can run identical logic and behave identically on a phone, and only the one that says out loud what it is for becomes a device. Industry figures put a De Novo classification at a median around $5 million and roughly five and a half years from concept to decision. So the honest company pays that, and the company that implies the same benefit while claiming nothing pays none of it, and both compete for a user who cannot see the difference. The money spent on proof transfers, in competitive effect, to whoever declined to gather any.

Then the second tax arrives, and almost nobody budgets for it. Validation requires reproducible output, so a product built to pass it is constrained to determinism. Woebot ran on pre-scripted rules-based therapy content written by clinicians, which was not a failure of ambition but the price of admission to the pathway it was on. When fluent generators arrived, those scripts read as archaic beside them, and the founder was explicit that the company wanted to use large language models and that the agency had not yet worked out how to regulate them. The discipline that earned the evidence is the same discipline that made the product feel old. Punished at the treasury, then punished again at the interface, and neither penalty had anything to do with whether the thing worked.

The market underneath all of this is thin to begin with. Across 93 mental health applications, one systematic analysis found a median daily open rate of 4.0 percent and median 30-day retention of 3.3 percent. Into that, a regulated product carries device overhead its unregulated neighbor does not. Where the lines have been drawn since, they fall in the same place rather than a better one. Utah’s H.B. 452, in effect since May 2025, defines a regulated mental health chatbot in terms of generative technology used in conversations a reasonable person would construe as therapy, and excludes tools that deliver scripted output or hand a person to a human. Read as a fact about how regulation works rather than as a map of where to hide, it says what the federal regime says. The trigger is the claim.

This is the case where restraint lost, and a corpus that only publishes its wins is running the same selective measurement it objects to elsewhere. The conclusion is not that proving things is a mistake. It is that proof is priced in two currencies, money and datedness, and a builder who budgets only the first will discover the second at the worst possible moment. Know which taxes you have agreed to pay before you agree to them. And notice what the trap actually is, because the naming decides what can be done about it. Nothing here is a property of evidence. It is a property of a market wired so that saying less costs less, and wiring is the kind of thing that gets rebuilt.