What we learned shipping an AI feature to 40,000 users

In the spring we launched an AI drafting assistant inside our customer-support product. Six weeks later it was used by 40,000 agents across 1,200 companies. This is what we learned, in roughly the order it hurt.

Roll out by percentage, not by date

We launched to 1% of accounts, then 5%, 25% and 100% over three weeks. Each step uncovered something: a timeout in one region, a language we had not tested, an enterprise customer whose admin had disabled third-party processing. Every one of those would have been an incident at 100%.

Users edit, and edits are data

The most valuable signal was not thumbs-up or thumbs-down. It was the diff between the draft we suggested and the reply the agent actually sent. We measured edit distance per draft. Drafts sent with minor edits meant we were helping; drafts deleted entirely meant we were not.

Latency matters more than we thought

Our first version took about six seconds to produce a full draft. Usage was flat. Streaming the response so the first words appeared within 700 milliseconds roughly doubled adoption, with no change to the model or prompt.

Guardrails need an owner

We had filters for personal data and policy violations, but nobody owned tuning them. False positives annoyed users for a week before anyone noticed. Now one engineer owns guardrail metrics on a weekly rotation.