Testing an AI Feature Without a User

To test a dictation feature by hand, you have to talk. Out loud, at your desk, again after every change, trying to say the same sentence the same way. To test a video capture feature, you have to find a video, play it, and wait for it. To test a conversational tool, you have to hold a conversation.

Features with a human in the loop have a test cycle measured in minutes. A test cycle measured in minutes is a test cycle you quietly stop running.

The cost is not the minutes

The minutes are the visible part. What actually hurts is what the slow loop does to your coverage.

You only ever test the happy path, because setting up the awkward case is more work than the change you were testing. You test in one environment, yours, on one device, with one voice and one accent and one room. And because the input is never identical twice, you cannot tell whether an output changed because your code changed or because you spoke differently. Every comparison is contaminated. That is the real damage: not slowness, but the loss of the ability to say "this is better than it was."

Feed a recording through the real pipeline

The fix is to replay a fixture, and the load-bearing word is real. The goal is not to mock out the model and assert that your glue code calls it. It is to hand recorded input to the actual pipeline and let everything downstream behave exactly as it would in production.

So inject as early as you can, at the first point after the hardware. Not "call the transcriber directly with this file," but "make the audio source produce these samples." The capture state machine, the chunking, the buffering, the cancel path, the handoff to the cleanup step, the clipboard logic: all of it should run, and none of it should know it is a test.

That placement matters because of where the bugs actually live. In our experience, very few failures in this kind of feature are the model being wrong. They are a race between stopping the recording and flushing the last chunk, a cancel that leaves a device handle open, a state machine that gets stuck when a step returns empty, a result that arrives after the window it belonged to has closed. A test that starts below those seams will pass on code that is broken.

Assert on shape, not on exact text

Do not assert that the transcript equals a string. Model output varies, and even a fixed local model changes across versions. A test that asserts exact text fails for reasons you do not care about, and a test that cries wolf is a test someone deletes.

Assert on properties that must hold for any correct output:

• Presence and plausible size. Non-empty, and of a length that makes sense for the duration of the input. A three-word transcript from a two-minute recording is a failure regardless of which three words they are.
• Anchor terms. Pick a handful of distinctive words that any correct transcription of that fixture must contain, and check for those. Proper nouns and numbers make good anchors. Common words do not.
• Format validity. If a step returns structured output, it parses, the required fields exist, and unexpected fields do not.
• Transformation invariants. For a cleanup step, the output should keep the anchors and should not grow substantially longer than its input. A step that invents content shows up immediately in that second check.
• Timing against the budget you claimed. If you told users this feels instant, assert the number that makes it instant, and let the test fail when a change makes it slow.
• The state machine and the cleanup. The pipeline passed through the expected states in order, released the device, and left no temporary file behind.

These assertions are weaker than string equality, which is precisely why they survive. They fail when something is broken and stay quiet when something merely changed.

Keep a small set of known-good cases

Five to fifteen fixtures, no more, each one earning its place. A good starting set covers a clean recording, a noisy one, a very short one, an empty or silent one, a long one, one in another language if you claim to support other languages, and one whose content contains something that looks like an instruction, so you can watch a transformation step treat it as text.

After that, the set should grow by bug. When something breaks in the wild, the smallest input that reproduces it becomes a fixture and stays forever. That is how a suite becomes an actual description of what your feature has to handle, rather than a set of cases someone imagined on a Tuesday.

Keep it small deliberately. A suite that runs in under a minute gets run before every commit. A suite that takes fifteen minutes gets run before releases, which means it finds problems a week after they were introduced.

A hidden developer flag beats clicking through the app

Put the replay path in the shipping application, behind a flag: an environment variable, a hidden menu item revealed with a modifier key, a debug pane that is not reachable from normal navigation. Pick a fixture, run it, watch the real interface respond.

This beats a standalone test harness for reasons that show up the first week. It exercises the real build with the real wiring and the real permission state, so it catches the packaging and entitlement problems a unit test never sees. It works on a teammate's machine when they cannot reproduce your bug, and on a customer's machine when you are diagnosing something and cannot ask them to keep talking into a microphone while you watch. And it costs almost nothing to maintain, because it is a few lines in front of the input source rather than a parallel copy of your app.

Guard it properly. Default off, unreachable from the normal interface, and the fixture path should be a local file rather than anything that can be pointed at a remote address. A debug hook that loads arbitrary input from the network is a feature you did not mean to ship.

What this does not catch

Be honest with yourself about the gap, because the failure mode of a good harness is trusting it too far.

Replay does not test the microphone. It does not test permission dialogs, the moment the operating system asks for screen or audio access and your app has to handle the denial gracefully. It does not test driver quirks, bluetooth headsets that resample, system audio routing when another app is playing, thermal behavior during a long real session, or the acoustic reality of a bad room. Tools that capture the screen also run into content that is protected from capture, which no fixture will reproduce.

Those still need a real device and a real person. The point of the harness is not to eliminate that pass. It is to change its frequency from every change to every release, and to make sure that when you do sit down with real hardware, you are looking for the things only real hardware can show you. Our capture tool Reel is a good example of the division: the transcript and insight path is fixture-testable end to end, while browser detection and recording permissions are not, and pretending otherwise would just move the surprises later.

The takeaway

If a feature needs a human to test it, the highest-value thing you can build next is the path that removes the human. Replay recorded input through the real pipeline, assert on the shape of the output instead of its exact text, keep a small fixture set that grows one bug at a time, and hide the trigger behind a developer flag in the shipping build.

The version of this work that fails is the one that tests a stub instead of the product. If your harness can pass while the app is broken, you have not built a test. You have built a second thing to maintain.