Teaching a Window to Do the Work

Somewhere in your week there is a piece of software with no API. A county records portal. A supplier ordering site from 2009. A desktop application your industry standardized on before anyone thought about integrations. You open it, click the same six things, type the same values, and close it.

That is the long tail, and it is where window automation earns its keep. Relay, our Mac tool, attaches to a browser or desktop app window and either works through a task you describe or replays a workflow you demonstrated once. It is genuinely useful, and it is genuinely brittle, and knowing which is which is the whole skill.

Why the long tail never gets an API

Integrations exist where there is commercial reason for them to exist. A vendor builds an API when enough customers ask, when a partner pays, or when their own mobile app needs one. Plenty of software fails all three tests. It serves a narrow trade, it has no competition to speak of, and the roadmap has been maintenance for years.

For that software, the user interface is the only interface. Which means automating it means driving the interface: reading what is on screen, clicking, typing, waiting. This is not a new idea, but doing it without a scripting engineer and a fragile selector file is.

Two ways to hand off a window

Relay gives you two modes, and they fail in different ways, so pick deliberately.

• Describe the task. You attach a window and say what you want done. The model looks at the screen and decides what to do next. This tolerates a surprising amount of variation, because it is reading the interface rather than following coordinates.
• Watch and learn. You demonstrate the workflow once. Relay captures supported clicks, typing, scrolling, and the intervals between actions. You review the recording, then save it as a routine you can run again.

The mental model worth holding: a description is flexible and less predictable, a recording is predictable and less flexible. Repetitive work with a stable screen wants a recording. Work where the screen shifts, or where a judgment call is required partway through, wants a description.

What a recording actually captures

A demonstration is a literal artifact. It is not a summary of your intent, it is the inputs you made and the timing between them. Three practical consequences follow.

First, start from a clean state. If you began the recording three clicks into a flow, the routine assumes those three clicks happened. Relay reads the attached window and prepares a saved routine's starting screen, and checks the live controls as it goes, but you are still better off recording from a place you can reliably return to.

Second, review the recording before you save it. Half of the bad routines we have seen were bad the moment they were captured: a stray click, a field filled in the wrong order, a step performed twice. Catching that at review is cheap. Catching it on the tenth run is not.

Third, know what is deliberately not captured. Protected fields are excluded, and an incomplete recording can require another demonstration. That is a feature, not an inconvenience, but it means some flows will always need you present for one step.

Where it gets brittle

Be clear eyed about this. Driving a user interface means depending on things that were never promised to stay the same.

• Layout changes. A vendor ships a redesign, moves a button, adds a consent banner, and a recorded routine walks into a wall. Timed replay has no opinion about what it is looking at.
• Timing. Recorded intervals worked on the network you had that day. A slow morning, a heavier page, a spinner that lasts a beat longer, and steps land before the screen is ready.
• What the app exposes. Support depends on the controls and content each application actually exposes to the system. A well behaved app with real, labeled controls automates far better than one that draws its own widgets into a canvas.
• The machine has to be there. Your Mac must stay awake and Relay must be running. This is desktop automation, not a server job.

The mitigation is not cleverness, it is expectation. Routines are maintenance items. Relay lets you delete old ones or record a replacement, and you should expect to do that occasionally, the same way you expect to re-do a spreadsheet formula when a report format changes.

Where we would not point it

Our line is simple, and we hold it even when it is inconvenient: automate the gathering, keep the human on the commit.

Anything that moves money, anything that is destructive, and anything that cannot be reversed with a normal undo should end with a person pressing the button. Submitting a payment. Approving an invoice. Deleting records. Sending something to a customer. The reason is not that the automation is untrustworthy in general, it is that these are exactly the cases where a single misread screen is expensive and unrecoverable, and where the savings from automating the last click are trivial compared to everything before it.

Good candidates look like the opposite: pulling the same report every Monday, copying values from one system into a form in another, checking a portal for a status change, preparing a record right up to the point of submission. Tedious, frequent, and safe to get wrong once.

The permissions and the tradeoff

Window automation requires real access. Relay uses Screen Recording and Accessibility permissions, which is what lets it read and control the window you attach. There is no lighter version of that. Anything that can see and click your screen has to be allowed to see and click your screen.

Where the work goes matters too. AI guided tasks send screenshots and task context to Claude through Juracich Path. Saved routines stay on your Mac, and timed replay uses the recorded inputs and intervals, though preparing the starting screen may still use AI. If you are automating a window with sensitive material in it, that is the sentence to read twice.

While a task runs, you can follow each action in a live activity view with timestamps, expand details, filter for issues, copy the log, and stop it. We would rather you watch the first several runs of anything new than trust a routine you have never seen execute.

The honest verdict

Window automation is a workaround, and workarounds deserve respect. If a real API exists, use the API. It will be faster, quieter, and it will not break because someone moved a button. Relay is for the case where no such option exists and the alternative is a person doing the same clicks forever.

Judged against that alternative, it holds up well. Judged as a replacement for integration work, it does not. Pick the tasks where a wrong click is cheap, demonstrate them carefully, review the recording, and keep your hand on anything that spends money.