The AI processing node

Some of what the hub does with a model takes too long to do while you wait. When you save a note, something has to read it and decide its category and tags. When a feature asks for a batch of model work, or a conversation needs an answer, something has to run the model. The AI processing node is the worker that does that, in the background, so the page you are on answers at once and the result arrives when it is ready.

Description

The node runs wherever a model runs, on a developer's own machine or on a server.

It holds a worker token. With that token it talks to the backend over the REST API, the same way any other program does, and the backend keeps a queue of jobs for it. It takes a job, runs it, and hands the result back.

Topics

The queue: where a job waits, and what happens to it from the moment it is added to the moment it is done.
The node: the worker itself and its configuration, and how it picks up and runs a job.
The worker token: the key the node holds, and how it borrows just enough authority to work a job.
As deployed: how a node runs on a machine, and what happens when there is more than one.

Tech Detail

APIs & protocols: REST

The node is a client, not a part of the backend. It holds a token, calls the REST API and runs a model. It has no connection to the database and no secret beyond its own token and its webhook secret, which is why it can run on a laptop.

The lease is the authority. The lease token proves the claim, the acting token lives exactly as long as the lease, and a lease that runs out returns the job to the queue. A node cannot hold a job, or the authority to work it, by accident.

Push to wake, poll to be sure. The webhook wakes the interactive loop, so an urgent job starts at once. The poll makes sure a job is never missed when a delivery is.

Duplicates are caught at the door. Saving a note three times puts one tagging job in the queue, not three.

The backend applies the result. The node returns data. The backend's applier writes it, inside the job's own tenancy. The node never writes a table.