The queue

A job moves through the queue in these steps:

1. Enqueue. A feature adds a job with a kind, a payload and a deduplication key. If a job with that key is already waiting, the feature gets that job back instead of a second one being added. The database raises a notice, aiproc_enqueued, and every registered webhook endpoint, kept in processing.webhook_endpoints, is sent a job.enqueued delivery. A job someone is waiting on is marked interactive, with a priority of 100.
2. Claim. A node asks for jobs, saying how many it wants and the lowest priority it accepts. The backend picks waiting jobs in a way that lets several nodes ask at once without taking the same job, marks each one claimed, and returns it with a lease of 30 seconds and a lease token. The lease token is the node's proof of the claim on every later call about that job.
3. Heartbeat. While it works, the node extends its lease.
4. Complete. The node sends the result. The backend stores it and runs the result appliers for that kind of job, inside the job's own tenancy. The database raises aiproc_job_done, which wakes anyone following the job's stream.
5. Fail. The node reports an error. The backend records it and puts the job back in the queue to try again after a wait, or, once the attempt limit is reached, marks it dead.
6. Reap. A background reaper returns any job whose lease has run out to the queue.

A program adds a job through the AI processing run call, which answers at once with the job's id and whether it was a duplicate. The program then follows the job on its stream, and falls back to polling if the stream is not available.