Workers AI

Workers AI is an inference API attached to the runtime the code is already running in. There is no separate vendor account and no API key in a secret store: a binding appears on the environment object and you call it with a model ID and an input. The catalogue is open-weight models, mostly small ones, with Llama 3.1 8B Instruct as the general text model, plus embedding models, image classification and description, and a moderation model. Billing is per unit of compute rather than per token, and requests run in the same data centre as the rest of the request.

How we use it

Objectify exposes it as product surface rather than as a hidden feature: endpoints for embeddings, indexing, vector search, chat and generation, moderation and image description, all gated to the two paid tiers, because inference is the one cost in that product that rises with use rather than sitting flat. Jonosakti uses four endpoints on Llama 3.1 8B Instruct, summarising a submission, suggesting a category for it, running a moderation check and helping someone write. In @Heartbeat it is a moderation pass over post and comment text at creation, and it fails open: if the binding is absent, the post goes through unchecked.

That last detail is the pattern we reuse. Anything in a write path is optional and has a defined behaviour when the model is unavailable, because otherwise a user's ability to publish depends on capacity in someone else's inference pool.

What it costs you

An 8B instruct model is good at short summaries, category suggestions and coarse moderation calls. It is not good at structured extraction, long context or anything where a wrong answer is expensive. Ask for JSON and you will get JSON most of the time, so the output goes through a schema check before it touches the database, which is one of the reasons every one of these builds carries runtime validation.

Model IDs are deprecated and the catalogue changes under you, so the ID belongs in config, not scattered through the code. Latency is a few hundred milliseconds on a good day and capacity errors are real, which means a timeout and a fallback on every call in a request path. The pricing unit does not map onto anything you can estimate in advance from a design document; we worked out the running costs of a social timeline with real traffic shapes rather than guessing, and moderation on every post is the line item that moves.

One operational trap worth naming: in @Heartbeat the AI binding is configured in the dashboard rather than in the deployment config, so nothing in the repository tells you whether moderation is actually running in production. Bindings belong in the config file that ships with the code, where a diff can show them.

When we would choose something else

If the model output is the product and accuracy is measurable, we go to a frontier hosted model instead: OpenAI models read a photograph of a meal and return macros in 360nutri, and no 8B model does that job. If the aim is retrieval rather than generation, embeddings are only half of it and you need an index to query, which is vector search. And if the requirement is really keyword matching over your own rows, a full-text index in SQLite is cheaper, faster and explainable to a customer who asks why a result appeared.