AI: LLMs, providers, the platform's AI and your own

How a language model reads and writes tokens and what each call costs, how the provider APIs differ and how one catalog hides them, how the platform's AI and your own key share the work and in what order, and how cost, caching, streaming and structured output are done safely - step by step, with pictures.

In short

A large language model (LLM) reads text as tokens and writes the most likely next ones, one at a time. {{brand}} does not train models; it calls them over plain HTTP, from five kinds of provider, through one small layer with just two clients. Two kinds of key can answer: the platform's AI, which {{brand}} pays for, and your own key - or your company's - which you pay the vendor for directly. Every call is checked, budgeted before it is made, and streamed back while it is written.

After this lesson you can Explain tokens, the context window and temperature, and price a model call by hand; talk to OpenAI, Anthropic and a local Ollama server with plain HTTP; design a bring-your-own-key system with a clear order of whose key answers; and reserve, cache, stream and structure model answers safely.

Words you need

How an LLM answers, in practice

You do not need the mathematics to use a model well. You need five facts: it reads tokens, it sees only one window, it writes one token at a time, randomness is a dial, and you pay for every token in both directions.

Reasoning models are different Newer reasoning models think in hidden tokens before they answer - you pay for those too - and some accept no temperature except their default. A robust client sends a temperature only when it may, and when a provider refuses one it asks once more without it.

Limits and failure modes

A model call can fail loudly, or succeed and still be wrong. Both are predictable. Know the limits before you design around a model.

What a call costs - a worked example

The formula is always the same: input tokens x input price + output tokens x output price. Prices are quoted per million tokens, so $0.15 per million is $0.00000015 per token. The prices below are examples for illustration; real ones change often, so look them up on the vendor's page.

Provider APIs: two dialects of one idea

Every vendor offers the same idea - send a list of messages, get text and token counts back - but the details on the wire differ. Two formats cover almost the whole market: OpenAI's chat completions, which most other vendors and open-source servers copied, and Anthropic's messages.

The provider catalog: differences as data

The classic design is the adapter pattern: the rest of the code talks to one small interface - complete a request, or stream it - and one adapter exists per wire format. Everything else that differs between vendors lives in a catalog of rows: the default address, the path, how the key is presented, the default model, the name of the max-tokens field and whether an address is required. {{brand}} has five provider rows and two adapters.

No built-in list of models Vendors release models faster than anyone updates a dropdown, and a stale list stops people using the model they pay for. The model is a free-text field with a sensible default, and the catalog never lists models.

The options in the market

What {{brand}} chose, and why

{{brand}} writes its own two clients on plain HTTP and keeps the vendor differences in a catalog. It also runs two AIs side by side: the platform's, which it pays for and budgets, and yours, which you pay for. The reasons:

The trade-off accepted When a vendor ships a new API feature, {{brand}} can use it only after someone writes the code for it. In exchange there is no SDK upgrade treadmill, and every vendor behaves the same inside the product - the same retries, the same budget, the same checks.

When another option is right: a product built around agents with memory and many tools gains from a framework; a company running dozens of models for many teams gains from a gateway; a product whose data may never leave the building should self-host.

Bring your own key: the design

In a multi-tenant product three parties can own the AI account: the platform (the operator), a tenant (an organization) and a user. All of them do the same work through the same code. Only who pays, what is limited and what may be reused differ.

Every AI call - a question, a report summary, a drafted card - asks one resolver whose key answers. It walks the same order every time and stops at the first that may answer:

Checked twice: defense in depth The resolver decides, and then a gate that every model call must pass checks the same rules again just before the request leaves. A new feature that forgot to ask the resolver still cannot send a tenant's data with a key the tenant ruled out. Rules enforced in one choke point cannot be skipped by accident.

Why answers from your own key are never shared

In any multi-tenant product that caches model answers, only answers the operator's own model wrote should be kept for other people. An answer written with a person's own key, or with an organization's key, is shown to the person who asked and then forgotten. The main reason is an attack called cache poisoning.

Policy at one choke point

A multi-tenant AI product usually lets each tenant's administrators decide what AI may do with their data. The policy is enforced at the one place every model call passes, whatever made the call - a question, a summary or a draft.

One choke point Policies written as checks inside each feature always leak somewhere. Put them where every call must pass - the same place that records usage and holds the budget - and a new feature obeys them on its first day.

Cost control: reserve the maximum before the call

The danger with AI cost is not one expensive call; it is many calls at once. Ten questions arrive together, each checks that the budget has room, each sees room, and together they overspend. The fix is the one airlines use for seats: reserve first, settle later.

Count money in micro-dollars Store money as integers of a millionth of a dollar. A price per million tokens is then exactly the micro-dollars of one token, sums never drift the way floating point does, and the rounding rule - always up - is written down in one place.

Caching answers safely

An answer that already exists costs nothing. The hard part is reusing it only when it would be the same answer for this person. {{brand}} always runs retrieval first - cheap database work, done with the asker's own permissions - and skips only the model call.

Why hashing the retrieved pieces is the trick If a card is added or edited, or the asker loses access to one, their retrieval returns different pieces, so the key changes and the old answer is simply never found. Permissions are enforced by the key itself - nothing has to remember to clear the cache.

Streaming with server-sent events

A long answer takes many seconds to write. Streaming sends it piece by piece, so the reader starts at once. Vendors stream to {{brand}} with server-sent events, and {{brand}} streams to the browser the same way: one ordinary POST request whose response stays open and carries events.

Why not the browser's EventSource? The built-in EventSource makes only GET requests and cannot send a body or custom headers. A question with its history needs a POST, so the page reads the stream with fetch and a small parser - which must cope with an event cut in half between two network chunks.

Structured output: JSON you can trust

Some features need data, not prose: a list of acceptance criteria, a drafted vacancy, a set of design changes. Asking politely for JSON works most of the time - and breaks in production. There are four techniques, from weakest to strongest, and {{brand}} uses the vendor's native one on each wire format.

With the forced tool call, the answer comes back as a content block of type tool_use whose input is the JSON object. With response_format, the message content itself is the JSON text.

Never stream a structured answer Half a JSON object is worth nothing. A request with a schema is answered whole, even when the caller asked for a stream.

Try it yourself: a model on your own computer

You can try every idea in this lesson for free on your own computer with Ollama, which serves open models behind an OpenAI-compatible API.

On Windows PowerShell and cmd, the quotes in the curl command above break; use the Node script below instead. It needs Node 18 or newer: save it as try.mjs and run node try.mjs.

Common mistakes

Interview questions

Related