Streaming

Set stream: true on a chat completion and THROUGHPUTS returns a stream of incremental chunks via Server-Sent Events (SSE). Use it for chat UIs, typewriter effects, and any path where time-to-first-token matters.

How it works

Instead of waiting for the full response, THROUGHPUTS sends chunks as the model generates them. Each chunk is a data: line containing a partial chat.completion.chunk object. The stream ends with data: [DONE].

Raw SSE example

curl https://api.throughputs.dev/v1/chat/completions \
  -H "Authorization: Bearer $THROUGHPUTS_API_KEY" \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Count to 5."}],
    "stream": true
  }'

data: {"id":"...","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"...","choices":[{"delta":{"content":"1"},"index":0}]}

data: {"id":"...","choices":[{"delta":{"content":"2"},"index":0}]}

...

data: [DONE]

With the OpenAI SDK

The SDK handles SSE parsing for you:

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Count to 5."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

const stream = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Count to 5." }],
  stream: true,
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
}

Why stream

Lower time-to-first-token. First chunk lands in 100–400ms instead of waiting for the full response.
Better UX for chat. Users see progress immediately.
Same cost. You pay per token regardless of streaming — there's no premium.

Cancellation

Cancel a stream by closing the connection from the client. You're billed only for tokens generated before cancellation.

If you cancel mid-stream, partial output is still yours to keep — but the upstream provider may have generated more tokens than you received. You are billed for X-Throughputs-Tokens-Out reported in the final chunk's trailing event, not for tokens you actually displayed.

Error events

If an error occurs mid-stream, THROUGHPUTS sends a final chunk with an error field before closing the connection:

data: {"error":{"type":"upstream_error","message":"Provider timeout"}}

Reconnect with backoff. The OpenAI SDK does this automatically.