Python SDK

pip install throughputs

Initialize

from throughputs import Throughputs

client = Throughputs(api_key="thp_live_xxx")

The base_url defaults to https://api.throughputs.dev/v1. Override it for testing or self-hosted deployments:

client = Throughputs(
    api_key="thp_live_xxx",
    base_url="https://api.throughputs.dev/v1",
)

Chat completions

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

print(response.choices[0].message.content)
print(response.throughputs_cost_usd)   # first-party field

Streaming

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Count to 5"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Fallback chains

If the primary model is down or slow, THROUGHPUTS can fall back to backups you specify:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    model_fallback=["claude-3-5-sonnet", "llama-3.1-70b"],
)

The routing layer tries each in order. The response's throughputs_model_served field tells you which one actually answered.

Async

import asyncio
from throughputs import AsyncThroughputs

async def main():
    client = AsyncThroughputs(api_key="thp_live_xxx")
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())

Error handling

from throughputs import ThroughputsError, RateLimitError, AuthenticationError

try:
    response = client.chat.completions.create(...)
except AuthenticationError:
    # invalid or revoked key
    ...
except RateLimitError as e:
    # 429 — back off and retry
    wait = e.retry_after
    ...
except ThroughputsError as e:
    # anything else
    print(e.status_code, e.message)