Common errors & how to fix them

Every failure the SDK can surface maps to one stable shape. The core formats a human-readable message, and on a 403 it also carries a machine-readable code (wrong_credential_type, project_scope_mismatch, scope_insufficient) so you can branch without parsing prose.

How that shape reaches your code depends on the language:

Python

Errors are typed exceptions, all subclassing InferenceKeyError:

from inferencekey.errors import (
    InferenceKeyError,   # base class for every SDK error
    PermissionDenied,    # 403 — wrong_credential_type / project_scope_mismatch / scope_insufficient
    AuthError,           # 401 — missing or invalid credentials
    ValidationError,     # a request argument failed validation (e.g. empty required field)
    ConfigurationError,  # client-side misconfiguration before any request
    ApiError,            # any other non-2xx response or transport failure
)

Catch the specific type, fall back to the base:

from inferencekey import ManagementClient, WorkloadSpec, Backend
from inferencekey.errors import PermissionDenied, ConfigurationError, InferenceKeyError

mgmt = ManagementClient.from_env(project="acme")
try:
    ref = mgmt.ensure(WorkloadSpec(
        name="support-bot", slug="support-bot",
        model="meta-llama/Llama-3.1-8B-Instruct", backend=Backend.VLLM,
        command="vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192",
    ))
except ConfigurationError as e:
    ...  # fix your setup; no request was made
except PermissionDenied as e:
    print(e)  # message includes the code, e.g. "permission denied [scope_insufficient]: ..."
except InferenceKeyError as e:
    ...  # anything else

TypeScript

The Node addon raises a plain Error whose message carries the core’s formatted string — including the bracketed code on a 403. Branch on the message:

import { ManagementClient, WorkloadSpec, Backend } from "@inferencekey/sdk";

const mgmt = ManagementClient.fromEnv({ project: "acme" });
try {
  const ref = await mgmt.ensure({
    name: "support-bot", slug: "support-bot",
    model: "meta-llama/Llama-3.1-8B-Instruct", backend: Backend.Vllm,
    command: "vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192",
  });
} catch (e) {
  const msg = (e as Error).message;
  if (msg.includes("[wrong_credential_type]")) { /* swapped tokens */ }
  else if (msg.includes("[project_scope_mismatch]")) { /* wrong project */ }
  else if (msg.includes("[scope_insufficient]")) { /* missing scope */ }
  else if (msg.startsWith("configuration error")) { /* fix setup; no request made */ }
  else if (msg.startsWith("authentication failed")) { /* 401 */ }
  else if (msg.startsWith("not found")) { /* 404 */ }
  else throw e;
}

No typed error classes in Node

The TypeScript binding maps every core failure to a single Error. The message text and the bracketed 403 code are stable — match on those rather than on an exception class.

Two tokens, two planes

Most credential errors come from mixing planes. ik_sdk_ is the control token — held by ManagementClient, scoped to one project, cannot call inference. ik_live_ is the data key — passed per workload to a DataClient endpoint, cannot provision. If an error mentions the wrong plane, you almost certainly swapped the two.

---

wrong_credential_type — you used a data key where a control token belongs (or vice-versa)

A 403 whose code is wrong_credential_type. The token is valid, but it is the wrong kind for the plane you called. This is also caught locally, before any request, when you hand an ik_live_ data key to ManagementClient.ensure() — and the token is redacted to its prefix in the message, never leaked in full.

Python

inferencekey.errors.PermissionDenied: permission denied [wrong_credential_type]: ik_live_proj1234… is a data key; the management plane requires an ik_sdk_ token

TypeScript

Error: permission denied [wrong_credential_type]: ik_live_proj1234… is a data key; the management plane requires an ik_sdk_ token

• Symptom — ensure() rejects an ik_live_ key, or the inference endpoint rejects an ik_sdk_ token.
• Cause — control and data tokens were swapped. ManagementClient needs ik_sdk_; an Endpoint needs ik_live_.
• Fix — set INFERENCEKEY_SDK_TOKEN to your ik_sdk_ token (read by ManagementClient) and pass an ik_live_ key as the workload’s api_key (Python) / apiKey (TypeScript). Never put a data key in INFERENCEKEY_SDK_TOKEN.

---

project_scope_mismatch — the token belongs to a different project

A 403 with code project_scope_mismatch. The credential is the right kind but is scoped to another project than the one in your request path.

Python

inferencekey.errors.PermissionDenied: permission denied [project_scope_mismatch]: token is not scoped to project "acme"

TypeScript

Error: permission denied [project_scope_mismatch]: token is not scoped to project "acme"

• Symptom — a correctly-typed token is refused for a specific project.
• Cause — project= / INFERENCEKEY_PROJECT (or spec.project) names a project the token cannot act on. An ik_sdk_ token is scoped to exactly one project.
• Fix — point the client at the project the token owns, or mint a token for the project you are targeting. Confirm the slug matches the dashboard exactly.

---

scope_insufficient — right token, right project, missing permission

A 403 with code scope_insufficient. The credential is valid and in the right project, but it lacks the scope for the operation (for example a read-only token attempting a write).

Python

inferencekey.errors.PermissionDenied: permission denied [scope_insufficient]: this credential cannot perform the requested operation

TypeScript

Error: permission denied [scope_insufficient]: this credential cannot perform the requested operation

• Symptom — the operation is refused even though the token type and project are correct.
• Cause — the token was issued with fewer scopes than the call requires.
• Fix — issue a token with the needed scope in the dashboard, then update your env var or the explicit token you pass in.

---

Missing project — ConfigurationError

Raised before any request when no project can be resolved. Provisioning needs a project; an endpoint needs a project + an ik_live_ key.

Python

inferencekey.errors.ConfigurationError: No project configured. Set INFERENCEKEY_PROJECT, pass project=, or set spec.project.

A missing data key is the same class:

inferencekey.errors.ConfigurationError: No ik_live_ key for workload "support-bot". Pass api_key= or set INFERENCEKEY_API_KEY.

TypeScript

Error: No project configured. Set INFERENCEKEY_PROJECT, pass project, or set spec.project.

A missing data key:

Error: No ik_live_ key for workload "support-bot". Pass apiKey or set INFERENCEKEY_API_KEY.

• Symptom — the client throws immediately, no network call.
• Cause — project (or, for inference, the ik_live_ key) was never supplied. Resolution order is explicit > env > spec.
• Fix — set INFERENCEKEY_PROJECT or pass project= / project, and supply the workload’s ik_live_ key via api_key= / apiKey or INFERENCEKEY_API_KEY.

Python

mgmt = ManagementClient.from_env(project="acme")          # explicit beats env
data = DataClient.from_env(project="acme")
ep   = data.endpoint("support-bot", api_key="ik_live_...") # per-workload key

TypeScript

const mgmt = ManagementClient.fromEnv({ project: "acme" });          // explicit beats env
const data = DataClient.fromEnv({ project: "acme" });
const ep   = data.endpoint("support-bot", { apiKey: process.env.SUPPORT_IK_LIVE });

---

Invalid spec — a required field is empty (ValidationError)

name, slug and model are required and validated locally (whitespace-only counts as empty).

Python

inferencekey.errors.ValidationError: validation error: name must not be empty

TypeScript

Error: validation error: name must not be empty

• Symptom — ensure() throws before sending anything.
• Cause — one of name / slug / model is empty or blank.
• Fix — give every required field a non-empty value. ensure() is idempotent on the explicit slug, so keep it stable across runs.

Special case: a secret in a spec field — ConfigurationError

The core scans every string field of a spec and refuses any value beginning with ik_live_ or ik_sdk_. A credential must never be inlined into a spec. This surfaces as a ConfigurationError (Python) / configuration error (TypeScript), not a validation error.

Python

# ❌ Don't put a token in a spec field
mgmt.ensure(WorkloadSpec(
    name="support-bot", slug="support-bot",
    model="meta-llama/Llama-3.1-8B-Instruct", backend=Backend.VLLM,
    command="vllm serve ... --api-key ik_live_secret",  # leaked secret
))

inferencekey.errors.ConfigurationError: configuration error: secrets must never go in specs

TypeScript

// ❌ Don't put a token in a spec field
await mgmt.ensure({
  name: "support-bot", slug: "support-bot",
  model: "meta-llama/Llama-3.1-8B-Instruct", backend: Backend.Vllm,
  command: "vllm serve ... --api-key ik_live_secret", // leaked secret
});

Error: configuration error: secrets must never go in specs

• Symptom — ensure() throws locally the moment a field looks like a credential.
• Cause — a token was placed in name, slug, model, command, description, vllm_version, worker_id, gpu_resource_id, or project.
• Fix — keep tokens out of the spec entirely. ik_sdk_ lives on the management client; ik_live_ is passed per workload at inference time. (A value that merely contains the substring elsewhere — e.g. "see ik_live_ docs" — is fine; only a field that starts with the prefix is rejected.)

---

Auth failed — 401

The credentials are missing or invalid at the server.

Python

inferencekey.errors.AuthError: authentication failed: invalid or expired credentials

TypeScript

Error: authentication failed: invalid or expired credentials

• Symptom — every call to a plane is rejected at the door.
• Cause — the token is wrong, revoked, expired, or truncated (a partial copy/paste). Unlike the 403 family, the server cannot even identify the credential.
• Fix — re-copy the full token from the dashboard, check INFERENCEKEY_BASE_URL points at the right environment, and confirm the token has not been rotated.

401 vs 403

A 401 (AuthError) means the credential could not be authenticated at all. A 403 (PermissionDenied) means it authenticated fine but may not do this — that is when you read the code: wrong_credential_type, project_scope_mismatch, or scope_insufficient.

---

Not found — 404

The targeted resource does not exist.

Python

inferencekey.errors.ApiError: not found: workload "support-bot" not found in project "acme"

TypeScript

Error: not found: workload "support-bot" not found in project "acme"

• Symptom — an inference call to data.endpoint(slug, ...) is rejected because the workload does not exist (or is in another project).
• Cause — a typo’d workload slug, the workload was never provisioned, or the project slug is wrong.
• Fix — provision it first with mgmt.ensure(...) and use the returned ref.workload_slug / ref.workloadSlug. ensure() is idempotent on the slug, so calling it again is safe.

Python

ref = mgmt.ensure(WorkloadSpec(
    name="support-bot", slug="support-bot",
    model="meta-llama/Llama-3.1-8B-Instruct", backend=Backend.VLLM,
    command="vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192",
))
ep = data.endpoint(ref.workload_slug, api_key="ik_live_...")

TypeScript

const ref = await mgmt.ensure({
  name: "support-bot", slug: "support-bot",
  model: "meta-llama/Llama-3.1-8B-Instruct", backend: Backend.Vllm,
  command: "vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192",
});
const ep = data.endpoint(ref.workloadSlug, { apiKey: process.env.SUPPORT_IK_LIVE });

---

server_shutting_down: retry the request

The InferenceKey server that was handling your request is restarting (a deploy or a restart) and could not finish it. Nothing is wrong with your request or your credentials: it is retryable, and the same request sent again reaches another server, or the same one once it is back.

The server gives every request still in flight about 24 seconds after it starts shutting down to finish. Whatever is still open at that point ends with this error instead of a silent connection drop, so you never take a truncated answer for a complete one. A stream also ends this way at once when the GPU worker producing it shuts down.

What reaches the wire

How the error looks depends on whether the response had started and on the route:

Request	What you get
Non-streaming call still waiting for its answer	HTTP 503, header Retry-After: 1, the JSON body below
Streaming chat completion (stream: true)	One error chunk, then data: [DONE]. A stream whose response has not started yet may get the 503 instead
/v1/messages (Anthropic-compatible), non-streaming	HTTP 503, header Retry-After: 1, Anthropic error body of type overloaded_error
/v1/messages, streaming	One event: error of type overloaded_error, and no [DONE] (Anthropic streams have none)
/v1/audio/speech with stream: true	Before any audio: the 503 above. After audio started: the chunked body is aborted without its terminating chunk, so your HTTP client reports a transfer error

The non-streaming 503 body:

503 Service Unavailable (Retry-After: 1)

{
  "error": {
    "message": "The server is shutting down before the request completed; retry the request.",
    "type": "server_error",
    "param": null,
    "code": "server_shutting_down"
  }
}

The last two events of a streaming chat completion cut this way:

POST .../v1/chat/completions (stream: true)

data: {"error":{"code":"server_shutting_down","message":"The server is shutting down before the stream completed; retry the request.","type":"server_error"}}

data: [DONE]

On /v1/messages the stream ends with:

POST .../v1/messages (stream: true)

event: error
data: {"error":{"message":"The server is shutting down before the stream completed; retry the request.","type":"overloaded_error"},"type":"error"}

and the non-streaming 503 body there is {"error":{"message":"The server is shutting down before the request completed; retry the request.","type":"overloaded_error"},"type":"error"}.

What you see in the SDK

The InferenceKey SDK does not retry by itself. A non-streaming call, or a stream that gets the 503 when it opens, raises the 503 with the raw body in the message:

Python

inferencekey.errors.ApiError: api error (status 503): {"error":{"message":"The server is shutting down before the request completed; retry the request.","type":"server_error","param":null,"code":"server_shutting_down"}}

A stream cut after it opened raises while you iterate. This SDK version does not decode an error chunk, so the message does not name the code (it is the same for any error chunk):

inferencekey.errors.ApiError: workload drifted from spec: choices[0]

TypeScript

Error: api error (status 503): {"error":{"message":"The server is shutting down before the request completed; retry the request.","type":"server_error","param":null,"code":"server_shutting_down"}}

A stream cut after it opened throws inside the for await loop. This SDK version does not decode an error chunk, so the message does not name the code (it is the same for any error chunk):

Error: workload drifted from spec: choices[0]

• Symptom: a call fails with 503 and server_shutting_down, or a stream stops early with an error, typically during a platform deploy.
• Cause: the server (or the GPU worker serving the stream) shut down before the answer was complete.
• Fix: retry the whole request after about one second (Retry-After: 1). For a stream, discard the partial output you already received: the retried request generates the reply again from the start, and it may differ from the cut one. Keep the number of attempts bounded.

Python

stream_with_retry.py

import os
import time
from inferencekey import DataClient
from inferencekey.errors import ApiError

data = DataClient.from_env(project="acme")
ep = data.endpoint("support-bot", api_key=os.environ["SUPPORT_IK_LIVE"])

def cut_by_restart(e: ApiError) -> bool:
    msg = str(e)
    # The 503 names the code; a stream cut mid-way surfaces as an undecoded error chunk.
    # That message is the same for every error code, so any mid-stream error chunk is
    # retried, a bounded number of times.
    return "server_shutting_down" in msg or msg.startswith("workload drifted from spec: choices[0]")

def stream_reply(prompt: str, attempts: int = 3) -> str:
    for attempt in range(1, attempts + 1):
        parts: list[str] = []  # start over on every attempt
        try:
            for chunk in ep.generate_text_stream(prompt=prompt):
                parts.append(chunk.text)
            return "".join(parts)
        except ApiError as e:
            if attempt == attempts or not cut_by_restart(e):
                raise
            time.sleep(1)  # Retry-After: 1

TypeScript

stream-with-retry.ts

import { DataClient } from "@inferencekey/sdk";

const data = DataClient.fromEnv({ project: "acme" });
const ep = data.endpoint("support-bot", { apiKey: process.env.SUPPORT_IK_LIVE });

function cutByRestart(e: unknown): boolean {
  const msg = (e as Error).message;
  // The 503 names the code; a stream cut mid-way surfaces as an undecoded error chunk.
  // That message is the same for every error code, so any mid-stream error chunk is
  // retried, a bounded number of times.
  return msg.includes("server_shutting_down") || msg.startsWith("workload drifted from spec: choices[0]");
}

async function streamReply(prompt: string, attempts = 3): Promise<string> {
  for (let attempt = 1; ; attempt++) {
    let text = ""; // start over on every attempt
    try {
      for await (const chunk of ep.generateTextStream({ prompt })) {
        text += chunk.text;
      }
      return text;
    } catch (e) {
      if (attempt >= attempts || !cutByRestart(e)) throw e;
      await new Promise((resolve) => setTimeout(resolve, 1000)); // Retry-After: 1
    }
  }
}

Calling the endpoint with the OpenAI or Anthropic SDK

The official OpenAI and Anthropic SDKs retry a 503 by themselves (they honour Retry-After), so a non-streaming call usually succeeds without any code on your side. They do not retry a stream that already started: the OpenAI Python SDK raises openai.APIError with e.code == "server_shutting_down" when it reads the error chunk, and the Anthropic SDKs surface the overloaded_error event as an error. Catch it, drop the partial output and send the request again. For a streamed /v1/audio/speech body, treat a transfer error the same way and request the whole file again.

---

Quick reference

What you see	Plane / status	Python type	Cause in one line
permission denied [wrong_credential_type]	403	PermissionDenied	Control vs data token swapped
permission denied [project_scope_mismatch]	403	PermissionDenied	Token scoped to another project
permission denied [scope_insufficient]	403	PermissionDenied	Token lacks the required scope
No project configured… / No ik_live_ key…	local	ConfigurationError	Project or data key never supplied
validation error: … must not be empty	local	ValidationError	Required spec field is empty
configuration error: secrets must never go in specs	local	ConfigurationError	A token was inlined into a spec field
authentication failed: …	401	AuthError	Credential missing, invalid, or expired
not found: …	404	ApiError	Workload/project does not exist
api error (status 503): …server_shutting_down… or a stream that stops early	503 / stream error	ApiError	The server restarted before the answer was complete: retry the request

In TypeScript every row above arrives as a single Error; match on the message text (and the bracketed code for the 403 rows).

Tokens & the two planes Where ik_sdk_ and ik_live_ each belong.

OnDrift reference What ensure() does when a live workload differs from your spec.

---

New to InferenceKey? Create an account or open the dashboard · Learn more at inferencekey.com.