DEV Community: chen qin

Your AI API Request Is Hanging? Add a Timeout With AbortController

chen qin — Thu, 13 Aug 2026 07:00:35 +0000

An AI API request should never be allowed to wait indefinitely.

Network problems, overloaded providers, or unexpectedly long generations can leave a request open much longer than your application should tolerate.

In JavaScript, AbortController provides a simple way to define a timeout and cancel the request.

A basic timeout wrapper

async function fetchWithTimeout(url, options = {}, timeoutMs = 15_000) {
  const controller = new AbortController();

  const timeoutId = setTimeout(() => {
    controller.abort();
  }, timeoutMs);

  try {
    return await fetch(url, {
      ...options,
      signal: controller.signal,
    });
  } finally {
    clearTimeout(timeoutId);
  }
}

You can use it with an OpenAI-compatible endpoint:

const response = await fetchWithTimeout(
  "https://your-api-endpoint/v1/chat/completions",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.AI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "your-model",
      messages: [
        {
          role: "user",
          content: "Explain exponential backoff in one paragraph.",
        },
      ],
    }),
  },
  30_000
);

Handle cancellation separately

An aborted request should not be treated like every other network failure:

try {
  const response = await fetchWithTimeout(url, options, 30_000);

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}`);
  }

  const data = await response.json();
  console.log(data);
} catch (error) {
  if (error.name === "AbortError") {
    console.error("The AI request exceeded its timeout.");
  } else {
    console.error("The request failed:", error);
  }
}

Separating AbortError makes it easier to log timeouts, display an appropriate message, and decide whether another attempt is safe.

Don’t retry blindly

A timeout and a retry solve different problems:

A timeout prevents your application from waiting indefinitely.
A retry creates a new request.

Automatically retrying an aborted request can duplicate work or produce unintended side effects. Before retrying, consider:

Is the operation safe to repeat?
Did the provider begin processing the original request?
Could the request create or modify data?
Is an idempotency key available?
How many attempts have already been made?
Should the retry use exponential backoff and jitter?

Retries should have a limit. They should also respect provider headers such as Retry-After when available.

Choose the timeout deliberately

There is no universal timeout value for every AI workload.

A short classification request may need a much lower limit than a long reasoning or generation task. Streaming responses may also need separate connection and inactivity timeouts.

Measure real production latency and choose limits based

Stop Sending the Full Chat History With Every AI Request

chen qin — Tue, 11 Aug 2026 09:04:05 +0000

Long-running AI conversations can quietly accumulate a lot of unnecessary context.

A common implementation simply appends every message and sends the entire array again:

const messages = [
  systemMessage,
  ...completeConversationHistory,
];

const response = await client.chat.completions.create({
  model: "your-model",
  messages,
});

This works initially, but the request grows after every turn. Token usage can increase, and additional input may also affect latency.

A more deliberate approach

Instead of keeping everything, build the request from the context the model currently needs:

const messages = [
  systemMessage,
  {
    role: "system",
    content: `Summary of earlier conversation: ${conversationSummary}`,
  },
  ...recentRelevantMessages,
];

const response = await client.chat.completions.create({
  model: "your-model",
  messages,
});

A practical context strategy might include:

Keeping the original system instructions
Keeping recent and relevant conversation turns
Summarizing older exchanges
Preserving tool results, identifiers, and decisions still required by the workflow
Measuring input tokens before sending the request

Don’t truncate blindly

The smallest possible context is not always the best context.

Removing an important tool result, user constraint, or earlier decision can produce an incorrect response. Context optimization should therefore be tested against response quality—not only token count.

For more complex applications, retrieval can also be used to restore relevant information when it is needed instead of attaching the complete history to every request.

The main idea

Treat the model’s context window as an input you actively manage, not as permanent storage for the entire conversation.

Send enough information for a reliable answer, but avoid repeatedly sending content that no longer contributes to the current task.

APIHubRelay provides an OpenAI-compatible API gateway for working with multiple AI models:

https://apihubrelay.com/

How do you manage long AI conversations in production: summarization, truncation, retrieval, or a combination?

Your Frontend API Key Is Not Secret: Move AI Requests to Your Backend

chen qin — Sun, 09 Aug 2026 04:58:32 +0000

How to Access Multiple AI Models Through One OpenAI-Compatible API

chen qin — Fri, 07 Aug 2026 10:43:32 +0000

How to Retry 429 Errors with Exponential Backoff in Python

chen qin — Tue, 04 Aug 2026 07:56:36 +0000

Stop Waiting for the Full AI Response: Stream Tokens in Python

chen qin — Mon, 03 Aug 2026 06:38:11 +0000

Most AI applications wait for the model to generate the complete answer before showing anything to the user.

For short answers, that may be acceptable. For longer responses, it can make the application feel slow—even when the model is already generating tokens.

Streaming solves this by displaying each part of the response as soon as it arrives.

The non-streaming version

A standard OpenAI-compatible request may look like this:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_API_KEY"],
    base_url=os.environ["AI_BASE_URL"],
)

response = client.chat.completions.create(
    model=os.environ["AI_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
)

print(response.choices[0].message.content)

This works, but nothing is printed until the complete response has arrived.

Stream the response

Enable streaming by adding stream=True:

stream = client.chat.completions.create(
    model=os.environ["AI_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
    stream=True,
)

The request now returns a sequence of chunks instead of one completed response.

Loop through those chunks and print the available content:

for chunk in stream:
    content = chunk.choices[0].delta.content

    if content:
        print(content, end="", flush=True)

print()

The user can now see the answer while it is being generated.

Complete example

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_API_KEY"],
    base_url=os.environ["AI_BASE_URL"],
)

stream = client.chat.completions.create(
    model=os.environ["AI_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content

    if content:
        print(content, end="", flush=True)

print()

Keeping the API key, base URL, and model name in environment variables also makes it easier to change providers without rewriting the application logic.

When streaming is useful

Streaming is especially helpful for:

AI chat interfaces
Coding assistants
Long-form generation
Command-line tools
Applications where perceived latency matters

Remember that model capabilities and streaming formats can vary between providers. Verify support for your selected model and handle empty chunks, connection failures, and interrupted streams before using this pattern in production.

I tested this pattern with an OpenAI-compatible endpoint through APIHubRelay.

What should the next example cover: streaming in Node.js, error handling, or automatic retries?

Building AI Products for a Global Audience: 7 Practical Lessons

chen qin — Sat, 01 Aug 2026 06:03:33 +0000

AI products are global from day one.

A developer may discover your product in English, configure it in Chinese, read the documentation in Japanese, and debug an API response with a distributed team across several countries.

That makes localization more than a translation task. It is part of the developer experience.

Here are seven practical lessons we learned while building a multilingual experience for an AI API platform.

1. Design for localization before translating

Avoid placing interface text directly inside components.

Use translation keys from the beginning:

t("navigation.documentation")
t("dashboard.apiKeys")
t("errors.invalidRequest")

This makes it much easier to add languages later without modifying every page.

2. Keep technical terms consistent

Terms such as API, token, endpoint, model, request, and response are already familiar to many developers.

Translating every technical term can make documentation harder to understand. A better approach is to keep standard developer terminology while translating the surrounding explanation.

Consistency is more important than literal translation.

3. Expect text length to change

A short English button label can become much longer in French or Russian.

Interfaces should therefore avoid fixed-width text containers. Navigation items, buttons, alerts, and pricing cards all need enough flexibility to accommodate different languages.

Test the real translations—not placeholder text.

4. Keep code examples language-neutral

Code snippets should remain easy to copy regardless of the selected interface language.

For example:

curl https://your-api-endpoint/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

Translate the explanation around the example, but avoid changing field names, endpoint paths, and commands.

5. Localize the complete onboarding journey

A translated homepage is only the beginning.

Developers may also need localized:

Registration guidance
API key instructions
Error messages
Billing explanations
Documentation navigation
Support information

If users switch back to an unfamiliar language immediately after registration, the localized landing page has not solved the real problem.

6. Make language switching predictable

The language selector should be visible, easy to recognize, and available without signing in.

Remembering the selected language also matters. Users should not need to change it every time they return.

We currently provide seven interface languages:

Simplified Chinese
English
French
Russian
Japanese
Vietnamese
Traditional Chinese

7. Measure the experience by locale

Traffic alone does not tell you whether localization works.

Useful measurements include:

Registration conversion by language
Documentation visits by locale
API key creation rate
Onboarding completion
Support requests by language
Errors encountered during setup

These signals help identify translations that are technically correct but still confusing.

What we are building

We applied these ideas while developing APIHubRelay, an OpenAI-compatible API gateway designed to make access to multiple AI models easier for developers.

The goal is simple: developers should be able to explore and use AI infrastructure without language becoming an unnecessary barrier.

If you are building a multilingual developer product, what localization issue surprised you the most?

Disclosure: I am involved in building APIHubRelay. AI tools were used to assist with the structure and language editing of this article; the technical points and final review were completed by the author.

How to Use One OpenAI-Compatible API for Multiple AI Models

chen qin — Wed, 29 Jul 2026 09:36:37 +0000

Modern AI applications rarely depend on only one model.

A team might use one model for reasoning, another for fast classification, and another for multimodal tasks. But integrating every provider separately introduces duplicated work:

Different authentication methods
Different request and response formats
Different SDKs and error structures
Separate usage dashboards
More code to maintain

An OpenAI-compatible API gateway can simplify this architecture by exposing one familiar interface for multiple models.

The basic idea

Instead of connecting your application directly to every model provider, your application connects to one gateway:

Your application
       ↓
OpenAI-compatible gateway
       ↓
OpenAI / Claude / Gemini / other models

Your authentication and request format remain consistent. In many cases, switching models only requires changing the model identifier.

JavaScript example

Install the OpenAI SDK:

npm install openai

Create a client that points to your gateway:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: process.env.AI_GATEWAY_BASE_URL,
});

async function generateResponse(model, prompt) {
  const response = await client.chat.completions.create({
    model,
    messages: [
      {
        role: "user",
        content: prompt,
      },
    ],
  });

  return response.choices[0].message.content;
}

const result = await generateResponse(
  process.env.AI_MODEL_ID,
  "Explain API gateways in three sentences."
);

console.log(result);

Set the environment variables using the values provided by your gateway:

AI_GATEWAY_API_KEY=your_api_key
AI_GATEWAY_BASE_URL=your_gateway_base_url
AI_MODEL_ID=your_model_id

Keeping these values outside the application code makes it easier to change endpoints and models between development and production.

Python example

Install the SDK:

pip install openai

Then initialize the client with a custom base URL:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url=os.environ["AI_GATEWAY_BASE_URL"],
)

response = client.chat.completions.create(
    model=os.environ["AI_MODEL_ID"],
    messages=[
        {
            "role": "user",
            "content": "Explain API gateways in three sentences.",
        }
    ],
)

print(response.choices[0].message.content)

Add a model-routing layer

Avoid scattering model identifiers throughout your code. Keep them in one configuration object:

const models = {
  fast: process.env.FAST_MODEL_ID,
  reasoning: process.env.REASONING_MODEL_ID,
  multimodal: process.env.MULTIMODAL_MODEL_ID,
};

const selectedModel = models.reasoning;

This lets you change providers without rewriting the rest of the application.

Handle failures consistently

A gateway reduces integration complexity, but production applications still need basic resilience:

async function requestWithRetry(model, prompt, retries = 2) {
  for (let attempt = 0; attempt <= retries; attempt++) {
    try {
      return await generateResponse(model, prompt);
    } catch (error) {
      if (attempt === retries) throw error;

      const delay = 500 * 2 ** attempt;
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

For production workloads, also consider:

Request timeouts
Rate-limit handling
Usage and cost monitoring
Logging without exposing prompts or API keys
Model fallbacks
Separate keys for development and production

Why we built APIHubRelay

We built APIHubRelay to provide developers with a unified, OpenAI-compatible gateway for leading AI models.

The goal is straightforward: integrate once, manage access in one place, and make it easier to test or switch models as applications evolve.

The website also supports multiple interface languages for international developers and teams.

You can learn more at:

https://apihubrelay.com/

Before copying the examples into production, replace the gateway URL and model identifiers with the exact values shown in the APIHubRelay documentation or dashboard.

Final thoughts

A unified API does not eliminate every difference between AI models. Models can still have different capabilities, context limits, pricing, and supported parameters.

However, standardizing the connection layer removes a large amount of repetitive integration work. It also gives teams a cleaner foundation for model routing, monitoring, and future provider changes.

Disclosure: I am involved with APIHubRelay. AI assistance was used to help structure and edit this article, and the final content was reviewed before publication.