What I underestimated when building an AI API gateway

I recently started building an independent API gateway for applications that use multiple AI providers.

At the beginning, I assumed the main challenge would be routing a request to the right model. The routing itself was relatively straightforward. The difficult part was making different providers behave consistently enough for an application to rely on them.

Here are the problems that took more time than expected:

1. Streaming responses

A non-streaming response is usually easy to normalize. Streaming is different. Events can arrive in different shapes, usage information may only appear in the final event, and a tool call can be split across several chunks.

If the client assumes that every chunk contains ordinary text, the integration eventually breaks.

2. Tool calls

Tool calls are especially difficult because the function name, arguments, and status may not arrive together. The application needs to accumulate partial data and only try to parse the arguments when the stream is complete enough.

Malformed or incomplete JSON also needs to produce a useful error instead of silently failing.

3. Content blocks

Some APIs represent content as a string. Others return an array of content blocks containing text, images, tool results, or other data.

Treating every response as a string works for simple demos, but not for a real application that supports multiple message formats.

4. Error handling

Different providers use different status codes, error fields, and messages. I had to normalize the errors while preserving enough information for debugging.

A generic “request failed” message is not useful. At the same time, returning raw provider responses can expose implementation details that the client should not depend on.

5. Retries and duplicate requests

Retries look simple until the request has side effects or consumes a large amount of tokens. A retry after a timeout does not necessarily mean the provider never processed the original request.

This makes idempotency, request tracking, and clear retry policies important, especially for agent workflows.

6. Model capabilities

Model names alone are not enough. Applications also need to know whether a model supports streaming, tools, vision, structured output, long context, or a particular message format.

I started maintaining a small capability matrix instead of assuming that all models behind a compatible endpoint behave the same way.

My current testing process includes:

• the same prompt in streaming and non-streaming modes
• normal text responses
• long responses
• tool calls with fragmented arguments
• malformed requests
• provider timeouts
• retry scenarios
• usage and billing metadata
• model-specific capability checks

The next part I’m working on is better observability: making failures easier to diagnose while avoiding the collection of sensitive prompt or API-key data.

I’m curious how other indie hackers handle this. Do you keep provider-specific logic inside one adapter, or do you expose the differences to the application layer?

For context, I’m building luoluoAI, an independent gateway with OpenAI-compatible and Claude Messages-compatible interfaces. I’m sharing the engineering problems here because the compatibility work has been more interesting than the initial routing layer.