How We Stopped Hemorrhaging Money on AI APIs — and Built a Governance Layer Instead
$3,472. That was our AI API bill for March 2026. For a 30-person dev team. With zero ability to explain which project burned what.
I remember staring at the number and thinking: we've built an entire CI/CD pipeline, infrastructure-as-code, monitoring dashboards for every microservice — and our AI spend is a black box.
The Wake-Up Call
It didn't start with the bill. It started with a Slack message.
Our backend lead, David, pinged me at 11pm on a Thursday: "Hey, is anyone else's Claude key throwing 429s? Mine just died mid-deploy."
I checked our shared 1Password vault. There were seven different Claude keys in there. Three belonged to people who'd left the company. One had billing attached to an expired corporate card. Nobody knew which key David was using.
That was February. In March, I sat down and tried to answer three questions:
- How many AI API keys are active across the team?
- What's the monthly spend, broken down by project?
- Where is our data going — are any of these third-party proxy services?
I couldn't answer any of them. Neither could anyone else.
What We Actually Found
I spent two weeks auditing. Here's what surfaced:
14 active API keys across OpenAI, Anthropic, Google, and two random proxy services someone had signed up for on a free trial. Four keys were tied to personal Gmail accounts. Two proxy services had zero public security documentation.
$3,472 in March. But the breakdown was the scary part: $1,840 went to a single project that we later realized was 40% cacheable queries. Another $600 was from a developer who'd configured his IDE plugin to use GPT-5 for every single autocomplete — including boilerplate.
No audit trail. When I asked the proxy service for access logs, they sent a CSV with IP addresses and timestamps. No user attribution. No project tagging. No way to map a request to a person.
And then there was the file leak. One of the proxy services was routing multiple teams through the same underlying API account. A developer on our team received a Claude response that contained a file path from another company's codebase — because Claude's context window had picked it up from a previous session. I still get chills thinking about that.
The First Attempt (That Failed)
My first instinct was "let's just consolidate everything into a single API account with usage quotas."
We tried. It broke within a week.
Developers hated it because they had to share rate limits. The security team hated it because a single compromised key would expose everything. Finance hated it because we still couldn't attribute costs. It was centralized access without centralized control — the worst of both worlds.
What Actually Worked
We took a step back and asked a different question: what would this look like if we treated it as infrastructure?
Not "how do we share keys." But: how do we issue credentials, route traffic, attribute costs, and maintain audit trails — the same way we handle database connections and API gateways.
The answer was a governance layer with four components:
| Layer | What It Does |
|---|---|
| Derived Credentials | Developers never touch raw API keys. They get scoped, time-limited credentials with model whitelists and rate limits. One-click revocation. |
| Local Proxy | All AI requests route through a lightweight proxy on the developer's machine. Traffic goes directly to the model provider — no third-party server in the path. |
| Cost Attribution | Every call is tagged by project, user, and model. You can answer "which project spent what on which model" in real time. |
| Audit Trail | Immutable, unified logs for every request. Exportable in minutes for compliance reviews. |
The local proxy was the critical architectural decision. We had team members (rightfully) paranoid about their prompts and code going through some external service. The proxy runs locally, handles routing and policy checks, and sends the request straight to Anthropic/OpenAI/Google. Our servers never see the data.
The Numbers, Before and After
| Metric | Before (March 2026) | After (June 2026) |
|---|---|---|
| Monthly AI spend | $3,472 | $2,081 |
| Active keys in the wild | 14 (unknown scope) | 0 raw keys exposed |
| Cost visibility | Zero per-project data | Real-time by project/model/user |
| Audit readiness | 2+ weeks to compile | Under 5 minutes |
| Key revocation on offboarding | Days to never | One click, instant |
The $1,391/month savings came from two changes: we identified that 15% of GPT-5 calls were simple code completions (switched to a cheaper model, saved ~$300/month), and we shut down the two proxy services (eliminated their markup, saved ~$1,100/month). The rest was just stopping abandoned keys from running.
What I'd Do Differently
Three things I wish someone had told me:
1. Don't start with cost — start with security.
Cost gets management's attention, but the real risk is data leakage. The file-path-in-context-window incident should never have happened. If I could go back, I'd frame the entire initiative around "we have no idea where our code is going" rather than "we're spending too much."
2. The proxy architecture is non-negotiable.
We evaluated two approaches: cloud-based API gateway vs. local proxy. Cloud-based was easier to deploy but meant every prompt passed through our infrastructure. Several team members flat-out refused. The local proxy added deployment complexity but eliminated the trust problem entirely.
3. Dev experience matters more than you think.
Our first version of the client required manual configuration of model endpoints. Developers hated it. We iterated to a single aicli auth command that auto-discovers project context. Adoption went from 40% to 95% in one sprint.
We Productized It
After we solved this internally, several founders in our network asked how we did it. The pattern kept coming up: teams of 20-200 people, heavy AI usage, zero governance.
So we built AiKey — a credential and governance platform for teams that have outgrown the ".env file with an API key" stage.
- Local proxy handles routing and compliance checks
- Control plane manages virtual key issuance, policy enforcement, audit aggregation
- Zero data passthrough — requests go directly from developer machine to model provider
If your team's AI API keys are still living in shared vaults, config files, and Slack DMs — fix it before someone leaves. The cleanup is 10x worse after the fact.
aikeylabs.com | Enterprise: aikeyfounder@gmail.com
AMA about the architecture, the proxy design, or the internal rollout. Happy to share what worked and what blew up in our faces.
The part about finding 7 different Claude keys in a shared vault with expired corporate cards hits the exact structural bleeding point. When developers leave IDE plugins configured for every single autocomplete without a local proxy audit layer, silent prompt replication and file-path leaks in context windows tear through backend margins. Since you built AiKey Labs to solve this governance gap, you might want to look at the production-grade backend logic I open-sourced to hard-lock these exact token leaks: https://github.com/koniJang/max-token-saver-light. No calls, 100% async.
Your point about starting with security rather than cost resonates. The file path leak across tenants is the kind of incident that makes governance non negotiable once you see it happen. What caught my attention was the local proxy decision. You mentioned team members refused a cloud gateway because of trust concerns. How did you handle the deployment complexity across different OS environments with the local proxy approach?
The network environment challenge makes sense. We deal with the same pattern at our agency, split tunnels and Zscaler create issues for any local proxy approach. Did you find a reliable way to detect the network environment automatically and adjust the proxy behavior, or did it end up being a config file per environment sort of thing?
Article doesn't go deep on OS-level deployment, but here's the short version: the proxy is a single binary, no dependencies, and we ship it via the same tooling the team already uses for runtime installs. macOS and Linux coverage from day one — that covers our entire eng team. Windows came later, primarily for the data team, and honestly it was the Go cross-compilation that made it manageable. One build pipeline, three targets.
The bigger challenge wasn't OS compatibility — it was network environments. VPNs, corporate proxies, Zscaler, split tunnels. That's where most of the support time went.