ILTER
Self-hosted AI gateway - MCP gateway, smart features
A few months back I was shipping LLM features into a few different projects, and I kept hitting the exact same wall every time. No idea what a feature was going to cost until the invoice showed up at the end of the month. One bad prompt loop from an agent could quietly burn through a day's budget while I was asleep. And customer data — emails, names, whatever — was going straight to OpenAI or Anthropic with zero controls in between.
I went looking for something to sit in front of my LLM calls and handle this, and everything out there was one of two things: a hosted SaaS that wanted my prompts (and my users' data) flowing through their servers too, or a self-hosted option that meant standing up a Python service, a Node frontend, a database, and Redis just to get a proxy running. I didn't want either. I wanted something I could drop in as a single binary and be done.
So I built ILTER. Self-hosted AI gateway, sits between your app and whatever provider you're using — OpenAI, Anthropic, Gemini, DeepSeek, OpenRouter, Ollama, whatever — one binary, nothing else to install.
What it actually does, roughly in the order I built it because I kept needing it:
💰 Budget Control — hard daily/monthly spending limits per API key. Hit the limit, requests get cut off, no end-of-month surprises.
🪪 PII Guard — emails, phones, SSNs, credit cards, names, all masked before they leave your network. Built this one for GDPR/HIPAA reasons, not because it sounded cool.
🌀 Agent Loop Detector — catches an agent going in circles by rate, repeated prompt fingerprint, cost burned, or session depth. Exists because I watched an agent make the same tool call 40 times in a row and almost paid for it.
⚡ Smart Router — every request gets scored for complexity in real time and routed to the right model tier, so you're not paying premium prices for "what's 2+2".
🌐 Smart Fallback — provider down or rate-limited, it fails over to another key or provider automatically, zero code changes.
🧩 MCP Gateway & Marketplace — connects your AI to your own APIs, CRM, or any MCP server with zero client-side changes, plus a marketplace to browse and one-click install community MCP servers.
🧠 Semantic Cache — repeated queries served from vector-search cache, with a SHA-256 exact-match fallback so it still works without an embedding model.
🛡️ Prompt Guardrails — blocks prompt injection, toxic content, and off-topic requests before they hit the model.
⏱️ Cron Engine — schedule recurring AI workflows with normal cron expressions, no external queue.
📡 Observability — Prometheus metrics and OpenTelemetry tracing out of the box, plugs into whatever you're already running.
📊 Dashboard — embedded straight into the binary — logs, keys, a chat playground, no separate frontend to run.
It's open source, one download, and I'm the only one building it right now — I run it on my own stuff, so everything above is something I actually needed, not something that sounded good on a roadmap.
Would genuinely love feedback from anyone here running LLM stuff in production — what's biting you that this doesn't solve yet?
3 Comments
What stands out is that nearly every feature came from something you personally ran into in production.
The next interesting evidence will come from seeing whether other teams encounter the same problems with the same intensity.
Exactly — every feature started as a fire I had to put out myself. Budget limits came after an agent quietly looped over a weekend and burned through way more API credit than it should have. PII masking came from realizing how much user data was leaving our network without anyone looking at it.
And you put your finger on the real open question. Honestly, I don't know yet whether these problems generalize with the same intensity — that's a big reason I open-sourced it instead of keeping it internal. What people actually turn on (which guardrails, which providers they route between) should tell me fairly quickly whether these are universal pains or just mine.
So if you've run into any of these in production, I'd genuinely love to know which one bit hardest — that's exactly the signal I'm watching for.
That’s helpful context. I appreciate you being open about what you know versus what you’re still trying to validate.
I’d like to continue the conversation outside the thread. What’s the best email to reach you on?