Building an AI Agent From Scratch (No Frameworks) — Lessons Learned
I recently experimented with AI agents, but instead of using frameworks like
LangChain or Autopen, I decided to implement a minimal agent in plain Python.
The goal wasn’t production-ready tooling — it was understanding the mechanics.
I wanted to see what an agent is really doing under the hood.
Key takeaways:
- Agents are mostly structured orchestration around an LLM.
- The core loop is Think → Act → Observe.
- Tool invocation, iterative reasoning, and memory handling can all be implemented in a few dozen lines of code.
- Frameworks like LangChain are convenient, but they hide the underlying mechanics.
Here’s a discussion I’d love to start: for those of you experimenting with AI agents, do you usually start from scratch to understand the logic, or do you rely on frameworks?
Really interesting perspective on the context management piece. That's often overlooked — most tutorials focus on the "how" but skip the "what". Without proper context injection, even the best LLM can't give useful responses. Your plain-Python approach sounds similar to how I'd approach learning async programming: understand the event loop before reaching for libraries that abstract it away.
That was exactly the plan, to fully understand what is an agent before going further. This is the post I was missing (or couldn't find) when I was learning basics of agents.
Awesome that you liked it!
This resonates. I went through the same exercise building a tech news aggregator with AI summaries — started with LangChain, then stripped it out once I realized I was fighting the abstraction more than using it.
What I've found:
The Think → Act → Observe loop is deceptively simple in concept but the real complexity lives in:
Context management — what goes into the prompt, what gets summarized, what gets dropped. Frameworks often have opinions here that don't match your use case.
Error recovery — when a tool call fails or returns unexpected output, handcrafted logic is often clearer than framework retry policies.
Observability — understanding why the agent took a path. With raw code, you can log exactly what you need. Frameworks sometimes bury this.
To your question: I usually start with a framework for prototyping (fast iteration), then rewrite the core loop in plain code once I know what I actually need. The framework teaches me the patterns; the rewrite teaches me the edge cases.
What was the hardest part of the implementation for you — the tool invocation plumbing, or the reasoning/memory piece?
@yamamoto7 to be honest, the hardest part of working with agents was actually understanding how they work and how to create them.
As you said - they are deceptively simple but also many courses or blogposts I visited was focusing on "look how fast you can do sth that is working for you".
The problem with that approach is that, sure, it can be done fast but would you understand it, truly understand it, this way? Maybe - yes, and I'm just not-that-clever 🤔.
I started creating agents, using Python next using Agents from open AI, CrewAI, did some testing with smolagents, was experimenting with visual WYSIWYG approach etc. and they were working. Getting stuff gone. They still do.
But all that time the fact that I didn't fully comprehend how this happens, how they function - bothered me.
Returning to your question "What was the hardest part of the implementation for you — the tool invocation plumbing, or the reasoning/memory piece?"
Overall - simple solution like the one I needed for the blogpost is well, simple. I intentionally didn't add any sophisticated logging etc. to make clear how things work. The said - if I would want to do it in customer-production-ready manner it would be totally different.
I hope it has done it's job, because, basically, I wrote a blogpost that I couldn't find when I was starting learning this stuff.
"I wrote a blogpost that I couldn't find when I was starting learning this stuff" — that's the best reason to write anything.
The "look how fast you can do something" tutorials are great for demos but terrible for understanding. Speed-to-working hides the decisions that matter.
Your post fills a real gap. Most agent content is either too abstract (theory papers) or too framework-specific (LangChain docs). The middle layer — here's what the code actually does, line by line — is rare.
Bookmarked your Medium article. Looking forward to seeing what you build next.
this resonates. i did something similar but on the input side rather than the agent side - wanted to understand what the human actually does during an AI coding session, so i
instrumented my own workflow. turns out it's shockingly repetitive: accept, reject, scroll, navigate, dictate. five actions, hundreds of times per session.
that led me to build a macOS app that maps those actions to a gamepad (vibepad.com). sounds like a joke but building it from scratch taught me the same lesson you're describing - you don't understand the interaction until you build the layer yourself. frameworks hide too much.
If you are interested, this are my findings on building agents from scratch https://medium.com/@kamil.tustanowski/ai-agents-101-from-concept-to-code-no-frameworks-required-2dfdaf66b6c1