Why LLMs alone aren't the answer?

I remember the moment clearly.

It was a little over two years ago, late in the evening. I was in my "R&D pod" with two of my reportees, stuck on a hard problem: Mainframe Modernization using LLMs. It was a tough nut to crack.

But, that evening we had a Eureka moment. My team rebuilt the solution as a multi-agent workflow using AutoGen, and the quality of the output improved dramatically. Same problem, same LLMs, completely different result.

Once I'd double-checked the output, I went through our solution codebase. It was pretty much what we had in the prior iterations. Except for one major difference. We had different agents working on sub-portions of the problem and THEN communicating with each other. Each agent had its own narrow goal. One agent generated, another critiqued, yet another tested via compilation and the loop repeated until the result held up. This was in essence the Evaluator-Optimizer pattern. Back then, my team and I didn’t know this name. We just experimented, found that it worked and started using it.

Once the highs of our euphoric breakthrough subsided, I soon started focusing on the unasked questions → “Why?” Why was this variant of a LLM based solution actually working?

Me: Did we change the underlying LLM?

My reportee: No, it’s the same. This is the best model we have so far for COBOL syntax comprehension.

Me: But then earlier the result was horrible. How come now it’s better?

Him: Hmm… Don’t know.

Me: Is it just because we broke down the larger problem into smaller sub-problems?

Him: Could be.

Eventually of course, we learnt that task decomposition and context engineering are key principles for solving complex problems using LLMs. This was before "Agentic AI" was a buzzword. We were tinkering with whatever we could get our hands on and learning every day. In hindsight, that's the best way to learn these concepts: by hitting the wall first.

The lesson I learnt: Just throwing the model at a problem expecting magic to happen was naive. Want to solve large problems? Be ready to do a lot of engineering.

Here’s an analogy that I often use to drive home this point. LLMs are like a powerful engine. But most people don’t buy an engine. They buy a car.

Today I'm Field CTO at Hazel AI, and such hard earned lessons by the entire team sit at the centre of everything we build. For example, we’ve seen this story unfold numerous times. Throw the AI at the problem. Then spend a whole lot of human SME effort in “fixing” the AI response. Instead of expending costly effort on this manual conformation process we adopt a different strategy. We leverage our own Mixture of Experts. We get the domain expert to do a detailed task breakdown. Then we have the AI expert perform the content engineering to neatly package just enough info for the AI agent to manage effectively while adhering to the overall goals.

Hazel is an agentic delivery engine for enterprise modernization: SAP S/4HANA programs, legacy application rewrites, cloud and data migrations. The kind of work that has always been sold in person-hours. We now provide the same under Services-as-Software.

We run a team of curated agents. Each one has its own goal, toolkit and operating instructions. They know each other's capabilities and call on their "colleagues" when needed, much like a well-run enterprise team.

Why agents? Because LLMs alone can't solve large, real-world enterprise problems. Someone has to build the harness that handles:

• Task decomposition
• Inter-agent collaboration
• Prompt versioning and dynamic model selection
• Memory management
• Private knowledge based ground
• Human-in-the-loop steering
• Error handling and checkpointing
• Evals tied into deployments
• Observability
• And a context layer that allows for meaningful inter-agent collaboration

None of this is trivial. It’s not sexy either. And ALL of it takes serious time and effort. A thin LLM wrapper makes a great demo. It falls apart the moment it meets a 40-interface S/4HANA rollout or a decade-old PL/SQL estate.

Our team has burnt the midnight oil building self-learning and self-correcting agents. Every customer problem we've hit has made those agents sharper, and that's the part I find most exciting: the system compounds. Each program we deliver makes the next one faster and more accurate.

The results show it. Across programs, Hazel has helped customers cut delivery cost by around 40% and delivery time by up to 50%. One customer finished application modernization work in three months that they had expected to take seven or eight.

It also changes the business model. Instead of selling hours, delivery organizations can encode their own methodology into agents that run under their brand, and commit to outcomes rather than effort.

The takeaway

If you consider just making an LLM call to solve complex problems; you will be disappointed. Go in with the mindset that you will need to marry the raw power of LLMs with classical software engineering practices; then victory will be yours.

So, if you are struggling with a problem that seems unsolvable with just plain strung-together LLM calls, drop us a line here.