When it comes to LLM, it's you who's using it wrong

I think we’re long past the discussion on whether LLM-assisted programming is better/faster (faster is better for some) than pure human typed one. LLM-assisted programming has won, at all kinds of tasks, and is clearly believed to be better than just doing it by hand.

But, how much better is still a topic of contention. And, rightfully so, given the estimations and velocity have started hinging on it.

It is in this question, that I see a dangerous assumption taken root. That, if someone is not having the same results as the loudest claims (Person A rewrote an entire project in a weekend, Company X clearing backlog of one year in one month, etc.), it’s the individual’s lack of skill (skillissue TM) that’s holding them back from releasing changes at the same pace.

The specifics of your existing code quality, bottlenecks in your process, the type of changes you’re making all go out the window and only the headline number takes root. This belief isn’t specific to companies or management. I see that even more among individual developers as well. I too went through that phase, where I was severely underwhelmed relative to the claims made and the touted productivity increase. Only after digging into everything involved did I understand there’s more to these claims than meets the surface.

Let’s say you use agentic development to identify a bug or add a new feature in an existing repository. You see the code it has produced is very subpar, lacks maintainability or downright wrong. You prompt again to fix it, do another round of review, find some more and you make those minor adjustments yourself and ship it.

For you, your experience is that LLM-assisted coding is good for certain kinds of tasks, but overall you find it gives you a modest 20-30% increase in the number of stuff you could work on.

And then you come across a thread on HackerNews and naturally, irrespective of the topic, LLMs comes into discussion and one says that they don’t find all that useful and then you get 10s of comments saying how they’re wrong and swear by the speed improvements they got. That the OP must be using the tool wrong, “did you add skills?”, “did you phrase your prompt right?”, “you probably didn’t do the plan/develop properly”, “you’re still doing plans? Agentic loops will fix this”, and on and on it goes. You close the tab feeling that you must be doing something wrong, otherwise how come these many people swear by this?

And so you go back, add/update skills, recurring skills, fancy skills, enlist subagent driven development, use frontier models, and still you find the AI drops the ball in some cases. You’re frustrated. You go back to thread and discover that for all of those 10 replies, there is one reply from a fellow engineer who had the same experience you did “I have the skills, I have the subagent, I use frontier models, but I still find LLM not that great on some usecases”.

“You probably didn’t prompt it right” posits one, while another speculates you’re ragebaiting. “They’re probably doing some kernel development, that’s why”. They all, in unison, come back and say “The tool is fine, it’s you who’s using it wrong”.

LLMs, like humans (I know how it reads, wait for it), are non-deterministic. The same way you don’t expect two humans to answer a question the same way, you can’t expect LLMs to perform the same way in all the scenarios. And when you combine two non-deterministic systems to produce something, there is bound to be a big gulf in the obtained result. Combining two variable systems compounds variance.

The problem is that a large group (knowingly or unknowingly I won’t speculate) treats LLMs as absolute, and as yet another tool, like a cloud service. In a cloud setup, for two identical workloads on an identical stack, you can expect to provision near the exact resources. You can’t do that with LLM.

Beyond just LLM’s contribution, there is a whole host of things that determine how much benefit someone gets out of it. What is their code’s shape, their engineering standards, the team’s strengths, business process, the type of task, and so on.

That is not to say that there’s no scope for tweaks one can use to improve but treating a group of people’s experience as absolute is where the discussion loses its nuance. One group believes that the ones who get the multi-fold improvements are all building greenfield or have shitty code standards while the other side thinks it’s skillissue or they’re AI naysayers and that is why the opposite side is not getting the output as themselves.

The reality, as evident, is nuanced. The best practices for using LLM and Agentic development are evolving by the day. The only sustainable path forward is to evaluate a workflow, integrate, tweak it to your situation, adopt and move on.

I do not believe in the future where LLMs write code, LLMs review, LLMs deploy and LLMs bugfix. Writing code is straightforward, engineering an application is not. So, for the foreseeable future, where LLMs <> Developers work hand-in-hand, a customised adaptation goes further than treating one workflow or one output as absolute.