Reflections on LLM Prototyping

As always, the feedback loops associated with software development are important in being productive. This development approach presented some ways to minimize time within feedback loops, mostly from the perspective of the LLM.

In the classic essay about maximizing developer effectiveness, Tim Cochran shared that High Effectiveness teams find the root cause for a defect within one day and that low effectiveness teams take four to seven days. I've built software for a long time and managed teams for a long time. That feels about right to right to me. Rather, that felt right; I think it's changing.

With the right setup, an LLM can make code changes, run tests, load a web page, see the web page, and inspect for issues. It can add debugging statements and inspect the output of the debugging statements as the code is executed. That takes the time to find a root cause for a defect WAY down. All from a prompt like "this does X and should do Y".

Now we're down to minutes to find and fix issues in software. Of course, you can't trust the solution without reviewing it. The ticket needs to be updated. Pull requests opened. Reviews addressed. So the work isn't done. But this takes a core part of what it means to be a developer, the "solving problems with (or in!) software" part, and dramatically boosts the speed of it.

There's a bit of sleight of hand in the last paragraph. I skipped the effort/knowledge/context required to be able to make a statement like "this does X and should do Y". For now, that type of goal setting I think is reserved for humans with experience in the domain and with the platform.

I also skipped that these setups are hard to replicate and hard to keep running with the current state of the technology. There are very few people who are great at using these tools given their recent introduction.

Engineers need to understand our stack really well and understand how to build things well using those tools. An understanding of the domain in which they work is also critical. Without engineers applying their knowledge, you get whatever the LLM gives you. It may solve exactly what you asked for but neglect a dozen important things or solve a dozen problems that don't to be solved right now. It may use technologies that are inappropriate or costly. Engineers bring judgement and taste to the table.

Given the nascent state of these technologies, teams adopting them need time to learn how to work with LLMs. The business shouldn't expect that immediately upon adoption these tools will improve productivity. No one is born knowing how to walk and engineers must learn how to best use these new tools.

Now is the time to start enabling teams and individuals to adopt and master these new tools. Much like the best time to plant a tree is twenty years ago, the best time to invest in the skills of your team is now.