modelstogether.com

Development teams and individual engineers that embrace LLMs in the development process are experiencing a fundamental shift in the time that it takes to solve problems with and in software. Here's what I learned by building prototypes in a controlled environment.

In Maximizing Developer Effectiveness, Tim Cochran put forward that High Effectiveness teams find the root cause for defects within one day and that Low Effectiveness teams take four to seven days. I've built software for a long time and managed teams for a long time. That feels about right to right to me. Rather, that felt right; I think it's changing.

For a lot of common cases, LLM-enabled teams who know their domain will find those defects in minutes. And generally, the effectiveness of engineers and teams who know how to wield these new tools will be off the charts relative to their peers who do not embrace them.

TLDR:

1. Quality control is critical. It takes the judgment, experience, and taste of an experienced engineer to get an LLM to produce something acceptable for your environment.
2. Iteration speed matters. If you haven't solved the slow iteration problems that people and teams have, you're way behind. You should fix those now because you'll shortly need to invest in improving the speed at which you provide context about the up-to-the-second state of a project to the LLM.
3. There is still no such thing as a free lunch. People need training and practice using these tools to fully take advantage of them. This won't happen overnight.

I built a service that lets LLMs talk to each other and a website to visualize those conversations.

Here I share the development log and my reflections on the process.