Why I ask AI coders to ask me questions

I’ve been using a whole host of AI program to write code over the past two years, but I’ve developed a style that is very different from other programmers with whom I’ve spoken. Based on that feedback, I’m writing up my ideas on prompt engineering for coding with the hope that others can benefit.

In my experience, using AI to write code is like having an enthusiastic, highly skilled undergraduate who has read all of the documentation and has several years of coding experience, but doesn’t have a good sense of elegance, style or architecture.

Lots of people I know give high-level instructions to the AI, it generates code, and then they iterate. This approach is great at getting code to work and adding features, but it’s lousy at creating code that is compact, well-designed, and maintainable.

My approach is different.

I start by writing several paragraphs for the AI about what I’m trying to accomplish and the different high-level architectures I’m considering. I ask the AI to evaluate the architectures and give me its opinion regarding the strengths and weaknesses of each. What libraries would the AI recommend? What should be stored in a database? How should the solution be architected and deployed? What are the broad outlines of my options?

I will pick one of the architectures and give the AI my rough sketch for the modular decomposition of the program. I will tell the AI the parts I’m thinking of using, how I want to handle technical issues like authentication, secrets management, data versioning and testing.

Next, I will tell the AI to ask me 10 to 20 questions about architectural choices that I haven’t thought about, that it needs to pin down before writing code. Invariably, the questions that I’m asked are detailed, insightful, and pivotal to the design. Sometimes the AI will give me choices for answers, sometimes it will ask the question and make a sceptic recommendation.

I answer the AI’s questions, after which I ask the AI if it has any follow-up questions for me. After a few rounds of this back-and-forth, I ask the AI to create high-level documentation for different aspects of the solution. I then review the documentation. This documentation lets me review the entire design and make corrections, but it also serves as a record that this or another AI will be able to use in the future when it comes back to do software maintenance.

Now, with all of this done, I ask the AI to create an initial implementation and a simple test suite.

I’ve discovered that a lot of the AIs are somewhat lazy programmers if you don’t hold them to high standards. They will repeat code, take questionable shortcuts, and engage in pretty poor programming practices. I establish strict coding standards---for example, requiring that code generate no program analyzer warnings --- and then repeatedly tell the AI that it has to follow these rules. (Sometimes it tries to disable a warning instead. Argh!)

This procedure has worked pretty well in practice, but it doesn’t work perfectly.

If there is interest, I can show how to use this procedure to create a simple in-browser video game. Let me know in the comments!

(written for a friend.)