Vibe Coding, Human Design, and Data Management
Coders — AI is coming for your jobs. Designers — you've got a few more years.
Last night I spent three hours trying to find a bug in my program—a bug I ultimately traced to a misunderstanding injected into my code by an AI coding assistant. My experience demonstrates not just the promise and potential pitfalls of using AI to write software, but the importance of design and data management as AI comes for our jobs.
I started using AI three years ago to help with programming tasks. I’ve been programming since 1973 and I’m pretty good at it, but these days there is such a proliferation of programming languages, frameworks, and deployment environments that it’s extraordinarily difficult for even the most skillful programmers to keep up.
“Vibe Coding” is the term that’s caught on to describe a new way of programming where the programmer simply describes what’s wanted and the AI churns it out. Small programs tend to come out perfect, but larger programs typically have a few bugs that can be eliminated during the course of a few back-and-forth discussions with the AI.
One style of vibe coding is to use a website like ChatGPT or Gemini. When it sees you are programming, it might show your program on the right and your dialogue on the left. You can even run your code! Another style is to use a software development environment like GitHub’s CoPilot or Cursor, a popular AI-assisted editor. These systems have the advantage of being able to reason over a much larger code base—dozens or even hundreds of files, and programs that have tens of thousands to hundreds of thousands of lines.
The current generation of AI systems are quite good at analyzing error messages and log files, and then suggesting ways of fixing a program. Cursor will even add “print” statements to trace a program’s execution and gain new insights, just the way a human programmer might. At some point I expect the consumer tools to start using debuggers; Microsoft recently published a technical report, “Debug-gym: an environment for AI coding tools to learn how to debug code like programmers,” describing their latest efforts in this area.
In my experience, the more detailed instructions that the AI receives, the better the job that it does. This is good news for experienced programmers. Without detailed instructions, the short attention span and inability to make detailed plans of today’s AI systems frequently leads them (and the programmer) down what’s technically known as a “rat hole,” which is bad news for people trying to learn software development. Both experts and beginners alike will benefit from techniques that have been proven to increase reliability and decrease costs, including test-driven development and data management.
For example, it’s easy for a novice programmer to describe a simple web-based application and have the AI come up with a working first draft. But if I want to create a website that’s going to be reliable and maintainable, I will describe in detail for the AI not just the website’s functionality, but the specific set of technologies that should be used to construct it. I will also specify software integrity controls that should be implemented to ensure correct operation, as well as end-to-end testing procedures that should be built into the code base from the beginning.
(If you are interested in what goes into writing reliable software, I recommend the 2014 article and presentation “Mars Code,” which describes how NASA writes code that has to run perfectly the first time, as well as the article “SQLite: past, present and future,” which explains how the world’s most popular database program may well also be the world’s best tested program.)
The second part of using AI reliably is data management—and in particular, source code management—so that it’s possible to track and review changes that the AI makes in your code base. Like many novice programmers, the AIs find it easier to keep adding more code, than to throw things away and start over. Sometimes they delete the wrong stuff. Sometimes they make gratuitous and seemingly innocuous changes in code that breaks core functionality, as happened to me last night.
I know several programmers who won’t touch AI generated code. I do not think that they will be able to keep up as expectations increase with these new tools. This isn’t surprising: Richard P. Gabriel made the same observation in his 1989 essay “The Rise of Worse is Better.” The ideas in the essay are still valid, but some of the terms may throw off the modern reader, so I also recommend his follow-up essay that puts it in context, “Worse is Better.”
In my case, the AI had injected a data-dependent bug that showed up on the production system but not in the test environment, because the two environments operate on different datasets. As a result, I may start making snapshots of the production data and adding them to my test environment. I will also add capabilities for better tests in the production environment itself, an idea that has many proponents but important risks as well.
All of this has significant implications for how we teach computer programming, and more broadly for other fields as well. Traditionally we have started with the simple mechanics of programming — variables, “if” statements and loops — and slowly introduced concepts like modularization and efficiency. I believe that we should flip this approach, and start with design principles, and then spiral out, showing how these principles are applied throughout our engineered systems.
For a real-world example, although it’s possible to have an AI write a completely functional application with a prompt like this — “write a web app that adds two numbers and displays the results,” I want today’s students to be able to write something like this:
“Please create a single page web app for deployment on Amazon Lambda using AWS SAM. The page should have side-by-side text fields, A and B, a push button C, and a result text field D. JavaScript validation running in the client assures that only decimal numbers can be entered into A or B. When the button is pressed, the client makes an API call to the server that receives the two numbers, validates them, and returns the numbers added. A and B are logged in a DynamoDB database along with the IP address of the call and the time. If there are more than 60 calls within a minute the system becomes unresponsive to the IP address for 5 minutes before accepting service again. There are three ways that this can be implemented on the Amazon AWS infrastructure: provide an analysis of the technical and cost differences. When the system is in the unresponsive state, the unresponsiveness should be detected by the web browser and a banner message is displayed. Make sure that the app works equally well for users who are low-vision and no-vision. Start this project by creating tests that independently test the back end, the front-end, as well as end-to-end tests. The back end should use Python Flask and pytest as a testing framework. Use poetry as a package manager. The front end should use React and Playwright as a testing framework. All tests should be able to run locally on the developer’s laptop or in GitHub using GitHub actions. Code coverage for the tests should be recorded in HTML pages that are stored as GitHub artifacts. Create a detailed plan for development that integrates between creating each test, running the test to demonstrate that it fails, and then implementing the test so that it works. After each step, perform a git commit with a detailed commit message. Finally, also create a testing script that tests the deployed system with a second AWS Lambda function that runs every 5 minutes and sends an alert email when the system goes offline, followed by an alert every hour that it remains offline, and a final alert when proper service is restored. That functionality will also require AWS DynamoDB to maintain state.”
(Despite what Sam says, I try to say “please” and “thank you” to my chatbots: I want to live in a world where people and machines are mutually kind and polite to each other. And it’s possible that the AIs will use this information to decide who to keep around as pets when they finally take over.)