Skip to content Skip to footer

The test comes before the code

In brief. Amazon had a working service against which to compare the one built by agents. To extend the same method you have to decide in advance what counts as a correct result. Amazon states that all the code of AWS Emulator was written by agents.

I opened GarageBand without knowing how a song is built. That was exactly why I wanted to try: pick a piece of software in which I could not find my way around and ask an agent to work in it. The request was a Nineties dance track. The result arrived, and I have talked about it in my talks.

I could listen to it. I could decide whether I liked it, whether it recalled the genre I had asked for, whether it deserved another attempt. It was a useful test to understand how much an agent could do inside an application. It did not prove it had carried out every step in the best way, and I did not need that to enjoy the experiment.

When software has to calculate the price of an order, the situation changes. A total cannot just have the right style. It has to respect rules, conditions and exceptions that someone knows how to verify.

Amazon already had the answer

On November 11, 2026 Amazon presented AWS Emulator, a program that reproduces locally the behavior of services in its cloud so that developers can test against them. The company states that agents wrote all the code. They had two references: the service specifications and the real cloud, against which to compare the emulator’s responses. Amazon · The development process

It is this availability of a comparison that makes the case interesting. A request can be run on the existing service and on the one under construction. If the results differ, there is something to investigate. Agents can prepare tests, find discrepancies and fix the program within a path that makes the expected result observable.

For a manager, the useful question comes before the programming language: where is the answer against which to compare the work? Sometimes it is in a system already in operation. Other times it is in a closed and approved file. In many cases it is spread among people who remember different agreements.

The discount had been promised verbally. The ERP shows the list price. The salesperson remembers another figure and the customer expects to pay it. Before automating the quote, someone has to establish which total the program should return. The agent can help retrieve the documents; the decision on the exception remains the company’s.

This is design work. Writing a rule forces you to specify who it applies to, until when and in what circumstances. The exception also has to become recognizable: an authorization, an expiry date, a limit. As long as it depends on a phone call to one person, the system has to know that the phone call is needed.

A good test can confirm an error

The reference against which we compare the new program can be wrong too, and passing every test does not remove that problem. Old software can embed an outdated rule. An approved file can contain a tolerated error. Reproducing it faithfully means automating the past, bad habits included.

This is why the source of the expected result has to be chosen, not simply found. If the project has to replicate an existing behavior, the comparison measures that fidelity. If it has to improve it, you need to state at which points the behavior must change. Asking at the same time to copy everything and to fix what is wrong, without specifying it, leaves to the machine a decision we have not made.

This is exactly where experienced people are needed. They can select representative cases, identify exceptions and explain why an apparently correct answer is not acceptable. Their contribution becomes reusable when it enters a test that can be repeated, instead of remaining a comment made during a demo.

The AWS documentation also makes clear that Emulator is intended for development and testing, not for production workloads. It further specifies that it does not reproduce permission evaluation the way the real service does and should not be used to verify security. These are essential limits: passing a test does not certify everything that test does not observe. AWS · Documentation and limits

A company building a quoting application has to draw the same distinction. A correct total does not prove that a person without permission cannot change the price list. A good answer on a simple order does not automatically cover a return, a different currency or an interruption halfway through an operation.

The first file

To start, the team can choose a bounded process and collect examples with a reasoned result. Before generating more code, it has to be able to explain by hand why each example is right. When two managers disagree, they have already found a useful problem to solve, without waiting for launch.

Among the correct results there can also be a stop: data is missing to calculate the price. The test has to reward this answer when it is the right one. Otherwise it teaches the system that producing a total anyway is worth more than recognizing missing information.

Then you verify that a fix does not break the cases already settled. The test bench grows with the process and preserves the decisions made. It is a less eye-catching investment than the first application that appears on screen, but it makes it possible to entrust it with real work.

With the song I could stop at listening. With a quote going out to a customer, someone has to be able to say why that number is correct even when nobody is in front of the screen any more.

Read next An agent’s identity is not enough to get the truck moving · One department’s productivity, another’s queue

Background Impact · the case file (in Italian)

Image: Amazon