When software stalls the starting grid – Why testing matters

A software problem at the Malaysian F1 race earlier this month makes a useful point about technology projects - testing needs to go beyond checking that software does what it is supposed to do under normal conditions.

The harder question is what happens when conditions change or several individually familiar conditions occur together.

What happened?

The FIA’s investigation found that some cars running at low engine speeds in certain circuit sectors suffered an unintended loss of power when a 250kW wet-weather power reduction was applied. The cars could not recover that power within those sectors, disrupting the race start procedure.

Before the restart, the FIA required all teams and manufacturers to apply a software update. This lifted the restrictions associated with the 250kW limit in the sectors where the problem had occurred.

The software had been tested. Teams and power unit manufacturers had run simulations, followed by full-scale testing during practice sessions at the Belgian and Dutch Grands Prix. Even so, the problem only emerged in competition.

For technology suppliers and customers, that is the uncomfortable part of the story. A successful test tells you something about the conditions tested. It does not necessarily tell you what will happen outside them.

Beyond the “happy path”

A business system might pass testing with a small set of clean data, then struggle with years of customer records containing duplicates, missing fields and old formats. An ordering system might work reliably until its connection to the warehouse drops halfway through an order. Or problems might only appear when a scheduled update coincides with a busy trading period, such as during Boxing Day sales.

These are the sorts of scenarios that can get overlooked when testing focuses on the “happy path”, where expected users take expected steps under normal conditions.

A useful test plan should also look at unusual inputs, limits, integrations, configuration changes and recovery after a failure. For that interrupted order, for example, does the system lose it, send it twice or hold it until the connection is restored? Calling something an “edge case” should not end the discussion. An uncommon event may still deserve close attention if the consequences are serious.

There will always be limits to what can be tested. The practical task is to decide which scenarios matter most, record what will and will not be covered, and understand the risks left over.

Put the testing arrangements in the contract

Testing often depends on work from both sides. The supplier knows the software; the customer knows how the business actually operates, including the workarounds and exceptions that may not appear in a requirements document.

The contract should turn that knowledge into a workable testing process. In particular:

• Scope: Which functions, integrations, configurations and operating conditions will be tested?

• Inputs: Who supplies the test environment, representative data and access to third-party systems and by when?

• Acceptance: What counts as a pass, how long does the customer have to test, and which defects prevent acceptance?

• Remediation: Who investigates a failure, fixes it and carries out retesting? Who bears any resulting delay costs?

• Updates: What regression testing is required to check that a change has not broken something that previously worked?

These details are worth settling before testing starts and are often set out in a statement of work.

The obligations also need sensible limits. A supplier cannot price an undefined commitment to test everything, and a customer gains little from acceptance criteria that leave too much room for argument.

Have a plan for failures that testing misses

The FIA identified the cause and released revised configurations within minutes of suspending the start procedure. It also committed to stronger testing and verification.

That response is another part of the lesson. Before software goes live, suppliers and customers should know how a problem will be detected, who needs to be contacted, and who can authorise a rollback or other recovery measures.

When a serious fault appears, the immediate priority is getting the system working again. The contract should help the parties to do that and not leave them working out who is responsible while the business waits.