By Ramon Schwammberger and the Platform Engineering Team

Modern platforms don’t just evolve — they accelerate. As release cycles get faster and infrastructure becomes more complex, even systems that have worked reliably for years can start to slow everything down. This is the story of how we evolved our testing system to support multi-platform environments, not by rebuilding it from scratch, but by carefully untangling and upgrading it while keeping the business moving.

We often say, “Never change a winning team.” Why touch something that works? It’s a mindset you see everywhere in IT. So, we focus on building new capabilities, pushing innovation forward, and delivering the next big thing—while the systems that got us here quietly remain unchanged in the background. That is, until they begin to hold us back – not because they suddenly break, but because everything around them moves faster.

That’s exactly what happened to our testing system. It wasn’t failing. In fact, it was still doing its job reliably. But at the same time, our platform was evolving – with faster release cycles of our software, of the underlying operating system, the kernel and tooling. That shift introduced a simple requirement on paper: we now had to make sure everything worked both on the existing OS version and the new version. In practice, that meant running the same tests in two different environments and being confident in both results.

The costs of a green field

We all wish for a green‑field scenario — the chance to build a new testing system from scratch, with today’s requirements in mind, clean boundaries, and proper platform awareness from the start. In reality, that path comes at a cost. We knew that building a “perfect” system would take significantly longer than evolving the one we already had. And while we would be working on that redesign, our fellow engineers would still need to validate their code on both operating system versions manually, effectively doubling their effort. For the company, this would have meant higher infrastructure costs, while customers would have experienced less reliable performance and slower feature delivery. Not to mention, that testing itself would get too complicated and intertwined. So, when push came to shove, we went for a brown-field approach.

Introducing a third dimension

Up until that point, infrastructure had been implicit. There was one version of the operating system and tooling. The testing system was only concerned with installing the correct software stack on top and running the tests against the various configurations. The moment we introduced a new version of the operating system, that assumption disappeared. Infrastructure was no longer fixed—it became something the system had to understand and act on. Switching from one version of the operating system to the other requires a reboot of the system. Developers were no longer just defining what should be tested. They were also defining under which conditions: configuration, software stack, operating system version. That meant infrastructure had to become a first-class input to the system.

Rather than trying to extend the existing implementation, we took a more deliberate step of redesigning one component responsible for triggering and scheduling tests. It was a collection of tightly coupled scripts. This made it very hard to understand the system or predict what the possible outcomes are. We decided to replace those scripts with one piece of software that deals with all the decisions and is easy to test. The key constraint was that everything around it had to keep working. Teams had already defined their tests, and asking them to rewrite them was not an option. So instead of changing the interface, we kept it exactly as it was and introduced a new implementation behind it. That allowed us to control the change. We could first replace the core behavior, verify that the output stayed the same, and only then begin introducing the new capability of running tests across multiple platforms.

Stress testing our untangling skills

It was at that point that the system started to reveal its hidden complexity. As soon as tests began running across both versions of the operating system, the that should have been completely independent suddenly stopped working. One example was the release process. Even when tests passed, the automated step that moved software into a production-ready state broke. Nothing in the change directly touched that logic. And yet, it was affected.

What this exposed was something we hadn’t fully appreciated before: the system was much more tightly coupled than it looked. The build, test and release steps were interconnected in ways that were not visible until something changed. Remember the collection of scripts that we replaced? The build and release steps had similar scripts and they all interacted in subtle ways. We didn’t try to untangle everything at once. That would have taken too long and would have brought us back to the green-field scenario we had consciously avoided. Instead, we did something more pragmatic. We replaced the component we needed, kept everything else stable, and used what we discovered to guide future improvements.

The coupling remained—but in smaller clusters.

Overcoming physical challenge

The next challenge was less architectural and more physical. Running the same tests in two environments is easy—if those environments can simply be duplicated. In the cloud, that is mostly a question of cost. But not all tests run in the cloud. Some depend on physical hardware. And hardware is limited—by rack space, by power, and by the fact that you can’t just duplicate it on demand. So instead of duplicating environments, we had to reuse them. The solution we implemented was straightforward in principle. For tests that required physical machines, the system would run them on the existing platform, reboot into the new operating system version, run the same tests again, and finally restore the original state. What matters here is not the idea itself—it’s the fact that it was fully automated. From the perspective of the engineers, nothing changed. They didn’t have to manually switch environments, repeat tests, or coordinate anything. The system handled the entire sequence for them.

Locked in reliability

But automation came with its own complexity. The testing process was not a single, linear flow. It consisted of many smaller steps, distributed across different systems. Some steps prepared infrastructure, others ran tests, and some required coordination across multiple machines. Duplicating all these steps for two versions of the operating system

In an ideal system, this would be solved at the design level—by restructuring the execution model and reducing fragmentation. But again, that would have required a much larger redesign. So we introduced a mechanism to control access instead. Whenever a test started running on a specific host, that host would be locked for the duration of the execution, ensuring that no other process could interfere. This wasn’t the cleanest solution. Locking introduces sequencing and limits parallel execution. It’s something you normally try to avoid. But in this context, it was the right trade-off. It allowed us to make the system reliable without stopping everything else.

Perfect is the enemy of good.

Looking back, the most interesting part of this work wasn’t any individual solution. It was the realization that systems rarely become complex because someone designs them that way. They grow that way—slowly, incrementally, and often for good reasons at the time. What we saw in our testing system was exactly that.

Today the system is not perfect. But it is fundamentally more capable than before. Tests can now run across multiple platforms without requiring manual intervention. Engineers no longer have to duplicate their work, and the system no longer slows down platform development—it supports it. We now have a much better understanding of where the system is tightly coupled and why. We have seen how complexity accumulates when responsibilities are not clearly separated, and how even small changes can expose hidden dependencies. This has made it easier to reason about the system—not just as it is today, but as it needs to evolve. The long-term direction is clear. The testing system should move closer to behaving like a platform—one where engineers define what they want to test, under which conditions, and the system takes care of the rest.

Getting there will take time. It will require further simplification and, eventually, a deliberate effort to reduce the remaining coupling between components. But that was never the goal of this phase. Instead of trying to fix everything at once, we focused on the change that mattered most—enabling tests to run across multiple platforms—and built around that. Not a perfect solution, but a meaningful step forward.