A beautiful mess

A tale of growing project needs

The beginning

Your new project works. You feel awesome! It’s succeeding at attracting a few users. There’s potential. You are getting new developers onboard.

You wanted things to be simple so you made your project run in docker. The build is reproducible. It is so much simpler to test. Testing is a simple GitHub Actions workflow file.

It starts simple

You started with Debian 10. Debian 11 came out but you still have 10 in production. Let’s support both for now. You now have two (2) build configurations.

Builds = debian_version

Some issues are tricky to diagnose. You create a debug version of your project to help debug issues. You now have four (2*2) build configurations.

Builds = debian_version x is_debug

You want your product to be secure! Since you use an unsafe language (C++), you want to use sanitizers¹ provided by clang/LLVM. Getting the build green is tricky, and running it is slow. To save on capacity, you only run in debug mode, even if you know that running on release would likely catch more race conditions. You now have six (2*(2+1)) build configurations.

Builds = debian_version x (is_debug NOR is_sanitizer)

ARMv8 (64 bits) VMs are the new hot thing. Early adopters request support to save on costs, but current customers can’t move yet. While at it, you create an ARMv6 (32 bits) version. It doesn’t cost much and it enables leveraging the old Raspberry Pi you had in a drawer. You now have eighteen (2*(2+1)*3) build configurations.

Builds = debian_version x cpu_arch_3 x (is_debug NOR is_sanitizer)

More OSes

The team grows. Your developers got the newest MacBook Pros with physical function keys on their keyboard. They are so happy! They would like to run the project locally and not inside docker to improve their edit-debug-test cycle. You don’t hate your developers, so you test this configuration. Some unlucky got Intel based laptops, others got the new M1’s. You don’t want developers to have to install Rosetta 2 so you build native for M1 too. You hope to never support RISC-V and start regretting ARMv6 but it’s too late now. You now have twenty four (((2*3)+(1*2))*(2+1)) build configurations.

Builds = (debian_version * cpu_arch_3) OR (is_macos * cpu_arch_2)) x (is_debug NOR is_sanitizer)

Your project uses python3. Your project promise compatibility with the two last python releases. Thankfully container users use whatever is preinstalled, but folks on macOS tend to use homebrew and have various degrees of broken local setup. You now have thirty (((2*3)+(1*2*2))*(2+1)) build configurations.

Builds = (debian_version * cpu_arch_3) OR (is_macos * cpu_arch_2 x python3_version)) x (is_debug NOR is_sanitizer)

Apple just released the newest macOS beta! It has new edge cases that have to be tested. It has to be tested on old Intel hardware too. You now have forty two (((2*3)+(2*2*2))*(2+1)) build configurations.

Builds = (debian_version * cpu_arch_3) OR (is_macos_beta * cpu_arch_2 x python3_version)) x (is_debug NOR is_sanitizer)

Reality sets in

VC money starts to run out. Fun time is over, time to pivot and create a product to start charging users. This is done with a private code base overlaid on the top of the public code. The code has to work in both configurations and you don’t want to break users of the open source code because it would create a backlash. You now have eighty four (((2*3)+(2*2*2))*(2+1)*2) build configurations.

Builds = (debian_version * cpu_arch_3) OR (is_macos_beta * cpu_arch_2 x python3_version)) x (is_debug NOR is_sanitizer) x is_public

While your development team uses both private and public trees on top of their corresponding main branches, releases are done on top of the latest release branch. The best way is to develop is against the last released public code but you need to test with tip-of-tree too. You now have one hundred and twenty six (((2*3)+(2*2*2))*(2+1)*(1+2)) build configurations.

Builds = (debian_version * cpu_arch_3) OR (is_macos_beta * cpu_arch_2 x python3_version)) x (is_debug NOR is_sanitizer) x (is_public OR is_stable)

Founders decided to cash in, partly due to VC pressure and lack of customer acquisition. It finally happens: you are acquired by Microsoft. They ask you to expand native support to Windows 10 and 11. Thankfully, nobody request Windows on ARM support … for now. You are not able to make sanitizers work when using MSVC++ and you just give up … for now. It doesn’t increase the number of configurations as badly as you feared. You now have one hundred fifty (((((2*3)+(2*2*2))*(2+1))+(2*2*2))*(1+2)) build configurations.

Builds = (((debian_version * cpu_arch_3) OR (is_macos_beta * cpu_arch_2 x python3_version)) x (is_debug NOR is_sanitizer) OR (_windows_version_ x python3_version x is_debug)) x (is_public OR is_stable)

How it ends?

Generally badly. Either you have a strong EngProd team and the build configurations are carefully curated or you end up with an organically grown mess. 🌿💐🌳

We didn’t even talk about Profile-Guided-Optimization (PGO), partial optimization, simplified debug symbols, various level of runtime checks or logging, and scaling to a constellation of various dependency versions.

How do you decide which configuration to test in practice?

How do you declare these test configurations?

How do you ensure maintainability?

Let’s talk about this next.

---

1. Fuchsia uses various sanitizers in its CI. They help make the OS more secure! ↩︎