Let's revisit "The Pyramid"
Jan 8, 2024
Like the Pyramids of Giza, the Testing Pyramid has remained largely unchanged for eons.
But whereas one is literally set-in-stone, the other was more of an idea. A pragmatic idea to help development teams succeed with agile. But much has changed since Mike Cohn first published Succeeding with Agile in 2009, and it might be worth another look.
Like a sandwich from 2009, a sniff-test is overdue.
I was a developer before 2009, and everything was a mess! Some places were probably more organized or modern. But what I saw most commonly was engineering teams made up entirely of developers. I can't remember where I was when I first saw a full-time tester in a dev team. Often, they were bought in for short periods to test everything manually, then disappeared. It was the Dark Ages!
Unit tests were rare, and it would be a few years before anything viable arrived in the end-to-end space.
So back when Succeeding with Agile was published, it was the best remedy. Development teams were writing better code by writing a bit more code to test their code... as they wrote the first code. And it works. Code quality improves without even running the unit test. This is partly due to the developer thinking about their code from a second perspective.
Develop Quality!
Companies want features and services to make money, so "testing" has always been hard for bosses to swallow.
"I'm already paying for developers; why should I also pay for testers just to tell me I'm being ripped off?" is a common sentiment - even if it's not expressed verbally.
Unsurprisingly, companies want the highest possible quality directly from the developers. And so, the Test Pyramid has become a foundational part of the development landscape.
High quality, but not high confidence
Quality code is quite an open term. We assume it means working, but different people will have different opinions. I've worked with many developers who will think quality is about how something is written (use of libraries, code structure, runtime efficiency etc). So naturally, when they write quality unit-tested code they skew towards their preferences.
This could sound contentious, so I'll give an example. Assume a function should have a known input, do some processing, and produce a known output. Simple, perfect, easy, without any room for error. They write what they consider quality code; they write an amazing comment, format the function to be easily understandable, and they write a few unit tests.
The code is peer-reviewed, and standard testing is performed. No one says anything, so it's promoted to live.
What's the problem?
Well, "should have a known input". This function doesn't exist in isolation; without looking at all the upstream code, that "should" requires a lot of trust. Another developer might take a different approach to quality, one that produces more resilient code, regardless of the unit testing.
They should have raised this at the pull request stage. But it's working; the unit test proves it. I'm sure all developers reading this have been in many conversations where one person says something is done, and another thinks it could be done better. They usually end with nothing changing.
Focus on confidence, not quality.
Confidence is what matters. You can have high-quality blocks of unit-tested code and still have zero confidence. Why? Because users are making you aware of issues.
What you need is end-to-end automation tests. A large automation regression pack that is run on every change. One that gets updated with every defect a user finds not only to validate that the specific defect is fixed but also ensures it can never regress.
"The Pyramid"
This can and should be done in every engineering team. But there's an obstacle. This thing that keeps getting in the way. It's like a really wide block of stone blocking your path. One that is hard to get around as it's just as wide as it is long. And you can't easily get over it either with it's perfect 60° slope... I'm describing a pyramid, "The Test Pyramid", let's move on.
I've heard "leaders" in testing say in public talks that they routinely delete automation tests with their PowerPoint slide of "The Pyramid" illuminated in garish colors behind them. I've heard others use "Shift-Left" as a way for QA Teams to blame developers for bad code. Yes, any bad code is the developer's fault, but the QA job is to support the engineering team by catching these before they go out, right?
We need to stop using "The Pyramid" as anything more than 2009s pragmatic idea to help improve quality. Its purpose is to increase testing efficiently, not set limits or not argue for its reduction.
So what's 2024s Pragmatic Idea?
Do more testing more quickly for less money.
This might sound like a joke or the start of the "sales pitch" part of the post, but it's not.
Two of the most significant differences between 2009 and now are:
The cost of good developers has rocketed
The cost of good automation testing has plummeted
Ask yourself; if you could have all of your most common user journeys as full end-to-end automation tests and be able to run them in single-digit minutes as part of your CI/CD. Or you could have 100% unit test coverage. What would give you more confidence?
In 2009, one was almost impossible and could double the engineering cost if pursued. The other would slow each feature down a little bit.
Now, in 2024, the benefits of unit tests remain the same, and you should do as much as you can while remaining efficient. Full end-to-end automation, on the other hand, has seen a revolution. Test automation has become so easy and so cost-effective it's astonishing to see "The Pyramid" being used as a prescribed ratio of end-to-end to unit tests.
We say don't limit your automation testing. Test as much as possible, then relax and enjoy your increased confidence.
You're overdue if you haven't investigated end-to-end automation testing in the last few years! Take a look at what's new in the codeless test automation space and give a few tools a go! Of course, I suggest starting with DoesQA.
