6 Key Parts of an Automated Regression Test Pack

What is an automation regression test pack?

Your automated regression test pack is your primary quality gate. It should run on an environment as close to production as possible and ideally before exploitative testing efforts start.

The outcome of the regression test pack is not to tell you everything is fine, "send it". But to save your team time while giving you the confidence to move on to the next phase, typically time-consuming manual testing. This should not be understated!

A good regression test pack will save time!

What are the 6 Key Parts of an Automated Regression Test Pack?

Every regression test pack should be shaped according to the industry and purpose of the software. But every good regression test pack will include the following 6 key parts of an automated regression test pack:

1 - Critical Business Flows

These are the areas that, if broken, fundamentally mean your service is down. If they occur on production, you would roll back a release.

For an e-commerce website, this would be adding items to a cart and performing a complete card checkout.

Most of these will likely be obvious to you, but it's always worth asking around to see if anyone else comes up with different items on the list. Some Critical Business Flows might be more hidden. Your company may have customers or partners with specific flows that need coverage.

2 - High-Risk Areas

These are areas of particular complexity or frequent change.

Continuing to use e-commerce as an example, this might be "discount stacking," where the user has six "3 for 2" items in their cart, plus a "10% off" discount code, and is trying to use up an old gift card before paying the rest of a credit card.

I suggest talking to developers to get a list of areas that send a shiver down their spine.

3 - Negative Testing

For every positive test, there are dozens of negative tests. Negative tests are those off of the "happy path" or "golden thread". These include validation messages, failed action errors, and anywhere where your application tries to nudge the user back onto a "happy path".

Here are some common examples:

• The inputted email address on a register form already has an account.

• An item in a user's basket becomes "out of stock" before checkout.

• A user tries to set a password that doesn't meet the minimum requirements.

Manual/behavioral testers are the go-to here. They might already have a list but will easily come up with many on the spot.

4 - Browser and Platform Coverage

Luckily, browsers have matured and are more similar and reliable than ever. But user fragmentation is still on the rise. This fragmentation mainly comes from mobile devices, specifically, the vast variation in screen sizes.

To give users the best experience at different sizes, most websites and web apps will include media queries to alter site behavior based on viewport size. For instance, the primary navigation will often be replaced with a "burger menu".

Using these alterations should be included in your automated regression test pack. Ensure your regression tests run on multiple browsers and various screen sizes.

Talk to whoever manages the analytics account. Ideally, you'll gain access, but if you still need to, ask them to provide a breakdown of the devices, browsers, and resolutions to help guide your prioritization.

5 - Regression Tests for Bug Fixes

Include tests that prove previous defects are still fixed. Your users don't want to see old bugs return, nor does anyone internally. Regressing on previously found and fixed issues can cause a lot of stress in what might be a well-functioning team otherwise.  

Murphy's law of "anything that can go wrong will go wrong" is a useful guide here. If something has previously shown itself to be possible to go wrong, it likely will again.

You'll probably be able to run a query on whatever tool you use to manage work. If you need to ask someone, your Scrum Master or whoever manages work tickets should be able to help. 

6 - Integrated User Journeys

Your automated regression test pack should be fully integrated, and you should write tests as if a real user is using your site.

Avoid tests that jump (navigate using the address bar) to a page and perform an action before jumping to the next. Users navigate, and so should your tests.

Avoid mocking wherever possible. Unit and integration testing is like tasting each ingredient of a cheeseburger; it's a good idea and will save you time and money if any ingredient turns out to be "off". But your customers will consume it differently. To truly know if you have a good cheeseburger, you need a finished cheeseburger with all its parts and take a big old bite.

This one is more of a guide on how to create your tests. They shouldn't be small blocks but connected chains. Analytics can help you again here, but I suggest talking to custom services, too. 

Room for a part 2!

I could have easily made this the 10 or 12 key parts of an automated regression test pack, covering localization, accessibility, security, and even SEO. But I'll save them for part 2, "Extending your automated regression test pack."

I hope you found this helpful. If it did, please share.