6-hour Production Automation Test Pack

100% of issues found by users happen on your production system. With the increasing rise of Shift-Left testing, we have seen the gap between testing and end-users widen into a chasm.

I'm not going to argue against shift-left testing (today 😜), but instead, highlight the growing need for some testing on the right-hand side: an automated test pack running on production.

Regardless of any other automated tests you have, a production test pack will bring you and your stakeholders peace of mind.

Let's list the requirements of our Production Automation Test Pack.

Reports

Let's start from the right and shift left 😊; we'll need reports. Ideally, we'll be able to receive an email or Slack notification if the pack ever fails. We'll want to view all the passing runs to help identify the route cause as fast as possible.

Runners

We need to be able to run this test pack every couple of hours. Runners should be fast, reliable, and run in parallel. We'll want the results to be quick and trustworthy and not have one issue blocking us from discovering more.

Schedule

We may want to avoid any extra site traffic during our busiest periods, and if we want our test to run every few hours, we will need a schedule.

CI/CD

We want to kick this production test pack off immediately after a live/prod deployment. There's no point waiting for the schedule if we know something new could cause issues.

The Tests

We need tests to run! Let's not worry about our 6-hour time restraint and figure out what we would want if we could.

— Functional Smoke Tests

We'll want to test site navigation, registration, login, and user account areas as a standard. If the site is e-commerce, we'll also want to create a few baskets of different products, like one-size items and gift cards, and check the checkout summaries and subtotals are all correct. We'll need to do calculations on dynamic values for this. We're not trying to create a full regression pack here, just a light smoke test (at least to start with).

— Lighthouse Tests

Accessibility and Performance metrics can easily drift down if you don't monitor them. We'll want a way to set the lowest acceptable lighthouse scores for these, and ideally, SEO, to make sure we catch any slow regressions. Production is the best place for these metric tests.

— Visual Regression

Certain pages and elements are not changed frequently, so Visual Regression would be perfect for validating that everything is as expected. Visual Regression also adds a massive layer of protection from the dangers of functional-only testing (but that's for another day).

Other factors to consider

Our production system might have bot protections; we may be unable to test anything if we can't get through the production firewall. Luckily, there's always a path through, whether that's IP white-listing (we'll need static IPs), a cookie (which we'll need to be able to set), or custom headers (sounds complex, but isn't).

Why the 6-hour time restraint?

Most people's work day is between 7 and 9 hours long, and given interruptions like meetings, the best you can hope for in a single day of actual work is about 6 hours. If you can make a task doable in a single day, it has a much higher chance of happening.

A production test pack brings so many benefits, even a 6-hour one, that it's far better to have something today than plan for something larger in the future, which then never happens.

But is this possible in 6 hours?

Yes, 100% you can do all this and more in 6 hours using DoesQA. Everything above is available right out of the box; all you need to do is create the elements and start drag-n-dropping the nodes into place.

Schedules, Slack integration, and CI/CD will only take minutes. Runner setup won't even take seconds; they're ready to go whenever you are. Every Node you could ever need is at your disposal. We use static IPs, so there are no headaches there, either.

But is it possible to do all this with code?

Maybe, we doubt it, but we have spoken to many die-hard selenium/cypress/playwright automation testers who have told us they can do everything DoesQA can do just as quickly in code. So I guess if the individual has the required skills, experience, caffeine, and the 6 hours.

If you can do this, we'd love to hear from you!

Conclusion

A production automation test pack is something everyone should want, regardless of all other automation tests taking place elsewhere.

You can start small, but you should start!

Do all this and much more with a DoesQA. If you get stuck at any point, reach out, and we'll jump in!