The first rule of Test Automation
Jul 26, 2024
Sam
Now, now, which one to choose. So many people have different reasons for choosing to automate their testing. Faster feedback? Dependency reduction? More coverage?
All are valid, but there is one key rule to starting any test…
Test Determinism
A test case should be repeatable. There is an age-old example of how this can be done badly.
Reset Password Journey
Given I am a registered user
And I am able to log in with my current credentials
When I update my password
Then I should be able to log in with my updated credentials
And my old credentials should no longer work
Automating the above scenario sounds pretty straightforward. However, some people take different approaches to this.
If your mind goes to “Cool, I have a user registered already, so will log in with those” - you have already fallen into a trap!
What Trap?
You are walking straight into a trap. What happens if:
The test fails on the “Then I should be able to log in with my updated credentials” step?
I run 2 of these tests at the same time?
The user is deleted
In order for you to have a known state - You have to ensure that you set up and tear down your state.
Setting up your state
In this example - would be to either:
Create a new user at the start / before running the test
Update the users password to a known state prior to starting the test (although good password storage principles would inhibit this)
Tearing down / Cleaning up
Not that this is a requirement, but it is always nice to leave the place how you found it!
Once your test is complete (regardless of pass or fail), either:
Delete the newly created user
Reset the password back (same issues as setup apply)
Teaching you to suck eggs.
Now you might be saying, “Sam, why are you saying something so obvious” - and you would be justified to do so. But this principle, or rule, can apply to a multitude of different situations.
Such as:
Placing an order on an e-commerce website, and validating an email is received
If you use the same email address each time, how do you know it’s failed?
Buying a product without first adding / setting stock can lead to inconsistent results
So although this may seem silly to mention, take a look over your tests and see if you find anywhere that you are missing this rule. Take a look, and then you can silently thank me, all whilst making your quality that bit better!
Now, now, which one to choose. So many people have different reasons for choosing to automate their testing. Faster feedback? Dependency reduction? More coverage?
All are valid, but there is one key rule to starting any test…
Test Determinism
A test case should be repeatable. There is an age-old example of how this can be done badly.
Reset Password Journey
Given I am a registered user
And I am able to log in with my current credentials
When I update my password
Then I should be able to log in with my updated credentials
And my old credentials should no longer work
Automating the above scenario sounds pretty straightforward. However, some people take different approaches to this.
If your mind goes to “Cool, I have a user registered already, so will log in with those” - you have already fallen into a trap!
What Trap?
You are walking straight into a trap. What happens if:
The test fails on the “Then I should be able to log in with my updated credentials” step?
I run 2 of these tests at the same time?
The user is deleted
In order for you to have a known state - You have to ensure that you set up and tear down your state.
Setting up your state
In this example - would be to either:
Create a new user at the start / before running the test
Update the users password to a known state prior to starting the test (although good password storage principles would inhibit this)
Tearing down / Cleaning up
Not that this is a requirement, but it is always nice to leave the place how you found it!
Once your test is complete (regardless of pass or fail), either:
Delete the newly created user
Reset the password back (same issues as setup apply)
Teaching you to suck eggs.
Now you might be saying, “Sam, why are you saying something so obvious” - and you would be justified to do so. But this principle, or rule, can apply to a multitude of different situations.
Such as:
Placing an order on an e-commerce website, and validating an email is received
If you use the same email address each time, how do you know it’s failed?
Buying a product without first adding / setting stock can lead to inconsistent results
So although this may seem silly to mention, take a look over your tests and see if you find anywhere that you are missing this rule. Take a look, and then you can silently thank me, all whilst making your quality that bit better!
Now, now, which one to choose. So many people have different reasons for choosing to automate their testing. Faster feedback? Dependency reduction? More coverage?
All are valid, but there is one key rule to starting any test…
Test Determinism
A test case should be repeatable. There is an age-old example of how this can be done badly.
Reset Password Journey
Given I am a registered user
And I am able to log in with my current credentials
When I update my password
Then I should be able to log in with my updated credentials
And my old credentials should no longer work
Automating the above scenario sounds pretty straightforward. However, some people take different approaches to this.
If your mind goes to “Cool, I have a user registered already, so will log in with those” - you have already fallen into a trap!
What Trap?
You are walking straight into a trap. What happens if:
The test fails on the “Then I should be able to log in with my updated credentials” step?
I run 2 of these tests at the same time?
The user is deleted
In order for you to have a known state - You have to ensure that you set up and tear down your state.
Setting up your state
In this example - would be to either:
Create a new user at the start / before running the test
Update the users password to a known state prior to starting the test (although good password storage principles would inhibit this)
Tearing down / Cleaning up
Not that this is a requirement, but it is always nice to leave the place how you found it!
Once your test is complete (regardless of pass or fail), either:
Delete the newly created user
Reset the password back (same issues as setup apply)
Teaching you to suck eggs.
Now you might be saying, “Sam, why are you saying something so obvious” - and you would be justified to do so. But this principle, or rule, can apply to a multitude of different situations.
Such as:
Placing an order on an e-commerce website, and validating an email is received
If you use the same email address each time, how do you know it’s failed?
Buying a product without first adding / setting stock can lead to inconsistent results
So although this may seem silly to mention, take a look over your tests and see if you find anywhere that you are missing this rule. Take a look, and then you can silently thank me, all whilst making your quality that bit better!
