Crafting A Perfect Quality Sandwich
May 23, 2024
Sam
The Quality Sandwich
This article may sound like the “Best BLT Recipe”. It’s not — But I assure you, it's just as tasty!
Many Quality teams worldwide face a similar problem: tests clash when running at any level of concurrency simultaneously.
The solution to this problem, while it appears complex, is actually really simple. Following easy steps, you can ensure that no matter the test case you are designing, you can be safe knowing that it will run at Unlimited* concurrency.
Why the asterisk, Sam? I hear you ask. Well, some environmental factors can impact your ability. Continue forward and see if you have any of these or if you can find a solution to said issues.
Now, let's move on to the ingredients for great automation testing at scale.
Your outer layer - Test Setup
In this “bread” layer, you set up everything needed for the test execution to be able to run without requiring exterior help. Need some products to add to a bag? Find them here, and then pass them onward in a known format for your tests to consume.
The filling - Test Execution
With the correct test data/setup configured, your execution steps should now use that and run whichever test they are meant to. Use the data provided and create tests that are dynamic enough not to be tied down to a particular environment or data set.
You now have something that can run on production, all the way down the stacks onto a local development machine.
The final layer - Test Teardown
With the test complete, regardless of the outcome, you should clean up anything you can to reset the application's state. While having completely fresh data every time is preferable, some integrations or functionality might hinder this. Clean up your mess so the next run can have a fresh start.
If executed correctly, you will end up with a quality sandwich. Now that you have been introduced to this concept, you might think that you are in a strange position.
Meat Bread Meat.
If your functional test code calls out to APIs or other areas to get/set information mid-test, you risk the tests being too tied to a particular data shape or value. If the API schema is updated for whatever reason, you are left with a failing test, but the issue is not what the test was designed to check; it is failing on data.
Like a sandwich in the real world, this works but is not the ideal format. Because it's nested so deep inside functional/test code, you end up losing the structure and compromising the confidence in the data.
Additionally, when running tests with concurrency, you may end up with race conditions, whereby one test relies on one state and another differently.
Setting up what you need in the functional filling layer outside allows you to control this with hooks such as Before Each or Before All, and similarly with the teardown, After Each or After All.
Your framework may not have these steps or hooks, but it's easy enough for you to add them yourself; before calling the test script, run another file, and after execution is complete, run the teardown functions.
After reading this, you may feel like a refactor is in order, but trust me, once you do, you will notice an immediate and massive difference in the agility of your automated tests.
The Quality Sandwich
This article may sound like the “Best BLT Recipe”. It’s not — But I assure you, it's just as tasty!
Many Quality teams worldwide face a similar problem: tests clash when running at any level of concurrency simultaneously.
The solution to this problem, while it appears complex, is actually really simple. Following easy steps, you can ensure that no matter the test case you are designing, you can be safe knowing that it will run at Unlimited* concurrency.
Why the asterisk, Sam? I hear you ask. Well, some environmental factors can impact your ability. Continue forward and see if you have any of these or if you can find a solution to said issues.
Now, let's move on to the ingredients for great automation testing at scale.
Your outer layer - Test Setup
In this “bread” layer, you set up everything needed for the test execution to be able to run without requiring exterior help. Need some products to add to a bag? Find them here, and then pass them onward in a known format for your tests to consume.
The filling - Test Execution
With the correct test data/setup configured, your execution steps should now use that and run whichever test they are meant to. Use the data provided and create tests that are dynamic enough not to be tied down to a particular environment or data set.
You now have something that can run on production, all the way down the stacks onto a local development machine.
The final layer - Test Teardown
With the test complete, regardless of the outcome, you should clean up anything you can to reset the application's state. While having completely fresh data every time is preferable, some integrations or functionality might hinder this. Clean up your mess so the next run can have a fresh start.
If executed correctly, you will end up with a quality sandwich. Now that you have been introduced to this concept, you might think that you are in a strange position.
Meat Bread Meat.
If your functional test code calls out to APIs or other areas to get/set information mid-test, you risk the tests being too tied to a particular data shape or value. If the API schema is updated for whatever reason, you are left with a failing test, but the issue is not what the test was designed to check; it is failing on data.
Like a sandwich in the real world, this works but is not the ideal format. Because it's nested so deep inside functional/test code, you end up losing the structure and compromising the confidence in the data.
Additionally, when running tests with concurrency, you may end up with race conditions, whereby one test relies on one state and another differently.
Setting up what you need in the functional filling layer outside allows you to control this with hooks such as Before Each or Before All, and similarly with the teardown, After Each or After All.
Your framework may not have these steps or hooks, but it's easy enough for you to add them yourself; before calling the test script, run another file, and after execution is complete, run the teardown functions.
After reading this, you may feel like a refactor is in order, but trust me, once you do, you will notice an immediate and massive difference in the agility of your automated tests.
The Quality Sandwich
This article may sound like the “Best BLT Recipe”. It’s not — But I assure you, it's just as tasty!
Many Quality teams worldwide face a similar problem: tests clash when running at any level of concurrency simultaneously.
The solution to this problem, while it appears complex, is actually really simple. Following easy steps, you can ensure that no matter the test case you are designing, you can be safe knowing that it will run at Unlimited* concurrency.
Why the asterisk, Sam? I hear you ask. Well, some environmental factors can impact your ability. Continue forward and see if you have any of these or if you can find a solution to said issues.
Now, let's move on to the ingredients for great automation testing at scale.
Your outer layer - Test Setup
In this “bread” layer, you set up everything needed for the test execution to be able to run without requiring exterior help. Need some products to add to a bag? Find them here, and then pass them onward in a known format for your tests to consume.
The filling - Test Execution
With the correct test data/setup configured, your execution steps should now use that and run whichever test they are meant to. Use the data provided and create tests that are dynamic enough not to be tied down to a particular environment or data set.
You now have something that can run on production, all the way down the stacks onto a local development machine.
The final layer - Test Teardown
With the test complete, regardless of the outcome, you should clean up anything you can to reset the application's state. While having completely fresh data every time is preferable, some integrations or functionality might hinder this. Clean up your mess so the next run can have a fresh start.
If executed correctly, you will end up with a quality sandwich. Now that you have been introduced to this concept, you might think that you are in a strange position.
Meat Bread Meat.
If your functional test code calls out to APIs or other areas to get/set information mid-test, you risk the tests being too tied to a particular data shape or value. If the API schema is updated for whatever reason, you are left with a failing test, but the issue is not what the test was designed to check; it is failing on data.
Like a sandwich in the real world, this works but is not the ideal format. Because it's nested so deep inside functional/test code, you end up losing the structure and compromising the confidence in the data.
Additionally, when running tests with concurrency, you may end up with race conditions, whereby one test relies on one state and another differently.
Setting up what you need in the functional filling layer outside allows you to control this with hooks such as Before Each or Before All, and similarly with the teardown, After Each or After All.
Your framework may not have these steps or hooks, but it's easy enough for you to add them yourself; before calling the test script, run another file, and after execution is complete, run the teardown functions.
After reading this, you may feel like a refactor is in order, but trust me, once you do, you will notice an immediate and massive difference in the agility of your automated tests.
