Building an Automation Framework
Tips & Tricks for Creating a Friendly Automation Testing Environment
We all know that all projects start small and in time, they gain complexity due to new features. This means that from a testing perspective, you will have a test suite that will never decrease, but only increase in terms of a number of tests and testing time. It would not be such a terrible scenario in terms of time if the QAs would have to test only one module at a time, but if there are multiple modules that need to be deployed and tested, it can get really frustrating and you could end up with delays due to large testing time.
When you take all these factors (multiple modules to be tested, lots of tests, lots of time spent testing, delays in releasing products) into account, you end up with quite a bit of a problem, which needs a solution ASAP. In this case, say hello to Automated Testing, the Holy Grail to all our previous presented issues.
There are a few steps that need to be followed when setting up an automation testing environment.
1. Create an Automation Framework mapped on your project’s needs
When deciding on how the automation setup should be constructed, you should take into consideration who will be developing the automation tests. Ideally, they will be developed by QAs, but if you need an automation suite up and running as soon as possible, developers might also need to be involved. For environment configuration and any other tools, you should take into account using the help of a DevOps person. In the end, the test suite will be maintained by QAs, so you should create an automation framework that can be easily maintained.
In the case of the project that I am working on, given the fact that we are Ruby developers, the best option was to create a Ruby-based Framework. The QAs working in the team had experience with Selenium, so we decided to use Selenium for the actual automated tests as the portable software-testing framework for web applications. Still, Selenium is quite tricky to use and in Ruby, we can enjoy the facilities offered by Capybara. No, we are not talking about the mammal Capybara, but about Capybara, the web-based test automation software that simulates scenarios for user stories and automates web application testing for behavior-driven software development.
As soon as you decide on the components of your automation framework, you can start thinking about the structure of your automated suite. Considering that QAs will maintain the automation project, could give you indications on what to decide. What would help is considering that the automation project should not be connected to the tested application, but they should be considered as two totally separate entities? In the particular case of the project that I am working on, we decided to create a separate project with a separate repository on Github, that would only test the application, without using anything related to the tested app.
Now that you have all the setup for your automation project, you can move on to priorities.
2. Decide which tests should be automated first
All applications have tests that are divided into multiple categories. These categories would mainly include tests that should be executed before each deploy (smoke tests) and tests that should be executed on major changes on a certain part of the application (regression tests).
If you take in consideration that there is a suite of tests that certainly needs to be performed before each deploy, you can conclude that you need to automate that suite of tests before any other.
Also, since this suite of tests is a high priority, the time needed to develop it should be as short as possible, so that you can use it in the near future. There is not that much use of an automated suite of tests if it’s going to take one year to implement. In the scope of the project that I am working on, in order to achieve the switch to automation fast, we came to the conclusion that involving the developers would speed up writing the tests, which proved to be the way to go.
3. Using a Mock Server
In general, tested applications are also integrated with different third-party services. When starting an automation project, you need to decide if the tests you are writing concern only your application or if they will also test the integration with these external services.
If you are targeting tests that only handle your application, you will probably need the benefits of a Mock Server. A mock server allows you to mock any server or service that you connect to over HTTP or HTTPS, such as a REST or RPC. It can easily recreate all types of responses to test applications easily and effectively.
By using a mock server, you can generate the responses that the tested application expects and handle them independently, having to only care about how your application behaves.
4. Establish a set of rules for writing tests
It is common knowledge that the higher number of people involved in developing a project, the higher number of styles for writing code. Soon, you could end up in a dish of spaghetti code, with a topping of multiple defined selectors. Even if without the code part, this meal might sound appealing, there is no way you can get rid of it. It would be best to prevent such a situation instead of having to deal with it.
One way to achieve a clean code structure, would be to establish a set of rules with the team before starting to write any tests.
You could build this list in a document that should be respected by the person that writes the tests and that is also checked by the tests reviewer. Reviewing code should be considered one of the main pillars for an easy maintainable project. A Code Review Process set in place could assure the tests are correct from a QA and development perspective. What might work for you, and has worked for our project, is to require the review to be performed by two people: a developer for the code side and a QA for the actual test side, at least until everyone is comfortable with the automation process.
Along with this list, you should encourage as much as possible the use of helper files that contain methods for common actions and targeted elements. This will encourage code reuse and enforce consistency of tests. Also, it will allow easy adaptation to changes that might appear at any time.
Tests should keep a general aspect, meaning that you should not refer to specific elements on the page, but retrieve the objects in a dynamic manner. For example, you should not rely on hardcoded values or exact rows in tables, but retrieve the exact data that you are working with, as an application is bound to change and your tests should not be affected.
5. Structure the project
I mentioned earlier that there might be multiple categories of tests in a project that might need to be performed at different times. This means that you will have to structure your suites as independent as possible so that you can run them by any rules you need.
For instance, in the automation project I helped develop, we have Smoke and Regression tests and we structured them in two separate folders (Smoke/Regression) that contain a subfolder for each area that needs testing. This way, we can run the tests we need when we need to, without having to cherry-pick them, which – trust me! – would cause some serious headaches.
6. Shorten the testing time
Automating tests is aimed at reducing overall testing time, but when the number of tests automated gets larger, testing time will start to increase and by fixing the initial problem, you will end up with another problem that needs to be solved: reduce the automated test run time.
I know, how could an automated suite not be enough to reduce the testing time? It might seem weird, but the final goal is to be able to run tests in the shortest amount of time so that if any issues occur, you would be able to fix them and re-run the suite in no time. In this case, you might foresee the answer: running tests in parallel.
You can achieve a test suite that can be run in parallel by paying attention to some aspects:
- Remember that when running tests in parallel, the rows in the DB change at a faster rate. This means that you will have to be careful to only handle the object that is targeted by the test.
- You need to be careful to not access the same resource in the same time from different places, because that might lead to flaky tests, which are difficult to debug.
- Use all the help you can get in order to guarantee a safe parallelization. For Ruby, there are a few helpers that can come quite in handy. Gem parallel_tests is one of those tools that targets exactly what you need for a parallel suite of tests. Amongst other features, it offers support for a customizable number of threads.
7. Submit the automation results to a Test Case Management app
After running a test suite, you will want to see the results in an easy to read manner, that can be analyzed in a report by anyone, even without any tech knowledge. For achieving such an outcome, you can use a Test Case Management tool, like TestRail, that offers support for integration with a Ruby project. More details on how you can benefit from TestRail’s API can be found at http://docs.gurock.com/testrail-api2/bindings-ruby.
From a developer’s perspective, writing automated tests is an opportunity to learn something new and exciting. It might become repetitive at a certain moment, but that gives you the chance to try and find new ways to improve the project and to find brand new challenges all by yourself.
This is a journey that will award you a large baggage of knowledge in what concerns your path as a developer, but also in what concerns teamwork.
I hope I have provided some useful insights in writing automated tests. If you have any questions regarding how an Automation Project in Ruby is developed, please do not hesitate to contact me at firstname.lastname@example.org. I might not have all the answers, but I will do my best in helping you.