End-to-End Testing and Creative Ways to Deploy It
04 August 2017
What is End-to-End testing (E2E)
End-to-end testing is a QA methodology used to test the flow & functionality of an application and make sure it performs as designed. The entire application is tested in a real-world scenario such as communicating with the database, network, hardware and other third party software.
How it works
To perform E2E testing, the application needs to be deployed on a server and be accessible from the platform that runs the actual tests. There are multiple E2E test runners & frameworks, but the most common ones are Selenium and Protractor. E2E tests are different from other testing methodologies (like unit tests) given the fact that no “pieces of code” are invoked directly, but rather the UI elements are invoking the application logic. This means that the E2E test runner is interacting with the UI to perform any actions. After an action is performed, the test scenario checks that the executed action was successful or not (UI elements change, DB records have changed, cookies were set, etc.).
E2E tests take a bit of time to run (as opposed to unit tests that run in a few minutes or less). This means that before a deployment, unit tests should run and prevent the build from being deployed if anything fails. Doing this prevents a long E2E test run and saves the team a tremendous amount of time.
The main benefit of having an E2E test suite is that it gives the team the confidence that the newly deployed build performs as expected. Of course, in order to achieve this, the test suite needs to be updated constantly according to the new changes in the application.
Another benefit is the fact that with an E2E test you can check that the UI elements are displayed/working as expected, as opposed to unit tests, where you can test that a piece of functionality is working as expected.
The ability to set up a proper deployment pipeline is the next big benefit. This allows for a controlled deployment environment, where a build can be pushed to the next pipeline server if all tests have passed. Let’s say your pipeline has 4 servers (dev, test, stage & production), a new version of the application is pushed to the dev server (if all unit tests pass). At this point, E2E tests will run against the dev server. As a best practice, on this server you should only run a sanity test suite, that will cover just the main functionality of the app using the happy flows and will finish in a matter of minutes. If all tests pass, you can be confident that all of the main functionality is working. Then you can proceed to deploying the same version to the test server. Here you will run the full test suite, that will take longer to run but will cover all functionality of your application. Again, if all tests pass, you can proceed to pushing the build to the next server, where you will run a sanity suite, just to make sure the deployment was successful.
Of course, having such a pipeline will result in a long test run time, especially for the full test suite (as a baseline, there was a test suite that took 3.5 hours to run), hence this is where running tests in parallel come in handy. The test runner can be configured to shard test files, and multiple tests will run in parallel, resulting in a quicker test run (after test sharding the tests that took 3.5 hours dropped to about 40 minutes). The ability to get changes into production safely and quickly in a sustainable way is cool, but it cannot be done without E2E testing. So if you are planning to get continuous deployments in place, E2E testing is a must.
Most E2E test runners will generate an inline report that will show some base metrics like the number of tests run, how many of them passed/failed along with some execution times. The E2E test frameworks can be configured to export reports to different formats (HTML, JSON, PDF, etc.), giving details for each test case in particular. They can even be configured to take a screenshot of the UI if the test fails, which can help debug the UI and see what’s wrong. After a test run, these reports can be sent via email to a set of people, and inform them on how the build performed.
Code coverage is one of unit testing output’s most loved features, but it is not well known that E2E testing can also generate code coverage. The trick is that the tests must run against a server that contains instrumented code, not against a real server hosting the web application.
The instrumented code contains tracing calls that can determine whether a piece of code was executed or not, thus generating code coverage. Most instrumentation tools will generate code coverage for statement/branch/functions/line coverage.
The main benefit of code coverage for E2E tests is that you can quickly see if you do not have test cases for some functionality of the application. With unit tests, you can reach 100% code coverage with no problem if you set your goals high, but with E2E tests that is not possible given the fact that your defensive code may not be triggered since there is no mocking involved.
Of course, by now you can see the full benefits of having E2E tests, and if I stop here, you can pretty much have the big picture. However, there’s one more thing:
The sky is the limit!
So with that being said, think of what’s missing from this methodology or what improvements you can add that will benefit your team/product and extend the functionality of the framework to suit your needs. Let me give you three examples:
- Slack integrationHaving a long running test suite (3.5 hours as mentioned before), it was very important for me to know when a test failed, so I can investigate, fix the problem and run the tests again, without having to wait a few hours for the tests to finish. At first, I kept an eye on the Jenkins logs where the tests were running, but that proved to be very inefficient & unproductive for me, so the need for a better integration arose. With a few tens of lines of code, I created a plugin that sent the team a slack notification once a test failed. The notification contained the test name error message, stack trace and a screenshot of the UI.
- Better reporting toolsThe initial test report was an HTML file that contained all tests and the results. If you wanted to see what tests failed, you needed to scroll through the whole document until you reached the failed test. Needless to say, this was annoying. By just wrapping a template around the test results, the report document now had collapsible HTML elements that were much more meaningful and provided a better way to see the results.
- Screenshot analyzerOne time we had to change the alignment of a button. By mistake, the new styles were also applied to other buttons, and that was a bad thing. Code review, unit tests & E2E tests did not catch this issue. The change was so small that no one saw it. As before, the need for a better integration test arose.Let’s have the following example, can you see the difference in the two screenshots below:
Probably not! How about now?
Ahh, much better, right?
What we ended up doing is using a PerceptualDiff (image comparison utility) to compare screenshots from the current test run with ones from the previous test run, and even if a single pixel changed in the UI, the test would fail, and the resulting screenshot would highlight the differences like above.
As mentioned above, if you put some effort, the sky is the limit, so:
THINK, IMPROVE & then ENJOY !