Types of Testing and Testing Pyramid
Types of Testing
Unit / Component
Lowest level of testing which tests individual methods and functions of the classes, components, or modules. Manual or automated (cheap and quick). Usually written and/or executed by the developer who is writing the code. Reviewed by peers. Mocking is used to emulate other classes, modules, backend cores, and micro apps.
Contract
Integration
Integration tests verify that different modules or services used by your application work well together. For example, it can be testing the interaction with the database or making sure that the micro services work together as expected. These types of tests are more expensive to run as they require multiple parts of the application to be up and running since they do not mock or stub integrated components.
Functional
Functional tests focus on the acceptance criteria being met. They only verify the output of an event and do not check the intermediate states of the system when performing that action.
There is sometimes a confusion between integration tests and functional tests as they both require multiple components to interact with each other. The difference is that an integration test may simply verify that you can query the database while a functional test would expect to get a specific value from the database as defined by the product requirements.
End-to-end (E2E)
End-to-end testing replicates a user behavior with the software in a complete application environment. End to end testing (E2E testing) refers to a software testing method that involves testing an application's workflow from beginning to end. This method basically aims to replicate real user scenarios so that the system can be validated for data integrity. It verifies that various use cases work as expected.
End-to-end tests are very useful, but they're expensive to perform and can be hard to maintain when they're automated.
NOTE: It is recommended to have a few key end-to-end tests and rely more on lower level types of testing (unit and integration tests) to be able to quickly identify breaking changes.
UI/GUI
GUI or UI tests check the Graphical User Interface of the Software. The purpose of Graphical User Interface (GUI) Testing is to ensure the functionalities of software application work as per specifications by checking screens and controls like menus, buttons, icons, etc.
Acceptance / User Acceptance Testing
Acceptance tests are formal tests executed to verify if a system satisfies the business requirements. They require the entire application to be up and running and focus on replicating user behaviors, but they can also go further and measure the performance of the system and reject changes if certain goals are not met.
Acceptance tests can come in different levels of granularity. Most of the time they will be rather high-level and test your service through the user interface.
Performance
Performance tests check the behaviors of the system when it is under significant load. These tests are non-functional and can have the various form to understand the reliability, stability, and availability of the platform. For instance, it can be observing response times when executing a high number of requests, or seeing how the system behaves with a significant of data.
Performance tests are usually, by nature, quite costly to implement and run, but they can help you understand if new changes are going to degrade your system.
Smoke
Smoke tests are basic tests that check basic functionality of the application. They are meant to be quick to execute, and their goal is to give you the confidence that the major features of your system are working as expected.
Smoke tests can be useful right after a new build is made to decide whether or not you can run more expensive tests, or right after a deployment to make sure that the application is running properly in the newly deployed environment.
Regression
Seeks to uncover software bugs after changes to the code (e.g. bug fixes or new functionality) have been made, by retesting the existing functionality and ensuring that it has not been degraded.
Upgrade
Testing technique that verifies if older versions of the software can be replaced by a newer version of the same software. It verifies it works and that user's learning is not challenged.
Installation
These tests examine the installation of the software rather than the use of the software. As such they are essentially behavioral tests, except the item under test is not the product, but the installer of the product.
Exploratory Testing
Exploratory testing is about exploring, finding out about the software, what it does, what it doesn’t do, what works and what doesn’t work. The tester is constantly making decisions about what to test next and where to spend the (limited) time. This is an approach that is most useful when there are no or poor specifications and when time is severely limited.
Minimal planning and maximum test execution.
It can be used as a check on the formal test process by helping to ensure that the most serious defects have been found. Short declaration of scope and short time boxed test effort.
Canary Testing
Canary tests are minimal tests that quickly and automatically verify that all the dependencies are active and ready. These tests should be simple and low maintenance.
Some examples are: simply scan configuration files for every URL, password and just ping them one by one against a predefined time threshold. Any log path mentioned in the configuration files can be scanned and checked for the required write permissions and available disk space. Any login and password can be checked, even though this may be more complicated.
A/B Testing
A/B testing is primarily used to review the effectiveness of a change and how the market reacts to the change. The new features are rolled out to a certain set of users.
A/B testing is NOT blue-green deployments. This is a common mistake. A/B testing is a way of testing features in your application for various reasons like usability, popularity, noticeability, etc, and how those factors influence the bottom line. It’s usually associated with the UI of the app, but of course the backend services need to be available to do this. You can implement this with application-level switches (ie, smart logic that knows when to display certain UI controls) and also using Canary releases.
Testing Pyramid
Keep these two things in mind when it comes to establishing an automation test suite.
Write tests with different granularity
The more high-level you get the fewer tests you should have
Essentially, write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end.
References
https://medium.com/@nathankpeck/microservice-testing-unit-tests-d795194fe14e
https://martinfowler.com/articles/practical-test-pyramid.html#TheImportanceOftestAutomation
http://istqbexamcertification.com/what-is-exploratory-testing-in-software-testing/
https://www.atlassian.com/continuous-delivery/different-types-of-software-testing
