Functional, non-functional testing, smoke testing, integration testing, performance testing, regression testing… Yes, we have them all! Yet we don’t divide our tests into categories by theoretical software testing methodologies. There is a difference between testing mobile applications and testing complex software like Tungsten Clustering and Replication. Of course we follow recommended testing practices, but our tests depend fully on our software architecture and its components.
We don’t use any public testing frameworks even though that would ease the testing process. This means more work for us, but gives us absolute freedom and control over our QA environment and tests.
In the previous blog post in this QA series, we described how we prepared a testing environment that fulfills our needs. Next, we want to deploy Tungsten software to those servers, install the topology, run tests, uninstall the topology and collect as many logs and as much information as possible.
Rubber Duck Method
Let’s focus now on the most advanced testing approach in our company - we test our software mostly as a whole system, installed on virtual servers, simulating realistic conditions and scenarios. Our testing tools and Tungsten software are independent units on those servers. Testing tools see Tungsten software as a black box, but Tungsten software allows them to look inside the box through an interface, which uncovers a lot of information. Also, the testing tools know how to read the log files of Tungsten components, or where to look for them in the environment.
Meet our main testing tool which is named Quackbook! At Continuent, besides the elegant swans in our logo, we love rubber ducks as a symbol of our QA. Quackbook is an intelligent testing tool, which is able to install, uninstall and test any topology in any environment. It fully understands the topology, its states, inputs, operations, and outputs. The tool is easily modifiable (it’s very easy to add new functionality) and offers a lot of possibilities and options. It’s a software written in pure Perl programming language without any external dependencies (libraries or modules).
From our perspective, we divide the tests into two important categories - testing features and testing scenarios. We can say that feature testing is also a kind of scenario testing. The difference is in test implementation. Automated tests for features are integrated right inside the source code of Quackbook. They test overall functionality from the top to bottom, from bottom to top and also from side to side. ;-) They test all supported topologies (Standalone cluster, Composite Active/Active and Composite Active/Passive), all supported installation methods (staging and INI installation method) and all supported cluster operations.
On the other hand, every testing scenario is a standalone script (which usually covers one use case) and uses available Quackbook functions to understand and operate a topology. These scenarios could be written in Perl, calling any function in Quackbook, or in a JSON notation understandable by Quackbook. Writing test scenarios is a very easy and fast way for developers to cover every developed functionality with a test. Not only are new features tested with these scenarios, but also fixed bugs, support cases, edge scenarios and non-standard configurations can be tested this way.
Running Tests Under Load
Once we have Tungsten Clustering installed on our servers, and before we start testing, one additional step is required - run a continuous load of queries into the database. For this purpose we use the tool Bristlecone, which generates mixed transactions. Continuent has released the Bristlecone Testing Tool under a GPL v2 license and the source code is available to the public.
Targeted Tungsten Components
In our QA, we test Tungsten Clustering as a system, but also separate components, parts and related tools:
- Tungsten Clustering as a whole system - all possible topologies, installation methods, SSL, servers configuration and roles, servers reporting chains, replication between servers and services, connections through connectors, and a lot of more...
- Cluster operations - switch, failover, recovery, backup and restore; including switch, failover and recovery on service level in Composite Active/Passive topologies.
- Tungsten Replicator - homogenous and heterogenous replication topologies.
- Tungsten Connector - establishing and testing connections using different programming languages and frameworks.
- TPM (Tungsten Package Manager) - installations, updates, upgrades, various TPM commands.
- tungsten_* tools - helpful scripts
Testing the upcoming version 7.0.0 of the Tungsten product suite constituted a major part of our QA work in recent months. This release will be full of new features, functionality and improved security. For this reason we expanded our QA with:
- Version 7.0-specific features including backwards compatibility
- Database level security - SSL between database and connectors
- IPv6 support
- API v2 tests
- Tungsten Cloud tests
- and a lot of more…
So... How Many Tests?
It’s hard to say what the “number of tests” really means. We may distinguish 3 groups based on the ‘level’:
- Test suite - group of scenarios, which are somehow related together (by topology, by component, etc.)
- Test scenario / tests for feature - group of commands and ‘checks’, to reproduce and fully exercise some feature or real life scenario
- ‘Checks’ - small (base) tests anytime during scenario lifetime, determining if current state is correct
In these terms, we may say that we have:
- 48 unique test suites, but including (reduced) matrix of 9 possible Linux distributions and 12 RDBMS versions, we run during release build 139 test suites in total
- During release testing we also have:
- 3,434 tested scenarios
- 156,503 checks (base tests)
Not counted in the above statistics, we also have JUnit tests during Tungsten product build time, which are testing on code level (specific code lines or functions).
At Continuent, we take QA very, very seriously! Our custom test system has hundreds of thousands of checks designed to exercise every part of the Tungsten product suite. Delivering the highest quality enterprise software designed to run in the most demanding environments is our goal each and every day.
For more information, feel free to reach out to us! Otherwise please visit these previous blogs in the QA series: