Comprehensive Review

Hi there! My name is Sergii Vashchenko, and I am the QA Practice Lead at Svitla Systems. I enjoy trying new tools, techniques, and theories on the projects I am a part of and would like to share my experience with the rest of our QA community.

In the last ten years, I've used many tools for test management. I already have a few favorites, but with 2024 making waves, I became interested in finding the right tool to suit current and modern project needs for less money.  

Nominees

Nowadays, there are tons of tools for test management. I had to narrow the scope, so I focused on the ten most popular tools based on the tech reviews from technical resources and QA forums. Then, I reviewed the tool documentation and picked up the five most promising ones for my current project: TestRails, X-Ray, Zephyr Scale,  AIO Tests, and QA Deputy. Many of them are some of the most prominent players on the market, and I'd be willing to bet good money that you have already worked with at least two of them. However, they have different philosophies and sets of features, and the pricing differs significantly. So it's fascinating to compare them side by side doing the same things for the same project.

Assessment Criteria

To make a comparison, I analyzed project requirements and prepared a list of crucial criteria: 

1. UX. The most crucial criterion for the tool is an outstanding user experience. Overall, it should speed up the process of writing tests and managing test executions. Nobody likes to do unnecessary clicks or to be unable to find the required stuff quickly.
2. Test reports and time tracking. Statistics and estimations are beneficial for making decisions. Therefore, having good reports is essential. 
3. Requirements traceability. The traceability between tests and user stories will help ensure that everything necessary has been considered. 
4. Gherkin language support. Test cases could be written in BDD style, so the tools should support the Gherkin language.
5. Automation. The integration with GitHub Actions and good documentation for the test automation API is a must. 
6. Performance.  The app should work quickly enough or it will become an annoying thing that will quite quickly distract engineers from their work.
7. Price. The tools should be relatively inexpensive for a small team. However, the team will start to grow fast in the future. 

Other additional criteria we include:

1. Jira application. The team prefers storing tests and other development tickets in one place, so direct test management from Jira was considered a big plus.
2. Test data import/export. It would be good to have the option to export the test cases and test suites into some standard formats like ".csv" or ".xml ."It will allow us to migrate the test base to a new test management tool if there is a need with minimum effort.
3. Free trial. Trying the app alone can provide a much more user experience than watching video demos or reading product documentation. 

Test Tool's Test Drive

The tool's test drive included the following things:

• Initial tool setup
• A test set for a new user story
• Test cycles
• Requirement traceability
• Time tracking
• Test reports
• Prepared tests execution

On average, I've used each tool for new projects for at least one week. Below is a short overview I've prepared for each tool, where I point out my impressions of each tool and omit some commonalities. Also, please remember that pros and cons are subjective, so take them with a pinch of salt.

TestRail

Home Page: TestRail: Test Management & QA Software for Agile Teams

Version under the test:  7.5.6 

Overview. TestRail is a standalone app, and you must consider that it takes a few minutes to register, get an instance, set up the integration with Jira, and start writing tests. You are unable to create and keep tests directly in Jira. However, installing a free Jira plugin can enhance your features. Overall, the tool allows you to link tests to Jira tickets and monitor test status from Jira. Additionally, it is possible to open bugs directly from the TestRail tool. 

The TestRail has a unique feature that others don't. I am talking about UI scripts. This feature allows you to add some additional things on your own. For example, you can make custom colors for keywords inside the test cases through reminders when some specific page/test is opened, etc. You can customize almost everything inside the app (add custom templates for test cases, a company logo for test reports, and much more). The TestRail API can be used with lots of test automation frameworks.

Pros:

1. Reliable, stable, and fast
2. Intuitive interface for tests, test execution, and test management overall
3. Clean and easy way to monitor the testing status 
4. Time tracking for each test as well as for test run/test plan
5. 50GB of free cloud storage for test data (screenshots, scripts, videos). Note: you can have more storage space, but each additional 25 GB will be billed in addition to your regular subscription
6. Custom fields for tests and test templates
7. User variables for the test automation
8. Good-looking reports
9. Many options for different tools integration and test automation

Cons: 

• Quite pricey ($37 per user monthly or $34/user monthly if you pay for a year). 
• The BDD test case format was unavailable in version 7.5.6, which is the one I tested. However, the BDD test documentation is already in the app documentation. According to emails from TestRail support, the feature is being actively developed, and the option is already present for early-access users. Meanwhile, the rest of the users can use a "free-text" format for the BDD tests.

Zephyr Scale

Home Page:Zephyr Scale - Test Management for Jira

Version under the test:  9.10.0

Overview. Zephyr is probably the most popular test management tool, as it has native support for Jira. It has three different versions: Squad, Scale, and Entreprise. The difference between those versions can be a topic of a separate article, so I will mention that the Scale has the most balanced set of features that fulfill all my criteria. Also, I wouldn't say I liked the Squad. It's so limited that it is incomparable to the Zephyr Scale. 

You can install Zephyr as a Jira app from the Jira marketplace and then enable Zephyr in the project settings. That's it. You are ready to start writing test cases and make test cycles and test runs with Zephyr default settings. Everything works buttery smooth out of the box. The design is minimalistic and super simple. You need to do a minimum of actions to create/run/manage tests. There is one feature related to testing sets that I actually didn't like. When you are working with test sets, you have the option to create folders and subfolders. However, when you select a folder, you will see test cases from the folder and all nested subfolders at once without grouping them by subfolders. So, you need to structure subfolders not and adequately keep test cases in the parent folder if you have any subfolders inside. 

The API for test automation is good enough for integration with all popular automation tools. Now, onto the subject of pricing. It's free if up to 10 users are registered in your Jira instance. It costs $4.78 per user per month if you have more. It's way cheaper than TestRail. 

Pros:

1. No need to install other plugins for Jira integration, all needed things are out of the box
2. UX for tool usage is one of the best on the market
3. Test tickets have their suffixes, e.g., (XXX-T1, XXX-T2), and by default, are shown only inside the Zephyr board or test runs 
4. Requirement management is out of the box
5. A "Traceability" tab inside each test case allows you to quickly add a link to any Jira ticket (e.g., Epic, User Story, Task, or Bug)
6. Time tracking feature for single tests as well as for test runs.A good set of default reports and the option to make custom reports using JQL (advanced Jira queries)
7. Free for teams with up to 10 people

Cons: 

• Fewer things can be customized compared to the other competitors
• Test data will consume Jira cloud storage
• Not all of the keywords are supported for BDD tests

Xray for Jira

Home Page: Xray - Native Test Management for Jira

Version under the test:  6.3.2

Overview. Xray is commonly considered a highly customizable alternative to Zephyr. It allows you to create separate test-type tickets for tests, executions, and test plans to manage them the same way as with all other Jira tickets. It has a lot of customization options, but the learning curve is relatively high. 

I had to read a lot of documentation and watch video demos to start working with the tool. You can set up everything for any development process, yet you must adjust many project settings. Also, it takes much more clicks to start a new test case compared with the other tools. Other than that, I had issues with slow-loading tests during the trial. However, the cached requests caused it, as everything started to work fine once I rebooted my router. I didn't have such issues with the other tools, and I hope the Xray team saw my feedback and has already fixed the problem. Pricing: it costs $10 per month if you have up to 10 users and $4.25 per user monthly if more users are registered in your Jira instance.

Pros:

1. Anything can be changed, added, or reordered
2. Different types of test styles, including data-driven and fully supported BDD
3. Advanced requirement traceability
4. Advanced Reports  / Document Composer. You can make almost any type of test document in any format you want

Cons: 

• The test runs looked unusual to me
• There are no test runtime tracking options from the box. However, it can be done with a third party plugin
• The separate test-type tickets heavily spam the project, quickly increasing ticket numbers overall. As a Jira SuperAdmin, you must modify Backlog and Board query settings to hide the newly created test types, create a separate QA/QC project, and keep tests there.

AIO Tests

Home Page:All-In-One Test Management App for Jira

Version under the test: 3.1.6

Overview. AIO Tests is a Jira-native app with many installations from the Atlassian marketplace and good reviews from users. The tool has a one-click installation, and you are good to go with the app's default settings. Minimalistic but has an enjoyable interface. Everything looks logical to me. There are two types of test cases: classic and BDD. The UX for BDD cases is the best of the compared tools. The tests are created with test suffixes, the same as in Zephyr Scale, so they are not shown by default on the board or backlog. Another interesting thing is that the app doesn't allow you to add created test cases to the test sets and test cycles if they are not reviewed and approved(published). It forces you to follow the review process. 

I would have preferred a better way to organize the test cycles. You can re-run tests inside a cycle several times, but only part of the cycle at a time. You can make a copy of the cycle as an option, but I used to use the cycles differently. However, the cycles in AIO Tests have a unique feature to execute them in offline mode. It's done by exporting the cycle into an excel file and then importing it back into the app.

Speaking about documentation, it's worth mentioning that it's pretty detailed and has many examples. I liked the chapter that explains how to make a beautiful dashboard with AIO widgets for thorough test process monitoring. Pricing: it's free for teams with up to 10 people but costs $350/year for teams with 10-50 people. 

Pros:

1. It is cheaper than other contenders if you have a team with 11- 50 people ($350/year which is equal to $29.17/month
2. It feels simple, but yet it is a powerful test management tool
3. Good for BDD tests
4. Free for teams with up to 10 people

Cons: 

• Test cycles can't be re-run
• Integration with other tools is limited compared to the other competitors, but it still covers most of the popular tools and automation frameworks

QADeputy

Home Page:QADeputy

Overview. I saw that QADeputy was mentioned on several websites as a good tool to try. However, I have yet to have a good experience with the tool. Jira integration allows you to see the test run status on the Jira side. There are no configuration settings at all. It looks like the company hasn't yet finished the Atlassian security self-assessment program, so the capabilities are limited. I briefly checked the test management stuff in the QADeputy instance to see what the app proposes. I wouldn't say I liked the UX, in all honesty. Whenever I tried to create a new test case or test set, I was stopped by a need to fill out some required fields that are commonly optional in most other tools by default. The BDD format is not yet supported—only free-text or step-based tests. The reports have just a few basic templates. Pricing: the tool is free for one user and costs $20 per user/month ($18 per user in case of an annual plan). 

Overall, the app needs to do much more to reach the level of other competitors mentioned in the article. The tool needs to meet my basement criteria. So, I will skip the cons and pros of this one. I hope it will improve and give it a try in a year or two with better results. 

Conclusion

TestRail, Zephyr Scale, Xray, and AOI Tests are all pretty good test management tools. They have pros and cons, but any tool can be used for most projects. All of them have plenty of features needed for test management. They all have good support for the popular project management and automation tools, reports, import and export of test cases, and so on.

Speaking about my own experience and from Svitla Systems' expertise in Quality Assurance, I recommend TestRail as it still feels like a safe option. However, the UI for some of you might need to be updated. Also, if you are into BDD tests, I am sure that 'AIO Tests' is worth trying instead. Zephyr Scale is a viable alternative. It delivers the same things as TestRail and AIO Tests, with a bit different test management flow. The advantages are that it is free for small teams and not so pricy for larger teams. 

Also, you can use Jira built-in things (JQL for the reports, labels, etc.) All 3 Jira-native apps (AIO, Zephyr, and Xray) will make the tests accessible for the whole development team, so developers can see which tests the QA team prepared for the features in advance.

Xray for Jira might be an option for you if you:

• Have a specific development process
• Have the full support of BDD format tests is a must-have
• Need to spend a bit more time on the app configuration 

As for me, QADeputy is not yet on the same level as the other competitors mentioned in the article. Still, I recommend you try it to form your own opinion. 

I hope the article helped you choose a new test management tool for your projects. 

May the quality be with you!