The Monotony of Testing

Software testing can be some of the least enviable work imaginable.

The gaming industry is a good example. Video games may be fun for the rest of us, but only after they’re really not fun, for a long time, for all the people who have to test them.

An open world RPG will have vast expanses of digital valleys, mountain ranges, and cities, every inch of which requires excruciating collision testing to ensure players don’t accidentally walk through walls or underneath oceans. A first-person shooter with multiple DLC packages can have so many permutations of game states that testing is all but impossible. One quality assurance (QA) tester spelled it out in an email to Jason Schreier of Kotaku:

Let’s say we have a game with 40 pieces of microtransactions/DLC. [. . .] Even if some or all these 40 DLC packages are simple unlocks of on-disc content, we have to test every possible combination of these 40 DLC packages, in all game modes, using different combinations of storage media (20 DLC packs on the hard drive, 15 on a USB flash drive, 5 on a memory unit, etc), mixing up what the host has versus what clients have in different online game modes, verifying graceful handling in the event some dipshit has all his DLC on an external storage device and decides to unplug that device, with each Title Update (1.01, 1.2, etc), with different save data, etc., etc.

It could (and did) get the point where it’s literally impossible for humans to physically test all possible combinations/situations. It’s something that would get exponentially more difficult as more variables are added — like if you can purchase DLC in-game.

Testing commercial apps isn’t quite as gruesome as with open world RPGs or FPS’ with 40 DLCs. Still, you’d be surprised how frightening a task it is to rid even relatively simple software of ordinary bugs.

Take a core method of software testing: “end-to-end.” End-to-end doesn’t focus on one particular component or dependency — rather, it’s about testing the entire program, altogether, front to back and side to side. The goal here is to smooth out all possible actions and sequences of actions a user might enact in using an app — opening one page to navigate to another, scrolling to a particular point and triggering an animation, and so on. As you can imagine, having to simulate every possible permutation of the user flow can become terribly monotonous work.

And yet, as if this work weren’t already difficult enough, testers can be some of the most overlooked and mistreated people in tech.

The Bugs in QA

The problems typically found in quality assurance (the umbrella field which includes software testing, quality control and improving development processes more generally) are numerous and varied. QA pros can be overworked, undervalued, and overestimated all at once.

Again, video games are a notable example. In gaming, where they’re arguably more important and necessary than in any other industry, testers are often treated worse than their colleagues with better job titles. There’s the infamous Rockstar Lincoln, for example, where employees worked overtime for years in a poor workplace environment. As employees told Kotaku:

I have never suffered from depression before working at Rockstar. Now some time after leaving it’s a recurring issue for me… One tester who worked below me told me he had gone to the doctor for help dealing with depression, was asked where he worked and when he replied Rockstar, the doctor said. ‘For god’s sake, another one.’

QA in commercial software is typically less dystopian than in gaming, for any number of financial and cultural reasons. Still, it’s too often considered an afterthought, a hassle or unnecessary expense. Last year, ‘The QA Lead’ spoke with testers about the burnout they faced all too often, recording some of their most common complaints, like being…

• Rushed

“I never get the time I need. And it is cut in half because developers overrun.”

• Ignored and/or underinformed

“Figuring out from backlog items what to test is almost impossible. And nobody mentions edge cases.”

• Undervalued

“Teams tell me, they are agile, they do not need testers.”

• Outright disliked

“Many do not want an honest report and especially not bad news. When you insist, it can get personal.”

An overused and occasionally misplaced criticism leveled at testers occurs when bugs make it into the released product. At first, the blame in such a situation seems obviously to be theirs. But if you know testing, you know that it’s simply not possible to catch everything. Inevitably, when the app is being used by millions of customers and not just a few QA people, previously unforeseen issues crop up.

Furthermore, the way testers and developers work together can cause problems which testers on their own can’t fix. In their article, The QA Lead recounts the story of a veteran who got squeezed between his developers and project managers during sprints:

Alan: The two weeks were intense. I am not happy with my progress. I wanted to have a first, simple performance test, but I am far from achieving this.

Markus: What hinders you?

Alan: I need support from developers, but they are loaded. I also needed much more time than expected to get to know the software and reporting the bugs I found.

Alan was behind not because he lacked skill or experience or drive, but because his developers were being rushed and thus his crucial line of communication to them had broken down. Just take a look at the following interaction:

Developer: Most bugs Alan found are either insignificant or no bugs at all. Let’s sift through them first before jumping to any conclusions.

Alan: Well it was difficult for me to understand what the software should be doing from the backlog items. A specification would really help.

Developer: Over my dead body! We don’t want Waterfall. It is mostly common sense. Just come and ask us. How is performance testing going?

Alan: I’m at a standstill. I cannot get the tool to run against the service layer with all that security. I need help.

To summarize, then, we have an underappreciated and overworked profession operating at a level many businesses nonetheless find subpar. When it isn’t working ideally for either side of the table, what can be done to help?

Is Automation the Solution?

One solution to QA is more automation. Machines are, after all, better and more efficient with monotonous work than humans are. To whatever extent the testing process is about rote repetition — clicking around an app, making sure the correct items load, and so on — it doesn’t make sense to hire labor.

And this is, largely, what’s already happening. End-to-end tests are typically designed by humans but carried out by the programs they write. It’s a marriage of human qualities — understanding the needs and context of the software — and machine qualities — doing the dirty work.

The question is how far can the machines go? Can more of the process be automated?

Recently, in my podcast for Wix.com, I asked some QA pros whether machines could do their jobs:

Oh, it’s a very funny question, yeah, because almost every testing conference, you get a lot of talks about artificial intelligence doing some part of QA and such. But [. . .] testing is about art. It’s about creativity. It’s about imagination, about thinking. And of course I think one day, machines will be able to learn from people’s actions. But still they will not be able to validate. They will not be able to think, to really challenge [. . .] for example, your user wants to have a red button and you create a red button. This is something that is pretty easy for a machine to check. But no machine will ever think how valid it is to actually have this red button. Maybe it should be green. [. . .] So this is something that a machine will never do and I believe that we will have work forever.

Past the bugs themselves, there’s a human element to software that machines can’t replicate.

[R]obots cannot predict emotions and they cannot feel [. . .] Nowadays, having plenty of products in each area on the market, it’s quite important to also [understand] the value of emotions for the [user]. So users love your product, not only use it.

QA can’t and shouldn’t be entirely automated, no matter how un-sexy it may be in contrast to development. But algorithms absolutely can be the motor that underpins the software testing process, taking as much of the repetitive work off humans’ plates as possible.

Either way, the worst thing to do is treat QA flippantly — as unnecessary, or some kind of burden to developers. The companies that don’t value their QA people — treating them as second-class and failing to support and understand their work — are shooting their own products and services in the foot.

Testing is, after all, the last line of defense for software programs. A bulwark against releasing bad stuff. A top priority in any development lifecycle, for every software company.