Why are reproduction steps helpful for fixing software bugs?

Question

Our company have a support team and a dev team. We require reproduction steps on every bug report, however sometimes the support team submit reports without and become frustrated when dev close them as "cannot reproduce / cannot fix".

Support then make the argument "if I take my car to the garage, the mechanic will look at the engine for me to diagnose and fix the problem, I don't need to know how my engine works, and I shouldn't have to spell out to the mechanic that my car makes odd noises".

What's a good way to explain WHY reproduction steps are important in response to this sort of challenge?

Searching for the topic finds lots of links with information on HOW to write good reproduction steps, but I cannot find any links that answer WHY they're important from the point of view of someone who is not a developer.

Edit: similar questions / sites / blogs the advise seems to amount "Mark it as cannot reproduce and they'll have to fix their work and resubmit the bug. Eventually they'll learn to do it right the first time", but I'd much prefer a constructive discussion on why I'm saying I can't fix it so that there's genuine understanding of what sorts of things the support team should be doing rather than completing a template by rote and being upset when their work is rejected.

Answer 1

To follow on from the car analogy, I've used the following in other contexts:

Say you took your car into the mechanic and said the battery keeps dying. He runs thorough tests on the battery and the electrical system and finds nothing wrong. The mechanic asks you how to reproduce the problem and you get annoyed at the question and say it just happens randomly for no reason. The mechanic digs deeper and eventually discovers you've been frequently leaving your headlights on all night, which as a brand new car owner, you didn't realize would drain the battery.

Now you might say that the car should prevent the battery from dying if you leave the lights on. That's a reasonable claim, and many modern cars do just that. However, it was still impossible to make the diagnosis without knowing the steps to reproduce the problem. And at least in this case, they mentioned the battery. Some bug reports are like, "my car is broken sometimes but it's fine now" and they don't know why you want more detail. Also, the best mechanics will have the conversation first and not even charge you.

It's not a perfect analogy, because in a computer application, "leaving the lights on" might not have been a user error, but an unintended side effect of another feature, or subtle interaction between two features. You also can add preventative measures and logging even if you don't precisely know what the cause is. And I would consider a mechanic very poor if they just returned your car without at least having a discussion about possible root causes.

Answer 2

True story: A family has a problem with their new car. Every sunday the whole large family meets for lunch, then someone drives to the ice cream parlor and buys ice cream for everyone. And then a strange thing happens: Whenever they buy vanilla ice cream, the car's engine won't start. If they buy any other ice cream, the car's engine starts. Mechanics couldn't find a fault.

Finally, the family invited someone quite higher up from the manufacturer for sunday lunch. And the guy drove with them to the ice cream parlor, vanilla ice cream was ordered, and the engine wouldn't start. This guy would have made an excellent software tester: He checked out the ice cream parlor. It turned out that they had a huge tub of vanilla ice cream that could be served quite quickly, while any other ice cream took longer. The fault had nothing to do with the ice cream flavour, but with how long the engine was turned off. Once they could reproduce the problem, the fix was easy.

But in the end, a bug that appears always and under all conditions is easily fixed. These bugs are most likely already gone before a tester ever sees the software. What is hard is problems that only happen under certain circumstances, and that will not appear 99% of the time. In that case, your bug report is not helping at all if you can't describe how you got the 1% case.

Good tools would also allow you to specify how reproducible a bug is. You may have a bug that even with your best efforts can only be reproduced 10% of the time. What does that mean? It means the developer won't give up if he tried it out three times and the bug didn't appear if he knows it is rare.

And when you write down steps to reproduce a bug: Consider that a developer cannot read your mind. If you don't write it down, he doesn't know about it. If you are not precise, he doesn't know what you are doing. If you are inaccurate, the bug may not be reproducible at all with your description.

PS. I've had bugs that would only happen to testers or even specific testers. For example: Tester does step 1, checks the screen, compares it to what his script says, does step 2, checks the screen etc. And there's a bug that only happens when you do step 1, 2, 3 and 4 several seconds apart, by the exact right amount. A second tester writes down the results, therefore takes longer, bug disappears. The developer knows all the steps, does them quickly in a row, bug disappears. Developer and Tester 2 go to Tester 1's desk and see him reproduce the problem. They ask him to do it slower, problem goes away. They ask him to do it faster, problem goes away. He does it at his own speed, bug comes up. This does happen.

Answer 3

Complexity and discoverability.

The mechanic fixes dozens of cars which are basically the same every single day. He has a list of tests, which is reasonably fast to check and covers detecting all expected and most unexpected defects.
Still, if the right back side flutters a bit beyond 50 mph, that's unlikely to be covered unless the driver highlights it. Actually, unless it is for the mandatory regular general inspections, not trying to describe your concerns to make sure they are addressed is at best odd, at worst irresponsible.

The software engineer has a far more amorphous test-target. First, it is one-of-a-kind. Yes, there is probably an automatic test-harness which is applied, but that that mostly covers the nuts and bolts, not the user-interaction, and it cannot be exhaustive. At least those will all work properly when released, so won't be reported by the user (anymore). All the inputs, all the dependencies, all the system-configurations, and myriad other outside influences, and on top of that it is hard to anticipate all possible interactions.
The coup de grace is that you cannot look at it, and say you have thoroughly tested everything, as even delineating parts of the application to check is non-trivial, and making sure you hit everything in that part is just about impossible. Try checking the border of an invisible shape for some kind of interaction. Better if that changes dynamically, best if the way you want to interact depends on that dynamic change. Try testing an ai, or anything else with mostly hidden highly dynamic state.

And that left out all the cases where the behavior is as designed and seems reasonable, but actually isn't, or at least doesn't help some group of users.

If the user says "Your software has a bug", the only thing the developer can say for moderately complex software is "probably. Where does it hurt, so I can look there."
And until that information is present, that's "no repro / cannot fix".

Answer 4

You can't fix a problem you can't see most of the time. It depends in the context of your company BUT the developer will need to reproduce the error so it can be investigated/fixed .

An example:

Reported Bug : The xml response gives the wrong dates for departure times

Reality: The xml response gives the wrong dates for departure times , when the train is from a certain company at a specific station .

If the developer cannot see the specifics of this bug(train company, station) , how is she gonna pin point the error and proceed with appropriate actions ? Imagine there are thousands of trains and stations responses all working properly.

Answer 5

Lets take your literal question aside for a moment. Your description gives me the impression the core problem is not the bug reports themselves, but the disconnection between the people who notice the bug, and the ones who have to fix them.

Ideally, a user may notice a behaviour of a program which looks like a bug to them, calls the support team, and they use some screen sharing tool, or screenshots, and real data to reproduce the issue, so the support team member can either help the user directly, or come to the conclusion they need to get a dev involved.

But if they cannot reproduce the issue together, the support team member can already close the ticket as "cannot reproduce". If they can, the support team member now can call a dev and do the same with them: show them what works wrong - for this, the support team member needs to make sure they can demonstrate the problem, for example, within their test environment. If they now cannot reproduce the issue again, the support team member can either speak to the user again, ask for more information, or close the issue themselves.

But you wrote:

sometimes the support team submit reports without and become frustrated when dev close them as "cannot reproduce / cannot fix"

That gives me the impression the support team member just try to write down their bug reports in words (not necessarily by reproducing the issue together with a user first), they don't try to reproduce issues in their environment and don't use screen sharing tools to explain the issue to the devs. And that's the real problem here - the communication workflow isn't ideal, there is too much "one way" communication, too much risk for "chinese whisper".

Interestingly, the answer given by @gnat in 2012 was pretty much going into the same direction: there needs to be a good bidirectional communication channel between users, support staff and devs. Of course, as that older other answer points out, such a channel can be established by using written text. Today screen sharing has become standard, it has made support communication a lot simpler. It is not a replacement for written tickets, but has proved itself as terribly useful.

Of course, reproducability is not only important for this communication chain, it is also important for the dev to be able to verify a bug fix works, but you asked for the non-dev perspective.

Answer 6

To look at this a slightly different way:

When a tester fills out a bug report, they typically can't write down exactly what the problem is. They write their interpretation of the problem. When a developer starts working on that ticket, they base their work on their interpretation of what the tester wrote. At each step, there's lots of room for error due to poor communication, faulty assumptions about how the program works, or mistaken observations. Reproduction steps describe the problem in an objective way that retains its accuracy, even through multiple layers of non-technical middlemen that may not all be fluent in the same language.

The other aspect is that reproduction steps provide a "definition of done". If I follow the steps and no longer see the problem, then I've fixed the issue. Without reproduction steps the situation is ambiguous. Did I fix it? Or am I just not doing the right thing to trigger the problem? It's hard to say. Situations like this are where you see bugs go through multiple cycles of fixed->re-opened before it gets fixed for good. It saves time for the tester and the developer if the tester provides the reproduction steps up front.

Answer 7

A few reasons:

1. "Trust, but verify": A large portion of bug reports are not actually bugs but rather mistaken user conceptions of how software should work. Sometimes, the problem is not even in the software they are complaining about: they mixed up the problem in their mind and remembered the flow incorrectly. Software developers know this, and ask for the reproduction steps as a form of verification. The repro steps offer more concrete proof that the issue is real and affects their software.
2. Ease of debugging: As a practical matter, it is much easier to resolve an issue if it can be reproduced consistently. You can attach a debugger and step through each line of code, watching the relevant variables change. Without repro steps, if you don't intimately understand the relevant code, you might not intuitively know where to find the bug. You might then need to add extra logging to all suspect code and wait for another occurrence.
3. Self-education: Developers are often tasked with maintaining code they did not write in a system that they do not use. Repro steps help educate the developer on the context of the problem and how the system is actually used. This is invaluable background that can help them understand potential side-effects of any fix they attempt.
4. Future regression testing: With good repro steps, a developer or QA engineer can more easily write an automated or manual test that can be added to the system to prevent future regressions of this bug.

I should also add that the car mechanic analogy is flawed because good auto mechanics do, in fact, ask for repro steps for issues that are intermittent or hard to diagnose.

Answer 8

There are two somewhat related reasons, I think.

First off, the car mechanic analogy is pretty badly flawed. For one thing, cars wear out in ways that software does not. Also, the mechanic is not the guy who designed and built the car. He (or she) is someone whose job it is to diagnose and repair problems. The role is more akin to QA than design.

Another thing to keep in mind that if someone who's reporting a bug doesn't think it's important to describe the bug carefully, or with reproduction steps, it's probably because they imagine that the bug is so obvious that no one could fail to miss it after even a cursory inspection. We'll come back to that.

Finally, of course, just about any modern software application is almost infinitely more complicated than a "simple" automobile. It's going to have gobs of different features and functions and usage patterns. How one user uses it may have almost no resemblance to how another does.

So the first of my reasons is that, like it or not, developers tend to imagine that their code is perfect. They tried hard to make it so, they tested it before they released it, it passed all its unit tests before they released it. And if nothing else, they know that their code works perfectly when it comes to the things they most commonly test it on. So if you hand a developer a bug report about his code, without much description, imagining it's obvious, the developer may or may not "put the code through its paces", but if he does, it's going to be with his preferred usage pattern -- the one he already knows it will pass handily. So he's totally predisposed to say, "Cannot reproduce, user error".

And then the other reason is is that users are probably making more heavy use of the older and more established features, while developers are always working with (designing and implementing and testing and using) the brand-new features that Marketing wanted in the new release. So if (as not uncommonly happens) a new feature inadvertently introduces a bug into an old, long-established feature, and a user notices and complains about it, and even if it is a glaring, showstopping bug that would be immediately noticed by any casual user, it's still not likely to be noticed by a developer, because the developers never use that old feature any more, they may have forgotten it even existed, because they're all consumed working on the brand-new features.

So, bottom line, careful descriptions and reproduction steps are necessary because the alternative -- just saying, "there's a bug, it's obvious, you'll see it right away" -- is too likely to be false.

Answer 9

To add to the other answers, sometimes of course you do have to investigate a problem that you don't know how to reproduce. We had a customer once with a database corruption, with no idea how it occurred; I was supporting the database software and had to work out a possible sequence of events that could have led to the corruption, working backwards from the symptoms. It took a week, and at the end of it, I had a possible explanation, but no evidence that it was the correct explanation. So the real reason developers ask for a repro is because it reduces the effort needed to find an explanation for the problem, and increases the likelihood that the explanation will be correct.

Answer 10

The Software Bug/Defect might not be a bug with the software itself, but a combination of the Software itself with a specific configuration.

Reproducibility steps do help verify that a specific issue is reproducible, and raises flags to the mechanic if, for example, you tell him the car doesn't have its steering wheel working properly on the road, but it does work in the mechanic's shop when they test the car there.

This could be thought of as similar to browser compatibility issues with a web based application (i.e. the "It works in newer browsers, but not in the browser the client is using" problem), but we can also see this elsewhere in the software configuration.

In some cases, the way a piece of software operates in production versus a staging server, versus a development server, could be widely variant, in cases where the software team recognizes that we don't want certain production aspects to be working as production level things, when in dev environments, or staging environments, but do want them working as production in production environments. For example, an E-commerce site doesn't make actual actionable orders with real money based on developer testing the workflow for adding an item to a shopping cart, or in paying in a shopping cart.

What reproducibility steps help with is determining if the error is in one of those steps. With a car, you might need to add additional details in case it turns out Your car is pulling a Volkswagen style effect in the mechanic's office and appearing to work properly at achieving their preferred goals.

It's been dubbed the "diesel dupe". In September [2015], the Environmental Protection Agency (EPA) found that many VW cars being sold in America had a "defeat device" - or software - in diesel engines that could detect when they were being tested, changing the performance accordingly to improve results. The German car giant has since admitted cheating emissions tests in the US.

In this case, you would likely need to tell your mechanic what was going wrong, especially if they're having issues reproducing it locally - they can't verify if it was broken before they sent it to the environment where the defect is being found.

Ideally, situations like this are rare in practice, but having steps to reproduce will allow a developer to verify that this is consistent behavior across environments, or whether this behavior comes across as a result of a configuration that they will need to look into why that configuration gives different behavior, especially if that behavior involves a different result than the testing behavior implies it should be.

Answer 11

Questions

As a thought experiment how many questions does your software ask and answer?

As an approximation search for words like: if, for, while, interface. Sum up their counts. That is a low ball-park figure.

Now when this software executes it plays a choose your own adventure story walking through each question and bouncing to this page or to that page.

A simple program that does 1 thing like copy a file, still has many paths it can take. A Large file vs a small file, from/to a fast/slow drive, across file systems, different filesystem paths.

A complex program has infinite paths, like just about any GUI program. Every user event occurs at different times, in different contexts, producing different layerings of response.

As a very rough thumbnail. Presume your system has roughly operational life time in seconds * factorial(the number of decisions) unique paths. There is a bug in one, or more of those paths. Find it.

If you aren't quaking about the factorial in that approximation, you really should be. Its get large really fast. The more complicated the program, the even quicker it grows. This is why a large program is guaranteed to have bugs, and why so much effort is put into making programs simpler and smaller.

Bug Reports

This is why it is so essential to describe as much information about the bug as possible.

The basics like:

• What was wrong.
• What it should have been.
• Which screen, report, extract, email.
• When it has occurred
• Who it has occurred to
• Which version

... are important. They help close down a lot of possibilities for where the bug could be lurking.

But in a complex software eco-system its just not enough.

Ideally there would be a testing tool that support have access to that allows them to recreate the situation inside an automated test. Even if the test is slow, and does way more than it really needs to. All of that can be worked on. The important part is that it exactly reproduces the issue.

Short of having that the next best tool is a manual test. Do this, then that, click this, and voila. This is also know as steps to reproduce. The more accurate these steps are, the easier it is going to be to find the bug. If these steps are inaccurate/not there then it is pure luck if the developer spots the bug.

---

Analogy

Also to the description of a mechanic and a driver, its the wrong analogy. Support are the mechanics. The driver of the vehicle is the client. Developers are the designers working on the car production factory.

Support do need to know what components make up the vehicle, and be able to describe accurately the effects of real world usage on those components. IE. Did you know the computer board A, burns out really quickly for high way driving users? In software speak Did you know that the last day of the month batch process X takes 3 times longer to run, but the input is the same size as every other day that week?

If support just communicates back to the developers. Hey your product is broken. you can expect them to be ignored, because of course its broken. It arrived broken, or it has been misused to the point of breaking. That's supports problem.

What is not supports problem is a structural defect with the product itself. Not with how it was delivered, or how it was used. They would communicate this exactly like a mechanic would to the car manufacturing company. In circumstance X, Y, Z. Following these normal system usage patterns. Some P% of the time Q happens. Doing F reduces P% or increases P%. If you do M the Q changes to Q2.

Answer 12

Simple tell them, "if I can't reproduce it, I'm unlikely to have the time to find and fix it."

Testers should always record everything they see (with details about / debug info on what they were doing at the time), even if they can't reproduce it. Once a bug is found, the test team need to figure out what is causing it or it may be impossible to fix. Having a working atmosphere where programmers come over and chat to testers makes a huge difference here as there's often a lot of background information from both sides that does not get communicated in a bug report.

Finding and fixing these bugs can be incredibly challenging for both sides. It may end up being a decision for the project lead - ignore it and put it out or devote more resources to finding and fixing it.

Answer 13

The fundamental problem with this analogue is that Support aren't the customer. They work for the "garage". They're part of the chain of people who are paid to be there in case of "car trouble". If they're getting paid, this analogue doesn't work.

Assuming they aren't rushing to rip up their paycheques, they need to take part in a division of responsibilities to help "fix the car".

Why are they best placed to be the ones to understand and reproduce?

Because they are closer to the customer. They can leverage their closer communication with the customer to figure out how the issue presents and they can use the knowledge they acquire during that process to provide very fast feedback to similar future queries. Passing the buck on this leads to support staff who don't understand the product, which isn't good for anyone.

Because doing this isn't a core competence of software development. Putting developers general work that they have no special ability at is a poor use of an expensive resource. Even if support people weren't cheaper (and let's assume they aren't, because that can come across as offensive), they should be better that this kind of work because it is part of their competence. Good support staff understand the product and how the users use the product. Working together, support and developers can make a better product for everyone by playing to their strengths.

So is it just up to them to change?

No. When Support are in the mode of just forwarding issues, it could be because Developers and Product professionals haven't provided them with the tools they need to understand the system and reproduce issues. They could be resistant to reproducing issues because of something like that, and I suspect you'd only find out that by talking to them.

Answer 14

Your own question is basically a bug report in disguise, and it lacks repro steps.

"There is a bug in our support team. They don't understand the purpose and meaning of repro steps."

So you say, but you have not given the repro steps by which this hypothesis was formed: what are the exact words that they said, in response to what, in what context?

Maybe they were just complaining about an overly rigid bug reporting style.

Bugs are rarely described without implying some of the repro steps. We have that as a separate field in a bug database to encourage detailed, precise, specific repro steps. This is because the issue may hinge on a key detail which is tangential to what appears to be the overall repro step.

However, some bugs really don't need elaborate repro steps; they can be inferred from the bug description. Suppose the title of the bug is Device burst into flames on power-up after Firmware C was flashed. All the info you need is contained in the topic; it hardly needs a Description field any more, let alone Repro Steps. That can literally be the entire bug report. (Still, repro steps could reveal some mistake, like that the submitter deviated from the recommended flashing procedure, or referenced the wrong firmware path name or something. But say none of that is the case. The background assumption is that the correct procedure was followed using the correct inputs; that's the implicitly understood repro step.)

If you have some bug submission validator that won't let you submit a bug without Repro Steps, that would be annoying if you're often submitting bugs which don't require them.

It could also be that the support people are frustrated because they are not getting clear repro steps from the customers, but when they escalate problems to the bug stage, those are required.

The support people may feel that since neither they nor the developers know what the repro steps are, the developers should be the ones to step up to bat and figure out the repro steps, since they know the product internals.

Remember, the support people are handling third parties. So when they talk about taking a car to the mechanic, they may be thinking of taking someone else's car to the mechanic, without a sufficiently detailed problem indication from the car owner. In that case, the mechanic is in the better position to figure out those details than the person who conveyed the car.

Another possibility is that the support people know the value of repro steps but are looking to engineering to help them explain it to customers, just like you're doing by going to StackExchange. That is important because if that is so, the explanation has to work for end users. Your support people are not getting repro steps, and would like some better scripts on how to obtain them, especially from certain irate customers who give them flippant mechanic analogies.

The question, "as a customer support agent, how do I handle customers who insist that the repro steps are a puzzle for support to figure out" is subtly different from "as an engineer, how do I convince my customer support people of the value and importance of repro steps?"

It's hard to believe that you would have people in support roles who are so unqualified that they question the value of repro steps per se. If so, that is a big problem in which the repro steps discussion is just a small part.