How to handle accountability for mistakes in a team as a manager?

Question

I currently have a position as a team leader inside a company (20-30 employees) who offer a service while using internal software. My team is of 5 developers and a product manager, and our main responsibility is to maintain and implement new features into this internal feature, which is the tool used by all employees every day.

As a background, all members of the team have been in the company ranging from 6 months to 2 years, while I hold the highest seniority inside the team. Something to note is that there's a difference in age between myself and some team members. I'm in my 20s while some of them are in their 30s, but with less work experience in the IT field. I report directly to the CTO of the company while handling the role of lead developer inside the team.

The problem I'm facing is the following: during the last few months, we've caused bugs while developing new features or fixing other bugs in the software. Some of these bugs didn't have a high effect on the business (for example a page not being displayed correctly) but others did have an impact on the revenue of the company. This means that we affected very important features.

On every bug that happened, I made sure that the person who caused the issue fixed it and knew its impact. My role inside the team is to review all new written code, so some of those bugs were also over-looked by myself during the review. Since we don't have a QA engineer, the product manager takes care of testing all features when delivered.

Every time, the employee apologized for causing the issue, and it's clear from analyzing them that they're issues that could have been solved if the code was tested enough or if the employee paid more attention. Every 2 weeks (or sprint, we use SCRUM), we hold a meeting to discuss what went well/not well, and I've told the team that we need to pay more attention to all changes and be more strict when developing new features to avoid problems.

Even after that, issues continue happening, so I want to know what will be the most effective way to communicate to the team that type of dynamic cannot continue happening. I have the following questions:

1. How can I effectively communicate to my team that we need to be accountable/careful for issues caused?
2. Should I use the opportunity inside the 1-on-1 meetings with each team member and discuss it personally instead of addressing the whole team?
3. How can I provide feedback to the product manager (who doesn't report to me) that they're not testing enough and need to be more attentive on that part of the process?
4. I've been discussing these issues with the CTO and followed her advice, but didn't improve the situation. How can I notify her that I need her intervention to communicate with the project manager while clearing that I'm holding myself accountable for the team as well? I want to avoid office politics issues where I appear as someone putting faults on someone else.

Notes:

• To avoid these issues, we're implementing currently automated tests to reduce the risk of bugs in the software.
• I'm holding myself accountable as well since these issues should be solved during the review process. Since I'm also participating in the development tasks, I end up with no time. I'm trying to change the team's workflow to be able to free up more of my time.

Answer 1

I want to know what will be the most effective way to communicate to the team that type of dynamic cannot continue happening.

This isn't an effective way of addressing the issue.

If you want developers to spend more time manually testing their changes, or generally be more cautious and careful when coding, this will mean a slower - most likely significantly slower - development process. It's very unlikely that any potential cutting of corners (if that's the reason for these bugs) is due to team members not caring about the quality of the product, so appealing to them to essentially work harder isn't a long-term solution. Telling them to reprioritise on manually testing for issues instead of coding new features is an option, but this may be hard to convince your bosses to accept. You're essentially offering a reduction in bugs in exchange for a permanent and significant reduction in productivity, as far as they're concerned. It's also a very inefficient way of trying to find bugs compared to automated testing.

What you need to do is to change the development process to include more automated testing, and a better code review process (you shouldn't be the only person doing code reviews). This will have an upfront cost on productivity but once the team settles into the new system, it can be overall faster (as well as less prone to bugs) than the current approach. There are other answers that discuss what changes might be useful, and other questions on this site about how to lead that transition/how to sell it to your boss, but from what you've mentioned in your question this is what you need to solve the problems you are having. And this is definitely something that, as team lead, you should be in responsible for.

Answer 2

If you want your team to be effective.. You need to remember that you blame the process, not the team member. As doing otherwise leads to a significant drop in team moral and serious case of both CYA (cover your ass) and NJH (new job hunting). Despite the mass layoffs at Facebook/Twitter its still pretty easy to find work.

From the sounds of it, your development process is atrocious. Your development process can be summed up as "we write code and pray to the machine god that it works as intended with no side effects." You don't have QA, automated unit tests, automated functional tests, or--it sounds like--a good test environment.

it also sounds like there might be a lot of technical debt/fragile code in your software. Fixing bugs/ and adding new features shouldn't impact anything except things directly related to the bug/ feature you are adding on a well designed system. In a well designed system you shouldn't need to do any testing outside of the code you are working on (you still do because good testing habits run thru your blood).

While this is ultimately on your team members as they should have pushed back against the companies lack of good development processes, against your PoS code base and for better coding practices, it sounds like many of them are not very experienced at software engineering, and so might not know better.

What you need to do? Have a discussion with your team and openly talk about the number of bugs and difficulties that they are experiencing while writing software. You need to get buy in from your team members to improve your development practices. Good development practices should be branded onto their hearts.

To implement these good development processes, improve your testing environment, write unit tests, and functional tests for existing features... If you have a GUI, automate testing that as well, there unit testing frameworks for everything. I would think about assigning someone on your team to spend half their time in charge of QA. Balance out the cost of bugs with the cost of slower development, and some other team members to building a bunch of regression tests... if bugs are costing you enough money spending the time/money to improve your code base will pay itself back.

Also start doing test driven development... Don't write code, and then test it to make sure it works. Write unit tests that test for the desired behavior then write code that implements that behavior.

All of this will slow down your rate of development, and cost time/money. But once you have paid back your technical debt, you should see an increase in efficiency and speed because good development/testing process give you coding confidence.

Answer 3

The Questor has written a good answer about the technical side of this - however here is what I'm going to address:

Leadership starts with yourself

If people see someone heading in what they think is the right direction, they will want to follow. It's up to you to set the standards for yourself and that will inspire people to follow/adhere to them.

There's two reasons I say this - firstly because I believe it to be true and secondly - in relation to your specific issue:

"I reviewed the code and missed the Bug"

Start there. Start with that conversation "Here is what I didn't do, and here's how I'm going to improve on it"

You will need to hold yourself accountable - but if people see that, they will want to do likewise.

All the technical problems can be fixed by technical solutions - this is what will fix the culture.

Answer 4

Having no dedicated QA is absolutely the problem. People making mistakes is unavoidable. These mistakes costing the company revenue, that’s avoidable by having QA.

One problem is that you have inexperienced developers. An experienced developer would have told you this a long time ago. And software developers cannot test their own code. It’s because having written the code, they subconsciously don’t want to find bugs. Finding bugs makes them look bad. With a QA person, finding bugs makes them look good.

Answer 5

Several key principles may help you here.

First, let's look at the team-level items:

1. Since you're using SCRUM, it might help to clarify your "Definition of Done" for your tasks. Does "Done" only include initial coding? Does it require internal testing, as well? And how thorough is the limited-audience public review prior to distribution of each module? If your developers think "done" means, "I got it to run," then there will always be buggy code. If, on the other hand, coders know that "done" means well-verified, and know that their code will be rigorously tested by some end user, with a potential for additional hassle and embarrassment (not shame--just the natural consequences of having to correct your own code after a recall), they will generally code more responsibly.
2. In front of your entire team, your product manager (PM) should be clear about the consequences of developing buggy code. There are a couple of principles here. First, according to the SCRUM model, the specification (including quality level) of the product is the responsibility of the PM. If the PM doesn't care about buggy code enough to specify the parameters, it's going to be harder for your team to care about buggy code. If all the PM cares about is getting modules developed, your team is going to respond to that. So find out if the PM cares (and if they don't, perhaps help them understand the consequences of emphasizing speed). Second, when a team understands the consequences of releasing buggy code, they will naturally choose to be more diligent. However, if you keep them shielded from the consequences, their motivation for correcting their behavior is low. So have the PM help them see the costs, even to the extent that it means passing the consequences down to the developers responsible for them (maybe not the financial costs, per se, but the additional workload, the hassle of having to revisit old code, the velocity impediments from switching tasks, etc.)

Next, let's look at the individual-level question you asked: Should you address the bug issue in 1-on-1 discussions with team members?

1. You want to present vision, standards, and processes generally, so that everyone gets to participate, gets the same instruction, and commits to the same goals.
2. You want to correct mistakes privately, clearly, and quickly. Help your developers know that their mistakes are not evidence that they're "bad" coders, but just that their approach is not as efficient as it could be. Your approach in a 1-on-1 can incorporate principles you have previously discussed in the group setting--reiteration can be helpful--but also make sure you work to understand any obstacles to a particular developer's compliance. Make it a 1-on-1 dedicated to helping them feel more successful, and if they are naïve to the needs of the team, see if you can invite them to "participate," to "help solve a problem."

Now, let's look at the CTO/PM/you triangle:

1. This is a tricky structure, and a bit not-SCRUM like. What makes it complicated? A) You and the PM both "lead" your team, but you don't necessarily have the same priorities. B) Further, lines of accountability are not clear/aligned--if code is buggy, does all of the blame fall to you? It should fall at least as much on the PM. C) The direct line from the CTO to you means that discussions about bugs are going to likely hit you first, because you can do things about the bugs more quickly.
2. To remedy these things, I recommend a re-alignment of accountability, with most of the changes moving toward the PM. A) You both still lead the team, but your role is to help your team meet the PM's objectives as fast and effectively as possible. You let them choose the direction (including level of quality), and you lead your team to that objective. B) While you are responsible for bugs in the code, the PM is responsible for releasing a buggy product. If they are incentivized by the speed at which you work (fixing forgotten bugs is SLOW work), and penalized when clients turn over because they encounter bugs, then they will choose to help you focus on cleaner code. Re-aligning the rewards/consequences will naturally help them to value the bug-free environment/process. C) It will likely feel ostentatious, but I recommend talking with the CTO about ways you can improve your processes, and NOT about bugs that need to be fixed. Conversations about bugs that need to be fixed should be had between the CTO and the PM. The CTO is kind of like your scrum master in this setting--they give you resources, remove obstacles, etc., but you get your list of tasks from the PM.

Finally, let's consider one personal-level item:

1. While there is always potential for politics and positioning, especially when resources are scarce, remember that a gracious, helpful perspective/mindset and approach will get you MUCH further than a cynical or critical one. If you want the PM's help, see if what you do can further their objectives somehow. If you need your team's buy-in, find out what's important to them. As you work together to help everyone win (including yourself), you'll succeed as a company.

Answer 6

As other answers have alluded: it is the team (organization) as a whole that bears the responsibility. The corollary is: how to improve the team so as to deliver higher-quality code and thus not impacting the business negatively? Pointing fingers is counterproductive; but if a pattern emerges, leadership should initiate corrective action.

A lot of good points were made re. what is wrong and what could be done; I want to add some more observations from my experience. I see some similarities between OP's circumstances and my own: team of ~7 developers, project ~2 years old (so no devs working on it longer than that), product used by the company's "resellers", Agile-ish process, no official/structured QA. Differences: older people, mostly 30-50-ish (so some more experience and maturity - I for one know a lot more than I did even at 30). Big international luxury goods org.

1. Complexity: initially the product was quite easy to grasp in totality, but as new features and integrations were added, one would only vaguely be aware of some other functionality, and not also the intricacies of how it was implemented. There's also a lot of concurrency, which is not always handled safely. This complexity is (IMHO) the biggest source of introduced bugs: something was added and tested just fine by the dev, but s/he was not aware of how some adjacent code is impacted by it, and some very subtle bugs got introduced. (No real finger-pointing about new bugs: a new bugfix story gets logged, estimated, done whenever; or if it's urgent, one or more of us would do a mob programming session and get it fixed ASAP.) Another point of personal opinion: Functional Programming makes it easier to reason about the flow of data and execution, unfortunately a lot of team members have not caught on yet, so you get state that gets mutated in one place affecting some other functionality, and it is not always quite visible that it does. Another BIG help would have been detailed documentation: e.g. architecture and functionality, so one could read up on what is expected and fit in. However this would take a lot more man hours, and it seems most orgs choose the tradeoff where they would rather respond to user needs very quickly.

2. Knowledge: Technology gets updated very fast, not all our team members have even a working knowledge of the WHOLE tech stack, and there seems a bit of penny-pinching reluctance from management to get people upskilled constantly. Time is under pressure... Also, it's not always feasible to self-study: some areas require access to expensive resources and/or real-world scenarios. Result: things break because people simply didn't know.

3. Testing: we do have a strict (automatically enforced) policy of unit testing and a minimum required code coverage threshold. Problem with metrics is that now people code to pass the metric... Still does not catch all bugs or regression. Again a personal opinion: TDD is difficult as I often find myself not knowing how something should be implemented, so there's a lot of experimenting until it works. Unit test are added afterwards. So in my case it would be perhaps better to use the unit/integration testing as a kind of "proof that functionality works", for all possible use cases (if only functional requirements were documented...).

4. Documentation: I mention documentation a lot. The sad truth is that if one's requirements are all spread out in (often badly worded) Jira stories, with changes in subsequent stories, it is nigh impossible to get it perfect or look up the "should have been" - people are again trying to keep it in their imperfect memories. Ideal would be to have them all compiled into an accessible, coherent source. (I'm fully convinced that if Documentation were part of the work products, Agile could easily accommodate updating them just as code is updated - you lose time now but gain clarity (and thus time) later - question remains with the tradeoff: is it worth it to you? And if not, then put up with the consequences.)

5. Code Reviews: The Github pull request tool makes it quite difficult to get an overall picture if it is a more complex change because only the changes are highlighted (and if you are in the web browser, you don't have access to all the helpful IDE features). While our whole team participates (2 approvals required), I feel in many cases I get a better idea of the code if checking it out and tracing through it, reading the IDE's hints, or even running it. And by that time others have already approved and the author has merged. Get a code review process that adds real value and isn't just a rubber stamp.

Long story short: nobody makes mistakes on purpose, but cognitive complexity makes them overlook things. Organizing things helps to simplify the complexity in a lot of disparate, orthogonal areas and different levels of abstraction. And to make things worse each human's cognitive abilities differ, so you have to cater for your team's lowest common denominator (which can potentially be a fresh-out-of-college new hire).

Answer 7

The thing with developers is that they (we) usually fix known problems, but what software development needs, is testing unknown problems. Like the others have said, QA is a step you can't skip. Especially a team of 5 developers is a large one, in my experience a small project with 2 developers is the maximum you can go without dedicated testers. It's not enough to test each feature with it's intended usage, you need to have a test plan to go through with every release to go through all test cases with intended usage, corner cases, unintended usage and impossible scenarios. It's also important to have someone from the target audience testing, which, as it is an internal tool for employees, might be already covered, but you need to make sure there is no knowledge gap that makes developers/testers miss important ways to use the application.

Another thing is, you simply can't have a requirement of "we have no bugs". Unknown things will always happen. "Test until there is no more bugs" is not a useful thing to tell to the development team. Neither is apologizing. What could be useful (in addition to having QA), is finding out what are the types of bugs most often happening. Is there a complicated swirl of features, and fixing bugs causes more bugs? You need refactoring and possibly improvements in architecture. Are the bugs appearing in different devices, OS's or browsers? Make sure to test all environments, here automatic tests could especially help. Is there lot of responsiveness issues and need to create exceptions for different screen sizes? You'd have most likely needed a better approach or libraries to begin with, if you are required to handle responsiveness issues manually case by case, bugs will be neverending. Are there beginner coders making simple code logic bugs? Code review process needs improvement. Performance? UX? Your project specific information is needed to understand where the issue is.

Answer 8

You don't fix process by burning witches

If every developer is producing code that adds bugs, the problem is almost certainly not every developer. The problem is the process of how you develop.

Even if it was only half of the developers that are producing bugs, it is a process and management problem.

If all you do is start punishing developers for releasing bugs, you'll get every developer who can get a job elsewhere doing so, leaving behind the rest who will almost all be unhappy.

You need to fix your process.

High Quality Code is Expensive

People will try to sell you "just do X" and your problem goes away. But your real problem is that high quality code is expensive compared to the code you are producing right now.

It can easily be 10 to 100 to 1000x as expensive as the process you describe.

Now, it does reduce the rate at which you introduce new bugs. But it will also massively increase the cost of fixing old bugs and introducing new features. Sometimes this cost increase is so large that the company cannot survive it, and it goes under.

I have no time to improve quality

If you (as an organization) have no time to do the development you are currently doing, then no, nothing you will do will prevent new bugs from occurring.

There is no cheap fix.

There are things you can do to marginally improve the quality of your bug fixes -- many things -- but it definitely won't prevent even the majority of bugs from getting through without a significant decrease in (at least short-term) productivity.

1. Serious and effective automated testing way more than doubles the cost of a code base. And deferring it doesn't make it cheaper. Currently your code base lacks automated testing. So tooling it with serious automated testing will require as much development effort as it took to create your code base in the first place, and probably more.

2. Extensive manual QA is an expensive process. QAing effectively most changes requires more person hours than writing it. And even getting it down to this level requires a bunch of it to be automated.

3. Full review of changes is more work than writing it.

4. Creating changes that don't break effective automated tests significantly more work than making a cowboy change that just works.

5. Entire domains of coding, like asynchronous code, graphics, and a bunch of others, require a high level of skill that can only be gained through a pile of expensive failures. Either you experience those failures internally, or you hire someone who has that expertise. If you don't want these failures to impact customers, this means an insanely thick buffer between developers and customers so you can catch these errors, possibly at the level of developing entire products and discarding them. When google buys a company they usually discard the code base, because it isn't up to google standards, but they hire the employees, because those employees know what they did wrong the first time.

Marginal, "Cheap" fixes

These should double development costs, but have a hope of reducing your regression problem.

1. Have every change done twice independently with no communication. Once both changes are fully approved, compare the two changes and determine which one is better and safer, or if a combined one would be even better.

Each changer has an incentive to find problems in the other change now, even if just out of pride.

2. Develop continuous deployment. Offer different levels of stability to your customers (nightly, weekly, monthly, and quarterly releases). Hire a large QA team (including engineers-in-test) to pound on your nightly builds and raise issues.

3. Require new automated tests for every new bug fix. When a bug is fixed, there must be an automated test that failed before the fix, and passed after the fix. These tests must be on behavior of some subsystem or API, and the failed/success test cannot be arbitrary. Ie, no "this vector has 3 elements" is a pass; it must be "the result is nonsense or incorrect".

Ideally the test should be cheap, but for some kinds of bug that isn't possible. An expensive test is still required.

Using git bisect or similar the location where the fix is introduced and (ideally) where the bug is introduced must be possible with an automated harness.

Run as many of these bug tests nightly as you can. If you can't run them all, run a random subset. If a regression bug happens, git bisect with the last time it passed to find when it was introduced.

New changes have to pass the entire suite of tests, even the more expensive ones. This may require a pile of computers whose only job is to run these tests.

...

All of these aren't cheap. But they are cheaper than a full on fix, and should reduce your rate of introducing new bugs.

On the other hand, telling an employee they are bad because they introduced a regression they should have fixed is free.

Answer 9

Your people are not the problem, your systems are

People make mistakes. People will always make mistakes. The solution to your problem is not to find people who don’t make mistakes because those people don’t exist.

You need to design and implement systems that expect mistakes and minimise their impact. Even then, your systems are designed and implemented by people. People who, as I believe I mentioned, will make mistakes. So your systems will fail.

You need to also be aware that there is a non trivial risk that a system you implement to avoid failure will introduce new sources of failure.

You can’t eliminate failure, you can only be prepared for it

Answer 10

Due to the complexity of software, it is unrealistic to expect zero bugs, however most bugs can be caught before they do any damage with a three stage process consisting of dev, code review and testing.

Each step in the process must be handled by a different person, for example a junior dev might write the initial code, a more senior dev will do the CR, and testing should be performed by a dedicated tester.

Due to this, there are three opportunities to catch the error, and while all three people are responsible, the blame is shared, so there is minimal drama.

But also if the code is going to affect a critical system, there should be an initial risk assessment done by the project manager which will tell you whether it is a good idea to slow down on this particular task and spend extra time on quality control.

All of this is assuming you're already following basic principles like being able to roll back the deployment easily, keeping backups, having up to date docs and ensuring the team is a cohesive unit. Avoiding a culture of blame will help with the last point, though there is only so much you can do if that blame culture comes from the upper management.