Handling a perceived over-reaction to a bug introduced by my team

Question

Background

I run my company's central data team. Other teams depend on the output of our daily refresh pipeline which includes jobs that both my team has written (such as building our core orders and customers tables) and also jobs that other teams have written (specific reporting tables built from those core tables). All failed jobs are pushed to a company-public Slack channel, and we have a support rota to investigate failures and inform affected teams.

One team in particular has some reporting jobs in our pipeline that are required for financial auditing. These jobs often fail due to errors introduced by this team or due to one particularly unreliable external data source. Any failures cause significant disruption to this team and others who depend on them, but they don't seem interested in improving these processes and refuse my help. Their jobs failing doesn't affect any other users of the central data platform so it's not a critical concern of mine.

The alerting Slack channel is often full of this team's job failures, and when the incident in question happened it was the third different failure we'd seen from this team's jobs in that week.

What Happened

Last week one of these jobs failed due to an error introduced by a member of my team - he was doing some routine background optimisation work in an upstream table that duplicated a small number of records. We informed the team immediately when the job failed and fixed it within a few hours. We also held an internal post-incident review and made a couple of technical and process changes to stop this specific issue from happening again.

This was the first time in the 18 month lifespan of my central data platform that any of these jobs have failed because of a bug introduced by my team.

I was therefore quite surprised to see a meeting invite in my calendar involving several senior stakeholders from the other team to discuss how my team can mitigate our impact on their important jobs. They also want us to inform them every time we do work that might impact their jobs.

My Thoughts

I've been doing this job for long enough to know that sometimes things go wrong and that it's a normal part of the engineering process. If there was a prior history of my team breaking things then I'd understand their concern, but there's not. Add to this that they don't seem to care much whenever their jobs break for other reasons, and I'm a bit miffed.

My questions are:

1. What do I mention in the meeting? Do I say "why are we having a meeting when this is the first time we've broken something, and we fixed it quickly anyway?" or do I simply treat it like a post-incident review and walk everyone through what happened and what steps we have in place to prevent it happening again? I don't want to anger anyone or have them think I'm being difficult (I get on well with these people), but I also don't want them thinking that my team is unreliable or risky.

2. Do I agree to inform them every time we do work that might impact their jobs? My team are constantly doing internal tasks around optimisation and refactoring that have no noticeable impact on external teams. I'm worried that telling this team every time we do such a task will a)result in them not understanding the importance when we need to change something that actually impacts them and b)add a tedious "contact this team" step into most of our tasks.

Note

My questions are around how I should handle this specific meeting. I'm not interested in advice about changing data ownership processes, that's already in progress.

Answer 1

do I simply treat it like a post-incident review and walk everyone through what happened and what steps we have in place to prevent it happening again?

Yes, this.

If you don't want them to make a bigger deal out of this incident than warranted, then don't you make it a bigger deal.

Don't be defensive. Don't point out that "they don't seem to care much whenever their jobs break for other reasons". Don't be miffed.

Go in with facts and explanations. Stay calm. Be sensitive to the disruption that was caused by the issue.

Propose a longer term solution, if the vibe in the room seems amenable.

Do I agree to inform them every time we do work that might impact their jobs?

Maybe. That depends on what they could do with the information, and how difficult it would be to produce.

An email saying "We are changing X tonight" is easy to produce. But it's hard to see how they could derive much value from it.

Answer 2

I think the important bit here is

reporting jobs in our pipeline that are required for financial auditing

Sometimes, the point isn't whether a job failed or not; the point is whether this was anticipated, reported, or accounted for. You mention that jobs of this team often fail; but due to their own changes that they are, presumably, aware of, so they can take remediation actions based on their runbook.

On the other hand, this failure was unaccounted for and more importantly unexpected; it likely revealed another source of potential errors that, depending on the organisation and the regulations, needs to be accounted for and have procedures in place. In other words, this signalled that "there's someone else that can modify, in a noticeable way, our processes". This isn't necessarily about being possessive; it can be just a realisation of another source of risk.

This is of course assuming best intentions from all parties; situations like these tend to get a bit political, even unintentionally.

I also don't want them thinking that my team is unreliable or risky.

Given what you say about the incident detection, fast response, notification of the team, and followup actions, it sounds like not describing what happened and evading the answer would actually hinder your goal. Instead, a summary of the timeline, especially how this was detected immediately and intentionally (ie not by accident but via automated monitoring), along with the post-incident actions will instead showcase your reliability (assuming they are receptive to that), instead of giving the impression that you are cavalier with auditing pipelines.

internal tasks around optimisation and refactoring that have no noticeable impact on external teams

Yet in this case it had. It's true that optimisation and refactoring is often the right thing to do, yet it doesn't come without risk. Perhaps in this case, there's additional risk that needs to be addressed (that would warrant this meeting).

Do I agree to inform them
not understanding the importance
tedious "contact this team" step

Is there a particular reason there's a constant stream of such tasks? Can't they be grouped when they affect this particular set of jobs and the work deployed in an agreed window of time (eg after the market closes)?

Also worth keeping an open mind when it comes to the value of this; you mention optimisation but I've seen teams being happier with signing off extra resources to ensure there are no breaks.

Finally, depending on the tone of the meeting, you could suggest that you take some extra time to work out the exact process - a manual step to notify depending on the description of the ticket isn't great either (isolating those pipelines and requiring eg managers sign-off during review, that would then contact the affected team pops in mind as a somewhat more robust alternative). This allows you to make the decision without being directly pressed (obviously freeze any changes to those jobs in the meantime).

Answer 3

Just show the data (if you have it).

I assume you folks have metrics and historical data around failed jobs and some pareto charts of the root causes. Show the data for all jobs and the data for this specific job.

Then explain what you have to prevent the problem from happening again. If you feel particularly miffed, you can add "The other root causes are outside of my domain and I have no visibility what's being done to address that".

Answer 4

At the end of your opportunity to speak at your meeting, announce that you're going to start publishing a weekly incident report that includes all teams. Precisely at that moment, pass out paper copies of a report (or share a PDF) that shows the same info over the last 90 days.

Put the numbers in a columnar (table) format. In the report, don't highlight any one team's activities. Sort by team name alphabetically, and not by number of incidents. This way no one can directly accuse you of throwing shade.

Don't add any commentary to the information you're sharing. Let the numbers speak for themselves. If anyone asks you about the other team's numbers, share your source of just the numbers and absolutely defer any "why" questions to be answered by the persons who run that other team. Again, no one can accuse you of throwing shade, and you can get out of this hot seat that the other team is trying to put you in.

Answer 5

If this team had 99 failures that were their own fault, and didn't do anything about it, and they had one failure that is your fault, and now they are all upset and want you to take costly actions, then it is clear that they want to shift blame from themselves to you.

So I'd do a quick count, and in the meeting say "team X's job failed yesterday, and that was our fault. In the three months before this, the same job failed 97 times. Each time it was team X's fault. We offered to help them to fix their problems, but they have not taking any action as far as we can see, and they have refused to accept our help. I suggest that it is much more effective to reduce all the other failures in their jobs first. ".

Answer 6

As pointed out in a few other comments, before you go into this meeting, you need to do some self-reflection on your team and their processes.

Last week one of these jobs failed due to an error introduced by a member of my team - he was doing some routine background optimisation work in an upstream table that duplicated a small number of records.

There is no such thing as routine background optimisation. Refactoring and optimisation tasks are critical areas of concern. All logic that depends on your logic assumes by now that it works, confirmation bias kicks in and other teams may have even removed checks and balances or might ignore notifications if it does start to fail.

• A software developer who assumes that their code is bug-free and does not perform thorough testing or debugging is a common example of confirmation bias in software engineering.

Your metrics for testing are not always likely to meet the organic requirements that other teams now have after using your code for so long. So even if all of your original test cases still succeed, you wont know what boundary conditions or quirks to your logic they may have adapted to.

• I once worked on a project porting to product to a new runtime. It was a requirement to replicate all of the existing bugs in the original software. Sounds crazy, but their staff and automated systems had already factored in and worked around those issues, if we fixed them now, all of the logic that depended on thos bugs would have to be reviewed and now fixed, which was well outside the scope of the project.

Due to the potential of this domino effect that your changes might have throughout the system, such changes should be coded and tested in isolated environments that simulate or replicate production loads. If your solution is split up so that there are multiple teams, then changes should only be introduced after scheduling deployments with all teams and performed in a way that you can rollback if there are any significant issues.

• If it aint broke, be careful that you're not making the situation worse!

We informed the team immediately when the job failed and fixed it within a few hours.

It is great that you communicated the issue and had it fixed, the irony of situations like this is that had you not raised it, they may never have known. But honesty is the best policy, by raising the alarm early you have given them opportunity to mitigate problems at their end as well.

We also held an internal post-incident review and made a couple of technical and process changes to stop this specific issue from happening again.

The main thing you need to take into this meeting is your findings from that review. You'll need a timeline of events and some detail about your new preventative measures to reassure everyone that it won't happen again, or at least to ensure you can identify the issue and have a strategy to rectify it even sooner.

What you cannot do...

You cannot really bring up any of the history about either your team's success record or the fact that their team gives you all sorts of issues. When their team causes issues, they know how to deal with it, by the sounds of things this is by contacting you. This situation is different, they can survive their own incompetence thanks mostly to you, but when you are incompetent, now they have zero trust in the system, the next issue they will not be certain if it is you or them.

Perceived over-reaction

You haven't really justified why this is an overreaction, for them your team has been stable for so long, "what has changed, why are we suddenly seeing issues now?" It is good that they do have a strong reaction, finance departments often will, it means that your work is important to them and they recognise that and potentially want to help.

In this meeting you need to build back the trust or goodwill that was lost, but don't do it by making promises you can't keep. Just state facts and confirm the communication channels and ask them if your response was adequate. Also ask them to evaluate your resolution and mitigation strategy, make them feel like part of the solution and have them confirm that your solution is infact adequate and can be trusted, don't tell them it is, make them confirm it.

Do I agree to inform them every time we do work that might impact their jobs? My team are constantly doing internal tasks around optimisation and refactoring that have no noticeable impact on external teams.

YES this is important, they clearly depend on your system for their day to day work! If you cannot explain the importance of a change to dependant teams, then is it really that important? It is a fact of your chosen delivery and maintenance model that contacting all affected teams is an important part of the process, because you have demonstrated that your quality control was not good enough to identify this issue before it was rolled out. This is part of earning that trust back.

But this is also a good time to reflect on why you are doing so many changes into the production environment. The first step is to batch changes together to reduce deployments, but you should liaise with the other teams and setup a staging or test environment so that you can test deployments before actually rolling them into production.

no noticeable impact on external teams

This is the key, this is your opinion. If you had tested extensively and verified that there will be no noticeable impact, then what was the purpose of the change? In your communication about the deployment you might use language like "We do not expect a noticeable impact" but you still need to follow with a brief explanation to justify that claim and how you have verified it.

The communication is not the tedious step, it is the testing that is causing you issues. Communication of release notes is critical and should never be skipped. Even if you have a great DevOps pipeline and automated deployments, you should still have a collated list of the features that have changed because when the deployment fails, we need to know where to go looking for the root cause of issues.

I appreciate it is more of a longer term objective, but this should be the objective of all teams that deploy into production environments.

Answer 7

I think that you should use this opportunity to really assess your process. In fact, what baffles me is that you push changes in production without even thinking about notifying your users. This is bad, no matter how exceptional your team is. This looks to me not as an unavoidable accident, but rather a flawed process that you should fix. When I read "he was doing some routine background optimisation" I felt almost sick: you don't do that. Please realise that what you call routine is not a good practice and should not happen outside release windows.

As I said, I'd use this occasion to introduce changes to your process. You definitely need to be more formal for new releases (you should agree a calendar with the user), organise proper testing windows and communicate with users the rationale for each new release. And this is what you should say in the meeting: the fact that the other team introduced bugs is not an excuse for you to not follow best practices.

Answer 8

I agree with you and Joe that, yes, you should do this:

do I simply treat it like a post-incident review and walk everyone through what happened and what steps we have in place to prevent it happening again?

However let me propose 4 points to hit specifically.

---

Root Cause

What went wrong? Who, What, When, Why, Where, etc.

Impact

How many stakeholders/applications/hosts/rows are affected? What are the exact effects that stakeholders should expect going forward?

Fix

How do we plan to fix the issue and redress stakeholders in the immediate and long term?

Prevention

How do we ensure that this issue or ones like it will not occur again?

---

I think on that last point you may get some pushback from the stakeholders, but if you show competence on the first 3 they will likely defer to your expertise, especially if you demonstrate a strong grasp of the impacts and the fix.

Answer 9

Do I say "why are we having a meeting when this is the first time we've broken something, and we fixed it quickly anyway?"

As others have already noted, do not do this, however emotionally satisfying it seems like it would be. In fact, I would suggest getting this out of your system. Pick someone not at work who is likely to take your side, a friend, spouse, significant other, or other family member. Say all these kinds of things to that person, then when they agree with you, take the other side and explain why they shouldn't.

I simply treat it like a post-incident review and walk everyone through what happened and what steps we have in place to prevent it happening again?

As already noted, do do this. This is just a perfectly normal meeting after a problem. Already said, but worth repeating.

All that said, you should take data and statistics to the meeting with you. Some things that might help in rebuttal to things that people might say:

1. How many failures versus how many successes. If your team is deploying without problems 99+% of the time, that's good. Be ready to say that if someone implies that your team is making mistakes.

2. How many of the failures were caused by the external agent, the team responsible for those reports, and your team. A response to someone complaining about the frequency of problems with those reports and blaming the problems on your team.

3. What your proposals are for making the reports more robust in general. In case someone asks what you would suggest they do.

4. How many of the changes that you made this year would have been reported at various standards of likelihood to impact the team. Also note which of those standards would have included this problem. I mean, you say that this was an upstream table, so it's not immediately obvious to me that you should have notified them even if you were notifying them of changes that might impact them. Perhaps it would be obvious if I knew more about your data structure. Anyway, discussing multiple notification standards and their likely impact would be useful if 99% of the changes that you would report had no actual impact on them at all.

   My electric utility company twice warned me of potential power outages from weather that might have caused issues. In both cases, I experienced no problem. Another time, they did not warn me (the wind storm was not predicted beforehand). My power was out for three days and my internet for five (a tree fell on the lines between my house and the street). Somewhat problematic, as I work from home. Net result is as you might expect. I tend to ignore the warnings, as they are prepared for those and there aren't problems. But I am thinking of mitigation steps for unexpected occurrences.

   Anyway, my point is that I suspect that you are right, they will quickly learn to ignore your warnings except when their stuff breaks for the normal reasons. Then they'll assume that it was the warned activity and not their own actions. This is likely to make them react slower to problems rather than allow them to better anticipate things that might go wrong.

   You might emphasize this by showing them the last twenty or hundred times that you would have notified them under each standard, including the incident. Then ask them which one was the notification related to that incident. See if they can recognize it under the way that you would have notified them. I.e. would "modify process populating table in [this] database to be more efficient" have let them know the potential impact to them? Even if you had added that that data would later be used to generate some data in their report(s).

I want to reiterate that this is probably not the time to be proactive in bringing things to their attention. For one thing, you may get them to start thinking about something that they hadn't considered yet. Also, stating information up front (other than the post-incident review) is likely to make them perceive you as defensive. They'll discount the information. If you use it in rebuttal, it will sound stronger. If you don't have to use it at all, then great. Perhaps they never even had that thought.

Answer 10

Its called a 'postmortem'.

Its not an over reaction. Its not an accusation.

The way you are responding is immature as you are making it personal and tribal.

I was previously part of a team and their reactions to postmortems was always one of personal attack and, in my view, not constructive. Its not about you or your job. its about the product. you are there to build and maintain a product and culture of continuous improvement.

Bring to the meeting: "What happened?" "What can we do to ensure this does not happen again?"

Irrelevant to this meeting is the ongoing problems with the customer team. Totally irrelevant. Call a postmortem with them the next time an error throws if it bothers you and solve the problem.

Answer 11

Others have answered most of the angles, but I'd like to pick up on one part that jumped out at me:

We also held an internal post-incident review and made a couple of technical and process changes to stop this specific issue from happening again.

I hope that I'm reading that wrongly, (but if I am, then others will too) - you need to make it clear to everyone else that your review attempted to ensure that other issues won't occur either, not JUST this specific issue.

We all know that it is impossible to predict all future issues, but if you can show that you are being pro-active, then you really don't need to worry about anything else. If the other team still grumble, I would ask for a representative to attend your meetings for both awareness and input.

Answer 12

Others have answered your first question pretty well, so I'll just address the second question about notifying them whenever you make a change.

People don't like surprises. When these jobs fail due to their bad input, it's not a surprise to them, they're used to it. But other types of failures are not normal to them.

When you make a change to an application, the other stakeholders for that application should be kept in the loop. They don't need to know the technical details if it will be over their heads (so they probably shouldn't be included in the VC notifications) , but a "heads up" is good practice. Something like

We tweaked the X job to improve performance. We don't expect a problem, but let us know if you see something wrong.

This shouldn't really be an extra step for this application, it should be routine for all your applications that impact other groups. Maybe there's even some way you can automate it.

Answer 13

Their jobs failing doesn't affect any other users of the central data platform so it's not a critical concern of mine.

This is the important feature of this issue. The other team depends on the quality and timeliness of your work, not the other way around. It's completely reasonable for them to be concerned with the delay caused by your team, and to follow up with you to make sure it doesn't happen again. How often the other team's jobs fail is not your concern insofar as it has nothing to do with the quality and timeliness of your work.

I also don't want them thinking that my team is unreliable or risky.

Did anyone on the other team explicitly say your team was unreliable or risky? If they explicitly say so then feel free to bring up your team's high reliability over the last 18 months, otherwise this is not even an issue.

My team are constantly doing internal tasks around optimisation and refactoring that have no noticeable impact on external teams. I'm worried that telling this team every time we do such a task will a)result in them not understanding the importance when we need to change something that actually impacts them and b)add a tedious "contact this team" step into most of our tasks.

Bring this up during your meeting with the stakeholders on the other team.

Answer 14

A lot of good answers already but I wanted to add a bit since I do the same job.

In the data world, we've adopted various software practices but have tended to build monoliths. One consideration to raise with stakeholders is the degree to which the financial audit team depends on "central" data models. There's a trade-off - rely on other people's stuff risk errors or build it yourself but have more work to do.

In addition you might want to limit which tables they can reference. Tables that can be publicly (used by people outside your team) referenced are like interfaces. This means people cannot just reference "shaky table to support fun emoji analysis" when doing serious financial stuff without you knowing. You can just agree the rules initially but you can embed them as code later (as part of your continuous integration flow).

In a past workplace, I grouped tables by which team owned them. Each teams tables are by default private and they have to be made public before a different team can reference them. Same for central team owned models. This stops people creating dependencies without both parties agreeing.

Answer 15

Just answering on a tiny detail.

I also don't want them thinking that my team is unreliable or risky.

Don't let this end up as an evaluation of your worth, neither from them or yourself. Quality of service is a big, unclear, difficult thing to handle. If a process is fragile, or somehow unreliable, that may be OK or very problematic. Neither answer condemns you, or your team, as "bad" in some way.

Desperately trying to make a fragile thing look solid, for ego or politics reasons, is a thankless place to be.