Why do I need SCRUM vs. a less formal, more lightweight process for my team?

Question

I'd like to begin my question by saying that I understand that SCRUM or some derivative of it is probably a good way to go for managing software development. It seems all the big companies and my managers use it or have used it, and I can't really argue with all that experience. However I'm struggling to understand the "whys" and all the reading and even my official SCRUM training at work is not doing the job for me. It's just all rhetoric. So I come here seeking answers.

Until now, I have developed in teams of 4-5 members very effectively, completely self-organized and without the need for any training, methodology, or special software. Just discussions in cubes, ad hoc meetings, and one-on-one code reviews. I am now in a position at work where we're being told SCRUM is the way to go, and everything that comes along with it. When they describe SCRUM to me, I read stuff like this:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That's great, but all of it seems like common sense to me. Why did this need codified? Then I'm told the methodology helps us respond to change. What specific aspects of SCRUM are allowing me to be so flexible that I was not previously achieving with my ad hoc meetings, cube discussions, and developer planning meetings? They explain the need to have a working deliverable every two weeks, or sprint. In my particular project, there is no "client", the software won't be finished for a year or more, and in the meantime I will probably only be demoing to upper management every month or less. So why the explicit need for a deliverable every other week? They emphasize the importance of the sprint planning meeting where the entire team lays out the stories and tasks for the next sprint. This is no different than the impromptu planning meetings I've had in the past. Why must it occur every other Monday, and why does the entire team have to be involved? I understand the concept of every member "owning" the product, but the fact is, only a few individuals can ever really contribute to breaking each story up into tasks, while the rest just watch idly.

Again, I understand that the majority of people are behind this process, and so it must work, and I need to get on board. I'd just like to understand why. Is my issue that I already practice these things and just don't like unnecessarily codifying them? Or perhaps I've yet to see the advantages of these techniques because they're being done improperly? Any real, personal information or advice on this, as opposed to the spiel I'm used to receiving, would be extremely appreciated.

Answer 1

I think there are two aspects to answer your question, but let me start with congratulating you for working with people who seem to be smart and competent enough to be able to pretty much work without a strongly defined process and still deliver a product. Unfortunately this isn't a case in all software teams, so maybe one of your issues with Scrum might be that you and your co-workers actually don't need a process dumped onto you to make you more effective. You might already be effective.

Other teams aren't and need a more strictly defined process and some guidelines to get them to get things done. This doesn't mean that these teams are more stupid or less capable, it just means that maybe they have problems self-organizing or working with discipline as a team. This can also be a simple learning experience when coming from a place where people work mostly alone to working together as a team. Scrum can help getting there, because it offers a few guidelines that are both easy enough to understand and follow, yet effective enough to put some pressure on the team to start getting it together.

Since Scrum doesn't say anything about the way that software development should be done it also leaves the team with the freedom to decide for themselves (e.g. you can still do a sprint applying a rather conservative waterfall method as long as you are done at the end of the sprint).

So the team is one issue. The other issue is management and management trust. Here, Scrum might be good because it's transparent and allows any stakeholders to see the progress the team makes in defined cycles. So it's not "we're making progress, unfortunately we can't show you anything right now, but believe us, we'll be done on time". This may be even true, but it can be reassuring for any managers to actually have a regular demo where they can see that progress has indeed happened.

Scrum is not a silver bullet. It may not work for some teams for a variety of reasons, maybe for some teams self-organization doesn't work out. Maybe for you it's the other way and it seems like a process dumped on an already productive and organized team.

When in doubt I would pretty much suggest you just try it and see. If it doesn't work and the greater part of the team doesn't like working that way, don't do it. However, check it out for a couple of months (I say a couple of months, because the first few sprints will be awkward anyway and you need time to adjust the details) and then re-evaluate.

Answer 2

Might be controversial, but Scrum is best to diminish management fears of Agile, or to use with an already under-performing team. If your team is running great, meeting goals, making money, and happy, Scrum is not going to buy you anything because all it will do is upset the good balance of activities that you do (and make your team successful). Scrum is not a silver bullet. In my experience with it, it only helps teams that had poor estimation and communication to begin with. A team working with realistic estimates in an environment of effective communication is only hindered by the process overhead of Scrum.

Believe it or not, good software teams did exist before Scrum came along. Scrum helps the bad ones get better.

Answer 3

Most of the answers here have already enumerated the reason, though a bit indirect. Anne's answer is especially illuminating when she touches on transparency. That is, allowing managers to see what's going on. And Schultz answer touches on this as well when he talks about people not being able to hide the fact that they are slacking off.

So I'd like to say what others are already saying but in a more direct language: the main goal of SCRUM is not to manage software development, the main goal of SCRUM is to measure software development.

Other systems have tried before and people have proposed countless metrics to try and measure software development but have failed. SCRUM turns the problem on its head and shifts the burden of measurement away from managers and onto developers themselves. The logic is simple: who better to estimate how long it takes to do something than those doing the work themselves?

Now, the problem with this is that programmers are well known for being too optimistic. Ask a programmer how long it takes to do something and he will typically underestimate how hard the task actually is. SCRUM provides the tools to control this:

• daily meetings to gauge progress and get a big-picture view of the project
• estimates are done in "points" instead of hours/days to abstract away time
• burn-down charts and tortise/hare charts to visualize the velocity of points
• stories and tasks on a board to get an overall view of workload
• sprints and iterations to act as deadlines so we can measure progress
• specific roles for scrum master, owner and team member to avoid the temptation to cheat

etc.

You may notice that all the above mainly does two things:

1. They measure work. Either work to be done or work being done or work completed.
2. They try very hard to avoid the problem of the overoptimistic programmer to get a better, more realistic estimate.

The longer you implement SCRUM the more accurate you will find your estimate to be. In fact, I personally believe running sprints + a backlog + a burn-down chart alone is enough to cure most programmers of making bad estimates on how long it takes to do something.

Answer 4

Personally I think the purpose of SCRUM is to satisfy older organizations where upper management cannot or will not get behind a leaner process. I've been working as an architect (Chicken) for about a year in a department that heavily utilizes SCRUM. My prior background is Silicon Valley startups most of which used a much leaner, ad hoc and highly iterative (sometimes weekly or even daily pushes) feature focused process. I find SCRUM, at least the way we implement it to be overkill in term of process (and in some ways more heavyweight than waterfall (at least from the developer perspective). To be faire, I will say that one aspect of our process that deviates is that our product owners are actually more akin to requirement analysts in the IT organization. As a result they tend to dull the information coming from the business and worse leave the business unaccountable to the development team (which requires regular successive infusions of user stories). Nonetheless, in my future startup, I wouldn't use a SCRUM. I'd probably only use it in the situation where management requires the added overhead.

Answer 5

I will not talk from a purist's perspective. I feel that you are able to execute it in somewhat similar to what Scrum says. However the main point is it is your ability to run the show. What will happen if you are on a vacation for a month?

I see scrum as mechanism to streamline all that you have been doing and put some defined aspects on that. So that in your absence somebody else can also replicate it and can replicate it to other projects as well. However scrum is not a silver bullet. I have seen many people who just started using Scrum (because it is in fashion) and got beaten badly because they didn't understand the essence of it.

PS: Scrum does not mandate that your sprint has to be two weeks long. It can be month long (your case).

Answer 6

Please see my comments to the question first.

SCRUM is an agile software development paradigm. As such, it's agile itself. It doesn't assume you must follow its classical model. And I doubt if anyone does actually. I used to work in several organizations and every team adapted it to their needs. It's not unusual there is no customer/consumer when it comes to releasing some public product/library/API. I never had one. In my case, our GM acted as one, which IMO was like having none.

Having 2 weeks sprints is tough. Very tough. 3 weeks is better but is really for experienced and familiar with the process team. We had 4 weeks or a month. That gave us enough time to "settle" so to speak in the beginning and have more confidence in the end due to more throughout testing. I liked that and I'd stick to 3 weeks at least.

The other team I was collaborating with, didn't have anything but backlog. They would get together, report on status and what's next and that's it. Once everything was done, they would come up with another backlog. They knew it wasn't SCRUM but it worked for them and that's what important.

Is it more lightweight? It definitely is. But it's not SCRUM. What I like about SCRUM is it promotes discipline. People feel pressure of delivering something everyday. Everyone knows what others do and he fails, everyone will know that. Even if one tries to cover that up (read lie), it becomes obvious much sooner than with other processes. So when you diverge and simplify as with that team, you have to be sure you do that with right people. Otherwise it may just fall apart really quickly degrading to meaningless status meetings where everyone would just stay and think "what do I do here? i know what I need to do so whatta hell?"

That's my two cents. I do my own SCRUM like development: plan work, split to tasks, estimate them, observer the progress. It really helps me to be on top of things. I applied some things from SCRUM to projects I outsourced and it worked out great for me.

Just... stay agile ;-)

Answer 7

I recommend you ignore scrum. In a couple of years a new fad will come along, and you will be less cynical and still be able to embrace it wholeheartedly. In fact you could quickly become an expert. Then you can make lots of money by writing a book on it and speaking at conferences.

Since a lot of things are cyclical, most likely this new fad will be a heavy weight process similar to RUP. What will have happened you see is that everyone will have followed light weight agile processes, and these will be blamed for their project failures. So of course the logical solution is that more up front planning and design is required !

Seriously though, I don't think you need Scrum. There is nothing in scrum that is better than other competing agile processes. Also it has a lot of stupid names for things.

Answer 8

That's great, but all of it seems like common sense to me. Why did this need codified?

Scrum is usually compared to older, more heavyweight methodologies. The methodologies tried to make the feedback-less waterfall work by enforcing more documents, more sign-off, and more planning up-front. The Agile manifesto (which you are quoting) was a reversal of those ideas.

Then I'm told the methodology helps us respond to change. What specific aspects of SCRUM are allowing me to be so flexible that I was not previously achieving with my ad hoc meetings, cube discussions, and developer planning meetings?

It sounds like you have an agile structure already. If you are already responding to change well, then you obviously don't need help. Some processes become so hidebound with procedure that getting a bug fixed requires a full analysis and functional design phase, and can't get into the project until next year, at the earliest.

They explain the need to have a working deliverable every two weeks, or sprint. In my particular project, there is no "client", the software won't be finished for a year or more, and in the meantime I will probably only be demoing to upper management every month or less. So why the explicit need for a deliverable every other week?

Original Scrum prescribes month-long sprints. There's a nasty trend towards Agile machismo in shortening sprints. ("Yeah, well our sprints are only one day...") The Customer/Client is whoever has the authority to say that the project is good to go, or needs more work. If you're demoing to upper management every month, they're probably your customer, and you're probably very close to Scrum already.

Based on your description of what your team is doing, Scrum is probably not much different. You might get some value out of standardizing, because it will be easier to explain to outsiders what's going on if you use the Scrum terms. Also, Scrum can be used a shield; for example, Scrum prescribe that technical decisions should be made by the team -- pointing out this principle can be a good way to get technical value that is otherwise hard to sell (Pair programming, for example.)

Scrum is basically an interface that your team can implement. If you do, then you have a good idea about how to communicate with those outside the team, and they have a good idea about how to communicate with you. Only you can know if your team needs this.

Answer 9

We don't use Scrum at work. We use a methodology founded in Agile and Lean which is similar in many respects. I've used this process in teams varying in size between 3-5 people including the lead. Although there are differences I think you may be able to help you figure out if Scrum is useful for your situation.

Making the Methodology Explcit

We make our process explicit because we review our process with each sprint wrap-up/review. Part of the wrap-up/review is to identify practices that aren't working for us. We also discuss practices that we think will work for us and if there is enough agreement we will try it out. We would not be able to do this without codifying our methodology.

Sign-off

This is the workhorse for our process. This guarantees what we write is what is needed. When we get a particular feature we go to our customer. The customer is whomever is going to use what your writing. In some cases you have to proxy the customer with someone who represents the customer (like product management). These are our steps, and until the last step is complete the feature is not done.

• Get the feature from the board, tracker, etc.
• Go talk to the customer and understand what they are looking for before writing anything.
• Implement the feature.
• Show the customer the working feature in it's final form Have the customer sign-off on the feature being complete.

Vertical Slices

We do all of our development in vertical slices. This supports the ability to do sign off with a completed feature as well as these other reasons.

• Amortizes integrations issues by rolling them in with each feature. Helps to eliminate crunch time at the end of a project.
• Allows us to cut out features easily because we eliminate alot of the cross dependencies.
• Allows us to cut off development if we need to change direction.
• We can do iterative releases, providing the customer with functionality early.

Acceptance TDD

We leverage unit test frameworks for acceptance tdd. This gives us many benefits.

• Large restructing doesn't cost alot of test reworking time.
• We assure customer functionality.
• We cover code integration.
• Support Vertical Slice development practice.

The Build is Always Releasable

After every push the code should be releasable. Even if the feature is incomplete, nothing should be broken. All the tests should run, and all previous functionality should be present. This is really an extension of our vertical slice development. As such it shares many of the same benefits.

• Allows us to cut out features easily because we eliminate alot of the cross dependencies.
• Allows us to cut off development if we need to change direction.
• We can do iterative releases, providing the customer with functionality early.

Continuous Integration

Every push generates a build via our CI build server. The build server checks out the code, runs through the entire test suite, tags the code, and packages it up for deployment. Reinforces our policy that the build is always releasable.

Point Estimation for Cards

Every bug, feature and task becomes "card". A card is the the smallest logical unit of work that has some customer benefit. We point these out according to our scale. Any which exceed our maximum point value or broken down further. We have found the larger the point value, the more deviation there can be in the time to completion, hence breaking large cards down further. If the card has enough ambiguity, it get's rounded up to the next point value in the scale. Each card must get signed off before it can be considered complete. Proper estimation support our ability to calculate Velocity.

Velocity

We have week long sprints. Every Friday we do work planning and prioritize the work for next week. Based on our velocity we have a good idea of how much "work" we can accomplish within the week. Our velocity is the mean and median of the total points completed within the week. Increases in standard deviation are analyzed for bad estimations (which are always trying to get better at), or increased interruptions (which I talk to the manager about). The velocity can also be used to estimated an accurate completion date for a project, but only if it's the sole project being worked on.

Incremental Improvement and Design

We also have a policy to leave the code in at least a little better state than how you found it. Refactor/redesign the code at the beginning of a card. The goal is to account for organic growth that can be prevalent with incremental development. We also refactor at the end per normal.

For the most part these are the rules we follow and why we follow them.

Answer 10

It sounds to me like you're in a very experienced, high functioning team. Congratulations, Scrum/Agile is basically formalizing what your team has intuited all this time.

I think what may be to your (entire) company's advantage is adopting Scrum as a "common ground", not between the members of your development team, but between your development team and the business department at large.

While Scrum Sprints are timeboxed, teams can select between the recommendation ranging from two weeks to 1 month. Any lesser and there would be too many Reviews and Retrospectives, and any more might hamper the business' ability to change direction within a year. It sounds like you've found your sweet spot of 1 month so push for that.

There's a lot of leeway for you to adjust Scrum parameters and I'm sure you can explain to your business that you're still doing Scrum the right way. One upside is that if you meet the business halfway, their involvement may yield positive support.