5

There are many development practices (TDD, continuous integration, cowboy-coding), principles (SOLID, layers of abstraction, KISS), and processes (RUP, Scrum, XP, Waterfall). I have learned you can't follow any of these blindly, but have to consider context and ROI (return on investment).

My question is: How do you know whether you are getting a good ROI by following a particular methodology? Metrics, guesstimation, experience? Do analytical methods exist? Or is this just the million-dollar question in software engineering that has no answer?

For clarification when I say ROI I mean overall business value. This does not necessary exclude developer happiness--if all the developers quit (or worse: become apathetic) after a 6-month death march you lose all the technical experience in the organization. If customers are unhappy you will need more customer support staff or lose business.

0

2 Answers 2

2

There are a lot of case studies on the DACS (Data & Analysis Center for Software) site, e.g.

WHAT IS THE ROI OF AGILE VS. TRADITIONAL METHODS?

An analysis of XP, TDD, Pair Programming, and Scrum (Using Real Options) by Dr. David F. Rico, PMP, CSM. Little is known about the costs and benefits of Agile Methods since their popularization in 1999, though 67% of projects use them and 75 books and 100s of papers have been written about them. The purpose of this article is to analyze the costs and benefits reported in studies of new product development approaches such as Agile Methods as compared to those of Traditional Methods. Over 300 articles on Agile Methods were examined; cost, schedule, productivity, quality, and customer satisfaction data were found in 69 studies; and ROI data were identified in 29 studies. Agile Methods ROI was four times more than expensive Traditional Methods, two times less than inexpensive ones, and the best Agile and Traditional Methods had equal ROI. However, it may not be proper to compare Traditional Methods optimized for productivity and quality to Agile Methods optimized for customer satisfaction, project success, and risk reduction.

(source: http://www.thedacs.com/databases/url/key/3990/4944/7956)

1
  • Thanks, I haven't read any other paper that performed such a comprehensive review. I am not convinced that the metrics used are accurate, but I will definitely investigate further. Commented Apr 28, 2012 at 14:53
1

How to define a "good" return on investment in relation to software development is probably the more relevant question.

ROI can have many meanings. In the simplest terms, you are assuming there will be a direct correlation between the cost to implement and use a system, and the resulting gains or losses once the system is in place. Much of this can be relative however. You might mean ROI for the life of a project, or for the life of a company, or you may wish to make a distinction between weeks, months, or even years that a system is in use.

Is your ROI going to be purely financially based, or will you consider other more ephemeral values as a part of your ROI definition. Things such as Customer satisfaction, or team and management satisfaction. How about the difference between hitting your deadlines, or missing them, or in being able to complete more or less project work? You may even consider a combination, where you assume that implementing a system will cost you financially up front, but that the cost may be worth the non fiscal outcomes.

Case studies and white papers rarely take all of this stuff into account when trying to determine if the effort will be worth the cost to adopt a new system of working, which leaves us at the mercy of people's perceptions, and which as a result often results in disaster for teams where a methodology champion manages to convince management that system-x is the panacea the company has been looking for, as often these champions are either novices, or non-programming managers. So where does that leave us?

There are no solid figures or hard and fast formula's that you can apply to determine if a specific methodology is the solution to all of your problems, and whether the risks will be worth the rewards. So how do we know that something can or will work without actually trying it? For that you need the situation to be analysed from the perspective of Experience, Logic, and an intimate understanding of the needs of all of the stakeholders of a project - by which I mean the programmers, company management, and the customers.

Anyone who has written software for 10-15 years will likely have experienced a number of methodologies, and will have experienced projects that have worked, and failed. This experience can give you a great basis for forming opinions about the value of a particular methodology, however without a good working knowledge of the needs of the stakeholders that will be affected by a methodology, even the experienced developer is likely to implement a system based on a predefined methodology that most closely the developer's own perceived needs.

The thing about Agility, is that in order to be truly Agile you need to understand that there is no one specific methodology that will satisfy absolutely everyone. Therefore, what is needed is to define a system that utilizes the practices that will most closely match the needs of all of the stake-holders. This means that a methodology needs to be tuned to satisfy the team, the company, the customer, and the project, all as a group, but also individually. In terms of determining ROI, this is a combination that makes measuring a "good return" very difficult to measure, as each stakeholder will view value and success differently.

In terms of relative success however, the success will only be known once the project is completed. While this may seem a little scary up front, being Agile means that you don't lock yourself in to a specific system. You tune your methodology up front, then you review often and tune the methodology as you go, until either the project ends, or the methodology requires no further tuning. By doing this, you maximize your chances that your methodology will evolve to a point where it will be of value to all of the stakeholders. As to how to measure the success of your methodology, that comes down to a number of factors. Are the developers happy working with the methodology? Does the management have enough control over the process? Does the customer get what they have paid for? Is the customer satisfied with the result? Are targets and outcomes definable, and can those targets/outcomes be realized? The most important measure however is whether all of the stakeholders can agree that the way things are done now is better than the way things were before.

I realize that I've described much of this as being after the implementation of a methodology. In real terms, the proof of a methodology can only be guaranteed by doing it. With experience however, you can apply your knowledge of the things you have done before that didn't work very well, and if you have learned valuable lessons from them, you can use this information to determine up front whether a particular approach will be worth your time and effort. Therefore, you can call it an educated guess, based on prior learning, in order to minimize risk. The result is that you can be reasonably certain of the value of any given methodology, and you would act on a decision to implement a methodology if you can accept the risk that your chosen method may fail, or may require further tuning. If you feel you can live with the risks, then you might believe it's likely to achieve a reasonable return on your investment, and implementing your chosen methodology will - with tuning - most likely work for you as you'll have a vested interest in making it so. This too is another intangible item that is almost impossible to objectively measure.

2
  • Thank you for the answer, I also believe evolutionary processes are the most successful. But are there any useful quantitative measures for "tuning a process" or is it based entirely on "feel" of the developers and stakeholders? Commented Apr 28, 2012 at 14:37
  • @GarrettHall It's less "feel" and more about observation. As the project progresses, you review the process frequently to identify causes for inefficiency. If the team notes that missing information causes delays, find out why the information isn't available and fix it. If the team has a lot of long meetings, either streamline the meetings, or remove them altogether. If customer interferes with development, manage the relationship with the customer. If tasks take too long to complete, break the tasks down. If something is difficult to make more efficient however, you may need to experiment.
    – S.Robins
    Commented Apr 28, 2012 at 23:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.