How to keep a big and complex software product maintainable over the years?

Question

I have been working as a software developer for many years now. It has been my experience that projects get more complex and unmaintainable as more developers get involved in the development of the product.

It seems that software at a certain stage of development has the tendency to get "hackier" and "hackier" especially when none of the team members that defined the architecture work at the company any more.

I find it frustrating that a developer who has to change something has a hard time getting the big picture of the architecture. Therefore, there is a tendency to fix problems or make changes in a way that works against the original architecture. The result is code that gets more and more complex and even harder to understand.

Is there any helpful advice on how to keep source code really maintainable over the years?

Answer 1

The only real solution to avoid code rot is to code well!

How to code well is another question. It's hard enough even if you're an excellent programmer working alone. In a heterogeneous team, it becomes much harder still. In outsourced (sub)projects... just pray.

The usual good practices may help:

1. Keep it simple.
2. Keep it simple. This applies especially to the architecture, the "big picture". If developers are having hard time to get the big picture, they are going to code against it. So make the architecture simple so that all the developers get it. If the architecture has to be less than simple, then the developers must be trained to understand that architecture. If they don't internalize it, then they shouldn't code in it.
3. Aim for low coupling and high cohesion. Make sure everyone in the team understands this idea. In a project consisting of loosely coupled, cohesive parts, if some of the parts becomes unmaintainable mess, you can simply unplug and rewrite that part. It's harder or near impossible if the coupling is tight.
4. Be consistent. Which standards to follow matters little, but please do follow some standards. In a team, everyone should follow the same standards of course. On the other hand, it's easy to become too attached with standards and forget the rest: please do understand that while standards are useful, they are only a small part of making good code. Don't make a big number of it.
5. Code reviews may be useful to get a team to work consistently.
6. Make sure that all tools - IDEs, compilers, version control, build systems, documentation generators, libraries, computers, chairs, overall environment etc. etc. - are well maintained so that developers don't have to waste their time with secondary issues such as fighting project file version conflicts, Windows updates, noise and whatever banal but irritating stuff. Having to repeatedly waste considerable time with such uninteresting stuff lowers the morale, which at least won't improve code quality. In a large team, there could be one or more guys whose main job is to maintain the developer tools.
7. When making technological decisions, think what it would take to switch the technology; which decisions are irreversible and which are not. Evaluate the irreversible decisions extra carefully. For example, if you decide to write the project in Java, that's a pretty much irreversible decision. If you decide to use some self-boiled binary format for data files, that's also a fairly irreversible decision (once the code is out in the wild and you have to keep supporting that format). But colors of the GUI can easily be adjusted, features initially left out can be added later on, so stress less about such issues.

Answer 2

Unit tests are your friend. Implementing them forces low coupling. It also means that the "hacky" parts of the program can easily be identified and refactored. It also means that any changes can be tested quickly to ensure they don't break existing functionality. This should encourage your developers to modify existing methods rather than duplicating code for fear of breaking things.

Unit tests also work as an extra bit of documentation for your code, outlining what each part should do. With extensive unit tests your programmers shouldn't need to know the whole architecture of your program to make changes and use exiting classes/methods.

As a nice side effect, unit tests will also hopefully reduce your bug count.

Answer 3

Everybody here is quick to mention code rot, and I completely understand and agree with this, but it still misses the bigger picture and the bigger issue at hand here. Code rot doesn't just happen. Further, unit tests are mentioned which are good, but they don't really address the problem. One can have good unit test coverage and relatively bug free code, however still have rotted code and design.

You mentioned that the developer working on a project has difficulty implementing a feature and misses the bigger picture of the overall architecture, and thus implements a hack into the system. Where is the technical leadership to enforce and influence the design? Where are the code reviews in this process?

You are not actually suffering from code rot, but you are sufferring from team rot. The fact is that it shouldn't matter if the original creators of the software are no longer on the team. If the technical lead of the existing team fully and truly understands the underlying design and is any good at the role of being a tech lead, then this would be a non-issue.

Answer 4

There are several things we can do:

Give one person overall responsibility for architecture. When choosing that person, ensure they have the vision and skill to develop and maintain an architecture, and that they have the influence and authority to help other developers follow the architecture. That person should be a seasoned developer who is trusted by management and who is respected by their peers.

Create a culture where all developers take ownership of the architecture. All developers need to be involved in the process of developing and maintaing architectural integrity.

Develop an enviroment where architectural decisions are easily communicated. Encourage people to talk about design and architecture - not just in the context of the current project, but in general, too.

The best coding practices make architecture easier to see from code - take time to refactor, to comment code, to develop unit tests, etc. Things like naming conventions and clean coding practices can help a lot in communicating architecture, so as a team you need to take time out to develop and follow your own standards.

Ensure that all necessary documentation is clear, concise, up-to-date and accessible. Make both high- and low-level architecture diagrams public (pinning them to the wall can help) and publically maintainable.

Finally (as a natural perfectionist) I need to recognise that architectural integrity is a worthy aspiration, but that there can be more important things - like building a team that can work well together and actually ship a working product.

Answer 5

The way I go about this issue is to snip it at the root:

My explanation will be using terms from the Microsoft/.NET, but will be applicable to any platform/toolbox:

1. Use standards for naming, coding, checkins, bug flow, process flow - basically anything.
2. Don't be afraid to say goodbye to team members who doesn't adhere to standards. Some developers simply cannot work within a defined set of standards and will become 5th column enemies on the battlefield to keep the code-base clean
3. Don't be afraid to allocate lower skilled team members to testing for long periods of time.
4. Use every tool in your arsenal to avoid checking in rotting code: this involves dedicated tools, as well as pre-written unit tests that test the build files, project files, directory structure, etc.
5. In a team of about 5-8 members, have your best guy do refactoring almost constantly - cleaning up the mess the others leave behind. Even if you find the best specialists in the field, you will still have a mess - it's unavoidable, but it can be constrained by the constant refactoring.
6. Do write unit tests and maintain them - do NOT depend on the unit tests to keep the project clean, they do not.
7. Discuss everything. Don't be afraid to spend hours to discuss things in the team. This will disseminate the information and will remove one of the root causes for bad code: confusion on technologies, goals, standards, etc.
8. Be very careful with consultants writing code: their code will, almost by definition, be the real shitty stuff.
9. Do reviews preferably as the process step before checkin. Don't be afraid to rollback commits.
10. Never use the open/close principle unless in the last stage before release: it simply leads to rotting code being left to smell.
11. Whenever a problem is encounted, take the time to understand it to the fullest before implementing a solution - most code rot comes from implementing solution to problems not fully understood.
12. Use the right technologies. These will often come in sets and be fresh: It's better to depend on a beta version of a framework you are ensured support from in the future, than to depend on extremely stable, but obsolete frameworks, that are unsupported.
13. Hire the best people.
14. Lay off the rest - you are not running a coffee shop.
15. If management is not the best architects and they interfere in the decision making process - find another job.

Answer 6

Clean rotted code by refactoring, while writing unit tests. Pay down (this) design debt on all the code you touch, whenever you:

• Develop a new feature
• Fix a problem

Greatly speed your test-first development cycle by:

• Refactoring to convert code modules to a scripting language
• Use fast, cloud-based test machines

Refactor code to use low coupling (of highly internally-cohesive units) by:

• Simpler, (more) functions (routines)
• Modules
• Objects (and classes or prototypes)
• Pure functions (without side effects)
• Preferring delegation, over inheritance
• Layers (with API's)
• Collections of small, one-purpose programs which can operate together

Organic growth is good; big up-front design is bad.

Have a leader who is knowledgeable about the current design. If not, read the project's code until you are knowledgeable.

Read refactoring books.

Answer 7

Simple answer: you can't.

That's why you should aim for writing small and simple software. It's not easy.

That's only possible if you think long enough about your seemingly complex problem to define it in as a simple and concise manner as possible.

Solution to problems that truly are big and complex can often still be solved by building upon small and simple modules.

In other words, as others pointed out, simplicity and loose coupling are the key ingredients.

If that's not possible or feasible, you are probably doing research (complex problems with no known simple solutions, or no known solution at all). Don't expect research to directly produce maintainable products, that's not what research is for.

Answer 8

I work on a code-base for a product that has been in continuous development since 1999, so as you can imagine it is pretty complex by now. The biggest source of hackiness in our codebase is from the numerous times we've had to port it from ASP Classic to ASP.NET, from ADO to ADO.NET, from postbacks to Ajax, switching UI libraries, coding standards, etc.

All in all we've done a reasonable job of keeping the code base maintainable. The main things we have done that contributed to that are:

1) Constant refactoring - If you have to touch a piece of code that is hacky or hard to understand, you are expected to take the extra time to clean it up and are given the leeway in the schedule to do so. Unit tests make this a lot less scary, because you can test against regressions more easily.

2) Keep a neat development environment - Be vigilant about deleting code that is no longer used, and don't leave backup copies/working copies/experimental code exist in the project directory.

3) Consistent coding standards for the life of the Project - Let's face it, our views on coding standards evolve over time. I suggest sticking with the coding standard you started with for the life of a project unless you have time to go back and retrofit all the code to comply with the new standard. It's great that you are over Hungarian notation now, but apply that lesson to new projects and don't just switch mid-stream on that new project.

Answer 9

Since you've tagged the question with project management, I've tried to add some non-code points :)

• Plan for turnover - assume that the entire development team will have disappeared by the time it hits its maintenance phase - no developer worth their salt wants to be stuck maintaining his / her system forever. Start preparing handover materials as soon as you have time.

• Consistency / uniformity cannot be stressed enough. This will discourage a culture of 'go it alone' and encourage new developers to ask, if they are in doubt.

• Keep it mainstream - technologies used, design patterns and standards - because a new developer to the team (at any level) will have more chance of getting up and running quickly.

• Documentation - especially architecture - why decisions were made, and coding standards. Also keep references / notes / roadmaps into documenting the business domain - you would be amazed how difficult it is for corporate business to explain what it is they do to a developer with no domain experience.

• Lay down the rules clearly - not just for your current development team, but think about future maintenance developers. If this means putting a hyperlink to relevant design and coding standard documentation on every page, so be it.

• Ensure that the architecture and especially code layers are clearly demarcated and separated - this will potentially allow for the replacement of layers of code as new technologies come along, for example, replace a Web Forms UI with an HTML5 jQuery UI, etc., which may buy a year or so of added longevity.

Answer 10

One property of highly maitainable code is function purity.

Purity means that functions should return the same result for the same arguments. That is, they should not depend on side effects of other functions. Additionally, it is useful if they do not have side effects themselves.

This property is easier to witness than coupling/cohesion properties. You don't have to go out of your way to achieve it, and I personally consider it more valuable.

When your function is pure, its type is a very good documentation by itself. In addition, writing and reading documentation in terms of arguments/return value is much easier than one mentioning some global state (possibly accessed by other threads O_O).

As an example of using purity extensively to help maintainability, you can see GHC. It is a large project about 20 years old where large refactorings are being done and new major features are still being introduced.

Last, I don't like the "Keep it simple" point too much. You can't keep your program simple when you are modelling complex things. Try making a simple compiler and your generated code will likely end up dead slow. Sure, you can (and should) make individual functions simple, but the whole program will not be simple as a result.

Answer 11

In addition to the other answers, I'd recommend layers. Not too many but enough to separate different types of code.

We use an internal API model for most applications. There is an internal API that connects to the database. Then a UI layer. Different people can work on each level without disrupting or breaking other parts of the applications.

Another approach is to get everyone read comp.risks and The Daily WTF so they learn the consequences of bad design and bad programming, and they will dread seeing their own code posted on The Daily WTF.

Answer 12

Since many of these answers seem to focus on biggish teams, even from the outset, I am going to put my view as part of a two-man development team (three if you include the designer) for a startup.

Obviously, simple designs and solutions are best, but when you have the guy that literally pays your salary breathing down your neck, you don't necessarily have time to think about the most elegant, simple and maintainable solution. With that in mind, my first big point is:

Documentation Not comments, code should be mostly self documenting, but things like design documents, class hierarchies and dependencies, architectural paradigms, etc. Anything that helps a new, or even existing, programmer to understand the code base. Also, documenting those odd pseudo-libraries that pop-up eventually, like "add this class to an element for this functionality" can help, since it also prevents people from re-writing functionality.

However, even if you do have a severe time limit, I find that another good thing to keep in mind is:

Avoid hacks and quick fixes. Unless the quick fix is the actual fix, it is always better to figure out the underlying problem to something, and then fix that. Unless you literally have a "get this working in the next 2 minutes, or you're fired" scenario, doing the fix now is a better idea, because you aren't going to fix the code later, you're just going to move onto the next task you have.

And my personal favorite tip is more of a quote, though I cannot remember the source:

"Code as if the person that comes after you is a homicidal psychopath that knows where you live"

Answer 13

• Be a scout. Always leave the code cleaner than you found it.

• Fix the broken windows. All those comments "change in version 2.0" when you're on version 3.0.

• When there are major hacks, design a better solution as a team and do it. If you can't fix the hack as a team, then you don't understand the system well enough. "Ask an adult for help." The oldest people around might have seen this before. Try drawing or extracting a diagram of the system. Try drawing or extracting the use cases that are particularly hacky as interaction diagrams. This doesn't fix it, but at least you can see it.

• What assumptions are no longer true that pushed the design in a particular direction? There might be a small refactoring hiding behind some of that mess.

• If you explain how the system works (even just one use case) and find yourself having to apologize of a subsystem over and over again, it's the problem. What behavoir would make the rest of the system simpler (no matter how hard it looks to implement compared to what is there). The classic subsystem to rewrite is one that pollutes every other subsystem with its operating semantics and implementation. "Oh, you have to groz the values before you feed them into the the froo subsystem, then you un-groz them again as you get output from the froo. Maybe all values should be groz'ed when read from the user & storage, and the rest of the system is wrong? This gets more exciting when there are two or more different grozifications.

• Spend a week as a team removing warnings so that real problems are visible.

• Reformat all the code to the coding standard.

• Ensure your version control system is tied to your bug tracker. This means future changes are nice and accountable, and you can work out WHY.

• Do some archeology. Find the original design documents and review them. They might be on that old PC in the corner of the office, in the abandoned office space or in the filing cabinet nobody ever opens.

• Republish the design documents on a wiki. This helps institutionalize knowledge.

• Write checklist-like procedures for releases and builds. This stops people having to think, so they can concentrate on solving problems. Automate builds wherever possible.

• Try continuous integration. The sooner you get a failed build, the less time the project can spend off the rails.

• If your team lead does not do these things, well that's bad for the company.

• Try to ensure all new code gets proper unit tests with measured coverage. So the problem can't get much worse.

• Try to unit test some of the old bits that are not unit tested. This helps cut back the fear of change.

• Automate your integration and regression test if you can. At least have a checklist. Pilots are smart and get paid lots and they use checklists. They also screw up pretty rarely.

Answer 14

One principle that has not been mentioned but that I find important is the open / closed principle.

You should not modify code that has been developed and tested: any such piece of code is sealed. Instead, extend existing classes by means of sub-classes, or use them writing wrappers, decorator classes or using whatever pattern you find suitable. But do not change working code.

Just my 2 cents.

Answer 15

Read and then re-read Code Complete by Steve McConnell. It's like a bible of good software writing, from initial project design down to a single line of code and everything in between. What I like most about it is that it is backed up by decades of solid data; it's not just the next best coding style.

Answer 16

I've come to call this the "Winchester Mystery House Effect". Like the house, it started simple enough, but over the years many different workers added on so many odd features without an overall plan that nobody really understands it anymore. Why does this staircase go to nowhere and why does that door only open one way? Who knows?

The way to limit this effect is to begin with a good design that's made where it's flexible enough to handle expansion. Several suggestions have already been offered on this.

But, often you'll take a job where the damage has already been done, and it's too late for a good design without performing an expensive and potentially risky redesign and rewrite. In those situations, it's best to try to find ways to limit the chaos while embracing it to some degree. It may annoy your design sensibilities that everything has to go through a huge, ugly, singleton 'manager' class or the data access layer is tightly coupled to the UI, but learn to deal with it. Code defensively within that framework and try to expect the unexpected when 'ghosts' of programmers past appear.

Answer 17

Code refactoring and unit testing are perfectly fine. But since this long-running project is running in to hacks, this means that the management is not putting its foot down to clean the rot. The team is required to introduce hacks, because somebody is not allocating sufficient resources to train people and analyze the problem/request.

Maintaining a long running project is as much a responsibility of project manager as an individual developer.

People don't introduce hacks because they like it; they are forced by circumstances.

Answer 18

I just want to place a non-technical issue and a (maybe) pragmatic approach.

If your manager doesn't care about technical quality (manageable code, simple architecture, reliable infrastructure, and so on), it becomes hard to improve the project. In this case it is necessary to educate said manager and convince to "invest" efforts into maintainability and addressing technical debt.

If you dream with code quality found in those books you also need a boss that is concerned about this.

Or if you just want to tame a "Frankenstein project" these are my tips:

• Organize and simplify
• Shorten functions
• Prioritize readability over efficiency (when acceptable of course, and some efficiency gains are too miserable to be kept)

In my experience, programming is entropic rather than emergent (at least in the popular imperative-structured paradigm). When people write code to "just work" the tendency is to lose its organization. Now organizing code requires time, sometimes much more than making it just work.

Beyond feature implementation and bug fixes, take your time for code clean-up.

Answer 19

I was surprised to find that none of the numerous answers highlighted the obvious: make the software consist of numerous small, independent libraries. With many small libraries, you can construct a big and complex software. If the requirements change, you don't have to throw the entire codebase away or investigate how to modify a big honking codebase to do something else than what it is currently doing. You just decide which of those libraries are still relevant after requirements change and how to combine them together to have the new functionality.

Use whatever programming techniques in those libraries that make the usage of the library easy. Note that e.g. any non-object-oriented language supporting function pointers is supporting actually object-oriented programming (OOP). So, e.g. in C, you can do OOP.

You can even consider sharing those small, independent libraries between many projects (git submodules are your friend).

Needless to say, each small, independent library should be unit-tested. If a particular library is not unit-testable, you are doing something wrong.

If you use C or C++ and dislike the idea of having many small .so files, you can link all libraries together into a larger .so file, or alternatively you can do static linking. The same is true for Java, just change .so into .jar.

Answer 20

Simple: reduce the maintenance costs of most of your code down to zero until you have a maintainable number of moving parts. Code that never needs to be changed incurs no maintenance costs. I recommend aiming for making code truly have zero maintenance cost, not trying to reduce the cost over many small and fussy refactoring iterations. Make it cost zero right away.

Okay, admittedly that's a lot, lot harder than it sounds. But it's not hard to get started. You can take a chunk of the codebase, test it, construct a nice interface over it if the interface design is a mess, and start growing the parts of the codebase that are reliable, stable (as in lacking reasons to change), while simultaneously shrinking the parts that are unreliable and unstable. Codebases that feel like a nightmare to maintain often don't distinguish the moving parts that need to be changed away from the parts that don't, since everything is deemed to be unreliable and prone to change.

I actually recommend going all the way of separating the organization of your codebase into "stable" and "unstable" parts, with the stable parts being a huge PITA to rebuild and change (which is a good thing, since they shouldn't need to be changed and rebuilt if they truly belong in the "stable" section).

It's not the size of a codebase that makes maintainability difficult. It's the size of the codebase that needs to be maintained. I depend on millions of lines of code whenever I, say, use the operating system's API. But that doesn't contribute to the maintenance costs of my product, since I don't have to maintain the source code of the operating system. I just use the code and it works. Code that I merely use and never have to maintain incurs no maintenance costs on my end.