by Angie Byron on February 19, 2007 // Short URL

Best practices in open source development

Introduction

Open source software has countless advantages over proprietary software. While there are disadvantages as well, such as the lack of built-in support contracts and no one to blame when things go wrong (which means no one to sue), in most cases moving to open source software is a smart move. You never need to just accept whatever functionality comes with it; you can modify the software to do whatever your heart desires, and you're never "locked in" with a particular vendor when you need additional functionality. There is a community of people developing, using, and testing the software, which tends to lead to higher quality and faster growth. Plus, in terms of initial costs, it's generally cheaper than proprietary software (often times, free).

However, gaining the full advantages of open source requires a fundamental shift in development practice. And people who are not "in the know" on this point can often get into trouble when they continue to work in a traditional way, and a negative attitude about open source software (it's not me, it's them!) can result.

This article provides some best-practice tips and advice that we at Lullabot employ while working on on development projects. While this article talks primarily around our experiences with Drupal, its lessons should still apply for anyone working with open source software.

Best Practice #1: Learn how to Extend The Software

Drupal has a little motto that goes something like:

If you have to "hack core" (change core files) to get Drupal to do what you want, you're probably doing something wrong.

This advice holds true for most other open source projects; always perform investigation into the tools and techniques the software makes available for customization. These may include a "skinning" or "themeing" system to customize the look and feel, the ability to extend functionality through modules, extensions, and add-ons, or configuration options that can be altered to fit your needs.

Your first goal when working with open source software should be to try and "get in the heads" of the developers of the software, and determine how they intend for the project to be extended. While it can appear easier on the surface to just bolt on a chunk of functionality you're missing, doing so means inadvertently undoing all the collective work and thinking the community around your project has poured into the problems you're facing.

Learn what tools and techniques are available to you, and you'll have a much easier time building your project.

Best Practice #2: Do. Not. Fork.

The second you change any of the default core files of an open source project in any way, even for something as small as changing the text here or there, you have created a "fork." This hurts you in a number of different ways:

  • Difficulty in getting support. Your changes may cause subtle bugs to appear, and you're going to have a very hard time finding other people to help you track them down, if they can't reproduce the problems themselves.
  • Difficulty in upgrading. Each change you make, no matter how minor, needs to be carried over each time you upgrade the site's code base. If you forget a change, your site behaves differently. If a security patch changes a line where you've made a customization, you then have to hunker down and do a bunch of analysis and testing as you attempt to merge the changes and ensure that you're still getting the proper fix.
  • Maintenance headaches. Because this is your code that you've written, you're on the hook for maintaining it, testing it, and improving it. More on this and why it's a bad thing in a bit.

Resist the urge to fork, and instead...

Best Practice #3: Participate within the community

There are many advantages to using open source software, but probably the biggest advantage is the virtually limitless resources involved in the open source ecosystem. Millions of people around the world are testing the software, fixing problems, and adding new features. It's simply impossible to reproduce this kind of momentum with a small team (or even a very large team).

Yet, many people only see open source software as a cheap (or free) alternative to building things from scratch, and completely ignore the community aspect of the software. This is detrimental, both to you and your clients, and to the software project itself.

Let's imagine a scenario where you find a bug in your open source project of choice. Your natural inclination might be to just troubleshoot and fix the bug in your local copy, and then move on with your day. After all, we're all busy people, and client deadlines aren't getting any shorter.

An alternate, and more effective, approach is to do the following:

  1. Search for the bug. When you very first encounter a bug, before you even attempt to fix it, your first instinct should always be to search for the bug in the issue queue or bug tracking software used by the project. It's possible someone has already found the bug, and reported a fix for it. If so, you just saved yourself some work.
  2. Report the bug. If the bug isn't there, the second thing you should do, again before you even attempt to fix it, is submit a detailed bug report. Often, developers are very active on issue queues, and they are also more familiar with their code base, so they can often find and get to the root of a bug before you can.
  3. Submit your fix. Assuming someone in the community didn't get a fix out to you before you figured it out, always submit your fix back. Not only is this good "karma," as you've just saved the next person from having to do this (and karma in an open source community is your very best asset), but often you can get good feedback from other developers as to whether or not your fix is the best solution, and either way, you help ensure the fix gets put into the "official" product, which means you no longer have to maintain it.

Let's use another example. A client needs some crazy new feature for a site you're building.

  1. Search for an existing module. Even if there's not a "100% fit," a 90% fit (or even 50% fit) is a better starting place than nothing.
  2. Submit a feature request. If there's a module that will work as a starting point, just as with bug reports, before you start coding anything submit your intent to work on the additional functionality. You may get other people interested who will offer co-development, testing, or even just some cold, hard cash toward its development. Or, the module author might chime in with how they already tried it, how it didn't work, and how to use X instead.
  3. Contribute new code (a new module for example) to the community. If there's not already something out there that fits your needs and you need to build it, submit it to the community in whatever they're using as a "forge" system (CVS, Subversion, etc.) If possible, the best thing to do is place your code there as early during the development as possible. This allows others to try out your changes and provide testing, bug reports, and usability feedback long before it reaches your client's site. Procrastination can create more custom code for you to maintain: lots of people have really good intentions of committing stuff back when all's said and done, but often it's easy to get side-tracked.
  4. Work out of the issue queue. When you add a new feature to your module, make a feature request for it, attach a patch, and commit it. When you fix a bug, make a bug report for it, attach a patch, and commit it. The discipline you show here will help pay off ten-fold by getting more eyes on the code, and by making changes small and easy to isolate should something go wrong.

By pushing fixes and features back "upstream," you help ensure that future sites you build will have the functionality you're looking for, rather than having to constantly re-invent the wheel. You also turn into a collaborator on the project, rather than a user.

Advantages of following best practices

Working in this manner has the following advantages:

  • It helps save you time (and saving time saves you money). You might find answers before you even begin your quest. Another developer can come along and code something before you even have the chance to do so.
  • It earns you karma. In addition to the "warm fuzzy feeling" that contributing gives you which is a reward in its own right, open source communities tend to be "meritocracies." Opinions of individuals are generally weighted by the community, either consciously or subconsciously, by the level of contribution they've given to a project. Gaining a reputation as a contributor often means your questions get answered sooner, and it also means more business for you as new clients see you being committed to the platform, and other contributors in the community refer additional work to you.
  • It increases the quality of your code. More eyes mean more testing, more feedback on the approaches you've taken to problem solving, and in the end more flexible, usable, and easy-to-understand code and functionality.
  • It lets the community help maintain your stuff. Putting stuff out in the community means you increase your chances of someone other than just you using your code. And that's a very good thing! These other people will contribute their own bug fixes and features to your module, and help test it for bugs. They can also help with upgrades between major versions. With custom code, you are the only person who is able to do this.

Disadvantages of following best practices

There are not really any, other than it takes a bit more time to do up front. You can help offset that by building that time into your contracts, and by client education. Explain to your clients about how open source is different, and that in order to get the very best "bang for their buck," it's important to leverage the community development model. This will help ensure that they enjoy all of the benefits of open source, including not being "locked in" to a proprietary fork, having the highest-quality code, and being "future proof" for upgrades.

"Real world" implications

Lullabot has built a number of sites using these best practices, which has resulted in thousands of contributions back to the Drupal community in the form of themes, modules, and core patches. That's a win for us, a win for our clients, and a win for Drupal. You can't get much better than that. :)

Angie Byron

Powered by Drupal!

Comments

Chris Johnson

Nicely done

Well said, Angie -- an easy-to-read, not-too-technical summary of what is good about participating in the open source process the Way It Was Meant To Be. ;-)

Reply

Roger Lopez

This overall philosophy is

This overall philosophy is what sets drupal apart from other communities. There are dozens of open source CMS's but the community surrounding Drupal is truly the difference.

Keep up the excellent work.

Reply

mfb

Thanks, I am giving this to

Thanks, I am giving this to a soon-to-be drupal developer today. Note, you may want to setup a separate stylesheet for printing the articles -- if you look at the print preview in firefox, you see that it tries to print the whole webpage on one page of paper, for some reason.

Reply

Bèr kessels

cheaper?

Maybe initial sales is cheaper. But FLOSS is no cheaper then free-/shareware. Nor is it any cheaper to deploy, its not cheaper to maintain. I is not cheaper to develop with, and probably not cheaper to get consultants and developers for.

FLOSS is not cheaper then closed source. In the end, the other benefits will probably bring you a lot of revenue, when one chooses open source, true. But that is the case for any good strategic choice.

To add to this Open Source says absolutely nothing about the price one needs to pay for it, it only says the code is open. While in practice this means most often the price is €0 too, nothing holds you from selling your open sourced application for €1 000 000 per piece. Open Source is not equal to GPL, keep that in mind too :)

Let's stop acting like cheapass Dutchies preferring Gratis over Quality. Lets start with not mentioning the price of Open source, and focus on its real benefits.

Bèr -there ain't no such thing as free- Kessels

Reply

angie

I've clarified that one sentence...

...so that it mentions initial cost, rather than a blanket cost.

Though I hope you read the rest of the article too, which is intended to specifically addresses this generally held misconception you point out of open source being "free." The goal is to provide some strategies so that developers working with FLOSS are working "with" the community rather than against (or beside) it, and has very little to do with whether or not open source software costs more than proprietary software. :)

Reply

Bèr kessels

sure i read the rest

And I think its a great article. Which is the reason why I wanted to stress that 'cheaper' part: to add my little value to an already great piece of writing.

Reply

dgtlmoon

On the subject of sueing

On the subject of sueing software vendors for what ever reason if there is a failure in the application, most licences you are granted with proprietry software actually leave you with less of a leg to stand on than opensource software, ever tried to sue microsoft for faulty software? atleast you can pay a developer (if you can find a good one) rather than lawyers (if you can find a good one and can be bothered) with GPL.......

Definately people who publish succesful opensource projects need to take some kind of responsibility for the project, either this is that the one or two developers are going to make sure their email always works or something like drupal that have a whole infrastructure for supporting their users and extending/integrating their needs

my two cents

Reply

Matthew Davidson

No Support / No Throat to Choke FUD

I agree; people in the free software / open source communities should stop repeating this FUD. I've been as guilty as anyone in the past. To pick on the paramount proprietary software vendor, from the Vista EULA (http://download.microsoft.com/documents/useterms/Windows%20Vista_Home%20Premium_English_d16c019b-fa71-4fc9-a51d-a0621bddb153.pdf):

"You can recover from Microsoft and its suppliers only direct damages up to the amount you paid for the software. You cannot recover any other damages, including consequential, lost profits, special, indirect or incidental damages."

That is, if the software destroys your business, you can get your couple of hundred bucks back.

And as for support, the license directs you to http://www.support.microsoft.com/common/international.aspx, which gives you some online documentation. If you want to talk to a human being, you can find out "How to purchase Microsoft support". They also direct you to "the Vista community", ie. online forums, and suggest that you might be entitled to OEM support.

Use free software (eg. Drupal), and you get online handbooks, forums, mailing lists, and you can optionally pay someone for support. Use proprietary software and you get the same, minus the benefits of freedom.

Reply

TheWhippinpost

Although I think there are

Although I think there are more disadvantages to open source than you allude to, good article.

My indoctrination into Drupal was via Civicspace (CS). I liked CS, though it was slower than I was comfortable with. Once I eventually found the time to get under the bonnet and make customisations, I gutted their supplied template.tpl.php file - BOOM! Much more responsive.

I subscribe to "The Drupal Way" as much as poss, and philosphically agree that it makes sense not to meddle with core files... but I still have felt (in my mind) that I needed to - Whether through ignorance, lack of time, or simply not knowing an alternative way of implementing the hack.

In my minds-eye, I have this battle that goes: If the system processes the core files, and then has some of the output over-ridden by tweaks made in template.tpl.php, that's double-the-work... hence why I came to believe, the CS template file slowed the system down.

I think the template.tpl.php way of doing things is also made slightly more complicated than is perhaps necessary by the concepts and practices of PHPTemplate callbacks, for newbies and/or non-PHP users.

And again, in my minds-eye, I see selected pieces of data being shunted to this template file, processed, and sent-on to eventually end up in node.tpl.php, page, or wherever... for this reason, mainly, and my experience with CS, I've become more comfortable making customisations directly in node.tpl, page.tpl etc... cut-out the middleman, so to speak.

I realise I may well be missing something here, or my understanding is flawed, but that perhaps should be a lesson (in the feedback/usability sense), as well as a pointer to people like myself.

Reply

Steven Wittens

Theming and other overrides

TheWhippinpost: when you supply an override in template.php, Drupal no longer calls the original theme function. You simply replace one piece of code with another. The loss in efficiency is negligable, and mostly comes down to you now having a piece of code in memory that is not used. This is usually not a problem for themable functions, but can be overcome by using a PHP cache like APC. In that case, there is no slowdown at all.

This does not hold in cases where you actively undo work by core (such as form alter), however, again, the efficiency hit is not as bad because if you e.g. remove a form element, all the work needed in processing, validating and displaying it is no longer performed.

Drupal's core follows a very 'lazy' code style, where we try to avoid doing work until the very last second. This means that module's changes often do not result in duplicate work being done.

Reply

Benjamin Melancon

The Business Model Difference

Great article. This is way late in the commenting but I wanted to make a point that underscores some of the ones made above.

Open source free software isn't a business model in itself, but it encourages business models of directly helping people rather than developing or acquiring special access to a resource and charging for that -- the difference between working for a living and living off others work, which is what you ultimately are doing even if you are the founder of a large corporation (which most executives and stockholders are not). Okay, that was a bit broader a political economics perspective than is relevant to an organization considering open source. ;-)

Lullabot isn't going to develop the best module in the world and sell it over and over again. If they did, webchick would leave and start Rollaby and use the open source code to solve clients' problems, improving the code as needed (as others would also be doing).

Closed source companies too often succumb to (seemingly) cutting off development once they have a marketable product, and squeezing maximum profit out of new sales and forced updates (which aren't always upgrades) until the product and service are so bad that a half-decent competitor starts to take away their market. Let's not harp on Microsoft for a change: anyone used Winfax (bought by Symantec) or ACT contact manager? You'll know what I'm talking about.

In my biased opinion, open source free software superiority grows from the economic base of using collective resources to meet individual needs combined with the community base of sharing and working together to meet collective needs.

~ ben, Agaric Design Collective

Reply

Jacques Mattheij

best practice for newbies

Hello there,

I read the above and I find myself agreeing with all of it but I still feel that I am falling well short of implementing changes without 'forking'. The reasons for this are twofold, first my 'demands' are quite often too complicated to expect a reasonable turnaround time from whoever maintains the relevant modules, second I don't feel comfortable with my level of experience with drupal (only a couple of weeks since I looked at it the first time).

It can also be very stifling to worry about the thousands of consequences of a change to the code when you do not yet have the experience necessary to judge the impact of that change.

Maybe when I've learned more about how drupal works internally (I'm playing around with a butchered set of modules right now to achieve a certain effect that I could not get any other way so far) then I'll feel more confident about making changes to modules and suggesting fixes or features. For now a fork (without 'fitting' myself in to the community) allows me to learn quickly and interactively how things work.

In the end that may mean that I end up doing things twice, once for 'trial purposes' to find an acceptable solution (acceptable to me, that is) and then a second time to see if there is a way to shoehorn that functionality into an existing module.

I hope that's an acceptable deviation from 'best practice'.

One of the disadvantages of the deviations from best practice in avoiding forks in drupal seems to be that there are many modules that all duplicate a lot of effort (access controls, revisioning & moderation as well as images come to mind) which would probably be better and more consistently served by merging a bunch of functionality into one well laid out and very configurable module. That would also help in presenting a more consistent interface to the users and administrators of the system.

best regards,

Jacques Mattheij

Reply

angie

I can see how that would /seem/ easier...

Certainly making your changes privately and privately seeing if they break your site is less intimidating than putting it out for the world to see.

But you're missing the entire point of working with an open source project. :)

Consider this: You're working with $foo module, and come across a bug. You have two options:

a) Fix the bug for your local site, continue on with your day.
b) Do a), but at the same time, roll a patch and submit it to the module issue queue.

If you do a), any fall-out bugs are yours to discover. Each time you upgrade the module to a new version, you have to keep track of what you've changed and re-create those changes so you don't miss out on important security fixes and such.
If you do b), then several people (including the module maintainer themselves, who know their code better than anyone) may comment back and point out flaws in your approach, or suggest alternate ways to fix it. Better still, once the fix is "solid", it'll likely be incorporated "upstream" so that when you next have to upgrade the module, you don't need to worry about maintaining your sets of changes.

http://www.lullabot.com/articles/strategies_for_patch_management has a lot more details on this approach.

Given that...

It can also be very stifling to worry about the thousands of consequences of a change to the code when you do not yet have the experience necessary to judge the impact of that change.

...is definitely true, it's not logical to me why one's first approach to this would be working without the help of people with this requisite experience. You have to fix the bug anyway. Why not do so with the help of the people who know the module best?

Reply