by Jerad Bitner on June 4, 2010 // Short URL

Site Development Workflow: Keep it in Code

The tools we use and the reasons we use them.

Almost a year ago now Development Seed had an article on some of the tools they were developing in order to address a very real, very important problem—that of the whole development to staging to live development process. Up until these tools were available, this process consisted of an archaic—and quite frankly, pain in the butt—method of either duplicating the clicks and changes you made through the Drupal UI in your various environments, or putting all of your changes that needed to be made to the database into update hooks that did very specialized queries, set variables, installed or uninstalled modules etc, etc. This came with a heavy price tag of testing your migration path over and over, resetting your database, and testing again. Oops! Small mistake there... change the code, reset the database, run it again, wash, rinse and repeat. These tools attempt to change all that and now that they're maturing, they've become a godsend for site builders and developers everywhere. Here is the workflow and process we are using at Lullabot and some of the finer points we've picked up along the way.

Version Control

Keeping your code in a revision control system is imperative to the seasoned developer and more than a good idea for those not as experienced. It's more important when you are working on a project with multiple developers so that code conflicts can be resolved instead of accidentally overwriting what your buddy is working on, but also becomes part of a developers toolbox in many other ways.

We're moving to using Git for all of our projects at Lullabot due to the power and flexibility it gives us. Personally I love it so much that I can't stand to use Subversion anymore. When I am involved in a project where everyone is using Subversion I use git-svn so that I can manipulate the Subversion repository on my local computer with Git. The ability to track Subversion branches with Git is my latest candy and super useful for those situations where you need to do frequent merges between branches. Git succeeds in merging where Subversion really falls behind. I don't know if you've ever tried to do this in Subversion, but even the RedBean book introduces it as a 'headache'. And it is. So I use Git for this.

Git is also great for it's quick and easy branching. It's actually become a habit for me to create a branch for each and every issue that I work on for a project. It's so easy to create a branch, and merge it back into HEAD, or the main trunk, that there just isn't a reason to NOT do it. It's quick, it's easy and it keeps your changes separated in a way that if you're not done with the feature branch you're working on and need to fix something in say, the main branch, you can simply switch back to the main branch, fix it, commit it and then go back to what you were doing in your feature branch. With Subversion this is a total PITA because it's such a heavy process. It copies every single file, creating a new physical directory structure and then just try to figure out those merge instructions. With Git it's as simple as

$ git branch [branch-name]
$ git checkout [branch-name]
[make changes]
$ git checkout master
$ git merge [branch-name]

Now that I've beaten that horse to death, let's move on to the actual modules that are making our lives easier these days.

Export it! Export it all!

Getting everything into code has gotten much easier now that Chaos Tools has come and cleared up some of the actual chaos around exporting objects into code. It provides a way to simply add a few keys to your data schema, implement a few hooks, and voilà! Your data object can be read from code, overwritten, re-exported or reverted back to code in an easy to read structure. If you're having trouble visualizing this, think of the way exporting views works and apply that to any arbitrary object that you might be able to think of. Yeah, cool. There are a few prerequisites (such as machine names instead of auto-incremental integer columns) that you need to make sure your tables have, but other then that it's actually pretty straightforward. There's even a nice tag on a lot of modules for Features integration that will give you a list of most of the modules that implement this on d.o - take a look, see how they work, and get your data exportable for goodness sake!

Settings

Have you ever made changes to your local site, or your development site that you then need to make to your live site? Maybe you had to change your default theme, or perhaps the default comment settings for a node type. Instead of clicking these setting on your local and having to go through the UI again and click these settings, checkout Strongarm. Strongarm makes these settings exportable into a custom module.

Layout

Context and Panels have become our layout tools of choice. We tend to choose one or the other based on the project and what the client needs. Personally I can lay out a website much faster with Panels, but there are quite a few occasions where Context is simply a better fit for the situation. Usually this is the case when a client does not need the ability to change the layout around themselves, and it's more of a cleanup job than a brand new site. If I want to rapidly prototype complex layouts I use Panels. If I just need to replicate some basic rules of block visibility, I use Context.

Boxes

Boxes has become a favorite of mine recently. It does two things really well that are a huge improvement over the traditional core Blocks. It allows for easy inline editing of content, and they're exportable. There is a module called fe_block within the Features Extra project that allows you to export blocks, but the problem with this is that they are not very good exports. In this case, it's a core decision to use an auto-incremental key on the boxes table (yes, blocks uses a boxes table, it's confusing, but bear with me) and this has the side effect that if you 'revert' a block and then read it from code, it is reinserted and the id changes automatically which means that anything that referenced that original block (like Context, or Skinr for example) no longer has the correct id to find what you're intending it to find. Boxes gets around this problem by using a machine name for it's unique identifier which plays much better with the exportable mindset. However, one thing that is pretty nice in fe_block is that you can export a block's settings. The actual block itself doesn't work so well, but having those setting is really nice. For instance, if you have a block that is provided by a view and you want to override the title attribute of that block, you can use fe_block to export those settings into code.

Bring it all together

Then we get to Features itself. This is what brings it all together and why it's so important to have all of those other module exportables. Features gives you a nice UI to pick and choose what all you want exported, and then it creates the module for you. That's right, it writes a module for you. Features also gives you an easy way to detect if any of those exported things have changed, and update them if they have. Features' Drush integration is awesome for allowing you to quickly determine what features are overridden ($ drush fr), revert them all ($ drush fra), or update them all ($ drush fu-all). Special thanks to James- aka: q0rban for allowing us to type $ drush fu [feature-name], for those days when things just aren't going too well and you need an expletive to help get you through the day. ;)

And finally...

So now that I've shown you some of the tools, explained the importance of version control, and harped on getting things into code, how does any of this solve the problem of getting all of your changes from development, to staging, to the live site? Well, here's an example workflow from a project I am currently working on. Using the aforementioned tools and the following workflow I was able to make extensive changes to an existing site on my local computer within a new Git branch, export all of my changes to code wrapped within a few features, turn these features on in the staging environment, and have an upgraded copy of the live site without writing a single upgrade path.

Tracking Subversion Branches with Git-svn

The current site I'm working with is in Subversion. It's Phase 2 of the project and this calls for a new branch of the code so that we can provide bug fixes to the current live site which is running on the trunk in our repository. A new Subversion branch was created for the architectural changes we're making to the site. Since we'll be making bug fixes to the trunk, we're going to need to keep merging those fixes into the new dev branch as well. And since merging is such a pain in Subversion, I'm going to use git-svn to work with this code repository. I want to be sure that the new Subversion branch is tracked in Git as a branch, so when I first checkout, or clone the repository I'm going to use the -T and -b operators.


- branches
  -- 2.x-dev
  -- original
- trunk


$ git svn clone [svn-repo-location] -T trunk -b branches


$ git branch -r

  2.x-dev
  original
  trunk

Working with Our Subversion Branch in Git

Checkout the dev branch of the project which is now using Git locally.
$ git checkout 2.x-dev

This new branch is connected to the Subversion branch and any changes that are committed to it locally with Git can be pushed into this new branch within the Subversion repository.
jbitner@sirkitbox_~_Sites_grammy365_public_html 2014 bash 2014 214?6-1
Notice in the screenshot above that it says: "Committing to https://grammys.unfuddle.com/svn/grammys_grammy365/branches/2.x-dev". You can see that I'm clearly not in a 'branches' directory, yet it certainly committed to the 2.x-dev branch.
You may also note that the branch indicator in my screenshot says "(context-2.x)". This is actually a branch of the 2.x-dev branch. So it's worth noting that a $ git svn dcommit here is directly connected to the subversion branch "2.x-dev" that I branched off of, and will commit changes to that branch as well. (Whew)

Fixing up the site

The current site I'm working on has a lot of custom blocks with PHP in them doing some things they probably shouldn't. The theme also needs to be redone, and the views and custom blocks need to be exported so that they're not being read from the database constantly. My buddy Jay Wolf is helping me out with the theme, so he is making changes to the development branch in Subversion while I'm using Git on my local. As soon as he had the base theme with all the regions I needed, I got busy with the Context. Once I had the blocks cleaned up and converted to boxes, and started using the Views correctly, I setup separate contexts for the current sections of the site and placed the boxes where their predecessors went. The boxes are using Skinr, and there are a few Quicktabs thrown in. We also want to make sure that the default theme is switched over to the new theme when this goes live.

Merging fixes from trunk

Oh! But then here comes Dave with a bug fix to trunk! The bug fix goes in and we need to merge that change back into our new dev branch.
jbitner@sirkitbox_~_Sites_grammy365_public_html 2014 bash 2014 213?5
A quick $ git merge master and we have the changes we just made to the master branch merged into our development branch and can continue on our way.

Creating Features

Ok, back to our dev branch. Now that we have everything laid out with context, we're going to create features that correspond to the different contexts, as well as a site feature that will hold some global settings. Let's create our site feature first. This will hold the default theme settings as well as a sitewide context.
G365 Sitewide Settings | GRAMMY 365 Members
Let's put this in /sites/all/modules/custom/features
g365_sitewide_settings
Now that we have our new 'feature' module, we enable it and then take a look at our Features list. You'll see that it is marked 'Default' which means it is reading the feature from code.
Features | GRAMMY 365 Members

And now that we have some basic site settings changed, we create our front page feature. Since the front page of this site holds a lot of the views and boxes referenced through our context, as soon as we tell our new front page feature to include the front page context, it also automatically detects some of the other elements that are referenced through this context. This feature is also wrapped up for us and we put it in the same place and enable it.

Moving the changes to the staging server

Our staging server is running the new Subversion branch already, so we add these new feature modules to the dev branch:

$ git add custom/features
$ git commit -am "adding new features"
$ git svn dcommit

These modules are then on the dev server when we $ svn up. Now we can simply enable our feature modules, and like magic, all of our changes are now working on the staging site!

Oh, and don't forget to clear the caches ;)
$ drush cc all

Jerad Bitner

Sr. Technical Project Manager

Want Jerad Bitner to speak at your event? Contact us with the details and we’ll be in touch soon.

Comments

Gambito

Great Articule

What a great article, thanks for the info, and specially thanks for name some modules you work with, it is so helpful for the new ones like me.

Reply

Jeremy Caldwell

Very informative

Thanks for taking the time to write this up Jerad, excellent read. Nice to see how things are managed from the dev to live site too and I'll definitely reference this post in the future.

Looking forward to playing with Context and Boxes now!

Reply

WizOne Solutions

Some great ideas for settings; what about special content types?

Hi there,

Great article. I've definitely got to try this stuff out.

One question though I often face is moving content that is essential for system operation. Let me give you an example. On a project I work on, we use a special type of node to encapsulate things like a course, a course term, a milestone log, etc. Sometimes I want to move one of these (say, a course round) from dev to staging to live. Is the only solution to re-create the content? I don't think it'd be too practical to give it a machine name (although perhaps it would be on a limited amount of content)...if I did give it a machine name, could it become exportable? That' d be pretty cool...

I guess that when it's only content though and not loads and loads of settings, it's still much better than the alternatives, since it's the functionality that matters the most.

Still, would love to hear your thoughts on this if you see my comment!

Kevin

Reply

sirkitree

Content types

Content types are exportable! That is, the structure of the content type is, like what fields it has attached to it. So if you have a custom node type that you created and through CCK have added certain types of fields to it, you can certainly encapsulate that in a feature. Which is really great because if you make changes to that same content type, say you add a new image field, you can update your export and push those changes live.

Beware that if you remove any fields on your custom content type, and then update your live site, you could stand to lose some data if you have any content already in place.

Reply

Kelly Bell

Drupal 7 integration/upgrade question

Thank you so much for this terrific article!

This is all so nice to have in one location and is great information, but I have one question: what does the upgrade path to Drupal 7 look like for these practices? Obviously for Git it's irrelevant, but for all the other techniques you discuss, I'm curious as to how this will impact - positively and negatively - my upgrade process.

Again, heartfelt thanks for taking the time to write this for us!

Reply

sirkitree

The practice and the tools.

The 'practice' is to make sure your changes are in code so that they can be moved between environments. The practice of doing this is important.

The practice is not dependent upon the tools, as we have been using this practice of getting things into code way before these tools came about.

How you go about this practice depends on your tools. Will these modules be ported? Probably at some point. Will new ones come along to change our toolbox? Probably at some point. These tools just make that practice much easier to execute.

Most of these tools are dependent upon ctools exportables. So if that base API stays the same between major versions, it should make any sort of upgrade process fairly clean. But API's do tend to change, mostly for the better, between major versions, so your guess is as good as mine at this point.

Reply

Ryan

Hey Jerad, great article.

Hey Jerad, great article. Only thing I get hung up on with one client is the content itself. How do you propose to migrate nodes from a site where they're being staged (e.g. informational pages, new classes / groups that are being prepared and pushed live, etc.)? Is there any hope? : P

Reply

sirkitree

What budda said...

I've looked into the Deploy module for this kind of stuff and it really does work well for moving content as long as the content's structure is the same. I.e. the content type has the same fields. It also lacks the ability to migrate content within certain fields, but is actively being worked on and maintained, so if you find a bug don't be afraid to work it out and you'll get some help.

We haven't had to worry about this too much as it's generally understood by our clients that if new content is created it can be done on the live site. But we do have a project coming up where we are certainly going to have to come up with a good solution for staging actual content and pushing it live. I'm sure we'll have an article on how we solve this problem when we do :)

Reply

dagmar

Input formats

Inputs formats are not exportable with features... Numeric ids, settings saved into variables...

A lot of modules uses Input Formats to work, I think the most common example is Wysiwyg. There is an issue to try to export wysiwyg settings using ctools + features.

But, the numeric ids are still a problem. If you import a wysiwyg option with the id 4, it may not work in other site that don't have an input format with this id.

For this reason I have recently uploaded into d.o a new module called Input Formats, it is an API that provides machine names for input formats and integrates with features.

If you can test it and give some feedback, it would be great.

Excellent article btw.

Cheers
Mariano

Reply

Mike

Nice module

Nice work dagmar!

Just saw this new module pop up on my module release feed this morning.

Reply

Greg

Enough!

I really have to get on and move to git. I've been putting it off for about a year now. Jerad, you've convinced me. Next week, I swear it!

Reply

Chip Hayner

What about Capistrano?

We've been able to use Capistrano to deploy our custom projects (written in Symfony) to our staging and production websites (one of our devs recently wrote a great post about our process: http://bit.ly/b7gbjN). Have you had any luck using Capistrano to deploy Drupal (to prevent having an SVN checkout on the live server)?

Thanks for the awesome write-up!

Reply

sirkitree

*shrug*

We haven't tested or used Capistrano, but I've personally been interested in it. I read your article but am still having a bit of trouble relating it to a Drupal install. Would love to see an actual example workflow using Capistrano to deploy a Drupal site!

Reply

Frank Carey

Hey Jarred, Wanted to

Hey Jarred,

Wanted to clarify that a 'svn copy' (Tags and branches in SVN) are stored as diff as well, and do not increase the size of the repository (as stated in podcast 85). I've been looking to do git as well, though. Have you been able to find something useful as far as migrating existing svn repositories to git (a way to preserve the previous commits and users)

Reply

Dave Lane

The Drupal Community is superb

Jerad,

This post is exactly what we've been looking for - we're struggling with some complex/multi-faceted Drupal sites on which we're doing lots of ongoing development... we've attempted to fit Drupal into the Scrum process, and it's been a rough ride as you can imagine.

In any case, we're already doing the mix of svn and git, and are interested to see your approach. We've also run into issues with missing export mechanisms - your references identify most of what we need... still looking for a tidy way to export menus that's able to handle the possibility of varying node ids, etc...

Thanks very much in any case - you guys are doing great work, and inspiring us out here in New Zealand.

Dave

Reply

Rafael Vega

Awesome!

Thanks so much, Jerad, I've been playing with this stuff all day and I'm feeling like a kid with new toys!

I have another question. Do you have a recommended "plain" or "base" theme that is adequate for starting theme development using Panes? (As oposed to starting a theme from scratch or starting from Kubrik).

I'm using Kubrik right now but I feel like tere's too much stuff there, something more minimal would be great.

Reply

Rafael Vega

Edit

I didn't mean to say "starting theme development" as in I'm learning how to develop themes.
I meant "start development of a theme" as in I'm starting to develop a new theme.

Reply

justinbryan

I hope you never stop! This

I hope you never stop! This is one of the best blog’s I’ve ever read. You’ve got some mad skill here, man. I just hope that you don’t lose your style because you’re definitely one of the coolest blogger’s out there. Please keep it up because the internet needs someone like you spreading the word.

Reply