Personal tools
You are here: Home → Reinout van Rees' weblog → Buildout development/production strategy

Buildout development/production strategy

by Reinout van Rees — last modified Dec 27, 2007 06:21 PM
Filed Under:

Buildout is great for development and for deployment. How to combine the two? Jean-Paul Ladage and me did some brainstorming on this and we'll show you what we came up with. I'm trying it now on a project and I'm pretty happy with it. Some prerequisites and assumptions:

  • Configuration is handled by buildout.
  • It shouldn't get in the way of easy development.
  • Deployment should be rigorous.

Production/preview are tags

Production and preview must be tagged buildouts. If preview is signed off on, it becomes the new production. In that case an "svn switch" to the relevant tag is done in the production buildout.

Variable bits

Production, preview and development differ in the port settings, zeo settings, debug mode, etcetera.

Production and preview do not differ in products, as a tagged buildout should progress from preview to production without changing. Production/preview and development do differ in products, naturally.

The following five buildout config files are needed to handle this:

  • production.cfg with production port number, zeo setup, etc.
  • preview.cfg with preview port number, zeo setup, etc.
  • stable.cfg with the products and eggs and zcml used by both preview.cfg and production.cfg.
  • unstable.cfg with the development products
  • devel.cfg with both development port number (just 8080 normally), debug mode.

unstable.cfg and devel.cfg could be combined, but experience has shown that it is harder that way to spot what you have to move from development to production. The extra hassle just isn't worth the worry about the extra file.

This way, with some care, a single buildout directory can be used for all three cases. It makes it easier to coordinate the move from development to production.

Externals

Externals in the form of downloadable tarballs and eggs are handled like a charm by buildout. No problems there. Problem cases are:

  • Products you develop yourself (svn trunk, mostly).
  • Products without a proper release from an external svn.

Handling svn externals in a buildout that is used for both production, preview and development is a pain. Keeping the svn checkout information inside buildout (with the infrae.subversion recipe) is the solution here. You'll need two infrae.subversion sections in case you have both products and eggs under development.

Note that we're talking "products" here. Keep the lib/python stuff as externals in src/* as they don't interfere if you don't enable them. I couldn't get those to work with infrae.subversion (chicken/egg problem as you can't reference those eggs in [buildout] if you still have to download them from the same section).

Tags: ,

Document Actions

svn switch question

Posted by http://duffyd.myopenid.com/ at Dec 28, 2007 12:38 AM
Hi Reinout,
Just a quick question re. your svn switch comment (call me dumb :) - so do you tag the same svn location that contains the buildout files or do you have the preview/production buildout .cfgs in different svn locations?
Thanks,
Tim Knapp

same location

Posted by Reinout van Rees at Jan 08, 2008 10:59 AM
It is the same location. If I make a release, I tag the buildout directory (with the changed stable.cfg) as release_2008-01-07 or so. I then do the "svn switch" in the preview checkout on the server. If everything's rosy I'll do exactly the same switch on the production checkout. So there's no difference between the preview and production checkout on the server, apart from symlinking buildout.cfg to production.cfg instead of preview.cfg.

Hope it is clear :-)

Config files

Posted by http://claytron.myopenid.com/ at Dec 28, 2007 05:53 PM
We've been working on almost the exact same thing here at Six Feet Up. What directory do you put the buildout configs in? We use ${buildout:directroy}/profiles as that seemed to be the way people were doing it around the internets.

We have the following configs:

base.cfg -- the configuration common to all the configs. this also contains a plone.recipe.runscript (currently trunk as I had to add in zeoserver support) part that can create a plone site based on some arguments.

local.cfg -- the local development config with the debug products added in.

debug.cfg -- all the debug products, ipzope and zopepy combined into one part that can be added in the local.cfg

dev.cfg/maint.cfg/production.cfg -- the three stages of our release process. these configs mainly contain overrides to the port, ip-address and locations of Zope.

There was one trick that is needed to use the base.cfg the way I did. If you extend the base.cfg and you want to modify it's parts then normally you have to write them all over again. What I ended up doing (on philikon's suggestion) was to create a variable in the buildout part in the base.cfg named base-parts. Then define the parts of the base.cfg as: parts = ${buildout:base-parts}. This way you can add in the debugging part in the local config like this: parts = ${buildout:base-parts} ${debugging/parts}. You would also have to use the same trick for the products definition in the zope instance part.

I've turned this all into a paster template that we use internally (as it still has some SFU specific pieces). The last thing I would like to do is use bundleman to bump versions and tag everything.

I'm glad to see that we are heading down the same path. Maybe we should put some examples in a public repo for others to be inspired by and possibly add to the ZopeSkel project a complete buildout solution.

claytron

location of config files

Posted by Reinout van Rees at Jan 08, 2008 11:07 AM
We're putting the *.cfg files just in the main buildout directory, exactly where the buildout.cfg would be, too. Didn't see much need to put them in a different directory. It are just 5 files anyway.

What we ought to do is to use that debug.cfg for some common development items. that makes it way easier to copy it between various projects if there's a new handy debug tool making the rounds.

What I'd like: a way to make local modifications. Instancemanager (the tool we used before buildout) searched for an optional PROJECTNAME-local.py. But there doesn't seem to be a way to look for an optional config file in buildout yet.

OT comments on your blog in general

Posted by http://rogererens.myopenid.com/ at Dec 29, 2007 02:00 AM
Hello Reinout,

I enjoy reading your blogs very much, thanks for your effort. However, there are two suggestions I'd like to offer:

1) Would it be possible to add a date to your blog entries? This way, it would be possible for a reader to evaluate how up-to-date a blog entry is that turns up in a search result.

2) Maybe it's nicer for readers to see your Weblog Archive in the left column sorted from newest to oldest. That way we don't have to scroll so much to get an idea about your a-little-bit-less-than-very-recent-entries.

Happy New Year,
Roger

good points

Posted by Reinout van Rees at Jan 08, 2008 11:10 AM
Hi Roger,

Good points, I've put them in my "fix up vanrees.org now that I've updated it to plone 3.0 and the new quills" project.

The date ought to be added to the template. Might be a local customization that I did where I forgot it.

The archive is waaaaaay too long as it is. I'm thinking about turning it into a google graph, btw. But the way you're asking: it seems you do like a date-based navigation (at least for the last months) in general?
Weblog Authors

Reinout van Rees

Location: Nieuwegein, the Netherlands
Reinout van Rees