Personal tools
You are here: Home Reinout van Rees' weblog Topics instancemanager

instancemanager

Easy backups with buildout

Filed Under:

I finally got around to copying the backup functionality of ye olde instancemanager to a buildout recipe. The result is a real handy recipe: collective.recipe.backup .

bin/repozo is a zope script to make backups of your Data.fs. Looking up the settings can be a chore. And you have to pick a directory where to put the backups. collective.recipe.backup provides sensible defaults for your common backup tasks. Making backups a piece of cake is important!

  • bin/backup makes a backup.
  • bin/restore restores the latest backup.
  • bin/snapshotbackup makes a full backup, separate from the regular backups. Handy for copying the current production database to your laptop or right before a big change in the site.

Just read the pypi page, especially the section on supported options, though you get perfectly decent backups by adding backup to your buildout's parts list and adding the following two lines:

  [backup]
  recipe = collective.recipe.backup

That's it!

Command line history

Filed Under:

Command line history turned up a few times in my feedreader this morning. Here's mine:

$ history|awk {a[$2]++ } END{for(i in a){print a[i] " " i}} |sort -rn|head
102 svn
85 cd
31 emacs
30 l
19 less
19 instancemanager
19 bin/supervisorctl
15 cp
14 timelog
13 bin/supervisord

"l" is of course some "ls" alias. "instancemanager" is there in favour of my now-regular "bin/buildout" as I was working in some older project on friday. "timelog" is my commandline version of gtimelog . And supervisord is an app that starts multiple programs at once and manages them (a zeo server, a zope instance via paste and varnish in my case, I worked on a new website this weekend).

Tags:

Open source cms, lock-in, licenses

Filed Under:

As customer, look at the license of the customizations to Plone that your provider makes for you.

Scott Paley talks about Choosing an appropriate web content management system . I've got two additional comments on two quotes:

Here's auote one: That's one of the nicest things about working with an open source CMS. You get to spend your money on customization and configuration, and not just on licensing.

Also spend your money on getting the extensions your provider makes for you released as more generic open source extensions. So not just for you, but for the generic public. Some of the items that I've been involved in that had at least partial customer funding: feedfeeder, instancemanager, azlinks, a security howto / httpslogin .

"I want to give something back" is pretty strong at some customers. Some only look at the price, but others allow you to book 4 hours for some extra documentation and packaging and putting it on plone.org. And moving it from Zest's internal svn repository to Plone's collective svn repository.

You also can get the advantage of feedback: some bugs are found, for instance.

The second quote: With the commercial systems, there is a significant amount of vendor lock-in you will experience, and typically you will not have the flexibility to customize commercial products the way you'd like.

See an earlier post by me about contracts and licenses .

Well, what's the lock-in to your provider? What's the license on the code? At Zest software we, in practice, place all the code under the GPL license. Not that we publish everything, but it gives both us and the customer all the freedom we want/need. They can take the code and go to one of our fellow plone companies in the Netherlands . They're free to do that, so they won't :-)

The customer can modify code themselves (some can/want that) without licensing/copyright hassles. Hey, they modified GPL code, so it stays GPL code. No problems. Two of our customers regularly hang out in our own IRC channel and at least three actively commit to our svn repository.

Another advantage: it is easy to share code snippets or whole examples with other plonistas.

Structuring plone projects 2: structuring instances

Filed Under:

I got some feedback on my previous entry, so it is time for some extra information and clarification.

I got (to date) two comments on how I structure plone instances, which prompts me to share some additional thoughts.

  • Keeping a load of plone sites in one zope instance? That's only doable if you've only got a stock plone. Perhaps one or two basic add-ons. As soon as you start to do custom development: separate those instances! It is just too much work to make sure your installer doesn't mess up other sites (or that your configure.zcml doesn't mess it up).
  • There's no need to keep all those instances in one homedir, though instancemanager makes it pretty easy to do that. If your company has its own server, why not create one homedir per project? At zest software we normally have one homedir per project. You can put everything in root-controlled /var/lib/* directories, but to me that's not terribly attractive or needed. If you use your server only for hosting zope instances, in my I there's nothing wrong with placing the instances in a homedir. Actually, it makes for handy instancemanager instance names: instancemanager update production, instancemanager preview --backup, etc.
  • Not everything needs to be in a home directory. Instancemanager puts backups into /home/username/backups/instancename by default, but you can override that in a side-wide config to be something like /var/lib/zopebackups/username/instancename.
  • To me, it is an absolute necessity to specify formally which add-on products and plone version make up an instance. I've had a few projects where I just zipped up the Products dir and used that as a bundle as I couldn't reliably figure out which add-on product versions I used (plone 2.0.5 from a .tgz or copied from an svn version? ploneformmailer 0.3 beta or 0.3 final with some local modifications?).
  • Something like instancemanager makes it real easy to test things out. Just copy your existing philips.py to philips3.py and swap plone 2.5 for plone 3.0 and start testing out that big customer project with plone 3.0. Easy as that. The easier you make that sort of stuff, the more you're going to do it. It was a snap for me to test various cachefu versions with both plone 2.5 and 3.0 because I just had to copy one file and change one line.
  • Setting up a well-working way of managing your production and development instances will take some time. I really hope the thoughts here (and in the previous article) help some people to get started, but you will have to find your own best way of working. And if you're in a bigger company, you'll have to make sure everyone can work with this. Most guys and galls at zest software use instancemanager as that helps getting everyone on track quickly, but the way we've structured things doesn't inhibit Daniel (for instance) from setting up his own instance by hand 'cause that's still easy to do with the bundles we've got set up. And a quick look at the instancemanager config gives you a readable list of add-on products and versions. No problems!

How I structure plone projects

Filed Under:

Some plone websites are pretty simple. Just a straight download of a plone version plus one custom product to customize things. Others consist of quite a number of add-on products and custom products. Here's how I structure them.

There is a rule number one: everything must be stored in svn. If you have pieces of configuration outside of svn, you're in for pain. I've had to work my way through Products/ directories trying to figure out which versions were in there. version.txt helps a bit, but once it is a version of which you cannot find a .tgz, you're done for. Or, if it is an svn checkout: is it safe to update it or does it need to be a specific revision?

Donjon (main tower structure)

Which pieces of configuration should you include? Well, which products are installed. Which plone version. The apache config file(s). Perhaps an awstats config. A cronjob for /etc/cron.d. Those config files are simple: just store them somewhere in your svn, I'll get to the location later. For the list of products and the plone version you've got basically three choices:

  • A "readme" file that lists your plone version and add-on products. You'd better keep those up to date!
  • Put the zipfiles in some svn directory of your choice. Your svn repository won't really like that, neither the backup of your svn repository.
  • Put the list of products (and download locations) in some script that downloads and installs them.

I use the last method, by using instancemanager. Basically a config file like this (URLs shortened to keep 'em on one line):

archive_sources = [
    {'source': 'qPloneGoogleSitemaps-0.3.0.tar.gz',
     'url': 'http://surf..../qPloneGoogleSitemaps-0.3.0.tar.gz',
     },
    {'source': 'RedirectionTool-1.1.tar.gz',
     'url': 'http://plon.....s/1.1/RedirectionTool-1.1.tar.gz',
     },

Instancemanager will reliably download those files and put them in your Products directory. Works for bundles, too. A good custom shell script or stuff like buildout can also help here. Nobody has to wonder which products and which versions to use: just look at (or use...) the config file.

For a small project with just one custom product, I normally add an instancemanager directory inside that custom product. Which, of course, is in your svn repository. In there, I put the instancemanager config with the specification of the project (other add-ons, plone version, zope version). Perhaps another directory for some documentation and an apache config file.

For a big project, you'll probably end up with a couple of specific svn checkouts of add-on products and a set of custom products. Svn bundles to the rescue! I normally make sure I only have to grab one svn checkout to set me up completely. Inside that checkout I've got the following directories:

myproject/externalproducts
An svn bundle of the external add-on products for which I need a specific checkout instead of a regular zip file. Perhaps a branch on which I'm doing fixes. Or a specific revision. The exact configuration is again explicitly in svn in this way!
myproject/products
An svn bundle of the add-ons I'm developing myself. Project-specific stuff. It somehow is handy to differentiate between "products" and "externalproducts" as you've got to take greater care with changes in products that other people are using than with changes in your own project-specific products. Put your DIYplonestyle theme in here, for instance.
integration
Here's where I put the configuration for bigger projects. I normally subdivide it further into subdirectories
integration/instancemanager
Instancemanager configuration files. Often a projectname.py for regular development and a specific production.py with the current stable production server's configuration.
integration/apache_etc
The apache configuration files.
integration/cron_d
Some cronjobs you want to put in to /etc/cron.d
integration/doc
Documentation for the whole project. Explanation of how to set up the server. Backup/restore documentation.

This way of setting up projects slowly developed during a few years of setting up projects like this. We also use it at zest software for the majority of our projects. One thing that's not fully crystallized yet is how to deal with production bundles and development bundles. The needs for this tend to differ wildly per project. Some only do an update once every two months, others try out a quick change in preview and put it into production right away as they need a more rapid cycle.

I hope this is of help to some people. It would be interesting to hear of other setups or recipes. If you don't have a blog yourself, feel free to email me.

ipython shell, workingenv and instancemanager

The ipython demonstration was quite impressive, but doing the activation and so by hand... That's what instancemanager is for.

Stefan showed off ipython on saturday. Pretty impressive. I've now added support for this in instancemanager.

BeerBook and Rob's finger torture device

Without workingenv (some info on Daniel's blog), ipython is harder to use. ipython needs to be started with the same python that your zope uses, so installing it globally eventually only gives you one ipython on your path. If you setup your zope as a workingenv, all python libraries you install are installed local to your zope. Also the ipython binary.

Workingenv in instancemanager: add a `use_workingenv = True' to your project's config file. An instancemanager PROJECTNAME fresh sets up your zope instance for workingenv. If your instance already exists: instancemanager PROJECTNAME --upgradezope fixes it up without destroying anything.

The workingenv is activated automatically when you start/stop/whatever zope. Also ipython is run with the workingenv activated. If you need to activate it yourself: do it by hand. instancemanager PROJECTNAME --activate prints out the line you have to paste in your shell (by nature, it can't be set from instancemanager itself).

Installing ipython should be easy if you've got instancemanager installed, as instancemanager requires setuptools to be installed itself :-) There are no other dependencies.

  • Download ipython from http://ipython.scipy.org/ and extract the tarball.
  • Activate the workingenv: instancemanager PROJECTNAME --activate. This places the correct easy_install right in front in your path.
  • easy_install /directory/where/you/extracted/ipython.
  • Download ipy_profile_zope.py and place it in your ~/.ipython directory (ah, run ipython once beforehand to get that directory).

If you want, you can type deactivate to deactivate your workingenv. Depends on how you use that. It is pretty new to me, so I'm still figuring out the handiest way to work with it.

Running ipython is real easy: instancemanager PROJECTNAME --ipython.

There are possible improvements, but it already seems to work pretty well.

Instancemanager for your plone 3.0

Filed Under:

Yep, instancemanager can create/manage your 3.0 instance just fine.

With all the recent ploneout and ploneenv explanations on how to set up plone 3.0, I'd thought it a good time to chip in with the news that instancemanager also works just fine with plone 3.0. Jean-Paul Ladage added a few lines of code recently that allows instancemanager to install stuff into your zope instance's lib/ directory.

Assumption: you've installed instancemanager and if needed, you've changed its config to suit your taste. After that, you only need to do instancemanager fresh plone30 and you're set. If needed it does an "svn checkout" of plone 3.0's sources. Here's the config:

  # If you've already got a checkout, adapt the location.
  symlinkbundle_sources = [
      {'source': 'plone30',
       'url': 'http://svn.plone.org/svn/plone/bundles/3.0'},
      {'source': 'plone30-lib',
       'url': 'http://svn.plone.org/svn/plone/bundles/3.0-lib',
       'pylib': True},
      ]
  # Adapt the zope version to what you've got.
  zope_version = '2.10.2'
  port = 3000  # Adapt to your taste
  plone_site_name = 'plone'
  user = 'test'
  password = 'test'

Works pretty much like charm. As an addition, I'm thinking about adding an optional call to workingenv so that it would be easy to install eggs locally to that specific zope instance. Support for eggs is pretty much needed, so I guess it will happen pretty soon :-)

Itch scratching and scratch frameworks

Scratching your own itch is an important reason to contribute to an open source project. Prerequisite is that said project allows it.

Joel Spolsky reviews a book about Chandler . I don't know if he's right, but the last time I looked at chandler they still seemed to be in the "early preview" stage. And I thought I've seen presentations on chandler's architecture on the last 4 europython conferences or so that I attended.

Anyway, I want to quote something from the end of his article:

The Chandler team also overestimated how much help they would get from volunteers. Open source doesn't quite work like that. It's really good at implementing copycat features, because there's a spec to work from: the implementation you're copying. It's really good at Itch Scratching features. I need a command line argument for EBCDIC, so I'll add it and send in the code. But when you have an app that doesn't do anything yet, nobody finds it itchy. They're not using it. So you don't get volunteers. Almost everyone on the Chandler dev team got paid.

That's one thing I noticed after I started instancemanager (see also some weblog entries). The first version was already pretty usable and a few people started to use it. And found a small thing lacking here and there. Something that lacks is an itch that needs scratching. And instancemanager provided a framework for hanging your personally-crafted itch-scratcher on :-)

Somehow it turned out to be a good place to attach all sorts of tiny helper scripts and handy additional functionality. The same with archgenxml, of course. It started out as a handy way to generate archetypes contenttypes, but at a certain moment somebody added a way to generate workflow with it. And then someone added CMFMember support. Lately Jean Jordaan added "remember" support. You can add little things here and there. Stuff like that turns a product into a small open source community.

Developer config bundle

Filed Under:

A bundle that sets resourceregistries into debug mode, that switches off cachefu and that installs a couple of add-ons.

You're using all sorts of new-fangled stuff that makes your plone website a lot faster. ResourceRegistries (portal_css and portal_javascripts) combine your stylesheets and spreadsheets into one file, which is handy for caching. You've set up cachefu and squid just right, making your customer very happy.

Now you continue working on the site and want to make some css improvements. Suddenly you have to bang up your keyboard, sacrifice your kitten and do a little rain dance just to get your css files to refresh. Darn resource registry! Darn cachefu! Darn squid that's not in front of my local development instance on my laptop!

Of course, you can uninstall cachefu and put resource registries into debug mode. But I'm testing locally with the customer database, which has all those for-production settings. And the generic setup profile also, of course, sets the production settings.

Solution

The solution: have a product set all those developer settings for you and install it only on your development machine. Fine. Time to get cracking. I created a bundle to do the following:

Resource registries
portal_css and portal_javascripts are put into debug mode. Now you get the real css and javascript files (uncached) instead of plone4321.css and so.
Cachefu
Cachefu is uninstalled, that's the handiest (though brute-force) way of not sending out caching headers that can cause caching side effects.
Products that are installed

Some products are installed that you normally don't want on your production site:

  • PloneErrorReporting provides more details about the error - right in your browser where they occur. It is bundled with plone, but not enabled by default.
  • Clouseau gives you a very nice in-browser javascript-based python prompt.
  • DocFinderTab - There is no svn source for it that I can find. Till lurker puts it in the collective (which he said he'd look into), I've just slapped the extracted tarball in here.
  • enablesettrace adds the possibility to put an import pdb;pdb.set_trace() into your python scripts.

Where to get it

You can do an svn checkout of the bundle at https://svn.plone.org/svn/collective/developerconfig/bundles/plone25develop.

Instancemanager

Maurits added a setting to instancemanager (for instancemanager, also see my posts). Set development_machine = True in your userdefaults.py on your development machine. Next, add the developer config bundle to your project's bundles, but mark it as for-development-only:

symlinkbundle_sources = [
    'preu/externalproducts',
    'preu/products',
    {'source': 'preu/plone25develop',
     'develop': True},
    ]

It is that last dictionary that does the trick. Items with develop=True will only get installed on development machines.

Instancemanager is not really needed, but it can make life much easier in this case.

Instancemanager: now also handy for server setups

Filed Under:

Instancemanager starts to get pretty good at handling server setups. Lots of improvements were made lately.

In the last few weeks loads of improvements were made to instancemanager, mainly aimed at improving instancemanager's server side story. Version 0.3 is out .

Instancemanager was originally aimed at managing your development zope instances. Starting/stopping, creating them, filling the Products/ directory, automatic quickreinstalling everything, that sort of tedious stuff. Something that's tedious is, by definition, something just begging to be rounded up, automated away and shot.

Some of the big changes and improvements:

Optparse
I now use python's standard optparse module to parse the commandline. With all those extra server-side options the homebrewn way was showing its age. Also passing along arguments to options wasn't really possible. So optparse to the rescue. It does mean changes to the way you have to call it. instancemanager start projectname is now instancemanager --zope start projectname.

There are some short options, like -z instead of --zope. And there's the shiny new multi-actions that will take away most or all of the pain of this change. instancemanager --help will show you the new syntax.

Zeo support
This is a biggie. Add a use_zeo = True to your project's config file and instancemanager will create both a zeo server and a zeo client (=zope server) for you. Way quicker when doing a restart, as the zeo server will just keep running with all your data still available. Also the solution for the real server work.

Hey, this is what instancemanager is all about: it is a bit of extra work to create zeo instances, so I normally didn't do it. Now it is only a one-line change in a config file away. Laziness incorporated.

Backup/restore
Jean-Paul Ladage added this. Backup your zope instance (incremental backup) and restore the latest backup. As is usual for instancemanager, there are sensible defaults: everything gets backed up to ~/backups/projectname/*, which is wildly open to customisation, though.

I added some more specific possibilities to a --repozo action: snapshot backups + restore, handy for taking a quick backup separate from the normal backups for use just before you muck up your instance with a new untested product. Also a forced full backup is available.

And a "sync" backup, in which you can copy/paste a running database to another instance: for instance when running a production and preview server in parallel .

Multi-actions
Your ticket to happiness. Perhaps the build-in "fresh" and "soft" commands suit you, perhaps not. They used to combine a few actions into one handy command. create, products, start, quickreinstall. If that wasn't your preferred way of doing things, you were out of luck.

Not anymore! You can have generic or per-project multi-actions. They're really just a list of things you'd pass to instancemanager on the commandline, so that's simple. It also means that you can disable the normal "fresh" and "soft" multi-actions, keeping your server instance free from accidental deletion.

The two server instances ("production" and "preview") that I deployed at philips (intranet for their research department) today only have a single "update" multi-action. Production does a snapshot backup, grabs the latest software and restarts itself. Preview copy/pastes production's database, grabs the latest software and restarts. Handy handy handy.

Special thanks to Maurits van Rees, Jean-Paul Ladage and Russ Ferriday who did a lot of work on this version.

Europython: lightning talks

Lots and lots of short talks.

MosheZ: Componentizing a stairway to heaven

Use components a lot, they are really really cool. (Sounds like zope3 components). One of the big tricks is to always access objects through an interface. ISomethingDoer(o).doSomething().

Raviolli pattern, Risotto pattern, Lasagna pattern (layered). Spaghetti pattern. I don't know what he meant by it, but it sure sounds funny :-)

Michael Hudson - pydoctor

Documentation generation tool. "API docs are basically impossible for mortals to generate", so he created something new. He didn't look at prior art and thought it all over fresh from the start. Extracting docstrings is the easiest part. It knows about zope.interface. It can cope moderately with from xxxx import *. It's still fairly small (2500 lines). Not release yet, no website. (I found an svn repo ).

Reinout van Rees - instancemanager

Instancemanager helps you to manager your zope/plone instances. At Zest software it replaced custom made scripts that had to be adapted for every project.

Some things it can do: create a zope instance, grap products and fill the Products directory, copy a pre-made Data.fs if available, start/stop zope, quickreinstall all products, etc. You can also do several things with a "combining" call: instancemanager vanrees fresh.

Products can be extracted from .tgz, .zip or can be symlinked from svn checkouts. Bundles can also be handled.

Future: two people use it on the server, so it needs a look at the safety aspect: wiping your production server isn't a good idea and instancemanager will currently happily oblige you if you ask it to do just that. Also: Jim Fulton talked about buildout today which does something similar.

Arigo? - pypy thunk and sharedref

Thunk is a way to do lazy evaluation. Sharedref allows python sessions to share variables: they're no copies, they're real variables. You can append something to a list in one running python and you get it updated in the other. Just a demo prototype, but much fun.

Philipp von Weitershausen - zope.testbrowser

Sometimes you just want to treat your web app as a black box, just sending http stuff at it from the outside, behaving as a real user.

Selenium is too slow and it can't be automated (with for instance buildbot). Zope.testbrowser is a programmable browser. There are three variants, one for http connections, one for talking directly to the zope3 publisher and a last one for the zope2 publisher.

All variants can be recorded with the testrecorder. correction 2006-07-05: you can install the server-side testbrowser in almost every webserver, you just have to figure out how to hook it up. It is easy to install inside zope, of course. Note: you need to install on the server which you are testing.

From the testrecorder, you can generate python doctests (and selenium tests, but the doctests are handier for automatic testing).

Mercurial revision control tool

(Talk by video)

Mercurial is written in python.

Sebastian Lopienski - A single word that's almost missing in the abstracts

He read all the abstracts and got some statistics out of it. A word you didn't see a lot is "security". Googling for "Python programming" versus "python security": security is 1.7% of the "programming" results. For perl it is 2.2% and for PHP it is 13%. Either they do a lot for security of they have a big security problem.

He build a small demo website giving the programmer advise based upon his chosen technologies. So: what do we think of the site and do we have input on python-related common problems and pitfals?

Python is faster than assembly. Really.

Speed is measured as distance divided by time.

Someone at CERN said that only 4% of the matter in the universe is known. Well, we can say that there are a few things that don't exist: the tooth fairy, santaclaus and completely specified projects.

So the only distance points that really matter are the business idea and the process is implemented, revenue is created.

The time python takes between those points way less than with assembly or C, so python is faster than assembly.

Niels Mache - Spreed conferencing application

Spreed is implemented in python for some 70-80%. Webcasts, conferencing, powerpoint sharing, etc. Spreed is available as software, as a service and as a hardware appliance.

Martijn Faassen - Update on lxml

Lxml is a python xml parser. Lxml is high performance, it is pythonic and it has lots of features. Not many of the other python xml libraries have all these three. lxml builds on libxml2, but has a much more pythonic interface than libxml2 itself has.

Some changes: there's a new maintainer that did loads of work. There is a 1.0 release and a 1.1 is in alpha. Lots of improvements. If you work with xml in python and you need high performance, pythonic api and lots of features: use lxml.

luis Belmar - itools catalog

itools has a catalog engine.

Philipp von Weitershausen - properties versus decorators

With "property" he means those python new-style-classes that allow you to have just obj.property instead of obj.getProperty() and so.

Decorators allow you to do things like:

  @property
  def feel():
      #xxxxxxx

Doesn't really help when you also have a setter. He tried something with @getproperty and @setproperty. Next try was using something like:

  class JamesBrown:
      class feel(classproperty):
          def getFeel(...)
              #xxxx             
          def setFeel(...)
              #xxxx             

Rob Collins - python software foundation and massage

PSF protects the python intellectual property, funds some research and organises pycon.

And.... Rob is going to raise money for the PSF tonight at the dinner by doing paid neck massages. (He's very good! I had one last year).

Mark Mc Mahon - PyWinAuto

Demo on what you can automate on windows with pywinauto .

Raymond Hettinger - interactive activation and competition

Treat a database as a neural net. You can investigate relationships, compensate for missing data, extrapolate values, etc.

Impossible to blog, all those screens of data, but it was pretty funny to see that it actually works a bit.

He'll have the python code up on the cookbook this evening.

One talk I didn't commit yet from this afternoon:

Gaëtan de Menten - Tiny ERP

Tiny ERP manages accounting, sales/purchases department, stock/production, customer relationship management (CRM), project management, etc. The advantage is that it is integrated and extensible. The target market is small to medium businesses, they're not yet targetting the big customers.

The project was given wider publicity in 2005. There are 5 full-time developers at Tiny and they have some 23 partners in 11 countries.

One convenient feature is automatic partner segmentation: 20% of your customers deliver 80% of your sales, so tiny ERP allows you to filter them out.

The architecture is client/server, but it is server oriented: all the logic is on the server. The database layer is postgreSQL, the object layer is python and the view layer is in XML. The client has almost no logic, it communicates with the server using xml-rpc. Workflows are also defined in XML, but you can generate an image from it.

Jim Fulton - zc.buildout

Interesting talk by Jim about Buildout, a tool that helps you control your eggs-based development. It sounds instancemanager-like :-)

At zope corporation they had problems with getting their heads around how to use eggs in a development environment. Their existing buildout environment was starting to show its age and it might just be the thing to manage those eggs.

Eggs are self-contained zero-installation python module installers. easy_install modulename installs the module for you. At the simples level they are simply directories that can by included in you sys.path.

The hard part is:

  • Deciding which eggs to use.
  • Building up your sys.path.

Setuptools is the heart of all this, it helps you build eggs, it handles dependencies. easyinstall finds distributions on the net and installs the build eggs into specified locations.

Some things he doesn't want:

  • Modify his python installation. Modifying that influences other projects that use the same
  • No runtime configuration (PYTHONPATH).
  • No accidental upgrades (setuptools figures out the dependencies, so installing another module might trigger updates to other modules which you might not always want).
  • Custom script generation.

Jim wanted greater control over the eggs that are used. Specific versions, so that they can test the exact software that they're going to ship. You don't always want the newest of everything.

He figured out that the old zc.buildout could be revamped and be used to manage the eggs. Buildout creates an assembly of parts, for instance to assemble databases, zeo servers, app servers, etc on multiple machines. The initial versions were make-based, which is a terrible scripting language. A few months ago he started on a 1.0 that would be made in python and that would manage eggs.

Buildout consists of recipes, which are small python classes or methods that do one thing (like installing something). Recipes are managed as eggs. So there's support for developing python software using developer eggs.

Each buildout has a configuration database build with python's configparser, a simple example:

 [buildout]
 develop = mkdir
 parts = mkdir
 log-level = INFO

 [data_dir]
 recipe = mkdir
 path = mystuff

It is under active development and it is ready for production. There will be recipes that are missing that you might have to write yourself. A near term goal is better control

Note by Reinout: I've made a program that partially does the same things, though on an entirely different basis and aimed a bit more at plone: "instancemanager"http://plone.org/products/instancemanager . See also the weblog entries on that subject .

Improvements for instancemanager: quickreinstall support

Filed Under:

The instancemanager now also supports quickinstalling your products. And... migration of your Plone.

I released the second version (0.2) of the instancemanager (see the first anouncement for a quick intro). The setup.py now also works with python 2.3, so if that bit you, you're clear now.

Most notable addition: it can quick-reinstall your products (optional). What can be reinstalled (so: not locked, not broken, not removed), is reinstalled.

What I'm finding to be very handy for testing migrations is that instancemanager tries to migrate your plone to a new version if it needs migration. This depends on you providing instancemanager with a copy of your Data.fs on which you want to try the migration. I'm moving my vanrees.org website from plone 2.1 to plone 2.5 and I'm already able to give it the old database, run instancemanager vanrees fresh to be greeted with a freshly setup Products/ directory and a fully migrated and quickisntall-updated site. Just one command!

The important point: this makes it way less time-consuming to test plone migrations, which means there will be more testing.

To get an idea, I'm giving my ~/.instancemanager/vanrees.py config file as an example:

 tgz_sources = [
     'DocFinderTab-0.5.2.tar.gz',
     'linguaplone.tgz',
     'plonelocalfolderng.tgz',
     'qplonecomments.tgz',
     'qplonegooglesitemaps.tgz',
     'redirectiontool.tgz',
     'Quills.tgz',
     ]

 tgz_basedir_template = '%(user_dir)s/svn/vanreesbundle/vanreesintegratie/products'

 tgzbundle_sources = [
     'reinoutproducts.tgz',
     ]

 tgzbundle_basedir_template = '%(user_dir)s/svn/vanreesbundle/vanreesintegratie/products'

 symlinkbundle_sources = [
     'vanreesbundle',
     'quillsbundle',
     'plone25',
     ]

 port = '9000'

 zope_version = '2.9.3'
 python = '/opt/local/bin/python2.4'
 user = 'reutel'
 password = 'kuch'

 plone_site_name = 'plone'
 main_products = ['vanrees',
                  'vanreestheme',
                  ]

I started to generate parts of a usage manual from the source code, just like I generated some archgenxml documentation from the sourcecode. I should automate this a bit more, uploading the documentation to plone.org takes way too much manual effort :-)