At Zest software, we're always refining our buildout strategy. For stable buildouts, you need to make it easy to do the right thing. And you need some mechanisms in place to keep the complexity manageable. The reality is that there are quite a lot of eggs involved in a typical plone project.

Two blog posts that outline what we're doing now:

• Splitting the buildout two ways about having stable/unstable.cfg for product configuration and devel/preview/production for server/instance configuration. Preview.cfg and production.cfg both extend stable.cfg; devel.cfg extends unstable.cfg. This keeps concerns separate.
• My brother Maurits' entry on creating repeatable buildouts which was mostly about pinning eggs in versions and how to do that reliably.

I was bothered by the long list of pinned versions in our buildout's [versions] part. Which are versions we picked ourselves and which are "just" currently used versions? To make the difference easier to spot, all you have to do is to make a small subdivision with comments. A quick "diff" will tell you what the difference between development and production is when you do that. Note that we've also started to pin versions in development.

In the unstable/development configuration:

 [versions]
 # Pin versions that we want to pin to a specific version
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1
 # Not specifically pinned versions.
 # List compiled 2008-07-07 by Reinout.
 # Remove this list and run buildout with '-n', afterwards run:
 # bin/buildout -vvvvv |sed -ne 's/^Picked: //p' | sort | uniq
 Products.CMFSquidTool = 1.5
 Products.CacheSetup = 1.2
 Products.PageCacheManager = 1.2
 Products.PolicyHTTPCacheManager = 1.2
 .....

In the stable configuration:

 [versions]
 # Current version numbers of our own products
 ren.app = 0.8
 ren.model = 0.8
 ren.theme = 0.8
 # Copied from the current unstable.cfg: specific pins.
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1
 # Copied from unstable.cfg: buildout-generated list.
 Products.CMFSquidTool = 1.5
 Products.CacheSetup = 1.2
 Products.PageCacheManager = 1.2
 Products.PolicyHTTPCacheManager = 1.2
 ...

This way, it is easy to spot what has to be copied where and which items you have to update yourself. In case you develop with the latest versions and only want to pin it in production, unstable/development would be just:

 [versions]
 # Pin versions that we want to pin to a specific version
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1

and stable/production:

 [versions]
 # Current version numbers of our own products
 ren.app = 0.8
 ren.model = 0.8
 ren.theme = 0.8
 # Copied from the current unstable.cfg: specific pins.
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1
 # Not specifically pinned versions.
 # Generate pin list by running the following command in development:
 # bin/buildout -vvvvv |sed -ne 's/^Picked: //p' | sort | uniq
 Products.CMFSquidTool = 1.5
 Products.CacheSetup = 1.2
 Products.PageCacheManager = 1.2
 Products.PolicyHTTPCacheManager = 1.2
 .....

I hope this gives some ideas to adapt it to your own situation!

Yesterday, monday, was our first day in our new office after the move: yeah, great office space. Every one of us looked happy and content. I was merrily working throughout the afternoon and suddenly noticed that it was 17:45 (I had to go home at 17:00 in order to eat before people arrived)... I just didn't look at the time. I blame it on the nice building :-)

The old location started to get cramped (see last photo), so we needed the extension. Jean-Paul and Esther (=bosses) did manage to find a good office that they're rightfully mighty proud of. Three random things that I especially like:

• Two entire walls filled with whiteboards. I love whiteboards and have a big one at home. Zest's new office takes it to a new level. We immediately put the whiteboards to good use to plan out a new customer project.
• Big Window. One of the long walls is all glass. From floor to ceiling. The programmers are on the second floor: when you come up the stairs the first thing you see is a big open space and a wide view outside. Loads of light, loads of openness. Gives me a feeling I'm capable of more: more overview of projects, better architectures, less bugs, whatever :-)
• Dead next to the metro (=subway). On a day, it shaves 2x10=20 minutes off my commute compared to the old office location. If I manage to be more strict in my morning routine, this bit of margin ought to allow me to be home on a better time. I have a 1 hour 35 commute (one-way) now (was 1h45) three days a week (I work at home two days a week). So I mostly got home just when dinner was finished. Those 20 minutes might make quite a difference!

There's some work to be done on the office, of course. But the office is off to a good start, seeing everyone's enthousiasm on the first day!

The location? Right here , but it is a brand new building, so google maps doesn't have it on its satelite photo yet :-)

I finally got around to copying the backup functionality of ye olde instancemanager to a buildout recipe. The result is a real handy recipe: collective.recipe.backup .

bin/repozo is a zope script to make backups of your Data.fs. Looking up the settings can be a chore. And you have to pick a directory where to put the backups. collective.recipe.backup provides sensible defaults for your common backup tasks. Making backups a piece of cake is important!

• bin/backup makes a backup.
• bin/restore restores the latest backup.
• bin/snapshotbackup makes a full backup, separate from the regular backups. Handy for copying the current production database to your laptop or right before a big change in the site.

Just read the pypi page, especially the section on supported options, though you get perfectly decent backups by adding backup to your buildout's parts list and adding the following two lines:

  [backup]
  recipe = collective.recipe.backup

That's it!

KSS best known for the inline editing in Plone. There's a lot more you can do with it. But most of the usage I've seen concerns modifying something. And if you modify something, you need a POST request in http land. The other request you can do, a GET request, is defined as something that you can do multiple times without changing anything. So you can press "reload" in your browser on ordinary pages just fine, but re-submitting a comment 20 times with POST results in 20 identical comments.

So KSS by default uses POST for its requests, as there's often a change involved. The downside is that POST requests aren't cacheable. So a simple KSS plugin that fetches a list of images from the server and that displays a random one from that list keeps on hitting the server. Even though that list of images changes only once a year or so.

Balasz Ree has a branch of KSS that allows GET requests. It only needs reviewing to land in KSS trunk. But we (= zest software ) needed it now, so Balasz cooked up kss.plugin.cacheability that allows GET to work in the current KSS version. It'll provide an easy migration path once the GET functionality itself has landed on trunk.

So if you have KSS requests that don't change a thing and that are expensive 'cause they're POST instead of GET: try it out!

Plone is rightfully known as a friendly community. We got some tasty proof of that at zest software this morning: we're moving our office later this week and got a congratulation cake from Goldmund, Wyldebeast & Wunderliebe, another plone company from the Netherlands. Thanks!

We've got a wonderful reason for our move: we're growing. It is getting too crowded in the office. We're staying in the same neighborhood near Rotterdam, just 200m closer to the metro. The office is about twice as big, so it ought to do for a while :-)

Plone is translated into a whopping amount of languages. Many add-on products also sport at least a few translations. This is all handled by zope's PlacelessTranslationService (PTS).

When starting up, all those translation files are loaded and information about them is stored in the zodb. That takes time and memory. A recent PTS release (at least included in plone 3.1.1) added a nifty trick to reduce that footprint in case you don't need those languages: a PTS_LANGUAGES=en,nl environment variable would make PTS use on the Dutch and English.

In case you use buildout, add the following to your instance part:

 [instance]
 recipe = plone.recipe.zope2instance
 ...
 environment-vars =
     PTS_LANGUAGES en, nl

I just discovered it in some mailinglist message last week and tried it out today. Great little feature. (Note that it seemed to want a comma AND a space when I tested it locally, might be a bug).

Six feet up just submitted a whole lot of sites to plone.net . As every submission now ends up in the reviewers' mailbox (via the cmfnotifications product), my mailbox was pretty effectively spammed :-) Time to look at the numbers again!

Wow, plone.net now has a whopping 1186 sites from 255 providers. That's a lot. At the 2007 plone conference (half a year ago or so) we just got our 200th provider. So 55 more companies listed themselves. Those 1186 sites is what freaks me out, though :-) Thanks for everyone that lists their plone sites there. Prospective customers can get a real good impression of plone. For instance, sorted by provider , industry or country  . What more do you want when investigating whether to get yourself a plone website or not?

If you want to list yourself as a provider, just log in to the site with your plone.org username, read the short guidelines and submit yourself and your sites. Welcome!

Tags: plone

Having fun with the python documentation generation tool sphinx today. I halfway hooked it up in my buildout, so I have a bin/sphinx-build script with all the eggs and python paths linked in.

Boom, semi-automatic python docstring extraction failed with an import error (on Products.statusmessages, Products.Archetypes, etc.). Aha, those are not in the Products/ dir in parts/zope2/lib/python (which is linked) but in parts/plone/*. It took me quite some time, but in the end (after looking at zope's own code) I added the following near the top of my sphinx conf.py config file:

 import Products 
 # ^^^ From zope's lib/python.
 Products.__path__.append(os.path.abspath('../../parts/plone'))
 # ^^^ Relative path in my situation.

I did not know that individual modules also had a path, I only knew about sys.path. It works :-)

Tags: plone

A couple of weeks ago I got triggered by a short article on working in project space . I've got a great project/todolist application (omnifocus) but there are days when I hardly look at it as I'm on the roll with a customer project. So I considered that a possible problem. The customer projects are managed in our extreme programming management website. No way I'm going to type over my tasks into omnifocus.

I'm now making a distinction:

• Projects that I want to work on for an extended period of time. It is   actually entirely ok to get into the flow of a customer project and spent a   full day on it. One day of focused work can make a huge difference on a   project. Getting into the flow is important.
• Loose tasks ("buy diapers") or projects that consist more of separate tasks   ("eventually I want to move my website from vanrees.org to   reinout.vanrees.org"). I can do those tasks anytime I want.

For me, omnifocus is great for the second type of tasks. I never have nothing to do if I want to do someting: just give me the list of things I want to brainstorm on or that I want to google.

An important note: maintaining the system you use for that second type of tasks is essential. Get all your commitments ("oh, I still have to ...") out of your head and put them in omnifocus/excel/paper. Only then can you really get into the project-flow, confident that you're not forgetting important things.

Writing it down so I don't forget it the next time I encounter it on my own laptop or a colleagues' laptop. Sometimes plone (or rather the zope process) crashes and gives you an OSX crash report. So: no python tracebacks, but an osx dialog.

If you see something like this, the cause is not in python code, but in some c library. Last week a colleague's laptop crashed because of some wrong sqlite library. I've seen it barf on a mysql lib, too.

Hard to figure out what's happening as you don't have any "nice" python traceback. If you look hard at the details provided by the osx dialog, you can often identify a likely culprit if you keep in mind that you have to look at some c library that your code interacts with (some _sqlite.so in my case). It does make you very appreciative of python as there's a whole class of programming problems that we don't have.

When you're working at the EU parliament, you apparently can only open microsoft word documents. When an EU citizen sends you an ISO ODF (more or less "open office") document, you cannot read it. And, the other way around, if as a citizen you want to watch the live streaming of the EU parliament sessions, you'll need microsoft's media player for it.

Wow, that's a pretty hefty lock-in. Despite many parliament decisions being in favor of openness (for instance for railway security systems). So there's now a petition , started by parliament members, for more openness. They're using a plone site for it, btw.

A good summary is at the ZEA partners website .

• Grok now uses the same list of versions as zope's KGS.
• zc.sourcerelease seems to work now: you can create a tarball from a buildout config file. The tarball contains all the downloaded sub-tarballs and eggs. No more downloads from loads of locations.
• Grokproject can now be run from paster. And it has become smaller.
• Documentation cleanup. Looked at possible grok website improvements (including some mails to the list). A couple of new volunteers for the website!
• five.grok has tests now that demonstrates that it works.
• The url() method can be passed a dict that results in GET parameters at the end of the URL. With the accompanying documentation in multiple places.
• Several how-to's added.
• grok.traversable takes care of a common coding pattern. It allows you to mark attributes and methods as traversable. So http://myobject/something will call the something() method on your object (once you configured it that way with a grok.traversable("something") directive).

• Grok's kss support improved (drag/drop, client actions, etc). And design decisions were made (sprints are great for getting everyone on the same track). Documentation is in a way better shape, as is kss itself.
• Not really grok-related, but Christian Theune finalized some bits and pieces on zeoraid : well-working sync for zeo servers! Sounds great.
• There'll be a zope.sqlalchemy package: that will just include low-level transaction integration. All the policy decisions that are now in all five different sqlalchemy integration packages can/will be build on top of zope.sqlalchemy. Great: unification in that area.
• The docgrok tool (build on top of sphinx) is finally merged.
• grokcore.component 1.0 is released. While writing documentation, Philipp found out a bug in grok which opened up other caves and cans of works. It is all fixed now :-)
• Guido Wesdorp talked a lot for two days :-)
• Permissions and roles work. Especially documentation needs/needed to happen: both the desired situation and the current situation.
• Tests for sql are apparently hard to set up. There's now some test setup with plain sqlalchemy, so without zope stuff in it.
• Buildout and virtualenv. Sometimes the buildout-based setup still fails because of interactions with system packages. Virtualenv might help here. There's work being done to check if it can be remedied.

WSGI is a specification for the the communication between the "gateway" (where http requests come in) and the "application" (the real thing that does something, in our case the zope publisher). The gateway outputs a python dict of environment variables plus a file-like object that's the request that's coming in. On the way out, you have the output (as an iterable of lines).

Important point: the web application no longer owns the process: no more "zopectl start". The whole process is handled by wsgi. It looks like standard CGI, but much more pythonic.

"Zopeproject" is an early go at running/configuring zope as a wsgi process. zope.app.wsgi.getWSGIApplication(/path/to/zope.conf). Zope.conf refers to the site.zcml, site.zcml loads the rest (for instance a demo grok application that he showed). You can start the application with "paster server deploy.ini" (or with apache's mod_wsgi etc). deploy.ini configures what you want to start via wsgi.

Another advantage is that zc.recipe.egg is enough: you don't actually need buildout and custom recipes to run it. Just zc.recipe.egg, a zope.conf, a site.zcml and a paste.ini.

There was some pushback "the repoze guys are doing it wrong the wrong way", "skeletons should not be used, so paster script is bad" and "that's middleware that's not allowed by wsgi". (I didn't agree with all of that, for the record. Putting the contents of a zcml fine as text in a buildout.cfg: yuck) zope.publisher.paste (mentioned next) removes the need for the most "offensive" generated file, btw.

There's something new out of Jim Fulton's fingers: zope.publisher.paste.Application(). It lets you configure the publication in paste.ini by passing parameters like database (zodb or not), the root object, the zcml file, etc. No more zope.conf, just paste.ini.

We can lower the amount of grok-specific code when we use wsgi. A lot of functionality can be hooked up at standard entry points so that we don't have to hook it up ourselves.

Tags: grokplone

Christian Theune - Pagelets

Normally, a layout template calls a provider that renders some template inside it. A pagelet is a template that instead "wraps" a layout template around it.

The pagelet is effectively defined (in zcml) as both a view and a content provider. It registeres one template as the "inner" template and another template as the "outer" layout template. The outer layout view on the pagelet looks up itself as a provider again in that layout template. So there's some going around in circles that's hard to get your head around (until you understand it, according to Chrisian).

In plone's context, it would be a different implementation of the "content" macro in the main_template.

Jean-Paul Ladage and Godefroid Chapelle - KSS and grok

Godefroid already did most of the kss-in-grok work. You'll have to add megrok.kss to your buildout to get the kss goodies. Add some KSS stuff to your html header and you're set.

Jean-Paul managed to get KSS working on views instead of only on contexts, something Godefroid was real happy with as he wanted to get that working for quite some time.

Apart from some demo effect breakage, everything is apparently working like a charm (client actions, server interaction, drag/drop, etc.)

Martijn Faassen showed a grok + sqlalchemy application he's been working on. This got everyone into the spirit and more importantly into the problem domain. Something he used a lot was custom traversal to navigate the relational database model with URLs. With some quick pseudocode:

  from zope.location.location import located

  class Something(grok.Model):
      def traverse(self.name):
          item = session.query(...).first() # or .one()
          if item == None:
              return
          return located(item)

Storm is an alternative to sqlalchemy. Storm seems to be oriented more towards the needs of big complex projects. Sqlalchemy seems much more active open source community-wise. Storm is less "friendly" as you have to / can configure everything, just like zope3. Sqlalchemy seems more grok-like in outlook. At least, that's what I extracted from the discussion.

An outcome of the discussion was that grok needs a to make it real easy to traverse to attributes. Unrelated to, but useful for, relational db integration. grok.traversable("something") was suggested. So this is in addition to the normal container traversal which effectively works with dictionary access (container["subitem_id"]). A shorthand that prevents you from having to write your own traverse() method in many common cases: so very grok-like.

A useful warning came up in the discussion. In zope, it is really easy to see everything as folders. In sql, you're making it all relational. Both are not 100% right object-oriented wise. Watch out for brainwashing based on your environment.

We went through the zope api docs for zope.app.container, writing down which sqlalchemy information we ought to use for treating the mapped sqlalchemy results as a container. This helped us to flush out some corner cases.

An essential point is to use sqlalchemy and its ORM mapper for the things that that is good at. As grok we won't do the sqlalchemy work, we'll just provide extra grok-specific functionality like traversal and folderish behaviour. Folderish behaviour means, as an example, in handling a "delete" correctly. Correctness is should be handled on the sqlalchemy level: if you configure your mapper right, the right thing will happen. That might sound dead-pan but it is, actually.

Grok also means: "not invented here: in a positive way". What other people figured out and did right is what we don't have to re-invent. So we won't invent it here.

Something that we'd need to figure out: how and where to define the classes, tables and mappers? Do we want it all easily in one class? Do we need to grok something?

Grok needs the fields for the add and edit forms. We didn't like defining an interface for the fields as that's partially double, partially potentially late-binding, etc. So something simple like:

 def form_fields(self):
     return grok.Fields(rdb.Schema(OurClass))

There was much more, but this ought to do for a summary. I didn't contribute much myself (especially since I have basically no grok experience!). I did learn a lot by listening, picking up useful concepts and ways of working. Nice sprint till now!

It is now slightly more than five years since my first blog post here on vanrees.org . 14 April 2003 . That's at least what my current blog software shows. I know I had some other items on an earlier website that was running on my university workstation, but I won't count that.

Nowadays the topic is mostly plone and some personal things (like video's of model railway shows). The plone stuff shows up on planet.plone.org, I filter out the rest. So if you're reading my weblog through planet.plone.org: you're missing (or not... :-) ) something.

The best feedback I got: my conference and sprint summaries. I'm apparently able to provide good accurate summaries in a ridiculously short turnaround (=practically live) time. Time to hone that skill in the next few days when I'm attending the grokkerdam sprint, though my brother Maurits is already doing that, so I might wanna try my hand at some limited video blogging instead :-)

Tags: plone, grok

Command line history turned up a few times in my feedreader this morning. Here's mine:

$ history|awk {a[$2]++ } END{for(i in a){print a[i] " " i}} |sort -rn|head
102 svn
85 cd
31 emacs
30 l
19 less
19 instancemanager
19 bin/supervisorctl
15 cp
14 timelog
13 bin/supervisord

"l" is of course some "ls" alias. "instancemanager" is there in favour of my now-regular "bin/buildout" as I was working in some older project on friday. "timelog" is my commandline version of gtimelog . And supervisord is an app that starts multiple programs at once and manages them (a zeo server, a zope instance via paste and varnish in my case, I worked on a new website this weekend).

Tags: plone

Some co-workers had problems with infrae.subversion in their buildout. I didn't get to the bottom of it, but apparently it depends on py 0.9.0 which, at least in buildout's eyes, disappeared from pypi after a recent 0.9.1 release.

If you have the same problems: a simple temporary fix is to put the next url in your buildout's "find-links" section: http://codespeak.net/download/py/py-0.9.0.tar.gz

Tags: plone

I'm playing with deliverance (hurray!) and repoze (hurray, too!) by using it for a brand new church website I'm doing in my own time. (See deliverance summary )

I found one big problem till now: kupu did not work. Kupu's edit screen is actually an iframe. So you can guess that existing content gets wrapped in the theme.... inside that edit iframe. I ignored the problem until today because I had to actually add content today.

The solution is easy: copy kupu's emptypage.pt into your skin product (for instance) and add the following line in the head section:

<meta http-equiv="x-deliverance-no-theme" content="1" />

I got that out of some deliverance mailinglist post. Blogging about it ought to make the solution more findable. Note that I didn't manage to use the same trick on the manage_main page (and friends). There'll probably be a more sane solution later. Deliverance/repoze development is proceeding at a nice pace!

If you want to know more about deliverance and repoze: head over to plone.tv for Paul Everitt's talk on deliverance and Tres Seaver's talk on repoze .

Tags: plone, deliverance, repoze