At Zest software, we're always refining our buildout strategy. For stable buildouts, you need to make it easy to do the right thing. And you need some mechanisms in place to keep the complexity manageable. The reality is that there are quite a lot of eggs involved in a typical plone project.

Two blog posts that outline what we're doing now:

• Splitting the buildout two ways about having stable/unstable.cfg for product configuration and devel/preview/production for server/instance configuration. Preview.cfg and production.cfg both extend stable.cfg; devel.cfg extends unstable.cfg. This keeps concerns separate.
• My brother Maurits' entry on creating repeatable buildouts which was mostly about pinning eggs in versions and how to do that reliably.

I was bothered by the long list of pinned versions in our buildout's [versions] part. Which are versions we picked ourselves and which are "just" currently used versions? To make the difference easier to spot, all you have to do is to make a small subdivision with comments. A quick "diff" will tell you what the difference between development and production is when you do that. Note that we've also started to pin versions in development.

In the unstable/development configuration:

 [versions]
 # Pin versions that we want to pin to a specific version
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1
 # Not specifically pinned versions.
 # List compiled 2008-07-07 by Reinout.
 # Remove this list and run buildout with '-n', afterwards run:
 # bin/buildout -vvvvv |sed -ne 's/^Picked: //p' | sort | uniq
 Products.CMFSquidTool = 1.5
 Products.CacheSetup = 1.2
 Products.PageCacheManager = 1.2
 Products.PolicyHTTPCacheManager = 1.2
 .....

In the stable configuration:

 [versions]
 # Current version numbers of our own products
 ren.app = 0.8
 ren.model = 0.8
 ren.theme = 0.8
 # Copied from the current unstable.cfg: specific pins.
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1
 # Copied from unstable.cfg: buildout-generated list.
 Products.CMFSquidTool = 1.5
 Products.CacheSetup = 1.2
 Products.PageCacheManager = 1.2
 Products.PolicyHTTPCacheManager = 1.2
 ...

This way, it is easy to spot what has to be copied where and which items you have to update yourself. In case you develop with the latest versions and only want to pin it in production, unstable/development would be just:

 [versions]
 # Pin versions that we want to pin to a specific version
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1

and stable/production:

 [versions]
 # Current version numbers of our own products
 ren.app = 0.8
 ren.model = 0.8
 ren.theme = 0.8
 # Copied from the current unstable.cfg: specific pins.
 SQLAlchemy = 0.4.5
 plone.recipe.plone = 3.1.1
 # Not specifically pinned versions.
 # Generate pin list by running the following command in development:
 # bin/buildout -vvvvv |sed -ne 's/^Picked: //p' | sort | uniq
 Products.CMFSquidTool = 1.5
 Products.CacheSetup = 1.2
 Products.PageCacheManager = 1.2
 Products.PolicyHTTPCacheManager = 1.2
 .....

I hope this gives some ideas to adapt it to your own situation!

Yesterday, monday, was our first day in our new office after the move: yeah, great office space. Every one of us looked happy and content. I was merrily working throughout the afternoon and suddenly noticed that it was 17:45 (I had to go home at 17:00 in order to eat before people arrived)... I just didn't look at the time. I blame it on the nice building :-)

The old location started to get cramped (see last photo), so we needed the extension. Jean-Paul and Esther (=bosses) did manage to find a good office that they're rightfully mighty proud of. Three random things that I especially like:

• Two entire walls filled with whiteboards. I love whiteboards and have a big one at home. Zest's new office takes it to a new level. We immediately put the whiteboards to good use to plan out a new customer project.
• Big Window. One of the long walls is all glass. From floor to ceiling. The programmers are on the second floor: when you come up the stairs the first thing you see is a big open space and a wide view outside. Loads of light, loads of openness. Gives me a feeling I'm capable of more: more overview of projects, better architectures, less bugs, whatever :-)
• Dead next to the metro (=subway). On a day, it shaves 2x10=20 minutes off my commute compared to the old office location. If I manage to be more strict in my morning routine, this bit of margin ought to allow me to be home on a better time. I have a 1 hour 35 commute (one-way) now (was 1h45) three days a week (I work at home two days a week). So I mostly got home just when dinner was finished. Those 20 minutes might make quite a difference!

There's some work to be done on the office, of course. But the office is off to a good start, seeing everyone's enthousiasm on the first day!

The location? Right here , but it is a brand new building, so google maps doesn't have it on its satelite photo yet :-)

I finally got around to copying the backup functionality of ye olde instancemanager to a buildout recipe. The result is a real handy recipe: collective.recipe.backup .

bin/repozo is a zope script to make backups of your Data.fs. Looking up the settings can be a chore. And you have to pick a directory where to put the backups. collective.recipe.backup provides sensible defaults for your common backup tasks. Making backups a piece of cake is important!

• bin/backup makes a backup.
• bin/restore restores the latest backup.
• bin/snapshotbackup makes a full backup, separate from the regular backups. Handy for copying the current production database to your laptop or right before a big change in the site.

Just read the pypi page, especially the section on supported options, though you get perfectly decent backups by adding backup to your buildout's parts list and adding the following two lines:

  [backup]
  recipe = collective.recipe.backup

That's it!

"Koekhappen" is a traditional Dutch children's game on birthdays. "Cake biting" would be the literal translation. You dangle a piece of cake from a wire overhead and the child gets to try and bite in it, possibly blindfolded.

Don't ask your technically educated husband (that is: me) to do that. Some people honestly don't believe there's such a thing as "overengineering". I took a solid piece of rope; attached three smaller cords to it; attached a short cocktail stick to the end; drilled nice round holes in a couple of cakes (with an apple drill); looped the cocktail stick through it.

The good thing is that we can reuse this every year. The other good thing is that I had fun.

I'm not a terribly good presenter, but there's one thing I don't have: fear of presenting as such. I'll tell you why in the next paragraph. Anxiety whether people will like what I'm going to tell: yes. But it doesn't matter to me whether there are 3, 30, 300 or 3000 people in the room. Well, I've only tested it till 300.

As a first-year student at Delft University, I joined a student club . Feburary was the month in which the club's birthday was celebrated. One of the activities was a simple debating contest which I participated in. The first round was at the club itself, with just 15 people around or so, so that didn't scare me.

What did scare me was the second round: at the theater evening with some 80 people present, right dead smack center on the stage. That prospect scared me.

When I came on stage, they put the main spotlight on me and my opponent. The result: you were blinded and couldn't see the audience. Only the bright light. So I could concentrate on the debating contest. No fear as you couldn't see the audience anyway. So I noticed I could speak for 80 people without being eaten or getting shot at.

The realisation that I talked center-stage in front of 80 people helped me a lot. The number of people never bothered me anymore after that. A valuable life experience!

I'm naturally introvert: all the basics to be afraid. But my fear of spotlights was cured by spotlights :-)

KSS best known for the inline editing in Plone. There's a lot more you can do with it. But most of the usage I've seen concerns modifying something. And if you modify something, you need a POST request in http land. The other request you can do, a GET request, is defined as something that you can do multiple times without changing anything. So you can press "reload" in your browser on ordinary pages just fine, but re-submitting a comment 20 times with POST results in 20 identical comments.

So KSS by default uses POST for its requests, as there's often a change involved. The downside is that POST requests aren't cacheable. So a simple KSS plugin that fetches a list of images from the server and that displays a random one from that list keeps on hitting the server. Even though that list of images changes only once a year or so.

Balasz Ree has a branch of KSS that allows GET requests. It only needs reviewing to land in KSS trunk. But we (= zest software ) needed it now, so Balasz cooked up kss.plugin.cacheability that allows GET to work in the current KSS version. It'll provide an easy migration path once the GET functionality itself has landed on trunk.

So if you have KSS requests that don't change a thing and that are expensive 'cause they're POST instead of GET: try it out!

Plone is rightfully known as a friendly community. We got some tasty proof of that at zest software this morning: we're moving our office later this week and got a congratulation cake from Goldmund, Wyldebeast & Wunderliebe, another plone company from the Netherlands. Thanks!

We've got a wonderful reason for our move: we're growing. It is getting too crowded in the office. We're staying in the same neighborhood near Rotterdam, just 200m closer to the metro. The office is about twice as big, so it ought to do for a while :-)

Plone is translated into a whopping amount of languages. Many add-on products also sport at least a few translations. This is all handled by zope's PlacelessTranslationService (PTS).

When starting up, all those translation files are loaded and information about them is stored in the zodb. That takes time and memory. A recent PTS release (at least included in plone 3.1.1) added a nifty trick to reduce that footprint in case you don't need those languages: a PTS_LANGUAGES=en,nl environment variable would make PTS use on the Dutch and English.

In case you use buildout, add the following to your instance part:

 [instance]
 recipe = plone.recipe.zope2instance
 ...
 environment-vars =
     PTS_LANGUAGES en, nl

I just discovered it in some mailinglist message last week and tried it out today. Great little feature. (Note that it seemed to want a comma AND a space when I tested it locally, might be a bug).

Maandag 2 juni was er een debat georganiseerd om te kijken of de bouw nu voor of achter loopt qua ICT. Onder andere georganiseerd door de nieuwbouw, waarvan ik ook twee eerdere samenvattingen heb gemaakt.

De discussie werd geleid door Anton Schippers van Kraan bouwcomputing . Hij had drie stellingen voorbereid waar een panel en de zaal op konden reageren

• Standaardisatie moet afgedwongen worden. Dat is de enige manier om ict toepassingen massaal doorgang te laten vinden in de bouw.
• Over twee jaar kan niemand in de sector meer om het BIM (bouw informatie model) heen.
• Veel bouwondernemingen hebben korte termijn strategiën. Er zijn dus geen noodzakelijke lange termijn investeringen in ict. Dat is funest voor innovatie door middel van ict.

Stelling 1: afdwingen standaardisatie

Ik maakte een opmerking over het ophakken in kleine stukken. Bestaande standaardisatieinitiatieven zijn eigenlijk allemaal groot en bouw-overkoepelend. De terminologie van de hele bouw moet erin gevangen worden. Dat krijg je toch nooit van de grond? Begin liever met kleine losse deelgebieden van de bouw en probeer het later aan elkaar te knopen. (Voordat ik het te lang maak: zie m'n proefschrift ).

Een andere probleem dat werd genoemd was wantrouwen. In de bouw wordt veel geregeld en gesjacherd: transparantie is dan niet gewenst. En als je je collega-bedrijven wantrouwt wil je ook niet dat ze bij "jouw" informatie kunnen. De informatie loopt op deze manier nooit de hele keten door.

Financiering is ook een probleem. "Maar er is toch genoeg geld in Bouwend Nederland?" "Tja, we zijn al zolang bezig... enige mogelijk is gewoon beginnen en proberen."

Er is een BIM case week geweest waar met 90 personen met BIM is gewerkt. Er ontstond vertrouwen en men begon beetje bij beetje samen te werken: ook qua ict communicatie.

ETIM is, refererende aan het ophakken in kleine stukjes, klein begonnen: met inkoopinformatie. Dat werkte en had success omdat het aantrekkelijk was. Omdat er geld mee verdiend kon worden. Belangrijk voor BIM: het moet ook gemakkelijk zijn, bijvoorbeeld voor de loodgieter. Bij Oranjewoud merkten ze dat het in de praktijk al lastig was om een website met wachtwoord voor de meest recente tekeningen gebruikt te krijgen.

Een ander probleem is dat men geen verdienmodel in de nieuwigheden ziet. De staandaard verdienpunten (uitknijpen, afdingen, enz) gaan weg zodra het transparant wordt. Eng.

Iemand kwam terecht met een bedrijfskundige kijk op de zaak: OF de ict toepassing is gewoon niet goed OF je krijgt van het goede product niet verkocht dat het goed is. In die redenatie hoor je geen enkele keer het woord standaardisatie: heeft dat er dan wel mee te maken? We keren in onszelf en komen met een oplossign waar geen behoefte aan is.

De samenvatting van Anton:

• Op deelprojecten kan je succes boeken. Goed punt om mee te beginnen.
• Vertrouwen is een belangrijke factor.
• De behoefte is er kennelijk niet direct.

Stelling 2: BIM over twee jaar onomkoombaar

Eigenlijk iedereen: nee. Voor wie is het het belangrijkste? Eigenlijk de hele bouw. Veel partijen hopen op goede data van het deel van de bouwketen dat voor hen zit (en hebben geen zin om zelf goede data aan de keten na hen te leven). Het enige dat lijkt te zorgen voor marktacceptatie is dwang van bijvoorbeeld de overheid (BIM door de USA overheid, STABU bestek door de NL overheid) of het verkopen van bouwmateriaal (zoals bij de succesvolle ETIM standaard).

Er zijn bedrijven die het nu, met bouwpartners, lekker BIM-achtig geïntegreerd doen en er 40% rendementsvoordeel uit krijgen. En langzaam wordt dat gekopieerd. Maar uiteindelijk heb je het nodig dat een grote partij het doet, niet een paar kleinere bedrijven die die 40% pakken. Concurrentie is iets waar je wakker van wordt. Ikea bouwt goedkope woningen in het Verenigd Koninkrijk: dan gaan UK aannemers echt wel even opletten.

Qua architecten moet je ze eerst allemaal van 2D naar 3D krijgen. Veel bureaus doen het trouwens al of zijn ermee bezig. Iemand uit de aannemerij zei trouwens dat hij graag standaardisatie en afspraken wilde: "steeds weer mooie paketten, maar ze sluiten niet op elkaar aan".

Anton: vroeger wilden automatiseerders niet samenwerken. Nu is het anders. Maar het zit ook op mensen vast: de wil moet er zijn.

Stelling 3: korte termijn visie, dus geen lange termijn ict investeringen

Voor veel bouwondernemers is het inderdaad zo. Veel kleine bedrijven zijn "gewoon aan het werk". Geen innovatie, zowel vakinhoudelijk als qua ict. Investeringen zijn vaak anti-cyclisch. Pas als het slecht gaat proberen ze te innoveren. Ze zijn er trouwens wel, de kleine bedrijven die innoveren. Een van de panelleden had er laatst 80 langs voor workshops.

Het zijn ook veelal juist de kleinere bedrijven die innoveren. Ze zijn van nature veel slagvaardiger.

De markt begint aan de leveranciers te zeggen dat ze het beter kunnen doen. De Rotterdamse havendienst die zaken aan de markt oplegt enzo. Er begint begrip te ontstaan dat er een mismatch is.

Het is een feit dat je tegenwoordig wat kan met BIM. Dat was 10 jaar geleden niet zo, daar moest je niet mee aankomen. Nu werkt het deels tenmiste! Onontkoombaar in twee jaar? Da's te snel, maar er begint wat te komen.

Slotopmerkingen

• Successen vieren zodat het niet te ontkennen is.
• BIM gaat er komen, maar het zullen meerdere BIMs zijn. Je kiest een leuke uit voor je project!
• BIM: leuke nieuwe uitdaging voor software bouwers.
• BIM kan sneller gaan dan we denken. Luister wel goed naar elkaar.
• Een BIM, da's gewoon de toekomst. Maar het is nog best lastig.

Maak het tastbaar, dan kan het groeien. En COMMUNICEER het.

Six feet up just submitted a whole lot of sites to plone.net . As every submission now ends up in the reviewers' mailbox (via the cmfnotifications product), my mailbox was pretty effectively spammed :-) Time to look at the numbers again!

Wow, plone.net now has a whopping 1186 sites from 255 providers. That's a lot. At the 2007 plone conference (half a year ago or so) we just got our 200th provider. So 55 more companies listed themselves. Those 1186 sites is what freaks me out, though :-) Thanks for everyone that lists their plone sites there. Prospective customers can get a real good impression of plone. For instance, sorted by provider , industry or country  . What more do you want when investigating whether to get yourself a plone website or not?

If you want to list yourself as a provider, just log in to the site with your plone.org username, read the short guidelines and submit yourself and your sites. Welcome!

Tags: plone

Having fun with the python documentation generation tool sphinx today. I halfway hooked it up in my buildout, so I have a bin/sphinx-build script with all the eggs and python paths linked in.

Boom, semi-automatic python docstring extraction failed with an import error (on Products.statusmessages, Products.Archetypes, etc.). Aha, those are not in the Products/ dir in parts/zope2/lib/python (which is linked) but in parts/plone/*. It took me quite some time, but in the end (after looking at zope's own code) I added the following near the top of my sphinx conf.py config file:

 import Products 
 # ^^^ From zope's lib/python.
 Products.__path__.append(os.path.abspath('../../parts/plone'))
 # ^^^ Relative path in my situation.

I did not know that individual modules also had a path, I only knew about sys.path. It works :-)

Tags: plone

I took a couple of hours yesterday evening to work through a couple of next actions for projects that I wasn't making any progress on. Nice to sit down in your study room, look at the list "@studyroom" and know you're going to check off a couple of actions in the next hours.

What I noticed: sometimes those next actions are the wrong actions. After completing a few of them I noticed some actions further down the list that were actually more practical/easy/fitting to do first. I would have gotten earlier progress on my projects if I maintained them a bit better and spend more time on the order of next actions.

So: if a project lingers on your list for weeks or months: look at that first action and see if you'd rather pick another action as the next one.

Tags: gtd

A couple of weeks ago I got triggered by a short article on working in project space . I've got a great project/todolist application (omnifocus) but there are days when I hardly look at it as I'm on the roll with a customer project. So I considered that a possible problem. The customer projects are managed in our extreme programming management website. No way I'm going to type over my tasks into omnifocus.

I'm now making a distinction:

• Projects that I want to work on for an extended period of time. It is   actually entirely ok to get into the flow of a customer project and spent a   full day on it. One day of focused work can make a huge difference on a   project. Getting into the flow is important.
• Loose tasks ("buy diapers") or projects that consist more of separate tasks   ("eventually I want to move my website from vanrees.org to   reinout.vanrees.org"). I can do those tasks anytime I want.

For me, omnifocus is great for the second type of tasks. I never have nothing to do if I want to do someting: just give me the list of things I want to brainstorm on or that I want to google.

An important note: maintaining the system you use for that second type of tasks is essential. Get all your commitments ("oh, I still have to ...") out of your head and put them in omnifocus/excel/paper. Only then can you really get into the project-flow, confident that you're not forgetting important things.

I've been quite distracted today by a fire at my former university (technical university Delft, the Netherlands). I studied there at the civil engineering faculty. My building was next to the architecture department's ("bouwkunde"). This morning that building caught fire. No casualties, but the building is a complete write-off. (Photo by praeseodymium )

In that picture there's one spot of visible fire. Three meters to the side and one floor down (approximately, I don't remember the very exact floor) was where I had the meeting with the chairman of my PhD committee. And another floor down was the office of one of my supervisors. Yuck. And the office next to hers and the one next to that: both PhD colleagues that I met quite often. Yuck. It comes pretty close that way.

It was only little more than a year ago that I was inside the building for some small conference...

Update: ok, those rooms that I mentioned: that part of the building just collapsed. You can see the dangling floorplates in this photo . Update 2: and there's a video of the collapse (after 40 seconds).

Tags: bouwkunde, tudelft

Writing it down so I don't forget it the next time I encounter it on my own laptop or a colleagues' laptop. Sometimes plone (or rather the zope process) crashes and gives you an OSX crash report. So: no python tracebacks, but an osx dialog.

If you see something like this, the cause is not in python code, but in some c library. Last week a colleague's laptop crashed because of some wrong sqlite library. I've seen it barf on a mysql lib, too.

Hard to figure out what's happening as you don't have any "nice" python traceback. If you look hard at the details provided by the osx dialog, you can often identify a likely culprit if you keep in mind that you have to look at some c library that your code interacts with (some _sqlite.so in my case). It does make you very appreciative of python as there's a whole class of programming problems that we don't have.

When you're working at the EU parliament, you apparently can only open microsoft word documents. When an EU citizen sends you an ISO ODF (more or less "open office") document, you cannot read it. And, the other way around, if as a citizen you want to watch the live streaming of the EU parliament sessions, you'll need microsoft's media player for it.

Wow, that's a pretty hefty lock-in. Despite many parliament decisions being in favor of openness (for instance for railway security systems). So there's now a petition , started by parliament members, for more openness. They're using a plone site for it, btw.

A good summary is at the ZEA partners website .

• Grok now uses the same list of versions as zope's KGS.
• zc.sourcerelease seems to work now: you can create a tarball from a buildout config file. The tarball contains all the downloaded sub-tarballs and eggs. No more downloads from loads of locations.
• Grokproject can now be run from paster. And it has become smaller.
• Documentation cleanup. Looked at possible grok website improvements (including some mails to the list). A couple of new volunteers for the website!
• five.grok has tests now that demonstrates that it works.
• The url() method can be passed a dict that results in GET parameters at the end of the URL. With the accompanying documentation in multiple places.
• Several how-to's added.
• grok.traversable takes care of a common coding pattern. It allows you to mark attributes and methods as traversable. So http://myobject/something will call the something() method on your object (once you configured it that way with a grok.traversable("something") directive).

• Grok's kss support improved (drag/drop, client actions, etc). And design decisions were made (sprints are great for getting everyone on the same track). Documentation is in a way better shape, as is kss itself.
• Not really grok-related, but Christian Theune finalized some bits and pieces on zeoraid : well-working sync for zeo servers! Sounds great.
• There'll be a zope.sqlalchemy package: that will just include low-level transaction integration. All the policy decisions that are now in all five different sqlalchemy integration packages can/will be build on top of zope.sqlalchemy. Great: unification in that area.
• The docgrok tool (build on top of sphinx) is finally merged.
• grokcore.component 1.0 is released. While writing documentation, Philipp found out a bug in grok which opened up other caves and cans of works. It is all fixed now :-)
• Guido Wesdorp talked a lot for two days :-)
• Permissions and roles work. Especially documentation needs/needed to happen: both the desired situation and the current situation.
• Tests for sql are apparently hard to set up. There's now some test setup with plain sqlalchemy, so without zope stuff in it.
• Buildout and virtualenv. Sometimes the buildout-based setup still fails because of interactions with system packages. Virtualenv might help here. There's work being done to check if it can be remedied.

WSGI is a specification for the the communication between the "gateway" (where http requests come in) and the "application" (the real thing that does something, in our case the zope publisher). The gateway outputs a python dict of environment variables plus a file-like object that's the request that's coming in. On the way out, you have the output (as an iterable of lines).

Important point: the web application no longer owns the process: no more "zopectl start". The whole process is handled by wsgi. It looks like standard CGI, but much more pythonic.

"Zopeproject" is an early go at running/configuring zope as a wsgi process. zope.app.wsgi.getWSGIApplication(/path/to/zope.conf). Zope.conf refers to the site.zcml, site.zcml loads the rest (for instance a demo grok application that he showed). You can start the application with "paster server deploy.ini" (or with apache's mod_wsgi etc). deploy.ini configures what you want to start via wsgi.

Another advantage is that zc.recipe.egg is enough: you don't actually need buildout and custom recipes to run it. Just zc.recipe.egg, a zope.conf, a site.zcml and a paste.ini.

There was some pushback "the repoze guys are doing it wrong the wrong way", "skeletons should not be used, so paster script is bad" and "that's middleware that's not allowed by wsgi". (I didn't agree with all of that, for the record. Putting the contents of a zcml fine as text in a buildout.cfg: yuck) zope.publisher.paste (mentioned next) removes the need for the most "offensive" generated file, btw.

There's something new out of Jim Fulton's fingers: zope.publisher.paste.Application(). It lets you configure the publication in paste.ini by passing parameters like database (zodb or not), the root object, the zcml file, etc. No more zope.conf, just paste.ini.

We can lower the amount of grok-specific code when we use wsgi. A lot of functionality can be hooked up at standard entry points so that we don't have to hook it up ourselves.

Tags: grokplone

Christian Theune - Pagelets

Normally, a layout template calls a provider that renders some template inside it. A pagelet is a template that instead "wraps" a layout template around it.

The pagelet is effectively defined (in zcml) as both a view and a content provider. It registeres one template as the "inner" template and another template as the "outer" layout template. The outer layout view on the pagelet looks up itself as a provider again in that layout template. So there's some going around in circles that's hard to get your head around (until you understand it, according to Chrisian).

In plone's context, it would be a different implementation of the "content" macro in the main_template.

Jean-Paul Ladage and Godefroid Chapelle - KSS and grok

Godefroid already did most of the kss-in-grok work. You'll have to add megrok.kss to your buildout to get the kss goodies. Add some KSS stuff to your html header and you're set.

Jean-Paul managed to get KSS working on views instead of only on contexts, something Godefroid was real happy with as he wanted to get that working for quite some time.

Apart from some demo effect breakage, everything is apparently working like a charm (client actions, server interaction, drag/drop, etc.)