Stefan showed off ipython on saturday. Pretty impressive. I've now added support for this in instancemanager.

Without workingenv (some info on Daniel's blog), ipython is harder to use. ipython needs to be started with the same python that your zope uses, so installing it globally eventually only gives you one ipython on your path. If you setup your zope as a workingenv, all python libraries you install are installed local to your zope. Also the ipython binary.

Workingenv in instancemanager: add a `use_workingenv = True' to your project's config file. An instancemanager PROJECTNAME fresh sets up your zope instance for workingenv. If your instance already exists: instancemanager PROJECTNAME --upgradezope fixes it up without destroying anything.

The workingenv is activated automatically when you start/stop/whatever zope. Also ipython is run with the workingenv activated. If you need to activate it yourself: do it by hand. instancemanager PROJECTNAME --activate prints out the line you have to paste in your shell (by nature, it can't be set from instancemanager itself).

Installing ipython should be easy if you've got instancemanager installed, as instancemanager requires setuptools to be installed itself :-) There are no other dependencies.

• Download ipython from http://ipython.scipy.org/ and extract the tarball.
• Activate the workingenv: instancemanager PROJECTNAME --activate. This places the correct easy_install right in front in your path.
• easy_install /directory/where/you/extracted/ipython.
• Download ipy_profile_zope.py and place it in your ~/.ipython directory (ah, run ipython once beforehand to get that directory).

If you want, you can type deactivate to deactivate your workingenv. Depends on how you use that. It is pretty new to me, so I'm still figuring out the handiest way to work with it.

Running ipython is real easy: instancemanager PROJECTNAME --ipython.

There are possible improvements, but it already seems to work pretty well.

Vincenzo: plone 3.0 ships with a new workflow. They've worked on the workflow transitions, also they fixed most of the tests.

Balasz: KSS! "Thanks to everyone who participated." This was a much bigger success than he hoped for, as a large amount of issues has been solved. Another improvement: browser-based tests with selenium. They instituted a policy at this sprint to add selenium tests for every issue that they tried to fix.

A bummer for him is that Kupu broke IE support. Kupu also uses the sarissa javascript library, but modify something in it... but keep the name. So KSS breaks. Grrrr.

Nate: plone4artists. They focused mostly on the audio product, they barely touched the video product. They did add the possibility to add a regular ATLink to a youtube video to your site which is then transformed in to a nice view. Load of extra tests. Also added: searching via the catalog for artists, track names, etc. Internationalisation (Japanese, Portuguese). Metadata you add to your audio contenttype ends up in the mp3 itself. And Jonathan made a great screencast.

Stefan: xml export/import. Work on exporting of references: they now simply pickle the references (tip from Daniel). They have some tests for importing, but it hasn't been implemented yet.

Anton: they looked at file/image widgets for formlib. They originally looked at p4a's audio/video, but made it more generic afterwards.

Rob: genericsetup and remember. Work started at the BBQ sprint was continued. Adding metadata (like versions) to genericsetup data, working towards supporting upgrading and migration steps and so. Some 75% seems finished. Membrane/remember had a problem: it always assumed that a content object contained the authentication data. This works OK unless you want to use LDAP or so. So this assumption had to be removed: they didn't actually get very far with this.

Kapil: he basically worked solo and continually helped other people. He himself did some work on plonecommerce, for which he needed something quick and very simple to attach some properties to a member (see friday's summary ). He's got it mostly working.

Reinout: cachefu. Getting the tests to run on both 2.5 and 3.0. Finalising the genericsetup support and adding unittests for it. Also a lot of code research, looking at zope 3, lovely.viewcache , getting great ideas from wiggy, philikon and optilude.

Belgian Bug Busters: loads of bug fixing! Often a lot of bugs were hiding behind an original one.

Ramon and Alec fixed security issues for the last two days.

Webdesigners know a lot about webpages and how to design and build them. On the other side there are the web content management systems that know how to maintain content. Those CMSs do make it harder to change the interface.

It is based on an idea of Paul Everitt, so by definition it has XSLT at the heart of it. It divides "content" and "theme". Templating still has a place on the CMS/content side of things, it are the looks that are to be separated out. The output of the content part is: html. The output of the theme part is: html and css.

Deliverance takes the html of the content part and the html+css of the theme part and hooks them up together based upon a mapping. Did we mention XSLT? Basically you grab identifiable chunks out of the content and inject them into the theme's html.

You can have deliverance running as either a proxy server or as wsgi. It has both a python and an xslt transformation engine, the xslt one is twice as fast at the moment. The python transformation engine does mean that you can do all python can do to the content that comes out of it eventually.

Deliverance has a couple of xml files that are loaded by a top-level rule xml file that defines which html items to grab from the source and where to inject them in the theme. You can even grab bits and pieces from different files. The demo he showed was to put plone.org's content in burningman's website. A comment he got was "you're helping phishing sites" :-)

A real example: http://openplans.org/projects/topp-the-organization is the content, http://topp.openplans.org is the site that gets created out of it. They also use it for some other sites, mixing the theme of the actual site with the openplans.org content management stuff.

According to what I heard, it seemed that Limi liked the approach and that a variant of this approach might end up as the default way to

Nate Aune gave a second demonstration. He deals a lot with small artists that have a small html-only website designed by the proverbial nephew. After a while, they want something more, but can't as they don't have a CMS. In cases like this, deliverance can allow you to manage your content in plone, while reusing the original html site's layout.

There are a couple of challenges. In a way, a lot of double work is being done. Plone renders a complete page, but afterwards most of it gets cut away. A way to handle this is to make the plone interface much more lightweight.

Rob showed a possible upcoming openplans architecture where deliverance is placed right up in front.

Plone4artists

First Rocky about plone4artists. They've actually been tracking progress on openplans.org. A group has been working on a formlib widget for uploading files. There was an existing widget, but that didn't cut it. Nice: Jonathan has made a new screencast that'll be put online later in the day. I saw it and it shows off the capabilities pretty well. Of course a lot of bugfixing.

I asked: at least 8 or 9 people are working on plone4artists. They've got more functional tests (all testbrowser tests). Folders now show an mp3 track listing for the mp3 files inside it. For the rest, they've looked at quite a number of things, but haven't really delved into it. Well, Rocky is very pleased about the progress. A beta for the audio part seems possible, but an alpha for the video part is doubtful, as everyone has been focusing on the audio part.

This sprint will make Rocky famous, as he's now got his own mp3 recording: the demo test file for the functional tests :-)

Kapil's member stuff

Kapil wants to have a very easy way to attach some properties to member objects. The properties are attached to members with a bit of zcml and a zope3 schema. Under the hood, a PAS plugin takes care of storing the data.

The difference with remember/membrane: it is simpler, but the members are regular plone members, not content objects: so no workflow. The code of ore.member is available in svn .

David Convent - export and import of content with genericsetup

He fixed some plone 3 bugs with Daniel. The ordering of contenttypes and action providers was off some.

For the rest he worked with Stefan and Ramon on import and export of content in plone. XMLforest and gsxml were the candidates and gsxml was the one they picked. Gsxml missed references, so they tried to add that. For that, they needed to touch marshall. The unittests of marshall weren't running and they tried to fix that.

After a while, Kapil showed them another generic setup based import/export product that handled references, but that also had limitations. In the end, they're using zexp/pickles to export the references folder, which seems to work (magically).

They believe they've got all the necessary parts, now all that's left to do is assemble them :-)

One very very very very handy tool they discovered is ipython for doing an interactive python session. I mentioned clouseau (the great in-python interactive javascript-enabled python shell) to Stefan and he said he used it until he found ipython and never touched clouseau afterwards. You need a special zope profile that Stefan will put online real soon or I'll have his hide. Interactive editing, tab completion. A traceback shows not only the line itself, but shows the line in context (2 lines before and after the offending line). not only the final line where it fails, but it does it for every line in the traceback.

Something they wanted me to write down: if you install python on the mac using darwinports, you get a buserror when using libxml2.

Alec Mitchell

Mostly walking around making sure that people who are fixing bugs know what exactly needs fixing and helping them getting started. Also he's working a bit on a relationship product that he maintains. At the moment, he's fixing security holes in plone with Ramon.

Balasz Ree - KSS

When I sat next to him he was getting frustrated by templates and macros that need a global context. They at least often need more than is provided/detectable inside the macro itself.

KSS status: loads of bugfixing happened. There has especially been a load of testing to really verify the bugs were gone.

Friday was pretty relaxed, as some 3/4 of the participants took the opportunity to visit the ruins of Pompeii in the afternoon. Did you know that Nate discovered proof that the Romans used Plone?

On thursday, we took the group photo. It is a very pleasant sprint, as is visible from the photo. Some 50 or 60 people turned up, with quite a number of first-time sprinters amongst them, which is great news for Plone's future. I haven't yet heard any negative sound about the sprint (apart from the speed and reliability of the internet connection, ah well). The hotel is good. The evening dinner is pretty good. The staff is very friendly. The bartender is pleasantly nuts.

I really should start to bug some people tomorrow moning on the various groups' sprint progress, otherwise I would only chat on about cachefu :-) As I haven't done that yet, you'll all forgive me that I mention that we've integrated the PageCacheManager settings into CacheFu's genericsetup profile. This brings the list of completed items from the original list to two: fixing the unittests (as far as currently practical, there are some plone 3.0 issues left) and finalising the genericsetup support.

I and Machiek did most of the work while pair programming. It was a first for me to have two full days of pair programming. I did some of the work on my own late in the evening, but that's probably OK. I'm enthousiastic about it. You both learn things. You both make simple mistakes that the other one catches immediately. You can discuss ways to approach the problem. You transfer knowledge. You see each other's way of working.

I got some more very valuable input from Philip and Wichert. In short: using events for hacking into the publishing process might be possible after all. Core to this is to raise an exception if cachefu detects that a "304 not modified" is in order of if a cached version of the page is available. There's code available (plone 3.0 has it build-in) that monkeypatches zope's publisher to look up views for errors instead of blindly spewing out an error page. Wow! The 3.0 code can be used for 2.5, too, so that looks promissing.

This is a really big change. For CacheFu. Getting this to work would be a great result of this sprint. We've got some days left to get it to work. Maciek, are you up to it? :-) *If there are others that want to help work on this: feel free. CacheFu is pretty important and there's a limit to what Maciek and me can accomplish in this sprint. There are also still the other useful tasks left... :-)

One other remark: I'm extremely happy with instancemanager . It is a tool for managing instances: creating them, backing them up, installing products (symlink, svn bundle, tgz), running tests, etcetera. All based on a reasonably simple config file. When I started this sprint, I had already a plone 3.0 config (see instancemanager for your plone 3.0 ) and a plain plone 2.5 config. After creating the "sorrento" bundle of cachefu, getting that installed in an instance took about a minute: copy plone25.py (the 2.5 instancemanager config) to cachefu.py, change the portnumber, add the sorrento bundle to the config, run instancemanager cachefu fresh. Done. Same for cachefu3.py, to get cachefu running inside a plone 3.0 site. No (in my eyes) elaborate in-location downloads of all software in a separate location as ploneout seems to do it. Just effectively a "mkzopeinstance" in a location of your choice while grabbing the bundles from your already-downloaded svn locations. Real quick and convinient.

Now, off to bed.

To reach more users, we need to reach them. For this we need more than technology: confidence, sustainability, quality, services. How do you train people? How long will plone exist? Things like that.

That was one of the reasons to start the ZEA network. It is an organisation of 22 partner companies. It involves promotion, some projects and now also open source research.

If we have a clear message to decision makers as a reliable technology, if we show up in a lot of research reports on open source: that is good for the partners and for plone. ZEA has pretty good relations with some universities. Being involved in research could also mean more funding.

The EU floss impact report was a study on the impact of free and open source software on innovation and competitiveness of Europe. ZEA helped that report because of its connections with and knowledge of those 22 SMEs that are a member of ZEA. (Xavier showed some of the figures from that report, there's a readable abstract on ZEA's website.

For ZEA, it is important to stay in touch with the researchers that make such reports because we can give them accurate information on our part of the IT sector. In the first draft of the EU report, zope/plone wasn't included and they also missed several case studies. ZEA helped them to include zope/plone and helped them with case studies. A core of that case study was the CommunesPlone project of some 20 Belgian local governments that also has a lot of contacts with other individual developers and SMEs. That kind of software ecosystem was exactly one of the things they wanted to show off in the report.

Current research in the EU - in so far that it has to do with open source - is mostly research on the impact of open source, the quality, the structure of projects, etcetera. There isn't yet much actual programming going on in this kind of research, but the percentage of actual programming is expected to rise.

There are all sorts of enlightening results coming from the analysis-type or results. Analysis of turnover of project contributors in an open source project: in a nice, regular pattern. Analysis of dependencies in projects. Project structure comparisons. Etcetera.

There is work to be done, though, to translate the findings of these research projects

In Switzerland, some Kantons (districts) are using plone for intranets and websites, there is also a UN website, and several other government agencies use it. Still it is a problem to promote plone. Some of the problems:

• IT departments are scared of open source as they've invested a lot in Microsoft and fear for their job if open source works well.
• Commercial players lobby well amongst decision makers. And it works.
• "Nobody is fired if a project based on Microsoft or Oracle fails".
• Big projects have to go through a very elaborate tendering process which is way too expensive for individual smaller companies.
• Open source is perceived as "linux on the desktop".
• There are not enough plone companies in Switzerland.

Open source also has its advantages. You can do guerrilla marketing because, if you find the right people, you can give them the open source software, install it for them and they'll run it "in secret". Afterwards, if it works, they will evangelise in their own organisation. You have freely distributable prototypes, so it is easy to try out. You can get excellent service, also you have an international community around it.

Plone is set up in a modular way, so something like PloneGov can be developed in small steps. Individual government instances can arrange for the creation of individual parts, which can be exchanged with the other government instances. You pool resources in this way and you can stay below the amount of money where you'd need the expensive tendering process.

They're now looking at some dossier/record management system within that PloneGov project. Bernard showed us the current state of the application. One of the things that they changed compared to core plone is a distinction between a "desktop" and a "repository". So no in-place editing of items, you first check out an item out of your repository and place it on your desktop before you can edit it. (Desktop in this case is just a personal working area inside plone, not your computer's desktop). When you're ready, you place it again into the repository. There's also a way to assign tasks to people (which uses archetypes' reference mechanism a lot).

The customers helped them a lot by telling them the things that were wrong with the current existing commercial systems. None of those systems is fully adapted to the organisations' needs.

As PloneGov, they hope to make it really easy for governments to pool relatively small amounts of money to get additional functionality. They are turning it into an open source project. In that way it also could be used in whole or in parts in other countries.

I pair-programmed on CacheFu today with Maciek Dziergwa. See yesterday's entry for a list of things we could work on.

Fixing unittests

Today's cachfu sprinting started out with fixing the failing unittests. In the first hour, we fixed two templates in CMFSquidTool and fixed one failing unittests: a test that failed if you didn't have an apache running on port 80 on your local machine. That's not very unittest-like, but ok. We now catch the error that occurs and the rest of the tests are running ok.

We added a few if/else statements to CacheSetup that conditionally import code for either plone 3.0 or 2.5, this makes sure the product is usable for both. The change was needed because plone3.0/zope2.10 uses a different page template implementation.

We ran into some remaining test problems:

• The rules defined in CacheSetup's genericsetup.xml seem to disappear from the cache configuration, somehow. strange. Only in plone 3.0, though.
• PolicyHTTPCacheManager also has a testing problem: the result is different in 3.0 and 2.5 due to a change to cmf 2.x. There's a strange message in the doctest: Starting with CMF 2.x any caching policies previously registered with the site are automatically applied to CMF Files and Images in their respective index_html, so we get the cache headers even before associating with our PolicyCache. Anyone know what to make of that?

Genericsetup

Genericsetup support was added by Rocky (and it works like a charm), it only missed the last configuration tab. Nice, bite-sized task. We discovered it was missing the data from the last configuration tab because that data comes from a separate tool (the pagecachemanager) in the plone root, not from AT contenttypes like the rest of the cachefu setup machinery. We didn't get it working yet, basically because there was a missing configure.zcml that prevented the testing mechanism from finding the exportimport adapters. Now I'm getting another error. We'll figure it out.

Handy comments we got during the day

During the day, I got a couple of very handy comments from people that I'm writing down here to preserve them.

• Wiggy suggested what is essential for caching is that someone implements a post-publish and pre-publish events for the zope2 and zope3 publishers. Once you have those you can easily reimplement cachefu cleanly. You want post-traversal-pre-publish so you can look for a cached rendering you can immediately serve and you want post-publish so you can tweak response headers and cache the final rendered output. If it got implemented during the sprint, it could potentially end up in zope 2.11, which would mean plone 3.5+.
• Wiggy pointed me at lovely.viewcache, that caches z3 views so you can strike that from your todo list. Zope3 views, no cachefu evilness.
• Stonor said z3c.responseheaders also looks relevant.
• During the lunch, I talked with Balasz and Alec. The conclusion was that adding the event to zope's publisher wasn't that good an idea. The publisher will change anyway (when twisted will get used). And you probably can't send a response back to the browser from within an event. Balasz has some code in a json server that patches zope's publisher. That might be a good stop-gap measure that replaces three monkeypatches with a single one. And it is something that will work in plone 2.5 and 3.0, too.

Plonegroups has unique features for democratic organising and collaborative deciding on texts. The code is owned by a foundation set up by Partecs from Rome: telematics freedom . It aims at democratically run organisations (postal stamp collectors club, democratic parties, etc.).

They're doing something funny with their license. The default current GPL license doesn't cut it because everyone can use and change the code without the need for distributing the changes as long as they run it behind a webserver. This is a known loophole in many open source/free software licenses. They're closing this loophole with some additional limitations ("this code is more free" :-) ).

They want to release a first alpha by the summer, based on plone 3.0.

They have added ways to comment on text. Individual words in the text are highlighted according to the amount of comments those words have gotten. Seems like a pretty good idea. You can suggest changes to texts and people can vote on it.

Rocky asked about the name. The reply was that they're happy to change the name if the plone foundation wants them to. It can go either way: it can be nice marketing or it can be a marketing risk.

He got some questions about the license. Everyone who wants to look at the code has to sign a very short non-disclosure-agreement. If you want to modify the code, you need an additional (and longer) modifier agreement.

Rob Miller mentioned the open planning project which hasn't got the same goal, but is mostly in the same area. So there could be quite a number of areas where they could share code. Open planning releases most of the code under the GPL (or zope's ZPL if needed). Using code from and mixing code with something with a somewhat modified GPL license sounds a bit hard to him. The reply was that this is a known problem of this type of licenses. Another point is that google/yahoo/etc can take all GPL code and use it to strengthen their commercial offerings without needing to give something back. Plonegroups thinks this is something that needs to be fought.

In the end I was left with the impression that they're taking away some of the rights I'm used to in order to give me more rights. But it still smelled like I ended up with less rights. It is probably something that is very hard to explain. I do doubt whether they can generate a lot of traction in this way, as you've got to get people to agree, which takes time: which is often in short supply.

The first sprint day was really a bit of an introduction day. Alec Mitchell started off with some general sprinting tips ("if you work alone, you're doing something wrong") and with a list of subjects people wanted to work on. There were a three big ones (genericsetup+xml, remember+membrane and kss) and some 8 smaller once, according to the preferences given by the participants in a pre-sprint questionaire. The suggestion was to try and join one of the smaller ones instead of swamping the big subjects.

In the end, I got the impression that everyone did swamp the big three :-) I myself was interested in working on one of the small ones: CacheFu improvements. The start of the sprint wasn't very structured, many people were catching up on email and stuff. I attended a good remember plus membrane presentation by Rob Miller. Afterwards I started to try out some plone 3.0 migrations of existing sites.

I had already indicated to Alec that I wanted to do CacheFu sprinting, so he came back to me in the afternoon and looked up his spreadsheet of questionaire results and hooked me up with the other person with CacheFu interest: Andrew Mleczko from Poland. At the dinner, Maciek Dziergwa also joined.

We discussed CacheFu and our various strengths. I was quite worried about the zope3 event stuff that's bound to be necessary, but Andrew thought that to be the easy part, so we're well matched together.

We aim at both plone 2.5 and 3.0 for the sprint's improvements as we want to try to improve things with events and we need 2.5's five version for that at the minimum. Probably we'll mostly work with plone 2.5, but I'm sure we'll check it works with 3.0, too.

Some of the stuff we identified that needs fixing or that could use some renewed attention in Cachefu:

Monkeypatching

CacheFu monkeypatches quite a lot. It patches the catalog to keep track of reindexes so that it can increment an internal counter. This counter is used as an indicator whether the navtree (or so) probably changed. It patches the pagetemplate and the pythonscript mechanism in order to check whether the page needs to be served at all or whether a simple "304 not modified" will suffice. All stuff that could possibly be handled with zope3 events. We hope.

ETags

CacheFu uses ETags a lot, but the build-in list of ETag components is hardcoded. It would be way handier to be able to register new etag components somewhere and have them show up in the CacheFu interface. Adapters? Zope3 utilities? I showed Andrew an example browserview where I generated ETag components. An approach like that would also allow other products to provide custom ETag components.

ETags 2

CacheFu currently doesn't look at modifications to either the default page of a folder or the chosen view on a content object. Changing this doesn't modify the catalog, so that doesn't help either. So we have to add that. (I've got some code lying around for that.)

Zope3 views

CacheFu doesn't cache zope3 views, probably because it doesn't patch the zope3 pagetemplate classes. And there are probably a couple of other reasons. Needs desperate fixing.

Unittests

Almost every CacheFu product gives one or two errors when running the tests, so that'll be a first thing that needs fixing.

Generic setup

Rocky added genericsetup support to CacheFu, which works like a charm. He only missed the last config tab of cachefu (for the memory cache), so that'll need to be added.

I looked at the bundles: there's a 2.5 and a 3.0 bundle for cachefu. The 2.5 one points at 1.0 branches, the 3.0 one at the trunk versions of the products. I don't think the next cachefu version will be 3.0-only, I'm pretty sure that a 2.5+3.0 release is the best plan.

In either case, we created a "sorrento-sprint" bundle at https://svn.plone.org/svn/collective/CacheFu/bundles/cachefu-sorrento that points at "sorrento" branches in each of the products (all copied from trunk) so that we can do our worst during this sprint without messing things up :-)

Rob Miller's introduction on remember

At the start of the plone Sorrento sprint, Rob Miller gave an introduction on membrane/remember.