Sunday's sprint wrapup! A nice day of sprinting. Everyone was busy, including all the new first-time sprinters. We had a great round of pastis before the wrapup thanks to AtReal :-) I really enjoyed the Good Stuff :-)

zodb benchmarking

Made some observations. Tried adding lots of simple objects in various batch sizes to get good metrics. Increasing the batch size made it detoriate exponentially. If you stay with a memory-only zodb you can get literally millions of insertions per minute. They suspected a big influence of the mac disk that they did the test on. They're going to continue tomorrow with plone content types, concurrent commits, etc. It also turned out that it is important to pack your Data.fs., as after adding 200k of objects some 30 percent of the database size was taken up with transaction info.

plone survey

Smallest team, 2.5 person. They can have both a 2.5 and 3.0 release out tomorrow. They're still working on KSS stuff to make it more usable, though. Fun to work on. Did a lot of internationalization work.

PrimaGIS

New buildout. Refactoring. Tilecache enable now, really fast now compared to what it is now. New demo creation on the way.

KSS

jquery problems: bugs in the jquery demo code. jQuery events started (success). "Back" button works better now. They tried working on "push" and have a working demo: it really worked, though I don't know how they managed it. Great great stuff.

Translation group

Lots of stuff working. Most of the fields. Hope to have most of it working tomorrow.

sqlalchemy

Bad day. Fighting the stack, including zopetestcase. Ported code to collective.lead (great as it is in Martin's book).

AT rest

Didn't get finished. Saw how to do it. Great help from Ramon: checked in plone.rest tonight. Also fighting against the stack.

getpaid

Completely converted to genericsetup. Got two bugs fixed. Got some 40 tests added, so ready for tomorrow's refactoring. Added some screencasts: they're going to market the hell out of this puppy.

KSS part 2

They've added Django support to kss! Within one day. (Big applause)

workflow tests

They've found out a lot of bugs in the basic product. Worked a lot on tests. Got them running. Now testing workflows is way easier. (Note to self

workflow

State workflow and activity workflow. Trying to switch between the two. Lots of work, but good to have the current results.

Functional tests

They have two new scenarios for generic reusable testing. And they now have a tutorial. Improved comments. Code cleanup. Buildout for selenium.

Vice

Less work for customizers needed, just override the adapter. Should make it easier. Functional tests have been greatly improved. Fixed some basic datetime tests. Polishing of the user interface.

New themes

ZopeSkel was fixed for windows. They worked on concrete designs and Spliter demoed them. They had a great working day. They thought the new guys to create good non-messy themes. And they created documentation for this. They even have disability support guys. The examples are at openplans , for example capri theme , napoli mock . Spliter seems to be doing a hell of a job of shepherding everyone towards good designs. One comment: document your CSS files lavishly with comments, they are filtered out by portal_css.

Commenting

They were stripped down to 2 people today and went over the generic concepts, which turned out to be a good idea. They coded some and can add comments programmatically. They also use parts of easycommenting for KSS and templates. Nothing is really finished, but they have a good foundation. Tomster is looking forward to continuing with this.

Archgenxml

It works. Plone 3.0 compatible, trunk is now safe again. Improvements to the generated tests.

Nuplone

Navigation tree improvements, for instance the whole active section now has a background. The livesearch has great css now. There's also good multi-resolution support, too, so it degrades quite gracefully. When changing position due to resizing of the window, the colors of for instance the calender portlet change. Wild applause because of that feature. Looked like the darkest magic that you can imagine. :-)

plone4artists

Continued with the tests and got most of the audio tests fixed. Audio was released today, Rocky released the video later this day. More translations added, the framework is ready for more translations. The p4a openplans page details the translation mechanism. Calendering added a nice day view, tomorrow they hope to improve the week view. Tomorrow they hope to get it over to plone4artists for a release. Something usable is on the horizon!

Extreme management

We worked on moving portlets to plone 3 portlets. Worked on the workflow (generic setup). Some KSS added for body texts and descriptions. Tried to get macros to work with KSS, but started to work on viewlets instead as that works better. Worked on splitting out bookings into a separate product, that'll be soon into the collective. Reusable for others!

blobfile support

They got conversion of regular files to blobs working. Demo failed as safari crashed :-) He assured it worked :-)

Schema extender

Lots of doc improvements in the doctest. Looks like excellent documentation. There's an additional adapter for modifying existing schemas. I questioned the difference with generic ISchema overriding: they answer was that this product allows you to plug those schema modifications in. No conflicts. Multiple products can do it at the same time.

Technorati tag: ploneconf2007

A real quick writeup of a real quick "what did we do today" session where everyone presented what they did. (Hard to write down so quickly, aargh! Probably made a few mistakes.)

getpaid

Cleanup of tests, new translations, more genericsetup. Excellent way to get everyone

kss

New tutorial which reflects plone 3.0-style development. People worked on resurrecting the event-push system. jQuery work. Django integration for kss: a new framework that they support.

sqlalchemy

Fixed copy/paste bug. New buildout recipe that makes it easier to set up. Lots of discussion about catalog integration. Better ideas about metadata fetching.

extreme management

Exchanged information about how we all handle projects in different organizations. Figuring out how to handle both. Slowly getting there code-wise. Backporting a KSS experiment to trunk. Looked at

ploneGIS

User documentation. A new buildout. Work on a new demo product. Work on layers.

syndication

Some success, some snags. A big timezone bug bit them again, but they made good progress. They can now tag an alpha, mostly due to their remote sprinter.

archgenxml

Polishing, bugfixing, finding a bug in archetypes itself (and fixing it). Documentation is now much better. Probably archgenxml works nice for 3.0 tomorrow. Ideas about breaking the dependency on archetypes.

Functional testing framework

Documentation update. Debugging code improved. Additional component for search testing. Started a buildout for selenium integration.

New themes for plone

Mostly setup today, making sure everyone the whole big team got set up. Worked out what should be handled in a theme. Their new theme has newtheme.pizza as a code name.

Commenting

Team of four, two architects, two coders. Figuring out how to handle comment spam filtering. New project page on openplans. They made a buildout for plone.commenting. They have their first tests running.

plone4artists

Tagging products for a 1.0 release. Fixing the last bugs and getting everything ready. Started on the translations. Working on plone 3.0 support. Work on blobfile support with especially nice handling for plone. The calender team has 10 bugs that need ironing out, 4 of them are done now. And they added a 1-minute screencast. They hope to get more of them done.

plone.relations

Work was done especially on plone.app.relations. Trying to tie into user objects. Adapter experiments and teaching people stuff.

Project name that I missed

Just getting started. Had lunch.

Blobfile support

Figuring how to best integrate it without breaking things. Did groundwork like adding a contenttype and a buildout. They hope to write migration scripts tomorrow morning for ATFile and ATImage. They idea is to add an add-on package for plone 3.0 which will hopefully end up in the next major release. When the 3.0 product is done, they plan a 2.5 product.

Internationalization

Trying to allow customization of translations through the web so that you can make small customizations.

Workflow

Unittesting for workflows basically sucks. They tried to work with Matt Hamilton's (?) idea for using a .csv file for testing.

archetypes schema extender

It is now out of proof-of-concept. Seems to be really working. Improving the interface and expanding the doctest. Will be the way for schema subtyping.

plonegov

They were too hungry to wait for this wrapup :-) Plonegov will merge the three projects that deal with openoffice integration.

Vidar started out by looking at the original genesis code/choices (like using Coral) and pretty rapidly did a good research on what else is available. He found some good tools, mostly based around eclipse and started hacking like there's no tomorrow. Just before my holiday (I just returned) I strongly suggested him not to add any more code but to package it all up nicely (as installing eclipse has been hell for me). Ease of installation is more important than adding one or two more features

The result blew me away. See the article on his weblog. Just one call to easy_install and it is installed. And it actually uses paster and its existing plone3 product skeleton.

Vidar worked hard and didn't need much outside encouragement. Smart, likes to dive in deep. Asks for help/advice when needed and listens to the sensible parts of the advice that people give him :-) He was a good fit with my style of mentoring (I've mentored a few students while I was a PhD student).

One possible snag regarding the original goal: the generator uses a special text format. Though, if you install eclipse, you can get a graphical editor working. And he's indicated that a transformation from UML is possible. So it is not 100% aligned with the original goal of genesis. What Vidar's done, though, looks like a way better basis than either Coral or the existing agx. And it bloody well works! Personally, I like the textual format, as it is versionable in subversion, for instance.

So: I'm real happy with how it all turned out! Especially the "easy_install" bit blew me away when I saw it last night. Vidar'll make more documentation in the days to come, so watch his blog when interested. And as he's done this gsoc project as part of his MSc study, he's finishing off a full report this week so there'll be enough documentation :-)

I'm mentoring Vidar Svansson for the Google summer of code. He's going to look at "genesis" again. Genesis is the name we use for a rewrite-archgenxml-from-scratch effort that hasn't gotten off the ground yet. One sprint, in which we selected coral as the underlying UML library, that's it. No real work done since.

Vidar's going to take a fresh look at it from a research perspective: he's doing it for his MSc thesis.

The gsoc is supposed to start next week, but Vidar has already started. Mostly reading up on zope3 and grok, trying to install coral and going to a code generation conference. I asked him a few days ago if he planned to be finished before the project started :-)

Coral is a pain to install. It is possible on linux if you use a pre-made rpm. Compiling on a mac: forget it. There's no public svn repository, so you have to use the tgz and figure out the changes since the last versions yourself. It sounds a bit risky.

At the conference, he discovered some new eclipse-based tool that looked OK to him, so he'll try things out with it. He says he'll be able to generate code for a simple case in a week or so...

End results that he/we aims for? A combination of actual code and scientific research. A theory on how to best model a plone web application. Code that shows it can be done :-)

In the recent weeks, I got quite a number of questions about archgenxml and plone 3.0. In summary:

• The good news is that archgenxml-generated products still pretty much work OK on plone 3.0. Archetypes itself didn't change that much and the majority of what archgenxml does is to feed info to archetypes.
• The number one problem on plone 3.0: the workflow. Archgenxml generates them the oldfashioned way: workflow files in the Extensions directory. That way of working has been deprecated with plone 3.0, workflows ought to be handled with genericsetup now: in profiles/default/workflow.xml.
• There is a workaround: install the product in plone 2.5, export the workflow.xml there and add it by hand to your product. Urgh.

Improving archgenxml has been in the planning for some time. Since that blog post in september 2006, progress has been slow, but in the last 2 weeks I've again put in quite some effort and I'm again happy coding along in good old train-driven-development commuter work :-) The stuff is all done on the svn trunk version. Some highlights of what's been done:

• Archgenxml is now something you install with python setup.py.
• Archgenxml uses the zope3 component architecture internally to handle things. At least, I've got a few interfaces, I'm using one adapter and one utility. It already makes things clearer. The choice for zope3 is a good one, I can tell. See one of the next blog posts.
• One horrible hack has been done to be able to import said zope3 machinery. I posted earlier about setup.py problems. Basically depending on zope.interface, zope.component and the rest of the stack almost guarantees that you mess up your python's site packages with versions that are incompatible with other zopes (like, your running instances...) that you're running with that python. You'd have to isolate bloody everything with workingenv to get this working. So start up archgenxml and the hack will introduce itself and tell you how to handle it :-)
• All the new code I wrote has doctests. And I try to add doctests for existing code that I copy/paste to new files. So hopefully this means that the quality will slowly rise.
• A completely empty genericsetup profile is generated and installed by the quickinstaller. Anything you put in there is in mortal danger as it'll probably get plowed under once I start adding genericsetup files with archgenxml.

Well, there's the current status! Ongoing work.

Joel Spolsky reviews a book about Chandler . I don't know if he's right, but the last time I looked at chandler they still seemed to be in the "early preview" stage. And I thought I've seen presentations on chandler's architecture on the last 4 europython conferences or so that I attended.

Anyway, I want to quote something from the end of his article:

The Chandler team also overestimated how much help they would get from volunteers. Open source doesn't quite work like that. It's really good at implementing copycat features, because there's a spec to work from: the implementation you're copying. It's really good at Itch Scratching features. I need a command line argument for EBCDIC, so I'll add it and send in the code. But when you have an app that doesn't do anything yet, nobody finds it itchy. They're not using it. So you don't get volunteers. Almost everyone on the Chandler dev team got paid.

That's one thing I noticed after I started instancemanager (see also some weblog entries). The first version was already pretty usable and a few people started to use it. And found a small thing lacking here and there. Something that lacks is an itch that needs scratching. And instancemanager provided a framework for hanging your personally-crafted itch-scratcher on :-)

Somehow it turned out to be a good place to attach all sorts of tiny helper scripts and handy additional functionality. The same with archgenxml, of course. It started out as a handy way to generate archetypes contenttypes, but at a certain moment somebody added a way to generate workflow with it. And then someone added CMFMember support. Lately Jean Jordaan added "remember" support. You can add little things here and there. Stuff like that turns a product into a small open source community.

Thanks to philikon, who pointed me to the zope3 distribution page, I've managed to get the zope3 component architecture stuff included in my setup.py. Included meaning that the zope3 stuff gets downloaded when you try to install the product (archgenxml 1.6 in this case) with python setup.py install.

Setuptools tells you how to declare dependencies. In the end I had to include the following in the setup.py:

...
install_requires="""
zope.interface
zope.component
zope.testing
""",
dependency_links = [
  'http://download.zope.org/distribution/'
  ],
...

Update: not really, it turns out. It works like a charm, until you start a zope 2.9.4 instance. There is some mismatch in the versions of the zope components between 2.9.4 and what distutils downloads. And distutils' packages are placed inside python itself and that wins from the packages bundles with zope 2.9.4...

I guess the problem is in zope.interface. The rest of the packages are 3.2, zope.interface has a 3.3 egg. There's a 3.2 .tgz, but if you set the dependency to the 3.2 version, setuptools complains about an absolute path in the egg it tries to make out of that .tgz.

There's an archgenxml idea that I'd like to pop onto those interested to get some feedback.

Basic premise: Give filesystem development more prominence.

Some issues that could bug you if you work with the code that archgenxml generates (or even just have to look at it):

• Loads of protected sections in every file that clutter it up.
• Custom code has to be put into those protected sections or it is removed upon re-generation.
• Well, apart from methods, but archgenxml is prone to shuffling these around so that the file is modified in subversion though no real changes were made.
• Most class-level customisations have to be done through tagged values. I mean the icon, the archetype_name, etc. These are often easier to change right inside the file.

What I hope is doable and acceptable: to give archgenxml the ability to leave the sourcefile alone as much as possible.

• No more protected sections, as archgenxml won't remove exisiting code.
• You can modify the archetype_name and friends, as archgenxml won't overwrite existing values (unless you specify them in your model).
• Archgenxml checks the file for import statements and only adds the ones that are missing.

This does mean that you have to do more housekeeping yourself. If you remove some dependency, you'll have to remove the import statement yourself, for instance.

Archgenxml will still generate contenttypes with good defaults, but it will be much less aggressive in maintaining its iron order inside its files. This might not be everybody's piece of cake. It might be horribly undoable. It might have to be a config setting.

Feedback! reinout@vanrees.org, your own blog, mailinglist, irc, whatever.

I've just cut a 1.6 branch of archgenxml, which will be the next version of archgenxml - 1.5.0 has been released last saturday. Time for some cleanup and modernisation!

• Use distutils and easy_install: python setup.py install and easy_install archgenxml. Done :-) Warning: that easy_install one is the version I'm hacking and slashing on now.

• Putting zope3 view classes or adapters in front of every code-generating dtml template. I'm expecting a lot from this relatively simple measure.

  The reason? It will allow us to extract some functionality out of the dtml templates. That in itself is not such a big deal. The big deal is that we can remove code from the XMIparser and the big ArchetypeGenerator.py file that is there just for presentation. There are a lot of special cases and if..else statements scattered all over the place.

  Slowly extracting all the non-core stuff into the view classes and adapters will make the XMI parser and the main generator way more accessible, moldable and improvable.

• Deprecating tagged values, stereotypes and commandline options. There are quite a lot of double items, almost unused options and deprecated stereotypes. This time we'll clear them out. This will mean some breakage here and there, but in a limited way that is perfectly acceptable.

  Example: cmfmember isn't there anymore in plone 2.5, so we can zap two stereotypes and five tagged values.

  Many generation options could be set on the command line, in a config file and in tagged values. We'll restrict many of them to just tagged values, which is logical as they can change throughout the model (like "author"). Work has begun, I've zapped 30% of the tagged values.

Apart from cleanup, some plone2.5 goodies will see the light. View class generation, that sort of thing.

Request for help. Apart from help getting 1.6 off the ground, 1.5.1 will be a bugfix release for 1.5.0. There's a good number of bugs in the issue tracker, I'm slowly working my way through them. Rejecting some, fixing some instantly, slotting some for 1.5.1, some for 1.6. I can't do all that work on my own, neither can Jens or Phil or Godefroid. So: we need more developers and more helpers with the issue tracker.

Fixing bugs is a great way to start contributing! It would help me a lot to have fewer 1.5 bugs to worry about. People are bugged by it. So they should be eradicated, mashed, and exterminated like the misbegotten horse droppings they are!

I just got the europython 2006 program and my talk is in: Generating content types and workflow with ArchGenXML , hurray! Monday morning, right before lunch, which is a good spot as far as I'm concerned.

For many people, ArchGenXML is the most attractive way to get started with Plone development. Generating your content types from a UML class diagram is easy and fast, especially as ArchGenXML sets up all the "bookkeeping code" for you: the __init__.py, the Install.py, etc. The presentation will cover this and I'll also show how to modify the generated code to your special needs.

ArchGenXML is great at generating a complete workflow out of UML state diagrams. You get good code that hardly ever needs modification from a UML diagram that you can readily show to your customer - and he'll be able to understand it (mostly). I'll show how to do this in the presentation.

Depending on the improvements made on ArchGenXML I hope to show our support for zope3 goodies in plone 2.5.

Oh, and allow me a brief recommendation of eXtremeManagement of projects with Plone that my brother Maurits gives on monday right after lunch :-)