Check your Plone site's cachefu setup: small improvements can help a lot. I assume you've got a basic cachefu setup pat down already, which only means adding your few custom content types to the "content" or "container" rules and saying you've got squid (and/or apache) in front. You get good results with that: this blog entry helps you pinpoint some extra improvements.

First the one and only indispensable tool: firefox' Livehttpheaders extension. Livehttpheaders shows you the raw http requests that your browser sends out. And it shows you the requests that comes back: are the caching headers that you would expect really there?

A word of warning about firebug for those that use it to monitor files being read (the "net" tab): it also shows the javascript and css files that are parsed. Also when fetched from the local browser cache. I was fooled by this once: "they should not be downloaded, why are they showing up?". Livehttpheaders disabused me of that false notion.

If I only have one screen, I normally open livehttpheaders in firefox' sidebar. Some problems are quicker to spot that way than by switching back and forth between windows. Disadvantage: horizontal scrolling if you want to look at details. The jackpot: dual screens. One for the website, one for livehttpheaders.

Now get cracking. Pick two or three typical pages.

• Load them and look at livehttpheaders: what gets loaded? Load them again: what is loaded now? Should some items (images, stylesheets) be cached? What did you expect? Write down the possible problem spots. If something is fishy, use livehttpheaders to check for faulty if-modified-since headers and so.
• What I find often: I forgot a custom content type. Or a special view is used on a content type, but cachefu doesn't know it should should also cache that view. In that case add the content type and/or the name of the view (or template).
• Double-check by doing a forced reload. Old locally cached copies don't always have the latest header information from your previous tweaks. Make sure you're not chasing long-solved ghosts.
• On the site I tested yesterday, squid wouldn't cache certain things I expected it to cache. Ah! We moved the site from 2.0 to 3.0 and there was an old authentication cookie that the current PAS doesn't use. But squid still looks for it, sees it, treats you as a logged-in users, sends you right over to plone: no squid caching. If something like that happens: purge your domain's cookies.
• Can you relax the caching somewhat? Cachefu's defaults are - of course - pretty conservative. Take images. (Images in your site, not filesystem images). Normally they are cached by squid; the browser is told to always check for newer versions. Most-times a quick "304 not modified" is given back. But still: a request for every image. For the site I optimized yesterday, I changed the used header set from "cache in squid for 24 hours" to "cache it in the browser for an hour". You give up potential freshness, but the images in this case would rarely change anyway. As the same pages (with quite some images) would be requested over and over again, this slashed 90% of the number of requests. Even if they would have given 304's, every omitted request still counts, especially in those numbers.

There you have it. A couple of hints to help you get started with basic CacheFu tweaking. You can get a lot of mileage out of it, without having to write custom rules or header sets. Though that is fun and satisfying, too.

Handouts are at http://plonebootcamps.com/resources/high-performance-plone/view, so I'm not going to write anything down. If you're in need of a speedup of your plone site, watch last year's talk (quite a lot is the same, only info on the latest cachefu version and on memoize is missing). I do guess that we'll get a video of this year's talk, too. Here, I'll only highlight a few things.

Plone is pretty efficient. It just does a lot, like checking for authentication and finding all resources that are needed, creating the dynamic navigation, grabbing the right CSS bits. Most of these things cannot be made really faster. So the basic strategy is not to make plone faster, but to make plone do less. The everything-is-fresh default setting of Plone is the right choice, of course, as it would otherwise not meet your expectations and it would lead to loads of bugreports.

Sweet spot for performance: 2 zope threads at a time (instead of the default of 4).

Use zeo ALL THE TIME. Also on development boxes.

Especially on linux, default hardware parameters are often very conservative. hdparm for instance. Linux distributions are conservative as they want to run on all hardware. There's some serious opportunity for improvements here.

Think about caching right at the start when coding. Add a few ramcaches with obvious names (cache_per_user_for_5_minutes) and set the cache time to 0 seconds during development. RAMcaches: they gave discover magazine a 20-fold increase, which was more than they got out of cachefu and squid.

Question: aren't RAMcaches slowly going away? All those browser views? Python scripts shouldn't be used anymore. Answer: Joel loves python scripts. It was only recently that the new view classes and so were cachable with memoize. Before that, ramcached python scripts out-performed browser views by order of magnitudes. And ramcache have excellent security.

Memoize: new caching mechanism for zope3. Works great, completely tweakable. Drawback: not tweakable by the customizer/integrator. He expects that to change, though.

Useful strategy: focus on the tiny: microcaching is very helpful. 5 seconds to 5 minutes is often unlikely to be noticed. It can reduce work with 500x.

Plone can be made really fast. With ramcache, memoize, cachefu and apache/squid/varnish in front, it can be blazingly fast. (Reinout: That's also my own experience: you can make Plone a helluvalot faster). Cachefu is in need of more developers and especially documentation writers. Plone's caching story needs to get more well-known. Joel especially asked for more bloggers :-)

Technorati tag: ploneconf2007

Vincenzo: plone 3.0 ships with a new workflow. They've worked on the workflow transitions, also they fixed most of the tests.

Balasz: KSS! "Thanks to everyone who participated." This was a much bigger success than he hoped for, as a large amount of issues has been solved. Another improvement: browser-based tests with selenium. They instituted a policy at this sprint to add selenium tests for every issue that they tried to fix.

A bummer for him is that Kupu broke IE support. Kupu also uses the sarissa javascript library, but modify something in it... but keep the name. So KSS breaks. Grrrr.

Nate: plone4artists. They focused mostly on the audio product, they barely touched the video product. They did add the possibility to add a regular ATLink to a youtube video to your site which is then transformed in to a nice view. Load of extra tests. Also added: searching via the catalog for artists, track names, etc. Internationalisation (Japanese, Portuguese). Metadata you add to your audio contenttype ends up in the mp3 itself. And Jonathan made a great screencast.

Stefan: xml export/import. Work on exporting of references: they now simply pickle the references (tip from Daniel). They have some tests for importing, but it hasn't been implemented yet.

Anton: they looked at file/image widgets for formlib. They originally looked at p4a's audio/video, but made it more generic afterwards.

Rob: genericsetup and remember. Work started at the BBQ sprint was continued. Adding metadata (like versions) to genericsetup data, working towards supporting upgrading and migration steps and so. Some 75% seems finished. Membrane/remember had a problem: it always assumed that a content object contained the authentication data. This works OK unless you want to use LDAP or so. So this assumption had to be removed: they didn't actually get very far with this.

Kapil: he basically worked solo and continually helped other people. He himself did some work on plonecommerce, for which he needed something quick and very simple to attach some properties to a member (see friday's summary ). He's got it mostly working.

Reinout: cachefu. Getting the tests to run on both 2.5 and 3.0. Finalising the genericsetup support and adding unittests for it. Also a lot of code research, looking at zope 3, lovely.viewcache , getting great ideas from wiggy, philikon and optilude.

Belgian Bug Busters: loads of bug fixing! Often a lot of bugs were hiding behind an original one.

Ramon and Alec fixed security issues for the last two days.

Friday was pretty relaxed, as some 3/4 of the participants took the opportunity to visit the ruins of Pompeii in the afternoon. Did you know that Nate discovered proof that the Romans used Plone?

On thursday, we took the group photo. It is a very pleasant sprint, as is visible from the photo. Some 50 or 60 people turned up, with quite a number of first-time sprinters amongst them, which is great news for Plone's future. I haven't yet heard any negative sound about the sprint (apart from the speed and reliability of the internet connection, ah well). The hotel is good. The evening dinner is pretty good. The staff is very friendly. The bartender is pleasantly nuts.

I really should start to bug some people tomorrow moning on the various groups' sprint progress, otherwise I would only chat on about cachefu :-) As I haven't done that yet, you'll all forgive me that I mention that we've integrated the PageCacheManager settings into CacheFu's genericsetup profile. This brings the list of completed items from the original list to two: fixing the unittests (as far as currently practical, there are some plone 3.0 issues left) and finalising the genericsetup support.

I and Machiek did most of the work while pair programming. It was a first for me to have two full days of pair programming. I did some of the work on my own late in the evening, but that's probably OK. I'm enthousiastic about it. You both learn things. You both make simple mistakes that the other one catches immediately. You can discuss ways to approach the problem. You transfer knowledge. You see each other's way of working.

I got some more very valuable input from Philip and Wichert. In short: using events for hacking into the publishing process might be possible after all. Core to this is to raise an exception if cachefu detects that a "304 not modified" is in order of if a cached version of the page is available. There's code available (plone 3.0 has it build-in) that monkeypatches zope's publisher to look up views for errors instead of blindly spewing out an error page. Wow! The 3.0 code can be used for 2.5, too, so that looks promissing.

This is a really big change. For CacheFu. Getting this to work would be a great result of this sprint. We've got some days left to get it to work. Maciek, are you up to it? :-) *If there are others that want to help work on this: feel free. CacheFu is pretty important and there's a limit to what Maciek and me can accomplish in this sprint. There are also still the other useful tasks left... :-)

One other remark: I'm extremely happy with instancemanager . It is a tool for managing instances: creating them, backing them up, installing products (symlink, svn bundle, tgz), running tests, etcetera. All based on a reasonably simple config file. When I started this sprint, I had already a plone 3.0 config (see instancemanager for your plone 3.0 ) and a plain plone 2.5 config. After creating the "sorrento" bundle of cachefu, getting that installed in an instance took about a minute: copy plone25.py (the 2.5 instancemanager config) to cachefu.py, change the portnumber, add the sorrento bundle to the config, run instancemanager cachefu fresh. Done. Same for cachefu3.py, to get cachefu running inside a plone 3.0 site. No (in my eyes) elaborate in-location downloads of all software in a separate location as ploneout seems to do it. Just effectively a "mkzopeinstance" in a location of your choice while grabbing the bundles from your already-downloaded svn locations. Real quick and convinient.

Now, off to bed.

I pair-programmed on CacheFu today with Maciek Dziergwa. See yesterday's entry for a list of things we could work on.

Fixing unittests

Today's cachfu sprinting started out with fixing the failing unittests. In the first hour, we fixed two templates in CMFSquidTool and fixed one failing unittests: a test that failed if you didn't have an apache running on port 80 on your local machine. That's not very unittest-like, but ok. We now catch the error that occurs and the rest of the tests are running ok.

We added a few if/else statements to CacheSetup that conditionally import code for either plone 3.0 or 2.5, this makes sure the product is usable for both. The change was needed because plone3.0/zope2.10 uses a different page template implementation.

We ran into some remaining test problems:

• The rules defined in CacheSetup's genericsetup.xml seem to disappear from the cache configuration, somehow. strange. Only in plone 3.0, though.
• PolicyHTTPCacheManager also has a testing problem: the result is different in 3.0 and 2.5 due to a change to cmf 2.x. There's a strange message in the doctest: Starting with CMF 2.x any caching policies previously registered with the site are automatically applied to CMF Files and Images in their respective index_html, so we get the cache headers even before associating with our PolicyCache. Anyone know what to make of that?

Genericsetup

Genericsetup support was added by Rocky (and it works like a charm), it only missed the last configuration tab. Nice, bite-sized task. We discovered it was missing the data from the last configuration tab because that data comes from a separate tool (the pagecachemanager) in the plone root, not from AT contenttypes like the rest of the cachefu setup machinery. We didn't get it working yet, basically because there was a missing configure.zcml that prevented the testing mechanism from finding the exportimport adapters. Now I'm getting another error. We'll figure it out.

Handy comments we got during the day

During the day, I got a couple of very handy comments from people that I'm writing down here to preserve them.

• Wiggy suggested what is essential for caching is that someone implements a post-publish and pre-publish events for the zope2 and zope3 publishers. Once you have those you can easily reimplement cachefu cleanly. You want post-traversal-pre-publish so you can look for a cached rendering you can immediately serve and you want post-publish so you can tweak response headers and cache the final rendered output. If it got implemented during the sprint, it could potentially end up in zope 2.11, which would mean plone 3.5+.
• Wiggy pointed me at lovely.viewcache, that caches z3 views so you can strike that from your todo list. Zope3 views, no cachefu evilness.
• Stonor said z3c.responseheaders also looks relevant.
• During the lunch, I talked with Balasz and Alec. The conclusion was that adding the event to zope's publisher wasn't that good an idea. The publisher will change anyway (when twisted will get used). And you probably can't send a response back to the browser from within an event. Balasz has some code in a json server that patches zope's publisher. That might be a good stop-gap measure that replaces three monkeypatches with a single one. And it is something that will work in plone 2.5 and 3.0, too.

The first sprint day was really a bit of an introduction day. Alec Mitchell started off with some general sprinting tips ("if you work alone, you're doing something wrong") and with a list of subjects people wanted to work on. There were a three big ones (genericsetup+xml, remember+membrane and kss) and some 8 smaller once, according to the preferences given by the participants in a pre-sprint questionaire. The suggestion was to try and join one of the smaller ones instead of swamping the big subjects.

In the end, I got the impression that everyone did swamp the big three :-) I myself was interested in working on one of the small ones: CacheFu improvements. The start of the sprint wasn't very structured, many people were catching up on email and stuff. I attended a good remember plus membrane presentation by Rob Miller. Afterwards I started to try out some plone 3.0 migrations of existing sites.

I had already indicated to Alec that I wanted to do CacheFu sprinting, so he came back to me in the afternoon and looked up his spreadsheet of questionaire results and hooked me up with the other person with CacheFu interest: Andrew Mleczko from Poland. At the dinner, Maciek Dziergwa also joined.

We discussed CacheFu and our various strengths. I was quite worried about the zope3 event stuff that's bound to be necessary, but Andrew thought that to be the easy part, so we're well matched together.

We aim at both plone 2.5 and 3.0 for the sprint's improvements as we want to try to improve things with events and we need 2.5's five version for that at the minimum. Probably we'll mostly work with plone 2.5, but I'm sure we'll check it works with 3.0, too.

Some of the stuff we identified that needs fixing or that could use some renewed attention in Cachefu:

Monkeypatching

CacheFu monkeypatches quite a lot. It patches the catalog to keep track of reindexes so that it can increment an internal counter. This counter is used as an indicator whether the navtree (or so) probably changed. It patches the pagetemplate and the pythonscript mechanism in order to check whether the page needs to be served at all or whether a simple "304 not modified" will suffice. All stuff that could possibly be handled with zope3 events. We hope.

ETags

CacheFu uses ETags a lot, but the build-in list of ETag components is hardcoded. It would be way handier to be able to register new etag components somewhere and have them show up in the CacheFu interface. Adapters? Zope3 utilities? I showed Andrew an example browserview where I generated ETag components. An approach like that would also allow other products to provide custom ETag components.

ETags 2

CacheFu currently doesn't look at modifications to either the default page of a folder or the chosen view on a content object. Changing this doesn't modify the catalog, so that doesn't help either. So we have to add that. (I've got some code lying around for that.)

Zope3 views

CacheFu doesn't cache zope3 views, probably because it doesn't patch the zope3 pagetemplate classes. And there are probably a couple of other reasons. Needs desperate fixing.

Unittests

Almost every CacheFu product gives one or two errors when running the tests, so that'll be a first thing that needs fixing.

Generic setup

Rocky added genericsetup support to CacheFu, which works like a charm. He only missed the last config tab of cachefu (for the memory cache), so that'll need to be added.

I looked at the bundles: there's a 2.5 and a 3.0 bundle for cachefu. The 2.5 one points at 1.0 branches, the 3.0 one at the trunk versions of the products. I don't think the next cachefu version will be 3.0-only, I'm pretty sure that a 2.5+3.0 release is the best plan.

In either case, we created a "sorrento-sprint" bundle at https://svn.plone.org/svn/collective/CacheFu/bundles/cachefu-sorrento that points at "sorrento" branches in each of the products (all copied from trunk) so that we can do our worst during this sprint without messing things up :-)