Planet Schooltool

Converting a gnarly SVN repository to GIT: success!

2012-02-20T20:17:00+00:00

I've received more feedback about my last night's post on gnarly svn to git migration than I've expected. Thanks to that feedback (and, mostly, Raffaele Salmaso for doing almost all the work and emailing the result to me) eazysvn is now on GitHub.

The rest of this post will describe the conversion (and verification) in detail, because if I ever need to do this again, I do not want to start from scratch.

Continue reading this post...

Converting a gnarly SVN repository to GIT: FAIL.

2012-02-19T21:13:00+00:00

eazysvn lives in a Subversion repository. I want to bring it (kicking and screaming) into the 21st century and put it on Github.

git-svn is unsuitable for the conversion, because in revision 50 I moved / to /trunk and added the traditional /tags and /branches. With git-svn I either get one third of the history that ignores everything before the layout switch, or I get directories named 'trunk', 'tags' and 'branches'.

Then I thought maybe hg would be smarter about the conversion, and then I could use hg-fast-export to convert hg to git. I enabled the hgsubversion extension:

$ sudo apt-get install hgsubversion
$ echo '[extensions'] >> ~/.hgrc
$ echo 'hgsubversion =' >> ~/.hgrc

(blog posts like this are a good reason why hg ought to steal the 'git config --global foo.bar=baz' syntax from git).

Then I converted the svn repository to Mercurial:

$ hg clone svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-hg

hg log -p ~~confirmed that hg handled the conversion nicely~~ looked right-ish at first glance, except the author information was nonsensical. To fix that:

$ rm -rf eazysvn-hg
$ echo 'mg = Marius Gedminas <marius@gedmin.as>' > AUTHORS
$ hg clone svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-hg -A AUTHORS

Unfortunately, a closer look at hg log now shows that two thirds of the history is lost: hg ignored everything before the layout restructuring, despite printing the log messages of those revisions as it went about the conversion. *sigh*.

Dear lazyweb, surely svn layout reorganization can't be such a rare thing that no tools in existence support it? What should I try next?

P.S. I also tried Bazaar, to see what it would do:

$ bzr branch svn+ssh://fridge/home/mg/svn/eazysvn eazysvn-bzr
Repository with UUID 4fc293c4-4eed-0310-a01a-b4ad72f90fad at svn+ssh://fridge/home/mg/svn/eazysvn contains fewer revisions than cache. This either means that this repository contains an out of date mirror of another repository (harmless), or that the UUID is being used for two different Subversion repositories (potential repository corruption).
bzr: ERROR: exceptions.KeyError: 'missing revision paths for 78'

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 946, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1150, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 699, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 721, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 1263, in run
    from_location)
  File "/usr/lib/python2.7/dist-packages/bzrlib/bzrdir.py", line 919, in open_tree_or_branch
    return bzrdir._get_tree_branch()
  File "/usr/lib/python2.7/dist-packages/bzrlib/controldir.py", line 410, in _get_tree_branch
    branch = self.open_branch(name=name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/remote.py", line 420, in open_branch
    branch_path = self._determine_relpath(name)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/remote.py", line 369, in _determine_relpath
    layout = repos.get_layout()
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 701, in get_layout
    return self.get_layout_source()[0]
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 720, in get_layout_source
    self._find_guessed_layout(self.get_config())
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/repository.py", line 743, in _find_guessed_layout
    revnum, self._hinted_branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 143, in repository_guess_layout
    return logwalker_guess_layout(repository._log, revnum, branch_path=branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 149, in logwalker_guess_layout
    logwalker.iter_changes(None, revnum, max(0, revnum-GUESS_SAMPLE_SIZE)), revnum, branch_path)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/layout/guess.py", line 104, in guess_layout_from_history
    for (revpaths, revnum, revprops) in changed_paths:
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/logwalker.py", line 60, in iter_all_changes
    revpaths = get_revision_paths(revnum)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/logwalker.py", line 295, in get_revision_paths
    return self.cache.get_revision_paths(revnum)
  File "/usr/lib/python2.7/dist-packages/bzrlib/plugins/svn/cache/tdbcache.py", line 187, in get_revision_paths
    raise KeyError("missing revision paths for %d" % revnum)
KeyError: 'missing revision paths for 78'

You can report this problem to Bazaar's developers by running
    apport-bug /var/crash/bzr.1000.2012-02-19T22:12.crash
if a bug-reporting window does not automatically appear.

N9 Hackathon in Vienna

2011-10-12T11:25:00+00:00

Last weekend I attended the N9 Hackathon in Vienna. Nokia kindly sponsored all food and accommodation costs and, at the very end, surprised me with an entirely unexpected gift of a N9 phone.

Vienna: great transportation system, delicious food (either that, or I was always extremely hungry when I ate), huge portions, restaurants open until midnight. Shame I didn't have time to see the city itself.

The N9 is a gorgeous phone; much more so in real life than in pictures.

After some hassle upgrading the Qt SDK (the provided upgrade tool managed to somehow remove the Qt Creator IDE while purporting to upgrade it; I had to reinstall the entire SDK) and flashing my N950 with Beta 2 firmware (Qt SDK 1.1.3 produces apps incompatible with the old Beta 1 firmware of the N950) I started prototyping a time tracking application in QML, while learning QML and the N9 design guidelines at the same time.

Converting the pretty pictures into QML was harder than I expected, but at the end of the second day I had something that looked like a native N9 application.

Most useful reference pages were:

The QML syntax introduction (which felt incomplete, but was almost adequate in the end).
The list of Harmattan-specific QML components.
The list of standard QML components.
The UI building blocks pages mentioned above (pretty pictures! pretty colours! I like shiny things!).
Harmattan Qt Components User Interface Guidelines: pixel and font sizes of the standard UI elements. (Ignore the "import UIConstants.js" red herrings; it's an internal thing apparently, and you can't use it directly from your 3rd-party apps. Unless you find and copy UIConstants.js into your project, after figuring out if the licence allows it, which seemed too much of a hassle for me to even start. So I hardcoded all the numbers directly, like a bad programmer who doesn't know about constants.)
The TutorialApplication sample, finding the sources of which was unexpectedly difficult -- a straight git clone of the qt-components repository gives you something too recent to run with the older qt-components version on the N9. I ended up using apt-get source qt-components in Scratchbox to download the source tarball of version 1.0~git20110525-1+0m6. Look in qt-components/examples/meego/.

Finally, workflow. QML is parsed at run time (application startup time, unless you delay the loading), which means no recompilation ought to be required to make changes, which means short development feedback cycles ought to be possible. So I was not happy about having to wait several seconds after hitting Run in Qt Creator, while it built a .deb, copied it to the device over wifi or USB networking, installed and ran it there. Deploying to the Qt Simulator is quicker, but not as much as I think it ought to be. Plus, the Qt Simulator apparently cannot simulate the landscape mode of the N9, and it lies about the default font size of QML Text elements (if you do not specify a pixelSize, text elements will look all right in the simulator, but ridiculously tiny on the N9).

In the end I cobbled up a shell script that rsyncs my updated QML files to the device and runs a short Python script (over ssh) to launch them. You need rsync and PySide installed on the device for this, obviously, as well as having SSH set up for passwordless logins. As a bonus, I can now do QML development during lunch, directly on the device, with vim, enjoying proper syntax highlighting. :)

Oh, and my code is up on Github.

Crackbook

2011-08-31T01:27:14+00:00

Shortly before starting my job at Google, I had invested a little time into writing a Chrome extension called Crackbook. It helps against wasting time on the web by delaying access to selected sites (see this XKCD cartoon for an illustration, and check out the hint too). Because access is just delayed rather than blocked, you are less likely to just disable the tool after a while, which is exactly what happened to the other tools I tried to use.

Having found a little free time in the evenings, I have just updated the extension. If you use Chrome, please try it and give feedback!

The extension also collects anonymous statistics on what domains people tend to mark as junk and how often the domains are hit. Some very basic results are available at crackbook.info. Now that a fair amount of data has been collected, a deeper statistical inspection is in the plans. Stay tuned.

Porting FBReader to Meego 1.2 Harmattan

2011-07-29T00:16:00+00:00

Andrew Olmsted built the first FBReader packages for Harmattan, after tweaking the build system a bit. The desktop version of FBReader already used Qt 4, and ran almost unmodified, but with some bugs (segfault on task switch) and ugly UI.

I started with the Ubuntu packages for FBReader, since they used a more sane build system for .debs (compared to upstream's funky shell script). Some tweaks were needed to make it build in Scratchbox: since GTK+ and Hildon libraries aren't available on Harmattan, I had to disable the building of -gtk and -maemo versions of libzlui. I also got to learn a new tool — quilt.

Fixing the segfault took a couple of days of debugging, studying the source code of both FBReader and Qt itself, and asking for help on IRC. Turns out FBReader was holding an active QPainter instance for too long, and its backing pixmap got destroyed (or, rather, converted from an OpenGL texture to a plain X11 pixmap) during a task switch, causing the crash. I'm probably describing this wrong BTW, but, in any case, adding QPainter::begin() and QPainter::end() calls in the paintEvent handler fixed the segfault.

Next, a small tweak in the .desktop file to make FBReader a single-instance application: change Exec=FBReader to Exec=single-instance /usr/bin/FBReader (I'm paraphrasing slightly).

Then, a more ambitious goal: making FBReader intercept volume keys and use them for scrolling. Google gave me a pointer to QmKeys, which was the wrong API to use here, but gave me a lead to qmkeyd2, which appears to be an open source daemon, which gave me a lead to sysuid, another open source daemon, which in turn gave me a lead to libresourceqt, and that was the right API at last.

Volume keys generate regular key events for XF86AudioRaiseVolume and XF86AudioLowerVolume, but they're also intercepted by qmkeyd2, which tells all subscribers (and sysuid is one) about them. Which subscriber gets to react is determined by the resource policy framework. So what I needed to do in FBReader was acquire the ScaleButtonResource when FBReader starts up or gets focus, and release it when FBReader quits or goes into background. That also required some IRC help until I discovered installEventFilter() and the ApplicationActivate/ApplicationDeactivate events. And QApplication::instance().

The various tools available in the developer firmware were invaluable: openssh, gdb, valgrind, strace, xev, xprop, lsof, netstat. Also, I would not have achieved my second goal without being able to look at the sources of Meego system components (qmkeyd, sysuid). Yay open source!

Here are my changes to the source code. You can find my modified Debian packaging files, as well as prebuilt binary packages (with full debug info, for gdb goodness), in my experimental harmattan apt repository. The UI is still ugly and non-native, but it doesn't matter much in fullscreen mode :) .

Note to self: when next building fbreader, make sure the 2 megabyte tags file doesn't end up in the .diff.gz. And speaking of crud in source packages, the vim package I built for Harmattan the other day contains the entire 50 meg .hg in the .orig.tar.gz. I need to figure out how to tell dh_make to omit it.

Nokia N950

2011-07-19T12:44:00+00:00

Last Thursday I received a package containing something called the Nokia N950 development kit. Sweet sweet hardware, shame it's not going to be sold to end users. The software is visibly an unfinished pre-release version, but shows great potential. There are almost no 3rd-party apps, which is why Nokia is loaning these N950s to random developers.

I intend to port GTimeLog to it. Although my more immediate need is to have FBReader, so that I can stop carrying both this one and my N900 with me everywhere. Also, vim would be nice.

I've already hacked up Lithuanian support to the virtual and hardware keyboards, thanks to the very nice design of Maliit. As a comparison, I've had my N900 for a year and a half, and I still can't type Lithuanian on it. XKB is not fun.

Amazon Kindle 3

2011-04-02T21:13:02+00:00

For the past couple years I had been reading very little, two or three books a year.

For the past month my average has been two books per week.

The culprit is the Amazon Kindle 3 I bought during my stay in the US. I just can not recommend it enough. For starters, it is quite affordable at $139. The image quality is wonderful, and reading in daylight or even in direct sunlight is pleasant. This is a welcome change from a backlit screen as the eyes get noticeably less strained. The format of the device is even more convenient than of a real book (the size is perfect for plain text, start thinking about the Kindle DX if and only if you intend to read PDFs). The Kindle is quite thin and convenient to hold (and you do not need to keep it from closing as with paperback bindings). Battery life is on the order of weeks. Extra functionality (e.g., web browsing) is available, but not convenient, but that is almost an advantage as distractions are kept at bay. The Amazon store is great, I have already shelled out over $50 with little remorse (my time is much more expensive than the measly $9.99 a typical book costs). There are quite a few free books around too. And to top it off, the "screensaver" with pictures of writers is a very nice touch.

The Kindle is also useful for going through longer articles you would normally be reading on your laptop. The Chrome extension Send to Kindle has worked well for me, do check it out.

The most important drawback of the Kindle is that it is slow to turn pages, so it is not easy to browse a book. Turning a page takes a second or so. This is extremely annoying when reading in non-linearly, e.g. taking in study material, when you constantly need to go back to check up details of previous material. A PDF interpreter is available, but the screen is too small to show a full PDF page (a Kindle DX or an iPad would probably be better) and scrolling is painful because of the delays. Wireless seems to drain the battery noticeably, there have been a few cases where I forgot to turn it off and found a dead Kindle a few days later. I wish the WiFi connection would manage itself intelligently somehow.

The Kindle drove the point home for me that dead-tree books will be going the way of vinyl soon, much sooner than I had expected. And for good reasons. Go order now if you do not have an e-book reader yet (and do not forget to order a sleeve too, given the time you will be spending carrying the device it will come in very handy).

Google Munich

2011-03-16T14:57:51+00:00

It looks like I have made it through most of the hurdles of the Google hiring process, and I should be starting my new job at Google's Munich office soon. Now I just need to haul my stuff to Munich and get the paperwork done. It sure looks like this is going to be a very interesting year.

Trip to Bay Area coming to an end

2011-03-06T04:37:46+00:00

My trip to the Bay Area is coming to the end. It has been surprisingly productive. Here's some things I have done in the nearly two weeks I have been here:

got acquainted with San Francisco
interviewed for a job at a startup
visited Stanford University & Berkeley University campuses
visited Google Mountain View and San Francisco offices
hanged out with folks from Google, from Twitter and from some very promising startups, also a Stanford professor, a researcher from Berkeley and a Berkeley undergrad student, among others
hanged out with people from The Singularity Institute for Artificial Intelligence
spent a day biking in the Golden Gate National Recreation Area

While I am getting a little travel-weary, I know I will be sad leaving this place.

Off to San Francisco!

2011-02-23T04:01:05+00:00

The interview at Munich seemed to go well. Well, at least, I had fun. We'll see how that turns out.

For now, I'm off again for another job interview, this time to Palo Alto. This will be my first time to the West Coast, so I have postponed the return flight for two weeks to familiarize myself with the proverbial area. If anyone's from around the San Francisco Bay Area, I would be very happy to meet over lunch/dinner/beer!

Interviewing with Google

2011-02-16T08:24:09+00:00

I somehow made it through Google's phone interviews in one piece, and now they are flying me in for an on-site interview in Munich (Germany), which will take place tomorrow. Keep your fingers crossed for me!

I will be landing in Munich in a few hours, and today I am free to look around the city. Anyone from here up for lunch or a beer?

Lithuanian (AltGr) keyboard for Windows

2010-12-19T22:59:43+00:00

In GNU/Linux I usually use an AltGr-based keyboard layout for entering Lithuanian characters. That is, you can enter Lithuanian diacritics by holding down AltGr (right Alt)). I had lived with separate modes for entering English and Lithuanian until I found out that having a single layout was a huge relief.

Windows does ship with a Lithuanian layout where AltGr switches to English characters, but I use the English layout much more, so I need it the other way around. So, I grabbed Microsoft's Keyboard Layout Creator and made the layout I needed. The result is available on a GitHub repository. You may also download the installer directly.

In Linux, keymaps, being plain text files, are more transparent, but to be fair, in this case the Windows way wins. Not only do you get a GUI tool to create the layout, but also an installer can be generated automatically for deployment of the layout.

Mobile App Camp

2010-12-19T19:13:00+00:00

On the last weekend my team and I participated in the Mobile App Camp (link in Lithuanian) organized by Omnitel, which is one of the dominant mobile operators in Lithuania. The event seems to be part of Omnitel's push for increasing the share of smartphones in the Lithuanian market, which has been lagging behind the European trends.

Our team consisted of Povilas Kytra, who is behind the TV.LT project, Mantas Kanaporis from A-Gain and me. In the weekend we built an app that shows the TV programme for the coming day for all the Lithuanian TV channels. (The app is not yet available on the Android Market, but we are working on it.)

Here's a screenshot of the main screen of the application:

The app was built using the standard Android Development Toolkit on Eclipse. The app gets the content from a Rails-based server containing a simple database and couple JSON views.

I had some experience of developing for Android before, but it was mostly about working with graphics on canvas, while in this app we used some standard GUI controls (with some nifty styling).

For the source control, we used a private repository on bitbucket.org. That one was a huge letdown: a 'hg push' or 'hg pull' would take ages (or at least that's how it seemed to me in comparison to GitHub), and we had no end of trouble with merging, partly due to the number of commands needed to get the repositories in sync (hg pull; hg update; hg merge; hg commit; hg push). Even Subversion would probably have worked better.

The event itself was great fun. It had been a while since I had last coded intensively for the entire weekend. There were quite a few decent ideas presented by the participants, and some of them were successfully implemented.

The rating system was somewhat disappointing though. The event was supposed to be a contest, with four predefined criteria for winning apps (still available on the website): uniqueness in the Lithuanian market, magnitude of the target group, value provided to the user and creativity. In the end, however, the jury nominated apps for three different awards (best app built on an existing database, best app built anew, and the "hope" nomination) with one app awarded from each category. Our app scored second in the first nomination, so we did not get an award, even though we would probably have been in the global top three, were the original criteria upheld.

To be fair, we did not stand much chance against the winner in our category, an app based on vaistai.lt, which sported a database of pharmaceuticals with detailed usage instructions, information about drug stores in Lithuania with maps and inventory status, and even a barcode scanner. Hats off to them. Another winner was "Alaus radaras" ("Beer radar") with locations of local beer bars and inventory info. The third one was "3 milijonai teisėjų" (Three million judges"), which, as far as I understood, was a conception for a basketball-throwing game (basketball is very big in Lithuania, it is a second national religion).

To conclude, it was a fun event and I wish we will be having more of those in Vilnius, even though the Monday after the long weekend was very unproductive.

Stuff I've been doing recently

2010-12-18T23:35:00+00:00

objgraph got

shiny documentation built with Sphinx.
Python 3.x support, thanks to Stefano Rivera.
Python 2.4 and 2.5 support, thanks to Daniel Blackburn.
assorted smaller improvements.

zodbbrowser got

support for all ZODB databases, not just those with a Zope 3/Bluebream-style root folder/local site.
the ability to cope better with broken objects (due to the way ZODB works, not having some of your modules on the Python pack can break unpickling; zodbbrowser now handles this kind of situation better).
assorted smaller improvements.
a slow but inevitable shift of focus from "use it as a plugin for your Zope 3 app" to "it's a standalone tool for inspecting ZODB contents". (Both use cases are still supported, and will be for the foreseeable future.)

imgdiff got

its first public release.
some experimental code to actually find and highlight the differing parts of the images:

This works better when both images are the same size, although there's experimental (and somewhat buggy) code to try all possible alignments. I could use some help here; image processing is not something I'm familiar with, and searching StackOverflow didn't help beyond reminding me of the existence of PIL's ImageChops.difference(), which is for same-sized images only. Many of the results there are about comparing photos, where things like lighting levels matter. What I need is a diff for computer-generated images, where some things may be shifted around a bit, by different amounts, but are essentially the same. Are there any two-dimensional sequence diff algorithms?

Windows (III): Packaging & nitpicks

2010-12-07T10:00:01+00:00

(This is a part of a series of posts on my recent experience with Windows; see Windows (I) and Windows (II): User Interface.)

Every GNU/Linux desktop distribution hangs on a package management system as a foundation, and as a result the packaging systems are quite advanced, while Windows is stuck with those pesky installation wizards. There have been several stabs to implement a package system for Windows (see here), but none seem to have caught on.

There are three main disadvantages on not having good package management: tedium of manual installation, no dependency resolution and no system-wide automatic updates. In practice they are less important than it might seem. Installation wizards are indeed tedious, but in terms of time I found them to be quick enough to get through. Moreover, apps tend to be less specialized and more encompassing than Linux utilities, and as a result you do not need to install as many things as in Linux. For the same reason dependency resolution is not as important: there are few dependencies in general. As for automatic updates, most high-profile apps have integrated mechanisms to deal with that, however, I found that those mechanisms tended to result in nagging the user with messages. After a while I started ignoring the update notifications. I suspect that happens to many users.

The one thing that Windows has going for it is simplicity. You download installation binaries and you run them, and that's it. Linux package systems tend to be very complicated, and if (when) they break, you will need serious sysadmin chops to deal with the problem. Also installing specific versions (either older ones, or newer ones not yet available in the distro) of specific apps/libraries can be unnecessarily complicated, and the dependency graph limits your actions. In my experience, Windows does well enough without all this complexity as long as you don't reinstall your system too often.

As for the other complaints, I frequently run into the restriction of moving/editing files opened by another application. Coming from Linux, that one is stupidly annoying. Furthermore, it is frustrating not to be able to move inactive (or stuck) background windows. I really miss my unified filesystem namespace. Also there's quite a bit of nagging in general on Windows which I would rather do without. In the end, these are not dealbreakers though.

In my next (and probably final) post I will discuss software development on Windows.

Windows (II): User Interface

2010-12-01T21:55:30+00:00

(This is a part of a series of posts on my recent experience with Windows. See Windows (I) for the first post.)

The Windows user interface is definitely acceptable. The charge that it is too colorful or toy-like is completely unfounded. Rather, perhaps geeks should spend less time looking at their gray Motif boxy controls. In terms of speed, the UI is generally more responsive than on Linux, though maybe less so than previous versions of Windows (that could be because my Intel video IGP is a slouch).

An interesting observation is that apps are almost as heterogeneous in terms of interface as in GNU/Linux, even though the standard controls are ubiquitous. Even apps by Microsoft can be separated into different "generations" of UI (e.g., folder windows, Control Panel, Microsoft Office). Also it is clear that generally more attention is paid to UI and usability by app developers, although perhaps not as much as in Apple products.

The greatest asset and the worst offender at the same time, by far, is the overall GUI orientation of the system. Needless to say, it aids discoverability, but reduces scripting capabilities. Problem is, even if you do not need to write scripts per se, command-line actions are useful because they can be repeated and chained very easily, using shell history. In Windows, I occasionally find myself doing much repetitive clicking that would likely be an "Alt-Tab Up Enter" (or sometimes just one key) sequence in Linux. Moreover, the lack of good standartized scripting is a huge pain during app deployments which tend be repetetive. I did not like Windows on servers before, and I do not like it now.

To be fair, my complaints about scripting capabilities may be partly moot because I have only used vanilla Windows batch files, and have not looked at all into Windows PowerShell, which is a new-generation scripting tool. The examples are fairly impressive. The language has a nice look and is clearly powerful, but also looks somewhat complicated which keeps me away until I find a good reason to learn it.

Initially I missed workspaces. The one app I found that was supposed to emulate workspaces would take several seconds to switch the workspace, so it was completely unusable. I learned to do without them rather quickly though. Actually I'm starting to think that using many workspaces is a sign that you are doing too many things at once and also they are an invitation to distraction (how many of you have a "blog reader" workspace right aside your "work" workspace, ready at your fingertips at a moment's notice?). Workspaces do have one killer feature: you can jump directly to a given workspace instead of cycling through windows. This is very useful when working with more than two apps, in which case without workspaces you are forced to think about the morphing Alt-Tab queue when switching windows.

Speaking about UI generations, I much prefer the ribbon toolbars of the new Office. The still-prevalent toolbars with zillions of old-style 32x32 (or 16x16) toolbar icons are really ugly. Typically only a few of the toolbar buttons are actually useful, and they make the user interface unnecessarily cramped and busy. The move to fewer and larger toolbar buttons is definitely on the right track.

The explorer context menus also tend to grow crazy long. Every app wants to get in there, and in the end you have a context menu that takes up half a screen vertically. Needless to say, most of those items are not used much. Sure, you can opt out during app installation, but at that time it is difficult to say how useful the context menu item will be, and there's no easy way (i.e. easier than just ignoring the cruft) to remove the entries afterwards.

In my coming posts I will cover package management and application development on Windows.

Windows (I)

2010-12-01T17:17:12+00:00

I have been helping with an overdue web app project lately, and the project uses ASP.NET MVC, so I have had to use Windows for most of my day. Since I had been using GNU/Linux almost exclusively since 2003, this is quite an adventure (imagine trying out GNU/Linux after a seven year break). It is not nearly as bad as I thought it would be, even though I am using the not-so-well-liked Windows Vista which came pre-installed with my computer.

The first thing that struck me is how much less important the OS is when compared to the situation a few years ago: a lot of the applications I depend on are web-based (GMail, Google Calendar, Google Docs, Springpad), and the ones that are not web-based are multiplatform and installable with a few clicks (GVim comes to mind). In fact, I do not think there is a single piece of software that I really missed in Windows. (An interesting corollary of this observation is that it should be equally easy to migrate to GNU/Linux nowadays.)

The one thing where the operating system matters is hardware support, and I do not think I have to say much here. After years of whack-a-mole GNU/Linux upgrades which would fix support for one component but break another (for things as essential as video and sound), it was absolutely weird having all hardware work properly at the same time (sound after suspend! no video glitches! high quality webcam picture!). There is no point recounting the reasons why hardware support in Linux is as it is, but now I am starting to think that Linux users do tend to suffer from a sort of Stockholm syndrome regarding hardware support; I know I did.

The other important thing that operating systems do is resource management, and I did not have problems here either. In fact, I liked the virtual memory behaviour much better: when Windows ran out of memory (I had filled up the smallish Windows partition and there was no space left for the swap file to grow), it started rejecting malloc calls from applications, but the system was kept alive and responsive, and I could close apps and clean up the mess without much trouble. In Ubuntu, with the default settings, whenever a runaway process ate too much RAM, the system would start thrashing so hard that it would be very difficult to fix. My standard procedure (switch to text console, log in, find and kill app using command line tools) would take minutes and was so annoying that I would sometimes just reset the computer instead. And in the end, even if I did not reset, an important app would possibly be killed anyway. I know there are kernel parameters for tweaking this behaviour (swappiness?), but I never got around to figuring them out as this situation does not arise that frequently, and testing is painful.

I did not like the Windows FS security model. ACLs are much more complicated and thus less transparent that the Unix approach, even though they are more powerful. My main gripe with them is that I do not know a good way to display them easily (as with ls -l in Unix). I have not had to deal with them much though.

I have had no stability issues. It seems that this argument should be put to rest.

Long blog posts are annoying, so I will leave the user interface, the developer experience and my complaints with the system for separate posts (edit: you can now read my next post about the UI).

Verdi's Requiem

2010-11-22T12:06:03+00:00

The choir I sing in (Pro Musica) will be performing on January 22nd in Bremen, Germany. Together with two local choirs we will perform Verdi's Requiem, which is absolutely fabulous. See for yourself:

(Also try Solti's version for better sound quality, but no video.)
It's going to be a busy month preparing for the concert. Verdi's melodies are not complicated, but he's crazy with dynamics: I think it's the first time I've seen a quintuple pianissimo (ppppp) or a quadruple fortissimo (ffff) in sheet music.

Laptop trouble

2010-11-21T16:48:30+00:00

My trusty somewhat-old laptop has been under the weather lately. It will not boot when it's cold, and it's fairly picky. ~20 °C seems too cold already; so I have to heat it up at little.

Now that winter is coming, this needs to be done almost every morning, and that is extremely annoying. Does anybody have an idea of what could be wrong? It seems that the problem is not just a loose connection, as the laptop goes up in stages:

1) when the power button is pressed the power LED will blink for a split-second and nothing will happen,
2) the LED will light up for slightly longer, but the laptop still will not boot,
3) the laptop starts booting but shuts off spontaneously after while on the BIOS screen,
4) the laptop goes through the BIOS screen, but shuts down before showing the boot menu.
Past that, apparently the CPU generates enough heat for stable operation.
This is fairly complicated behaviour for a hardware problem. I wonder, maybe the power supply is at fault? Any ideas, anyone? This is a good excuse to buy a new laptop, but I would rather not splurge right before Christmas season.

Living a monastic life for 10 days

2010-11-04T03:33:38+00:00

About a month ago, at the end of August, I finished a 10-day Vipassana meditation course. This was probably one of the more extreme activities I have ever participated in, and it's well worth a blog post.

I found out about the course in a ridiculously trivial way, by following a link in a Reddit comment (I wonder if I could track down that comment...). The website referenced by the comment contained (almost) no new-agey mumbo-jumbo and looked legit in general. I also appreciated the fact that this was a relatively serious and intensive course, which should be enough to form a reasonably founded opinion on meditation in general. When learning sporadically, in small bits, it is hard to say whether the material itself is no good, or you just do not understand it yet and need to study more; I was quite skeptical towards the practice, but wanted to give it fair trial. I was quite impressed that there was a course available in Lithuania, a few hours by train from where I live. Finally, the course was free (by tradition Vipassana is taught without monetary compensation). I went ahead and signed up for a standard introductory 10-day course.

Vipassana (also called insight meditation) is a meditation technique allegedly coming back from Gautama Buddha himself (well, all the traditions like to claim that origin...). It has been experiencing a renaissance, partly due to an organization set up by a meditation teacher S. N. Goenka which has founded meditation centers all over the world.

The essence of Vipassana is a method of teaching your mind to act instead of merely reacting to outside stimuli such as pain, anger of other people or hardships. The technique provided is to monitor your sensations, watch them with equanimity and suspend any reaction, and slowly it sinks in that all things are temporary and you can choose what to react to and how. That's the basic idea; I don't want to go deep into exposition of Vipassana as there is plenty of information about that (also at the same time the information is almost useless because you have to experience it to really understand it). Instead, I will sure some of the more personal remarks and experiences.

The course took place in an cottage near Kretinga, housing around 50 people. The living conditions were absolutely great, and this would have been a true holiday if not for the regime and the rules. The day would start very early, at 4 AM (ending at around 9:30 PM), and we'd do about 12 hours of meditation in total inbetween, with occasional breaks, but the most gruesome part is that no communication at all was allowed with anyone except the manager of the camp (and the teacher during a specified time). That gets to you after a few days, but still, there are people around you, so it's bearable; on the other side, the senses (touch, smell) become very acute. The food was taken care of by volunteers (students who have finished the course), and I could say it was quite good actually. The goal, of course, was to keep the environment from interfering with meditation, and I think it was done very well.

The presentation of the actual technique also did not disappoint. Each day a new step would be introduced, with the whole day to practice before moving on. Looking back, the pacing worked well. Every day in the evening there were lectures on theory. They were somewhat informative, but also sometimes annoyingly primitive and verbose. The little mistakes, holes in logic and slight doses of mumbo-jumbo almost drove me nuts by the middle of the course, until I learned not to react to them. Despite the shortcomings of presentation, the simplicity and lucidity of the practice and lightness of theoretical baggage really appealed to me. The whole organisation in general gave the impression of complete transparency, with no strings attached, no financial interests in play and no agitation for further involvement.

While there are many meditation techniques, I am happy that I've gone with Vipassana in particular for one reason: Vipassana meditation uses no artificial tools to help you concentrate, such as mantras, visualisations, counting, etc. Such methods make concentration much easier, but they also create a dependence on the methods themselves. I appreciate the purity of Vipassana in this regard.

Not everything in the practice is roses though. First, a particular worldview supported by Vipassana is presented as "scientific" and supposedly "the right way to live", and if even if you don't agree, "the truth" of Vipassana will become obvious to you with time. This is a distinctive mark of ideologies. The thing that makes everything break is the reincarnation, which is generally assumed as true, even if it's not shoved down the student's throat. Every rationalization of a world view has a "white lie" (usually related to the concept of eternity somehow) at the core, and in this context reincarnation is it. Given what we know now about the relationship between the mind and the body, reincarnation is absolutely ridiculous and completely untenable, and that breaks all "moral" reasoning for practicing meditation as opposed to doing something else. I'm not the first to bring up this problem, there's plenty of material about it online.

The second slippery idea is that the concept of "negative happiness" is central in Vipassana, that is: happiness is defined as lack of suffering. This point is not that unreasonable actually, from personal experience I could indeed call the state of non-suffering happiness, but there is definitely something missing when compared to the state of mind when I'm truly, positively happy. Whether you consider that something important or not, it's definitely there.

Third, Vipassana suggests that it is only you who is at fault when your inner state is disturbed; the world affects you as much as you let it, and in order to find happiness you should learn to control the "interface" rather than the outside world, which will usually not behave as you want. This may be correct, but sometimes it could just be easier to fix the world rather than change yourself, and I have the suspicion that it this is much more frequently the case in the modern world compared to the pre-modern environment. (This is one of the points which break when reincarnation is rejected, as with reincarnation you have the whole eternity to fix yourself.)

While the complaints are serious, they do not mean complete rejection of Vipassana but rather integration among the various other techniques available. The actual meditation technique is still great for gaining and maintaining mental composure, reducing anxiety and improving well-being. Now, if I could only get used to meditating regularly...

If you have the chance, I highly recommend you try out an introductory 10-day course. There is little to lose and a lot to gain. And if you do go ahead, I have two hints. First, do not even think about leaving the course before it's over. Any hardships are temporary and will be overcome quickly. Second, use the time with the teacher, those interactions were very helpful to me. Also, given the "noble silence", talking to someone once in a while will not hurt.

Riga marathon

2010-09-16T23:03:56+00:00

I participated in the Riga marathon and ran the full 42km! The time (4:39) isn't great at all, but I'm happy to have finished at all!

Profiling with Dozer

2010-08-07T03:06:00+00:00

Dozer is mostly known for its memory profiling capabilities, but the as-yet unreleased version has more. I've talked about log capturing, now it's time for

Profiling

This WSGI middleware profiles every request with the cProfile module. To see the profiles, visit a hidden URL /_profiler/showall:

What you see here is heavily tweaked in my ~~fork~~ branch of Dozer; upstream version had no Cost column and didn't vary the background of Time by age (that last bit helps me see clumps of requests).

Here's what an individual profile looks like:

The call tree nodes can be expanded and collapsed by clicking on the function name. There's a hardcoded limit of 20 nesting levels (upstream had a limit of 15), sadly that appears not to be enough for practical purposes, especially if you start profiling Zope 3 applications...

You can also take a look at the WSGI environment:

Sadly, nothing about the response is captured by Dozer. I'd've liked to show the Content-Type and perhaps Content-Length in the profile list.

The incantation in development.ini is

[filter-app:profile]
use = egg:Dozer#profile
profile_path = /tmp/profiles
next = main

Create an empty directory /tmp/profiles and make sure other users cannot write to it. Dozer stores captured profiles as Python pickles, which are insecure and allow arbitrary command execution.

To enable the profiler, run paster like this:

$ paster serve development.ini -n profile

Bonus feature: call graphs

Dozer also writes a call graph in Graphviz "dot" format in the profile directory. Here's the graph corresponding to the profile you saw earlier, as displayed by the excellent XDot:

See the fork where the "hot" red path splits into two?

On the left we have Routes deciding to spend 120 ms (70% total time) recompiling its route maps. On the right we have the actual request dispatch. The actual controller action is called a bit further down:

Here it is, highlighted. 42 ms (24% total time), almost all of which is spent in SQLAlchemy, loading the model object (a 2515 byte image stored as a blob) from SQLite.

A mystery: pickle errors

When I first tried to play with the Dozer profiler, I was attacked by innumerable exceptions. Some of those were due to a lack of configuration (profile_path) or invalid configuration (directory not existing), or not knowing the right URL (going to /_profiler raised TypeError). I tried to make Dozer's profiler more forgiving or at least produce clearer error messages in my ~~fork~~ branch, e.g. going to /_profiler now displays the profile list.

However some errors were very mysterious: some pickles, written by Dozer itself, could not be unpickled. I added a try/except that put those at the end of the list, so you can see and delete them.

Does anybody have any clues as to why profile.py might be writing out broken pickles?

Update: as Ben says in the comments, my changes have been accepted upstream. Yay!

Capturing logs with Dozer

2010-08-07T02:27:00+00:00

Dozer is mostly known for its memory profiling capabilities, but the as-yet unreleased version has more:

Log capturing

This WSGI middleware intercepts logging calls for every request. Here we see a toy Pylons application I've been working on in my spare time. Dozer added an info bar at the top:

When you click on it, you get to see all the log messages produced for this request. I've set SQLAlchemy's loglevel to INFO in my development.ini, which produces:

(Why on Earth does SQLAlchemy think I want to see the memory address of the Engine object in my log files, I don't know. The parentheses contain argument values for parametrized queries, of which there are none on this page.)

Upstream version displays absolute timestamps (of the YYYY-MM-DD HH:MM:SS.ssssss variety) in the first column; my fork shows deltas in milliseconds. The incantation in development.ini is

[filter-app:logview]
use = egg:Dozer#logview
next = main

which makes it disabled by default. To enable, you run paster like this:

$ paster serve development.ini -n logview

(Upstream version lacks the paste entry point for logview; it's in my fork, for which I submitted a pull request weeks ago like a good open-source citizen. Incidentally, patches for stuff I maintain have been known to languish for years in my inbox, so I'm not one to throw stones.)

Next: profiling with Dozer.

Update: Tom Longson blogged about this back in 2008! And his CSS is prettier.

irclog2html is now on PyPI

2010-08-06T01:32:00+00:00

irclog2html, the IRC log to HTML converter, is now (finally!) available from the Python Package Index.

In other news, logs2html now copies irclog.css to the destination directory (if it doesn't exist there already). I've been noticing logs produced with irclog2html on random places, and sometimes they were unstyled; hopefully this will become rare now.

Python course projects

2010-07-29T11:43:43+00:00

This spring I read a course on Python in Vilnius University, Department of Mathematics and informatics. The course was not mandatory, which I suspect is one of the reasons why my students were quite a bright and motivated bunch. I began the course with a brief introduction to the Python language and the standard library, then dedicated a few lectures to Django and in the remainder I covered various libraries (GUI toolkits, Pygame) and techniques (testing, debugging, optimization).

One of the major tasks of the course (and the deciding factor in the final valuation) was to develop a project using Python. The students were given complete freedom over the topic of the project and the tools used, as long as Python was involved. The quality of the result varied, of course; had I given concrete tasks, the low bar would definitely have been higher. However, I am happy with how things turned out.

Probably the best-executed project was 1000 Online, an online multiplayer card game (see rules). It was developed by Vytautas Karpavičius. Vytautas apparently had prior experience with Python and Django, nevertheless, the quality of the app (and the code behind it) is impressive. Among the other projects, Gegute.lt, a Lithuanian clone of Reddit is still up and running. It's nothing too fancy, but looks quite clean for a university project. Other projects perhaps were not perfectly executed, but were very interesting conceptually. Jonas Keturka developed a bot (BitBucket) for the online strategy game Travian. Marius Damarackas developed a web-based guess-the-song game. He used YouTube for the song material and Last.fm (IIRC) to make you guess among similar songs, so that it would not be too easy. Andrius Chamentauskas developed a Puyo Pop clone (GitHub) with PyGame, complete with multiplayer and an AI. Paulius Budzinskas worked towards his course paper by developing a bacteria movement modelling and visualisation app (GitHub; Lithuanian only). David Abdurachmanov developed a spell-checker bot for WikiAnswers.com, with a really baroque and sophisticated architecture. I believe he has applied to Google Summer of Code with that project; I wonder how that is working out.

For reference, you can find links to all student repositories on GitHub/BitBucket in this list. Beware: some of them may be documented/commented in Lithuanian.

Another interesting effect of the course was that people started using Python in other related courses, for example in Mathematical Modelling and Numeric Methods. I even heard that the lecturer was impressed and expressed interest in learning Python. Another lecturer, this time a statistician, was interested in teaching and using SAGE at lab practice. Feeling such 'ripples' going through the department sure felt nice.

By the way, the university also offers lectures on Ruby, read by Saulius Grigaitis. I took some inspiration from him for the course form and content.

Serving ePub files to Android 1.6

2010-07-22T23:23:00+00:00

If you click on an ePub download link in the Android browser and get back an error saying "Cannot download. The content type is not supported on the device", then make sure the web server is setting the Content-Type header to "application/epub+zip" and not "application/octet-stream". When you do this, Android will launch FBReaderJ automatically (provided that you have it installed, of course).

N900 connection sharing the hard way

2010-07-20T22:41:00+00:00

My N900 has a SIM card with a flat-rate 3G data plan. My laptop hasn't. What do I do when I want to use the Internet on my laptop somewhere that doesn't have WiFi? Well, there are many options:

Option 1: N900 as a USB modem

Use the provided USB cable to connect the N900 to the laptop. Choose "PC Suite" mode on the N900 when you get the USB connection menu. The laptop now sees your N900 as a bog-standard USB 3G modem. Use Network Manager to connect to the internet.

Pros: no extra setup required. The N900 and the laptop can both access the Internet at the same time.

Cons: you have to use a USB cable (I hate cables). You cannot ssh into your N900 (and ssh is my primary file transfer protocol between the laptop and the M900).

Option 2: N900 as a Bluetooth DUN modem

Install Bluetooth DUN support from Maemo Extras. Then use it like you would any other phone that has Bluetooth DUN.

Pros: no cables.

Cons: Bluetooth is the worst technology ever. I never had it work reliably. Plus, Network Manager in Ubuntu 10.04 doesn't support Bluetooth DUN (it supports only Bluetooth PAN, as far as I know).

Option 3: N900 as a WiFi access point with Joikuspot

I haven't tried this.

Pros: simple (hopefully), no cables required.

Cons: Joikuspot is non-free. I'm not an absolute zealot, but I will avoid closed-source stuff when open-source alternatives are available.

Option 4: N900 as a WiFi access point with Mobilehotspot

I haven't tried this either.

Pros: it's an open-source app available from Maemo Extras. No cables required.

Cons: requires a non-standard kernel (or so I've heard). Way outside my comfort level.

Option 5: N900 as a WiFi access point with shell scripts

Here's the shell script I run on my N900: share-wifi. It sets up an ad-hoc WiFi network, and starts a DHCP and DNS server (dnsmasq). Sadly, it cannot set up connection sharing (NAT), so I rely on OpenSSH as a SOCKS5 proxy. The whole setup is like this:

You want the latest firmware (PR 1.2) to avoid this bug.
You need to have OpenSSH installed on the N900. Also, setting up key-based authentication makes it more convenient.
The script assumes that you've set up sudo on the N900 so that you can run any command as root.
You need to have wireless-tools installed. It's in the main SSU repository so you should be able to sudo apt-get install it (if it's not preinstalled; I don't remember).
On the N900 run share-wifi in a terminal (optionally passing a WiFi channel number from 1 to 11, in case you need to avoid interference with nearby networks).
On the laptop connect to the new n900 WLAN and run ssh -D 1080 user@n900. You will get a shell session; the SOCKS proxy will be active while it is open.
Reconfigure your laptop to use a SOCKS5 proxy on localhost:1080. For GNOME systems I've a couple of shell scripts: proxy-on and proxy-off. For applications that do not use the GNOME proxy settings (such as Subversion access over SSH), use tsocks.
When done, hit Ctrl-C on the N900 to terminate the sharing script.

Pros: no non-free software or custom kernel required. No cables.

Cons: complicated to set up. No WLAN power savings available for ad-hoc networks, so battery life is extremely poor (~2 hours). But, hey, no cables!

ImportError: No module named _md5

2010-07-06T22:20:00+00:00

If you're using virtualenv, and after a system upgrade you get errors like

...
  File "...", line ...
    from hashlib import md5
  File "/usr/lib/python2.6/hashlib.py", line 63, in __get_builtin_constructor
     import _md5
ImportError: No module named _md5

this means that the copy of the python executable in your virtualenv/bin directory is outdated and you should update it:

$ cp /usr/bin/python2.6 /path/to/venv/bin/python

or, better yet, recreate the virtualenv.

Booting ISO images from a USB drive

2010-05-03T17:24:00+00:00

Dear lazyweb, I would like to download an arbitrary ISO image (say, a Ubuntu 10.04 Desktop CD) into a directory of a USB flash drive, and then make that USB flash drive boot that ISO image. I do not want to

re-partition or re-format the flash drive (this eliminates usb-creator, AFAIU)
extract the contents of the ISO image into the root of the USB drive (this eliminates unetbootin)
skip the ISO's bootloader and directly boot the kernel+initramfs from the ISO (eliminates this recipe, and this recipe)

I just want a bootloader on the USB to read the VFAT filesystem, mount the ISO image as a loop device, then chain-load the bootloader from that ISO. Bonus points for having a menu letting me choose one of several ISO images. Running a script to edit a text file (say, grub's config) to get that menu is fine.

Is this even possible? If not, can I at least have two out of three (no partition/extraction, but skipping intrinsic bootloader is fine)?

Re: Web frameworks considered useful.

2010-04-18T01:57:00+00:00

Martijn Faassen defends web frameworks in a rather longish post (you can tell it's 5 AM in the morning and I've nearly defeated the unread post queue in Google Reader). I'd like to propose a condensed version. Consider this slogan:

Simple things should be easy; complicated things should be possible.

Frameworks make simple things easy. Good frameworks keep the complicated thing possible; poorly-designed frameworks make the complicated thing more difficult than necessary; bad frameworks make even simple things complicated.

Doing everything from scratch merely makes things possible, but rarely easy.

Review: Grok 1.0 Web Development

2010-04-04T19:30:00+00:00

Disclaimer: I received a free review copy of this book. The book links are affiliate links; I get a small amount from any purchase you make through them.

Grok is a Python web framework, built on top of the Zope Toolkit, which is the core of what used to be called Zope 3 and is now rebranded as BlueBream. Confused yet? Get used to it: the small pluggable components are the heart and soul of ZTK, and the source of its flexibility. It's not surprising that people take the same approach on a larger scale: take Zope 3 apart into smaller packages and reassemble them into different frameworks such as Grok, BlueBream or repoze.bfg.

The Grok book by Carlos de la Guardia introduces the framework by demonstrating how to create a small but realistic To-do list manager. I like this technique, and it works pretty well. The author covers many topics:

creation of a new project
simple views with Zope Page Templates
automatic form generation from schemas (with tweaks)
catalogs and indexes (my favourite chapter)
security: users, roles, permissions; authentication and authorization
extremely pluggable page layouts with viewlets and pagelets
basic ZODB, blobs, ZEO, database packing, backups with repozo
SQL databases, integration with SQLAlchemy (including a common transactional model)
component architecture: adapters and utilities
Martian: extending Grok by defining custom component directives
very short intro to testing (zope.testing, unit tests and doctests, functional tests with zope.testbrowsing) and debugging (pdb; AJAXy debugger, which looks exactly like the Pylons one with an uglier skin)
deployment (my second favourite chapter): paster, apache and mod_proxy, mod_wsgi, pound, squid, varnish, scalable deployments.

Some important topics like internationalization, time zones, testing with Selenium, and (especially) database migration (which is pretty specific for ZODB) were not covered.

If you want to learn about Grok, this book will be useful, but there's a caveat: there's the usual slew of typographical mistakes and other errors I've come to expect from books published by Packt. It's their third book I've seen; all three had surprisingly high numbers of errors. Some had more, others had fewer. The Grok book was on the high side and the first one where I was tempted to record a "WTFs per page" metric. The mistakes are easy to notice and correct, so they didn't impede my understanding of the book's content. Disclaimer: I've been working with Zope 3 for the last six-or-so years, so I was pretty familiar with the underlying technologies, just not the thin Grok convenience layer. If minor errors annoy you, stay away. I haven't noticed any major factual errors, although there were what I would consider some pretty important omissions:

ZODB is not as transparent as people tell you. There are many gotchas, especially if you want to refactor your code without throwing away old databases.
bin/buildout is free to recursively remove anything under parts. Keeping your database there is fine only if you don't mind occasionally starting from scratch.
repozo does not back up blobs.
The ZODB transaction conflict resolution depends on being able to repeat requests several times; this is important if your code has external side effects (e.g. sends emails, creates files, pings 3rd party websites over HTTP). Packages like megrok.rdb or zope.sendmail take care of this; it'd be nice to be shown how to do that for your own code before you discover this issue the hard way when your app starts charging people's credit cards three times every now and then.
You need to make sure you send out object events at appropriate times, or your catalog indexes won't be updated.
Permission and role grants are persistent: if you delete a user and then create a new one with the same username, the new user will have all the roles and permissions granted to the old one. If you implement user deletion, you need to explicitly remove old grants.
The Zope security model expects every object to have a valid __parent__ attribute; permission/role grants will not work properly on objects without a __parent__. Most of the time this is taken care of automatically, but when it's not, you can get really confusing errors.
applySkin should only be used for browser requests; blindly calling it from a traversal event handler can break WebDAV/XML-RPC. (Incidentally, I should file a bug about that; it should abort if you pass a non-browser request instead of silently converting it into a browser request.)
Allowing end-users to specify ++skin++ in the URL can be a security hole.

Overall, Grok is pretty nice, especially compared to vanilla Zope 3. However, when compared to frameworks like Pylons or Django, Grok appears more complex and seemingly requires you to do additional work for unclear gain. For example, chapter 8 has you writing three components for every new form you add: one for the form itself, one for a pagelet wrapping the form, and one for a page containing the pagelet. Most of that code is very similar with only the names being different. I'm sure there are situations where this kind of extreme componentization pays off (e.g. it lets you override particular bits on particular pages to satisfy a particular client's requests, without affecting any other clients), but the book doesn't convincingly demonstrate those advantages. Again, I may be biased here since I've been enjoying those advantages for the past six years, without ever having felt the pain of doing similar customizations with a less flexible framework. (It's a gap in my professional experience that I'm itching to fill.)

Update: some other reviews on Planet Python.

Update 2: Another review (well, part 1 of one, but I got tired waiting for part 2).

Review: Python Testing: Beginner's Guide

2010-03-13T18:54:00+00:00

I've been testing (as well as writing) Python code for the last eight years, so a book with the words Begginer's Guide prominently displayed on the cover isn't something I'd've decided to buy for myself. Nevertheless I jumped at the offer of receiving a free e-copy for reviewing it.

Short summary: it's good book. I learned a thing or two from it. I don't know well it would work as an introductionary text for someone new to unit testing (or Python). Some of the bits seemed overcomplicated and underexplained, parts of the example code/tests seemed to contain design decisions received from mysterious sources.

Incidentally, Packt uses a simple yet effective method for watermarking e-books: my name and street address are displayed in the footer of every page. What's funny is that the two non-ASCII characters in the street name are replaced with question marks. It's not a data entry problem: the website that let me download those books shows my address correctly, so it must be happening somewhere in the PDF production process. I didn't expect this kind of Unicode buggyness from a publisher. Then again there were occasional strange little typographical errors in the text, like not leaving a space in front of an opening parenthesis in an English sentence, or using a never-seen-before +q= operator in Python code. I was also left wondering how the following sentence (page 225) could slip past the editing process:

doctest ignores everything between the Traceback (most recent last call).

Thankfully those small mistakes did not detract from the overall message of the book.

I liked the author's technique of showing subtly incorrect code, letting the reader look at it and miss all the bugs, and then showing how unit or integration tests catch the bugs the reader missed. I'm pretty sure there's at least one remaining bug that the author missed in the example package (storing a schedule doesn't erase old data), which could serve for a new chapter on regression testing if there's a second edition.

Summary of topics covered:

Terms: unit testing, integration testing, system testing.
Basics of doctest and unittest, their strengths and weaknesses.
Using mocks (with Mocker).
Using Nose.
Test-Driven Development with lots of example code.
Using Twill.
Integration testing with lots of example code.
Using coverage
Post-commit hooks to run tests with Bazaar, Mercurial, Git, Darcs, Subversion.
Continuous integration with Buildbot

I found the TDD cycle a bit larger than I generally like, but I believe it's a matter of taste, and perhaps a shorter cycle wouldn't work as well in a written medium.

I found it a bit jarring how the Twill chapter intrudes between the two chapters showing unit testing and integration testing of the same sample package. I think it would've been better to swap the order of chapters 8 and 9.

I liked the technique presented for picking subsets of the code for integration tests, although I wonder how well it would work on a larger project.

Topics not covered:

Functional testing (which is very close but not exactly the same as system testing).
Regression testing (page 46 contains advice about this without mentioning the term regression testing).
Continuous integration with Hudson (simpler to set up than buildbot, easily covers 80% of cases).

As you can see these holes are all rather small.

Probably the biggest weakness of the book is the complexity of some things shown:

writing mocks for pure unit tests
mocking other instances of the same class under test
even occasionally mocking self, which needs tricks like calling a method's im_func directly
mocking __reduce_ex__ so you can pickle mocks in an integration test, instead of using real classes or simple stubs.
testing the same code multiple times: unit tests, several sets of integration tests that test ever-increasing subsets of classes
Buildbot instead of Hudson

Seeing the repetitive and redundant mock code in the first few doctest examples I started asking what's the point?, but the book failed to provide a compelling answer (the answer provided—it's easier to locate bugs—works just as well for integration tests that focus on individual classes). And there are good answers for that question, like instant feedback from your unit test suite. Are they worth the additional development effort? Maybe that depends on the developer. I don't think they would help me, so I tend to stick with low-level integration tests I call "unit tests" (as well as system tests; it's always a mistake to keep all your tests in a single level). I'm slightly worried that this book might give the wrong impression (testing is hard) and turn away beginning Python programmers from writing tests altogether.

Overall I do not feel that I have wasted my time reading Python Testing. I look forward to reading the other reviews that showed up on Planet Python. I gathered that not all reviewers were happy with the book, but avoided reading their reviews in order not to influence my own.

Update: I especially liked this review by Brian Jones. The lack of awkward page breaks in code examples is something that I only noticed after reading a different book, which is full of such awkward breaks, sigh.

Update 2: The book links are now affiliate links; I get a small amount from any purchase you make through them.

You've got to love profiling

2010-03-06T17:49:00+00:00

Yesterday I slashed 50% of run time from our applications functional test suite by modifying a single function. I had no idea that function was responsible for 50% of the run time until I started profiling.

Profiling a Python program is getting easier and easier:

$ python -m cProfile -o prof.data bin/test -f

runs our test runner (which is a Python script) under the profiler and stores the results in prof.data.

$ runsnake prof.data

launches the RunSnakeRun profile viewer, which displays the results visually:

The square map display of RunSnakeRun, with the 'render_restructured_text' function highlighted.

Who knew that ReStructuredText rendering could be such a time waster? A short caching decorator and the test suite is twice as fast. The whole exercise took me less than an hour. I should've done it sooner.

Other neat tools:

pstats from the standard library lets you load and display profiler results from the command line (try python -m pstats prof.data).
pyprof2calltree converts Python profiler data files to a format that the popular profiler visualization tool kcachegrind can understand. It's somewhat less useful now that RunSnakeRun exists.
profilehooks by yours truly has decorators for easily profiling individual functions instead of entire scripts.
keas.profile and repoze.profile hook up the profiler as WSGI middleware for easy profiling of web apps.

Bye, bye, free time!

2010-03-05T20:02:00+00:00

Things I've taken up to do in the nearest future:

Read and review Python Testing: Beginner's Guide and Grok 1.0 Web Development for Packt. (The links are trackable to my blog, but I'm not getting anything out of it. Other than free copies of the e-books, which I already received, in exchange for a promise to review them on this blog.)
Help Reportlab folks set up continuous integration (most likely Hudson, since Buildbot, while powerful, has a steep learning curve).
Think about becoming the buildbotmaster for Zope. Originally I intended to volunteer to set up a few buildbots for various Zopeish projects (ZTK, BlueBream, Grok, Zope 2) since half of the existing ones were down or broken. Then various people fixed some of the broken ones and other people chimed in mentioning existing buildbots that nobody else knew about. There is a need for somebody to coordinate all this activity: make sure we have up-to-date test results for all kinds of projects, aggregate them in one place, chase up build slaves for exotic OSes (i.e. Windows)... I don't think I'm well suited for this kind of organisational activity.
Push along the various scratch-my-itch open source projects (GTimeLog, irclog2html, zodbbrowser).
No idea what, but I've been wanting to do something for Maemo. Something small, given the copious amounts of free time I have.
Then there's the paying work. On the plus side, there are opportunities for fun there (today I slashed functional test run time by a half, by adding a small caching decorator in front of a single function. RunSnakeRun and cProfile rule!)
You know what, scratch the Zope buildbotmaster idea. Maybe I can do something technical there, e.g. a cron script to ping the various buildbot, scrape HTML/parse emails and aggregate build results. Maybe.
I hope I don't get burnout again. Because that would suck. Again. Been there, done that, didn't even get a T-shirt.

I really ought to read Getting Things Done. Reading it has been on my todo-list for years.

Oopsie

2010-03-03T16:45:00+00:00

Sorry for flooding Planet Maemo -- it was a side effect of changing this feed's URL to only include posts tagged "maemo". I'm not sure if the fault is PyBlosxom's or the aggregator's

As a penance, here's a Terminal trick for you:

LABELS='[Tab,Esc,Enter,PgUp,PgDn,F2,VKB]'
KEYS='[Tab,Escape,KP_Enter,Page_Up,Page_Down,F2,Return]'
gconftool -s /apps/osso/xterm/key_labels --type list --list-type string "$LABELS"
gconftool -s /apps/osso/xterm/keys --type list --list-type string "$KEYS"

This changes the toolbar to have three extra keys (Enter, F2, and a key that acts like Enter when the hardware keyboard is open, and opens the virtual keyboard if the hardware keyboard is closed).

Update: added screenshot:

Nokia N900 Terminal app with new toolbar buttons

Weekly Zope developer IRC meetings

2010-03-03T10:09:00+00:00

On Tuesday we started what will hopefully become a tradition: weekly IRC meetings for Zope developers. Topics covered include buildbot organization and maintenance, open issues with the ZTK development process, and the fate of Zope 3.5 (= BlueBream 1.0).

There are IRC logs of the meeting, and Christian Theune posted a summary to the mailing list.

My take on this can be summed up as: Zope ain't dead yet! The project has fragmented a bit (Zope 2, Zope Toolkit, Grok, BlueBream, Repoze), but we all share a set of core packages and we want to keep them healthy.

Next meeting is also happening on a Tuesday, at 15:00 UTC on #zope in FreeNode.

Latin-1 or Windows-1252?

2010-01-07T20:29:00+00:00

Michael Foord wrote about some Latin-1 control character fun in a blog that's hard to read (the RSS feed syndicated on Planet Python is truncated, grr!) and hard to reply (~~no comments on the blog!~~ my Chromium's AdBlock+ hid the comment link so I couldn't find it), but never mind that.

Unfortunately the data from the customers included some \x85 characters, which were breaking the CSV parsing.

0x85 is a control character (NEXT LINE or NEL) in Latin-1, but it's a printable character (HORIZONTAL ELLIPSIS) in Microsoft's code page 1252, which is often mistaken for Latin-1. I would venture a suggestion that the encoding of the customer data was not latin-1 but rather cp1252.

>>> '\x85'.decode('cp1252')
u'\u2026'

GTimeLog: not dead yet!

2009-12-17T22:22:00+00:00

Back in 2004 I wrote a small Gtk+ app to help me keep track of my time, and called it GTimeLog. I shared it with my coworkers, put it on the web (on the general "release early, release often" principles), and it got sort-of popular before I found the time to polish it into a state where I wouldn't be ashamed to show it to other people.

Fast-forward to 2008: there are actual users out there (much to my surprise), I still haven't added the originally-envisioned spit and polish, haven't done anything to foster a development community, am wracked by guilt of not doing my maintainerly duties properly, which leads to depression and burnout. So I do the only thing I can think of: run away from the project and basically ignore its existence for a year. Unreviewed patches accumulate in my inbox.

It seems that the sabbatical helped: yesterday, triggered by a new Debian bug report, I sat down, fixed the bug, implemented a feature, applied a couple of patches languishing in the bug tracker, and released version 0.3 (which was totally broken thanks to setuptools magic that suddenly stopped working; so released 0.3.1 just now). Then went through my old unread email, created bugs in Launchpad and sent replies to everyone. Except Pierre-Luc Beaudoin, since his @collabora.co.uk email address bounced. If anyone knows how to contact him, I'd appreciate a note.

There are also some older changes that I made before I emerged out of the funk and so hadn't widely announced:

There's a mailing list for user and developer discussions (if there still are any ;).
GTimeLog's source code now lives on Launchpad (actually, I mentioned this on my blog once).

Android apps

2009-12-16T22:56:17+00:00

I've recently gotten myself an Android phone. The Android app market is not as mature as iPhone's App Store, but there is a lot of applications, and plenty of chaff to separate the wheat from. I found most of the apps I use by browsing "top Android apps" lists on the web, so I am putting up a similar list here in case someone finds it useful.

Everyday apps
- Astrid Task/Todo List - a great To-Do list app that integrates with Remember The Milk
- Quick Calendar - an "upcoming events" widget
- CardioTrainer - nice jogging app, records routes using GPS, integrates with music player
Language tools
- Thesaurus Free - a thesaurus
- Free Dictionary Org - reference dictionary
- QuickDic German Dictionary - an English-German-English dictionary
Media content
- Listen - an app to find audio content online
- Mother TED - search and view TED talks
- TuneWiki - music, song lyrics
Games
- Lumbricidae - guide a worm using your accelerometer
- Flying High - plane flying (also accelerometer-based)
- miniMatcher - memory game
- Iconic Memory - another memory game
Cool apps
- Phonalyzr - pretty phone call (and SMS) stats
- Shazam - identify a song by recording a short snippet; possibly one of the coolest apps available
- SnapTell - look up product information by scanning a barcode (or even by snapping a photo!)
- Backgrounds - background images
- Google Sky Map - shows constellations
- Layar Reality Browser 3.0 - a fancy-looking Augmented Reality app
Music
- Sonorox - tune composition for everyone (the pentatonic scale is great, isn't it?)
- Loops! Lite - another music composition app
- Tuner - gStrings - a guitar tuner
Miscellaneous
- Task Manager - a process monitor utility, useful for checking up on app resource usage
- Currency - a currency converter
- Sleep Logger - a very simple, but useful app for tracking your sleep habits

Unix is an IDE, or my Vim plugins

2009-12-08T22:23:00+00:00

Unix is an IDE. I do my development (Python web apps mostly) with Vim with a bunch of custom plugins, shell (in GNOME Terminal: tabs rule!), GNU make, ctags, find + grep, svn/bzr/hg/git.

The current working directory is my project configuration/state. I run tests here (bin/test), I search for code here (vim -t TagName, find + grep), I run applications here (make run or bin/appname). I can multitask freely, for example, if I'm in the middle of typing an SVN commit message, I can hit Ctrl+Shit+T, get a new terminal tab in the same working directory, and look something up. No aliases/environment variables/symlinks/scripts making changes to config files. I can work on multiple projects at the same time. I can work remotely (over ssh).

Gary Bernhardt's screencasts on Vimeo show how productive you can get if you learn Vim and tailor it to your needs. I have Vim scripts that let me

See the name of the class and function that I'm editing in the statusbar, even if the class/function definition is offscreen: pythonhelper.vim.
See all pyflakes warnings and errors in a list as soon as I press F2 to save the file: python_check_syntax.vim.
Add a "from foo.bar import Something" line at the top of the file if I press F5 when my cursor is on Something, looking up the package and module from ctags: python-imports.vim.
Switch between production code and unit tests with a single key if the project uses one of several conventions for tests (e.g. ./foo.py <-> ./tests/test_foo.py): py-test-switcher.vim.
Generate a command line for running one particular unit test (the one my cursor is inside) and copy it into the system clipboard, so I can run that test by Alt-Tabbing into my terminal window and pasting. py-test-runner.vim.
Open the right file and move the cursor to the right line if I triple-click a line of traceback in a shell (or an email) then press F7 in my gvim window: py-test-locator.vim.
Compare my version of the code with the pristine version in source control in an interactive side-by-side diff that lets me revert bits I no longer want: vcscommand.vim.
Highlight which lines of the source are covered by my tests, if I have coverage information in trace.py format: py-coverage-highlight.vim.
Show the signature of a function/class's __init__ when I type the name of that class/function and an open parenthesis (looked up from tags): py-function-signature.vim.
Fold code into an outline so I only see names of methods or classes instead of their full bodies: vimrc, function PythonFoldLevel.
Fold diff files so I can see whole hunks/files and can delete those with a single key (well, two keys -- dd). Useful for reviewing large diffs (tens of thousands of lines): vimrc, function DiffFoldLevel.

Some of these come from www.vim.org, some I've written myself, some I've taken and modified a little bit to avoid an irritating quirk or add a missing feature. Some things I don't have (and envy Emacs or IDE users for having -- like an integrated debugger for Python apps, and, generally, integration with other tools, running in the background).

It's been my plan for a long time to polish my plugins, release them somewhere (github? bitbucket? launchpad?) and upload to vim.org, but as it doesn't seem to be happening, I thought I'd at least put an svn export of my ~/.vim on the web.

Displaying multiline text in Zope 3

2009-12-01T17:52:00+00:00

zope.schema has Text and TextLine. The former is for multiline text, the latter is for a single line, as the name suggests. Zope 3 forms will use a text area for Text fields and an input box for TextLine fields. Display widgets, however, apply no special formatting (other than HTML-quoting of characters like <, > and &), and since newlines are treated the same way as spaces in HTML, your multiline text gets collapsed into a single paragraph.

Here's a pattern I've been using in Zope 3 to display multiline user-entered text as several paragraphs:

import cgi

from zope.component import adapts
from zope.publisher.browser import BrowserView
from zope.publisher.interfaces import IRequest


class SplitToParagraphsView(BrowserView):
    """Splits a string into paragraphs via newlines."""

    adapts(None, IRequest)

    def paragraphs(self):
        if self.context is None:
            return []
        return filter(None, [s.strip() for s in self.context.splitlines()])

    def __call__(self):
        return "".join('<p>%s</p>\n' % cgi.escape(p)
                        for p in self.paragraphs())

View registration

<configure
    xmlns="http://namespaces.zope.org/zope">

  <view
      for="*"
      name="paragraphs"
      type="zope.publisher.interfaces.browser.IBrowserRequest"
      factory=".views.SplitToParagraphsView"
      permission="zope.Public"
      />

</configure>

and usage

<p tal:replace="structure object/attribute/@@paragraphs" />

Update: The view really ought to be registered twice: once for basestring and once for NoneType. I was too lazy to figure out the dotted names for those (or check if zope.interface has external interface declarations for them), so I registered it for "*". You should know that this makes the view available for arbitrary objects (but won't work for most of them, since they don't have a splitlines method), and that it is, sadly, accessible to users who may try to hack your system by typing things like @@paragraphs in the browser's address bar. Ignas Mikalajūnas offers an alternative solution using TALES path adapters.

Maemo Summit 2009

2009-10-11T22:22:00+00:00

The second Maemo Summit is over.

Nokia surprised everyone on the first day by handing out 300 pre-release N900s to the participants. I'm so happy now that after a long period of wavering I finally decided to come to the summit! The device is much better than I expected/feared (and I haven't even put a SIM card in yet). We're supposed to provide feedback and will have to send the devices back to Nokia in 6 months. (Nokia insisted on loan contracts signed in blood, kidding, but there are contracts.)

The tiny pixels are beautiful. It's what, 266 pixels per inch? Even older 225 dpi devices spoiled me: both the first generation iPhone and the first generation Kindle displays seemed very coarse and pixellated.

The user interface is very smooth. Having a composition manager improves apparent responsiveness: even if the app is swapped out and not ready to redraw, switching between windows appears to be instant since the picture is cached. And there's no flicker while the apps are redrawing. (Flickering during redraw is one of the main reasons I did not buy a S60 phone and stayed with good old S40.) Speaking of swapping, it's barely noticeable. You can run more apps than fit in RAM without having to suffer. The flash memory is noticeably faster than in a N810. And there's more of it (32 gigs: 28 gig partition for user data, the rest for the system: swap, applications, config files, etc.)

The design and usability of the user interface have improved a lot since the N810. The UI is pretty. Many of the apps are now convenient to use. Pervasive kinetic scrolling is sweet (except when you have really long lists or web pages, then it takes forever to reach the end).

Finally there are PIM-y things people missed in older Maemo releases: calendaring, contacts that can record all kinds of information (such as phone numbers).

All right, enough gushing. There were some irritating things too. For example, Bluetooth support is buggy/incomplete in the pre-release firmware, so it's hard to transfer files. Calendar/contacts sync with S40 phones does not work either. GPS is utterly useless when you're offline (no maps, or at least I haven't found a way to pre-download and cache them; also very long fix times without network assistance). Since I have no desire to pay extortionist roaming charges of my provider (2.5 EUR per megabyte), and haven't had a chance to go look for a prepaid SIM card, I usually have either WiFi or GPS coverage, but not both.

As you can guess, playing the device diverted a part of my attention from the presentations somewhat. I tried to compensate for that by reporting on the talks on IRC (using xchat on the device). I think the strategy backfired; IRC is rather disruptive and the channel is quite busy lately.

Escaping hotel firewalls with ssh over port 80

2009-10-11T20:09:00+00:00

I booked a stay at a particular hotel because the web page said "Free WiFi". It didn't say "all outgoing ports firewalled except for port 80 and a few other (useless) ones". Not having SSH access is most painful. Luckily, there's a solution.

You need a web server running Apache and SSH. Enable mod_proxy and mod_proxy_connect and add this to the first (i.e. default) virtual host configuration:

<VirtualHost whatever:80>
...

  # allow ssh to localhost over http proxy
  ProxyRequests on
  AllowCONNECT 22
  <Proxy localhost>
    Order allow,deny
    Allow from all
  </Proxy>

</VirtualHost>

Reload Apache configuration. The setup is done. (Instructions based on Tunneling SSH over HTTP(S) by Dag Wieers.)

On the client side you need proxytunnel. Sadly, it's not packaged for Ubuntu yet, but compiling from sources is trivial. Edit ~/.ssh/config and add an entry for your proxied ssh connection:

Host pmyservername
ProxyCommand proxytunnel -q -p myserver.mydomain.com:80 -d localhost:22

That's it. Now you can ssh pmyservername. (The p prefix is a reminder that I'm using a proxied connection: ssh fridge versus ssh pfridge. Also it reminds me of Terry Pratchett's Pyramids.).

For extra fun (e.g. IRC) use ssh's built-in SOCKS5 proxy: ssh -D 1080 pmyservername. Then tell the apps to use a SOCKS5 proxy on localhost. Since telling each app to use a proxy (and then, later, telling it to stop using it) is a big *pain*, and some apps (e.g. ssh) don't support proxies directly, a wrapper like tsocks is handy. Edit /etc/tsocks.conf and set the default socks server to 127.0.0.1, then use it to run apps:

$ tsocks xchat-gnome
$ tsocks bzr push lp:myprojectname

tsocks is packaged for Ubuntu.

If your hotel doesn't have free WiFi, a prepaid SIM card with 3G access could be cheaper than roaming charges. Apparently you can get one with a virtually unlimited (for a short stay, anyway) data plan for 27 EUR in Amsterdam.

Piano improv #2

2009-09-28T02:47:10+00:00

Here's a recording of a recent improvisation. It's very simple, but I'm quite happy with it.

Pylons and SQL schema migration

2009-09-21T16:44:00+00:00

I'm at the point in my hobby project where I'd like to be able to change my models without losing all my test data. And I'm too lazy to do manual dumps and edit the SQL in place before reimporting it.

I want a system

that is transparent to the user: if my database is at schema version 1, and my code is at version 3, I want it to be automatically upgraded to version 3 on server startup.
that is not too hard on the programmer: dropping a numbered Python or SQL script in a directory ought to be sufficient to define a transition from schema version X to schema version X+1.
that handles errors gracefully: makes a backup of the database with the old schema version; runs my script in a transaction and aborts that transaction if the conversion fails (while showing me enough information to debug the problem).
allows prototyping without having to increment the schema number for every little change I make to the models; I should be the one who decides that a new schema is ready to go out to the world.

I've been glancing at SQLAlchemy-Migrate, since I've been brought up to believe NIHing is Bad. But Migrate is scary. I have to admit that the longer I stare at its documentation, the less I can describe why I think so. All those shell commands—but there's an API for invoking them from Python, so maybe I can achieve my goals. I'll have to try and see.

Pylons with zc.buildout, continued

2009-09-15T20:31:00+00:00

Last time I mentioned that running bin/buildout with the -N flag makes it run faster (since it skips looking for newer versions to upgrade). You can tell buildout to do this by default by putting 'newest = false' into the [buildout] section of buildout.cfg. We'll be running bin/buildout a lot now, since we'll be making changes to the project environment, so this will save wear and tear on the '-', 'N' and Shift keys. (And, by the way, I'm not trying to soak up Google juice by repeating the word 'buildout' a lot, honest!)

I will omit bzr commits from this narrative as it's getting long; you can assume that every self-contained change was committed separately.

tests

First, I want a bin/test script to run the test suite. Pylons uses nose, so we need to tell buildout to install the nosetests script (under a different name, since I'm used to typing bin/test no matter what test runner a project happens to use):

$ bzr diff
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-15 19:49:11 +0000
+++ buildout.cfg	2009-09-15 19:49:18 +0000
@@ -8,5 +8,8 @@
 recipe = zc.recipe.egg
 eggs = Pylons
        PasteScript
+       nose
        asharing
 interpreter = python
+scripts = paster
+          nosetests=test

$ bin/buildout
...
Generated script '/tmp/AlliterationSharing/bin/paster'.
Generated script '/tmp/AlliterationSharing/bin/test'.
...
$ bin/test

----------------------------------------------------------------------
Ran 0 tests in 0.276s

OK

ctags

Documentation is good, but sometimes you want to look at the source code of the framework. There's a tool called ctags that builds a database of identifiers. The popular text editors Vim and Emacs can then use the tags database to jump to a definition of any name with a single keystroke (Ctrl-] in vim, M-. in emacs).

Building the tags database is complicated by each Python package being installed into a separate directory. There's a buildout recipe called z3c.recipe.tag that finds those directories and lets you build a unified tags file. We'll also ask buildout to make sure it unzips any packages distributed as .egg files, since ctags doesn't process those:

$ bzr diff
@@ -1,8 +1,9 @@
 [buildout]
 develop = .
-parts = pylons
+parts = pylons ctags
 
 newest = false
+unzip = true
 
 [pylons]
 recipe = zc.recipe.egg
@@ -13,3 +14,7 @@
 interpreter = python
 scripts = paster
           nosetests=test
+
+[ctags]
+recipe = z3c.recipe.tag:tags
+eggs = ${pylons:eggs}

$ bin/buildout
...
Generated script '/tmp/AlliterationSharing/bin/ctags'.
...
$ bin/ctags

omelette

ctags lets you find classes and functions by name; it doesn't let you find packages or modules. There's another recipe, collective.recipe.omelette that creates a tree of symlinks mirroring the Python package structure (here 'unzip = true' also comes in handy):

$ bzr diff
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-15 20:04:42 +0000
+++ buildout.cfg	2009-09-15 20:05:30 +0000
@@ -1,6 +1,6 @@
 [buildout]
 develop = .
-parts = pylons ctags
+parts = pylons ctags omelette
 
 newest = false
 unzip = true
@@ -18,3 +18,7 @@
 [ctags]
 recipe = z3c.recipe.tag:tags
 eggs = ${pylons:eggs}
+
+[omelette]
+recipe = collective.recipe.omelette
+eggs = ${pylons:eggs}

$ bin/buildout 
...
$ ls -l parts/omelette
...

The symlink tree is created under parts/omelette/. For example, if you want to see what webhelper tags were available, you can open parts/omelette/webhelper/html/builder.py in your editor and see.

Makefile

This is getting long (and not everyone may be interested¹), but one long post is easier to skip than five medium ones in a row, so I'll continue.

¹ Sorry, Planet Maemo! There's an RSS feed of posts tagged 'maemo', if you can figure out the URL, which is very well hidden by PyBlosxom, *sigh*.

Wouldn't it be nice if new developers could check out your project and start it up with just a couple of commands? Make is a time-tested tool that works well for this:

$ cat Makefile
# Just remember that you need to use real tabs, not spaces, in a Makefile

PYTHON = python

.PHONY: all
all: bin/paster

.PHONY: run
run: bin/paster
        bin/paster serve development.ini --reload

.PHONY: test check
test check: bin/test
        bin/test

.PHONY: tags
tags: bin/ctags
        bin/ctags

bin/paster bin/test bin/python bin/ctags: bin/buildout
        bin/buildout

bin/buildout: bootstrap.py
        $(PYTHON) bootstrap.py

Now all you need to do after checking out is run 'make' to set up a working development environment. 'make run' or 'make test' will also do that, if necessary, so this one-liner is sufficient to get a working Hello World application on port 5000:

$ bzr branch lp:~mgedmin/+junk/AlliterationSharing && cd AlliterationSharing && make run

Try it! You'll get a Bazaar branch with all the history of this little blog project.

Starting a Pylons project with zc.buildout

2009-09-13T13:13:00+00:00

For software development I prefer buildout to virtualenv. This is because buildout has a text file describing the state of your working environent, which can be versioned and used later to recreate it, as well as during development to modify the environment slightly.

To start a new Pylons project, first create an empty directory. Let's call our new project AlliterationSharing¹, because everybody is sick of 'foo' and 'bar'.

¹ Generated by randomly picking two words from /usr/share/dict/words, then chosen over among 120 other variants that weren't as good.

$ mkdir -p ~/src/AlliterationSharing
$ cd ~/src/AlliterationSharing

Now create a file called buildout.cfg with the following content:

$ cat buildout.cfg
[buildout]
parts = pylons

[pylons]
recipe = zc.recipe.egg
eggs = Pylons
       PasteScript
interpreter = python

Download bootstrap.py to it and run it to get bin/buildout. Note: you can chose which Python version you want to use by running bootstrap.py with it. All other scripts under bin/ will be generated by buildout and will use the same Python interpreter.

$ wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
$ python bootstrap.py
Creating directory '.../AlliterationSharing/bin'.
Creating directory '.../AlliterationSharing/parts'.
Creating directory '.../AlliterationSharing/eggs'.
Creating directory '.../AlliterationSharing/develop-eggs'.
Generated script '.../AlliterationSharing/bin/buildout'.

Run bin/buildout to install Pylons into your sandbox.

$ bin/buildout
Installing pylons.
Generated script '.../AlliterationSharing/bin/paster'.
Generated interpreter '.../AlliterationSharing/bin/python'.

Aside: buildout has this very nice feature where it can share Python packages between projects. This will save you enormous amounts of time that would otherwise be spent downloading and unpacking eggs. To make use of this facility, create a file ~/.buildout/default.cfg with

$ cat ~/.buildout/default.cfg 
[buildout]
eggs-directory = /home/mg/tmp/buildout-eggs
# XXX replace /home/mg with the full path of *your* home directory
# it would be much nicer if buildout let me use ~ or $HOME
# see https://bugs.launchpad.net/zc.buildout/+bug/190260

Another useful trick is to pass the -N flag to bin/buildout, which will tell it not to bother looking for newer versions of packages on the Internet when there's already an existing version installed in your eggs directory.

Back to business: now you've got two new scripts: bin/python and bin/paster. You can use the first one to play with the interactive Python console where you can now import pylons and all the dependencies; it has no other value.

Now is a good point to add the files you've created into a version control system. I'll arbitrarily use Bazaar.

$ bzr init .
$ bzr add bootstrap.py buildout.cfg
$ bzr ignore bin parts eggs develop-eggs .installed.cfg
$ bzr commit -m "Create AlliterationSharing project"

Run bin/paster create -t pylons to create a skeleton project.

$ bin/paster create -t pylons asharing
$ bzr ignore *.egg-info
$ bzr add asharing
$ bzr commit -m "Generated project files with paster create"

Now paster creates a directory structure that I don't like:

AlliterationSharing/
  buildout.cfg
  bin/
  asharing/
    setup.py
    README.txt
    MANIFEST.in
    asharing/
      __init__.py
      config/
      controllers/
      templates/
      public/

I'd like the README and setup.py to be in the top level, and I dislike repeating 'asharing' twice in directory names. I'll move some files around

$ cd asharing/
$ bzr mv development.ini docs MANIFEST.in README.txt setup.* test.ini ../
$ bzr rm ez_setup.*
$ cd ..
$ bzr mv asharing src
$ bzr ci -m "Moved some files around"

Now the tree looks like this:

AlliterationSharing/
  buildout.cfg
  setup.py
  README.txt
  MANIFEST.in
  bin/
  src/
    asharing/
      __init__.py
      config/
      controllers/
      templates/
      public/

We have to tell setup.py where to find the source tree

$ bzr diff
=== modified file 'MANIFEST.in'
--- MANIFEST.in	2009-09-13 13:04:00 +0000
+++ MANIFEST.in	2009-09-13 13:05:59 +0000
@@ -1,3 +1,3 @@
-include asharing/config/deployment.ini_tmpl
-recursive-include asharing/public *
-recursive-include asharing/templates *
+include src/asharing/config/deployment.ini_tmpl
+recursive-include src/asharing/public *
+recursive-include src/asharing/templates *

=== modified file 'setup.py'
--- setup.py	2009-09-13 13:04:00 +0000
+++ setup.py	2009-09-13 13:04:40 +0000
@@ -17,7 +17,8 @@
         "SQLAlchemy>=0.5",
     ],
     setup_requires=["PasteScript>=1.6.3"],
-    packages=find_packages(exclude=['ez_setup']),
+    packages=find_packages('src', exclude=['ez_setup']),
+    package_dir={'': 'src'},
     include_package_data=True,
     test_suite='nose.collector',
     package_data={'asharing': ['i18n/*/LC_MESSAGES/*.mo']},

(I'm not sure if you also need to change package_data and/or setup.cfg; it's possible that I left i18n in a broken state. Can somebody comment on this?)

And we have to tell buildout that we've got a new Python package to enable in the project environment

$ bzr diff buildout.cfg 
=== modified file 'buildout.cfg'
--- buildout.cfg	2009-09-13 12:57:21 +0000
+++ buildout.cfg	2009-09-13 13:08:05 +0000
@@ -1,8 +1,10 @@
 [buildout]
+develop = .
 parts = pylons
 
 [pylons]
 recipe = zc.recipe.egg
 eggs = Pylons
        PasteScript
+       asharing
 interpreter = python

Now you can re-run bin/buildout and start your hello-world project

$ bzr commit -m "Include the new package in the build"
$ bin/buildout -N
$ bin/paster serve --reload development.ini

Happy hacking!

To be continued: telling buildbot to create bin/test; using ctags and omelette.

Footnotes done well

2009-09-11T22:58:00+00:00

I like the way footnotes are implemented here: Snakes on the Web by Jacob Kaplan-Moss.

(Recorded with byzanz. My gif-fu is nonexistent or I would make it loop, but with a sufficiently long delay at the end to avoid irritation. Now you have to reload the whole page if you missed the animation.)

I'm somewhat ambivalent about the animation effect. On one hand, shiny! On the other hand, hitting tiny clickable areas is not good usability. Still, shiny!

Footnotes are kind of a personal pet-peeve of mine.

Local changes to buildout.cfg

2009-08-03T17:26:00+00:00

Most of Python packages in the Zope world use Buildout:

svn co svn+ssh://svn.zope.org/repos/main/plone.z3cform/trunk plone.z3cform
cd plone.z3cform
python2.4 bootstrap.py
bin/buildout
bin/test -pvc

Now suppose you want to change the buildout environment somehow, e.g. use the current development version of zope.testing instead of whatever is specified in buildout.cfg. Don't edit the existing buildout.cfg (you might accidentally commit your local debug changes), instead create a new cfg file, e.g. test.cfg:

[buildout]
extends = buildout.cfg
develop += ../zope.testing

[versions]
# override any existing version pins
zope.testing =

Now re-run buildout

bin/buildout -c test.cfg
bin/test -pvc

And the tests should be run with the newest zope.testing.code.

Only this does not work with plone.z3cform, and I have no clue why. It generally works with other packages (at least those that use the zc.recipe.testrunner rather than collective.recipe.z2testrunner). Buildout is like that sometimes :(

Python-related updates for the last couple of months

2009-07-24T23:14:00+00:00

Went to EuroPython, met new people, had a great time.

Updated gtkeggdeps, the interactive Python package dependency browser. Collaborated with Thomas Lotze, who maintains the engine (tl.eggdeps) that gtkeggdeps wraps, to resolve API mismatches. Moved the sources to launchpad.net, added a test suite, made it use zc.buildout for convenient development.

Moved the source repository of gtimelog, the simple desktop time tracker, to launchpad.net. Failed to do anything else with it. :-(

Tried to work on xdot, wrestled with git-svn merges, failed abysmally. Asked upstream to upload xdot to PyPI.

Released ZODB Browser, but this deserves a separate post.

Sent a bunch of pyflakes patches from my old branch upstream, created trac tickets for the rest. Wrestled with bzr-svn merges, failed abysmally.

df

2009-07-14T22:33:00+00:00

Modern Linux system have all sorts of fake filesystems cluttering the output of df and mount: tmpfs, bind mounts, fuse for ~/.gvfs, etc. I have only one real partition on my laptop, yet mount returns 22 lines of output.

Question: are there any df-like utilities that filter out all the crap and show only interesting bits? The standard df as well as pydf both display 8 lines instead of 1. Discus is worse: it shows 20. GUI utilities like Baobab also suffer from this confusion, especially bind mounts.

Ironically, Ubuntu's update-motd gets confused by Ubuntu's private user directories and displays disk stats for ~/Private as if it were a real partition.

Surprising old-style class behaviour

2009-05-21T19:12:00+00:00

Some anonymous Planet Python poster (at least I couldn't find the author's name on the blog) Christian Wyglendowski asks about a surprising difference between old-style and new-style classes. Since the comments on their blog are closed (which you find out only after pressing Submit), I'll answer here.

The question, slightly paraphrased: given a class

class LameContainerOld:
    def __init__(self):
        self._items = {'bar':'test'}
 
    def __getitem__(self, name):
        return self._items[name]
 
    def __getattr__(self, attr):
        return getattr(self._items, attr)

why does the 'in' operator work

>>> container = LameContainerOld()
>>> 'foo' in container
False
>>> 'bar' in container
True

when the equivalent new-style class raises a KeyError: 0 exception? Also, why does __getattr__ appear to be called to get the bound __getitem__ method of the dict?

>>> container.__getitem__
<bound method LameContainerNew.__getitem__ of {'bar': 'test'}>

What actually happens here is that LameOldContainer.__getattr__ gets called for special methods such as __contains__ and __repr__. This is why (1) the 'in' check works, and (2) it appears, at first glance, that you get the wrong __getitem__ bound method. If you pay close attention to the output, you'll see that it's the __getitem__ of LameOldContainer; it's just that repr(LameOldContainer()) gets proxied through to the dict.__repr__ when you don't expect it:

>>> container
{'bar': 'test'}

Special methods never go through __getattr__ for new-style classes, therefore neither __contains__ nor __repr__ are proxied if you make the container inherit object. If there's no __contains__ method, Python falls back to the sequence protocol and starts calling __getitem__ for numbers 0 through infinity, or until it gets an IndexError exception.

Enabling comments in PyBlosxom

2009-05-16T03:13:00+00:00

I've just spent the whole night setting up blog comments. PyBlosxom doesn't make it painless, sadly, more like the opposite.

First: don't be scared by the list of comment-related plugins on the PyBlosxom site. There's only one important plugin: comments. All others depend on it and enhance its functionality. The last three or four times I was about to add comments to my blog I got scared at step one: evaluate the available plugins. Don't repeat my mistake!

Second, follow the instructions carefully. There's no shortcut.

Third, fix what's broken. Be prepared to debug the source code. print >> sys.stderr, "message" is your friend.

Fourth, fiddle with the look (CSS and HTML).

Fifth, write a blog post and eagerly await your first comments.

Step 3 screams for an explanation, doesn't it? Problem 1: the comments plugin requires that you use categories in your blog. I'm not (I'm holding out for tags). Workaround: comment out if entry['absolute_path'] check in cb_story and cb_story_end.

Problem 2: the AJAX post returns "Empty response from server". Workaround: modify cb_story_end to call readComments directly if entry['num_comments'] is None, since cb_story, which usually does the read, is not called during the AJAX post.

Problem 3: if you enable comment moderation (by setting comment_draft_ext to a different value from comment_ext), the AJAX post returns "Empty response from server" once more. Workaround: modify cb_prepare to notice this case and set data['moderated'] = True, create a new template comment-moderated and render it in cb_story_end just like the preview template is rendered; also modify __shouldOutput to return True when rendering comment-moderated.

I'll post patches to the pyblosxom mailing tomorrow, unless I forget. It's 6 am already, and I'm kind of sleepy. I just hope I haven't inadvertently broken my RSS feed or flooded any planets.

Oh, and a helpful hint: don't name the post you're writing comments.txt, or the #comments anchor will point to the start of the story instead of the comments.

Buildbot issues on Ubuntu Hardy

2009-05-15T12:33:00+00:00

Update: The story continues, but solution is not in sight yet.

I upgraded a buildbot slave to Ubuntu 8.04 (Hardy) recently and now I'm getting a strange intermittent failure: sometimes cp -r /local/dir /nfs/mounted/dir fails ("process killed by signal 1", i.e. SIGHUP).

I wonder if NFS is relevant or incidental to the issue?

Google finds an old thread from 2005, with a workaround (usepty=False), but I'd like to understand the problem before applying random fixes.

So far three different build steps doing cp -r have failed during 10 days. I've now changed them all to cp -rv, so I can at least see if the failure is in the middle of the copy or at the end, if it fails again.

Update: so far 4 build steps have failed on 6 separate occasions:

May  5 02:31: cp -r local-dir1 nfs-mounted-dir1  
May  6 02:31: cp -r local-dir1 nfs-mounted-dir1  
May  6 04:33: cp -r local-dir2 nfs-mounted-dir2  
May 15 02:00: cp -r local-dir3 nfs-mounted-dir3  
May 17 04:32: rm -rf nfs-mounted-dir4            
May 20 04:31: rm -rf nfs-mounted-dir4

I see no particular correlation between step duration and results, e.g. the rm -rf step usually takes between 2.2 and 4.6 seconds. The two SIGHUPs happened after 2.4 seconds.

They all make no output. When I changed the cp steps and added a -v, they stopped failing, but that could be just a coincidence.

We're having an email conversation with Jean-Paul Calderone ("exarkun") about the possibility of this being PTY-related, with no clear resolution so far.

And, hey, now this blog supports comments ;)

Lęšis.lt

2009-05-14T02:21:05+00:00

A friend of mine has recently launched a contact lens e-shop, lesis.lt. It only ships in Lithuania at the moment, but if you're based here, have a look, you might find it useful.

Expert Python Programming

2009-05-08T03:09:00+00:00

It's been a while since the last Expert Python Programming review on Planet Python. Y'all might've forgotten about this book by now. Time for a reminder? (Actually, I'm just lazy busy, and this is why this review hasn't appeared sooner.)

I received a free PDF copy of this book from Packt Publishing, with the understanding that I'll post a review on my blog. This is it. Short summary: it is a good book marred by a lot of mostly inconsequential little mistakes. I'd give it four stars out of five.

Aside: the PDF that I could download was personalized and had my name and address in the footer of every page. A very nice form of DRM that did not restrict my software choices for reading the book (Evince and also PDF Reader on Nokia Internet Tablets).

I bring it up here because it seems that Packt could've also applied fixes for the known errata to the personalized version, yet missed that opportunity. Perhaps it's technically more difficult than slapping a footer on every page. Or maybe it's better if everyone buying the book, whether in paper or in PDF, gets to see the same text.

The author (Tarek Ziade) covers a wide range of topics in the book, ranging from syntax (probably useful for those who've been programming in Python for quite a few years, and didn't have the time to keep up with the language changes before picking up this book) to style, source code organization, project infrastructure, software life cycle, documentation, testing and optimization, and finally ending with a review of some of the popular design patterns. The middle parts were the most interesting for me personally. I learned a thing or two, disagreed with the author on a few minor points (which are mostly a matter of preference), and managed to finish the book despite constant irritating little pricks I feel when I notice an error (I confess I'm a pedant. A missing space after a colon drives me up the wall).

As an example of the disagreement: I have an aversion to code-generating tools where you have to edit the generated code by hand. I could say more, but this is a topic for another time. Next, I strongly dislike sudo easy_install since it scribbles onto the part of the filesystem exclusively reserved for your OS's package management tools. And I don't think porting the original 23 design patterns to other programming languages is a good way to describe what those languages are about. (Also, set tabstop=4 in your .vimrc? Heresy! The Right Thing To Do is set softtabstop=4, as all right-thinking Vim users will doubtlessly agree. All hail the one true text editor! Oh dear, now I'm glad I don't have comments on this blog...)

The goodies: Chapter 1 (the bits about PYTHONSTARTUP on page 19) gave me persistent history for my interactive Python prompt, nicely complementing the coloured prompt and tab-completion I already had snarfed from somewhere else on the net (probably Peter Norvig's Python IAQ). Chapter 12 provided good examples of how to do profiling for time (page 281) and memory (page 291). I like Tarek's @profile decorator (measure time, pystones and memory at the same time). My profilehooks module was not mentioned, *sniff* ;-). Chapter 13 told me about Queue.join and task_done that snuck into the stdlib with Python 2.5 without me noticing.

I haven't mentioned topics covered in the book that I was already familiar with, such as setuptools, virtualenv, zc.buildout, Sphinx, Nose, Buildbot, or Mercurial. Yet, in my opinion, those are the most useful parts of the book. The breadth of the topics is amazing: I could hardly think of something that every serious Python programmer should know that isn't wasn't mentioned. I believe the depth was exactly right: mention solutions that are available, show how they feel when used and what they can do, point to the relevant web page and then stop. And not only tools, the descriptions of workflows (how to organize your source trees, how to develop software consisting of multiple packages, how to make releases), while hardly universal, are invaluable.

One thing prevents this from being a perfect book: errata. At around page 95, according to my notes, I invented a new metric of book quality: WTFs per page, It's closely related to WTFs per minute, but independent of your reading speed. At around page 165 I got tired of making a note of every little thing that I noticed and started just reading. This was considerably more enjoyable. I hope there's a second edition will all the bugs shaken out. To that end, I should go through my notes again and submit them via the online errata form. Yay, more work...

Piano improv

2009-04-15T16:40:39+00:00

My today's experiments on piano:

Filing bugs on GUI apps

2009-04-09T19:29:00+00:00

Ubuntu people want users to use apport to report bugs. There's a command-line tool called 'ubuntu-bug' that you can use if you know the name of the package or at least the name of executable. There's a "Report a problem" menu item in many, but not all GUI apps.

Here's what you can do if the GUI app in question doesn't have that menu item, and you don't remember what it's called, and you're the same sort of a crazy command-line person that I am:

$ xprop|grep PID

Click on the app's window. Watch that shell command return a line that looks like

_NET_WM_PID(CARDINAL) = 807

Now run

$ ps 807  # substitute the real number

You'll see the command name, e.g.

  PID TTY      STAT   TIME COMMAND
  807 ?        S      0:02 /usr/lib/indicator-applet/indicator-applet --oaf-acti

Now you can run

$ ubuntu-bug /usr/lib/indicator-applet/indicator-applet

If you're not running Jaunty, you'll need to do one more step to find the name of the Ubuntu package:

$ dpkg -S /usr/lib/indicator-applet/indicator-applet
indicator-applet: /usr/lib/indicator-applet/indicator-applet

You can use that with apport-cli or on the Launchpad online bug reporting form.

For some (many?) programs you can short-circuit this trail by looking at the WM_CLASS property instead of the _NET_WM_PID property:

$ xprop|grep CLASS
WM_CLASS(STRING) = "indicator-applet", "Indicator-applet"

While there is no requirement for the window class name to match the name of the Ubuntu package or the name of the executable, it may give you a reasonable guess.

Upgrade to Ubuntu Jaunty

2009-04-05T23:28:00+00:00

Ubuntu 9.04 is going to be released in around three weeks. As usual I couldn't wait (and saw that some bugs that were irritating me every day were fixed in Jaunty), so I upgraded to the current beta.

After a little hiccough at the beginning, the upgrade was the smoothest Ubuntu upgrade I've ever had: I spent those two and a half hours browsing the web, watching screencasts and chatting on IRC, while update-manager worked in the background. Firefox was mostly very responsive, only stuttering when update-manager got around to unpacking openoffice.org-common. There were few debconf or conffile questions (one from sysstat 2 minutes into the upgrade, then a conffile question after 1 hour and 20 minutes, then two more after 15 minutes, and one more 5 minutes later. And the last one 10 minutes later). There were no ugly theme changes or failing GNOME applet error messages during the upgrade. Near the end X-Chat automatically started showing new-style notifications (beautiful!) and Firefoxes nicely asked to be restarted with a fold-down notification bar.

Nice. Now, after a reboot things were not so nice: I couldn't login. After typing in my password and a couple of mode changes I was kicked back to the GDM prompt. I panicked and started logging into the text consoles and trying to run startx, quite in vain, since when I just tried gdm again it worked fine.

The intel video driver feels slower, as promised by the release notes, but it's acceptable as long as you don't try to rotate the external screen. Then it's horrible and unusable—a regression since Intrepid. I'll have to retry with UXA.

Compiz failed to enable a plugin (GNOME Compatibility), so a couple of key bindings didn't work (Alt+F1 to get the menu, Alt+F2 to get the run dialog, my custom keybinding to open a terminal) until someone on FreeNode told me what to enable.

The Flash plugin is now swfdec, and it is unable to cope with Youtube music videos—the sound is all choppy. I'm wondering if this is swfdec's fault, pulseaudio's fault (it's common knowledge that all audio problems stem from pulseaudio, right? ;) or X.org's fault (top shows it's X that's eating 90% CPU when swfdec is trying to play a video).

A lot of very irritating bugs are gone. I don't need to restart Compiz after playing with xrandr. X doesn't crash after I play with xrandr. Two-finger scrolling with the Synaptics touchpad doesn't produce phantom scroll-down-17-pages events when I take my fingers off the touchpad. The GNOME panels don't migrate to the external screen when I play with xrandr (but one of them jumps from the bottom to the top when I play some more). The new splash screen has a pretty gradient for its progress bar (but is displayed off-center, maybe because I added vga=872 to my GRUB kernel options list to avoid ugly stretching of text consoles). X.org no longer distorts the aspect ratio of 1024x768 when stretching it to fit the 1280x800 screen—now I get sensible black bars on the sides. The new notification bubbles are beautiful!. I could stare at them all day. (But the new indicator applet is ugly.)

Overall I'm happy. A bunch of very irritating bugs were replaced with a smaller bunch of somewhat less irritating bugs. The intel video slowdown scares me a bit, though, but the prettiness of the notification bubbles outweigh everything else. What can I say, I like pretty things—Ooh, shiny!

Submitting patches the Launchpad way

2009-04-05T21:45:00+00:00

Today I happened to read about lazr.enum in a mailing list. I went to the PyPI page and saw raw ReStructuredText markup instead of a nicely formatted page. Now I know from prior experience that this happens when the package's description has an error in the markup. I thought I'd report a bug and provide a patch.

Leap of knowledge: since I know lazr.enum was created by the Launchpad.net team I could safely assume they were keeping the sources in Launchpad. Therefore I was pretty sure I could get them with

$ bzr branch lp:lazr.enum

so I ran that command and it worked.

Next I looked at setup.py to see how it produces the long_description field. It was reading the contents of a couple of text files, one of them being src/lazr/enum/README.txt. I looked at that and saw a .. toc-tree: directive that does not exists in plain docutils (it's a Sphinx extension).

I added up a couple of lines to setup.py to strip that out, tested it (with setup.py --long-description > test.rst; restview test.rst) committed to my local branch, and created a bug report in Launchpad. Then I was a bit lost, since I didn't know how to make my fix available. Attach a patch? Maybe, but I wanted to see if this distributed version control thing is good for anything else.

I thought that first I'd make that branch public, and then see if there was a way to link it to the bug report. I ran

$ bzr push lp:~mgedmin/lazr.enum/pypi-fix

which took a few seconds to create a new public branch on Launchpad with my fix in it (it would be nice if I didn't have to explicitly specify my Launchpad username and the project name—both of which bzr already knows—and just specify the name of the branch). Then I went back to my bug report and saw an option to link it to a branch. There was a search field in the popup that found my "~mgedmin/lazr.enum/pypi-fix" easily enough when I pasted it into the search box.

After clicking on the branch, I saw a "propose a merge" option. I did that and Launchpad sent an email to the developers asking them to merge my fix.

I made one mistake, I think: I should've created the bug report first, and then mentioned the bug number in my commit message (with bzr commit --fixes=NNN, although here I'm suddenly not sure if the bug number should be left bare, or prefixed with something like "lp" to indicate it was a Launchpad bug number?).

Other than that it was a pretty smooth experience. When will I be able to do that for Ubuntu packages?