EuroPython 2008 was fun. I presented two talks (My God, it's Full of Files -- Pythonic filesystem abstractions and Version Control for Du^H^HDevelopers) and one lightning talk (RST+S5 for your slides), participated in a bunch of open space sessions, listened to about 13 talks, took a bunch of pictures, but most importantly had interesting hallway conversations with interesting people.

As usual, PyPy was heavily represented, and seems to be making nice progress toward being the nice and featureful default Python implementation of the future. I especially liked the restricted execution features and the LLVM backend. The zc.buildout talk made me decide I will try to replace (part of?) one custom deploy mechanism with zc.buildout -- most likely I'll end up rewriting most of the current things as zc.buildout recipes, but hopefully some of the pre-existing recipes will be useful, and hopefully I can then later reuse the recipies I create for this setup.

Personally, I think my talks went ok. I understand videos will be available later, as soon as transcoding etc are finished. I'm anxious to see them myself, as I'm still finetuning my public speaking skills. I'm learning, though -- this year I had no trouble staying within my time slot, even when I was adjusting verbosity on the fly.

For some reason, I felt underprepared for the filesystem API talk, but ultimately people liked the idea of a consistent Pythonic filesystem API enough that we had an open space session on it, and people were enthusiastic about a sprint to prototype the API. Which is what we ended up doing, too -- I'll blog separately about the results of that.

My decentralized version control talk seemed to me to go over more smoothly; I guess that's just because I've been thinking about version control and project management a lot lately, so it was easy to talk about the topic in a relaxed way. On the other hand, it wasn't as much a call to action, and it really was overly generic, so I didn't get as strong audience participation there. We did have an interesting conversation about branch management strategies and such, though. I consciously tried to keep the talk on a generic level, as I felt a pure git talk would have alienated some listeners, but I did end up feeling restricted by that. There was some interest on a Teach me git -style session, but what we ended up doing was just talking one on one about getting started with git, during the sprints. Sorry if I missed any one of you -- grab me on #git to continue, or find me in future conferences ;)

I was requested to organize an open space session for Twisted Q&A, and that is exactly what we did. We went through a bunch of things related to asynchronous programming concepts, Deferreds, working with blocking code and libraries, database interfaces, debugging and unit testing.

I was also pulled in to another Twisted open space session, that was mostly about what greenlets are and how to use them. I tried to explain the differences between classical Deferreds, deferredGenerator/inlineCallbacks, and greenlets, to the best of my understanding. As a summary, with greenlets any function you call can co-operatively yield execution (I mean yield in the scheduling meaning, giving away your turn to run, not in the Python generator meaning -- interestingly inlineCallbacks etc actually make those be the same thing... my kernel instincts make me want to say "sleep"). Yielding in any subroutine means anything you do may end up mutating your objects -- which is the root evil behind threading we wanted to get away from. All the other mechanisms keep the top-level function in explicit control of yielding. Around that time, most people left for lunch, but about three of us stayed and talked about debugging Deferreds and network packet processing with twisted.pair and friends.

One of the interesting hallway conversations was about what happens when upstream web hosting listed on PyPI is failing. It seems PyPI already does some sort of mirroring, but even that might not be enough. Many companies seem to be bundling eggs of their dependencies in their installation package, which sounds like a good setup for commercial click-to-install deployment. But it would still be good to see a CPAN-style mirror network for PyPI, and at least some people seemed even motivated to donating servers and bandwidth. Personally, I'm mostly spoiled by the combination of Debian/Ubuntu and decentralized version control, and my level of paranoia is too high to automatically install unverified software from the internet anyway. My primary motivation in the conversation was to point out that PyPI already has some sort of mirroring/upload setup, and that you'd really want to specify exact versions and SHA-1 hashes of your dependencies. Optionally, you could delegate the known good hash storage to PyPI (assuming you trusted PyPI not to attack you), but that would require a full Debian-style signature chain from a trusted key, or you'd be owned by anyone capable of MITM attacks, DNS forgery, or cracking a PyPI mirror.

I published slides for my first Europython talk. Note the slides might not easy to understand without the actual talk -- the video streaming at Ustream will work if the network infrastructure can take it, and a downloadable video should be available later.

My God, it's Full of Files

Pythonic filesystem abstractions: An overview of different filesystem(-like) APIs in Python and attempts for unifying them.

There's a lot of different filesystem(-like) APIs in Python. I intend to provide an overview of existing projects, their status and capabilities, and hopefully inspire you to work on improving things.

Read more...

So Google has their MapReduce, and the people behind CouchDB are throwing around their ideas. I spent some time thinking about incremental mapreduce around July, and it's time I type out that page full of scribbles.

First of all: I think the ideas thrown out by Damien above aren't really mapreduce, As Google Intended. The real power of mapreduce is in its inherent combination of parallelism and chainability, output of one mapreduce is input to another, each processing step can run massively in parallel with each other, etc. The proposed design is like a one-iteration retarded cousin of mapreduce.

With that bashing now done (sorry), here's what I was thinking:

The way I imagined building an incremental mapreduce mechanism, without storing the intermediate data and just recomputing chunks that are out-of-date (which would be lame), is to add one extra concept into the system: call it "demap". It will basically create "negative entries" for the old data. This is basically what Damien did by providing both the old and new data map calls, all the time, just said differently, and I think my way might make the average call a lot simpler. And I don't see any reason why my version wouldn't be parallelizable, chainable, and generally yummy.

Read the article

For years now, my primary machine has been a laptop. I've been avoiding running Xen [1] because the hypervisor it injects between hardware and Linux hasn't been very friendly for power saving. Modern laptops are too often hot enough without any extra help.

Well, I'm happy to say that as of now, KVM is definitely stable enough to replace my use of Xen for test setups. I expect that shortly it will become what I recommend for server use, too. And because it lets Linux be Linux, instead of doing any oddities with the hardware, all the powersaving etc goodness still works perfectly.

So far, I've stumbled on two things:

1. The version of KVM in Ubuntu, even gutsy, is just too old. With a bit of fiddling with the patches, KVM v46 built a perfectly working deb.

2. The isolinux graphical bootup, used in Ubuntu, crashes KVM (and with v28, it crashes the host machine -- beware!). See the bug report. I got around that by using mini.iso, but you could always fall back to debootstrap; that was all we ever really had with Xen.

And while I have the soapbox: I have a dream. It's KVM running with SDL/VNC graphics in a window that's resizeable all the way to the virtual machine, with XRandR. Please make it happen!

Patches

So, the edited version of the patches:

• 01_use_bios_files_in_usr_share_kvm.patch
• 06_no_system_linux_kvm_h.patch
• from-debian-qemu/04_do_not_print_rtc_freq_if_ok.patch

And just remove from-debian-qemu/62_linux_boot_nasm.patch, it seems to have made it upstream.

A brief update of things I've been working on:

Snakepit

Snakepit is a port of (part of) HiveDB to Python and SQLAlchemy. It will help you write database-backed applications that need to scale further than one database server, or even a master-slave setup, can take you. It's MIT licensed, that is, pretty much as free-for-all as it can be. And it's still work in progress, so don't be too harsh yet ;)

Read the README, browse the gitweb, or grab the source with git clone git://eagain.net/snakepit.git

gitosis

gitosis aims to make hosting git repos easier and safer. It manages multiple repositories under one user account, using SSH keys to identify users. End users do not need shell accounts on the server, they will talk to one shared account that will not let them run arbitrary commands. gitosis is licensed under the GPL.

First real release will come as soon as I have to time to go through a couple of really minor nits. It's been self-hosting for a long time now.

For more, read the README, browse the gitweb, or grab the source with git clone git://eagain.net/gitosis.git

Updated to drop --use-separate-remote from git clone, it's the default.

Updated to add --read-only to git-shell-enforce-directory.

I've run repeatedly into cases where I want to provide services to people without really trusting them. I do not want to give them shell access. I don't want to even create separate unix user accounts for them at all. But I do want to make sure the service they use is safe against e.g. password sniffing.

Instead of trying to run the version control system over HTTPS (like Subversion's mod_dav_svn that will only work with Apache, which I don't run), I want to run things through SSH. SSH is the de facto unix tool for securing communications between machines.

Now, I said I don't want to create a unix user account for every developer using the version control system. With SSH, this means using a shared account, usually named by the service it provides: svn, git, etc. To identify different users of that account, do not give the account a password, but use SSH keys instead. To avoid giving people full shell access, use a command="..." when adding their public key to ~/.ssh/authorized_keys.

For Subversion, I submitted an enhancement to add --tunnel-user, to make sure the commit gets identified as the right user, and then used command="..." with the with arguments, like this (all on one line):

command="/srv/example.com/repo/svn/svnserve -t
  --root /srv/example.com/repo/svn/view/examplegroup
  --tunnel-user jdoe" ssh-rsa ... jdoe@example.com

Where the view directory is a bunch of symlinks to the actual repositories, allowing me to do group-based access control.

With git, the author of the changeset is recorded way before the SSH connection is opened. Without building some sort of access control in git hooks on the server, every developer can pretty much ruin the repository by overwriting branches with bogus commits. What they will not have is access outside of the repository, or a way to actually remove the old commits from the disk (unless you run git prune on the server). The distributed nature of git makes this reasonably easy to detect, and pretty much trivial to recover from. For any real trust in the code, you should look at signed tags anyway. The included wrapper allows you to have read-only users, but provides no detailed access control against developers with write access; they just won't be able to escape to the rest of the filesystem.

So, with that introduction out of the way, let's get to configuring:

1. Install git on the server:

   sudo apt-get install git-core git-doc
   

2. Create the directory structure store the repositories and related files:

   sudo install -d -m0755 \
        /srv/example.com/repo/git \
        /srv/example.com/repo/git/.ssh \
        /srv/example.com/repo/git/repos \
        /srv/example.com/repo/git/view
   

3. Create the shared user account for this service:

   sudo adduser \
        --system \
        --home /srv/example.com/repo/git \
        --no-create-home \
        --shell /bin/sh \
        --gecos 'git version control' \
        --group \
        --disabled-password \
        git
   

4. Set up a script that makes sure only relevant git commands can be run via SSH, and to limit the visible section of the filesystem to things you actually want to give access to; put this file in /usr/local/bin/git-shell-enforce-directory (download) and chmod a+x it

   #!/usr/bin/python
   
   # Copyright (c) 2007 Tommi Virtanen <tv@eagain.net>
   #
   # Permission is hereby granted, free of charge, to any person
   # obtaining a copy of this software and associated documentation files
   # (the "Software"), to deal in the Software without restriction,
   # including without limitation the rights to use, copy, modify, merge,
   # publish, distribute, sublicense, and/or sell copies of the Software,
   # and to permit persons to whom the Software is furnished to do so,
   # subject to the following conditions:
   #
   # The above copyright notice and this permission notice shall be
   # included in all copies or substantial portions of the Software.
   #
   # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
   # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
   # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
   # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
   # BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
   # ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
   # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
   # SOFTWARE.
   
   # Enforce git-shell to only serve repositories
   # in the given directory. The client should refer
   # to them without any directory prefix.
   # Repository names are forced to match ALLOW.
   
   import sys, os, optparse, re
   
   def die(msg):
       print >>sys.stderr, '%s: %s' % (sys.argv[0], msg)
       sys.exit(1)
   
   def getParser():
       parser = optparse.OptionParser(
           usage='%prog [OPTIONS] DIR',
           description='Allow restricted git operations under DIR',
           )
       parser.add_option('--read-only',
                         help='disable write operations',
                         action='store_true',
                         default=False,
                         )
       return parser
   
   ALLOW_RE = re.compile("^(?P<command>git-(?:receive|upload)-pack) '[a-zA-Z][a-zA-Z0-9@._-]*(/[a-zA-Z][a-zA-Z0-9@._-]*)*'$")
   
   COMMANDS_READONLY = [
       'git-upload-pack',
       ]
   
   COMMANDS_WRITE = [
       'git-receive-pack',
       ]
   
   def main(args):
       os.umask(0022)
   
       parser = getParser()
       (options, args) = parser.parse_args()
       try:
           (path,) = args
       except ValueError:
           parser.error('Missing argument DIR.')
       os.chdir(path)
   
       cmd = os.environ.get('SSH_ORIGINAL_COMMAND', None)
       if cmd is None:
           die("Need SSH_ORIGINAL_COMMAND in environment.")
   
       if '\n' in cmd:
           die("Command may not contain newlines.")
   
       match = ALLOW_RE.match(cmd)
       if match is None:
           die("Command to run looks dangerous")
   
       allowed = list(COMMANDS_READONLY)
       if not options.read_only:
           allowed.extend(COMMANDS_WRITE)
   
       if match.group('command') not in allowed:
           die("Command not allowed")
   
       os.execve('/usr/bin/git-shell', ['git-shell', '-c', cmd], {})
       die("Cannot execute git-shell.")
   
   if __name__ == '__main__':
       main(args=sys.argv[1:])
   

5. Create your first repository:

   cd /srv/example.com/repo/git/repos
   sudo install -d -o git -g git -m0700 myproject.git
   sudo -H -u git env GIT_DIR=myproject.git git init
   

   (with git older than v1.5, use init-db instead of init)

6. Set up an access control group and give it access to that repository:

   cd /srv/example.com/repo/git/view
   sudo install -d -m0755 mygroup
   cd mygroup
   sudo ln -s ../../repos/myproject.git myproject.git
   

   You can also use subdirectories of view/mygroup to organize the repositories hierarchically.

   Note, one SSH public key will belong to exactly one group, but if necessary you can create a separate group for each account for absolute control.

   Note, access to repository implies write access to repository, at least for now. You could make

7. Get an SSH public key from a developer and authorize them to access the group:

   cd /srv/example.com/repo/git
   sudo vi .ssh/authorized_keys
   

   How the developer generates their key is out of scope here.

   Add a line like this, with the public key in it (all on one line, broken up in the middle of word to make sure there is no misunderstanding about when to use a space and when not to):

   command="/usr/local/bin/git-shell-enforce-directory /srv/exampl
     e.com/repo/git/view/mygroup",no-port-forwarding,no-X11-forwar
     ding,no-agent-forwarding,no-pty ssh-rsa ... jdoe@example.com
   

   Or to allow only read-only access, add --read-only as an option.

8. You can now push things to the repository with:

   git push git@myserver.example.com:myproject.git mybranch:refs/heads/master
   

   Note that before the first push, your server-side repository will not contain even an initial commit, and can't really be cloned.

9. Now the developer can clone the repository:

   git clone git@myserver:myproject.git
   

   or to avoid some behavior of older git that I consider confusing (needs git v1.5 or newer):

   git clone -o myserver git@myserver:myproject.git
   

   They will probably want to set up ssh-agent to avoid typing the passphrase all the time.

And you're done! Good luck with your adventures with git, and welcome to the 21st century and to distributed version control systems.

Tomorrow, if everything goes well, I will buy a used car from an individual in California. Here's a checklist of things for that, to help others in similar situations. Some things may have already been omitted because they weren't relevant for me, so you may want to independently browse the websites I'm using as sources. I'm skipping everything related to finances and haggling. I'm not covering cases where the car isn't in good condition. Also, buying from dealerships is different. Good luck.

(Updated to mention REG 262.)

1. Find an ad on craigslist or whatever. I liked http://losangeles.listpic.com/car/

2. Call the seller. Don't just email, start evaluating the seller. Things to ask: http://www.carbuyingtips.com/questions.html (though that list is horribly extensive; I'd rather just pick the three best-looking candidates and do that thing on the spot; you'll need to doublecheck anyway to make sure the seller wasn't lying).

   Ask for:

   • the VIN of the car (usually 17 characters, on windshield)
   • full name of the seller (you'll need it anyway for the check)

3. Check the CARFAX report for the car. Go for the $24.95 30-day option, you should always look at more than one car.

   • compare odometer, accident history etc with what seller said (better yet, just avoid accident cars, the risk isn't worth it)
   • make sure it's not an old junk car poorly repaired ("salvage" title)

4. Check the smog check history (fails are a sign of trouble). In California, in most of the cases, the seller is required to provide a certificate of check newer than 90 days old -- it seems the information flow to website is slow enough that this latest check does not show up, or something.

5. Check that the car hasn't suffered storm damage. (Annoying website demands cookies for a simple form. Suck.)

6. Make an appointment and go see the car. In good sunlight, you want to see the car.

   • for some things to check, read more, especially On-the-Lot Checklist, Road and Test Checklist
   • you have to drive the car; leave a photocopy of your drivers license if needed for assurance
   • bring a car nut friend who knows what to check; play good cop bad cop (this checklist is not as good, as I am not a car nut)
   • on the lot:
     • paint chips
     • cracked windows
     • accident damage
     • signs of water damage
     • tires
     • check the VIN you were given against windshield, doors, engine, dashboard, major body parts; if they don't match you are dealing with a criminal
   • while driving:
     • brake hard, is it even
     • rev the engine, does it sound healthy
     • make plenty of starts and stops
     • go through a manual gearbox, there should be no grinding noises
     • see if the car will drive straight with your hands off the wheel
     • how does the clutch feel?
     • listen for noises throughout the test drive
   • after driving:
     • check for leaks
   • to be really careful, you should check a lot of things like
     • AC
     • windows up/down
     • radio/cd/speakers
     • etc

7. Take the car to a mechanic of your choosing for evaluation. Alternatively, choose to trust on service from brand name vendor.

8. Check the price against Kelley Blue Book etc.

9. Now you're in business. Only bureaucracy and doublechecking left.

10. If you know what you're willing to pay, go to your bank and get a cashiers check. Or two alternatives, if that works out. Otherwise, you will need to go back to the bank after haggling, and without a deposit the seller may have sold the car to someone else. Or something. Tuff.

    (Some people say haggling at dealerships is easier if you show up with a cashiers check just a bit short from what they're asking.)

    Don't pay with cash, there's even less chance of getting it back than cancelling a check.

    If you happen to be a seller reading this, do the actual sale in a bank to be sure you aren't being cheated.

11. Download and print PDF forms from DMV.

    • Bill of Sale (PDF)
    • Statement of Facts may interest you (PDF)
    • things the seller needs: Notice of Release of Liability (PDF)

12. In California, sellers are required to provide a smog certificate. Make sure you get one. Stuff on smog checks: 1, 2, 3. Frankly, I'm still a bit confused myself when I will need to do a smog check.

13. Check that the registration is current and that the car wasn't repurchased under the California "Lemon law". (Err, how? As I understand it should say so in the Certificate of Title paper)

14. The seller should find the "pink slip" aka Certificate of Title

    • if said paper does not have form fields for ownership transfer and odometer reading, you need form "REG 262" from the DMV, which is printed on special paper and not available as PDF. SUCK!
    • check seller name against his drivers license
    • fill it out with both of your info and both sign it; for instructions search for "Where do I sign?" on this page
    • if a bank or something still owns a chunk of the car, their signature is also needed on the pink slip; I'd be inclined to avoid the complexity
    • fill in the odometer value, both sign (read more)
    • seller keeps the Notice of Transfer and Release of Liability part and submits to DMV.
    • buyer fills in the back of the title to transfer ownership

15. Things to ask for before leaving

    • is a special wheel lug key needed, get it
    • are there extra keys
    • how does the car alarm work
    • on a convertible, how does the roof work
    • any "tricks" you should know

16. Now you're ready to leave with your new car! Check that you have

    • Certificate of Title, signed by both, also odometer section
    • Bill of Sale, signed by both
    • maintenance records
    • smog certification
    • owners manuals, repair manuals
    • spare tire, jack

    (sources 1, 2)

17. Seller has 5 days to submit Notice of Transfer and Release of Liability to DMV. Do it online.

18. Buyer has 10 days days to report ownership change to DMV and max 30 days to pay the fees. Read more: checklist, things to send to DMV, more info.

19. Taxes?

General resources:

• The other side of the story: seller howto. The person you are buying from should have done all of this, and this is what you can demand from him. Also, see the links and actual transaction guidance inside.
• Used car buying tips at http://www.carbuyingtips.com/used.htm

More SCALE5x: Dave Neary is talking about OpenWengo. Note to self: Wengo = TelCo, WengoPhone = software, OpenWengo = project developing WengoPhone -- or something. At least it's not just .org for community and .com for services, even if the names are way too close to eachother.

Good quote (not his, didn't catch the name):

"People don't want to buy a quarter-inch drill. They want a quarter-inch hole!"

He recommended this blog for anyone interested in user interface design: http://headrush.typepad.com/

Choice quotes:

"Cross Platform (but sound on Linux is a disaster)"

"Surprisingly, for Microsoft, it's not SIP... pure SIP." (talking about MSN Messenger)

They intend to implement XMPP-based transport mechanisms. Mentioned inkboard, an Inkscape extension(?) for whiteboard-style sharing of drawing over the internet.

They have games over OpenWengo (I guess XMPP?), like chess.

"Oh did I mention sound on Linux is horrible?"

Heh, we're calling audience members during the talk. From France. And it didn't work ;)

OpenWengo has cross-platform video conferencing. Wow.

More SCALE5x: Sam Folk-Williams is doing a talk called Xen Virtualization in Red Hat Enterprise Linux 5 and Fedora Core 6: An overview for System Administrators (UNGH!). And demonstrates why I hate "big" companies like Red Hat: they sent a non-technical, but well-practised, person to talk about Xen. He sounds convincing, but ended up explaining Xen domU migration without understanding the concept of shared storage. Gah.

Also note how talk summary promises live demos of Xen integration features only Red Hat has, and how the actual talk contained no such thing. If I didn't have wireless right now I'd be annoyed. Thank you SCALE5x organizers, the wifi is just great.

So I'm at SCALE5x, listening to Ron Gorodetzky talk about what he learned about sysadmining for Digg and Revision3 (who try to be an "Internet television network"; in effect, they distribute loads of big files). Most of the tools he mentioned I already knew, but it was nice to get independent reviews of "hey I think this is good". Here's what I took home from his talk:

• He really thinks highly of the OSCon 2005 talk Livejournal's Backend (A history of scaling) (PDF).

• He liked memcached and MogileFS.

• Between the lines I understood Revision3 has outsourced their big bandwidth use -- the CDNs he mentioned by name were Cachefly (the color scheme hurts even my eyes and real designers think I'm colorblind), BitGravity (caution hideous flash site) and of course Akamai.

• He spoke about outsourcing data center operations, using things like Amazon EC2 and S3. I need to come up with a budget and time to play with EC2.

• He stressed the importance of setting up KVMs etc properly for the data center.

• Set up your infrastructure and plan for scaling before you get popular, because you will be too busy to do them afterwards. That's nice, I like building things scalable from scratch.

• Specific infrastructure management tools:

  • Puppet -- seems pretty much a reimplementation of cgengine
  • Bcfg2 -- smells like academentia to me
  • ISconf -- from the Bootstrapping an Infrastructure people, seems to be based on the idea of a p2p distributed cache that stores pretty much a version control history of commands ran.

  As usual, I haven't yet seen anything that would actually seem to work in the real world, unless you give up everything you already have (like package management etc), and do things 100% their way.

  His suggestion: as the tools are based on very different worldviews, look at everything and try to pick the one that matches your opinions.

• One thing he wouldn't skimp on: "Don't skimp on RAM."

• At Revision3, they use long-life server hardware and don't upgrade the servers, instead they go for a full new deployment.

Tired of managing n+1 passwords? Hate having an extra network port open on that server box? Want to have automated replication of email to your laptop in a Unix command line geek-friendly fashion?

Here's how to make OfflineIMAP synchronize mail between local and remote Maildirs.

• on the client:

  • create an SSH key pair with no passphrase:

    ssh-keygen -t rsa -N '' -f ~/.ssh/imap-preauth-key
    

• on the server:

  • install Binc IMAP on the server; no need to have it actually listen for network connections

  • I store my mail as ~/.Mail on the server; create a ~/.bincimap on the server and adjust to fit:

    Mailbox {
      depot = "IMAPdir",
      umask = "0077",
      path = ".Mail",
    }
    

  • create a shell script ~/bin/imapd-preauth that'll start the IMAP daemon in a preauthenticated mode; note that OfflineIMAP wants a certain style of handshake bincimapd doesn't know how to do, so we fix that with sed:

    #!/bin/sh
    set -e
    
    export BINCIMAP_LOGIN=PREAUTH+FAKE
    bincimapd|sed --unbuffered '1s/^FAKE OK PREAUTH/* PREAUTH/'
    

    Make the script executable (duh).

  • authorize the previously generated SSH key to run only the above script -- add the following to ~/.ssh/authorized_keys (split here for readability, make it all one line; replace THINGS to fit):

    command="/home/USERNAME/bin/imapd-preauth",no-port-forwarding,
    no-X11-forwarding,no-agent-forwarding,no-pty SSHPUBLICKEYHERE
    

• on the client:

  • tell OfflineIMAP about the preauthenticated IMAP connection:

    [Account SOMETHING]
    localrepository = local-SOMETHING
    remoterepository = remote-SOMETHING
    
    [Repository local-SOMETHING]
    type = Maildir
    localfolders = ~/data/mail/SOMETHING
    
    [Repository remote-SOMETHING]
    type = IMAP
    remotehost = HOSTNAME
    preauthtunnel = env -u SSH_AUTH_SOCK ssh -q -i ~/.ssh/imap-preauth-key %(remotehost)s fake-command
    

That should be it! Have fun.

(And if you just broke it, feel free to give one of the halves to me.)

Here's a nice video on what people mean when they say Web 2.0: http://youtube.com/watch?v=6gmP4nk0EOE

I like the style of it, though it is too fast paced if you don't already know what it is talking about.

(Found via Mickipedia)

Sixwordscifi.com is so much fun it has to be wrong, somehow. Here are my favorites so far (yes, one of them is mine):

Back me up before I die.

boogah

Even at light speed, I wait.

jefft

All alone in his light cone.

tv

I now know what I want from my next phone. And it'll totally blow the whole phone concept out of the water.

Start with a mostly-open hardware platform like Neo1973, add Linux (OpenMoko) on top. And no need to cram in a clumsy qwerty keypad, just carry a one-hand keyboard when you care about it. Less clumsy when you don't want more than a phone, full SSH sweetness when you want it. The phone itself is purely touch-screen, and the keyboard can actually get respectable WPM with real keypress feedback. And the big part is, because the phone is actually Open, plugging all this in is not a big problem! That's just great!

I have a shoulder strap-style bag, manufactured and used by the German army, bought as surplus and dyed black. My 12" thinkpad fits perfectly inside of it. But the strap is sewn in place, and seems to fail every few years -- and now's the time.

The bag is really good, and I am going to get it fixed, but that doesn't mean I can't look at alternatives. So, I need something that fits a 12" thinkpad, isn't too big, preferably comes in black, and is otherwise non-attention grabbing and doesn't look like it'd contain a laptop. According to Lenovo, my laptop is about 268x211x20mm.

My options:

• small messenger bag: 1, 2
• hard-shell attache case (not really my style, but a 12" one in black might rock it -- though I'm not going to buy a 15" case with internal padding to make a 12" laptop not bounce around, I want a smaller case too; it'd still be plenty big for the obligatory dead tree notes and docs; for pics, see 3, 4
• the manliest purse ever from Maxpedition: I'd be willing to carry that (in black of course;) if it fit a 12".. And I don't think it will.
• more classical military style: M-51 Engineers Field Bag, Urban Explorer Black Canvas Shoulder, etc..
• something from http://booqbags.com/ but the style doesn't really smack me in the face with want
• something from Timbuk2, but they really don't seem my style

None of those really work for me. I guess considering the bags I already own, I might go for something I don't have. Mmm, a smallish hard-shell attache case in black, with a good enough shoulder strap that I can sling it over my head. That might just do it. Now where do I get one?

Update: It seems the script had gone missing at some point. It's back.

As you may or may not have noticed, I do a bunch of stuff for Revver. I ended up writing a sort of a tutorial to the Revver API, and as I like to collect all kinds of code samples here, I thought I should crossblog it here. The original is on the Revver developer blog.

One day, I was on a slow internet connection and wanted to upload a few files. I wanted something more batch-oriented than the web-based upload, and I have a personal bias against most current Java runtimes. So I decided to use the cool new API and write a video upload client , and will walk you through what it does in this blog entry. Feel free to "just" use the tool, but hopefully this will also help you in writing your own API clients.

First of all, I wanted to write something that's usable just about everywhere. I tend to use Python, so that's what the tool is written in. The Python standard library didn't seem to be able to do HTTP POST file upload (think web forms) of large files, so I ended up using curl for that. This should work on any Linux/OS X/etc box with Python and curl installed. All you Ubuntu/Debian people just get to say sudo apt-get install curl and that's it.

So, let's dive right in. The tool is imaginatively named revver-upload-video. The first bit is the command line parser. Don't be intimidated by the length, this is pretty much boilerplate code, the actual API-using bits are really small. The full file is 155 lines, total.

There are basically three kinds of options: mandatory, optional and for developer use. Mandatory options are enforced later on, and developer options are mostly meant for playing with the staging environment and reusing upload tokens from previous, failed, uploads.

#!/usr/bin/python
"""
Guerrilla command line video upload tool.
"""
import optparse, getpass, urlparse, urllib, xmlrpclib, subprocess

def getParser():
    parser = optparse.OptionParser(
        usage='%prog --title=TEXT --age-rating=NUM [OPTIONS] FILE..',
        description='Upload videos to Revver')

    parser.set_defaults(
        api_url='https://api.revver.com/xml/1.0',
        upload_url='http://httpupload.revver.com/',
        login=getpass.getuser(),
        )
    parser.add_option('--login',
                      help='login name to use (default: %s)' %
                      parser.defaults['login'])
    parser.add_option('--passphrase-file',
                      help='read passphrase from (prompt if not given)',
                      metavar='FILE')
    parser.add_option('--age-rating',
                      help='MPAA age rating (mandatory)',
                      type='int')
    parser.add_option('--title',
                      help='title for the video (mandatory)',
                      metavar='TEXT')
    parser.add_option('--tag',
                      help='tags (mandatory, repeat for more tags)',
                      metavar='KEYWORD',
                      action='append')

    parser.add_option('--author',
                      help='author of the video',
                      metavar='FULLNAME')
    parser.add_option('--url',
                      help='website for extra info')
    parser.add_option('--credits',
                      help='extra credits',
                      metavar='TEXT')
    parser.add_option('--description',
                      help='a brief description',
                      metavar='TEXT')

    parser.add_option('--api-url',
                      help='API URL to contact (developers only)')
    parser.add_option('--upload-url',
                      help='Upload URL to send the file to (developers only)')
    parser.add_option('--upload-token',
                      help='use preallocated token (developers only)',
                      metavar='HEX',
                      action='append')

    return parser

If you've used optparse before, there's not much interesting there. It just instantiates a parser object and returns it, for the main function to use. Nothing there touches the Revver API yet.

Next up, we have some utility functions. getPassphrase will read a passphrase form the file given to --passphrase-file=, or prompt the user for one. getAPI instantiates an XML-RPC client object with the login and passphrase, and caches it in options so if you call getAPI more than once, you're still only prompted for the passphrase at most once. Nothing in revver-upload-video uses that, but these are meant to be reusable functions.

def getPassphrase(filename=None):
    if filename is not None:
        f = file(filename)
        passphrase = f.readline().rstrip('\n')
        f.close()
    else:
        passphrase = getpass.getpass('Passphrase for video upload: ')

    return passphrase

def getAPI(options):
    api = getattr(options, 'api', None)
    if api is None:
        passphrase = getPassphrase(filename=options.passphrase_file)

        (scheme, netloc, path, query, fragment) =
                urlparse.urlsplit(options.api_url,
                                  allow_fragments=False)
        query = urllib.urlencode([('login', options.login),
                                  ('passwd', passphrase)])
        url = urlparse.urlunsplit((scheme, netloc, path, query, fragment))
        api = xmlrpclib.Server(url)
        options.api = api
    return api

All right, now we're getting to the actual meat. getToken calls the API method video.getUploadTokens to allocate an upload token, that lets you upload a file to the Revver archive.

def getToken(api):
    url, tokens = api.video.getUploadTokens(1)
    assert len(tokens)==1
    token = tokens[0]
    return url, token

createMedia creates a new video in the archive from your uploaded file by calling video.create in the API. It also adds metadata like your website URL to the video. Finally, it returns the media id of the newly-created video.

def createMedia(options, token):
    data = {}
    if options.credits is not None:
        data['credits'] = options.credits
    if options.url is not None:
        data['url'] = options.url
    if options.description is not None:
        data['description'] = options.description
    if options.author is not None:
        data['author'] = options.author
    api = getAPI(options)
    media_id = api.video.create(token,
                                options.title,
                                options.tag,
                                options.age_rating,
                                data)
    return media_id

Finally, we have the main function, and the bits that call it when you run the tool. Here, we actually parse the command line arguments, enforce the presence of the mandatory options, and bail out unless you gave it actual files to upload.

For each file given on the command line, we either use one of the tokens given to us with --upload-token=, or get one from the API with getToken. Then we join the upload URL and the token to get the place to upload the file to, and run curl as a subprocess to do the actual upload. Checking that curl worked takes 8 lines, and then we use createMedia to actually create the video. And that's it!

def main(progname, args):
    parser = getParser()
    (options, args) = parser.parse_args()

    if options.login is None:
        parser.error('You must pass --login=LOGIN')
    if options.title is None:
        parser.error('You must pass --title=TEXT')
    if not options.tag:
        parser.error('You must pass --tag=KEYWORD')
    if options.age_rating is None:
        parser.error('You must pass --age-rating=NUM')
    if not args:
        parser.error('Pass files to upload on command line')

    for filename in args:
        if options.upload_token:
            token = options.upload_token.pop(0)
            url = options.upload_url
        else:
            api = getAPI(options)
            url, token = getToken(api)
            print '%s: allocated token %s' % (progname, token)

        upload_url = urlparse.urljoin(url, token)
        retcode = subprocess.call(['curl',
                                   '-F', 'file=@%s' % filename,
                                   '--',
                                   upload_url,
                                   ])
        if retcode < 0:
            print >>sys.stderr, '%s: upload aborted by signal %d' % (
                progname, -retcode)
            sys.exit(1)
        elif retcode > 0:
            print >>sys.stderr, '%s: upload failed with code %d' % (
                progname, retcode)
            sys.exit(1)
        print '%s: used token %s for %s' % (progname, token, filename)

        media_id = createMedia(options, token)
        print '%s: created media %r from %r' % (progname,
                                                media_id,
                                                filename)

if __name__ == '__main__':
    import os, sys
    main(progname=os.path.basename(sys.argv[0]),
         args=sys.argv[1:])

At this point, we have all we need to do the same thing as the web-based upload, or the Java upload client. And you can do something similar, by yourself. This file is copyright Revver, Inc, but licensed under the MIT license -- that means you can use it as a base for writing your software, without any real restrictions. Download the whole thing here: revver-upload-video.

I had a bit of fun thinking up of puntastic DNS domain names. I ended up registering eagain.net, the old tv.debian.net name will soon start redirecting there. Need to set up email too.. Vanity domains are soo much fun.

For the rare non-nerd reading this, EAGAIN is the error code you get when you are doing asynchronous programming with non-blocking sockets and would block next. Err, let's just say "I write async code and it's a neat insider joke", ok?

For the nerds out there, here's a bunch of wild ideas I had while figuring out what domain name to register. Many of them are invalid (too short), and most of the good ones are already taken, but in case you need some inspiration:

• asyn.ch
• dot.at
• ex.plo.de / im.plo.de
• plea.se
• fal.se
• belong.us
• chi.hu (as in Chihuahua, our dogs..)
• celci.us
• blo.gr
• co.de
• co.ff.ee
• oh.no
• wh.ee
• wh.at
• sh.it
• stre.am (taken)
• up.stre.am
• down.stre.am
• upstre.am
• em.ploy.ee
• s.tre.am
• 3x2.net (as in triple-double-w)
• tribledouble.net
• 2by4.net (as in clue-by-four)
• be.am (la.ser.be.am, scotty.up.be.am ;-)
• dre.am
• progr.am
• pu.sh
• pu.bli.sh
• form.at
• anon.ymo.us
• pla.net
• mag.net
• carwa.sh
• blo.at
• cave.at
• repe.at
• pho.to
• oct.et
• pron.to
• plu.to
• lot.to
• dit.to
• adju.st
• ang.st
• arti.st
• ava.st (type like a pirate)
• broadca.st
• inner.net
• comm.it
• mis.info
• foc.us
• fung.us
• geni.us
• gur.us
• octop.us
• styl.us
• stat.us
• tor.us
• line.br
• deco.de
• while.do
• un.do
• weir.do
• if.then.fi
• cook.ie
• newb.ie
• zomb.ie
• disk.io
• comm.it
• subm.it
• bat.ch
• bran.ch
• epo.ch
• fet.ch
• zil.ch
• gra.ph
• boo.st
• fibona.cc
• from.to
• justa.com

And for the Finns:

• mi.au (cat lovers)
• vai.nu (dog lovers)
• kir.nu
• masen.nu
• tie.dos.to (taken, by non-Finn)
• hit.to

I'm not going to say anything about the Cook Island's subdomain for commercial entities.

If your domUs have networking trouble with TCP, or some other protocol that ends up needing fragmentation such as large ICMP pings, you need to read this.

If it seems TCP handshakes complete, but no data is transferred -- especially, no actual data gets sent out from the domU -- you're likely hitting a bug in how Xen interacts with TCP segmentation offload.

The bug seems to depend on the actual network interface card the traffic is going out from. I hear tg3 is one of the cards that triggers it, and I'm seeing it on my home box with 8139too's.

The fix is pretty simple, but hard to figure unless you know what to look for: inside the domU, run

ethtool -K eth0 tx off

for each interface affected.

See http://wiki.xensource.com/xenwiki/XenFaq#head-4ce9767df34fe1c9cf4f85f7e07cb10110eae9b7 for the very small amount of extra information that is out there.

Life sucks and then your computer breaks. As soon as a new X release is out, and it seems dual head on iBook is a possibility, the darn thing decides to fry its logic board. Again. Thankfully, Apple may make the repair for free, if the symptoms match the manufacturing problem. Again.

Well, the good news is that after 3 years of using the iBook, I got a new laptop. A Lenovo Thinkpad x60s, weighing just 1.3kg. It's so light I always think I forgot to put it in the backpack.

Ubuntu Dapper (flight 7) seems to work pretty well on the x60s. Trouble spots so far:

• install CD corrupted display during X autoconfig: screen was in text mode, mostly black, with two or three character-size grey rectangles -- hitting enter blindly let it continue and reboot the machine

• suspend and hibernation fail on resume

• wlan hanged once, and didn't recover until I rebooted into Windows -- unngh. Look at this:

  ipw3945: Error sending SCAN_ABORT_CMD: time out after 500ms.
  ipw3945: Radio Frequency Kill Switch is On:
  Kill switch must be turned off for wireless networking to work.
  ipw3945: Error sending ADD_STA: time out after 500ms.
  ipw3945: Error sending RATE_SCALE: time out after 500ms.
  

  After that, any attempt to use the interface ended with:

  ADDRCONF(NETDEV_UP): eth1: link is not ready
  

• hotplugging the UltraBay docking station does not seem to work in Linux

I especially love the dual headedness, after fighting with the ATI driver in the iBook.

Now I need to see about hooking the fingerprint reader up to PAM.

Finally, after two years of hacks, my iBook 2.2 knows how to multihead! And no silly clone mode only, totally different image and external output at 1600x1200 at 85Hz. This is nice! Thank you X.org people for version 7, thank you X Strike Force!

Update: well, now suspending fails and booting the machine results in a black screen in over half of the tries. Bah.

After render_fragment, dialtone mentioned render_pattern, that would get one or many patterns from the page and put them in the current tag. Well, that's easy to write:

def render_pattern(self, name):
    """
    Find and render a pattern.

    Example:

    <span nevow:pattern="foo">
      I'm very repetititive.
    </span>
    <ul>
      <li nevow:render="pattern foo">
        this text will get removed when rendering
      </li>
      <li nevow:render="pattern foo"/>
    </ul>
    """
    def f(ctx, data):
        doc = self.docFactory.load(ctx)
        patterns = inevow.IQ(doc).allPatterns(name)
        return ctx.tag.clear()[patterns]
    return f

Updated to adapt doc to inevow.IQ before calling allPatterns.

This Nevow renderer came up on #twisted.web. Thanks to rwall and dialtone for input.

def render_fragment(self, name):
    """
    Find and render a fragment, with optional docFactory.

    Find a fragment factory from self via attributes named
    fragment_* and replace content of current tag with said
    fragment.

    If pattern docFactory is found under this tag, pass it as
    docFactory to the fragment factory.

    Example:

    class MyFrag(rend.Fragment):
        ...

    class MyPage(rend.Page):
        fragment_foo = MyFrag
        ...

    and give MyPage a template with

    <!-- no docFactory -->
    <div nevow:render="fragment foo">
      this text will get removed when rendering
    </div>

    <!-- with docFactory -->
    <div nevow:render="fragment foo">
      this text will get removed when rendering
      <span nevow:pattern="docFactory">
        but this whole tag will be passed as docFactory to MyFrag.
      </span>
    </div>
    """
    def f(ctx, data):
        callable = getattr(self, 'fragment_%s' % name, None)
        if callable is None:
            callable = lambda ctx, *args: ctx.tag[
             "The fragment named '%s' was not found in %r." % (name, self)]
        kwargs = {}
        try:
             docFactory = ctx.tag.onePattern('docFactory')
        except stan.NodeNotFound:
             pass
        else:
             kwargs['docFactory'] = loaders.stan(docFactory)
        return ctx.tag.clear()[callable(**kwargs)]
    return f

This Nevow renderer has saved me a lot of time:

def render_if(self, ctx, data):
    r=ctx.tag.allPatterns(str(bool(data)))
    return ctx.tag.clear()[r]

Use it like this:

<nevow:invisible nevow:render="if" nevow:data="items">
  <ul nevow:pattern="True"
      nevow:render="sequence">
    <li nevow:pattern="header">The items are a-coming!</li>
    <li nevow:pattern="item">(the items will be here)</li>
  </ul>
</nevow:invisible>

And now, if the list returned by data_items is empty, there will be no <ul> tag at all in the output.

I just realized non-boolean tests may be wanted -- for example, test if a string matches a regexp. You could do that by mangling the data before render_if, but that's not nice, because then you don't have access to the original data inside nevow:pattern="True". So, instead let's parametrize the test:

def render_ifparam(self, name):
    tester = getattr(self, 'tester_%s' % name, None)

    if tester is None:
        callable = lambda context, data: context.tag[
             "The tester named '%s' was not found in %r." % (name, self)]
        return callable

    def f(ctx, data):
        r=ctx.tag.allPatterns(str(bool(tester(data))))
        return ctx.tag.clear()[r]

    return f

Note how we still cast the return value of the tester to boolean. You could avoid that and call the renderer render_switch. Adding support for Deferred tests would be quite easy, too. The only ugly part is I don't know of any way to make the same renderer work nicely for nevow:render="if" and nevow:render="ifparam foo".

[Updated to add return f, also renamed second render_if to render_ifparam to clarify things a bit. Thanks k3mper.]

I've just finished my review of the book Twisted Network Programming Essentials by Abe Fettig. Now go read it.

You may also be interested in an earlier review I wrote about Foundations of Python Network Programming.

This entry is about a local software developer gathering, and written in Finnish. My apologies if it is complete gibberish to you, but atleast you can stare at the pretty ä dots.

Mikä?

Vapaamuotoinen tapaaminen ohjelmistoja työkseen ja/tai harrastuksekseen tekeville, tai muuten aiheesta kiinnostuneille.

Tutustutaan ihmisiin, puhutaan mukavia, syödään ruokaa. Jos haluat kertoa hienosta uudesta softasta, jota olet juuri tekemässä, löydät varmaan jonkun samanmielisen. Jos tarvitset apua hankalaan ongelmaan, joku varmaan on joskus tehnyt jotain samankaltaista. Eikä ihmisten tunteminen ainakaan haittaa urakehitystäkään.

Sillä ei ole väliä onko työkalusi C, Perl, Java, Python, Ruby vai PHP; tai Linux, BSD, OS X vai jopa Windows. Vapaat/avoimet ohjelmistot ovat monelle meistä tärkeitä, joten niiltä et kokonaan pysty välttymään, mutta tarkoitus on vain saada samanhenkisiä ihmisiä kokoon.

Missä?

Turun keskustassa oleva ravintola Harald, katso kartta.

Meitä kiinnostaa eniten Turun seudun toiminta, mutta ajatuksia "road showsta" on heitetty ilmaan, eli jatkossa kehittäjätapaaminen saattaa olla sinunkin lähikuppilassasi.

Koska?

Nyt lauantaina, 26.11. n. klo 12:30 alkaen. Niin pitkään kun intoa riittää.

Seuraava kerta varmaan joskus tammikuussa, ja siitä sitten eteenpäin vaikka kuukauden tai parin välein.

Kuka?

Tällä tapahtumalla ei ole virallista järjestäjää, eikä se liity minkään yhdistyksen tjms. toimintaan. Minä aloin asiasta tutuille puhumaan, Tero Kuusela on tehnyt lähes kaiken valmistelutyön.

Tällä hetkellä aiheesta kiinnostuneiden ihmisten taustoja ja kiinnostuksia: VSTKY, Linux-Aktivaattori, Debian, Python, Linux kernel, Google Summer of Code, jne..

Tahtoo!

Liity postituslistalle. Listan osoite on turku-dev@lists.inoi.fi ja liittyminen tapahtuu lähettämällä viesti osoitteeseen turku-dev-subscribe@lists.inoi.fi ja vastaamalla vahvistus-pyyntöön.

Ihmismäärän arvioimiseksi pyydän, että ilmoitat tulostasi etukäteen osoitteeseen Tero Kuusela <tero@teroajk.net>.

"It's simply a more convenient syntax."

"MochiKit is full of more convenient syntax."

The MochiKit screencast is great. I think screencasts are a great way to introduce people to new software.

I just registered at Technorati, and it requires me to post an entry claiming my Technorati Profile.

..and when I registered an alternate URL for it, they required me to do it all over again: Technorati Profile.

>>> def simple(): yield 'a'
...
>>> ', '.join(simple())
'a'
>>> def horrible():
...     if ' ' not in False: yield 'a'
...
>>> ', '.join(horrible())
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: sequence expected, generator found
>>>

But it does accept generators!

(Yes, I know what triggers it to say that. It's still horribly misleading.)

<ronwalf> ok, I give up... How do I get the AVATAR_LOGIN stuck
between the SessionWrapped resource ul and the current
resource url

Well, we aim to please.

def getActionURL(ctx):
    request = inevow.IRequest(ctx)
    current = url.URL.fromRequest(request).clear()
    root = request.getRootURL()
    root = url.URL.fromString(root)
    assert root is not None
    root = root.pathList()
    me = current.pathList(copy=True)
    diff = len(me) - len(root)
    assert diff >= 0
    action = current
    if diff == 1:
        action = action.curdir()
    else:
        while diff > 1:
            diff -= 1
            action = action.parent()
    action = action.child(guard.LOGIN_AVATAR)
    for element in me[len(root):]:
        action = action.child(element)
    return action

Comment from ronwalf (on IRC) on 2005-09-17T00:06:57:

<ronwalf> Better.  after root = root.pathList()
<ronwalf> if root == ['']: root = []