Tv's cobwebhttp://eagain.net/blog/2020-01-21T20:49:33-07:00Tommi Virtanentv@eagain.netBetter CoreOS in a VM experiencehttp://eagain.net/blog/2016/03/31/coreos-in-a-vm.html2016-03-31T16:37:27-08:002020-01-21T20:49:33-07:00<p>(This was all tested with CoreOS beta <code>991.2.0</code>, the stable <code>835.13.0</code>
fails to mount <code>9p</code> volumes.)</p>
<p><a href="https://coreos.com/">CoreOS</a>
comes with
<a href="https://coreos.com/os/docs/latest/booting-with-qemu.html">instructions how to run it in QEMU</a>
.
After the setup, it comes down to something like</p>
<pre tabindex="0"><code>#!/bin/sh
exec ./coreos_production_qemu.sh \
-user-data cloud-config.yaml \
-nographic \
"$@"
</code></pre><p>with <code>cloud-config.yaml</code> looking like</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="c">#cloud-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">hostname</span><span class="p">:</span><span class="w"> </span><span class="l">mytest</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">users</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">jdoe</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">groups</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">sudo</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">rkt</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ssh_authorized_keys</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">"ssh-ed25519 blahblah jdoe@foo.example.com"</span><span class="w">
</span></span></span></code></pre></div><p>I used that for many experiments, but felt it was less than ideal.</p>
<p>I wanted to move beyond <code>-net user</code>. That just means calling QEMU
directly, instead of using <code>coreos_production_qemu.sh</code>, no big deal.
But it meant I would be writing my own QEMU runner, no matter what.</p>
<p>I also wanted more efficient usage of my resources -- after all, the
whole point of me running several virtual machines on one physical
machine is to make these test setups more economical.</p>
<p>The provided QEMU images waste <em>disk</em> and <em>bandwidth</em>. Every VM stores
two copies of the CoreOS <code>/usr</code> image, just like a physical machine
would. Copy-on-write trickery on the initial image will not help
beyond the first auto-update, as each VM independently downloads and
applies the updates. This means if you run a small test cluster of say
5 VMs, you'll end up with 10 copies of CoreOS, and 5x the bandwidth
usage needed.</p>
<p>Imitating physical computers with virtual machines is great if you're
trying to learn how the CoreOS update mechanism works, but once you're
to the point of wanting to just run services, it's simply not needed.</p>
<p>CoreOS does have a supported mode where it does not use the <code>USR-A</code>
and <code>USR-B</code> partitions:
<a href="https://coreos.com/os/docs/latest/booting-with-pxe.html">PXE booting</a>
,
starting a computer by requesting the software over the network. I
could even skip the virtual networking and use this
<a href="http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe.sh">with QEMU by launching the kernel and initrd directly</a>
,
no need for PXE itself. However, this is wasteful in another way: it
holds the complete <code>/usr</code> partition contents in RAM, using about
180MB. Once per each VM. There is also an annoying delay of 15+
seconds in VM startup, presumably related to the large initrd image,
and later the kernel spends 1.2 seconds uncompressing it into a
<code>tmpfs</code> (measured on a i5-5300U laptop).</p>
<p>Digging into the PXE image, I find that it actually stores the <code>/usr</code>
contents as a <code>squashfs</code> -- which is a real filesystem that can be
stored on block devices, as opposed to just unpacking a <code>cpio</code> to a
<code>tmpfs</code>. The PXE image does what's called a "loopback mount", where a
file is treated like a block device. In the PXE scenario, the file is
held in RAM in a <code>tmpfs</code>; I can just put those bytes on a block
device, and boot that!</p>
<p>(The
<a href="https://coreos.com/os/docs/latest/booting-with-iso.html">Live CD</a>
seems to also hold <code>/usr</code> contents in <code>tmpfs</code> just like the PXE
variant, even though it <em>could</em> fetch them on demand from the ISO. The
<code>squashfs</code> image is random-access, unlike the usual <code>cpio.gz</code> that's
used for <code>initramfs</code> contents. In later versions, CoreOS could switch
their ISO images to use the trick I'll explain below -- at the cost of
physical machines needing to spin up a CD more often than once per
boot. The live CD has another downside that made me avoid it: to pass
kernel parameters, I'd have to resort to kludges like creating a boot
floppy image with <code>syslinux</code> and the right parameters on it.)</p>
<p>So, I set about fixing the wasted disk and bandwidth problem. Here's a
story of an afternoon project.</p>
<h2 id="using-the-usr-image-directly">Using the <code>/usr</code> image directly</h2>
<p>Instead of holding an extra copy of the <code>/usr</code> image data in RAM, we
can make it available as a block device, and load blocks on demand.</p>
<p>For that, we need the <code>/usr</code> <code>squashfs</code> image as a standalone file,
not inside the <code>cpio</code>. It's not available as a separate download, but
we can extract it from the PXE image:</p>
<pre tabindex="0"><code>wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz.sig
gpg --verify coreos_production_pxe.vmlinuz.sig
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz.sig
gpg --verify coreos_production_pxe_image.cpio.gz.sig
zcat coreos_production_pxe_image.cpio.gz \
| cpio -i --quiet --sparse --to-stdout usr.squashfs \
>usr.squashfs
</code></pre><h2 id="prepare-a-root-filesystem">Prepare a root filesystem</h2>
<p>We also need to make prepare a disk image that will be used for
storing the root filesystem. CoreOS won't boot right with a fully
blank disk. If it had, I would have used <code>qcow2</code> as the format, but
now I need to provide some sort of structure for the root filesystem,
so let's go with a <code>raw</code> disk image.</p>
<p>I might have been able to set up the right GPT partition UUIDs for the
initrd to <code>mkfs</code> things for me, but that seemed too complicated, and I
doubted it'd support my "just the root" scenario as well as their
nine-partition layout.</p>
<p>To keep it simple, we won't bother to use partitions; the whole block
device is just one filesystem.</p>
<pre tabindex="0"><code>>rootfs.img
chattr +C rootfs.img
truncate -s 4G rootfs.img
mkfs.ext4 rootfs.img
</code></pre><h2 id="prepare-user_data">Prepare <code>user_data</code></h2>
<p>This was previously done inside <code>coreos_production_qemu.sh</code> with a
temp dir, but we'll just pass a directory as <code>virtfs</code> following the
"config drive" convention. Let's move our previous file into the right
place:</p>
<pre tabindex="0"><code>mkdir -p config/openstack/latest
mv cloud-config.yaml config/openstack/latest/user_data
</code></pre><h2 id="finally-run-qemu">Finally, run QEMU</h2>
<pre tabindex="0"><code>qemu-system-x86_64 \
-name mycoreosvm \
-nographic \
-machine accel=kvm -cpu host -smp 4 \
-m 1024 \
\
-net nic,vlan=0,model=virtio \
-net user,vlan=0,hostfwd=tcp::2222-:22,hostname=mycoreosvm \
\
-fsdev local,id=config,security_model=none,readonly,path=config \
-device virtio-9p-pci,fsdev=config,mount_tag=config-2 \
\
-drive if=virtio,file=usr.squashfs,format=raw,serial=usr.readonly \
-drive if=virtio,file=rootfs.img,format=raw,discard=on,serial=rootfs \
\
-kernel coreos_production_pxe.vmlinuz \
-append 'mount.usr=/dev/disk/by-id/virtio-usr.readonly mount.usrflags=ro root=/dev/disk/by-id/virtio-rootfs rootflags=rw console=tty0 console=ttyS0 coreos.autologin'
</code></pre><p>You'll be greeted with the Linux bootstrap messages and finally</p>
<pre tabindex="0"><code>This is mycoreosvm (Linux x86_64 4.4.6-coreos) 06:14:10
SSH host key: SHA256:t+WkofIWxkARu1hezwPnS/vgTJXUcPidA3UxKr+1uGA (DSA)
SSH host key: SHA256:cT32H33EVCHSnrCRsB+I9GG7AgXQWfyjk7JFuEzAqFU (ECDSA)
SSH host key: SHA256:NFgc7BLbeyS3SslpscSSNHNzc7lXzx6vKqBmUp+5T7Q (ED25519)
SSH host key: SHA256:pK8Dknoib61FnIwMQ6u4F4FxeSMIRq9zYsrJd0N3MPY (RSA)
eth0: 10.0.2.15 fe80::5054:ff:fe12:3456
mycoreosvm login: core (automatic login)
CoreOS stable (991.2.0)
Last login: Fri Apr 1 06:02:25 +0000 2016 on /dev/tty1.
Update Strategy: No Reboots
core@mycoreosvm ~ $
</code></pre><p>Success!</p>
<p>As usual with QEMU, press <code>C-a x</code> to exit.</p>
<p>Stay tuned for part 2, where we will make the VM even leaner.</p>
Unmarshaling a JSON array into a Go structhttp://eagain.net/blog/2016/01/13/go-json-array-to-struct.html2016-01-13T13:22:06-08:002020-01-21T20:49:33-07:00<p>Sometimes, you see heterogeneous JSON array like</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">[</span><span class="s2">"Hello world"</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="kc">false</span><span class="p">]</span>
</span></span></code></pre></div><p>Dealing with such an array in Go can be very frustrating. A
<code>[]interface{}</code> hell is just about as painful as the
<code>map[string]interface{}</code> hell (See my
<a href="/articles/go-dynamic-json/">earlier article about that</a>
).</p>
<p>The natural way to deal with data like that in Go would be a struct
like</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">Notification</span> <span class="kd">struct</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">Message</span> <span class="kt">string</span>
</span></span><span class="line"><span class="cl"> <span class="nx">Priority</span> <span class="kt">uint8</span>
</span></span><span class="line"><span class="cl"> <span class="nx">Critical</span> <span class="kt">bool</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>See how much more <em>meaning</em> we've added?</p>
<p>Now, you can't just <code>json.Unmarshal</code> an array into a struct. I'll show
you how to make that work.</p>
<p>
<a href="/articles/go-json-array-to-struct/">Read more...</a>
</p>
Go JSON unmarshaling based on an enumerated field valuehttp://eagain.net/blog/2016/01/12/go-json-kind.html2016-01-12T15:54:27-08:002020-01-21T20:49:33-07:00<p>In a
<a href="/articles/go-dynamic-json/">previous article</a>
, we talked about
marshaling/unmarshaling JSON with a structure like</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"type"</span><span class="o">:</span> <span class="s2">"this part tells you how to interpret the message"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"msg"</span><span class="o">:</span> <span class="p">...</span><span class="nx">the</span> <span class="nx">actual</span> <span class="nx">message</span> <span class="nx">is</span> <span class="nx">here</span><span class="p">,</span> <span class="k">in</span> <span class="nx">some</span> <span class="nx">kind</span> <span class="k">of</span> <span class="nx">json</span><span class="p">...</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>Last time, we left a repetitive switch statement in the code, where
each message type was unmarshaled very explicitly. This time, we'll
talk about ways to clean that up.</p>
<p>
<a href="/articles/go-json-kind/">Read more...</a>
</p>
Dynamic JSON in Gohttp://eagain.net/blog/2015/08/27/go-dynamic-json.html2015-08-27T08:59:34-07:002020-01-21T20:49:33-07:00<p>Go is a statically typed language. While it can represent dynamic
types, making a nested <code>map[string]interface{}</code> duck quack leads to
very ugly code. We can do better, by embracing the static nature of
the language.</p>
<p>The need for dynamic, or more appropriately <em>parametric</em>, content in
JSON often arises in situations where there's multiple kinds of
messages being exchanged over the same communication channel. First,
let's talk about <em>message envelopes</em>, where the JSON looks like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"type"</span><span class="o">:</span> <span class="s2">"this part tells you how to interpret the message"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"msg"</span><span class="o">:</span> <span class="p">...</span><span class="nx">the</span> <span class="nx">actual</span> <span class="nx">message</span> <span class="nx">is</span> <span class="nx">here</span><span class="p">,</span> <span class="nx">as</span> <span class="nx">some</span> <span class="nx">kind</span> <span class="k">of</span> <span class="nx">json</span><span class="p">...</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>
<a href="/articles/go-dynamic-json/">Read more...</a>
</p>
Notes from SCALE12xhttp://eagain.net/blog/2014/02/24/scale12x.html2014-02-24T11:21:49-08:002020-01-21T20:49:33-07:00<p><em>(Sidenote: this little blog engine had bitrotted pretty bad.. I
reimplemented it with markdown, go & bootstrap, and it's much more
pleasant to work with now. Time for new content!)</em></p>
<p>I spent Saturday & Sunday at the
<a href="https://www.socallinuxexpo.org/scale12x">Southern California Linux Expo (SCALE)</a>
,
and here's my very personal report of how I experienced it.</p>
<p>SCALE is not your typical tech conference, it brings in very diverse
groups of people. The organizers are actively trying to reach out to
e.g. kids that are in that "might grow up <em>interested</em> in things" age.
Just about every age group, techie background, and personal interest
is present -- the common theme really is only Linux (and a few
BSD-based vendors trying to sell their gear). Of course this means
that SCALE won't ever serve <em>my</em> desires perfectly -- but it serves
the community well, and the <em>feel</em> of the conference is very friendly
and engaging.</p>
<h2 id="friday">Friday</h2>
<p>First of all, I was too busy to go on Friday, and the streaming video
had some sort of audio codec trouble, so I won't comment about content
of the
<a href="http://www.socallinuxexpo.org/scale12x/devops-day-la">devops day</a>
.
What I will say is that I'm impressed by the strength of the devops
presence at SCALE. It's becoming a significant backbone of SCALE, year
by year. Kudos to the organizers. And they're at it all year long --
the local ops-oriented meetups have a great community going. Heartily
recommended, whether you carry a pager or not. Also see
<a href="http://hangops.com/">hangops</a>
.</p>
<p>SCALE also hosted another sub-event on Friday called
<a href="https://www.socallinuxexpo.org/scale12x/infrastructurenext">Infrastructure.Next</a>
,
<a href="https://twitter.com/infranext">@infranext</a>
. It looked interesting,
though I fear overpresence of Red Hat and vendor agenda. I'm still
waiting for slides and/or video of
<a href="https://www.socallinuxexpo.org/scale12x/presentations/how-assemble-cutting-edge-cloud-stack-minimal-bleeding">How to Assemble A Cutting Edge Cloud Stack With Minimal Bleeding</a>
.
(The archived live streams for all three days are useless because of
audio problems.)</p>
<p>I also missed Greg Farnum's
<a href="http://www.socallinuxexpo.org/scale12x/presentations/fundamentals-ceph-greg-farnum">talk</a>
on <a href="http://ceph.com/">Ceph</a>
. I worked at <a href="http://inktank.com">Inktank</a>
for almost two years, and this technology is one of a kind, and a good
indicator of what direction the future lies. If you deal with >20
machines, you should definitely take time to look into Ceph.</p>
<h2 id="saturday">Saturday</h2>
<p>Saturday started off with a talk about
<a href="https://www.socallinuxexpo.org/scale12x/presentations/what-linux-can-learn-solaris-performance-and-vice-versa">SmartOS vs Linux performance tooling</a>
(<a href="http://www.slideshare.net/brendangregg/what-linux-can-learn-from-solaris-performance-and-viceversa">slides</a>
).
There wasn't much <em>new</em> there this time around, but Brendan is a good
speaker, and SmartOS is probably the most serious server-side
alternative to Linux I'd personally consider these days, so it's good
to keep tabs on what they've been working on.</p>
<p>My interestests drew me next to the
<a href="https://www.socallinuxexpo.org/scale12x/presentations/presto-sql-engine-big-data">talk</a>
about <a href="http://prestodb.io/">Presto</a>
(<a href="https://www.socallinuxexpo.org/sites/default/files/presentations/scale%2012%20theme.pdf">slides</a>
).
Takeaways:</p>
<ul>
<li>batch and interactive systems have fundamentally different needs,
e.g. for monitoring grace periods, how and when maintenance can be
performed; they require a different ops culture.</li>
<li>Dain shared background on Facebook's internal networking challenges,
and how data center power limits forced them to essentially trade
off other servers for Presto servers, to avoid network bottlenecks.</li>
<li>Presto is integrating the BlinkDB research on approximate queries,
e.g. <10% error for 10-100x faster queries sounds like a very good
trade-off.</li>
<li>many "big data" stores don't store enough statistics about index hit
rates to guide query planning</li>
</ul>
<p>I'm sad I missed
<a href="https://www.socallinuxexpo.org/scale12x/presentations/beyond-hypervisor-technical-roadmap-open-virtualization-linux-kvm">Beyond the Hypervisor</a>
(<a href="https://www.socallinuxexpo.org/sites/default/files/presentations/beyond%20the%20hypervisor.pdf">slides</a>
)
due to a schedule conflict.</p>
<p>The
<a href="https://www.socallinuxexpo.org/scale12x/presentations/whats-new-openldap">OpenLDAP talk</a>
(<a href="https://www.socallinuxexpo.org/sites/default/files/presentations/2014SCALE-hyc.pdf">slides</a>
)
was really largely about <a href="http://symas.com/mdb/">LMDB</a>
, and that's
what I came for. LMDB is a library that implements a key-value store,
with an on-disk B-tree where read operations happen purely through a
read-only <code>mmap</code>. This is a really nice architecture, pretty much as
good as a btree gets -- that is, it's probably happiest with
read-mostly workloads, and probably at its worst with small writes to
random keys. Pretty much the opposite of LevelDB, there. I wish the
benchmarks were less biased, but that seems to be the unavoidable
nature of benchmarking. LMDB has a lot of the kind of mechanical
sympathy that may remind you of
<a href="https://www.varnish-cache.org/trac/wiki/ArchitectNotes">Varnish</a>
: all
aspects of caching are offloaded to the kernel, and data can be
accessed in zero-copy fashion because the read-only mmap prevents
accidents. For Go programmers, <a href="https://github.com/boltdb/bolt">Bolt</a>
is a reimplementation of the design in pure Go, avoiding the Cgo
function call overhead, and offering a <em>much</em> nicer api than the
direct wrapper <a href="https://github.com/szferi/gomdb">szferi/gomdb</a>
. My
quick microbenchmarks say that, <em>when used from Go</em>, Bolt can be
faster.</p>
<p>Next up was
<a href="https://www.socallinuxexpo.org/scale12x/presentations/high-volume-metric-collection-visualization-and-analysis">High volume metric collection, visualization and analysis</a>
.
If I could take back those 20 minutes, I would.</p>
<p>I spent the rest of the day catching up with old friends and making
new ones.</p>
<h2 id="sunday">Sunday</h2>
<p>Clint Byrum is now at HP and working on <em>TripleO</em>, a project that aims
to make OpenStack do bare-metal deploys, and then run a public-facing
OpenStack on top of that. His
<a href="https://www.socallinuxexpo.org/scale12x/presentations/openstack-deploy-thyself-tripleo">talk</a>
was a good status report
(<a href="https://www.socallinuxexpo.org/sites/default/files/presentations/SCALE12x%20TripleO%20Clint%20Byrum.pdf">slides</a>
),
but in situations like this I always end up wanting more details.</p>
<p>For the next slot, I bounced between three different talks, not 100%
happy with any one of them. First,
<a href="https://www.socallinuxexpo.org/scale12x/presentations/hadoop-2-enabling-modern-data-architecture-through-efficient-multi-tenant">Hadoop 2</a>
(<a href="http://www.slideshare.net/ddkaiser/scale-12-x-multitenant-hadoop-2-workloads-with-yarn">slides</a>
)
was an intro to YARN et al that started off like an apologist "I swear
Hadoop and Java don't <em>really</em> suck as much as they seem to". Mark me
down as unconvinced.</p>
<p>Second,
<a href="https://www.socallinuxexpo.org/scale12x/presentations/configuration-management-101">Configuration Management 101</a>
was a good effort from a Chef developer to be party neutral, and
talked about the common things you find in all the common CM
frameworks. His references to
<a href="https://en.wikipedia.org/wiki/Promise_theory">promise theory</a>
are
pretty much dead on, and in the 3 years since I fiddled momentarily
with <a href="https://github.com/tv42/troops">https://github.com/tv42/troops</a>
, my thoughts have gone more and
more into thinking about distributed CM as an eventual consistency
problem. With Juju-inspired notifications about config changes, using
more gossip & vector clock style communication to update peers on e.g.
services provided, this might result in something very nice. That one
is definitely on the ever-growing itches to scratch list.</p>
<p>Third,
<a href="https://www.socallinuxexpo.org/scale12x/presentations/seven-problems-linux-containers">Seven problems of Linux Containers</a>
was an OpenVZ-biased look into remaining problems. Some of it was a
bit ridiculous -- who says containers <em>must</em> share a filesystem, just
mount one for each container if you want to -- and some of it was just
too OpenVZ-specific to be interesting. Still, a good topic, and OpenVZ
was groundbreaking work.</p>
<p>For the next slot, I returned from lunch too late to fit in the packed
rooms, and enjoyed breathing too much to try harder. I watched three
talks, mostly from open doorways. The hotel's AC was not really
keeping up anymore at this point, and only the main room was pleasant
to be in.</p>
<p><a href="https://www.socallinuxexpo.org/scale12x/presentations/big-data-visualization">Big Data Visualization</a>
left me wishing that 1) it wasn't fashionable to say "big data" 2)
he'd have shown more visualizations 3) he'd talked about the <em>hard</em>
parts.</p>
<p><a href="https://www.socallinuxexpo.org/scale12x/presentations/zfs-101-aka-zfs-cool-and-why-you-should-be-using-it">ZFS 101</a>
(<a href="https://www.socallinuxexpo.org/sites/default/files/presentations/scale2014.pdf">slides</a>
)
is interesting to me mostly to see what people think about & want from
storage. Btrfs is really promising in this space, feature-wise; it
still has implementation trouble like IO stalls, but the integrated
snapshots and RAID are just so much more useful and usable than any
combination of hardware RAID, software RAID, and LVM. Snapshots really
need to be a first-class concern. So far, my troubles with Btrfs are
of a magnitude completely comparable to my troubles with the
combination of LVM, LVM snapshots, HW-RAID cards dying, and SW-RAID1
sometimes booting the drive that was meant to be disabled. All in all,
I find the "not yet stable" argument a bit boring; there's a whole lot
of code and complexity in Btrfs, but it also removes the need for a
whole lot of other kinds of code and complexity. If nothing else, the
ZFS/Btrfs feature set should be a design template for future efforts;
I understand e.g. <a href="https://en.wikipedia.org/wiki/F2FS">F2FS</a>
has a
very specific design goal (think devices rather than full computers),
but not supporting snapshots in a new filesystem design is a bummer.</p>
<p>And finally, I spent time in Jordan Sissel's
<a href="https://www.socallinuxexpo.org/scale12x/presentations/fpm-package-software-and-be-happy">fpm talk</a>
.
<code>fpm</code> is a tool that converts various package formats into other
package formats, a lot like
<a href="http://kitenet.net/~joey/code/alien/">Alien</a>
. Jordan's viewpoint on
this is a frustrated admin who just wants the damn square peg to fit
in the round hole, and <code>fpm</code> is the jigsaw & hammer that'll make that
happen. I fundamentally disagree with him about the role of packaging;
the whole <em>point</em> of packaging is destroyed if the ecosystem has too
many bad packages, and the reason e.g. Debian packaging can be a lot
of work is not because cramming files in an archive should be hard,
but because making all that software work together and upgrade
smoothly actually is a difficult problem. But Jordan is an
entertaining speaker, and his point is valid; there are plenty of
cases where you don't care about the quality of the resulting package.
Just.. please don't distribute them, ok?</p>
Slides for my recent talkshttp://eagain.net/blog/2011/11/17/talks.html2012-01-08T11:21:32-08:002020-01-21T20:49:33-07:00<p>I just put up a bunch of slides from talks I've presented lately:</p>
<ul>
<li>
<p>
<a href="/talks/concurrency-oh-my/">concurrency-oh-my/</a>
Concurrency, Parallelism, Events, Asynchronicity, Oh My:
concepts and Python applications of concurrency,
2011-11-10 at <a href="http://socal-piggies.org/">SoCal Piggies</a>
.</p>
</li>
<li>
<p>
<a href="/talks/ceph-overview/">ceph-overview/</a>
Ceph Overview for the OpenStack Conference,
2011-10-07 at <a href="http://www.openstack.org/">OpenStack Conference</a>
.</p>
</li>
<li>
<p>
<a href="/talks/teuthology/">teuthology/</a>
Teuthology: a multi-machine test runner,
2011-07-01 at <a href="http://www.dreamhost.com/">DreamHost</a>
.</p>
</li>
<li>
<p>
<a href="/talks/intro-to-boto/">intro-to-boto/</a>
Introduction to Boto, and an even briefer intro to Gevent,
2011-07-01 at <a href="http://www.dreamhost.com/">DreamHost</a>
.</p>
</li>
</ul>
Keyboards influenced by touchscreenshttp://eagain.net/#2011-04-30_keyboards_influenced_by_touchscreens2011-04-30T12:00:00-07:002020-01-21T20:49:33-07:00<p>By now, you have probably used a touchscreen keyboard. We've come a
long way from the clumsy "kiosk" computers that brought touchscreen
keyboards to mainstream, a decade or two ago. But a classic keyboard
with physical keys is still preferable for the tactile feedback we get
from pressing the keys. Until touchscreens can provide that, we'll be
using traditional keyboards for a while.</p>
<p>But how do touchscreen keyboards differ from physical keyboards, and
what ideas could we copy from them to improve the user experience of
traditional keyboards? Well, for one, most touchscreen keyboards these
days don't do key repeat -- instead, they'll pop up a menu of
alternatives, often the same base letter with with various accents,
diacritics and umlaut. And you slide your finger to pick one of these
options.</p>
<p>
<img src="/blog/2011/04/30/popup.png" alt="Popup">
</p>
<p>Now, that sounds good. I for one don't know how to type the various
variations of the letters on a US keyboard, and as a Finn I actually
need "ä" and "ö" sometimes. I can type them in Emacs, but not in my IM
application or web browser.</p>
<p>Here's the idea: we don't really use autorepeat on the A-Z
characters. Instead, make a long press of a letter key bring up an
on-screen menu with variations of the basic letter.</p>
<p>On a touchscreen, choosing from the menu is immediate and fairly
intuitive. While using the mouse for that would be straight forward --
and probably what a first time user will try -- nobody wants to type
like that. We're stuck with one finger holding down the original key.
We need to make a selection from up to about 9 alternatives. Here's
two easy ideas (assuming the initial highlighted alternative is the
base letter):</p>
<ul>
<li>make space move to the next highlighted alternative, or wrap around
to the left</li>
<li>make the four primary home row keys of the <em>other</em> hand highlight
alternatives 1-4; make pressing the same home row key again act like
space, above</li>
</ul>
<p>For example, say I want to type "ä". My options are (assuming US
keyboard layout):</p>
<ol>
<li>
<p>hold "a", wait till menu pops up, while still holding "a" press
space 4 times until "ä" is highlighted, release "a"</p>
</li>
<li>
<p>hold "a", wait till menu pops up, while still holding "a" press
";", release "a"</p>
</li>
</ol>
<p>I think I'd use that more than auto-repeat. What about you?</p>
Deploy toolshttp://eagain.net/#2011-02-27_studying-poni2011-02-27T21:25:00-08:002020-01-21T20:49:33-07:00<p>I've been looking at the world of deployment tools lately. Outside of
Puppet and Chef (and ignoring the old beards Bcfg2 & Cfengine), what
other things are there?</p>
<p><a href="http://fabfile.org/">Fabric</a>
lets you write Python functions to
describe "tasks" to be run. The Python functions are run on a client
machine -- for example, the sysadmin's laptop -- and each task can be
directed to operate on hosts or roles (groups of hosts), over SSH.
The functions can run remote commands with <code>run("echo hello, world")</code> and <code>sudo("chmod u=rw,go=r /etc/passwd")</code>. Fabric is a
very useful piece of the puzzle, but doing more complex operations one
shell command at a time gets frustrating. I keep wanting something
that can run whole chunks of Python easily, on the target machine.
Fabric also does nothing to solve the problems of e.g. multiple admins
running a deploy command at the same time.</p>
<p><a href="http://samuelks.com/kokki/">Kokki</a>
is closer to Chef in it's style
(and literally, "Chef" in Finnish). It's a framework for writing
cookbooks, with actions like <code>File("/etc/greeting", content="hello, world")</code> in them. Then a configuration for a machine can invoke
certain recipes. Kokki seems to be aimed fully at running things
locally; that is, if you're deploying things, you'd run Kokki on the
target machine. Kokki is still in fairly early development, it's
website and source code don't match each other at all, and many of the
cookbooks no longer work with the current version. It also still
inherits a bit too many non-Pythonic elements from Chef, for my
tastes. Still, to this Pythonista, it looks very promising, and I will
be exploring it further.</p>
<p><a href="http://melor.github.com/poni/">Poni</a>
("Pony" in Finnish -- I want
one too!) is another Python project that takes a very different tack.
It is built on a command-line tool that lets you define your
infrastructure through hierarchical collections of key-value settings;
that is, you describe the whole multi-machine service with database
servers, load balancers, app servers and all. You can use inheritance
much like other deploy tools use cookbooks. The command-line tool
seems to be meant to be used for <em>everything</em>; the stored data is not
really meant to be edited directly. While I appreciate the polish of
the command-line tool, the editor-hostility comes off a bit
odd. Especially so when the getting started guide has me "uploading"
template files and Python source code to Poni's internal configuration
storage. Am I really supposed to have two copies of these files?</p>
<p>Once you have your infrastructure defined, Poni provides you two main
methods to actually make changes: you can create files based on
templates (that have a strong mechanism for referring to any values
from the configuration, including things like sharing the database
connection information between the DB server and the client config),
or you can run custom functions ("control commands"). The Python
functions run locally, but Poni provides a remote execution framework
very similar to the one in Fabric, though at least for now it is
significantly more verbose. And, to my disappointment, doesn't really
allow running full Python functions remotely either.</p>
<p>Somewhat confusing is the difference between the "create a file" and
the "run control command" functionality. It is not quite clear how the
whole is intended to orchestrate the full deployment, and the examples
are both lacking and misleading. For example, right now the <a href="http://melor.github.com/poni/examples/puppet.html">Puppet
deployment example</a>
requires you to run a command to create some files, watch it fail, run
another command to install software, then run the first command again
to create the rest of the files. (Kind of weird to deploy Puppet with
Poni in the first place..)</p>
<p>There is one thing about Poni that I am already starting to
dislike. Currently, every identifier you need to refer to on the
command line is given as a regexp, and commands act on all
matches. This leads to high risk for operator errors: for example, the
documentation itself uses <code>$find("webshop/frontend")</code> as an example;
yet that would also match <code>webshop/frontendforsomethingelse</code>. I do
hope the author changes his mind about regexps everywhere.</p>
<p>Much like Kokki, Poni is very early on in its development; it's
command-line tools and things like variable referencing are top notch,
but the picture is very much not complete yet. But this one is
definitely a project to watch.</p>
<hr>
<p>For posterity: I filed a bunch of issues about the things I bumped
into:</p>
<ul>
<li><a href="https://github.com/melor/poni/issues/2">https://github.com/melor/poni/issues/2</a>
</li>
<li><a href="https://github.com/melor/poni/issues/3">https://github.com/melor/poni/issues/3</a>
</li>
<li><a href="https://github.com/melor/poni/issues/4">https://github.com/melor/poni/issues/4</a>
</li>
<li><a href="https://github.com/melor/poni/issues/5">https://github.com/melor/poni/issues/5</a>
</li>
<li><a href="https://github.com/melor/poni/issues/6">https://github.com/melor/poni/issues/6</a>
</li>
</ul>
EuroPython 2008 videos are uphttp://eagain.net/#2008-08-11_ep-talk-videos2008-08-11T18:27:00+03:002014-02-05T20:01:15-08:00<p>Update: Well, they used to be. blip.tv pulled a nasty stunt and
removed tons of user content. And <a href="http://europython.org/">EuroPython</a>
itself has a nasty habit of not providing archived websites for
previous years. This all has linkrotted. Don't you hate the internets?</p>
<p>My talks were:</p>
<ul>
<li>
<a href="/talks/pythonic-fs/">My God, it's Full of Files -- Pythonic filesystem abstractions</a>
</li>
<li>
<a href="/talks/version-control-for-du-developers/">Version Control for Du^H^HDevelopers</a>
</li>
</ul>
EuroPython 2008 wrap uphttp://eagain.net/#2008-07-19_europython20082008-07-19T23:17:00+03:002008-07-20T14:02:00+03:00<div class="next-pull-left next-breather"></div>
<p>
<img src="/blog/2008/07/19/europython2008-lightning-talks.jpg" alt="Lightning talks">
</p>
<p><a href="http://www.europython.org/">EuroPython 2008</a>
was fun. I presented two talks (<a href="http://eagain.net/talks/pythonic-fs/">My God, it's Full
of Files -- Pythonic filesystem abstractions</a>
and <a href="http://eagain.net/talks/version-control-for-du-developers/">Version Control
for Du^H^HDevelopers</a>
) and one lightning talk (<em>RST+S5 for your
slides</em>), participated in a bunch of open space sessions, listened to
about 13 talks, took a bunch of pictures, but most importantly had
interesting hallway conversations with interesting people.</p>
<div class="next-pull-right next-breather"></div>
<p>
<img src="/blog/2008/07/19/europython2008-buildout.jpg" alt="&quot;Buildout for Pure Python Project&quot;, Carsten Rebbien">
</p>
<p>As usual, <a href="http://codespeak.net/pypy/">PyPy</a>
was heavily represented, and seems to be making nice
progress toward being the nice and featureful default Python
implementation of the future. I especially liked the restricted
execution features and the <a href="http://llvm.org/">LLVM</a>
backend. The <a href="http://pypi.python.org/pypi/zc.buildout">zc.buildout</a>
<a href="http://registration.europython.eu/talk_abstracts.html#32">talk</a>
made
me decide I will try to replace (part of?) one custom deploy mechanism
with <code>zc.buildout</code> -- most likely I'll end up rewriting most of the
current things as <code>zc.buildout</code> recipes, but hopefully some of the
pre-existing recipes will be useful, and hopefully I can then later
reuse the recipies I create for this setup.</p>
<p>Personally, I think my talks went ok. I understand videos will be
available later, as soon as transcoding etc are finished. I'm anxious
to see them myself, as I'm still finetuning my public speaking
skills. I'm learning, though -- this year I had no trouble staying
within my time slot, even when I was adjusting verbosity on the fly.</p>
<p>For some reason, I felt underprepared for the <a href="http://eagain.net/talks/pythonic-fs/">filesystem API talk</a>
,
but ultimately people liked the idea of a consistent Pythonic
filesystem API enough that we had an open space session on it, and
people were enthusiastic about a sprint to prototype the API. Which is
what we ended up doing, too -- I'll blog separately about the results
of that.</p>
<p>My <a href="http://eagain.net/talks/version-control-for-du-developers/">decentralized version control talk</a>
seemed to me to go over more
smoothly; I guess that's just because I've been thinking about version
control and project management a lot lately, so it was easy to talk
about the topic in a relaxed way. On the other hand, it wasn't as
much a <em>call to action</em>, and it really was overly generic, so I didn't
get as strong audience participation there. We did have an interesting
conversation about branch management strategies and such, though. I
consciously tried to keep the talk on a generic level, as I felt a
pure <code>git</code> talk would have alienated some listeners, but I did end
up feeling restricted by that. There was some interest on a <code>Teach me git</code> -style session, but what we ended up doing was just talking one
on one about getting started with <code>git</code>, during the sprints. Sorry
if I missed any one of you -- grab me on <code>#git</code> to continue, or find
me in future conferences ;)</p>
<div class="next-pull-right next-breather"></div>
<p>
<img src="/blog/2008/07/19/europython2008-twisted-qa.jpg" alt="Twisted Q&amp;A session">
</p>
<p>I was requested to organize an open space session for Twisted Q&A, and
that is exactly what we did. We went through a bunch of things
related to asynchronous programming concepts, Deferreds, working with
blocking code and libraries, database interfaces, debugging and unit
testing.</p>
<p>I was also pulled in to another Twisted open space session, that was
mostly about what greenlets are and how to use them. I tried to
explain the differences between classical Deferreds,
<code>deferredGenerator</code>/<code>inlineCallbacks</code>, and greenlets, to the best
of my understanding. As a summary, with greenlets any function you
call can co-operatively yield execution (I mean yield in the
scheduling meaning, giving away your turn to run, not in the Python
generator meaning -- interestingly <code>inlineCallbacks</code> etc actually
make those be the same thing... my kernel instincts make me want to
say "sleep"). Yielding in any subroutine means anything you do may end
up mutating your objects -- which is the root evil behind threading we
wanted to get away from. All the other mechanisms keep the top-level
function in explicit control of yielding. Around that time, most
people left for lunch, but about three of us stayed and talked about
debugging Deferreds and network packet processing with
<code>twisted.pair</code> and friends.</p>
<div class="next-pull-left next-breather"></div>
<p>
<img src="/blog/2008/07/19/europython2008-lobby.jpg" alt="Lobby area">
</p>
<p>One of the interesting hallway conversations was about what happens
when upstream web hosting listed on <a href="http://pypi.python.org/pypi">PyPI</a>
is failing. It seems PyPI
already does some sort of mirroring, but even that might not be
enough. Many companies seem to be bundling eggs of their dependencies
in their installation package, which sounds like a good setup for
commercial click-to-install deployment. But it would still be good to
see a <a href="http://cpan.org/">CPAN</a>
-style mirror network for PyPI, and at least some people
seemed even motivated to donating servers and bandwidth. Personally,
I'm mostly spoiled by the combination of Debian/Ubuntu and
decentralized version control, and my level of paranoia is too high to
automatically install unverified software from the internet anyway.
My primary motivation in the conversation was to point out that PyPI
already has some sort of mirroring/upload setup, and that you'd really
want to specify exact versions and SHA-1 hashes of your dependencies.
Optionally, you could delegate the known good hash storage to PyPI
(assuming you trusted PyPI not to attack you), but that would require
a full Debian-style signature chain from a trusted key, or you'd be
owned by anyone capable of MITM attacks, DNS forgery, or cracking a
PyPI mirror.</p>
EuroPython 2008 talk #1: My God it's Full of Fileshttp://eagain.net/#2008-07-07_ep2008-pythonic-fs2008-07-07T10:31:00+03:002020-01-21T20:49:33-07:00<p>I published slides for my first <a href="http://www.europython.org/">Europython</a>
talk. Note the slides
might not easy to understand without the actual talk -- the video
streaming at <a href="http://ustream.tv/europython">Ustream</a>
will work if the network infrastructure can take
it, and a downloadable video should be available later.</p>
<p><strong>My God, it's Full of Files</strong></p>
<p><em>Pythonic filesystem abstractions: An overview of different
filesystem(-like) APIs in Python and attempts for unifying them.</em></p>
<p>There's a lot of different filesystem(-like) APIs in Python. I intend
to provide an overview of existing projects, their status and
capabilities, and hopefully inspire you to work on improving things.</p>
<p>
<a href="/talks/pythonic-fs/">Read more...</a>
</p>
Incremental mapreducehttp://eagain.net/#2007-11-24_incremental-mapreduce2007-11-24T03:48:00+02:002020-01-21T20:49:33-07:00<p>So Google has their <a href="http://research.google.com/archive/mapreduce.html">MapReduce</a>
, and the people behind <a href="http://couchdb.com/">CouchDB</a>
are
throwing around <a href="http://groups.google.com/group/couchdb/browse_thread/thread/3ccb595c69ac606d?hl=en">their ideas</a>
. I spent some time thinking about
incremental mapreduce around July, and it's time I type out that page
full of scribbles.</p>
<p>First of all: I think the ideas thrown out by Damien above aren't
really mapreduce, As Google Intended. The real power of mapreduce is
in its inherent combination of parallelism and chainability, output of
one mapreduce is input to another, each processing step can run
massively in parallel with each other, etc. The proposed design is
like a one-iteration retarded cousin of mapreduce.</p>
<p>With that bashing now done (sorry), here's what I was thinking:</p>
<p>The way I imagined building an incremental mapreduce mechanism,
without storing the intermediate data and just recomputing chunks that
are out-of-date (which would be lame), is to add one extra concept
into the system: call it "demap". It will basically create "negative
entries" for the old data. This is basically what Damien did by
providing both the old and new data map calls, all the time, just said
differently, and I think my way might make the average call a lot
simpler. And I don't see any reason why my version wouldn't be
parallelizable, chainable, and generally yummy.</p>
<p><a href="http://eagain.net/articles/incremental-mapreduce/">Read the article</a>
</p>
KVM, the virtualization mechanism, rockshttp://eagain.net/#2007-10-12_kvm-1-xen-02007-10-12T22:59:00+03:002020-01-21T20:49:33-07:00<p>For years now, my primary machine has been a laptop. I've been
avoiding running <a href="http://www.xensource.com/">Xen</a>
<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> because the <code>hypervisor</code>
it injects between hardware and Linux hasn't been very friendly for
power saving. Modern laptops are too often hot enough without any
extra help.</p>
<p>Well, I'm happy to say that as of now, <a href="http://kvm.qumranet.com/">KVM</a>
is definitely stable
enough to replace my use of Xen for test setups. I expect that shortly
it will become what I recommend for server use, too. And because it
lets Linux be Linux, instead of doing any oddities with the hardware,
all the powersaving etc goodness still works perfectly.</p>
<p>So far, I've stumbled on two things:</p>
<ol>
<li>
<p>The version of KVM in Ubuntu, even <code>gutsy</code>, is just too old. With a
bit of fiddling with the patches, KVM v46 built a perfectly working
<code>deb</code>.</p>
</li>
<li>
<p>The <code>isolinux</code> graphical bootup, used in <code>Ubuntu</code>, crashes KVM
(and with v28, it crashes the host machine -- beware!). See the <a href="https://bugs.launchpad.net/ubuntu/+bug/83642">bug
report</a>
. I got around
that by using <a href="http://archive.ubuntu.com/ubuntu/dists/gutsy/main/installer-amd64/current/images/netboot/">mini.iso</a>
,
but you could always fall back to <code>debootstrap</code>; that was all we
ever really had with Xen.</p>
</li>
</ol>
<p>And while I have the soapbox: <em>I have a dream</em>. It's KVM running
with SDL/VNC graphics in a window that's <strong>resizeable</strong> all the way
to the virtual machine, with XRandR. Please make it happen!</p>
<p><strong>Patches</strong></p>
<p>So, the edited version of the patches:</p>
<ul>
<li>
<a href="/blog/2007/10/12/01_use_bios_files_in_usr_share_kvm.patch">01_use_bios_files_in_usr_share_kvm.patch</a>
</li>
<li>
<a href="/blog/2007/10/12/06_no_system_linux_kvm_h.patch">06_no_system_linux_kvm_h.patch</a>
</li>
<li>
<a href="/blog/2007/10/12/04_do_not_print_rtc_freq_if_ok.patch">from-debian-qemu/04_do_not_print_rtc_freq_if_ok.patch</a>
</li>
</ul>
<p>And just remove <code>from-debian-qemu/62_linux_boot_nasm.patch</code>, it
seems to have made it upstream.</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p>What's up with <a href="http://www.xensource.com/">http://www.xensource.com/</a>
, by the
way? That website is just a pile of horrible non-informative
enterprise speak. Utterly useless, and that tends to alienate the
techie community pretty fast. At least it's alienating me. <a href="#fnref:1" class="footnote-backref" role="doc-backlink">↩︎</a></p>
</li>
</ol>
</div>
Snakepit and gitosis, things I've been working onhttp://eagain.net/#2007-10-12_snakepit-and-gitosis2007-10-12T01:46:00+03:002020-01-21T20:49:33-07:00<p>A brief update of things I've been working on:</p>
<p><strong>Snakepit</strong></p>
<p><code>Snakepit</code> is a port of (part of) <a href="http://www.hivedb.org/">HiveDB</a>
to Python and <a href="http://www.sqlalchemy.org/">SQLAlchemy</a>
.
It will help you write database-backed applications that need to scale
further than one database server, or even a master-slave setup, can
take you. It's MIT licensed, that is, pretty much as free-for-all as
it can be. And it's still work in progress, so don't be too harsh yet
;)</p>
<p>See <a href="https://github.com/tv42/snakepit">https://github.com/tv42/snakepit</a>
for more.</p>
<p><strong>gitosis</strong></p>
<p><code>gitosis</code> aims to make hosting <code>git</code> repos easier and safer. It
manages multiple repositories under one user account, using SSH keys
to identify users. End users do not need shell accounts on the server,
they will talk to one shared account that will not let them run
arbitrary commands. <code>gitosis</code> is licensed under the GPL.</p>
<p>First real release will come as soon as I have to time to go through a
couple of really minor nits. It's been self-hosting for a long time
now.</p>
<p>See <a href="https://github.com/tv42/gitosis">https://github.com/tv42/gitosis</a>
for more.</p>
Rotaclock -- a unique clock where the whole wrist display the timehttp://eagain.net/blog/2007/06/06/rotaclock.html2007-06-06T15:09:00-08:002015-09-11T17:09:52-07:00<p><em>(This post didn't originally get published due to technical problems
with the graphics. It was written in 2007 and was rescued & inserted
into the archive in 2015.)</em></p>
<p>Imagine a wristwatch with nothing but the wristband. Time flows around
your wrist; parts of the wristband light up to indicate what time it
is, and you can watch seconds race loops around your wrist.</p>
<p>The concept may need some tuning, as some of the readings are
awkwardly behind your wrist. Perhaps there's a gap there that the
indicators jump over, around the closing mechanism of the wrist band.</p>
<p>
<img src="/blog/2007/06/06/rotaclock.dia.svg" alt="rotaclock">
</p>
Git for Computer Scientistshttp://eagain.net/#2007-03-29_git-for-computer-scientists2007-03-29T18:10:00-08:002007-03-29T18:14:00-08:00<p>I wrote a <a href="http://eagain.net/articles/git-for-computer-scientists/">brief introduction</a>
to <a href="http://git.or.cz/">git</a>
for people who are not scared by words like <a href="http://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed Acyclic Graph</a>
. Read it
here: <a href="http://eagain.net/articles/git-for-computer-scientists/">Git for Computer Scientists</a>
</p>
Howto host `git` on your Linux boxhttp://eagain.net/#2007-03-22_howto-host-git2007-03-23T17:43:00-07:002008-03-19T22:02:00+02:00<div class="panel panel-danger">
<div class="panel-heading">
<h3 class="panel-title">Warning</h3>
</div>
<div class="panel-body">
<p><strong>This solution is obsolete. Use <a href="https://github.com/tv42/gitosis">gitosis</a> instead!</strong></p>
<p><strong>Gitosis is basically
<code>git-shell-enforce-directory</code>'s big brother, and an
actual software project. <a
href="https://github.com/tv42/gitosis">Use it</a>.</strong></p>
</div>
</div>
<p><strong>Updated</strong> to drop <code>--use-separate-remote</code> from <code>git clone</code>, it's
the default.</p>
<p><strong>Updated</strong> to add <code>--read-only</code> to <code>git-shell-enforce-directory</code>.</p>
<p>I've run repeatedly into cases where I want to provide services to
people without really trusting them. I do not want to give them shell
access. I don't want to even create separate unix user accounts for
them at all. But I do want to make sure the service they use is safe
against e.g. password sniffing.</p>
<p>Instead of trying to run the version control system over HTTPS (like
<a href="http://subversion.tigris.org/">Subversion</a>
's <code>mod_dav_svn</code> that
will only work with Apache, which I don't run), I want to run things
through <a href="http://openssh.com/">SSH</a>
. SSH is the de facto unix tool
for securing communications between machines.</p>
<p>Now, I said I don't want to create a unix user account for every
developer using the version control system. With SSH, this means using
a shared account, usually named by the service it provides: <code>svn</code>,
<code>git</code>, etc. To identify different users of that account, do not give
the account a password, but use SSH keys instead. To avoid giving
people full shell access, use a <code>command="..."</code> when adding their
public key to <code>~/.ssh/authorized_keys</code>.</p>
<p>For Subversion, I submitted an enhancement to add <code>--tunnel-user</code>,
to make sure the commit gets identified as the right user, and then
used <code>command="..."</code> with the with arguments, like this (all on one
line):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">command="/srv/example.com/repo/svn/svnserve -t
</span></span><span class="line"><span class="cl"> --root /srv/example.com/repo/svn/view/examplegroup
</span></span><span class="line"><span class="cl"> --tunnel-user jdoe" ssh-rsa ... jdoe@example.com
</span></span></code></pre></div><p>Where the <code>view</code> directory is a bunch of symlinks to the actual
repositories, allowing me to do group-based access control.</p>
<p>With <a href="http://git.or.cz/">git</a>
, the author of the changeset is
recorded way before the SSH connection is opened. Without building
some sort of access control in <code>git</code> hooks on the server, every
developer can pretty much ruin the repository by overwriting branches
with bogus commits. What they will not have is access outside of the
repository, or a way to actually remove the old commits from the disk
(unless you run <code>git prune</code> on the server). The distributed nature
of <code>git</code> makes this reasonably easy to detect, and pretty much
trivial to recover from. For any real trust in the code, you should
look at signed tags anyway. The included wrapper allows you to have
read-only users, but provides no detailed access control against
developers with write access; they just won't be able to escape to the
rest of the filesystem.</p>
<p>So, with that introduction out of the way, let's get to configuring:</p>
<h2 id="install-git-on-the-server">Install <code>git</code> on the server</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">sudo apt-get install git-core git-doc
</span></span></code></pre></div><h2 id="create-the-directory-structure-store-the-repositories-and-related-files">Create the directory structure store the repositories and related files</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">sudo install -d -m0755 <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> /srv/example.com/repo/git <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> /srv/example.com/repo/git/.ssh <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> /srv/example.com/repo/git/repos <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> /srv/example.com/repo/git/view
</span></span></code></pre></div><h2 id="create-the-shared-user-account-for-this-service">Create the shared user account for this service</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">sudo adduser <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --system <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --home /srv/example.com/repo/git <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --no-create-home <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --shell /bin/sh <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --gecos <span class="s1">'git version control'</span> <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --group <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> --disabled-password <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span> git
</span></span></code></pre></div><h2 id="limit-access">Limit access</h2>
<p>Set up a script that makes sure only relevant <code>git</code> commands can be
run via SSH, and to limit the visible section of the filesystem to
things you actually want to give access to; put this file in
<code>/usr/local/bin/git-shell-enforce-directory</code>
(
<a href="/blog/2007/03/22/git-shell-enforce-directory">download</a>
) and <code>chmod a+x</code> it</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="ch">#!/usr/bin/python</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Copyright (c) 2007 Tommi Virtanen <tv@eagain.net></span>
</span></span><span class="line"><span class="cl"><span class="c1">#</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Permission is hereby granted, free of charge, to any person</span>
</span></span><span class="line"><span class="cl"><span class="c1"># obtaining a copy of this software and associated documentation files</span>
</span></span><span class="line"><span class="cl"><span class="c1"># (the "Software"), to deal in the Software without restriction,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># including without limitation the rights to use, copy, modify, merge,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># publish, distribute, sublicense, and/or sell copies of the Software,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># and to permit persons to whom the Software is furnished to do so,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># subject to the following conditions:</span>
</span></span><span class="line"><span class="cl"><span class="c1">#</span>
</span></span><span class="line"><span class="cl"><span class="c1"># The above copyright notice and this permission notice shall be</span>
</span></span><span class="line"><span class="cl"><span class="c1"># included in all copies or substantial portions of the Software.</span>
</span></span><span class="line"><span class="cl"><span class="c1">#</span>
</span></span><span class="line"><span class="cl"><span class="c1"># THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,</span>
</span></span><span class="line"><span class="cl"><span class="c1"># EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF</span>
</span></span><span class="line"><span class="cl"><span class="c1"># MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND</span>
</span></span><span class="line"><span class="cl"><span class="c1"># NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS</span>
</span></span><span class="line"><span class="cl"><span class="c1"># BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN</span>
</span></span><span class="line"><span class="cl"><span class="c1"># ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN</span>
</span></span><span class="line"><span class="cl"><span class="c1"># CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE</span>
</span></span><span class="line"><span class="cl"><span class="c1"># SOFTWARE.</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Enforce git-shell to only serve repositories</span>
</span></span><span class="line"><span class="cl"><span class="c1"># in the given directory. The client should refer</span>
</span></span><span class="line"><span class="cl"><span class="c1"># to them without any directory prefix.</span>
</span></span><span class="line"><span class="cl"><span class="c1"># Repository names are forced to match ALLOW.</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">optparse</span><span class="o">,</span> <span class="nn">re</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">die</span><span class="p">(</span><span class="n">msg</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span> <span class="o">>></span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">,</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">msg</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">getParser</span><span class="p">():</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">optparse</span><span class="o">.</span><span class="n">OptionParser</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">usage</span><span class="o">=</span><span class="s1">'%prog [OPTIONS] DIR'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">description</span><span class="o">=</span><span class="s1">'Allow restricted git operations under DIR'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--read-only'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'disable write operations'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span><span class="o">=</span><span class="s1">'store_true'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">parser</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">ALLOW_RE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s2">"^(?P<command>git-(?:receive|upload)-pack) '[a-zA-Z][a-zA-Z0-9@._-]*(/[a-zA-Z][a-zA-Z0-9@._-]*)*'$"</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">COMMANDS_READONLY</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="s1">'git-upload-pack'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">COMMANDS_WRITE</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="s1">'git-receive-pack'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">os</span><span class="o">.</span><span class="n">umask</span><span class="p">(</span><span class="mi">0022</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">getParser</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">options</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">path</span><span class="p">,)</span> <span class="o">=</span> <span class="n">args</span>
</span></span><span class="line"><span class="cl"> <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Missing argument DIR.'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">cmd</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'SSH_ORIGINAL_COMMAND'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">cmd</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">die</span><span class="p">(</span><span class="s2">"Need SSH_ORIGINAL_COMMAND in environment."</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span> <span class="ow">in</span> <span class="n">cmd</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">die</span><span class="p">(</span><span class="s2">"Command may not contain newlines."</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">match</span> <span class="o">=</span> <span class="n">ALLOW_RE</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">match</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">die</span><span class="p">(</span><span class="s2">"Command to run looks dangerous"</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">allowed</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">COMMANDS_READONLY</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="ow">not</span> <span class="n">options</span><span class="o">.</span><span class="n">read_only</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">allowed</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">COMMANDS_WRITE</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="s1">'command'</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">allowed</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">die</span><span class="p">(</span><span class="s2">"Command not allowed"</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">os</span><span class="o">.</span><span class="n">execve</span><span class="p">(</span><span class="s1">'/usr/bin/git-shell'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'git-shell'</span><span class="p">,</span> <span class="s1">'-c'</span><span class="p">,</span> <span class="n">cmd</span><span class="p">],</span> <span class="p">{})</span>
</span></span><span class="line"><span class="cl"> <span class="n">die</span><span class="p">(</span><span class="s2">"Cannot execute git-shell."</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">main</span><span class="p">(</span><span class="n">args</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
</span></span></code></pre></div>
<h2 id="create-your-first-repository">Create your first repository</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="nb">cd</span> /srv/example.com/repo/git/repos
</span></span><span class="line"><span class="cl">sudo install -d -o git -g git -m0700 myproject.git
</span></span><span class="line"><span class="cl">sudo -H -u git env <span class="nv">GIT_DIR</span><span class="o">=</span>myproject.git git init
</span></span></code></pre></div><p>(with <code>git</code> older than v1.5, use <code>init-db</code> instead of <code>init</code>)</p>
<h2 id="set-up-an-access-control-group-and-give-it-access-to-that-repository">Set up an access control group and give it access to that repository</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="nb">cd</span> /srv/example.com/repo/git/view
</span></span><span class="line"><span class="cl">sudo install -d -m0755 mygroup
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> mygroup
</span></span><span class="line"><span class="cl">sudo ln -s ../../repos/myproject.git myproject.git
</span></span></code></pre></div><p>You can also use subdirectories of <code>view/mygroup</code> to organize
the repositories hierarchically.</p>
<p>Note, one SSH public key will belong to exactly one group, but if
necessary you can create a separate group for each account for
absolute control.</p>
<p>Note, access to repository implies write access to repository, at
least for now. You could make</p>
<h2 id="authorize-users">Authorize users</h2>
<p>Get an SSH public key from a developer and authorize them to access the group</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="nb">cd</span> /srv/example.com/repo/git
</span></span><span class="line"><span class="cl">sudo vi .ssh/authorized_keys
</span></span></code></pre></div><p>How the developer generates their key is out of scope here.</p>
<p>Add a line like this, with the public key in it (all on one line,
broken up in the middle of word to make sure there is no
misunderstanding about when to use a space and when not to):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">command="/usr/local/bin/git-shell-enforce-directory /srv/exampl
</span></span><span class="line"><span class="cl"> e.com/repo/git/view/mygroup",no-port-forwarding,no-X11-forwar
</span></span><span class="line"><span class="cl"> ding,no-agent-forwarding,no-pty ssh-rsa ... jdoe@example.com
</span></span></code></pre></div><p>Or to allow only read-only access, add <code>--read-only</code> as an option.</p>
<h2 id="you-can-now-push-things-to-the-repository-with">You can now push things to the repository with</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">git push git@myserver.example.com:myproject.git mybranch:refs/heads/master
</span></span></code></pre></div><p>Note that before the first push, your server-side repository will
not contain even an initial commit, and can't really be cloned.</p>
<h2 id="now-the-developer-can-clone-the-repository">Now the developer can clone the repository</h2>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">git clone git@myserver:myproject.git
</span></span></code></pre></div><p>or to avoid some behavior of older git that I consider confusing
(needs <code>git</code> v1.5 or newer):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">git clone -o myserver git@myserver:myproject.git
</span></span></code></pre></div><p>They will probably want to set up <code>ssh-agent</code> to avoid typing the
passphrase all the time.</p>
<p>And you're done! Good luck with your adventures with <code>git</code>, and
welcome to the 21st century and to distributed version control
systems.</p>
Howto buy a used car in Californiahttp://eagain.net/#2007-02-20_buy-a-car2007-02-20T10:40:00-08:002007-02-20T23:59:00-08:00<p>Tomorrow, if everything goes well, I will buy a used car from an
individual in California. Here's a checklist of things for that, to
help others in similar situations. Some things may have already been
omitted because they weren't relevant for me, so you may want to
independently browse the websites I'm using as sources. I'm skipping
everything related to finances and haggling. I'm not covering cases
where the car isn't in good condition. Also, buying from dealerships
is different. Good luck.</p>
<p>(Updated to mention REG 262.)</p>
<ol>
<li>
<p>Find an ad on craigslist or whatever. I liked
<a href="http://losangeles.listpic.com/car/">http://losangeles.listpic.com/car/</a>
</p>
</li>
<li>
<p>Call the seller. Don't just email, start evaluating the seller.
Things to ask: <a href="http://www.carbuyingtips.com/questions.html">http://www.carbuyingtips.com/questions.html</a>
(though
that list is horribly extensive; I'd rather just pick the three
best-looking candidates and do that thing on the spot; you'll need
to doublecheck anyway to make sure the seller wasn't lying).</p>
<p>Ask for:</p>
<ul>
<li>the VIN of the car (usually 17 characters, on windshield)</li>
<li>full name of the seller (you'll need it anyway for the check)</li>
</ul>
</li>
<li>
<p>Check the <a href="http://www.carfax.com/">CARFAX</a>
report for the car.
Go for the $24.95 30-day option, you should always look at more
than one car.</p>
<ul>
<li>compare odometer, accident history etc with what seller said
(better yet, just avoid accident cars, the risk isn't worth it)</li>
<li>make sure it's not an old junk car poorly repaired ("salvage" title)</li>
</ul>
</li>
<li>
<p>Check the <a href="http://www.smogcheck.ca.gov/VEHTESTS/PUBTSTQRY.ASPX">smog check history</a>
(fails are
a sign of trouble). In California, in most of the cases, the seller
is required to provide a certificate of check newer than 90 days
old -- it seems the information flow to website is slow enough that
this latest check does not show up, or something.</p>
</li>
<li>
<p>Check that the car hasn't suffered <a href="http://www.autocheck.com/consumers/stormDamageAction.do">storm damage</a>
.
(Annoying website demands cookies for a simple form. Suck.)</p>
</li>
<li>
<p>Make an appointment and go see the car. In good sunlight, you want
to see the car.</p>
<ul>
<li>
<p>for some things to check, <a href="http://autorepair.ca.gov/StdPage.asp?Body=/geninfo/publications/Used_Car_Guide-Jan_1996.htm">read more</a>
,
especially <code>On-the-Lot Checklist</code>, <code>Road and Test Checklist</code></p>
</li>
<li>
<p>you <em>have</em> to drive the car; leave a photocopy of your drivers
license if needed for assurance</p>
</li>
<li>
<p>bring a car nut friend who knows what to check; play good cop bad cop
(this checklist is not as good, as I am <em>not</em> a car nut)</p>
</li>
<li>
<p>on the lot:</p>
<ul>
<li>paint chips</li>
<li>cracked windows</li>
<li>accident damage</li>
<li>signs of water damage</li>
<li>tires</li>
<li>check the VIN you were given against windshield, doors, engine,
dashboard, major body parts; if they don't match you are
dealing with a criminal</li>
</ul>
</li>
<li>
<p>while driving:</p>
<ul>
<li>brake hard, is it even</li>
<li>rev the engine, does it sound healthy</li>
<li>make plenty of starts and stops</li>
<li>go through a manual gearbox, there should be no grinding noises</li>
<li>see if the car will drive straight with your hands off the wheel</li>
<li>how does the clutch feel?</li>
<li>listen for noises throughout the test drive</li>
</ul>
</li>
<li>
<p>after driving:</p>
<ul>
<li>check for leaks</li>
</ul>
</li>
<li>
<p>to be really careful, you should check a lot of things like</p>
<ul>
<li>AC</li>
<li>windows up/down</li>
<li>radio/cd/speakers</li>
<li>etc</li>
</ul>
</li>
</ul>
</li>
<li>
<p>Take the car to a mechanic of your choosing for
evaluation. Alternatively, choose to trust on service from brand
name vendor.</p>
</li>
<li>
<p>Check the price against <a href="http://www.kbb.com/">Kelley Blue Book</a>
etc.</p>
</li>
<li>
<p>Now you're in business. Only bureaucracy and doublechecking left.</p>
</li>
<li>
<p>If you know what you're willing to pay, go to your bank and get a
cashiers check. Or two alternatives, if that works out. Otherwise,
you will need to go back to the bank after haggling, and without a
deposit the seller may have sold the car to someone else. Or
something. Tuff.</p>
<p>(Some people say haggling at dealerships is easier if you show up
with a cashiers check just a bit short from what they're asking.)</p>
<p>Don't pay with cash, there's even less chance of getting it back
than cancelling a check.</p>
<p>If you happen to be a seller reading this, do the actual sale in a
bank to be sure you aren't being cheated.</p>
</li>
<li>
<p>Download and print <a href="http://www.dmv.ca.gov/forms/formsmost.htm">PDF forms from DMV</a>
.</p>
<ul>
<li>
<p><code>Bill of Sale</code> (<a href="http://www.dmv.ca.gov/forms/reg/reg135.pdf">PDF</a>
)</p>
</li>
<li>
<p><code>Statement of Facts</code> may interest you (<a href="http://www.dmv.ca.gov/forms/reg/reg256.pdf">PDF</a>
)</p>
</li>
<li>
<p>things the seller needs: <code>Notice of Release of Liability</code> (<a href="http://www.dmv.ca.gov/forms/reg/reg138.pdf">PDF</a>
)</p>
</li>
</ul>
</li>
<li>
<p>In California, sellers are required to provide a smog certificate.
Make sure you get one. Stuff on smog checks:
<a href="http://www.autorepair.ca.gov/stdPAge.asp?Menu=/includes/Menu_GenInfo.htm&Body=/Geninfo/Factsheets/Program_Areas.htm">1</a>
,
<a href="http://www.autorepair.ca.gov/StdPage.asp?Body=/smogcheck/doineed.htm">2</a>
,
<a href="http://www.dmv.ca.gov/vr/smogfaq.htm">3</a>
.
Frankly, I'm still a bit confused myself when <em>I</em> will need to do a smog check.</p>
</li>
<li>
<p>Check that the registration is current and that the car wasn't
repurchased under the California "Lemon law". (Err, how? As I
understand it should say so in the <code>Certificate of Title</code> paper)</p>
</li>
<li>
<p>The seller should find the "pink slip" aka <code>Certificate of Title</code></p>
<ul>
<li>
<p>if said paper does not have form fields for ownership transfer
and odometer reading, you need form "REG 262" from the DMV, which
is printed on special paper and not available as PDF. SUCK!</p>
</li>
<li>
<p>check seller name against his drivers license</p>
</li>
<li>
<p>fill it out with both of your info and both sign it; for
instructions search for "Where do I sign?" on <a href="http://www.dmv.ca.gov/pubs/brochures/fast_facts/ffvr32.htm">this page</a>
</p>
</li>
<li>
<p>if a bank or something still owns a chunk of the car, their
signature is also needed on the pink slip; I'd be inclined to
avoid the complexity</p>
</li>
<li>
<p>fill in the odometer value, both sign (<a href="http://www.dmv.ca.gov/pubs/brochures/reg/odometer.htm">read more</a>
)</p>
</li>
<li>
<p>seller keeps the <a href="http://www.dmv.ca.gov/pubs/brochures/reg/nrl.htm">Notice of Transfer and Release of Liability</a>
part and
submits to DMV.</p>
</li>
<li>
<p>buyer fills in the back of the title to transfer ownership</p>
</li>
</ul>
</li>
<li>
<p>Things to ask for before leaving</p>
<ul>
<li>is a special wheel lug key needed, get it</li>
<li>are there extra keys</li>
<li>how does the car alarm work</li>
<li>on a convertible, how does the roof work</li>
<li>any "tricks" you should know</li>
</ul>
</li>
<li>
<p>Now you're ready to leave with your new car! Check that you have</p>
<ul>
<li>
<p><code>Certificate of Title</code>, signed by both, also odometer section</p>
</li>
<li>
<p><code>Bill of Sale</code>, signed by both</p>
</li>
<li>
<p>maintenance records</p>
</li>
<li>
<p>smog certification</p>
</li>
<li>
<p>owners manuals, repair manuals</p>
</li>
<li>
<p>spare tire, jack</p>
</li>
</ul>
<p>(sources <a href="http://www.dmv.ca.gov/vr/checklists/ownership.htm">1</a>
,
<a href="http://www.carbuyingtips.com/used.htm">2</a>
)</p>
</li>
<li>
<p>Seller has 5 days to submit <code>Notice of Transfer and Release of Liability</code>
to DMV. Do it <a href="https://mv.dmv.ca.gov/nrl/welcome.do">online</a>
.</p>
</li>
<li>
<p>Buyer has 10 days days to report ownership change to DMV and max 30
days to pay the fees. Read more: <a href="http://www.dmv.ca.gov/vr/vr_info.htm#BM2522">checklist</a>
, <a href="http://www.dmv.ca.gov/vr/checklists/ownership.htm">things to send
to DMV</a>
,
<a href="http://www.dmv.ca.gov/pubs/brochures/fast_facts/ffvr32.htm">more info</a>
.</p>
</li>
<li>
<p>Taxes?</p>
</li>
</ol>
<p>General resources:</p>
<ul>
<li>
<p>The other side of the story: <a href="http://wehow.ehow.com/how_2003079_sell-used-car-california.html">seller howto</a>
. The
person you are buying from should have done all of this, and this is
what you can demand from him. Also, see the links and actual
transaction guidance inside.</p>
</li>
<li>
<p>Used car buying tips at <a href="http://www.carbuyingtips.com/used.htm">http://www.carbuyingtips.com/used.htm</a>
</p>
</li>
</ul>
SCALE5x: Talk summary of the OpenWengo talkhttp://eagain.net/#2007-02-10_scale5x-openwengo2007-02-10T17:05:00-08:002007-02-19T19:42:00-08:00<p>More <a href="http://socallinuxexpo.com/scale5x/">SCALE5x</a>
: <a href="http://socallinuxexpo.com/scale5x/speakers/speakers_neary.php">Dave Neary</a>
is talking about <a href="http://openwengo.org/">OpenWengo</a>
. Note to
self: Wengo = TelCo, WengoPhone = software, OpenWengo = project
developing WengoPhone -- or something. At least it's not just <code>.org</code>
for community and <code>.com</code> for services, even if the names are way too
close to eachother.</p>
<p>Good quote (not his, didn't catch the name):</p>
<pre><code>"People don't want to buy a quarter-inch drill.
They want a quarter-inch hole!"
</code></pre>
<p>He recommended this blog for anyone interested in user interface
design: <a href="http://headrush.typepad.com/">http://headrush.typepad.com/</a>
</p>
<p>Choice quotes:</p>
<pre><code>"Cross Platform (but sound on Linux is a disaster)"
"Surprisingly, for Microsoft, it's not SIP... pure SIP."
(talking about MSN Messenger)
</code></pre>
<p>They intend to implement XMPP-based transport mechanisms. Mentioned
<code>inkboard</code>, an Inkscape extension(?) for whiteboard-style sharing of
drawing over the internet.</p>
<p>They have games over OpenWengo (I guess XMPP?), like chess.</p>
<p>"Oh did I mention sound on Linux is horrible?"</p>
<p>Heh, we're calling audience members during the talk. From France.
And it didn't work ;)</p>
<p>OpenWengo has cross-platform video conferencing. Wow.</p>
SCALE5x: Talk summary of the horribly named Red Hat Xen talkhttp://eagain.net/#2007-02-10_scale5x-redhat-xen2007-02-10T15:45:00-08:002007-02-10T15:49:00-08:00<p>More <a href="http://socallinuxexpo.com/scale5x/">SCALE5x</a>
: <a href="http://socallinuxexpo.com/scale5x/speakers/speakers_folkwilliams.php">Sam Folk-Williams</a>
is doing a talk called <em>Xen
Virtualization in Red Hat Enterprise Linux 5 and Fedora Core 6: An
overview for System Administrators</em> (UNGH!). And demonstrates why I
hate "big" companies like Red Hat: they sent a non-technical, but
well-practised, person to talk about Xen. He sounds convincing, but
ended up explaining Xen domU migration <em>without understanding the
concept of shared storage</em>. Gah.</p>
<p>Also note how talk summary promises live demos of Xen integration
features only Red Hat has, and how the actual talk contained no such
thing. If I didn't have wireless right now I'd be annoyed. Thank you
SCALE5x organizers, the wifi is just great.</p>
SCALE5x: Talk summary of Admin++, what root never told youhttp://eagain.net/#2007-02-10_scale5x-admin2007-02-10T15:19:00-08:002020-01-21T20:49:33-07:00<p>So I'm at <a href="http://socallinuxexpo.com/scale5x/">SCALE5x</a>
, listening to <a href="http://socallinuxexpo.com/scale5x/speakers/speakers_gorodetzky.php">Ron Gorodetzky</a>
talk about what he
learned about sysadmining for <a href="http://digg.com/">Digg</a>
and <a href="http://revision3.com/">Revision3</a>
(who try to be an
"Internet television network"; in effect, they distribute loads of big
files). Most of the tools he mentioned I already knew, but it was nice
to get independent reviews of "hey I think this is good". Here's what
I took home from his talk:</p>
<ul>
<li>
<p>He really thinks highly of the OSCon 2005 talk <code>Livejournal's Backend (A history of scaling)</code>
(<a href="http://danga.com/words/2005_oscon/oscon-2005.pdf">PDF</a>
).</p>
</li>
<li>
<p>He liked <a href="http://danga.com/memcached/">memcached</a>
and <a href="http://www.danga.com/mogilefs/">MogileFS</a>
.</p>
</li>
<li>
<p>Between the lines I understood Revision3 has outsourced their big
bandwidth use -- the CDNs he mentioned by name were <a href="http://www.cachefly.com/">Cachefly</a>
(the
color scheme hurts even my eyes and real designers think I'm
colorblind), <a href="http://www.bitgravity.com/">BitGravity</a>
(caution hideous flash site) and of course
<a href="http://www.akamai.com/">Akamai</a>
.</p>
</li>
<li>
<p>He spoke about outsourcing data center operations, using things like
Amazon <a href="http://www.amazon.com/ec2">EC2</a>
and <a href="http://www.amazon.com/s3">S3</a>
. I need to come up with a budget and time to
play with EC2.</p>
</li>
<li>
<p>He stressed the importance of setting up KVMs etc properly for the
data center.</p>
</li>
<li>
<p>Set up your infrastructure and plan for scaling before you get popular,
because you will be too busy to do them afterwards. That's nice, I like
building things scalable from scratch.</p>
</li>
<li>
<p>Specific infrastructure management tools:</p>
<ul>
<li>
<p><a href="http://reductivelabs.com/projects/puppet/">Puppet</a>
-- seems pretty much a reimplementation of cgengine</p>
</li>
<li>
<p><a href="http://trac.mcs.anl.gov/projects/bcfg2">Bcfg2</a>
-- smells like academentia to me</p>
</li>
<li>
<p><a href="http://trac.t7a.org/isconf/">ISconf</a>
-- from the <code>Bootstrapping an Infrastructure</code> people, seems to be
based on the idea of a p2p distributed cache that stores pretty
much a version control history of commands ran.</p>
</li>
</ul>
<p>As usual, I haven't yet seen anything that would actually seem to
work in the real world, unless you give up everything you already
have (like package management etc), and do things 100% their way.</p>
<p>His suggestion: as the tools are based on very different worldviews,
look at everything and try to pick the one that matches your
opinions.</p>
</li>
<li>
<p>One thing he wouldn't skimp on: "Don't skimp on RAM."</p>
</li>
<li>
<p>At Revision3, they use long-life server hardware and don't upgrade
the servers, instead they go for a full new deployment.</p>
</li>
</ul>
IMAP over SSH Howtohttp://eagain.net/#2007-02-09_imap-over-ssh2007-02-09T21:11:00-08:002020-01-21T20:49:33-07:00<p>Tired of managing <code>n+1</code> passwords? Hate having an extra network port
open on that server box? Want to have automated replication of email
to your laptop in a Unix command line geek-friendly fashion?</p>
<p>Here's how to make <a href="https://github.com/OfflineIMAP/offlineimap">OfflineIMAP</a>
synchronize mail between local and
remote <a href="http://cr.yp.to/proto/maildir.html">Maildirs</a>
.</p>
<ul>
<li>
<p>on the client:</p>
<ul>
<li>
<p>create an SSH key pair with no passphrase:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> ssh-keygen -t rsa -N <span class="s1">''</span> -f ~/.ssh/imap-preauth-key
</span></span></code></pre></div></li>
</ul>
</li>
<li>
<p>on the server:</p>
<ul>
<li>
<p>install <a href="http://bincimap.org/">Binc IMAP</a>
on the server; no need to have it actually
listen for network connections</p>
</li>
<li>
<p>I store my mail as <code>~/.Mail</code> on the server; create a
<code>~/.bincimap</code> on the server and adjust to fit:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Mailbox {
</span></span><span class="line"><span class="cl"> depot = "IMAPdir",
</span></span><span class="line"><span class="cl"> umask = "0077",
</span></span><span class="line"><span class="cl"> path = ".Mail",
</span></span><span class="line"><span class="cl">}
</span></span></code></pre></div></li>
<li>
<p>create a shell script <code>~/bin/imapd-preauth</code> that'll start the
IMAP daemon in a preauthenticated mode; note that OfflineIMAP
wants a certain style of handshake <code>bincimapd</code> doesn't know how
to do, so we fix that with <code>sed</code>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="cp">#!/bin/sh
</span></span></span><span class="line"><span class="cl"><span class="cp"></span><span class="nb">set</span> -e
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">BINCIMAP_LOGIN</span><span class="o">=</span>PREAUTH+FAKE
</span></span><span class="line"><span class="cl">bincimapd<span class="p">|</span>sed --unbuffered <span class="s1">'1s/^FAKE OK PREAUTH/* PREAUTH/'</span>
</span></span></code></pre></div><p>Make the script executable (duh).</p>
</li>
<li>
<p>authorize the previously generated SSH key to run <em>only</em> the above
script -- add the following to <code>~/.ssh/authorized_keys</code> (split
here for readability, make it all one line; replace THINGS to
fit):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">command="/home/USERNAME/bin/imapd-preauth",no-port-forwarding,
</span></span><span class="line"><span class="cl">no-X11-forwarding,no-agent-forwarding,no-pty SSHPUBLICKEYHERE
</span></span></code></pre></div></li>
</ul>
</li>
<li>
<p>on the client:</p>
<ul>
<li>
<p>tell OfflineIMAP about the preauthenticated IMAP connection:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Account SOMETHING]</span>
</span></span><span class="line"><span class="cl"><span class="na">localrepository</span> <span class="o">=</span> <span class="s">local-SOMETHING</span>
</span></span><span class="line"><span class="cl"><span class="na">remoterepository</span> <span class="o">=</span> <span class="s">remote-SOMETHING</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Repository local-SOMETHING]</span>
</span></span><span class="line"><span class="cl"><span class="na">type</span> <span class="o">=</span> <span class="s">Maildir</span>
</span></span><span class="line"><span class="cl"><span class="na">localfolders</span> <span class="o">=</span> <span class="s">~/data/mail/SOMETHING</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">[Repository remote-SOMETHING]</span>
</span></span><span class="line"><span class="cl"><span class="na">type</span> <span class="o">=</span> <span class="s">IMAP</span>
</span></span><span class="line"><span class="cl"><span class="na">remotehost</span> <span class="o">=</span> <span class="s">HOSTNAME</span>
</span></span><span class="line"><span class="cl"><span class="na">preauthtunnel</span> <span class="o">=</span> <span class="s">env -u SSH_AUTH_SOCK ssh -q -i ~/.ssh/imap-preauth-key %(remotehost)s fake-command</span>
</span></span></code></pre></div></li>
</ul>
</li>
</ul>
<p>That should be it! Have fun.</p>
<p>(And if you just broke it, feel free to give one of the halves to me.)</p>
Web 2.0 Explainedhttp://eagain.net/#2007-02-05_web20-explained2007-02-05T08:42:00-08:002020-01-21T20:49:33-07:00<p>Here's a nice video on what people mean when they say Web 2.0:
<a href="http://youtube.com/watch?v=6gmP4nk0EOE">http://youtube.com/watch?v=6gmP4nk0EOE</a>
</p>
<p>I like the style of it, though it is too fast paced if you
don't already know what it is talking about.</p>
<p>(Found via <a href="http://www.mickipedia.com/?p=711">Mickipedia</a>
)</p>
Six Word Scifihttp://eagain.net/#2006-12-02_sixwordscifi2006-12-02T21:38:00-08:002020-01-21T20:49:33-07:00<p><a href="http://sixwordscifi.com/">Sixwordscifi.com</a>
is so much fun it has to be wrong, somehow.
Here are my favorites so far (yes, one of them is mine):</p>
<blockquote>
<p>Back me up before I die.</p>
<p>— <a href="http://sixwordscifi.com/archives/uncategorized/boogah/16">boogah</a>
</p>
</blockquote>
<blockquote>
<p>Even at light speed, I wait.</p>
<p>— <a href="http://sixwordscifi.com/archives/uncategorized/jefft/33">jefft</a>
</p>
</blockquote>
<blockquote>
<p>All alone in his light cone.</p>
<p>— <a href="http://sixwordscifi.com/archives/uncategorized/tv/40">tv</a>
</p>
</blockquote>
The Phone Killer Phonehttp://eagain.net/#2006-12-02_phone-killer2006-12-02T15:46:00-08:002020-01-21T20:49:33-07:00<p>I now know what I want from my next phone. And it'll totally blow the
whole phone concept out of the water.</p>
<p>Start with a mostly-open hardware platform like <a href="http://www.linuxdevices.com/news/NS2986976174.html">Neo1973</a>
, add Linux
(<a href="http://www.openmoko.com/">OpenMoko</a>
) on top. And no need to cram in a clumsy qwerty keypad,
just carry a <a href="http://www.frogpad.com/information/bluefroginfo.asp">one-hand keyboard</a>
when you care about it. Less clumsy
when you don't want more than a phone, <em>full</em> SSH sweetness when you
want it. The phone itself is purely touch-screen, and the keyboard can
actually get respectable WPM with real keypress feedback. And the big
part is, because the phone is actually <em>Open</em>, plugging all this in
is not a big problem! That's just great!</p>
I need a baghttp://eagain.net/#2006-11-17_a-new-bag2006-11-17T11:30:00-08:002006-11-26T22:14:00-08:00<p>I have a shoulder strap-style bag, manufactured and used by the German
army, bought as surplus and dyed black. My 12" thinkpad fits perfectly
inside of it. But the strap is sewn in place, and seems to fail every
few years -- and now's the time.</p>
<p>The bag is really good, and I am going to get it fixed, but that
doesn't mean I can't look at alternatives. So, I need something that
fits a 12" thinkpad, isn't too big, preferably comes in black, and is
otherwise non-attention grabbing and doesn't look like it'd contain a
laptop. According to Lenovo, my laptop is about 268x211x20mm.</p>
<p>My options:</p>
<ul>
<li>small messenger bag: <a href="http://www.chromebags.com/minimetro.php">1</a>
, <a href="http://www.amazon.com/Manhattan-Portage-West-Side-Laptop/dp/B000BT0ERE/sr=1-5/qid=1163791646/ref=sr_1_5/103-0331664-0463044?ie=UTF8&s=apparel">2</a>
</li>
<li>hard-shell attache case (not really my style, but a 12" one in
black might rock it -- though I'm not going to buy a 15" case with
internal padding to make a 12" laptop not bounce around, I want a
smaller case too; it'd still be plenty big for the obligatory dead
tree notes and docs; for pics, see <a href="http://www.the-gadgeteer.com/review/matias_laptop_armor_cases_review">3</a>
, <a href="http://www.ebags.com/samsonite_business_cases/4_aluminum_attache_computer_case/product_detail/index.cfm?modelid=15017">4</a>
</li>
<li><a href="http://www.consumating.com/blog/2006/06/the_manliest_purse_ever.html">the manliest purse ever</a>
from <a href="http://maxpedition.com/">Maxpedition</a>
: I'd be willing to
carry that (in black of course;) if it fit a 12".. And I don't
think it will.</li>
<li>more classical military style: <a href="http://www.amazon.com/rtc8612-M-51-Engineers-Field-Olive/dp/B000B5NGT6/sr=1-8/qid=1163725696/ref=sr_1_8/102-0025330-9204136?ie=UTF8&s=apparel">M-51 Engineers Field Bag</a>
,
<a href="http://www.amazon.com/Urban-Explorer-Black-Canvas-Shoulder/dp/B0006FMIIG/ref=pd_sim_a_4/102-0025330-9204136">Urban Explorer Black Canvas Shoulder</a>
, <a href="http://www.amazon.com/Military-Style-Canvas-Messenger-Case/dp/B000H29BVK/ref=pd_sbs_a_2/102-0025330-9204136">etc</a>
..</li>
<li>something from <a href="http://booqbags.com/">http://booqbags.com/</a>
but the style doesn't really
smack me in the face with want</li>
<li>something from <a href="http://www.timbuk2.com/tb2/retail/catalog.htm?categoryId=1">Timbuk2</a>
, but they really don't seem my style</li>
</ul>
<p>None of those really work for me. I guess considering the bags I
already own, I might go for something I <em>don't</em> have. Mmm, a smallish
hard-shell attache case in black, with a good enough shoulder strap
that I can sling it over my head. That might just do it. Now where do
I get one?</p>
A Revver command line video upload toolhttp://eagain.net/#2006-11-13_revver-upload-video2006-11-13T20:27:00-08:002008-07-20T13:33:00+03:00<p><strong>Update</strong>: It seems the script had gone missing at some point.
<a href="/blog/2006/11/13/revver-upload-video">It's
back</a>
.</p>
<p>As you may or may not have noticed, I do a bunch of stuff for Revver.
I ended up writing a sort of a tutorial to the Revver API, and as I
like to collect all kinds of code samples here, I thought I should
crossblog it here. The <a href="http://web.archive.org/web/20061118134747/http://developer.revver.com/?p=16">original</a>
is on the <a href="http://web.archive.org/web/20061116085509/http://developer.revver.com/">Revver developer blog</a>
.</p>
<p>One day, I was on a slow internet connection and wanted to upload a
few files. I wanted something more batch-oriented than the web-based
upload, and I have a personal bias against most current Java runtimes.
So I decided to use the cool new API and write a
<a href="/blog/2006/11/13/revver-upload-video">video
upload client</a>
, and will walk you through what
it does in this blog entry. Feel free to "just" use the tool, but
hopefully this will also help you in writing your own API clients.</p>
<p>First of all, I wanted to write something that's usable just about
everywhere. I tend to use <a href="http://python.org/">Python</a>
, so that's what the tool is written
in. The Python standard library didn't seem to be able to do HTTP POST
file upload (think web forms) of large files, so I ended up using
<a href="http://curl.haxx.se/">curl</a>
for that. This should work on any Linux/OS X/etc box with Python
and curl installed. All you <a href="http://ubuntu.com/">Ubuntu</a>
/<a href="http://debian.org/">Debian</a>
people just get to say
<code>sudo apt-get install curl</code> and that's it.</p>
<p>So, let's dive right in. The tool is imaginatively named
<code>revver-upload-video</code>. The first bit is the command line
parser. Don't be intimidated by the length, this is pretty much
boilerplate code, the actual API-using bits are really small. The
full file is 155 lines, total.</p>
<p>There are basically three kinds of options: mandatory, optional and
for developer use. Mandatory options are enforced later on, and
developer options are mostly meant for playing with the staging
environment and reusing upload tokens from previous, failed,
uploads.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="ch">#!/usr/bin/python</span>
</span></span><span class="line"><span class="cl"><span class="s2">"""
</span></span></span><span class="line"><span class="cl"><span class="s2">Guerrilla command line video upload tool.
</span></span></span><span class="line"><span class="cl"><span class="s2">"""</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">optparse</span><span class="o">,</span> <span class="nn">getpass</span><span class="o">,</span> <span class="nn">urlparse</span><span class="o">,</span> <span class="nn">urllib</span><span class="o">,</span> <span class="nn">xmlrpclib</span><span class="o">,</span> <span class="nn">subprocess</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">getParser</span><span class="p">():</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">optparse</span><span class="o">.</span><span class="n">OptionParser</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">usage</span><span class="o">=</span><span class="s1">'%prog --title=TEXT --age-rating=NUM [OPTIONS] FILE..'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">description</span><span class="o">=</span><span class="s1">'Upload videos to Revver'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">set_defaults</span><span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">api_url</span><span class="o">=</span><span class="s1">'https://api.revver.com/xml/1.0'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">upload_url</span><span class="o">=</span><span class="s1">'http://httpupload.revver.com/'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">login</span><span class="o">=</span><span class="n">getpass</span><span class="o">.</span><span class="n">getuser</span><span class="p">(),</span>
</span></span><span class="line"><span class="cl"> <span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--login'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'login name to use (default: </span><span class="si">%s</span><span class="s1">)'</span> <span class="o">%</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">defaults</span><span class="p">[</span><span class="s1">'login'</span><span class="p">])</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--passphrase-file'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'read passphrase from (prompt if not given)'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'FILE'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--age-rating'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'MPAA age rating (mandatory)'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="s1">'int'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--title'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'title for the video (mandatory)'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'TEXT'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--tag'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'tags (mandatory, repeat for more tags)'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'KEYWORD'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span><span class="o">=</span><span class="s1">'append'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--author'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'author of the video'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'FULLNAME'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--url'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'website for extra info'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--credits'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'extra credits'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'TEXT'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--description'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'a brief description'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'TEXT'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--api-url'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'API URL to contact (developers only)'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--upload-url'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'Upload URL to send the file to (developers only)'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_option</span><span class="p">(</span><span class="s1">'--upload-token'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">'use preallocated token (developers only)'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">metavar</span><span class="o">=</span><span class="s1">'HEX'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span><span class="o">=</span><span class="s1">'append'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">parser</span>
</span></span></code></pre></div><p>If you've used <code>optparse</code> before, there's not much interesting
there. It just instantiates a parser object and returns it, for the
main function to use. Nothing there touches the Revver API yet.</p>
<p>Next up, we have some utility functions. <code>getPassphrase</code> will read a
passphrase form the file given to <code>--passphrase-file=</code>, or prompt
the user for one. <code>getAPI</code> instantiates an XML-RPC client object
with the login and passphrase, and caches it in <code>options</code> so if you
call <code>getAPI</code> more than once, you're still only prompted for the
passphrase at most once. Nothing in <code>revver-upload-video</code> uses that,
but these are meant to be reusable functions.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">getPassphrase</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">filename</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">f</span> <span class="o">=</span> <span class="n">file</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">passphrase</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">passphrase</span> <span class="o">=</span> <span class="n">getpass</span><span class="o">.</span><span class="n">getpass</span><span class="p">(</span><span class="s1">'Passphrase for video upload: '</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">passphrase</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">getAPI</span><span class="p">(</span><span class="n">options</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">api</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">options</span><span class="p">,</span> <span class="s1">'api'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">api</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">passphrase</span> <span class="o">=</span> <span class="n">getPassphrase</span><span class="p">(</span><span class="n">filename</span><span class="o">=</span><span class="n">options</span><span class="o">.</span><span class="n">passphrase_file</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">scheme</span><span class="p">,</span> <span class="n">netloc</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">query</span><span class="p">,</span> <span class="n">fragment</span><span class="p">)</span> <span class="o">=</span> \
</span></span><span class="line"><span class="cl"> <span class="n">urlparse</span><span class="o">.</span><span class="n">urlsplit</span><span class="p">(</span><span class="n">options</span><span class="o">.</span><span class="n">api_url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">allow_fragments</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">query</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">([(</span><span class="s1">'login'</span><span class="p">,</span> <span class="n">options</span><span class="o">.</span><span class="n">login</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">'passwd'</span><span class="p">,</span> <span class="n">passphrase</span><span class="p">)])</span>
</span></span><span class="line"><span class="cl"> <span class="n">url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="o">.</span><span class="n">urlunsplit</span><span class="p">((</span><span class="n">scheme</span><span class="p">,</span> <span class="n">netloc</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">query</span><span class="p">,</span> <span class="n">fragment</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"> <span class="n">api</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">Server</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">api</span> <span class="o">=</span> <span class="n">api</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">api</span>
</span></span></code></pre></div><p>All right, now we're getting to the actual meat. <code>getToken</code> calls
the API method <code>video.getUploadTokens</code> to allocate an <code>upload token</code>, that lets you upload a file to the Revver archive.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">getToken</span><span class="p">(</span><span class="n">api</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">url</span><span class="p">,</span> <span class="n">tokens</span> <span class="o">=</span> <span class="n">api</span><span class="o">.</span><span class="n">video</span><span class="o">.</span><span class="n">getUploadTokens</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">tokens</span><span class="p">)</span><span class="o">==</span><span class="mi">1</span>
</span></span><span class="line"><span class="cl"> <span class="n">token</span> <span class="o">=</span> <span class="n">tokens</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">url</span><span class="p">,</span> <span class="n">token</span>
</span></span></code></pre></div><p><code>createMedia</code> creates a new video in the archive from your uploaded
file by calling <code>video.create</code> in the API. It also adds metadata
like your website URL to the video. Finally, it returns the media id
of the newly-created video.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">createMedia</span><span class="p">(</span><span class="n">options</span><span class="p">,</span> <span class="n">token</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">data</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">credits</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">data</span><span class="p">[</span><span class="s1">'credits'</span><span class="p">]</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">credits</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">url</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">data</span><span class="p">[</span><span class="s1">'url'</span><span class="p">]</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">url</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">description</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">data</span><span class="p">[</span><span class="s1">'description'</span><span class="p">]</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">description</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">author</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">data</span><span class="p">[</span><span class="s1">'author'</span><span class="p">]</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">author</span>
</span></span><span class="line"><span class="cl"> <span class="n">api</span> <span class="o">=</span> <span class="n">getAPI</span><span class="p">(</span><span class="n">options</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">media_id</span> <span class="o">=</span> <span class="n">api</span><span class="o">.</span><span class="n">video</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">token</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">title</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">tag</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">age_rating</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">data</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">media_id</span>
</span></span></code></pre></div><p>Finally, we have the <code>main</code> function, and the bits that call it when
you run the tool. Here, we actually parse the command line arguments,
enforce the presence of the mandatory options, and bail out unless you
gave it actual files to upload.</p>
<p>For each file given on the command line, we either use one of the
tokens given to us with <code>--upload-token=</code>, or get one from the API
with <code>getToken</code>. Then we join the upload URL and the token to get
the place to upload the file to, and run <code>curl</code> as a subprocess to
do the actual upload. Checking that <code>curl</code> worked takes 8 lines, and
then we use <code>createMedia</code> to actually create the video. And that's
it!</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">progname</span><span class="p">,</span> <span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">getParser</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">options</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">login</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'You must pass --login=LOGIN'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">title</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'You must pass --title=TEXT'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="ow">not</span> <span class="n">options</span><span class="o">.</span><span class="n">tag</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'You must pass --tag=KEYWORD'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">age_rating</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'You must pass --age-rating=NUM'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="ow">not</span> <span class="n">args</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Pass files to upload on command line'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">filename</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">upload_token</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">token</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">upload_token</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">url</span> <span class="o">=</span> <span class="n">options</span><span class="o">.</span><span class="n">upload_url</span>
</span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">api</span> <span class="o">=</span> <span class="n">getAPI</span><span class="p">(</span><span class="n">options</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">url</span><span class="p">,</span> <span class="n">token</span> <span class="o">=</span> <span class="n">getToken</span><span class="p">(</span><span class="n">api</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: allocated token </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">progname</span><span class="p">,</span> <span class="n">token</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">upload_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="o">.</span><span class="n">urljoin</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">token</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">retcode</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">([</span><span class="s1">'curl'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">'-F'</span><span class="p">,</span> <span class="s1">'file=@</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">filename</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="s1">'--'</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">upload_url</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">])</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">retcode</span> <span class="o"><</span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span> <span class="o">>></span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">,</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: upload aborted by signal </span><span class="si">%d</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">progname</span><span class="p">,</span> <span class="o">-</span><span class="n">retcode</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">elif</span> <span class="n">retcode</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span> <span class="o">>></span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">,</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: upload failed with code </span><span class="si">%d</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span>
</span></span><span class="line"><span class="cl"> <span class="n">progname</span><span class="p">,</span> <span class="n">retcode</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: used token </span><span class="si">%s</span><span class="s1"> for </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">progname</span><span class="p">,</span> <span class="n">token</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="n">media_id</span> <span class="o">=</span> <span class="n">createMedia</span><span class="p">(</span><span class="n">options</span><span class="p">,</span> <span class="n">token</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="nb">print</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">: created media </span><span class="si">%r</span><span class="s1"> from </span><span class="si">%r</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">progname</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">media_id</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="n">filename</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">sys</span>
</span></span><span class="line"><span class="cl"> <span class="n">main</span><span class="p">(</span><span class="n">progname</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span>
</span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
</span></span></code></pre></div><p>At this point, we have all we need to do the same thing as the
web-based upload, or the Java upload client. And you can do something
similar, by yourself. This file is copyright Revver, Inc, but licensed
under the <a href="http://www.opensource.org/licenses/mit-license.php">MIT license</a>
-- that means you can use it as a base for
writing your software, without any real restrictions. Download the
whole thing here:
<a href="/blog/2006/11/13/revver-upload-video">revver-upload-video</a>
.</p>
New domain namehttp://eagain.net/#2006-09-30_domain-fun2006-09-30T20:03:00+03:002006-11-13T20:17:00-08:00<p>I had a bit of fun thinking up of puntastic DNS domain names. I ended
up registering <a href="http://eagain.net/">eagain.net</a>
, the old
<code>tv.debian.net</code> name will soon start redirecting there. Need to set up
email too.. Vanity domains are soo much fun.</p>
<p>For the rare non-nerd reading this, <code>EAGAIN</code> is the error code you
get when you are doing asynchronous programming with non-blocking
sockets and would block next. Err, let's just say "I write async
code and it's a neat insider joke", ok?</p>
<p>For the nerds out there, here's a bunch of wild ideas I had while
figuring out what domain name to register. Many of them are invalid
(too short), and most of the good ones are already taken, but in
case you need some inspiration:</p>
<ul>
<li>asyn.ch</li>
<li>dot.at</li>
<li>ex.plo.de / im.plo.de</li>
<li>plea.se</li>
<li>fal.se</li>
<li>belong.us</li>
<li>chi.hu (as in Chihuahua, our dogs..)</li>
<li>celci.us</li>
<li>blo.gr</li>
<li>co.de</li>
<li>co.ff.ee</li>
<li>oh.no</li>
<li>wh.ee</li>
<li>wh.at</li>
<li>sh.it</li>
<li>stre.am (taken)</li>
<li>up.stre.am</li>
<li>down.stre.am</li>
<li>upstre.am</li>
<li>em.ploy.ee</li>
<li>s.tre.am</li>
<li>3x2.net (as in triple-double-w)</li>
<li>tribledouble.net</li>
<li>2by4.net (as in clue-by-four)</li>
<li>be.am (la.ser.be.am, scotty.up.be.am ;-)</li>
<li>dre.am</li>
<li>progr.am</li>
<li>pu.sh</li>
<li>pu.bli.sh</li>
<li>form.at</li>
<li>anon.ymo.us</li>
<li>pla.net</li>
<li>mag.net</li>
<li>carwa.sh</li>
<li>blo.at</li>
<li>cave.at</li>
<li>repe.at</li>
<li>pho.to</li>
<li>oct.et</li>
<li>pron.to</li>
<li>plu.to</li>
<li>lot.to</li>
<li>dit.to</li>
<li>adju.st</li>
<li>ang.st</li>
<li>arti.st</li>
<li>ava.st (type like a pirate)</li>
<li>broadca.st</li>
<li>inner.net</li>
<li>comm.it</li>
<li>mis.info</li>
<li>foc.us</li>
<li>fung.us</li>
<li>geni.us</li>
<li>gur.us</li>
<li>octop.us</li>
<li>styl.us</li>
<li>stat.us</li>
<li>tor.us</li>
<li>line.br</li>
<li>deco.de</li>
<li>while.do</li>
<li>un.do</li>
<li>weir.do</li>
<li>if.then.fi</li>
<li>cook.ie</li>
<li>newb.ie</li>
<li>zomb.ie</li>
<li>disk.io</li>
<li>comm.it</li>
<li>subm.it</li>
<li>bat.ch</li>
<li>bran.ch</li>
<li>epo.ch</li>
<li>fet.ch</li>
<li>zil.ch</li>
<li>gra.ph</li>
<li>boo.st</li>
<li>fibona.cc</li>
<li>from.to</li>
<li>justa.com</li>
</ul>
<p>And for the Finns:</p>
<ul>
<li>mi.au (cat lovers)</li>
<li>vai.nu (dog lovers)</li>
<li>kir.nu</li>
<li>masen.nu</li>
<li>tie.dos.to (taken, by non-Finn)</li>
<li>hit.to</li>
</ul>
<p>I'm not going to say anything about the Cook Island's subdomain for
commercial entities.</p>
In case your Xen domU's have networking troublehttp://eagain.net/#2006-05-21_xen-tcp-hangs2006-05-21T23:25:00+03:002020-01-21T20:49:33-07:00<p>If your <code>domU</code>s have networking trouble with TCP, or some other
protocol that ends up needing fragmentation such as large ICMP pings,
you need to read this.</p>
<p>If it seems TCP handshakes complete, but no data is transferred --
especially, no actual data gets sent out from the domU -- you're
likely hitting a bug in how Xen interacts with TCP segmentation
offload.</p>
<p>The bug seems to depend on the actual network interface card the
traffic is going out from. I hear tg3 is one of the cards that
triggers it, and I'm seeing it on my home box with 8139too's.</p>
<p>The fix is pretty simple, but hard to figure unless you know what to
look for: inside the domU, run</p>
<pre><code>ethtool -K eth0 tx off
</code></pre>
<p>for each interface affected.</p>
<p>See
<a href="http://wiki.xensource.com/xenwiki/XenFaq#head-4ce9767df34fe1c9cf4f85f7e07cb10110eae9b7">http://wiki.xensource.com/xenwiki/XenFaq#head-4ce9767df34fe1c9cf4f85f7e07cb10110eae9b7</a>
for the very small amount of extra information that is out there.</p>
iBook--, Thinkpad++http://eagain.net/#2006-05-21_thinkpad-rocks2006-05-21T22:41:00+03:002006-05-22T07:36:00+03:00<p>Life sucks and then your computer breaks. As soon as a new X release
is out, and it seems
<a href="/blog/2006/04/22/dualhead-ibook.html">dual head on iBook is a possibility</a>
, the darn
thing decides to fry its logic board. Again. Thankfully, Apple may
make the repair for free, if the symptoms match the <a href="http://www.apple.com/support/ibook/faq/">manufacturing
problem</a>
. Again.</p>
<p>Well, the good news is that after 3 years of using the iBook, I got a
new laptop. A Lenovo Thinkpad x60s, weighing just 1.3kg. It's so light
I always think I forgot to put it in the backpack.</p>
<p>Ubuntu Dapper (flight 7) seems to work pretty well on the
x60s. Trouble spots so far:</p>
<ul>
<li>
<p>install CD corrupted display during X autoconfig:
screen was in text mode, mostly black, with two or three
character-size grey rectangles -- hitting enter blindly
let it continue and reboot the machine</p>
</li>
<li>
<p>suspend and hibernation fail on resume</p>
</li>
<li>
<p>wlan hanged once, and didn't recover until I rebooted into Windows
-- unngh. Look at this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">ipw3945: Error sending SCAN_ABORT_CMD: time out after 500ms.
</span></span><span class="line"><span class="cl">ipw3945: Radio Frequency Kill Switch is On:
</span></span><span class="line"><span class="cl">Kill switch must be turned off for wireless networking to work.
</span></span><span class="line"><span class="cl">ipw3945: Error sending ADD_STA: time out after 500ms.
</span></span><span class="line"><span class="cl">ipw3945: Error sending RATE_SCALE: time out after 500ms.
</span></span></code></pre></div><p>After that, any attempt to use the interface ended with:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">ADDRCONF(NETDEV_UP): eth1: link is not ready
</span></span></code></pre></div></li>
<li>
<p>hotplugging the UltraBay docking station does not seem to work in
Linux</p>
</li>
</ul>
<p>I especially love the dual headedness, after fighting with the ATI
driver in the iBook.</p>
<p>Now I need to see about hooking the fingerprint reader up to PAM.</p>
My iBook has two headshttp://eagain.net/#2006-04-22_dualhead-ibook2006-04-23T18:50:00+03:002006-05-02T22:04:00+03:00<p>Finally, after two years of
<a href="/articles/ibook-cloned-monitor/">hacks</a>
, my iBook 2.2 knows how to
multihead! And no silly clone mode only, totally different image and
external output at 1600x1200 at 85Hz. This is <em>nice</em>! Thank you
<a href="http://x.org/">X.org</a>
people for version 7, thank you <a href="http://wiki.debian.org/XStrikeForce">X Strike Force</a>
!</p>
<p><strong>Update:</strong> well, now suspending fails and booting the machine results
in a black screen in over half of the tries. Bah.</p>
`render_pattern`: Repeat patterns easily in Nevow templateshttp://eagain.net/#2005-12-21_nevow_render_pattern2005-12-21T19:18:00+02:002005-12-27T11:48:00+02:00<p>After <code>render_fragment</code>, <code>dialtone</code> mentioned <code>render_pattern</code>,
that would get one or many patterns from the page and put them in the
current tag. Well, that's easy to write:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">render_pattern</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"""
</span></span></span><span class="line"><span class="cl"><span class="s2"> Find and render a pattern.
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> Example:
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> <span nevow:pattern="foo">
</span></span></span><span class="line"><span class="cl"><span class="s2"> I'm very repetititive.
</span></span></span><span class="line"><span class="cl"><span class="s2"> </span>
</span></span></span><span class="line"><span class="cl"><span class="s2"> <ul>
</span></span></span><span class="line"><span class="cl"><span class="s2"> <li nevow:render="pattern foo">
</span></span></span><span class="line"><span class="cl"><span class="s2"> this text will get removed when rendering
</span></span></span><span class="line"><span class="cl"><span class="s2"> </li>
</span></span></span><span class="line"><span class="cl"><span class="s2"> <li nevow:render="pattern foo"/>
</span></span></span><span class="line"><span class="cl"><span class="s2"> </ul>
</span></span></span><span class="line"><span class="cl"><span class="s2"> """</span>
</span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">doc</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">docFactory</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">patterns</span> <span class="o">=</span> <span class="n">inevow</span><span class="o">.</span><span class="n">IQ</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span><span class="o">.</span><span class="n">allPatterns</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">clear</span><span class="p">()[</span><span class="n">patterns</span><span class="p">]</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">f</span>
</span></span></code></pre></div><p><em>Updated</em> to adapt doc to <code>inevow.IQ</code> before calling
<code>allPatterns</code>.</p>
`render_fragment`: Reusable fragment embedding in Nevow templateshttp://eagain.net/#2005-12-21_nevow_render_fragment2005-12-21T18:40:00+02:002005-12-21T22:45:00+02:00<p>This <a href="http://nevow.org/">Nevow</a>
renderer came up on #twisted.web. Thanks to <code>rwall</code> and
<code>dialtone</code> for input.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">render_fragment</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"""
</span></span></span><span class="line"><span class="cl"><span class="s2"> Find and render a fragment, with optional docFactory.
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> Find a fragment factory from self via attributes named
</span></span></span><span class="line"><span class="cl"><span class="s2"> fragment_* and replace content of current tag with said
</span></span></span><span class="line"><span class="cl"><span class="s2"> fragment.
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> If pattern docFactory is found under this tag, pass it as
</span></span></span><span class="line"><span class="cl"><span class="s2"> docFactory to the fragment factory.
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> Example:
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> class MyFrag(rend.Fragment):
</span></span></span><span class="line"><span class="cl"><span class="s2"> ...
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> class MyPage(rend.Page):
</span></span></span><span class="line"><span class="cl"><span class="s2"> fragment_foo = MyFrag
</span></span></span><span class="line"><span class="cl"><span class="s2"> ...
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> and give MyPage a template with
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> <!-- no docFactory -->
</span></span></span><span class="line"><span class="cl"><span class="s2"> <div nevow:render="fragment foo">
</span></span></span><span class="line"><span class="cl"><span class="s2"> this text will get removed when rendering
</span></span></span><span class="line"><span class="cl"><span class="s2"> </div>
</span></span></span><span class="line"><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="cl"><span class="s2"> <!-- with docFactory -->
</span></span></span><span class="line"><span class="cl"><span class="s2"> <div nevow:render="fragment foo">
</span></span></span><span class="line"><span class="cl"><span class="s2"> this text will get removed when rendering
</span></span></span><span class="line"><span class="cl"><span class="s2"> <span nevow:pattern="docFactory">
</span></span></span><span class="line"><span class="cl"><span class="s2"> but this whole tag will be passed as docFactory to MyFrag.
</span></span></span><span class="line"><span class="cl"><span class="s2"> </span>
</span></span></span><span class="line"><span class="cl"><span class="s2"> </div>
</span></span></span><span class="line"><span class="cl"><span class="s2"> """</span>
</span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">callable</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'fragment_</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">name</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">callable</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">callable</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">ctx</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"The fragment named '</span><span class="si">%s</span><span class="s2">' was not found in </span><span class="si">%r</span><span class="s2">."</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="p">)]</span>
</span></span><span class="line"><span class="cl"> <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl"> <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">docFactory</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">onePattern</span><span class="p">(</span><span class="s1">'docFactory'</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">except</span> <span class="n">stan</span><span class="o">.</span><span class="n">NodeNotFound</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="k">pass</span>
</span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">kwargs</span><span class="p">[</span><span class="s1">'docFactory'</span><span class="p">]</span> <span class="o">=</span> <span class="n">loaders</span><span class="o">.</span><span class="n">stan</span><span class="p">(</span><span class="n">docFactory</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">clear</span><span class="p">()[</span><span class="n">callable</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)]</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">f</span>
</span></span></code></pre></div>`render_if`: Conditional Parts in Nevow Templateshttp://eagain.net/#2005-12-17_nevow_render_if2005-12-17T16:59:00+02:002006-03-22T22:52:00+02:00<p>This <a href="http://nevow.org/">Nevow</a>
renderer has saved me a lot of time:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">render_if</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">r</span><span class="o">=</span><span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">allPatterns</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="nb">bool</span><span class="p">(</span><span class="n">data</span><span class="p">)))</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">clear</span><span class="p">()[</span><span class="n">r</span><span class="p">]</span>
</span></span></code></pre></div><p>Use it like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-xml" data-lang="xml"><span class="line"><span class="cl"><span class="nt"><nevow:invisible</span> <span class="na">nevow:render=</span><span class="s">"if"</span> <span class="na">nevow:data=</span><span class="s">"items"</span><span class="nt">></span>
</span></span><span class="line"><span class="cl"> <span class="nt"><ul</span> <span class="na">nevow:pattern=</span><span class="s">"True"</span>
</span></span><span class="line"><span class="cl"> <span class="na">nevow:render=</span><span class="s">"sequence"</span><span class="nt">></span>
</span></span><span class="line"><span class="cl"> <span class="nt"><li</span> <span class="na">nevow:pattern=</span><span class="s">"header"</span><span class="nt">></span>The items are a-coming!<span class="nt"></li></span>
</span></span><span class="line"><span class="cl"> <span class="nt"><li</span> <span class="na">nevow:pattern=</span><span class="s">"item"</span><span class="nt">></span>(the items will be here)<span class="nt"></li></span>
</span></span><span class="line"><span class="cl"> <span class="nt"></ul></span>
</span></span><span class="line"><span class="cl"><span class="nt"></nevow:invisible></span>
</span></span></code></pre></div><p>And now, if the list returned by <code>data_items</code> is empty, there
will be no <code><ul></code> tag at all in the output.</p>
<p>I just realized non-boolean tests may be wanted -- for example, test
if a string matches a regexp. You could do that by mangling the data
before <code>render_if</code>, but that's not nice, because then you don't have
access to the original data inside <code>nevow:pattern="True"</code>. So,
instead let's parametrize the test:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">render_ifparam</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">tester</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'tester_</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">name</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">tester</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">callable</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">context</span><span class="p">,</span> <span class="n">data</span><span class="p">:</span> <span class="n">context</span><span class="o">.</span><span class="n">tag</span><span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"The tester named '</span><span class="si">%s</span><span class="s2">' was not found in </span><span class="si">%r</span><span class="s2">."</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="p">)]</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">callable</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">r</span><span class="o">=</span><span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">allPatterns</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="nb">bool</span><span class="p">(</span><span class="n">tester</span><span class="p">(</span><span class="n">data</span><span class="p">))))</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">ctx</span><span class="o">.</span><span class="n">tag</span><span class="o">.</span><span class="n">clear</span><span class="p">()[</span><span class="n">r</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">f</span>
</span></span></code></pre></div><p>Note how we still cast the return value of the tester to boolean. You
could avoid that and call the renderer <code>render_switch</code>. Adding
support for Deferred tests would be quite easy, too. The only ugly
part is I don't know of any way to make the same renderer work nicely
for <code>nevow:render="if"</code> and <code>nevow:render="ifparam foo"</code>.</p>
<p>[Updated to add <code>return f</code>, also renamed second <code>render_if</code> to
<code>render_ifparam</code> to clarify things a bit. Thanks <code>k3mper</code>.]</p>
My review of Twisted Network Programming Essentialshttp://eagain.net/#2005-11-29_snakeball2005-11-29T02:12:00+02:002005-11-30T17:23:00+02:00<p>I've just finished
<a href="/articles/review-snakeball/">my review</a>
of the book <a href="http://fettig.net/tnpe/">Twisted Network
Programming Essentials</a>
by Abe Fettig. Now
<a href="/articles/review-snakeball/">go read it</a>
.</p>
<p>You may also be interested in an
<a href="/articles/review-pynet/">earlier review</a>
I wrote about
<a href="http://www.complete.org/publications/pynet/">Foundations of Python Network Programming</a>
.</p>
turku-dev: Kehittäjätapaaminen Turussahttp://eagain.net/#2005-11-22_turku-dev2005-11-22T15:48:00+02:002005-11-25T15:33:00+02:00<p>This entry is about a local software developer gathering,
and written in Finnish. My apologies if it is complete
gibberish to you, but atleast you can stare at the pretty
ä dots.</p>
<h2 id="mikä">Mikä?</h2>
<p>Vapaamuotoinen tapaaminen ohjelmistoja työkseen ja/tai
harrastuksekseen tekeville, tai muuten aiheesta kiinnostuneille.</p>
<p>Tutustutaan ihmisiin, puhutaan mukavia, syödään ruokaa. Jos haluat
kertoa hienosta uudesta softasta, jota olet juuri tekemässä, löydät
varmaan jonkun samanmielisen. Jos tarvitset apua hankalaan ongelmaan,
joku varmaan on joskus tehnyt jotain samankaltaista. Eikä ihmisten
tunteminen ainakaan haittaa urakehitystäkään.</p>
<p>Sillä ei ole väliä onko työkalusi C, Perl, Java, Python, Ruby vai PHP;
tai Linux, BSD, OS X vai jopa Windows. Vapaat/avoimet ohjelmistot ovat
monelle meistä tärkeitä, joten niiltä et kokonaan pysty välttymään,
mutta tarkoitus on vain saada samanhenkisiä ihmisiä kokoon.</p>
<h2 id="missä">Missä?</h2>
<p>Turun keskustassa oleva ravintola <a href="http://www.ravintolaharald.com/index2.html">Harald</a>
, katso <a href="http://opaskartta.turku.fi/cgi-bin/GifMap.dll?Theme=-%20Opaskartta&West=69522.7&South=104645.0&East=70422.8&North=105445.1&Height=400&Width=450&Command=DisplayLink&Language=fin&Info=105045.0,69966.7,Harald&Page">kartta</a>
.</p>
<p>Meitä kiinnostaa eniten Turun seudun toiminta, mutta ajatuksia "road
showsta" on heitetty ilmaan, eli jatkossa kehittäjätapaaminen saattaa
olla sinunkin lähikuppilassasi.</p>
<h2 id="koska">Koska?</h2>
<p>Nyt lauantaina, 26.11. n. klo 12:30 alkaen. Niin pitkään
kun intoa riittää.</p>
<p>Seuraava kerta varmaan joskus tammikuussa, ja siitä sitten
eteenpäin vaikka kuukauden tai parin välein.</p>
<h2 id="kuka">Kuka?</h2>
<p>Tällä tapahtumalla ei ole virallista järjestäjää, eikä se liity
minkään yhdistyksen tjms. toimintaan. Minä aloin asiasta tutuille
puhumaan, Tero Kuusela on tehnyt lähes kaiken valmistelutyön.</p>
<p>Tällä hetkellä aiheesta kiinnostuneiden ihmisten taustoja ja
kiinnostuksia: VSTKY, Linux-Aktivaattori, Debian, Python, Linux
kernel, Google Summer of Code, jne..</p>
<h2 id="tahtoo">Tahtoo!</h2>
<p>Liity postituslistalle. Listan osoite on <a href="mailto:turku-dev@lists.inoi.fi">turku-dev@lists.inoi.fi</a>
ja
liittyminen tapahtuu lähettämällä viesti osoitteeseen
<a href="mailto:turku-dev-subscribe@lists.inoi.fi">turku-dev-subscribe@lists.inoi.fi</a>
ja vastaamalla vahvistus-pyyntöön.</p>
<p>Ihmismäärän arvioimiseksi pyydän, että ilmoitat tulostasi etukäteen
osoitteeseen Tero Kuusela <a href="mailto:tero@teroajk.net">tero@teroajk.net</a>
.</p>
The MochiKit screencast is very nicehttp://eagain.net/#2005-11-21_mochikit2005-11-21T16:47:00+02:002020-01-21T20:49:33-07:00<p>"It's simply a more convenient syntax."</p>
<p>"MochiKit is full of more convenient syntax."</p>
<p>The <a href="http://mochikit.com/">MochiKit</a>
<a href="http://mochikit.com/screencasts/MochiKit_Intro-1.html">screencast</a>
is great. I think screencasts are a great
way to introduce people to new software.</p>
I registered at Technorati.comhttp://eagain.net/#2005-11-21_technorati2005-11-21T01:37:00+02:002005-11-21T01:56:00+02:00<p>I just registered at <a href="http://technorati.com/">Technorati</a>
, and it requires me to post an
entry claiming my <a href="http://technorati.com/claim/iezctc84pk">Technorati Profile</a>
.</p>
<p>..and when I registered an alternate URL for it, they required
me to do it all over again:
<a href="http://technorati.com/claim/ykh5rdp8gx">Technorati Profile</a>
.</p>
New website templatehttp://eagain.net/#2005-11-20_new-website2005-11-20T17:34:00+02:002020-01-21T20:49:33-07:00<p>Just finished a new website layout. I'm reasonably pleased with it.</p>
Python is confusinghttp://eagain.net/#2005-11-02_python-is-confusing2005-11-02T14:40:00+02:002020-01-21T20:49:33-07:00<pre tabindex="0"><code class="language-pycon" data-lang="pycon">>>> def simple(): yield 'a'
...
>>> ', '.join(simple())
'a'
>>> def horrible():
... if ' ' not in False: yield 'a'
...
>>> ', '.join(horrible())
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sequence expected, generator found
>>>
</code></pre><p>But it <em>does</em> accept generators!</p>
<p>(Yes, I know what triggers it to say that. It's still horribly
misleading.)</p>
Using *nevow.guard* the smart wayhttp://eagain.net/#2005-09-16_getActionURL2005-09-16T22:59:00+02:002005-09-17T00:06:57+00:00<pre><code><ronwalf> ok, I give up... How do I get the AVATAR_LOGIN stuck
between the SessionWrapped resource ul and the current
resource url
</code></pre>
<p>Well, we aim to please.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">getActionURL</span><span class="p">(</span><span class="n">ctx</span><span class="p">):</span>
</span></span><span class="line"><span class="cl"> <span class="n">request</span> <span class="o">=</span> <span class="n">inevow</span><span class="o">.</span><span class="n">IRequest</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">current</span> <span class="o">=</span> <span class="n">url</span><span class="o">.</span><span class="n">URL</span><span class="o">.</span><span class="n">fromRequest</span><span class="p">(</span><span class="n">request</span><span class="p">)</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">root</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">getRootURL</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">root</span> <span class="o">=</span> <span class="n">url</span><span class="o">.</span><span class="n">URL</span><span class="o">.</span><span class="n">fromString</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">assert</span> <span class="n">root</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl"> <span class="n">root</span> <span class="o">=</span> <span class="n">root</span><span class="o">.</span><span class="n">pathList</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">me</span> <span class="o">=</span> <span class="n">current</span><span class="o">.</span><span class="n">pathList</span><span class="p">(</span><span class="n">copy</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">diff</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">me</span><span class="p">)</span> <span class="o">-</span> <span class="nb">len</span><span class="p">(</span><span class="n">root</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">assert</span> <span class="n">diff</span> <span class="o">>=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span> <span class="o">=</span> <span class="n">current</span>
</span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">diff</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span> <span class="o">=</span> <span class="n">action</span><span class="o">.</span><span class="n">curdir</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="n">diff</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="cl"> <span class="n">diff</span> <span class="o">-=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span> <span class="o">=</span> <span class="n">action</span><span class="o">.</span><span class="n">parent</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span> <span class="o">=</span> <span class="n">action</span><span class="o">.</span><span class="n">child</span><span class="p">(</span><span class="n">guard</span><span class="o">.</span><span class="n">LOGIN_AVATAR</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">me</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">root</span><span class="p">):]:</span>
</span></span><span class="line"><span class="cl"> <span class="n">action</span> <span class="o">=</span> <span class="n">action</span><span class="o">.</span><span class="n">child</span><span class="p">(</span><span class="n">element</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">action</span>
</span></span></code></pre></div><p>Comment from ronwalf (on IRC) on 2005-09-17T00:06:57:</p>
<pre><code><ronwalf> Better. after root = root.pathList()
<ronwalf> if root == ['']: root = []
</code></pre>
Turuxi gathering looks a bit too boring :(http://eagain.net/#2005-09-16_turuxi2005-09-16T15:23:18+02:002020-01-21T20:49:33-07:00<p>The <a href="http://turuxi.org/">Turuxi</a>
<a href="http://twiki.linux-aktivaattori.org/view/Turuxi/TuruxiTapaaminen16092005">gathering</a>
today looks like a bit too boring for me,
personally. I think I'll go xycle instead.</p>