I just put up a bunch of slides from talks I've presented lately:
By now, you have probably used a touchscreen keyboard. We've come a long way from the clumsy "kiosk" computers that brought touchscreen keyboards to mainstream, a decade or two ago. But a classic keyboard with physical keys is still preferable for the tactile feedback we get from pressing the keys. Until touchscreens can provide that, we'll be using traditional keyboards for a while.
But how do touchscreen keyboards differ from physical keyboards, and what ideas could we copy from them to improve the user experience of traditional keyboards? Well, for one, most touchscreen keyboards these days don't do key repeat -- instead, they'll pop up a menu of alternatives, often the same base letter with with various accents, diacritics and umlaut. And you slide your finger to pick one of these options.
Now, that sounds good. I for one don't know how to type the various variations of the letters on a US keyboard, and as a Finn I actually need "ä" and "ö" sometimes. I can type them in Emacs, but not in my IM application or web browser.
Here's the idea: we don't really use autorepeat on the A-Z characters. Instead, make a long press of a letter key bring up an on-screen menu with variations of the basic letter.
On a touchscreen, choosing from the menu is immediate and fairly intuitive. While using the mouse for that would be straight forward -- and probably what a first time user will try -- nobody wants to type like that. We're stuck with one finger holding down the original key. We need to make a selection from up to about 9 alternatives. Here's two easy ideas (assuming the initial highlighted alternative is the base letter):
For example, say I want to type "ä". My options are (assuming US keyboard layout):
I think I'd use that more than auto-repeat. What about you?
I've been looking at the world of deployment tools lately. Outside of Puppet and Chef (and ignoring the old beards Bcfg2 & Cfengine), what other things are there?
Fabric lets you write Python functions to describe "tasks" to be run. The Python functions are run on a client machine -- for example, the sysadmin's laptop -- and each task can be directed to operate on hosts or roles (groups of hosts), over SSH. The functions can run remote commands with run("echo hello, world") and sudo("chmod u=rw,go=r /etc/passwd"). Fabric is a very useful piece of the puzzle, but doing more complex operations one shell command at a time gets frustrating. I keep wanting something that can run whole chunks of Python easily, on the target machine. Fabric also does nothing to solve the problems of e.g. multiple admins running a deploy command at the same time.
Kokki is closer to Chef in it's style (and literally, "Chef" in Finnish). It's a framework for writing cookbooks, with actions like File("/etc/greeting", content="hello, world") in them. Then a configuration for a machine can invoke certain recipes. Kokki seems to be aimed fully at running things locally; that is, if you're deploying things, you'd run Kokki on the target machine. Kokki is still in fairly early development, it's website and source code don't match each other at all, and many of the cookbooks no longer work with the current version. It also still inherits a bit too many non-Pythonic elements from Chef, for my tastes. Still, to this Pythonista, it looks very promising, and I will be exploring it further.
Poni ("Pony" in Finnish -- I want one too!) is another Python project that takes a very different tack. It is built on a command-line tool that lets you define your infrastructure through hierarchical collections of key-value settings; that is, you describe the whole multi-machine service with database servers, load balancers, app servers and all. You can use inheritance much like other deploy tools use cookbooks. The command-line tool seems to be meant to be used for everything; the stored data is not really meant to be edited directly. While I appreciate the polish of the command-line tool, the editor-hostility comes off a bit odd. Especially so when the getting started guide has me "uploading" template files and Python source code to Poni's internal configuration storage. Am I really supposed to have two copies of these files?
Once you have your infrastructure defined, Poni provides you two main methods to actually make changes: you can create files based on templates (that have a strong mechanism for referring to any values from the configuration, including things like sharing the database connection information between the DB server and the client config), or you can run custom functions ("control commands"). The Python functions run locally, but Poni provides a remote execution framework very similar to the one in Fabric, though at least for now it is significantly more verbose. And, to my disappointment, doesn't really allow running full Python functions remotely either.
Somewhat confusing is the difference between the "create a file" and the "run control command" functionality. It is not quite clear how the whole is intended to orchestrate the full deployment, and the examples are both lacking and misleading. For example, right now the Puppet deployment example requires you to run a command to create some files, watch it fail, run another command to install software, then run the first command again to create the rest of the files. (Kind of weird to deploy Puppet with Poni in the first place..)
There is one thing about Poni that I am already starting to dislike. Currently, every identifier you need to refer to on the command line is given as a regexp, and commands act on all matches. This leads to high risk for operator errors: for example, the documentation itself uses $find("webshop/frontend") as an example; yet that would also match webshop/frontendforsomethingelse. I do hope the author changes his mind about regexps everywhere.
Much like Kokki, Poni is very early on in its development; it's command-line tools and things like variable referencing are top notch, but the picture is very much not complete yet. But this one is definitely a project to watch.
---
For posterity: I filed a bunch of issues about the things I bumped into:
EuroPython 2008 videos are available.
My God, it's Full of Files -- Pythonic filesystem abstractions
See the slides.
Hosted at blip.tv, download MP4.
Version Control for Du^H^HDevelopers
See the slides.
Hosted at blip.tv, download MP4.
Or see all talks from EuroPython 2008.
EuroPython 2008 was fun. I presented two talks (My God, it's Full of Files -- Pythonic filesystem abstractions and Version Control for Du^H^HDevelopers) and one lightning talk (RST+S5 for your slides), participated in a bunch of open space sessions, listened to about 13 talks, took a bunch of pictures, but most importantly had interesting hallway conversations with interesting people.
As usual, PyPy was heavily represented, and seems to be making nice progress toward being the nice and featureful default Python implementation of the future. I especially liked the restricted execution features and the LLVM backend. The zc.buildout talk made me decide I will try to replace (part of?) one custom deploy mechanism with zc.buildout -- most likely I'll end up rewriting most of the current things as zc.buildout recipes, but hopefully some of the pre-existing recipes will be useful, and hopefully I can then later reuse the recipies I create for this setup.
Personally, I think my talks went ok. I understand videos will be available later, as soon as transcoding etc are finished. I'm anxious to see them myself, as I'm still finetuning my public speaking skills. I'm learning, though -- this year I had no trouble staying within my time slot, even when I was adjusting verbosity on the fly.
For some reason, I felt underprepared for the filesystem API talk, but ultimately people liked the idea of a consistent Pythonic filesystem API enough that we had an open space session on it, and people were enthusiastic about a sprint to prototype the API. Which is what we ended up doing, too -- I'll blog separately about the results of that.
My decentralized version control talk seemed to me to go over more smoothly; I guess that's just because I've been thinking about version control and project management a lot lately, so it was easy to talk about the topic in a relaxed way. On the other hand, it wasn't as much a call to action, and it really was overly generic, so I didn't get as strong audience participation there. We did have an interesting conversation about branch management strategies and such, though. I consciously tried to keep the talk on a generic level, as I felt a pure git talk would have alienated some listeners, but I did end up feeling restricted by that. There was some interest on a Teach me git -style session, but what we ended up doing was just talking one on one about getting started with git, during the sprints. Sorry if I missed any one of you -- grab me on #git to continue, or find me in future conferences ;)
I was requested to organize an open space session for Twisted Q&A, and that is exactly what we did. We went through a bunch of things related to asynchronous programming concepts, Deferreds, working with blocking code and libraries, database interfaces, debugging and unit testing.
I was also pulled in to another Twisted open space session, that was mostly about what greenlets are and how to use them. I tried to explain the differences between classical Deferreds, deferredGenerator/inlineCallbacks, and greenlets, to the best of my understanding. As a summary, with greenlets any function you call can co-operatively yield execution (I mean yield in the scheduling meaning, giving away your turn to run, not in the Python generator meaning -- interestingly inlineCallbacks etc actually make those be the same thing... my kernel instincts make me want to say "sleep"). Yielding in any subroutine means anything you do may end up mutating your objects -- which is the root evil behind threading we wanted to get away from. All the other mechanisms keep the top-level function in explicit control of yielding. Around that time, most people left for lunch, but about three of us stayed and talked about debugging Deferreds and network packet processing with twisted.pair and friends.
One of the interesting hallway conversations was about what happens when upstream web hosting listed on PyPI is failing. It seems PyPI already does some sort of mirroring, but even that might not be enough. Many companies seem to be bundling eggs of their dependencies in their installation package, which sounds like a good setup for commercial click-to-install deployment. But it would still be good to see a CPAN-style mirror network for PyPI, and at least some people seemed even motivated to donating servers and bandwidth. Personally, I'm mostly spoiled by the combination of Debian/Ubuntu and decentralized version control, and my level of paranoia is too high to automatically install unverified software from the internet anyway. My primary motivation in the conversation was to point out that PyPI already has some sort of mirroring/upload setup, and that you'd really want to specify exact versions and SHA-1 hashes of your dependencies. Optionally, you could delegate the known good hash storage to PyPI (assuming you trusted PyPI not to attack you), but that would require a full Debian-style signature chain from a trusted key, or you'd be owned by anyone capable of MITM attacks, DNS forgery, or cracking a PyPI mirror.
I published slides for my first Europython talk. Note the slides might not easy to understand without the actual talk -- the video streaming at Ustream will work if the network infrastructure can take it, and a downloadable video should be available later.
My God, it's Full of Files
Pythonic filesystem abstractions: An overview of different filesystem(-like) APIs in Python and attempts for unifying them.
There's a lot of different filesystem(-like) APIs in Python. I intend to provide an overview of existing projects, their status and capabilities, and hopefully inspire you to work on improving things.
So Google has their MapReduce, and the people behind CouchDB are throwing around their ideas. I spent some time thinking about incremental mapreduce around July, and it's time I type out that page full of scribbles.
First of all: I think the ideas thrown out by Damien above aren't really mapreduce, As Google Intended. The real power of mapreduce is in its inherent combination of parallelism and chainability, output of one mapreduce is input to another, each processing step can run massively in parallel with each other, etc. The proposed design is like a one-iteration retarded cousin of mapreduce.
With that bashing now done (sorry), here's what I was thinking:
The way I imagined building an incremental mapreduce mechanism, without storing the intermediate data and just recomputing chunks that are out-of-date (which would be lame), is to add one extra concept into the system: call it "demap". It will basically create "negative entries" for the old data. This is basically what Damien did by providing both the old and new data map calls, all the time, just said differently, and I think my way might make the average call a lot simpler. And I don't see any reason why my version wouldn't be parallelizable, chainable, and generally yummy.
For years now, my primary machine has been a laptop. I've been avoiding running Xen [1] because the hypervisor it injects between hardware and Linux hasn't been very friendly for power saving. Modern laptops are too often hot enough without any extra help.
Well, I'm happy to say that as of now, KVM is definitely stable enough to replace my use of Xen for test setups. I expect that shortly it will become what I recommend for server use, too. And because it lets Linux be Linux, instead of doing any oddities with the hardware, all the powersaving etc goodness still works perfectly.
So far, I've stumbled on two things:
1. The version of KVM in Ubuntu, even gutsy, is just too old. With a bit of fiddling with the patches, KVM v46 built a perfectly working deb.
2. The isolinux graphical bootup, used in Ubuntu, crashes KVM (and with v28, it crashes the host machine -- beware!). See the bug report. I got around that by using mini.iso, but you could always fall back to debootstrap; that was all we ever really had with Xen.
And while I have the soapbox: I have a dream. It's KVM running with SDL/VNC graphics in a window that's resizeable all the way to the virtual machine, with XRandR. Please make it happen!
Patches
So, the edited version of the patches:
And just remove from-debian-qemu/62_linux_boot_nasm.patch, it seems to have made it upstream.
| [1] | What's up with http://www.xensource.com/, by the way? That website is just a pile of horrible non-informative enterprise speak. Utterly useless, and that tends to alienate the techie community pretty fast. At least it's alienating me. |
A brief update of things I've been working on:
Snakepit
Snakepit is a port of (part of) HiveDB to Python and SQLAlchemy. It will help you write database-backed applications that need to scale further than one database server, or even a master-slave setup, can take you. It's MIT licensed, that is, pretty much as free-for-all as it can be. And it's still work in progress, so don't be too harsh yet ;)
Read the README, browse the gitweb, or grab the source with git clone git://eagain.net/snakepit.git
gitosis
gitosis aims to make hosting git repos easier and safer. It manages multiple repositories under one user account, using SSH keys to identify users. End users do not need shell accounts on the server, they will talk to one shared account that will not let them run arbitrary commands. gitosis is licensed under the GPL.
First real release will come as soon as I have to time to go through a couple of really minor nits. It's been self-hosting for a long time now.
For more, read the README, browse the gitweb, or grab the source with git clone git://eagain.net/gitosis.git
I wrote a brief introduction to git for people who are not scared by words like Directed Acyclic Graph. Read it here: Git for Computer Scientists
Warning
This solution is obsolete. Use gitosis instead!
Gitosis is basically git-shell-enforce-directory's big brother, and an actual software project. Use it.
Updated to drop --use-separate-remote from git clone, it's the default.
Updated to add --read-only to git-shell-enforce-directory.
I've run repeatedly into cases where I want to provide services to people without really trusting them. I do not want to give them shell access. I don't want to even create separate unix user accounts for them at all. But I do want to make sure the service they use is safe against e.g. password sniffing.
Instead of trying to run the version control system over HTTPS (like Subversion's mod_dav_svn that will only work with Apache, which I don't run), I want to run things through SSH. SSH is the de facto unix tool for securing communications between machines.
Now, I said I don't want to create a unix user account for every developer using the version control system. With SSH, this means using a shared account, usually named by the service it provides: svn, git, etc. To identify different users of that account, do not give the account a password, but use SSH keys instead. To avoid giving people full shell access, use a command="..." when adding their public key to ~/.ssh/authorized_keys.
For Subversion, I submitted an enhancement to add --tunnel-user, to make sure the commit gets identified as the right user, and then used command="..." with the with arguments, like this (all on one line):
command="/srv/example.com/repo/svn/svnserve -t
--root /srv/example.com/repo/svn/view/examplegroup
--tunnel-user jdoe" ssh-rsa ... jdoe@example.com
Where the view directory is a bunch of symlinks to the actual repositories, allowing me to do group-based access control.
With git, the author of the changeset is recorded way before the SSH connection is opened. Without building some sort of access control in git hooks on the server, every developer can pretty much ruin the repository by overwriting branches with bogus commits. What they will not have is access outside of the repository, or a way to actually remove the old commits from the disk (unless you run git prune on the server). The distributed nature of git makes this reasonably easy to detect, and pretty much trivial to recover from. For any real trust in the code, you should look at signed tags anyway. The included wrapper allows you to have read-only users, but provides no detailed access control against developers with write access; they just won't be able to escape to the rest of the filesystem.
So, with that introduction out of the way, let's get to configuring:
Install git on the server:
sudo apt-get install git-core git-doc
Create the directory structure store the repositories and related files:
sudo install -d -m0755 \
/srv/example.com/repo/git \
/srv/example.com/repo/git/.ssh \
/srv/example.com/repo/git/repos \
/srv/example.com/repo/git/view
Create the shared user account for this service:
sudo adduser \
--system \
--home /srv/example.com/repo/git \
--no-create-home \
--shell /bin/sh \
--gecos 'git version control' \
--group \
--disabled-password \
git
Set up a script that makes sure only relevant git commands can be run via SSH, and to limit the visible section of the filesystem to things you actually want to give access to; put this file in /usr/local/bin/git-shell-enforce-directory (download) and chmod a+x it
#!/usr/bin/python
# Copyright (c) 2007 Tommi Virtanen <tv@eagain.net>
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
# Enforce git-shell to only serve repositories
# in the given directory. The client should refer
# to them without any directory prefix.
# Repository names are forced to match ALLOW.
import sys, os, optparse, re
def die(msg):
print >>sys.stderr, '%s: %s' % (sys.argv[0], msg)
sys.exit(1)
def getParser():
parser = optparse.OptionParser(
usage='%prog [OPTIONS] DIR',
description='Allow restricted git operations under DIR',
)
parser.add_option('--read-only',
help='disable write operations',
action='store_true',
default=False,
)
return parser
ALLOW_RE = re.compile("^(?P<command>git-(?:receive|upload)-pack) '[a-zA-Z][a-zA-Z0-9@._-]*(/[a-zA-Z][a-zA-Z0-9@._-]*)*'$")
COMMANDS_READONLY = [
'git-upload-pack',
]
COMMANDS_WRITE = [
'git-receive-pack',
]
def main(args):
os.umask(0022)
parser = getParser()
(options, args) = parser.parse_args()
try:
(path,) = args
except ValueError:
parser.error('Missing argument DIR.')
os.chdir(path)
cmd = os.environ.get('SSH_ORIGINAL_COMMAND', None)
if cmd is None:
die("Need SSH_ORIGINAL_COMMAND in environment.")
if '\n' in cmd:
die("Command may not contain newlines.")
match = ALLOW_RE.match(cmd)
if match is None:
die("Command to run looks dangerous")
allowed = list(COMMANDS_READONLY)
if not options.read_only:
allowed.extend(COMMANDS_WRITE)
if match.group('command') not in allowed:
die("Command not allowed")
os.execve('/usr/bin/git-shell', ['git-shell', '-c', cmd], {})
die("Cannot execute git-shell.")
if __name__ == '__main__':
main(args=sys.argv[1:])
Create your first repository:
cd /srv/example.com/repo/git/repos
sudo install -d -o git -g git -m0700 myproject.git
sudo -H -u git env GIT_DIR=myproject.git git init
(with git older than v1.5, use init-db instead of init)
Set up an access control group and give it access to that repository:
cd /srv/example.com/repo/git/view
sudo install -d -m0755 mygroup
cd mygroup
sudo ln -s ../../repos/myproject.git myproject.git
You can also use subdirectories of view/mygroup to organize the repositories hierarchically.
Note, one SSH public key will belong to exactly one group, but if necessary you can create a separate group for each account for absolute control.
Note, access to repository implies write access to repository, at least for now. You could make
Get an SSH public key from a developer and authorize them to access the group:
cd /srv/example.com/repo/git
sudo vi .ssh/authorized_keys
How the developer generates their key is out of scope here.
Add a line like this, with the public key in it (all on one line, broken up in the middle of word to make sure there is no misunderstanding about when to use a space and when not to):
command="/usr/local/bin/git-shell-enforce-directory /srv/exampl
e.com/repo/git/view/mygroup",no-port-forwarding,no-X11-forwar
ding,no-agent-forwarding,no-pty ssh-rsa ... jdoe@example.com
Or to allow only read-only access, add --read-only as an option.
You can now push things to the repository with:
git push git@myserver.example.com:myproject.git mybranch:refs/heads/master
Note that before the first push, your server-side repository will not contain even an initial commit, and can't really be cloned.
Now the developer can clone the repository:
git clone git@myserver:myproject.git
or to avoid some behavior of older git that I consider confusing (needs git v1.5 or newer):
git clone -o myserver git@myserver:myproject.git
They will probably want to set up ssh-agent to avoid typing the passphrase all the time.
And you're done! Good luck with your adventures with git, and welcome to the 21st century and to distributed version control systems.
Tomorrow, if everything goes well, I will buy a used car from an individual in California. Here's a checklist of things for that, to help others in similar situations. Some things may have already been omitted because they weren't relevant for me, so you may want to independently browse the websites I'm using as sources. I'm skipping everything related to finances and haggling. I'm not covering cases where the car isn't in good condition. Also, buying from dealerships is different. Good luck.
(Updated to mention REG 262.)
Find an ad on craigslist or whatever. I liked http://losangeles.listpic.com/car/
Call the seller. Don't just email, start evaluating the seller. Things to ask: http://www.carbuyingtips.com/questions.html (though that list is horribly extensive; I'd rather just pick the three best-looking candidates and do that thing on the spot; you'll need to doublecheck anyway to make sure the seller wasn't lying).
Ask for:
Check the CARFAX report for the car. Go for the $24.95 30-day option, you should always look at more than one car.
Check the smog check history (fails are a sign of trouble). In California, in most of the cases, the seller is required to provide a certificate of check newer than 90 days old -- it seems the information flow to website is slow enough that this latest check does not show up, or something.
Check that the car hasn't suffered storm damage. (Annoying website demands cookies for a simple form. Suck.)
Make an appointment and go see the car. In good sunlight, you want to see the car.
Take the car to a mechanic of your choosing for evaluation. Alternatively, choose to trust on service from brand name vendor.
Check the price against Kelley Blue Book etc.
Now you're in business. Only bureaucracy and doublechecking left.
If you know what you're willing to pay, go to your bank and get a cashiers check. Or two alternatives, if that works out. Otherwise, you will need to go back to the bank after haggling, and without a deposit the seller may have sold the car to someone else. Or something. Tuff.
(Some people say haggling at dealerships is easier if you show up with a cashiers check just a bit short from what they're asking.)
Don't pay with cash, there's even less chance of getting it back than cancelling a check.
If you happen to be a seller reading this, do the actual sale in a bank to be sure you aren't being cheated.
Download and print PDF forms from DMV.
In California, sellers are required to provide a smog certificate. Make sure you get one. Stuff on smog checks: 1, 2, 3. Frankly, I'm still a bit confused myself when I will need to do a smog check.
Check that the registration is current and that the car wasn't repurchased under the California "Lemon law". (Err, how? As I understand it should say so in the Certificate of Title paper)
The seller should find the "pink slip" aka Certificate of Title
Things to ask for before leaving
Now you're ready to leave with your new car! Check that you have
Seller has 5 days to submit Notice of Transfer and Release of Liability to DMV. Do it online.
Buyer has 10 days days to report ownership change to DMV and max 30 days to pay the fees. Read more: checklist, things to send to DMV, more info.
Taxes?
General resources:
More SCALE5x: Dave Neary is talking about OpenWengo. Note to self: Wengo = TelCo, WengoPhone = software, OpenWengo = project developing WengoPhone -- or something. At least it's not just .org for community and .com for services, even if the names are way too close to eachother.
Good quote (not his, didn't catch the name):
"People don't want to buy a quarter-inch drill. They want a quarter-inch hole!"
He recommended this blog for anyone interested in user interface design: http://headrush.typepad.com/
Choice quotes:
"Cross Platform (but sound on Linux is a disaster)"
"Surprisingly, for Microsoft, it's not SIP... pure SIP." (talking about MSN Messenger)
They intend to implement XMPP-based transport mechanisms. Mentioned inkboard, an Inkscape extension(?) for whiteboard-style sharing of drawing over the internet.
They have games over OpenWengo (I guess XMPP?), like chess.
"Oh did I mention sound on Linux is horrible?"
Heh, we're calling audience members during the talk. From France. And it didn't work ;)
OpenWengo has cross-platform video conferencing. Wow.
More SCALE5x: Sam Folk-Williams is doing a talk called Xen Virtualization in Red Hat Enterprise Linux 5 and Fedora Core 6: An overview for System Administrators (UNGH!). And demonstrates why I hate "big" companies like Red Hat: they sent a non-technical, but well-practised, person to talk about Xen. He sounds convincing, but ended up explaining Xen domU migration without understanding the concept of shared storage. Gah.
Also note how talk summary promises live demos of Xen integration features only Red Hat has, and how the actual talk contained no such thing. If I didn't have wireless right now I'd be annoyed. Thank you SCALE5x organizers, the wifi is just great.
So I'm at SCALE5x, listening to Ron Gorodetzky talk about what he learned about sysadmining for Digg and Revision3 (who try to be an "Internet television network"; in effect, they distribute loads of big files). Most of the tools he mentioned I already knew, but it was nice to get independent reviews of "hey I think this is good". Here's what I took home from his talk:
He really thinks highly of the OSCon 2005 talk Livejournal's Backend (A history of scaling) (PDF).
Between the lines I understood Revision3 has outsourced their big bandwidth use -- the CDNs he mentioned by name were Cachefly (the color scheme hurts even my eyes and real designers think I'm colorblind), BitGravity (caution hideous flash site) and of course Akamai.
He spoke about outsourcing data center operations, using things like Amazon EC2 and S3. I need to come up with a budget and time to play with EC2.
He stressed the importance of setting up KVMs etc properly for the data center.
Set up your infrastructure and plan for scaling before you get popular, because you will be too busy to do them afterwards. That's nice, I like building things scalable from scratch.
Specific infrastructure management tools:
As usual, I haven't yet seen anything that would actually seem to work in the real world, unless you give up everything you already have (like package management etc), and do things 100% their way.
His suggestion: as the tools are based on very different worldviews, look at everything and try to pick the one that matches your opinions.
One thing he wouldn't skimp on: "Don't skimp on RAM."
At Revision3, they use long-life server hardware and don't upgrade the servers, instead they go for a full new deployment.
Tired of managing n+1 passwords? Hate having an extra network port open on that server box? Want to have automated replication of email to your laptop in a Unix command line geek-friendly fashion?
Here's how to make OfflineIMAP synchronize mail between local and remote Maildirs.
on the client:
create an SSH key pair with no passphrase:
ssh-keygen -t rsa -N '' -f ~/.ssh/imap-preauth-key
on the server:
install Binc IMAP on the server; no need to have it actually listen for network connections
I store my mail as ~/.Mail on the server; create a ~/.bincimap on the server and adjust to fit:
Mailbox {
depot = "IMAPdir",
umask = "0077",
path = ".Mail",
}
create a shell script ~/bin/imapd-preauth that'll start the IMAP daemon in a preauthenticated mode; note that OfflineIMAP wants a certain style of handshake bincimapd doesn't know how to do, so we fix that with sed:
#!/bin/sh
set -e
export BINCIMAP_LOGIN=PREAUTH+FAKE
bincimapd|sed --unbuffered '1s/^FAKE OK PREAUTH/* PREAUTH/'
Make the script executable (duh).
authorize the previously generated SSH key to run only the above script -- add the following to ~/.ssh/authorized_keys (split here for readability, make it all one line; replace THINGS to fit):
command="/home/USERNAME/bin/imapd-preauth",no-port-forwarding,
no-X11-forwarding,no-agent-forwarding,no-pty SSHPUBLICKEYHERE
on the client:
tell OfflineIMAP about the preauthenticated IMAP connection:
[Account SOMETHING]
localrepository = local-SOMETHING
remoterepository = remote-SOMETHING
[Repository local-SOMETHING]
type = Maildir
localfolders = ~/data/mail/SOMETHING
[Repository remote-SOMETHING]
type = IMAP
remotehost = HOSTNAME
preauthtunnel = env -u SSH_AUTH_SOCK ssh -q -i ~/.ssh/imap-preauth-key %(remotehost)s fake-command
That should be it! Have fun.
(And if you just broke it, feel free to give one of the halves to me.)
Here's a nice video on what people mean when they say Web 2.0: http://youtube.com/watch?v=6gmP4nk0EOE
I like the style of it, though it is too fast paced if you don't already know what it is talking about.
(Found via Mickipedia)
Sixwordscifi.com is so much fun it has to be wrong, somehow. Here are my favorites so far (yes, one of them is mine):
Back me up before I die.boogah
Even at light speed, I wait.jefft
All alone in his light cone.tv
I now know what I want from my next phone. And it'll totally blow the whole phone concept out of the water.
Start with a mostly-open hardware platform like Neo1973, add Linux (OpenMoko) on top. And no need to cram in a clumsy qwerty keypad, just carry a one-hand keyboard when you care about it. Less clumsy when you don't want more than a phone, full SSH sweetness when you want it. The phone itself is purely touch-screen, and the keyboard can actually get respectable WPM with real keypress feedback. And the big part is, because the phone is actually Open, plugging all this in is not a big problem! That's just great!
I have a shoulder strap-style bag, manufactured and used by the German army, bought as surplus and dyed black. My 12" thinkpad fits perfectly inside of it. But the strap is sewn in place, and seems to fail every few years -- and now's the time.
The bag is really good, and I am going to get it fixed, but that doesn't mean I can't look at alternatives. So, I need something that fits a 12" thinkpad, isn't too big, preferably comes in black, and is otherwise non-attention grabbing and doesn't look like it'd contain a laptop. According to Lenovo, my laptop is about 268x211x20mm.
My options:
- small messenger bag: 1, 2
- hard-shell attache case (not really my style, but a 12" one in black might rock it -- though I'm not going to buy a 15" case with internal padding to make a 12" laptop not bounce around, I want a smaller case too; it'd still be plenty big for the obligatory dead tree notes and docs; for pics, see 3, 4
- the manliest purse ever from Maxpedition: I'd be willing to carry that (in black of course;) if it fit a 12".. And I don't think it will.
- more classical military style: M-51 Engineers Field Bag, Urban Explorer Black Canvas Shoulder, etc..
- something from http://booqbags.com/ but the style doesn't really smack me in the face with want
- something from Timbuk2, but they really don't seem my style
None of those really work for me. I guess considering the bags I already own, I might go for something I don't have. Mmm, a smallish hard-shell attache case in black, with a good enough shoulder strap that I can sling it over my head. That might just do it. Now where do I get one?
Update: It seems the script had gone missing at some point. It's back :download:<revver-upload-video>.
As you may or may not have noticed, I do a bunch of stuff for Revver. I ended up writing a sort of a tutorial to the Revver API, and as I like to collect all kinds of code samples here, I thought I should crossblog it here. The original is on the Revver developer blog.
One day, I was on a slow internet connection and wanted to upload a few files. I wanted something more batch-oriented than the web-based upload, and I have a personal bias against most current Java runtimes. So I decided to use the cool new API and write a video upload client, and will walk you through what it does in this blog entry. Feel free to "just" use the tool, but hopefully this will also help you in writing your own API clients.
First of all, I wanted to write something that's usable just about everywhere. I tend to use Python, so that's what the tool is written in. The Python standard library didn't seem to be able to do HTTP POST file upload (think web forms) of large files, so I ended up using curl for that. This should work on any Linux/OS X/etc box with Python and curl installed. All you Ubuntu/Debian people just get to say sudo apt-get install curl and that's it.
So, let's dive right in. The tool is imaginatively named revver-upload-video. The first bit is the command line parser. Don't be intimidated by the length, this is pretty much boilerplate code, the actual API-using bits are really small. The full file is 155 lines, total.
There are basically three kinds of options: mandatory, optional and for developer use. Mandatory options are enforced later on, and developer options are mostly meant for playing with the staging environment and reusing upload tokens from previous, failed, uploads.
#!/usr/bin/python
"""
Guerrilla command line video upload tool.
"""
import optparse, getpass, urlparse, urllib, xmlrpclib, subprocess
def getParser():
parser = optparse.OptionParser(
usage='%prog --title=TEXT --age-rating=NUM [OPTIONS] FILE..',
description='Upload videos to Revver')
parser.set_defaults(
api_url='https://api.revver.com/xml/1.0',
upload_url='http://httpupload.revver.com/',
login=getpass.getuser(),
)
parser.add_option('--login',
help='login name to use (default: %s)' %
parser.defaults['login'])
parser.add_option('--passphrase-file',
help='read passphrase from (prompt if not given)',
metavar='FILE')
parser.add_option('--age-rating',
help='MPAA age rating (mandatory)',
type='int')
parser.add_option('--title',
help='title for the video (mandatory)',
metavar='TEXT')
parser.add_option('--tag',
help='tags (mandatory, repeat for more tags)',
metavar='KEYWORD',
action='append')
parser.add_option('--author',
help='author of the video',
metavar='FULLNAME')
parser.add_option('--url',
help='website for extra info')
parser.add_option('--credits',
help='extra credits',
metavar='TEXT')
parser.add_option('--description',
help='a brief description',
metavar='TEXT')
parser.add_option('--api-url',
help='API URL to contact (developers only)')
parser.add_option('--upload-url',
help='Upload URL to send the file to (developers only)')
parser.add_option('--upload-token',
help='use preallocated token (developers only)',
metavar='HEX',
action='append')
return parser
If you've used optparse before, there's not much interesting there. It just instantiates a parser object and returns it, for the main function to use. Nothing there touches the Revver API yet.
Next up, we have some utility functions. getPassphrase will read a passphrase form the file given to --passphrase-file=, or prompt the user for one. getAPI instantiates an XML-RPC client object with the login and passphrase, and caches it in options so if you call getAPI more than once, you're still only prompted for the passphrase at most once. Nothing in revver-upload-video uses that, but these are meant to be reusable functions.
def getPassphrase(filename=None):
if filename is not None:
f = file(filename)
passphrase = f.readline().rstrip('\n')
f.close()
else:
passphrase = getpass.getpass('Passphrase for video upload: ')
return passphrase
def getAPI(options):
api = getattr(options, 'api', None)
if api is None:
passphrase = getPassphrase(filename=options.passphrase_file)
(scheme, netloc, path, query, fragment) = \
urlparse.urlsplit(options.api_url,
allow_fragments=False)
query = urllib.urlencode([('login', options.login),
('passwd', passphrase)])
url = urlparse.urlunsplit((scheme, netloc, path, query, fragment))
api = xmlrpclib.Server(url)
options.api = api
return api
All right, now we're getting to the actual meat. getToken calls the API method video.getUploadTokens to allocate an upload token, that lets you upload a file to the Revver archive.
def getToken(api):
url, tokens = api.video.getUploadTokens(1)
assert len(tokens)==1
token = tokens[0]
return url, token
createMedia creates a new video in the archive from your uploaded file by calling video.create in the API. It also adds metadata like your website URL to the video. Finally, it returns the media id of the newly-created video.
def createMedia(options, token):
data = {}
if options.credits is not None:
data['credits'] = options.credits
if options.url is not None:
data['url'] = options.url
if options.description is not None:
data['description'] = options.description
if options.author is not None:
data['author'] = options.author
api = getAPI(options)
media_id = api.video.create(token,
options.title,
options.tag,
options.age_rating,
data)
return media_id
Finally, we have the main function, and the bits that call it when you run the tool. Here, we actually parse the command line arguments, enforce the presence of the mandatory options, and bail out unless you gave it actual files to upload.
For each file given on the command line, we either use one of the tokens given to us with --upload-token=, or get one from the API with getToken. Then we join the upload URL and the token to get the place to upload the file to, and run curl as a subprocess to do the actual upload. Checking that curl worked takes 8 lines, and then we use createMedia to actually create the video. And that's it!
def main(progname, args):
parser = getParser()
(options, args) = parser.parse_args()
if options.login is None:
parser.error('You must pass --login=LOGIN')
if options.title is None:
parser.error('You must pass --title=TEXT')
if not options.tag:
parser.error('You must pass --tag=KEYWORD')
if options.age_rating is None:
parser.error('You must pass --age-rating=NUM')
if not args:
parser.error('Pass files to upload on command line')
for filename in args:
if options.upload_token:
token = options.upload_token.pop(0)
url = options.upload_url
else:
api = getAPI(options)
url, token = getToken(api)
print '%s: allocated token %s' % (progname, token)
upload_url = urlparse.urljoin(url, token)
retcode = subprocess.call(['curl',
'-F', 'file=@%s' % filename,
'--',
upload_url,
])
if retcode < 0:
print >>sys.stderr, '%s: upload aborted by signal %d' % (
progname, -retcode)
sys.exit(1)
elif retcode > 0:
print >>sys.stderr, '%s: upload failed with code %d' % (
progname, retcode)
sys.exit(1)
print '%s: used token %s for %s' % (progname, token, filename)
media_id = createMedia(options, token)
print '%s: created media %r from %r' % (progname,
media_id,
filename)
if __name__ == '__main__':
import os, sys
main(progname=os.path.basename(sys.argv[0]),
args=sys.argv[1:])
At this point, we have all we need to do the same thing as the web-based upload, or the Java upload client. And you can do something similar, by yourself. This file is copyright Revver, Inc, but licensed under the MIT license -- that means you can use it as a base for writing your software, without any real restrictions. Download the whole thing here: revver-upload-video.
I had a bit of fun thinking up of puntastic DNS domain names. I ended up registering eagain.net, the old tv.debian.net name will soon start redirecting there. Need to set up email too.. Vanity domains are soo much fun.
For the rare non-nerd reading this, EAGAIN is the error code you get when you are doing asynchronous programming with non-blocking sockets and would block next. Err, let's just say "I write async code and it's a neat insider joke", ok?
For the nerds out there, here's a bunch of wild ideas I had while figuring out what domain name to register. Many of them are invalid (too short), and most of the good ones are already taken, but in case you need some inspiration:
- asyn.ch
- dot.at
- ex.plo.de / im.plo.de
- plea.se
- fal.se
- belong.us
- chi.hu (as in Chihuahua, our dogs..)
- celci.us
- blo.gr
- co.de
- co.ff.ee
- oh.no
- wh.ee
- wh.at
- sh.it
- stre.am (taken)
- up.stre.am
- down.stre.am
- upstre.am
- em.ploy.ee
- s.tre.am
- 3x2.net (as in triple-double-w)
- tribledouble.net
- 2by4.net (as in clue-by-four)
- be.am (la.ser.be.am, scotty.up.be.am ;-)
- dre.am
- progr.am
- pu.sh
- pu.bli.sh
- form.at
- anon.ymo.us
- pla.net
- mag.net
- carwa.sh
- blo.at
- cave.at
- repe.at
- pho.to
- oct.et
- pron.to
- plu.to
- lot.to
- dit.to
- adju.st
- ang.st
- arti.st
- ava.st (type like a pirate)
- broadca.st
- inner.net
- comm.it
- mis.info
- foc.us
- fung.us
- geni.us
- gur.us
- octop.us
- styl.us
- stat.us
- tor.us
- line.br
- deco.de
- while.do
- un.do
- weir.do
- if.then.fi
- cook.ie
- newb.ie
- zomb.ie
- disk.io
- comm.it
- subm.it
- bat.ch
- bran.ch
- epo.ch
- fet.ch
- zil.ch
- gra.ph
- boo.st
- fibona.cc
- from.to
- justa.com
And for the Finns:
- mi.au (cat lovers)
- vai.nu (dog lovers)
- kir.nu
- masen.nu
- tie.dos.to (taken, by non-Finn)
- hit.to
I'm not going to say anything about the Cook Island's subdomain for commercial entities.
If your domUs have networking trouble with TCP, or some other protocol that ends up needing fragmentation such as large ICMP pings, you need to read this.
If it seems TCP handshakes complete, but no data is transferred -- especially, no actual data gets sent out from the domU -- you're likely hitting a bug in how Xen interacts with TCP segmentation offload.
The bug seems to depend on the actual network interface card the traffic is going out from. I hear tg3 is one of the cards that triggers it, and I'm seeing it on my home box with 8139too's.
The fix is pretty simple, but hard to figure unless you know what to look for: inside the domU, run
ethtool -K eth0 tx off
for each interface affected.
See http://wiki.xensource.com/xenwiki/XenFaq#head-4ce9767df34fe1c9cf4f85f7e07cb10110eae9b7 for the very small amount of extra information that is out there.
Life sucks and then your computer breaks. As soon as a new X release is out, and it seems dual head on iBook is a possibility, the darn thing decides to fry its logic board. Again. Thankfully, Apple may make the repair for free, if the symptoms match the manufacturing problem. Again.
Well, the good news is that after 3 years of using the iBook, I got a new laptop. A Lenovo Thinkpad x60s, weighing just 1.3kg. It's so light I always think I forgot to put it in the backpack.
Ubuntu Dapper (flight 7) seems to work pretty well on the x60s. Trouble spots so far:
install CD corrupted display during X autoconfig: screen was in text mode, mostly black, with two or three character-size grey rectangles -- hitting enter blindly let it continue and reboot the machine
suspend and hibernation fail on resume
wlan hanged once, and didn't recover until I rebooted into Windows -- unngh. Look at this:
ipw3945: Error sending SCAN_ABORT_CMD: time out after 500ms.
ipw3945: Radio Frequency Kill Switch is On:
Kill switch must be turned off for wireless networking to work.
ipw3945: Error sending ADD_STA: time out after 500ms.
ipw3945: Error sending RATE_SCALE: time out after 500ms.
After that, any attempt to use the interface ended with:
ADDRCONF(NETDEV_UP): eth1: link is not ready
hotplugging the UltraBay docking station does not seem to work in Linux
I especially love the dual headedness, after fighting with the ATI driver in the iBook.
Now I need to see about hooking the fingerprint reader up to PAM.
Finally, after two years of hacks, my iBook 2.2 knows how to multihead! And no silly clone mode only, totally different image and external output at 1600x1200 at 85Hz. This is nice! Thank you X.org people for version 7, thank you X Strike Force!
Update: well, now suspending fails and booting the machine results in a black screen in over half of the tries. Bah.
I never thought I'd fall for the "small dogs must wear cute clothes" phenomenon, but I think I'm going to have to buy this.
After render_fragment, dialtone mentioned render_pattern, that would get one or many patterns from the page and put them in the current tag. Well, that's easy to write:
def render_pattern(self, name):
"""
Find and render a pattern.
Example:
<span nevow:pattern="foo">
I'm very repetititive.
</span>
<ul>
<li nevow:render="pattern foo">
this text will get removed when rendering
</li>
<li nevow:render="pattern foo"/>
</ul>
"""
def f(ctx, data):
doc = self.docFactory.load(ctx)
patterns = inevow.IQ(doc).allPatterns(name)
return ctx.tag.clear()[patterns]
return f
Updated to adapt doc to inevow.IQ before calling allPatterns.
This Nevow renderer came up on #twisted.web. Thanks to rwall and dialtone for input.
def render_fragment(self, name):
"""
Find and render a fragment, with optional docFactory.
Find a fragment factory from self via attributes named
fragment_* and replace content of current tag with said
fragment.
If pattern docFactory is found under this tag, pass it as
docFactory to the fragment factory.
Example:
class MyFrag(rend.Fragment):
...
class MyPage(rend.Page):
fragment_foo = MyFrag
...
and give MyPage a template with
<!-- no docFactory -->
<div nevow:render="fragment foo">
this text will get removed when rendering
</div>
<!-- with docFactory -->
<div nevow:render="fragment foo">
this text will get removed when rendering
<span nevow:pattern="docFactory">
but this whole tag will be passed as docFactory to MyFrag.
</span>
</div>
"""
def f(ctx, data):
callable = getattr(self, 'fragment_%s' % name, None)
if callable is None:
callable = lambda ctx, *args: ctx.tag[
"The fragment named '%s' was not found in %r." % (name, self)]
kwargs = {}
try:
docFactory = ctx.tag.onePattern('docFactory')
except stan.NodeNotFound:
pass
else:
kwargs['docFactory'] = loaders.stan(docFactory)
return ctx.tag.clear()[callable(**kwargs)]
return f
This Nevow renderer has saved me a lot of time:
def render_if(self, ctx, data):
r=ctx.tag.allPatterns(str(bool(data)))
return ctx.tag.clear()[r]
Use it like this:
<nevow:invisible nevow:render="if" nevow:data="items">
<ul nevow:pattern="True"
nevow:render="sequence">
<li nevow:pattern="header">The items are a-coming!</li>
<li nevow:pattern="item">(the items will be here)</li>
</ul>
</nevow:invisible>
And now, if the list returned by data_items is empty, there will be no <ul> tag at all in the output.
I just realized non-boolean tests may be wanted -- for example, test if a string matches a regexp. You could do that by mangling the data before render_if, but that's not nice, because then you don't have access to the original data inside nevow:pattern="True". So, instead let's parametrize the test:
def render_ifparam(self, name):
tester = getattr(self, 'tester_%s' % name, None)
if tester is None:
callable = lambda context, data: context.tag[
"The tester named '%s' was not found in %r." % (name, self)]
return callable
def f(ctx, data):
r=ctx.tag.allPatterns(str(bool(tester(data))))
return ctx.tag.clear()[r]
return f
Note how we still cast the return value of the tester to boolean. You could avoid that and call the renderer render_switch. Adding support for Deferred tests would be quite easy, too. The only ugly part is I don't know of any way to make the same renderer work nicely for nevow:render="if" and nevow:render="ifparam foo".
[Updated to add return f, also renamed second render_if to render_ifparam to clarify things a bit. Thanks k3mper.]
I've just finished my review of the book Twisted Network Programming Essentials by Abe Fettig. Now go read it.
You may also be interested in an earlier review I wrote about Foundations of Python Network Programming.
This entry is about a local software developer gathering, and written in Finnish. My apologies if it is complete gibberish to you, but atleast you can stare at the pretty ä dots.
Vapaamuotoinen tapaaminen ohjelmistoja työkseen ja/tai harrastuksekseen tekeville, tai muuten aiheesta kiinnostuneille.
Tutustutaan ihmisiin, puhutaan mukavia, syödään ruokaa. Jos haluat kertoa hienosta uudesta softasta, jota olet juuri tekemässä, löydät varmaan jonkun samanmielisen. Jos tarvitset apua hankalaan ongelmaan, joku varmaan on joskus tehnyt jotain samankaltaista. Eikä ihmisten tunteminen ainakaan haittaa urakehitystäkään.
Sillä ei ole väliä onko työkalusi C, Perl, Java, Python, Ruby vai PHP; tai Linux, BSD, OS X vai jopa Windows. Vapaat/avoimet ohjelmistot ovat monelle meistä tärkeitä, joten niiltä et kokonaan pysty välttymään, mutta tarkoitus on vain saada samanhenkisiä ihmisiä kokoon.
Meitä kiinnostaa eniten Turun seudun toiminta, mutta ajatuksia "road showsta" on heitetty ilmaan, eli jatkossa kehittäjätapaaminen saattaa olla sinunkin lähikuppilassasi.
Nyt lauantaina, 26.11. n. klo 12:30 alkaen. Niin pitkään kun intoa riittää.
Seuraava kerta varmaan joskus tammikuussa, ja siitä sitten eteenpäin vaikka kuukauden tai parin välein.
Tällä tapahtumalla ei ole virallista järjestäjää, eikä se liity minkään yhdistyksen tjms. toimintaan. Minä aloin asiasta tutuille puhumaan, Tero Kuusela on tehnyt lähes kaiken valmistelutyön.
Tällä hetkellä aiheesta kiinnostuneiden ihmisten taustoja ja kiinnostuksia: VSTKY, Linux-Aktivaattori, Debian, Python, Linux kernel, Google Summer of Code, jne..
Liity postituslistalle. Listan osoite on turku-dev@lists.inoi.fi ja liittyminen tapahtuu lähettämällä viesti osoitteeseen turku-dev-subscribe@lists.inoi.fi ja vastaamalla vahvistus-pyyntöön.
Ihmismäärän arvioimiseksi pyydän, että ilmoitat tulostasi etukäteen osoitteeseen Tero Kuusela <tero@teroajk.net>.
"It's simply a more convenient syntax."
"MochiKit is full of more convenient syntax."
The MochiKit screencast is great. I think screencasts are a great way to introduce people to new software.
I just registered at Technorati, and it requires me to post an entry claiming my Technorati Profile.
..and when I registered an alternate URL for it, they required me to do it all over again: Technorati Profile.
Just finished a new website layout. I'm reasonably pleased with it.
>>> def simple(): yield 'a'
...
>>> ', '.join(simple())
'a'
>>> def horrible():
... if ' ' not in False: yield 'a'
...
>>> ', '.join(horrible())
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: sequence expected, generator found
>>>
But it does accept generators!
(Yes, I know what triggers it to say that. It's still horribly misleading.)
<ronwalf> ok, I give up... How do I get the AVATAR_LOGIN stuck
between the SessionWrapped resource ul and the current
resource url
Well, we aim to please.
def getActionURL(ctx):
request = inevow.IRequest(ctx)
current = url.URL.fromRequest(request).clear()
root = request.getRootURL()
root = url.URL.fromString(root)
assert root is not None
root = root.pathList()
me = current.pathList(copy=True)
diff = len(me) - len(root)
assert diff >= 0
action = current
if diff == 1:
action = action.curdir()
else:
while diff > 1:
diff -= 1
action = action.parent()
action = action.child(guard.LOGIN_AVATAR)
for element in me[len(root):]:
action = action.child(element)
return action
Comment from ronwalf (on IRC) on 2005-09-17T00:06:57:
<ronwalf> Better. after root = root.pathList()
<ronwalf> if root == ['']: root = []