Tv's projects

winner go gets it all

21 November 2013

Tommi Virtanen

Gentlementlemen

2

Go forth and create new repos

github.com/tv42 (because 26**2 is not enough for everyone)

Ask for more when it gets interesting.

demand -- Download, build, cache and run a Go app easily
staged -- Run a command with the Git staged files
jog -- Structured logging library
birpc -- Bi-directional RPC library, including JSON-over-WebSocket
topic -- in-process single-topic pub-sub
adhoc-httpd -- Quick & dirty HTTP static file server
adhoc-httpd-upload -- Quick & dirty HTTP file upload server
humanize-bytes -- Command-line utilities to convert "MiB" etc to raw numbers, and back
json-point -- Command-line tool to extract parts of JSON documents, with JSON Pointer syntax
slug -- Create slugs of text or URLs, for use in e.g. URL path segments
base58 -- Encode numeric ids to pretty tokens for e.g. URLs
seed -- Demo of how to easily seed PRNGs with some entropy
twackup -- Back up your tweets
didyouseethis -- Retweet anything with certain keywords
upstart-socket -- Go library to listen on a socket handed in by Upstart (/systemd etc)
maildir-fix -- Fix maildirs after git has pruned empty dirs
msgpack-json -- Command-line utilities to convert between msgpack and json
spindown-idle-disks -- Spin down idle SATA disks. Because hdparm -S 120 just won't work
3

demand -- Download, build, cache and run a Go app easily

github.com/tv42/demand

Put in ~/bin:

#!/usr/bin/env demand
go:
  import: github.com/tv42/humanize-bytes/cmd/bytes2human

Then run

$ chmod a+x ~/bin/bytes2human
$ bytes2human 65536
64KiB
4

demand: usage

Usage of demand:
  demand [OPTS] SPEC_PATH [ARGS..]

Only build, not run command:
  demand -build [OPTS] SPEC_PATH..

Options:
  -build=false: only build, do not run command (can pass multiple spec files)
  -gopath=false: use GOPATH from environment instead of downloading all dependencies
  -upgrade=false: force upgrade even if older version exists

Use as an interpreter:
  #!/usr/bin/env demand
  go:
    import: GO_IMPORT_PATH_HERE
5

demand: it's only 260 lines

github.com/tv42/demand/blob/master/main.go

6

demand: cram tests

github.com/tv42/demand/blob/master/cram/run.t

  $ cat >foo <<EOF
  > go:
  >   import: github.com/tv42/humanize-bytes/cmd/bytes2human
  > EOF
  $ XDG_CONFIG_HOME="$PWD/home" DEMAND_CACHE_DIR="$PWD/cache" demand foo 65536
  64KiB
7

staged -- Run a command with the Git staged files

Ever forget to commit a new file?

Ever commit only half the change?

Usage of /home/tv/bin/staged:
  /home/tv/bin/staged COMMAND..
8

staged: demo 1/2

  $ git init --quiet
  $ cat >demo.go <<'EOF'
  > package demo
  > func foo() int {
  >     return 42
  > }
  > EOF
  $ cat >demo_test.go <<'EOF'
  > package demo
  > 
  > import "testing"
  > 
  > func TestFoo(t *testing.T) {
  >     if g, e := foo(), 42; g != e {
  >         t.Errorf("bad foo: %v != %v", g, e)
  >     }
  > }
  > EOF
9

staged: demo 2/2

  $ git add demo_test.go        # forget to add demo.go
  $ go test                     # ignores the problem
  PASS
  ok * (glob)
  $ staged go test              # detects the problem
  # * (glob)
  ./demo_test.go:6: undefined: foo
  FAIL\t* [build failed] (esc) (glob)
  staged: command failed: exit status 2
  [1]
  $ staged ls                   # no demo.go
  demo_test.go
  $ git add demo.go
  $ staged go test
  PASS
  ok * (glob)
10

staged: it's only 220 lines

github.com/tv42/staged/blob/master/main.go

11

jog -- Structured logging library

github.com/tv42/jog

Often, ease of processing is more important than human eyeball friendliness.

{"Time":"2013-04-23T20:51:17.633016Z","Type":"example.com/webshop#login","Data":{"User":"jdoe","IP":"12.34.56.78"}}
12

jog: demo

type start struct {
    Enthusiasm string
}

type stop struct {
    Style  string
    Points float32
}

// Example output:
//
//     {"Time":"2013-04-23T20:51:17.632909Z","Type":"main#start","Data":{"Enthusiasm":"high"}}
//     {"Time":"2013-04-23T20:51:17.633016Z","Type":"github.com/tv42/jog/example/greetings#greet","Data":{"Message":"Hello, world!","ID":"43235433"}}
//     {"Time":"2013-04-23T20:51:17.633047Z","Type":"main#stop","Data":{"Style":"commendable","Points":9.7}}
func main() {
    log := jog.New(nil)
    log.Event(start{Enthusiasm: "high"})
    greetings.Greet(log)
    log.Event(stop{Style: "commendable", Points: 9.7})
}
13

jog: it's only 110 lines

github.com/tv42/jog/blob/master/jog.go

14

birpc -- Bi-directional RPC library, including JSON-over-WebSocket

github.com/tv42/birpc
chat example

cd ~/go/src/github.com/tv42/birpc/examples/chat
./run

localhost:8000/

15

birpc: messages

{"id":"1","fn":"Arith.Add","args":{"A":1,"B":1}}
{"id":"1","result":{"C":2}}
{"id":"1","error":{"msg":"Math is hard, let's go shopping"}}
type Message struct {
    // 0 or omitted for untagged request (untagged response is illegal).
    ID uint64 `json:"id,string,omitempty"`

    // Name of the function to call. If set, this is a request; if
    // unset, this is a response.
    Func string `json:"fn,omitempty"`

    // Arguments for the RPC call. Only valid for a request.
    Args interface{} `json:"args,omitempty"`

    // Result of the function call. A response will always have
    // either Result or Error set. Only valid for a response.
    Result interface{} `json:"result,omitempty"`

    // Information on how the call failed. Only valid for a
    // response. Must be present if Result is omitted.
    Error *Error `json:"error,omitempty"`
}
16

birpc: wetsock, jsonmsg

wetsock for JSON over WebSocket messages

jsonmsg for (lines of) json over any io.ReadWriteCloser

17

birpc: the RPC part

func (c *Chat) Message(msg *Incoming, _ *nothing, ws *websocket.Conn) error {
    log.Printf("recv from %v:%#v\n", ws.RemoteAddr(), msg)

    c.broadcast.Broadcast <- Outgoing{
        Time:    time.Now(),
        From:    msg.From,
        Message: msg.Message,
    }
    return nil
}
18

birpc: optional argument stuff

func (c *codec) FillArgs(arglist []reflect.Value) error {
    for i := 0; i < len(arglist); i++ {
        switch arglist[i].Interface().(type) {
        case *websocket.Conn:
            arglist[i] = reflect.ValueOf(c.WS)
        }
    }
    return nil
}
19

birpc: oneshotlisten

Used in tests to avoid TCP.

    pipe_client, pipe_server := net.Pipe()
    fakeListener := oneshotlisten.New(pipe_server)
    done := make(chan error)
    go func() {
        defer close(done)
        done <- server.Serve(fakeListener)
    }()
    ws, _, err := websocket.NewClient(
        pipe_client,
    err = <-done
    if err != nil && err != io.EOF {
        t.Fatalf("http server failed: %v", err)
    }
20

birpc: it's 530 lines

$ wc -l birpc.go message.go wetsock/wetsock.go
  387 birpc.go
   54 message.go
   90 wetsock/wetsock.go
  531 total

$ wc -l examples/chat/chat.js examples/chat/main.go
 135 examples/chat/chat.js
 142 examples/chat/main.go
 277 total
21

topic -- in-process single-topic pub-sub

github.com/tv42/topic

Remember this?

func (c *Chat) Message(msg *Incoming, _ *nothing, ws *websocket.Conn) error {
    log.Printf("recv from %v:%#v\n", ws.RemoteAddr(), msg)

    c.broadcast.Broadcast <- Outgoing{
        Time:    time.Now(),
        From:    msg.From,
        Message: msg.Message,
    }
    return nil
}

godoc.org/github.com/tv42/topic (local)

lossy is good, because lossy is realistic

22

adhoc-httpd -- Quick & dirty HTTP static file server

github.com/tv42/adhoc-httpd

mkdir junk
adhoc-httpd -port 8000 junk

50 lines, but oh so convenient

23

adhoc-httpd-upload -- Quick & dirty HTTP file upload server

github.com/tv42/adhoc-httpd-upload

adhoc-httpd-upload -port 8001 junk

This time DIR is mandatory, so you don't mess up $HOME.

111 lines

Time for a dramatic demo

24

humanize-bytes -- Convert "MiB" etc to raw numbers, and back

github.com/tv42/humanize-bytes

$ human2bytes -h
Usage of /home/tv/bin/human2bytes:
  /home/tv/bin/human2bytes [NUMBER..]
  /home/tv/bin/human2bytes <FILE
  -sloppy=false: continue past parse errors
$ human2bytes 100M
100000000
$ human2bytes 100MiB
104857600

$ bytes2human -h
Usage of /home/tv/bin/bytes2human:
  /home/tv/bin/bytes2human [NUMBER..]
  /home/tv/bin/bytes2human <FILE
  -si=false: use base-10 SI units instead of base-2 units like MiB
  -sloppy=false: continue past parse errors
$ bytes2human 992342522
946MiB
$ du -b * | sort -nr | head -3 | bytes2human
639KiB    drama.gif
56KiB    gopher.jpg
5.1KiB    talk.slide

180 lines

25

json-point -- Command-line tool to extract parts of JSON documents

github.com/tv42/json-point

JSON Pointer spec: simple descend, no matching.

Library by Dustin. Keep your JSON as []byte as long as you can.

$ echo '{"foo":{"quux":"thud"}}' | json-point /foo
{"quux":"thud"}
$ echo '{"foo":{"quux":"thud"}}' | json-point /foo/quux
"thud"

$ echo '{"foo":"bar", "quux":"thud"}' | json-point -pretty /foo
bar
26

slug -- Create slugs of text or URLs, for use in e.g. URL path segments

github.com/tv42/slug

func ExampleSlug() {
    fmt.Println(slug.Slug("Rødgrød med fløde"))
    fmt.Println(slug.Slug("BUY NOW!!!11eleven"))
    fmt.Println(slug.Slug("../../etc/passwd"))
    // Output:
    // rødgrød-med-fløde
    // buy-now-11eleven
    // etc-passwd
}
func ExampleURLString() {
    s, err := slug.URLString("https://www.example.com/foo/index.html")
    fmt.Println(s, err)
    // Output:
    // example-com-foo <nil>
}

100 lines

27

base58 -- Encode numeric ids to pretty tokens for e.g. URLs

github.com/tv42/base58

$ base58-encode 2345235235
4zeWJg
$ base58-decode 4zeWJg
2345235235

Good example of lib + multiple mains

lib 70 + cmd 50 = 120 lines

28

seed -- Demo of how to easily seed PRNGs with some entropy

github.com/tv42/seed

package main

import (
    "fmt"

    "github.com/tv42/seed"
)

func main() {
    d6 := seed.Rand().Int31n(6)
    fmt.Println(d6)
}

silly 33 lines

29

twackup -- Back up your tweets

github.com/tv42/twackup

$ cd ~/Private/personal/tweets/
$ twackup tv .
Fetching tweets newer than 373615153020817408
Fetching url https://api.twitter.com/1.1/statuses/user_timeline.json?[...]
Fetching url https://api.twitter.com/1.1/statuses/user_timeline.json?[...]
Saved 17 new tweets
Fetching tweets older than 1417093
Fetching url https://api.twitter.com/1.1/statuses/user_timeline.json?[...]
$ json-point /text <401067312066732032.json
"I have 22 browser tabs open. On my phone. #0thworldproblem #browsertabitis"

340 lines

30

didyouseethis -- Retweet anything with certain keywords

github.com/tv42/didyouseethis

Twitter Streaming API client

Saves & retweets tweets matching keywords

The good kind of stalker

Run this on a protected account to keep track of mentions of interesting things

Because Twitter Search sucks

680 lines

31

didyouseethis: watchdog is cute

    dog := watchdog.New(90 * time.Second)
        select {
        case line, ok := <-lines:
            if !ok {
                err = <-readError
                log.Fatalf("error reading stream: %s\n", err)
            }

            dog.Pet()
        case <-dog.Bark:
            log.Fatalf("stream timeout")
        }
32

upstart-socket -- Go library to listen on a socket handed in by Upstart (/systemd etc)

github.com/tv42/upstart-socket

start on socket PROTO=inet PORT=5000
setuid foo
setgid foo
exec /some/where/over/the/rainbow/food

It gives you a Listener:

l, err := upstart.Listen()

30 lines

33

maildir-fix -- Fix maildirs after git has pruned empty dirs

github.com/tv42/maildir-fix

Because I'm crazy enough to keep my mail in Git

$ ls Spam
cur/  tmp/
$ maildir-fix Spam
$ ls Spam
cur/  new/  tmp/

$ maildir-fix -depot=.

In Git hooks

120 lines

34

msgpack-json -- Command-line utilities to convert between msgpack and json

github.com/tv42/msgpack-json

$ echo '{"foo": 42}'|json2msgpack >temp
$ hexdump -C temp
00000000  81 a3 66 6f 6f cb 40 45  00 00 00 00 00 00        |..foo.@E......|
0000000e
$ msgpack2json <temp
{"foo":42}

60 lines

35

spindown-idle-disks -- Spin down idle SATA disks. Because hdparm -S 120 just won't work

github.com/tv42/spindown-idle-disks

description    "spin down idle disks"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
exec /usr/local/sbin/spindown-idle-disks -idle=30m /dev/sdb /dev/sdc /dev/sdd /dev/sde

150 lines

36

Summa summarum

(literally)

$ perl -ne '$n+=$1 if /(\d+) lines/; END {print "$n\n"}' talk.slide
3094
37

Thank you

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)