Tv's projects
winner go gets it all
21 November 2013
Tommi Virtanen
Tommi Virtanen
github.com/tv42 (because 26**2 is not enough for everyone)
Ask for more when it gets interesting.
demand -- Download, build, cache and run a Go app easily staged -- Run a command with the Git staged files jog -- Structured logging library birpc -- Bi-directional RPC library, including JSON-over-WebSocket topic -- in-process single-topic pub-sub adhoc-httpd -- Quick & dirty HTTP static file server adhoc-httpd-upload -- Quick & dirty HTTP file upload server humanize-bytes -- Command-line utilities to convert "MiB" etc to raw numbers, and back json-point -- Command-line tool to extract parts of JSON documents, with JSON Pointer syntax slug -- Create slugs of text or URLs, for use in e.g. URL path segments base58 -- Encode numeric ids to pretty tokens for e.g. URLs seed -- Demo of how to easily seed PRNGs with some entropy twackup -- Back up your tweets didyouseethis -- Retweet anything with certain keywords upstart-socket -- Go library to listen on a socket handed in by Upstart (/systemd etc) maildir-fix -- Fix maildirs after git has pruned empty dirs msgpack-json -- Command-line utilities to convert between msgpack and json spindown-idle-disks -- Spin down idle SATA disks. Because hdparm -S 120 just won't work
Put in ~/bin:
#!/usr/bin/env demand go: import: github.com/tv42/humanize-bytes/cmd/bytes2human
Then run
$ chmod a+x ~/bin/bytes2human $ bytes2human 65536 64KiB
Usage of demand: demand [OPTS] SPEC_PATH [ARGS..] Only build, not run command: demand -build [OPTS] SPEC_PATH.. Options: -build=false: only build, do not run command (can pass multiple spec files) -gopath=false: use GOPATH from environment instead of downloading all dependencies -upgrade=false: force upgrade even if older version exists Use as an interpreter: #!/usr/bin/env demand go: import: GO_IMPORT_PATH_HERE
github.com/tv42/demand/blob/master/main.go
6github.com/tv42/demand/blob/master/cram/run.t
$ cat >foo <<EOF > go: > import: github.com/tv42/humanize-bytes/cmd/bytes2human > EOF $ XDG_CONFIG_HOME="$PWD/home" DEMAND_CACHE_DIR="$PWD/cache" demand foo 65536 64KiB
Ever forget to commit a new file?
Ever commit only half the change?
Usage of /home/tv/bin/staged: /home/tv/bin/staged COMMAND..
$ git init --quiet $ cat >demo.go <<'EOF' > package demo > func foo() int { > return 42 > } > EOF $ cat >demo_test.go <<'EOF' > package demo > > import "testing" > > func TestFoo(t *testing.T) { > if g, e := foo(), 42; g != e { > t.Errorf("bad foo: %v != %v", g, e) > } > } > EOF
$ git add demo_test.go # forget to add demo.go $ go test # ignores the problem PASS ok * (glob) $ staged go test # detects the problem # * (glob) ./demo_test.go:6: undefined: foo FAIL\t* [build failed] (esc) (glob) staged: command failed: exit status 2 [1] $ staged ls # no demo.go demo_test.go $ git add demo.go $ staged go test PASS ok * (glob)
github.com/tv42/staged/blob/master/main.go
11Often, ease of processing is more important than human eyeball friendliness.
{"Time":"2013-04-23T20:51:17.633016Z","Type":"example.com/webshop#login","Data":{"User":"jdoe","IP":"12.34.56.78"}}
type start struct { Enthusiasm string } type stop struct { Style string Points float32 } // Example output: // // {"Time":"2013-04-23T20:51:17.632909Z","Type":"main#start","Data":{"Enthusiasm":"high"}} // {"Time":"2013-04-23T20:51:17.633016Z","Type":"github.com/tv42/jog/example/greetings#greet","Data":{"Message":"Hello, world!","ID":"43235433"}} // {"Time":"2013-04-23T20:51:17.633047Z","Type":"main#stop","Data":{"Style":"commendable","Points":9.7}} func main() { log := jog.New(nil) log.Event(start{Enthusiasm: "high"}) greetings.Greet(log) log.Event(stop{Style: "commendable", Points: 9.7}) }
github.com/tv42/jog/blob/master/jog.go
14
github.com/tv42/birpc
chat example
cd ~/go/src/github.com/tv42/birpc/examples/chat ./run
{"id":"1","fn":"Arith.Add","args":{"A":1,"B":1}} {"id":"1","result":{"C":2}} {"id":"1","error":{"msg":"Math is hard, let's go shopping"}}
type Message struct { // 0 or omitted for untagged request (untagged response is illegal). ID uint64 `json:"id,string,omitempty"` // Name of the function to call. If set, this is a request; if // unset, this is a response. Func string `json:"fn,omitempty"` // Arguments for the RPC call. Only valid for a request. Args interface{} `json:"args,omitempty"` // Result of the function call. A response will always have // either Result or Error set. Only valid for a response. Result interface{} `json:"result,omitempty"` // Information on how the call failed. Only valid for a // response. Must be present if Result is omitted. Error *Error `json:"error,omitempty"` }
wetsock for JSON over WebSocket messages
jsonmsg for (lines of) json over any io.ReadWriteCloser
func (c *Chat) Message(msg *Incoming, _ *nothing, ws *websocket.Conn) error { log.Printf("recv from %v:%#v\n", ws.RemoteAddr(), msg) c.broadcast.Broadcast <- Outgoing{ Time: time.Now(), From: msg.From, Message: msg.Message, } return nil }
func (c *codec) FillArgs(arglist []reflect.Value) error { for i := 0; i < len(arglist); i++ { switch arglist[i].Interface().(type) { case *websocket.Conn: arglist[i] = reflect.ValueOf(c.WS) } } return nil }
Used in tests to avoid TCP.
pipe_client, pipe_server := net.Pipe()
fakeListener := oneshotlisten.New(pipe_server) done := make(chan error) go func() { defer close(done) done <- server.Serve(fakeListener) }()
ws, _, err := websocket.NewClient( pipe_client,
err = <-done if err != nil && err != io.EOF { t.Fatalf("http server failed: %v", err) }
$ wc -l birpc.go message.go wetsock/wetsock.go 387 birpc.go 54 message.go 90 wetsock/wetsock.go 531 total $ wc -l examples/chat/chat.js examples/chat/main.go 135 examples/chat/chat.js 142 examples/chat/main.go 277 total
Remember this?
func (c *Chat) Message(msg *Incoming, _ *nothing, ws *websocket.Conn) error { log.Printf("recv from %v:%#v\n", ws.RemoteAddr(), msg) c.broadcast.Broadcast <- Outgoing{ Time: time.Now(), From: msg.From, Message: msg.Message, } return nil }
godoc.org/github.com/tv42/topic (local)
lossy is good, because lossy is realistic
22mkdir junk adhoc-httpd -port 8000 junk
50 lines, but oh so convenient
23github.com/tv42/adhoc-httpd-upload
adhoc-httpd-upload -port 8001 junk
This time DIR
is mandatory, so you don't mess up $HOME
.
111 lines
Time for a dramatic demo
24github.com/tv42/humanize-bytes
$ human2bytes -h Usage of /home/tv/bin/human2bytes: /home/tv/bin/human2bytes [NUMBER..] /home/tv/bin/human2bytes <FILE -sloppy=false: continue past parse errors $ human2bytes 100M 100000000 $ human2bytes 100MiB 104857600 $ bytes2human -h Usage of /home/tv/bin/bytes2human: /home/tv/bin/bytes2human [NUMBER..] /home/tv/bin/bytes2human <FILE -si=false: use base-10 SI units instead of base-2 units like MiB -sloppy=false: continue past parse errors $ bytes2human 992342522 946MiB $ du -b * | sort -nr | head -3 | bytes2human 639KiB drama.gif 56KiB gopher.jpg 5.1KiB talk.slide
180 lines
25JSON Pointer spec: simple descend, no matching.
Library by Dustin. Keep your JSON as []byte
as long as you can.
$ echo '{"foo":{"quux":"thud"}}' | json-point /foo {"quux":"thud"} $ echo '{"foo":{"quux":"thud"}}' | json-point /foo/quux "thud" $ echo '{"foo":"bar", "quux":"thud"}' | json-point -pretty /foo bar
func ExampleSlug() { fmt.Println(slug.Slug("Rødgrød med fløde")) fmt.Println(slug.Slug("BUY NOW!!!11eleven")) fmt.Println(slug.Slug("../../etc/passwd")) // Output: // rødgrød-med-fløde // buy-now-11eleven // etc-passwd }
func ExampleURLString() { s, err := slug.URLString("https://www.example.com/foo/index.html") fmt.Println(s, err) // Output: // example-com-foo <nil> }
100 lines
27$ base58-encode 2345235235 4zeWJg $ base58-decode 4zeWJg 2345235235
Good example of lib + multiple mains
lib 70 + cmd 50 = 120 lines
28package main import ( "fmt" "github.com/tv42/seed" ) func main() { d6 := seed.Rand().Int31n(6) fmt.Println(d6) }
silly 33 lines
29$ cd ~/Private/personal/tweets/ $ twackup tv . Fetching tweets newer than 373615153020817408 Fetching url https://api.twitter.com/1.1/statuses/user_timeline.json?[...] Fetching url https://api.twitter.com/1.1/statuses/user_timeline.json?[...] Saved 17 new tweets Fetching tweets older than 1417093 Fetching url https://api.twitter.com/1.1/statuses/user_timeline.json?[...] $ json-point /text <401067312066732032.json "I have 22 browser tabs open. On my phone. #0thworldproblem #browsertabitis"
340 lines
30Twitter Streaming API client
Saves & retweets tweets matching keywords
The good kind of stalker
Run this on a protected account to keep track of mentions of interesting things
Because Twitter Search sucks
680 lines
31 dog := watchdog.New(90 * time.Second)
select { case line, ok := <-lines: if !ok { err = <-readError log.Fatalf("error reading stream: %s\n", err) } dog.Pet()
case <-dog.Bark: log.Fatalf("stream timeout") }
github.com/tv42/upstart-socket
start on socket PROTO=inet PORT=5000 setuid foo setgid foo exec /some/where/over/the/rainbow/food
It gives you a Listener
:
l, err := upstart.Listen()
30 lines
33Because I'm crazy enough to keep my mail in Git
$ ls Spam cur/ tmp/ $ maildir-fix Spam $ ls Spam cur/ new/ tmp/ $ maildir-fix -depot=.
In Git hooks
120 lines
34$ echo '{"foo": 42}'|json2msgpack >temp $ hexdump -C temp 00000000 81 a3 66 6f 6f cb 40 45 00 00 00 00 00 00 |..foo.@E......| 0000000e $ msgpack2json <temp {"foo":42}
60 lines
35github.com/tv42/spindown-idle-disks
description "spin down idle disks" start on runlevel [2345] stop on runlevel [!2345] respawn exec /usr/local/sbin/spindown-idle-disks -idle=30m /dev/sdb /dev/sdc /dev/sdd /dev/sde
150 lines
36(literally)
$ perl -ne '$n+=$1 if /(\d+) lines/; END {print "$n\n"}' talk.slide 3094