Dynamic JSON in Go

Go is a statically typed language. While it can represent dynamic types, making a nested map[string]interface{} duck quack leads to very ugly code. We can do better, by embracing the static nature of the language.

The need for dynamic, or more appropriately parametric, content in JSON often arises in situations where there's multiple kinds of messages being exchanged over the same communication channel. First, let's talk about message envelopes, where the JSON looks like this:

{
	"type": "this part tells you how to interpret the message",
	"msg": ...the actual message is here, in some kind of json...
}

Generating JSON with different message types

Marshaling data structures into JSON with multiple types of message bodies with a separate envelope is easy with interface{}. To generate this:

{
	"type": "sound",
	"msg": {
		"description": "dynamite",
		"authority": "the Bruce Dickinson"
	}
}
{
	"type": "cowbell",
	"msg": {
		"more": true
	}
}

We can use these Go types:

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

type Envelope struct {
	Type string
	Msg  interface{}
}

type Sound struct {
	Description string
	Authority   string
}

type Cowbell struct {
	More bool
}

func main() {
	s := Envelope{
		Type: "sound",
		Msg: Sound{
			Description: "dynamite",
			Authority:   "the Bruce Dickinson",
		},
	}
	buf, err := json.Marshal(s)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s\n", buf)

	c := Envelope{
		Type: "cowbell",
		Msg: Cowbell{
			More: true,
		},
	}
	buf, err = json.Marshal(c)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%s\n", buf)
}

Output:

{"Type":"sound","Msg":{"Description":"dynamite","Authority":"the Bruce Dickinson"}}
{"Type":"cowbell","Msg":{"More":true}}

Nothing special there.

Unmarshal into dynamic hell

If you just ask to unmarshal the above JSON objects into the Envelope type from above, you'll end with Msg being a map[string]interface{}. That's not very nice to use, and will make you regret your life choices:

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

const input = `
{
	"type": "sound",
	"msg": {
		"description": "dynamite",
		"authority": "the Bruce Dickinson"
	}
}
`

type Envelope struct {
	Type string
	Msg  interface{}
}

func main() {
	var env Envelope
	if err := json.Unmarshal([]byte(input), &env); err != nil {
		log.Fatal(err)
	}
	// for the love of Gopher DO NOT DO THIS
	var desc string = env.Msg.(map[string]interface{})["description"].(string)
	fmt.Println(desc)
}

Output:

dynamite

Unmarshaling the very explicit way

Previously, I used to recommend changing the Envelope type, like this:

type Envelope {
	Type string
	Msg  *json.RawMessage
}

json.RawMessage is a useful type that lets you postpone the unmarshaling; it just stores the raw data as a []byte.

This lets you explicitly control the unmarshaling of Msg, and thus delay that until after you've branched out based on the value of Type.

The downside of that is that now you need to explicitly marshal Msg first, or you need separate EnvelopeIn and EnvelopeOut types, where EnvelopeOut still has Msg interface{}.

We can do better than that.

Combining the powers of *json.RawMessage and interface{}

So, how to combine the good aspects of the two approaches above? By putting a *json.RawMessage in the interface{} field!

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

const input = `
{
	"type": "sound",
	"msg": {
		"description": "dynamite",
		"authority": "the Bruce Dickinson"
	}
}
`

type Envelope struct {
	Type string
	Msg  interface{}
}

type Sound struct {
	Description string
	Authority   string
}

func main() {
	var msg json.RawMessage
	env := Envelope{
		Msg: &msg,
	}
	if err := json.Unmarshal([]byte(input), &env); err != nil {
		log.Fatal(err)
	}
	switch env.Type {
	case "sound":
		var s Sound
		if err := json.Unmarshal(msg, &s); err != nil {
			log.Fatal(err)
		}
		var desc string = s.Description
		fmt.Println(desc)
	default:
		log.Fatalf("unknown message type: %q", env.Type)
	}
}

Output:

dynamite

How to put everything at the top level

While I heartily recommend you put the parametric body under a single key, sometimes you work with pre-existing formats that just don't do that.

Please use the earlier style if you can.

{
	"type": "this part tells you how to interpret the message",
	...the actual message is here, as multiple keys...
}

We can cope with that by unmarshaling the data twice:

package main

import (
	"encoding/json"
	"fmt"
	"log"
)

const input = `
{
	"type": "sound",
	"description": "dynamite",
	"authority": "the Bruce Dickinson"
}
`

type Envelope struct {
	Type string
}

type Sound struct {
	Description string
	Authority   string
}

func main() {
	var env Envelope
	buf := []byte(input)
	if err := json.Unmarshal(buf, &env); err != nil {
		log.Fatal(err)
	}
	switch env.Type {
	case "sound":
		var s struct {
			Envelope
			Sound
		}
		if err := json.Unmarshal(buf, &s); err != nil {
			log.Fatal(err)
		}
		var desc string = s.Description
		fmt.Println(desc)
	default:
		log.Fatalf("unknown message type: %q", env.Type)
	}
}
dynamite

Parting thoughts

Hope this inspired you to clean up your JSON handling. Also see the follow-up article for how to avoid the switch statements you see here.

I have more related topics in store, but please let me know what you find interesting! You can reach me at tv@eagain.net.