Custom-marshal Golang structs with flattening

How to marshal a struct field that doesn’t implement the Marshaler interface

Recently, I needed to marshal a Go struct to JSON and BSON (binary JSON, a serialization format developed by MongoDB), but one of the fields in my struct was an interface that needed special handling.

Typically, it’s easy enough to provide custom marshaling functions: just make CustomType implement json.Marshaler and bson.Marshaler.

Whenever MyStruct needs to be marshaled, the marshaler will call our implementation of the marshal functions to marshal the value for myStruct.Custom. All we need to do is provide MarshalJSON and MarshalBSON functions for every implementation of CustomType.

Unfortunately, in my case, this was not an option. I already had dozens of implementations of my custom type, and a handy function that took any CustomType implementation and made it marshalable. (Note: for demo purposes, this function will just return the string constant “field prepared by MarshalableCustomType”.)

I didn’t want to provide MarshalJSON/MarshalBSON functions for each implementation of CustomType; I just wanted the marshaler to call MarshalableCustomType.

Mirror struct

A common way to do this is to create an auxiliary struct used just for marshaling MyStruct:

play with this code: https://play.golang.org/p/9RjAMpuHQSy

MyStruct’s MarshalJSON function creates an instance of myStructMirror that’s exactly the same as the original, except for the Custom field, which is created by calling MarshalableCustomType. Then, we marshal the myStructMirror to get what we want.

This works well, but there’s one obvious annoyance: every time we want to add a field that doesn’t need custom marshaling to MyStruct, we must also add it to myStructMirror and MarshalJSON (as well as MarshalBSON, UnmarshalJSON, andUnmarshalBSON) in order to ensure the new field gets marshaled and unmarshaled properly.

This makes adding a new field cumbersome and error-prone. While it’s possible to mitigate this with unit tests or compile-time checks, it would be preferable for future engineers to not have to add new fields to multiple places.

Ideally, our auxiliary struct would be modified such that we only specify the Custom field and its special marshaling behavior. The struct should then fall back to the default behavior for all other fields. To do this, we’ll take advantage of the flattening feature of our marshaling libraries. The code is slightly different for JSON and BSON, so we’ll address them separately.

Flattening JSON

For every field in a struct, Go’s JSON marshaler will create a new key/value pair in the serialized JSON. There’s one exception to this rule: embedded structs. If a field is an embedded struct of a parent, the child struct’s fields will be flattened, and included on the parent’s level.

An unflattened vs flattened JSON object.

Here’s a simple example of flattening:

play with this code: https://play.golang.org/p/4G0D_BIEIs_N

Here, ChildStruct is embedded in ParentStruct. When ParentStruct is marshaled, instead of creating a new key called “ChildStruct”, all the fields in ChildStruct are simply treated as if they were part of the parent.

We can take advantage of flattening to improve upon the mirror struct strategy from before. Instead of adding the fields from myStruct to myStructMirror, we’ll start by creating a type alias for MyStruct called myStructAlias, and embed it in the mirror struct instead.

Now, when we marshal MyStruct, Field1 and Field2 will be marshaled the way they normally would, but Custom will get marshaled using MarshalableCustomType. All will be placed at the same level in the output JSON.

play with this code: https://play.golang.org/p/EgsB9h0rSSZ

Flattening BSON

We can use the same principle with BSON, but the specifics are a bit different. The MongoDB BSON library respects an inline struct tag which provides the same behavior as JSON flattening on any struct field, so we don’t need to bother with the type alias for an embedded struct. The code is super simple:

Et voila! When we try to marshal into BSON, we get our new custom-marshal output:

play with this code: https://play.golang.org/p/mITs1ZuKsmj

Note: an older version of the mongo-driver BSON package would complain because Custom shows up twice on the top-level. To solve this, suppress it from the original MyStruct so that it doesn’t conflict with the marshaling of myStructMirrorBSON.Custom.

Final Thoughts

In this post, we mostly addressed marshaling, but of course most things that are marshaled need to be unmarshaled as well. Assuming you have an UnmarshalCustomType function that does the inverse of MarshalCustomType, it should be fairly simple to write UnmarshalJSON and UnmarshalBSON functions that unmarshal into the auxiliary struct and then convert into MyStruct. Give it a try yourself!

When possible, it’s still preferable to implement the json.Marshaler or bson.Marshaler interfaces so that you don’t need auxiliary structs for marshaling at all. However, when auxiliary structs are unavoidable, embedding and flattening structs can save you from having to add new fields in many different places — preventing major headaches.

Special thanks to Jarrett Gaddy and Chris Warth for reviewing an earlier version of this post.