Importing Protobuf with Go Modules

Protocol buffers and gRPC are established standards for building modern web services. Go modules are now the default way of versioning packages in Go projects. So how can you import and reuse existing .proto definitions from your versioned dependencies while using Go modules?

While working on a Go project, I wanted to reuse existing protobuf definitions. This makes sense when interacting with an existing system (like Kubernetes) via an existing library (like client-go) and you want to further work with the retrieved objects. For example embed these in your own protobuf messages and send to another service for further processing. Recreating the definitions would be a lot of work and prone to running out of sync. And also, I’m lazy… so why would I want to do that?

Protocol buffers allow you to import definitions using the import keyword, but the import path has to point directly to a local .proto file. The protoc compiler does not recognize Go-style import paths. In both former cases the paths of dependencies within the given directory ($GOPATH/src or vendor) will match the import ones. With Go modules the actual location of files looks something like $GOPATH/pkg/mod/k8s.io/client-go@v0.0.0-20190830034949-326ca7a019c8 and no longer matches the path in the import directive. This makes sense, as protocol buffers format is not specific to Go and it is not an issue when using GOPATH or Go vendoring, but it won’t work well with Go modules.

To demonstrate this let’s look at following example. Definition from k8s.io/api/core/v1 package is being imported, and its Pod message embedded into newly declared Info message.

1
2
3
4
5
6
7
8

syntax = "proto3";
import "k8s.io/api/core/v1/generated.proto";

message Info {
  string id = 1;
  string message = 2;
  k8s.io.api.core.v1.Pod pod = 3;
}

So I wanted to reuse definitions from existing Go packages, didn’t know how to do that with Go modules and I didn’t find any good examples (that’s also why I decided to write this article). Some of the examples I came across - Argo CD or Istio. Both quite complex and relying on GOPATH or vendoring. The projects I’ve found mostly used non-trivial build scripts to address this issue and either did not reflect versions properly, required too much manual maintenance or relied on vendor directory.

Requirements

I started by thinking how would ideal solution look like and came up with a set of requirements:

• Correct versions - the imported Proto definitions must match the version of the relevant Go packages.
• Go modules - the solution has to work with Go modules.
• No Go vendoring - I don’t want to commit all the dependencies into my repository.
• Future-proof - the solution should still work when a new dependency is added or existing one updated, without any manual intervention.

First iteration - symlinks

When running protoc generator, one of the parameters is the import path and all import paths are resolved as relative to these. So what if we can symlink all the used Go modules under the expected paths?

Luckily the go command gives us arsenal of tools to work with Go source code. There’s go list, which lists all used modules (the -m flag) and their properties. You can even parametrize its output. For a list of all the options and output variables run go help list. We’re going to use {{ .Path }} – the Go module path, and {{ .Dir }} – path to directory containing package sources.

1
2
3
4
5

$ go list -f "{{ .Path }} {{ .Dir }}" -m all
github.com/stepanstipl/go-protobuf-import-example /Users/stepan/Projects/go-protobuf-import-example
cloud.google.com/go /Users/stepan/go/pkg/mod/cloud.google.com/go@v0.38.0
github.com/Azure/go-autorest/autorest /Users/stepan/go/pkg/mod/github.com/!azure/go-autorest/autorest@v0.9.0
...

With this information we can populate protobuf import directory with correct paths. Below is commented script that creates the required directory structure and symlinks to real directories:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/env sh
set -emou pipefail

# Set parent directory to hold all the symlinks
PROTOBUF_IMPORT_DIR='protobuf-import'
mkdir -p "${PROTOBUF_IMPORT_DIR}"

# Remove any existing symlinks & empty directories 
find "${PROTOBUF_IMPORT_DIR}" -type l -delete
find "${PROTOBUF_IMPORT_DIR}" -type d -empty -delete

# Download all the required dependencies
go mod download

# Get all the modules we use and create required directory structure
go list -f "${PROTOBUF_IMPORT_DIR}/{{ .Path }}" -m all \
  | xargs -L1 dirname | sort | uniq | xargs mkdir -p

# Create symlinks
go list -f "{{ .Dir }} ${PROTOBUF_IMPORT_DIR}/{{ .Path }}" -m all \
  | xargs -L1 -- ln -s

The script will create symlinks under the directory set in$PROTOBUF_IMPORT_DIR variable. Make sure you use the same protoc plugin as the imported packages. The example uses protoc-gen-gofast from gogoprotobuf, as opposed to default golang/protobuf. Also, if you’re not using gRPC, omit the plugins=grpc bit. Protobuf compiler should then be called with -I flag pointing to that directory:

1
2

$ protoc -I ./protobuf-import -I ./ ./my.proto \
   --gofast_out=plugins=grpc:./.

And voila! We have compiled our protocol buffer definition that references another third-party definition into a Go code.

Simpler - go mod vendor

Although I initially mentioned that I prefer not to use vendoring, we can still use go mod vendor command to populate vendor directory, without using it in the build process and without committing it to VCS. In this case it will be done by copying files, instead of creating symlinks.

After running go mod vendor, we can compile the protobuf definition into Go code same way as previously, just pointing the compiler to the vendor directory:

1
2
3
4
5

$ go mod vendor
go: finding github.com/stepanstipl/go-protobuf-import-example/pb latest

$ protoc -I ./vendor -I ./ ./my.proto \
   --gofast_out=plugins=grpc:./.

The good part is that you don’t need a custom script anymore, the bad part is that the process is a bit heavier on disk space, as all the files are duplicated. Go will use cached versions of modules to populate the vendor directory, so no network bandwidth is consumed.

In case of the project I was working on, a project with 126 modules, the difference was negligible (as expected the go mod vendor was a bit slower, and resulted in additional 33 MB of copied files).

1
2
3
4
5
6
7
8

$ go list -m all | wc -l
126
     
$ time ./protobuf-import.sh
./protobuf-import.sh  0.53s user 1.20s system 109% cpu 1.587 total

$ time go mod vendor
go mod vendor  0.58s user 1.44s system 124% cpu .618 total

Example

I’ve created a minimal gRPC server example, that is exposing service with embedded Pod message from Kubernetes, You can find this demo at stepanstipl/go-protobuf-import-example. Explore the protobuf-import.sh script from above and simple Makefile used to demonstrate both methods. Run make proto-link or make proto-vendor to generate the Go protobuf files after cloning the repo.

Summary

I hope this will help someone in a similar situation, as I didn’t find any good answer on versioning and import third-party protobuf definitions together with Go modules. Both methods are quite simple, but personally, I will probably end up using the go mod version one. The overhead seems to be small enough and you don’t have to maintain any additional scripts.

How do you version imported protobuf definitions? I would love to hear any comments or any questions you have. Follow me on @stepanstipl and happy coding!

References