Review of hosted CI for Go

Review of a few hosted Continuous Integration services offering Go support

I’ve been working on the Continuous Integration testing of Google Chrome and Chromium since 2008. As a matter of fact Chromium has a very large Buildbot based CI infrastructure that is open source. You can get a glimpse at chromium-build.appspot.com.

Chromium finished migrating to Swarming based task execution hosted on Google AppEngine [edit] in early 2019.

In the context of my personal project pre-commit-go, I had the chance to try out a few hosted Continuous Integration services offering Go support. I sampled 4 that could be used for free for open source projects.

Since pre-commit-go is a project that does check the state of the repository when running and also installs dependencies via go get as part of its normal operation, its smoke tests tend to be more complex than usual and I did hit many corner cases, which is even more interesting in the context of this review.

Travis

Badge:

Advantages

• It is one of the oldest service and is very complete.
  • It has very low chance of disappearing.
• Lets you to run tests against multiple versions of Go, even against tip! This is very useful for libraries that want to support older version of Go like 1.2. This means your build is run once per Go version requested. Minor versions (e.g. 1.4 vs 1.4.2) are supported.
• Generally speaking, Travis does the right thing for Go settings, the repository is checked out inside $GOPATH at its expected location and $GOPATH/bin is in $PATH. This results is a very low fuzz setup.
  • These environments are clearly shown as part of the build output.

Disavantages

• You can’t SSH in to debug a build so you have to do multiple commits to figure out when there’s a problem.
• There is no way to enable Travis without a .travis.yml. Not everyone is comfortable with this. On the other hand, this clearly states what is supported and what is the testing scenario, unlike other services which can have configuration modified out-of-band.
• Can’t disable email notifications without a commit, e.g. updating .travis.yml.
  • IMHO email notification settings do not belong inside the repository.
• Build failure email requires a click through to find out what failed.

CircleCI

Badge:

Advantages

• Lets you SSH into the bot for 30 minutes to debug a failure! The workflow is that you run a build then you request to rerun it with SSH enabled. Once done, you cancel the build to terminate the docker.
• Lets you do 4 parallel builds for free, which is quite nice. When a bot is running a SSH enabled build, this takes a parallel build slot so you can enable up to 4 SSH enabled builds in parallel.
• Doesn’t require to check in any file in the repository.
  • Optionally support checked in circle.yml configuration file.
• Build failure email includes a dump of the step output. Surprisingly, it’s the only service doing this. It is very useful.

Disadvantages

• Can’t specify Go version. As of 2015-06-29, it is using 1.4 while 1.4.2 was released a 4 months ago! This is problematic.
• It uses build output caching which does get in the way, especially due to the weird layout that CircleCI uses for repositories.
• Uses a multivalue $GOPATH: /home/ubuntu/.go_workspace:/usr/local/go_workspace.
• Checks out in ~/PROJECT_NAME with your project name instead of inside $GOPATH.
  • This means that readlink -f . returns a path outside of $GOPATH.
• In general, it is quite immature:
  • It used to use ~/.go_project/src/<repo/path> which contained a symlink to the project checkout that is directly in ~/ but this changed mid-June 2015 without alert!
  • For a while, ~/.go_workspace/bin was not in $PATH. You had to add it manually if needed.
• Can’t completely disable email notifications.

Codeship

Badge:

Advantages

• Lets you SSH into the bot for an unspecified amount of time to debug a failure. Unlike CircleCI, you give one build at a time.
• Doesn’t require to check in any file in the repository.

Disadvantages

• Can’t specify Go version. As of 2015-06-29, it is using 1.4 while 1.4.2 was released a 4 months ago! This is problematic.
• Current working directory is outside $GOPATH. It is ~/clone which is a symlink to the right directory inside $GOPATH. cd’ing to the right directory works just fine but that is not Go idiomatic at all.
• Advertizes a limit of 100 builds per month for free builds.
• Build failure email requires a click through to find out what failed.

DroneIO

Badge:

Unlike other services, drone can be run stand alone on your own VM; https://github.com/drone/drone. On the other hand, it is relatively barebone compared to the other services.

Advantages

• Doesn’t require to check in any file in the repository.

Disadvantages

• It uses a git template which gets in the way if you ever run git in a smoke test. As long as you do not do smoke tests that involves git, you should be fine, which is the normal case for projects.
• Can’t specify Go version. As of 2015-06-29, uses 1.4.2, which is the current version. Since Drone is itself written in Go, it is expected that it follows the latest released version closely.
• You can’t SSH in to debug a build so you have to do multiple commits to figure out when there’s a problem.
• Build failure email requires a click through to find out what failed.

Conclusion

I sampled 4 hosted services but there are many others. Many include support for deployment, which is not covered in this review but is an excellent feature. For example I personally use the codeship’s deployment feature as part of webskel.

This is very nice of these services to offer free builds for open source projects. This permits running tests on pull requests and trying out the service before paying. For small open sources projects, this permits having a no-fuzz no-maintenance CI system for free, which is incredible.

On the other hand, throughout the years a few hosted services failed and disappeared. This is something to take in account as you settle for an hosted service.

As for which service to use, it depends on your use-case. I found SSH support to be immensely useful but both Codeship and CircleCI use non Go idiomatic checkout layout and both use an outdated version of Go. Both issues are very concerning. Travis is very feature complete, no fuzz and works fine for open source projects but many found is slow to their taste and it requires you to check in a .travis.yml file in your repository. Drone is a bit of an outlier as it is open source and can be run by yourself in a docker image but I haven’t tested this use case.

If you have to run tests on a Go project, check out pre-commit-go as a free open source tool to run tests efficiently.