October 25, 2019

Go Modules overview

Go modules it is only two files:

  • go.mod
  • go.sum

and some new tools in go cmd:

  • go mod init creates a new module, initializing the go.mod file that describes it.
  • go build, go test, and other package-building commands add new dependencies to go.mod as needed.
  • go list -m all prints the current module’s dependencies.
  • go get changes the required version of a dependency (or adds a new dependency).
  • go mod tidy removes unused dependencies.

Now, let's dive into back-end staff of the go modules. Continue reading text below.

Checksum Database: overview

Modules introduced the go.sum file, which is a list of SHA-256 hashes of the source code.
When you add a version of a dependency that you’ve never seen before to your module, the go command fetches the code and adds lines to the go.sum file on the fly.
The problem is that those go.sum lines aren’t being checked against anyone else. For example, when a proxy intentionally served malicious code targeted to you.
Go's solution is a global source of go.sum lines, called a checksum database, which ensures that the go command always adds the same lines to everyone's go.sum file.
Even the author of a module can’t move their tags around or otherwise change the bits associated with a specific version from one day to the next without the change being detected.

Checksum Database: how it works

The checksum database is served by sum.golang.org,
and is built on a Transparent Log (or “Merkle tree”) (https://research.swtch.com/tlog) of hashes backed by Trillian (https://github.com/google/trillian).
The main advantage of a Merkle tree is that it is tamper proof and has properties that don’t allow for misbehavior to go undetected, which makes it more trustworthy than a simple database.

Modules Proxy Server: overview

It's a special server which already used by default in go modules.
It store code for all dependencies in your go.mod file and their dependencies in their go.mod files.

This means you don’t have to use any vendor/ folder anymore.
You don't need to install any VCS tools for fetch dependent modules. It uses go's net/http for it.

By default it use Google's proxy server. You can review it from new env var GOPROXY.

Articles:

Modules versioning

The version referenced in a go.mod may be:

  • an explicit release tagged in the repository (for example, v1.5.2),
  • it may be a pseudo-version based on a specific commit (v0.0.0-20170915032832-14c0d48ead0c).

! Once you start tagging your repo with versions, it's important to keep tagging new releases as you develop your module.
Do not delete version tags from your repo. If you find a bug or a security issue with a version, release a new version.
Once you release a version, do not change or overwrite it.

Modules versioning: SemVer overview

- Increment the MAJOR version when you make a backwards incompatible change to the public API of your module. This should only be done when absolutely necessary.
- Increment the MINOR version when you make a backwards compatible change to the API, like changing dependencies or adding a new function, method, struct field, or type.
- Increment the PATCH version after making minor changes that don't affect your module's public API or dependencies, like fixing a bug.

Modules versioning: pseudo-version

Pseudo-versions are useful when a user needs to depend on a project that has not published any semantic version tags.

Example:

go get github.com/CyCoreSystems/ari@b5b364958e10bdff2dc374dea2b85981b3e3e37c

Modules versioning: rules

! v0 major versions and pre-release versions do not guarantee backwards compatibility.

! A v1 major version communicates to users that no incompatible changes will be made to the module's API. If there are changes to the API, they will be backwards compatible (for example, adding a new field to a struct).
Sometimes, maintaining backwards compatibility can lead to awkward APIs. That's OK. An imperfect API is better than breaking users' existing code.

Modules versioning: complications with major versioning

How to start

run go mod init

How to tag a new major version v0

1. run go mod tidy
2. go test ./...
3. git tag {the version}
4. push the tag to the origin repository

Known Problems

Problem #1: Go can't verify your private repository's go.sum in sum.golang.org

If you run go run / go build with dependencies of some private modules, you get error as following:

go run/go build: verifying reading https://sum.golang.org/lookup/{PRIVATE_MODULE}: 410 Gone

Solution

Add your private domains into Go's special environment variable as below:

export GOPRIVATE="go.kolesa-team.org"

Future changes

They will try to solve it in the next minor version of golang:

The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either.

Related issues:

Problem #2: Some modules are already has versions greater that v0 or v1

Example:

go get github.com/CyCoreSystems/[email protected] 
> github.com/CyCoreSystems/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v5

Future changes

  1. Wait from authors of a module to move their greater versions under the folder like:
    "github.com/CyCoreSystems/ari/v5"
  2. Wait for a tag with the latest version of the module, like:
    "v5.0.0"

Solution

  1. Download the module by locking for a some commit's id
    For example, get an ID of the last commit in the master branch and do this:
    go get github.com/CyCoreSystems/ari@b5b364958e10bdff2dc374dea2b85981b3e3e37c
    > go: extracting github.com/CyCoreSystems/ari v0.0.0-20191010021544-b5b364958e10

Bad advice

  • Donwload the module by defining master branch
    It won't help, because go modules will be lock by the latest tag in the repository.
    It'll be the same behaviour with what you've already tried by locking as latest version: go get github.com/CyCoreSystems/ari@latest
    which is same with just go get github.com/CyCoreSystems/ari

Documentation:

Related issues:

Problem #3: You have to install git or other VCS for download dependencies without go.mod file

The first thing the Go command does in module mode (whether running go build, go test, go list, ...) is load the go.mod of the main module and any other modules that are transitively required. If go.mod files are not present in the module cache, they are fetched via GOPROXY or VCS tools.

Related issues: