🐤 How to test Microservice by using a Canary with Golang λ-functions?

In this article we will discuss the possibility of testing a microservice using Canary based on Golang λ-functions.

The Canary is an effective way of interface-testing, the features of which are:

1. Periodicity. Tests can be run with a certain period, or on a one-time basis.
2. Logging. The test results are logged and stored on some storage or in a specially prepared database.
3. Visualization. The test results can be presented in the form of dynamic and active graphs.

Despite the fact that the main purpose of Canaries is testing, which allows you to check the quality of web services, Canaries can also be used not only for testing, but also for launching some services that are planned to be performed in accordance with some routine maintenance.

Currently, AWS has the Amazon CloudWatch Synthetics protocol, which allows you to run testing using AWS::Synthetics::Canary. The main differences of the protocol are:

• When creating Canary, an initializing AWS is automatically generated AWS::Lambda::Function (λ-canary), written with Puppeteer.
• λ-canary is designed to run tests, process responses, log CloudWatch Metrics and save results to S3.
• The resulting metrics can be used to create AWS::CloudWatch::Alarm with sending messages to Slack and other corporate messengers.

We can present the work of Canary, in the form of the following scheme:

Problem Definition

Despite the fact that Canary is quite simple to create, nevertheless, when using this tool, at least four global problems can be identified.

• The problem of adding NPM packages. The Puppeteer is a node library, which provides a high-level API for managing Chromium or Chrome over the DevTools protocol. However, if testing requires the implementation of non-standard things, then the only way to expand the existing functionality is to upload the missing packages to the node_modules (NM) directory. The weight of the archive with packed NM can significantly exceed the size of the tests themselves, which leads to the need to upload the NM archive to S3 storage. There are studies showing that the size of an archive with NM can reach 700 mb [Polak A., Reduce node modules for better performance, 2019].
• The problem of packaging tests. When the number of tests is small, they can be written directly in the source code of λ-canary. With an increase in the volume of testing, the problem of the growth of the source code of λ-canary is solved by distributing interface tests into separate files, with their placement in specific resources - Layers (λ-layers). The volume and number of λ-layers are fixed, and certain rules must be observed for the names of λ-layers, such as the impossibility of the existence of two or more λ-layers with the same names.
• The problem of degradation of the testing architecture. Sometimes there is a need to test a pair of interacting HTTP requests within the same Canary. Therefore, the only place to exchange responses between requests will be the main code λ-canary. This blurs the original meaning of λ-canary, as a function that deals only with running tests, interpreting and representing results.
• The problem of support for λ-canary. Sometimes testing needs to be carried out at the time of development of a particular microservice. The approach of offering to place testing in λ-layers does not allow you to dynamically change the test code. The code in the λ-layer is closed from the developer, and the content in the λ-layer is changed by updating the entire content of the λ-layer with a version update. Often a whole team is working on a project, and such an unobvious process of working on the source code of tests requires a strict declaration of logging rules, indicating the name of the test, the time of failure, the version of the λ-layer, and other things.

High Level Solution

The problems described above can be solved by adding additional λ-function, which can host code for testing.

The size of one λ-package can vary from 5 to 10 MB.

λ-functions can communicate with each other, avoiding the return of intermediate results in λ-canary. Moreover, there are no restrictions on the number of created λ-functions. Such a mechanism ensures a clean testing environment, making testing easy to maintain.

λ-functions can be used in other services, both as independent microservices and functions for interface-testing.

Changing the microservice entails the need to change the λ-function used to test a specific endpoint. Therefore, maintenance and development of service testing becomes relatively simple.

However, this model has its drawbacks:

• Increasing the number of λ-functions. The testing development process is expanding. It is necessary to take into account the nuances when creating new λ-function for a variety of microservices. Newly created λ-functions can repeat each other if several teams are engaged in the development of microservices.
• A rather inconvenient built-in service for working with the λ-function. Since AWS does not have a code editor for Lambda with Golang, the adjustment of the tests is provided by updating them and uploading the archive to S3. This is still better than searching for a test in a set of Layers and updating its contents, but not good enough to call this approach ideal.

In fact, we are bound by the Amazon protocol, somehow forcing us to use suboptimal ways of working. It would be much easier if Canary testing existed independently of Amazon, especially since Canaries-testing is a unified way to run tests, as well as interpret and represent test results.

Implementation

The architecture of the S3 directory in which the test results will be placed looks like this

/root s3
|--...
|--/<bucket>
|----/<aws-region>
|------/<canary name>
|--------/logs
|--------/source

The above method of storing test results provides differentiation of the work of the λ-function from different services and allows you to unify the interaction model of the λ-canary and the launched λ-functions.

For the normal functioning of λ-canary and λ-functions, it is necessary to create the correct set of roles that allow you to work with the created directory on S3, read and write information to the AWS CloudWatch log.

The creation of roles, the addition of directories to S3, the loading and updating of the necessary λ-package in the λ-function, the preparation of the initiating λ-canary, as well as the creation of new λ-functions can be described using the Golang standalone microservice.

Canary creation is possible using the library github.com/aws/aws-sdk-go/service/synthetics . Let's give as a small example, a block of code that allows you to create a Canary.

func CreateCanary(session *session.Session, config *aws.Config, service *models.ServiceParameters) error {
	/*
		Prepare a Canary:
		Send to the S3 already existed *.zip file with the executable Golang file.
		It can be a small HealthCheck, for example.

		Also create a synthetics parameters for the Canary:
	*/
	s3LocationLogs := "s3://" + service.Bucket + "/" + service.Region + "/" + service.Canary.Name + "/logs/"
	s3LocationFile := "s3://" + service.Bucket + "/" + service.Region + "/" + service.Canary.Name + "/source/" + service.Canary.Function + ".zip"
	prepareCanaryCode(session, config, service)
	parameters := synthetics.CreateCanaryInput{
		ArtifactS3Location: aws.String(s3LocationLogs),
		Code: &synthetics.CanaryCodeInput{
			Handler:  aws.String(service.Canary.Function.Handler),
			S3Bucket: &service.Bucket,
			S3Key:    aws.String(s3LocationFile)
		},
		ExecutionRoleArn: aws.String(service.Role.ARN),
		Name:             aws.String(service.Canary.Name),
		RuntimeVersion:   aws.String(service.Canary.RuntimeVersion),
		Schedule: &synthetics.CanaryScheduleInput{
			DurationInSeconds: aws.Int64(60),
			Expression:        aws.String(service.Canary.Expression),
		},
	}
	/*
		Create a Synthetics Client with prepared parameters.
	*/
	client := synthetics.New(session, config)
	if _, err := client.CreateCanary(&parameters); err != nil {
		fmt.Println("Failed to create a Canary. Error:", err.Error())
		return err
	}
	return nil
}

A model ServiceParameter contains an information about Lambdas, Canary, AWS Credentials, S3 and etc. For example, we can create an object service with parameters

*
	Variable title is the Title of the Canary.
	This may be the name of the service that needs to be tested, or something else.
*/
title   := "yellow-canary"

service := models.ServiceParameters{
	Region:  "us-west-2",
	Service: title,
	Bucket:  title + "-us-west-2",
	Policy:  title + "-policy",
	Role: models.Role{
		Name: title + "-role",
	},
	Canary: models.Canary{
		Name:           title + "-canary",
		Function:       "canary",
		Handler:        "handler",
		RuntimeVersion: "syn-nodejs-puppeteer-3.4",
		Expression:     "rate(5 minutes)",
	},
	Lambdas: []models.Lambda{
		{
			Name:     "get-healthcheck-lambda",
			Filename: "get-healthcheck.zip",
			Runtime:  "go1.x",
			Handler:  "get-healthcheck",
			Role: models.Role{
				Name: title + "-lambda-role",
			},
		},
		...
	},
}

In order to run Canary, we have to send the necessary λ-package to S3. The preparation of the λ-package is described in the article «‎H‎ow to create a λ-function for the Golang Microservice?».

It is also necessary to prepare the λ-package for the λ-canary

zip -r canary.zip <repo root>/nodejs/*

Note that the λ-package for λ-canary will be placed in the /source directory of the created partition in S3 storage. The source code of the λ-canary can process the results of the execution of the λ-function, such as, for example, interpret and display the received code, create an alert based on it, and much more.

This approach differentiates the execution of the λ-function from interface testing, and interface testing from the interpretation of tests.

Repository

See the sample of a Golang λ-function with a Database Connection in repo:

🍃 GO TO REPO 🍃

Contents

1. Polak A., Reduce node modules for better performance, 2019.