Contents of a bundle image are:

• Kubernetes Custom Resource Definitions (CRDs)
• ClusterServiceVersion (CSV)
• Specification of operator’s dependencies
• Operator metadata like its name, version, channels, etc.

The control loops associated with the operator are defined in its Controller Manager. It is an executable that contains one or more custom controllers.

The Operator Lifecycle Manager (OLM) pulls the bundle image from a registry and installs it on the cluster.

Operator SDK

Operator SDK is a project under Operator Framework that provides tools for building, testing, and packaging operators using the operator-sdk utility.

Using Operator SDK we can create operators based on Ansible Roles, Go Programming Language, or Helm Charts.

Creating a Go-based Operator

In this article, I will create a Memcached operator using operator-sdk CLI.

Memcached is a memory caching system, often used by developers to increase the performance of API calls or databases. It stores data as key-value pairs in RAM. The Memcached operator will make the following changes to the cluster

• Create a Memcached custom resource.
• Add a controller manager for Memcached resources.
• Implement APIs to interact with custom resources.

Prerequisites:

• GNU Make (make)
• Docker
  • DockerHub/Quay.io or any other public container image registry account
• Minikube or any Kubernetes cluster
  • kubectl CLI utility
• Go
• Operator SDK

My environment details

$ cat /etc/os-release | head -n 5
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

$ docker version
Client: Docker Engine - Community
 Version:           23.0.3

$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

$ go version
go version go1.20.4 linux/amd64

$ operator-sdk version
operator-sdk version: "v1.28.0", commit: "484013d1865c35df2bc5dfea0ab6ea6b434adefa", kubernetes version: "1.26.0", go version: "go1.19.6", GOOS: "linux", GOARCH: "amd64"

Initializing Operator Project

Kubebuilder provides a standardized way of creating Kubernetes API using Go. It generates the CRDs associated with the API in an organized file structure.

init subcommand from operator-sdk will generate custom resources, API, and controller manager for an operator based on the basic kubebuilder project layout.

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

Artifacts generated by the command

• YAML manifests for custom resources, controllers, Prometheus integration, and Role Based Access Control (RBAC) resources
• Scorecard tests
• Dockerfile for an image containing the binary of the controller manager
• go.mod containing the definition of the Go module with basic dependencies
• Makefile for building, distributing, and deploying the operator
• main.go contains logic for the controller manager
• PROJECT file containing the operator’s metadata (domain, project layout, name, repo, and version)
• README.md for documentation

--domain flag is used to specify a prefix of labels assigned to custom resources created by the operator.

--repo refers to the Go module to be used for the operator, it needs to be specified if the project directory is outside $GOPATH/src.

Other flags for the init subcommand:

• --project-name: To specify the name of the operator
• --project-version: To specify the operator version
• --owner: To specify the owner’s name
• --fetch-deps: To toggle dependency installation by OLM during deployment

Implementing API

Implementing an API for interacting with the custom resources created by the operator

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

After executing this command a new API resource will be added to the PROJECT file

resources:
- api:
    crdVersion: v1
    namespaced: true
  controller: true
  domain: example.com
  group: cache
  kind: Memcached
  path: github.com/example/memcached-operator/api/v1alpha1
  version: v1alpha1

The API and controller’s implementations will be stored in api/v1alpha1/memcached_types.go and controllers/memcached_controller.go respectively.

The creation of the controller could be toggled using the --controller flag. It is true by default. --group flag is used to specify the group of the API resources created. The value of the --version flag will specify the API version and --kind specifies the type of API to be implemented. --resource toggles the creation of API resource’s YAML manifests.

Building an Operator Image

The difference between an Operator Image and an Operator Bundle Image is that an operator image could be used to deploy an operator directly on the cluster as a Deployment (it contains the binary of the controller manager) whereas the bundle image stores the necessary metadata, custom resources, and APIs associated with the operator (also used for deployment, but through OLM).

make is an automation utility commonly used for processes like compiling/building applications. The Makefile in the operator project defines multiple targets like docker-build and docker-push for building and pushing the operator image to the registry respectively.

Keep in mind that you have to be logged in to the registry from your container engine before executing the following command

make docker-build docker-push IMG="docker.io/bovem/memcached-operator:v0.0.1"

For development purposes, if you want to test the bundle image outside the cluster you can use the following command

make install run

Building the Operator Bundle Image

Makefile target bundle will create a bundle/ directory in the project’s root containing manifests (CRDs) and metadata associated with the operator. A Containerfile named bundle.Dockerfile will be created as well.

Targets bundle-build and bundle-push will build and push the bundle image respectively.

make bundle bundle-build bundle-push BUNDLE_IMG="docker.io/bovem/memcached-operator-bundle:v0.0.1"

Files inside an Operator Bundle Image

manifests directory

Contains CRDs including the ClusterServiceVersion.

metadata directory

annotations.yaml in the metadata directory stores the operator’s metadata. This includes the path of the manifests and metadata directory, channels, etc. dependencies.yaml specifies the dependencies to be satisfied before installation of the operator is initiated. These could be dependencies on other operators or specific API/Custom Resources.

bundle.Dockerfile

The base image of the bundle is scratch. Inside the container image, the path to the manifest and metadata directory is test/ and test/metadata respectively.

Deploying an Operator

Direct Deployment using Operator Image

Makefile provides a target deploy for deploying the operator

make deploy IMG="docker.io/bovem/memcached-operator:v0.0.1"

After the deployment is completed successfully, we can create Memcached custom resource

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

Executing the make command with the undeploy target will uninstall the operator from the cluster

make undeploy

OLM Deployment using Operator Bundle Image

Before deploying the operator through the bundle image we have to make sure that OLM is installed on the cluster. To do that we can use the command

operator-sdk olm status

If OLM is missing it could be installed easily from the Operator SDK itself

operator-sdk olm install

Once OLM is installed, operator deployment is as easy as

operator-sdk run bundle docker.io/bovem/memcached-operator-bundle:v0.0.1

and uninstalling it

operator-sdk cleanup docker.io/bovem/memcached-operator-bundle:v0.0.1

Validating the Operator Bundle Image

Validating an OBI or the bundle directory ensures

• The manifests directory contains all the required CRDs including CSV.
• Data present in the files inside the manifests directory matches the provided data.
• The bundle format is valid.
• Permissions and Configurations of the operator are valid for an OLM-enabled cluster.
• Any additional validations defined with the operator are satisfied.

To validate a bundle the image or bundle directory path could be passed as an argument

operator-sdk bundle validate docker.io/bovem/memcached-operator-bundle:v0.0.1

or

operator-sdk bundle validate ./bundle

Testing Operator Bundle Image using Scorecard

Developers can define tests for their operator projects using the scorecard.

By default scorecard contains the following tests:

• Basic Test Suite (basic-check-spec-test): Tests for spec block in all CRDs.
• OLM Test Suite:
  • olm-bundle-validation-test: Validates bundle manifests
  • olm-crds-have-validation-test: All CRDs contain a validation section containing validation for each spec and status field.
  • olm-crds-have-resources-test: All CRDs have a resources section
  • olm-spec-descriptors-test: Every field in the CRD’s spec section has a descriptor listed in CSV
  • olm-status-descriptors-test: Every field in the CRD’s status section has a descriptor listed in CSV

Tests are defined in config/scorecard/bases as stages and executed on pods created on the cluster. At each stage, the tests are executed parallelly or sequentially. The result file is generated in JSON/XML/Text format.

scorecard subcommand is used to trigger test execution

operator-sdk scorecard docker.io/bovem/memcached-operator-bundle:v0.0.1

Resources

Operator Bundle
What is the Manager
What is Memcached
Kubebuilder
What’s in a basic project?
Go Operator Tutorial
operator-sdk init
operator-sdk create api
operator-sdk olm
operator-sdk run
operator-sdk bundle validate
operator-sdk scorecard