An Operator Bundle Image (OBI) is created to package custom resources and metadata associated with an operator. It’s like any other container image only difference is that it couldn’t be executed but it could be distributed through an OCI-compliant image registry.
Contents of a bundle image are:
- Kubernetes Custom Resource Definitions (CRDs)
- ClusterServiceVersion (CSV)
- Specification of operator’s dependencies
- Operator metadata like its name, version, channels, etc.
The control loops associated with the operator are defined in its Controller Manager. It is an executable that contains one or more custom controllers.
The Operator Lifecycle Manager (OLM) pulls the bundle image from a registry and installs it on the cluster.
Operator SDK is a project under Operator Framework that provides tools for building, testing, and packaging operators using the
Using Operator SDK we can create operators based on Ansible Roles, Go Programming Language, or Helm Charts.
Creating a Go-based Operator
In this article, I will create a Memcached operator using
Memcached is a memory caching system, often used by developers to increase the performance of API calls or databases. It stores data as key-value pairs in RAM. The Memcached operator will make the following changes to the cluster
- Create a
- Add a controller manager for
- Implement APIs to interact with custom resources.
- GNU Make (
- DockerHub/Quay.io or any other public container image registry account
- Minikube or any Kubernetes cluster
- Operator SDK
My environment details
$ cat /etc/os-release | head -n 5 PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy $ docker version Client: Docker Engine - Community Version: 23.0.3 $ minikube version minikube version: v1.30.1 commit: 08896fd1dc362c097c925146c4a0d0dac715ace0 $ go version go version go1.20.4 linux/amd64 $ operator-sdk version operator-sdk version: "v1.28.0", commit: "484013d1865c35df2bc5dfea0ab6ea6b434adefa", kubernetes version: "1.26.0", go version: "go1.19.6", GOOS: "linux", GOARCH: "amd64"
Initializing Operator Project
Kubebuilder provides a standardized way of creating Kubernetes API using Go. It generates the CRDs associated with the API in an organized file structure.
init subcommand from
operator-sdk will generate custom resources, API, and controller manager for an operator based on the basic kubebuilder project layout.
mkdir memcached-operator cd memcached-operator operator-sdk init --domain example.com --repo github.com/example/memcached-operator
Artifacts generated by the command
- YAML manifests for custom resources, controllers, Prometheus integration, and Role Based Access Control (RBAC) resources
- Scorecard tests
Dockerfilefor an image containing the binary of the controller manager
go.modcontaining the definition of the Go module with basic dependencies
Makefilefor building, distributing, and deploying the operator
main.gocontains logic for the controller manager
PROJECTfile containing the operator’s metadata (domain, project layout, name, repo, and version)
--domain flag is used to specify a prefix of labels assigned to custom resources created by the operator.
--repo refers to the Go module to be used for the operator, it needs to be specified if the project directory is outside
Other flags for the
--project-name: To specify the name of the operator
--project-version: To specify the operator version
--owner: To specify the owner’s name
--fetch-deps: To toggle dependency installation by OLM during deployment
Implementing an API for interacting with the custom resources created by the operator
operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller
After executing this command a new API resource will be added to the
resources: - api: crdVersion: v1 namespaced: true controller: true domain: example.com group: cache kind: Memcached path: github.com/example/memcached-operator/api/v1alpha1 version: v1alpha1
The API and controller’s implementations will be stored in
The creation of the controller could be toggled using the
--controller flag. It is
true by default.
--group flag is used to specify the group of the API resources created.
The value of the
--version flag will specify the API version and
--kind specifies the type of API to be implemented.
--resource toggles the creation of API resource’s YAML manifests.
Building an Operator Image
The difference between an Operator Image and an Operator Bundle Image is that an operator image could be used to deploy an operator directly on the cluster as a Deployment (it contains the binary of the controller manager) whereas the bundle image stores the necessary metadata, custom resources, and APIs associated with the operator (also used for deployment, but through OLM).
make is an automation utility commonly used for processes like compiling/building applications. The
Makefile in the operator project defines multiple targets like
docker-push for building and pushing the operator image to the registry respectively.
Keep in mind that you have to be logged in to the registry from your container engine before executing the following command
make docker-build docker-push IMG="docker.io/bovem/memcached-operator:v0.0.1"
For development purposes, if you want to test the bundle image outside the cluster you can use the following command
make install run
Building the Operator Bundle Image
bundle will create a
bundle/ directory in the project’s root containing manifests (CRDs) and metadata associated with the operator. A Containerfile named
bundle.Dockerfile will be created as well.
bundle-push will build and push the bundle image respectively.
make bundle bundle-build bundle-push BUNDLE_IMG="docker.io/bovem/memcached-operator-bundle:v0.0.1"
Files inside an Operator Bundle Image
Contains CRDs including the
annotations.yaml in the
metadata directory stores the operator’s metadata. This includes the path of the manifests and metadata directory, channels, etc.
dependencies.yaml specifies the dependencies to be satisfied before installation of the operator is initiated. These could be dependencies on other operators or specific API/Custom Resources.
The base image of the bundle is
scratch. Inside the container image, the path to the manifest and metadata directory is
Deploying an Operator
Direct Deployment using Operator Image
Makefile provides a target
deploy for deploying the operator
make deploy IMG="docker.io/bovem/memcached-operator:v0.0.1"
After the deployment is completed successfully, we can create
Memcached custom resource
kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
make command with the
undeploy target will uninstall the operator from the cluster
OLM Deployment using Operator Bundle Image
Before deploying the operator through the bundle image we have to make sure that OLM is installed on the cluster. To do that we can use the command
operator-sdk olm status
If OLM is missing it could be installed easily from the Operator SDK itself
operator-sdk olm install
Once OLM is installed, operator deployment is as easy as
operator-sdk run bundle docker.io/bovem/memcached-operator-bundle:v0.0.1
and uninstalling it
operator-sdk cleanup docker.io/bovem/memcached-operator-bundle:v0.0.1
Validating the Operator Bundle Image
Validating an OBI or the
bundle directory ensures
manifestsdirectory contains all the required CRDs including CSV.
- Data present in the files inside the
manifestsdirectory matches the provided data.
- The bundle format is valid.
- Permissions and Configurations of the operator are valid for an OLM-enabled cluster.
- Any additional validations defined with the operator are satisfied.
To validate a bundle the image or
bundle directory path could be passed as an argument
operator-sdk bundle validate docker.io/bovem/memcached-operator-bundle:v0.0.1
operator-sdk bundle validate ./bundle
Testing Operator Bundle Image using Scorecard
Developers can define tests for their operator projects using the scorecard.
By default scorecard contains the following tests:
- Basic Test Suite (
basic-check-spec-test): Tests for
specblock in all CRDs.
- OLM Test Suite:
olm-bundle-validation-test: Validates bundle manifests
olm-crds-have-validation-test: All CRDs contain a validation section containing validation for each spec and status field.
olm-crds-have-resources-test: All CRDs have a
olm-spec-descriptors-test: Every field in the CRD’s
specsection has a descriptor listed in CSV
olm-status-descriptors-test: Every field in the CRD’s
statussection has a descriptor listed in CSV
Tests are defined in
config/scorecard/bases as stages and executed on pods created on the cluster. At each stage, the tests are executed parallelly or sequentially. The result file is generated in JSON/XML/Text format.
scorecard subcommand is used to trigger test execution
operator-sdk scorecard docker.io/bovem/memcached-operator-bundle:v0.0.1
Thank you for taking the time to read this blog post! If you found this content valuable and would like to stay updated with my latest posts consider subscribing to my RSS Feed.
What is the Manager
What is Memcached
What’s in a basic project?
Go Operator Tutorial
operator-sdk create api
operator-sdk bundle validate