Traditionally, network functions were deployed on proprietary hardware with application-specific integrated circuits and installed on the telco’s premise (baremetal deployment). Such network functions are called Platform Network Functions (PNFs).

PNF deployments present the following challenges:

• Hardware for network functions could not be moved easily and redeployed.
• The deployment of PNFs required knowledge of proprietary hardware.
• Services deployed on top of PNFs are tightly coupled with the hardware. Customization, scaling, and configuration of services could require change/upgrade of hardware components.
• Access to PNFs is limited to the private cloud setup by the telco.

Virtualized Network Functions (VNFs)

The concept of Network Function Virtualization (NFV) was introduced in 2012 by a group of leading telcos. It proposed the transition of network functions from proprietary hardware to virtual machines. Individual network functions could be provisioned as a Virtual Machine on top of a hypervisor (Virtualized Deployment), such network functions are called Virtualized Network Functions (VNF).

Benefits of VNFs over PNFs

• VNFs could be deployed on any generic hardware as long as they are supported by the hypervisor. This eliminates the need for proprietary hardware for deployment.
• VNFs could be provisioned and moved easily.
• The scalability of the network is improved.
• Overall reduction in power and cooling requirements.

Other than VNFs the NFV architecture comprises MANO (Management, Automation, and Network Orchestration) and NFVi (Network Function Virtualization infrastructure).

MANO is a framework for the management and orchestration of all the resources and their lifecycle in NFV architecture. This includes the compute, storage, and network resources. NFVi refers to the infrastructure that provides the resources to VNFs an example could be OpenStack.

Software Defined Network (SDN) could be created on top of VNFs to further ease the management and configurability of the network.

Example of a VNF: Infoblox

Infoblox provides cloud networking solutions through products that provide a unified network view, global load balancing, reporting, analytics, etc.

The NFV provided by Infoblox has the following features:

• DNS, DHCP, and IP Address Management (IPAM) services.
• DNS Cache Acceleration Services (vDCA) improves the performance of DNS resolution across the network.
• Advanced DNS Protection (vADP) for networks deployed on OpenStack.
• Integration with other Infoblox products like Infoblox IPAM and Infoblox vDiscovery.

Cloud-Native Network Functions (CNFs)

Now, network functions are transitioning from virtual machines to containers (Containerized Deployment). This will provide further benefits over PNFs like

• Improved scalability and efficient usage of resources.
• Ability to use a mix of private, public, and hybrid cloud as containers could be deployed and migrated easily on any cloud environment.
• Edge devices could also be included in the network to improve its reach and latency to the end users.

Although CNIs could be used to create and configure network interfaces between containers, for telco’s use case a network function container could require network interfaces that could interact with the specialized hardware directly. In such cases, an operator could be defined with custom resources that could procure the hardware directly to the workloads (containers) as a network interface. CNFs themselves could also be packaged as Kubernetes operators.

Example of a CNF: Nokia Cloud Mobility Manager

Nokia’s Cloud Mobility Manager is a control plane network function for packet networks following the 3GPP (3rd Generation Partnership Project) standard.

It is designed to be deployed in a cloud-native environment as VNF (in OpenStack KVM or VMware vCloud environments) or as CNF (in OpenShift). It provides the following benefits

• Performance and scalability depending on the workloads
• Reliability and resiliency with fault monitoring, recovery, and overload control protection
• Multi-generation access (2G/3G/4G/5G)

Transition from VNFs to CNFs

The journey from VNFs to CNFs for telcos is not easy due to the following challenges:

• Virtualization of a workload is relatively easier compared to its containerization.
• A container isolates the process running inside it using namespaces. Some of the network function’s workloads might not run as expected inside an isolated namespace.
• Containers share the kernel with their host and traditionally the deployment of network functions requires kernel hacks.
• The architecture of telco networks is complex. Migrating it to a container-based architecture would not be easy.

In addition to this telcos work in a highly regulated environment where they are expected to have:

• Service Level Agreements (SLAs) for the availability of the network. Usually, this is 99.999% (~5.26 minutes in a year).
• Maintain the backward compatibility of the network with up to 50 years of legacy requirements.
• Tight regulation from governments and authorities.

To ease this transition we have CNF Test Suite and CNF Testbed

CNF Test Suite

The CNF Test Suite is defined by the Telecom User Group (TUG) and Cloud Native Network Function Working Group (CNF WG) for telcos to test their network functions deployed in a cloud-native environment for cloud-native principles:

• Configuration: The configuration for CNF should be defined in a declarative manner in ConfigMaps, Operators, or any other resource.
• Compatibility, Installability & Upgradability: CNFs should work with Certified Kubernetes Products. CNFs should be distributed & deployed as Containers, Operators, or Helm Charts.
• Microservice: CNFs should be built as microservices.
• State: The state of CNFs should be stored in a custom resource or a database like etcd.
• Reliability, Resilience & Availability: Failures are inevitable in a non-carrier grade cloud environment. So a CNF should be reliable, resilient, and available during such failures.
• Observability & Diagnostics: CNFs should provide endpoints and data to establish the observability stack (metrics, tracing, and logging).
• Security: The containers associated with CNFs should be isolated from their host and other containers. They should also be verified against common CVE and other vulnerabilities.

CNF Testbed

CNF Testbed provides reference code and test suites for benchmarking the performance and resiliency between network functions deployed as VNFs and CNFs.

It provisions the infrastructure for a Kubernetes and OpenStack cluster. After that, it deploys similar workloads on both environments to benchmark their performance using testing tools such as NFVbench.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

What is telco cloud?
What is NFV?
What is NFV MANO?
What are Virtual Network Functions (VNFs)?
What is Network Functions Virtualization (NFV)? | VMware Glossary
About virtual network functions over VPC
Infoblox Network Function Virtualization
VNF and CNF, what’s the difference?
Nokia Cloud Mobility Manager
Lessons Learnt Developing and Deploying a Cloud Native Network Function - Paul Graham
cncf/cnf-testsuite
Testing cloud native best practices with the CNF Test Suite
cncf/cnf-testbed
Cloud Native Network Function (CNF) Testbed - Taylor Carpenter & Denver Williams
K8s Best Practices for Telco Apps - Taylor Carpenter & Bill Mulligan
Building CNF applications with OpenShift Pipelines
Keynote: E2E 5G Cloud Native Network - Heather Kirksey, Azhar Sayeed & Fu Qiao
Understanding edge computing for telecommunications

A Container Storage Interface (CSI) plugin is implemented by the storage providers (Amazon, Google, IBM) as an interface to provision and mount volumes for workloads when requested by container orchestrators. The CSI plugin provisions the volume, procures it from the node hosting the container, and mounts it to the requesting container. It standardizes the process of allocating storage for containers between different orchestrators.

Volume Plugin

Before CSIs, the Volume Plugins were used to link the storage solutions and container orchestrators.

Limitations of Volume Plugins

• Volume Plugins were a part of Kubernetes core. Thus, their development and release were tightly coupled with Kubernetes.
• Instability and reliability issues with volume plugins could lead to the failure of core Kubernetes components.
• Every container orchestrator required a different implementation of the plugin.
• Orchestrators had to communicate with the storage solution throughout the lifecycle of the volume to avoid failures.
• Volume Plugins had to be open source.

Lifecycle of a Volume with Volume Plugins

Volume Plugins

CSI Specification

The CSI specification defines interfaces for implementing a plugin

• The plugin should work seamlessly across all container orchestrators.
• Ability to dynamically provision and unprovisioned volumes.
• Mounting, unmounting, attaching, and detaching volumes on the node.
• Ability to work with block and mountable volumes.
• Performing snapshots of volumes, that could be used for provisioning a new volume.

Architecture of a CSI

The orchestrator interacts with the CSI plugin through gRPCs, a cross-platform Remote Procedure Call framework introduced by Google.

Remote Procedure Calls (RPCs) are used in distributed systems to execute a procedure (subroutine) on a remote system. The CSI’s RPCs are idempotent i.e. a call should not change the result beyond its initial application.

RPC endpoints

There are three sets of gRPCs provided by the CSI Plugin: Identity, Controller, and Node service.

Identity Service RPCs

The Identity Service provides gPRC endpoints that allow an orchestrator to query the capabilities of the plugin, its health, and the metadata associated with the plugin.

gRPC endpoints provided by the identity service:

• GetPluginInfo: Returns the plugin metadata
• GetPluginCapabilities: Returns the plugin capabilities (fetched from controller service)
• Probe: Returns the health of the plugin

Controller Service RPCs

Controller Service groups all the gRPC endpoints responsible for

• Creating and deleting the volume
• Publishing and unpublishing the volume to the nodes
• Creating snapshots of the volumes

gRPC endpoints provided by the controller service:

• CreateVolume: Creates the volumes on the storage solution. The volume could be created as an empty new volume or from an existing snapshot or from an existing volume.
• DeleteVolume: Deletes a provisioned volume on the storage solution. Although the volume is deleted its snapshots are functional and available as sources for new volumes.
• ControllerPublishVolume: Performs the necessary operations to make the volume available to the requesting node.
• ControllerUnpublishVolume: Reverts the operations performed in ContainerPublishVolume.
• ValidateVolumeCapabilities: Verifies the provisioned volume for capabilities requested by the orchestrator.
• ListVolumes: Returns information about all the volumes.
• ControllerGetVolume: Called by the orchestrator to fetch the current information associated with a specific volume.
• GetCapacity: Queries the storage pool for its current capacity.
• ControllerGetCapabilities: Returns the details of the capabilities of the controller service.
• CreateSnapshot: Creates a snapshot of an existing volume.
• DeleteSnapshot: Deletes the volume snapshot.
• ListSnapshots: Returns details of all snapshots.
• ControllerExpandVolume: Using this RPC the orchestrator can request for expansion in the size of a volume.

Node Service RPCs

These RPCs are called from the node where the volume is requested. They are responsible for the actions performed on the volume after it is mounted on the node.

gRPC endpoints provided by the controller service:

• NodeStageVolume: After the workload (container) is scheduled on the orchestrator it calls NodeStageVolume on the volume provided by ContainerPublishVolume. The volume is mounted on the node on stage_path temporarily.
• NodeUnstageVolume: Reverts the actions performed by NodeStageVolume.
• NodePublishVolume: After NodeStageVolume is called the NodePublishVolume mounts the volume on target_path to be consumed by the workload.
• NodeUnpublishVolume: Reverts the actions performed by NodeUnpublishVolume.
• NodeGetVolumeStats: Returns the capacity statistics for a volume.
• NodeGetCapabilities: Called by the orchestrator to check the capabilities of the node services.
• NodeGetInfo: Returns the information for the node where the volume has to be mounted.
• NodeExpandVolume: Expands the volume mounted on the node. This call should be performed after NodeStageVolume and NodePublishVolume.

Implementation of a CSI

CSI specification provides examples of different implementations for plugins.

Centralized Controller Plugin

Centralized Controller CSI Plugin

The controller service RPCs could be packaged in a Controller Plugin located on the master node of the orchestrators. The node service RPCs are bundled as Node Plugin located on each worker node (or any node that could request volumes).

Headless Plugin

Headless CSI Plugin

The Controller and Node Plugin could be placed together inside the worker node.

Controller Plugin

Controller CSI Plugin

CSI could be deployed with just the Controller plugin on the worker node that provides both controller and node services.

Node Plugin

Node CSI Plugin

CSI could also be deployed with just a Node plugin on the worker node providing just the node services.

Volume lifecycle with CSI

Lifecycle of a Volume managed by CSI Plugin

Mounting a Volume on a Container

1. A Workload is scheduled by the container orchestrator.
2. The orchestrator calls CreateVolume RPC to provision a volume on the storage solution.
3. Once the volume is provisioned on the storage solution the orchestrator calls ContainerPublishVolume to make it available to the node.
4. After the volume is available to the node, the orchestrator calls NodeStageVolume to mount the volume on the node at stage_path.
5. Finally, the orchestrator calls NodePublishVolume to mount the volume on target_path. Thus making it available for the container.

Deleting a Volume

1. The volume to be deleted is unpublished by the orchestrator by calling NodeUnpublishVolume RPC.
2. Once unpublished from the container the volume is unstaged by NodeUnstageVolume.
3. The unstaged volume is then unpublished from the node by ContainerUnpublishVolume.
4. Finally, the unpublished volume is deleted by DeleteVolume RPC.

Example of a CSI: Intel Optane Persistent Memory PMEM-CSI Storage Driver

Intel Optane is a tiered persistent memory technology. It is used as a high-performance storage for desktop (improving system boot time) and enterprise (CAD, Modeling, Media editing) workloads.

The PMEM-CSI storage driver is provided by Intel to configure the Optane Persistent Memory as a storage solution for Kubernetes. The persistent volumes could be provisioned dynamically on the Optane memory and mounted on the containers present on the cluster.

Using this driver the developers can take advantage of Optane memory in their cache data store like Memcached Operator.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Container storage 101: What is CSI and how does it work? | Computer Weekly
Understanding the Container Storage Interface (CSI) | by anoop vijayan maniankara
Container Storage Interface: Present and Future - Jie Yu, Mesosphere, Inc.
Container Storage Interface (CSI) for Kubernetes
Introduction - Kubernetes CSI Developer Documentation
Container Storage Interface (CSI) for Kubernetes GA
container-storage-interface/spec
Persistent Memory
intel/pmem-csi
Persistent Memory (PMEM) CSI
Redis-pmem operator
memcached with PMEM

The Container Network Interface (CNI) project provides specifications and libraries for implementing a plugin-based solution for managing network interfaces for containers. The runtime executes the CNI plugins provided as binary executable files.

A network configuration is passed to the runtime as a JSON file. It contains the details of the CNI plugins and the network interfaces to be configured.

Example of a network configuration file from CNI specification

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

{
  "cniVersion": "1.0.0",
  "name": "dbnet",
  "plugins": [
    {
      "type": "bridge",
      // plugin specific parameters
      "bridge": "cni0",
      "keyA": ["some more", "plugin specific", "configuration"],
      
      "ipam": {
        "type": "host-local",
        // ipam specific
        "subnet": "10.1.0.0/16",
        "gateway": "10.1.0.1",
        "routes": [
            {"dst": "0.0.0.0/0"}
        ]
      },
      "dns": {
        "nameservers": [ "10.1.0.1" ]
      }
    },
    {
      "type": "tuning",
      "capabilities": {
        "mac": true
      },
      "sysctl": {
        "net.core.somaxconn": "500"
      }
    },
    {
        "type": "portmap",
        "capabilities": {"portMappings": true}
    }
  ]
}

Runtime executes the CNI plugins with the following environment variables

• CNI_COMMAND: Operation for CNI (ADD/DELETE/CHECK/VERSION)
• CNI_ARGS: Arguments for CNI
• CNI_PATH: Path to CNI binaries.
• CNI_IFNAME: Name of the network interface to be configured
• CNI_NETNS: Path to the container’s network namespace
• CNI_CONTAINERID: Container ID assigned by the runtime

CNI Operations

According to the CNI specification, a CNI plugin should have ADD, DELETE, CHECK, and VERSION operations.

ADD

If the environment variable passed to the runtime by the CNI plugin is CNI_COMMAND=ADD then it will create or configure the network interfaces specified in the configuration.

Execution flow of ADD operation

The interfaces will be created or configured in the order specified in the plugins key inside the network configuration. The result from the previous plugin execution will be passed to the next plugin and the final result will be stored with runtime.

So from the example above the bridge plugin will be executed first, followed by the tuning plugin, and finally the portmap plugin.

DELETE

The DELETE operation will undo operations performed by the ADD, deleting the created interfaces and reverting the configurations. It executes CNI plugins in the reverse order of their specification in network configuration. So the changes performed by the portmap plugin will be reverted first then the tuning plugin and lastly the bridge plugin.

Execution flow of DELETE operation

It has to be preceded by an ADD operation because it takes the final result stored by the last ADD operation as input from container runtime.

CHECK

CHECK operation probes the container and network interfaces for their health and current status. The container runtime will go through the changes performed by each CNI plugin and validate its functionality. Similar to the DELETE operation it also uses the result from the previous ADD operation as input from container runtime.

VERSION

The VERSION operation returns the version of the CNI plugin.

Reference CNI Plugins

Network Interface Plugins

• bridge: Creates a virtual switch that connects all the containers on a host. An IP address is assigned to each container by the bridge.
• macvlan: Creates a sub-interface from the host interface. Each virtual interface connected to macvlan has its own MAC address.

• ipvlan: Similar to macvlan but all connected devices share the same MAC address. Before transmitting a packet the kernel driver inspects the IP address of the destination virtual interface.
• ptp: Creates a point-to-point link between the container and its host by creating a Virtual Ethernet (veth) pair where one end resides inside the container and the other on the host.

• host-device: An existing device could be moved from the host’s network namespace to the container’s network namespace.
• vlan: Creates a Virtual Local Area Network (VLAN) interface between the host and the container.

Windows Plugins

These plugins are created specifically for Microsoft Windows containers.

• win-bridge: All containers will be connected to a bridge network that has an endpoint on the host’s network namespace.
• win-overlay: An Overlay network allows containers from different hosts to act as if they are on the same host. Using the win-overlay plugin all containers on a host could be connected to an Overlay network.

IP Address Allocation Plugin

• dhcp: The IP leased by the DHCP server has to be renewed periodically. The dhcp plugin creates a daemon separate from the container to perform these requests.
• host-local: Allocates IPv4 and IPv6 addresses out of a provided range of addresses.
• static: Allocates static IPv4 and IPv6 addresses to the containers.

Other Plugins

• tuning: Used with other plugins to change system controls or interface attributes (like MTU and MAC address)
• portmap: Forwards traffic from the host’s network ports to the container’s network ports.
• bandwidth: Configures traffic bucket filter (tbf) to limit bandwidth on ingress and egress traffic.
• sbr: Configures source based routing (sbr) where the application (source) decides the correct network interface for outgoing traffic.
• firewall: Creates firewall rules to allow traffic to the container’s IP address.

Example of 3rd Party CNI Plugin: Weave

The Weave is a CNI plugin created by Weaveworks that creates an overlay mesh network that allows Dockers containers running on different hosts to connect.

Features of Weave:

• Containers from different hosts and data centers will act as if they are on the same network without any additional configuration.
• Application containers (running on any host) could be exposed to the outside network.
• It can create networks between legacy systems and containers.
• Traffic could be encrypted.
• Supports Amazon ECS, Apache Mesos, Kubernetes, etc.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Kubernetes CNI Explained
Introduction to CNI, the Container Network Interface Project - Bryan Boreham & Dan Williams
Deep Dive: CNI - Bryan Boreham, Weaveworks & Dan Williams, Red Hat
Kubernetes Networking: The Complete Guide | Tigera
Container Networking: What You Should Know
cni/SPEC.md at main · containernetworking/cni · GitHub
Kubernetes Networking: How to Write a CNI Plugin From Scratch - Eran Yanay, Twistlock
A brief overview of the Container Network Interface (CNI) in Kubernetes | Enable Sysadmin
CNI Plugins Overview
Network Plugins | Kubernetes
containernetworking/plugins
Introduction to Weave Net

Helm is a package manager for Kubernetes. It is implemented in Go and installed as a binary helm. It interacts with the Kubernetes cluster using Kubernetes API.

Charts

Helm distributes Kubernetes-based applications in a format called Chart.
Charts can deploy all kinds of Kubernetes resources such as Deployments, Pods, Services, Persistent Volumes, etc.

Structure of a Helm Chart

Chart’s directory contains the following files and sub-directories:

• Chart.yaml contains the chart’s metadata
• charts/ directory contains dependencies (other charts)
• crds/ directory contains the custom resources required by the chart
• templates/ stores the templates for Kubernetes resources
• values.yaml stores the configuration values for templates

Chart.yaml

The chart’s metadata includes

• Chart’s name
• Application’s version
• Chart’s type (application or library)
  • application charts package Kubernetes applications that could be installed directly on the cluster.
  • library charts are used by other charts to extend their functionality.
• Compatible Kubernetes versions
• Keywords related to the chart, its homepage, and link to the chart’s source code
• Dependencies
• Contact information of chart’s maintainers

templates and values

Resource templates are based on the Go template convention. Example of a Deployment resource template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: {{ .Values.replicaNum }}
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: {{ .Values.containerName }}
        image: {{ .Values.containerImage }}
        ports:
        - containerPort: {{ .Values.containerPort }}

The values to be substituted in this template will be provided from values.yaml file in the chart’s root directory. Example of a values.yaml file for the template

1
2
3
4

replicaNum: "3"
containerName: "nginx-test"
containerImage: "docker.io/library/nginx:1.14.2"
containerPort: "80"

The structure of the values.yaml file is defined in values.schema.json. Chart developers can also define the datatype of each value. Example of a values.schema.json file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

{
  "$schema": "https://json-schema.org/draft-07/schema#",
  "properties": {
    "replicaNum": {
      "type": "string"
    },
    "containerName": {
      "type": "string"
    },
    "containerImage": {
      "type": "string"
    },
    "containerPort": {
      "type": "string"
    }
  },
  "required": [
    "replicaNum",
    "containerName",
    "containerImage",
    "containerPort",
  ],
  "title": "Values",
  "type": "object"
}

crds

Custom Resources (CRs) are installed before any other resource is created by the helm chart. Some limitations imposed on the CRs created by the helm charts:

• YAML manifests of CRs could not be templated for values.
• CRs would not be reinstalled with a chart.
• Upgrades and rollbacks do not impact the CRs installed by the chart.
• Helm doesn’t delete CRs upon uninstallation.

Chart Repository

Helm charts are distributed through HTTP servers called Repositories. The metadata of available charts in the repository is stored in index.yaml.

A repository could be added using

1

helm repo add jenkins-x https://storage.googleapis.com/chartmuseum.jenkins-x.io 

Here jenkins-x is the name of the helm chart repository and https://storage.googleapis.com/chartmuseum.jenkins-x.io is the link to the repository.

Once a repository is added, it is recommended to update the information of available charts in the local cache.

1

helm repo update

The repo command also provides list and remove for listing and removing repositories respectively.

Searching Helm Charts in Repositories and ArtifactHub

To search a helm chart within the added repositories the search command could be used

1

helm search repo tekton

ArtifactHub is a web platform that provides access to multiple Kubernetes-based applications including helm charts. helm search can also be used for searching packages in ArtifactHub

1

helm search hub tekton

Installing Helm Charts

Each instance of a helm chart installation is called a Release.

A release will be created after executing the following command.

1

helm install tekton-pipelines jenkins-x/tekton

Here, tekton-pipelines is the name of the release and jenkins-x/tekton is the name of the helm chart to be installed.

To install a local helm chart, the chart name will be substituted with the path to the chart’s directory or archive

1

helm install test-helm-chart ./test-helm-chart

Configuration values are passed from a file using the --values flag.

During development, if developers want to inspect the manifests created during release and avoid installing the chart on a cluster then they can run the install command with --debug and --dry-run flags.

Upgrading and Rolling Back Releases

Helm performs the least invasive upgrades on chart releases i.e. updates only the changed resources. The upgrade command is used to upgrade a chart release

1

helm upgrade -f newValues.yaml test-helm-chart ./test-helm-chart

Release test-helm-chart will be updated with values in newValues.yaml.

On every upgrade, the chart release’s revision number will be incremented by 1. To view the history of releases with their respective revision numbers

1

helm history test-helm-chart

For some reason, if the chart upgrade doesn’t go as planned it could be rolled back to a previous revision number.

1

helm rollback test-helm-chart 1

will rollback the test-helm-chart release to its first revision.

Uninstalling Helm Chart

The uninstall command will remove all the resources created by the helm chart release from the Kubernetes cluster (except CRs). It will also wipe the history of the release, to retain history the --keep-history flag could be used with the command.

1

helm uninstall test-helm-chart

Example of a Helm Chart: Grafana Helm Chart

Grafana is an analytics tool that provides utilities for creating dashboards with graphs and alerts. Using Grafana developers can create dashboards for monitoring services, presenting business metrics, etc.

Grafana’s Helm Chart will perform the following changes to the Kubernetes cluster

• Create pods and other workload resources required by Grafana
• Create services and security policies
• Create ConfigMaps and Secrets for pods
• Create PersistentVolumes
• Create RBAC resources

Creating Helm Charts

The boilerplate code for helm charts is generated using create command.

1

helm create test-helm-chart

will create a directory test-helm-chart containing basic chart resources (Chart.yaml, charts/, templates/, values.yaml, etc.).

Testing a Helm Chart

Developers can define tests for their helm charts in the templates directory (or templates/tests for better organization).

Tests are defined as Kubernetes Pods with the following annotation.

1
2

  annotations:
    "helm.sh/hook": test

Test pods should finish with exit code 0 to be marked successful.

To execute tests defined with a release

1

helm test test-helm-chart

Packaging Helm Charts

Charts are packaged as GZIP compressed archives with extension tgz. The package subcommand will create the archive from the chart’s directory

1

helm package ./test-helm-chart

The archive could be signed (--sign) by its developer using a key passed with a --key flag.

Distributing Helm Charts

Chart Repositories

The chart repository server contains

• an index.yaml containing chart’s metadata
• charts packaged as archives (for example test-helm-chart-0.0.1.tgz)
• chart’s provenance files (test-helm-chart-0.0.1.tgz.prov) containing integrity and validation data

A publicly available helm chart repository could be created using GitHub pages. Public helm chart repositories could also be added to ArtifactHub.

OCI-based registries

Some OCI-based registries like DockerHub support helm chart distribution. To distribute a chart through DockerHub the user has to login to the registry through helm

1

helm registry login -u bovem docker.io

Here bovem is the DockerHub username and docker.io is the link to the DockerHub registry. To push the chart to DockerHub it has to be packaged as an archive

1

helm push test-helm-chart-0.0.1.tgz oci://docker.io/bovem

Similarly, a specific version of the chart could be pulled from DockerHub using the pull command.

1

helm pull oci://docker.io/bovem/test-helm-chart --version 0.0.1

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Using the DNF software package manager
Helm
Helm Architecture
Installing Helm
Charts
The Chart Repository Guide
Tekton’s Helm Chart
ArtifactHub
Grafana’s Helm Chart

Contents of a bundle image are:

• Kubernetes Custom Resource Definitions (CRDs)
• ClusterServiceVersion (CSV)
• Specification of operator’s dependencies
• Operator metadata like its name, version, channels, etc.

The control loops associated with the operator are defined in its Controller Manager. It is an executable that contains one or more custom controllers.

The Operator Lifecycle Manager (OLM) pulls the bundle image from a registry and installs it on the cluster.

Operator SDK

Operator SDK is a project under Operator Framework that provides tools for building, testing, and packaging operators using the operator-sdk utility.

Using Operator SDK we can create operators based on Ansible Roles, Go Programming Language, or Helm Charts.

Creating a Go-based Operator

In this article, I will create a Memcached operator using operator-sdk CLI.

Memcached is a memory caching system, often used by developers to increase the performance of API calls or databases. It stores data as key-value pairs in RAM. The Memcached operator will make the following changes to the cluster

• Create a Memcached custom resource.
• Add a controller manager for Memcached resources.
• Implement APIs to interact with custom resources.

Prerequisites:

• GNU Make (make)
• Docker
  • DockerHub/Quay.io or any other public container image registry account
• Minikube or any Kubernetes cluster
  • kubectl CLI utility
• Go
• Operator SDK

My environment details

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

$ cat /etc/os-release | head -n 5
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

$ docker version
Client: Docker Engine - Community
 Version:           23.0.3

$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

$ go version
go version go1.20.4 linux/amd64

$ operator-sdk version
operator-sdk version: "v1.28.0", commit: "484013d1865c35df2bc5dfea0ab6ea6b434adefa", kubernetes version: "1.26.0", go version: "go1.19.6", GOOS: "linux", GOARCH: "amd64"

Initializing Operator Project

Kubebuilder provides a standardized way of creating Kubernetes API using Go. It generates the CRDs associated with the API in an organized file structure.

init subcommand from operator-sdk will generate custom resources, API, and controller manager for an operator based on the basic kubebuilder project layout.

1
2
3

mkdir memcached-operator
cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

Artifacts generated by the command

• YAML manifests for custom resources, controllers, Prometheus integration, and Role Based Access Control (RBAC) resources
• Scorecard tests
• Dockerfile for an image containing the binary of the controller manager
• go.mod containing the definition of the Go module with basic dependencies
• Makefile for building, distributing, and deploying the operator
• main.go contains logic for the controller manager
• PROJECT file containing the operator’s metadata (domain, project layout, name, repo, and version)
• README.md for documentation

--domain flag is used to specify a prefix of labels assigned to custom resources created by the operator.

--repo refers to the Go module to be used for the operator, it needs to be specified if the project directory is outside $GOPATH/src.

Other flags for the init subcommand:

• --project-name: To specify the name of the operator
• --project-version: To specify the operator version
• --owner: To specify the owner’s name
• --fetch-deps: To toggle dependency installation by OLM during deployment

Implementing API

Implementing an API for interacting with the custom resources created by the operator

1

operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

After executing this command a new API resource will be added to the PROJECT file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

resources:
- api:
    crdVersion: v1
    namespaced: true
  controller: true
  domain: example.com
  group: cache
  kind: Memcached
  path: github.com/example/memcached-operator/api/v1alpha1
  version: v1alpha1

The API and controller’s implementations will be stored in api/v1alpha1/memcached_types.go and controllers/memcached_controller.go respectively.

The creation of the controller could be toggled using the --controller flag. It is true by default. --group flag is used to specify the group of the API resources created. The value of the --version flag will specify the API version and --kind specifies the type of API to be implemented. --resource toggles the creation of API resource’s YAML manifests.

Building an Operator Image

The difference between an Operator Image and an Operator Bundle Image is that an operator image could be used to deploy an operator directly on the cluster as a Deployment (it contains the binary of the controller manager) whereas the bundle image stores the necessary metadata, custom resources, and APIs associated with the operator (also used for deployment, but through OLM).

make is an automation utility commonly used for processes like compiling/building applications. The Makefile in the operator project defines multiple targets like docker-build and docker-push for building and pushing the operator image to the registry respectively.

Keep in mind that you have to be logged in to the registry from your container engine before executing the following command

1

make docker-build docker-push IMG="docker.io/bovem/memcached-operator:v0.0.1"

For development purposes, if you want to test the bundle image outside the cluster you can use the following command

1

make install run

Building the Operator Bundle Image

Makefile target bundle will create a bundle/ directory in the project’s root containing manifests (CRDs) and metadata associated with the operator. A Containerfile named bundle.Dockerfile will be created as well.

Targets bundle-build and bundle-push will build and push the bundle image respectively.

1

make bundle bundle-build bundle-push BUNDLE_IMG="docker.io/bovem/memcached-operator-bundle:v0.0.1"

Files inside an Operator Bundle Image

manifests directory

Contains CRDs including the ClusterServiceVersion.

metadata directory

annotations.yaml in the metadata directory stores the operator’s metadata. This includes the path of the manifests and metadata directory, channels, etc. dependencies.yaml specifies the dependencies to be satisfied before installation of the operator is initiated. These could be dependencies on other operators or specific API/Custom Resources.

bundle.Dockerfile

The base image of the bundle is scratch. Inside the container image, the path to the manifest and metadata directory is test/ and test/metadata respectively.

Deploying an Operator

Direct Deployment using Operator Image

Makefile provides a target deploy for deploying the operator

1

make deploy IMG="docker.io/bovem/memcached-operator:v0.0.1"

After the deployment is completed successfully, we can create Memcached custom resource

1

kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

Executing the make command with the undeploy target will uninstall the operator from the cluster

1

make undeploy

OLM Deployment using Operator Bundle Image

Before deploying the operator through the bundle image we have to make sure that OLM is installed on the cluster. To do that we can use the command

1

operator-sdk olm status

If OLM is missing it could be installed easily from the Operator SDK itself

1

operator-sdk olm install

Once OLM is installed, operator deployment is as easy as

1

operator-sdk run bundle docker.io/bovem/memcached-operator-bundle:v0.0.1

and uninstalling it

1

operator-sdk cleanup docker.io/bovem/memcached-operator-bundle:v0.0.1

Validating the Operator Bundle Image

Validating an OBI or the bundle directory ensures

• The manifests directory contains all the required CRDs including CSV.
• Data present in the files inside the manifests directory matches the provided data.
• The bundle format is valid.
• Permissions and Configurations of the operator are valid for an OLM-enabled cluster.
• Any additional validations defined with the operator are satisfied.

To validate a bundle the image or bundle directory path could be passed as an argument

1

operator-sdk bundle validate docker.io/bovem/memcached-operator-bundle:v0.0.1

or

1

operator-sdk bundle validate ./bundle

Testing Operator Bundle Image using Scorecard

Developers can define tests for their operator projects using the scorecard.

By default scorecard contains the following tests:

• Basic Test Suite (basic-check-spec-test): Tests for spec block in all CRDs.
• OLM Test Suite:
  • olm-bundle-validation-test: Validates bundle manifests
  • olm-crds-have-validation-test: All CRDs contain a validation section containing validation for each spec and status field.
  • olm-crds-have-resources-test: All CRDs have a resources section
  • olm-spec-descriptors-test: Every field in the CRD’s spec section has a descriptor listed in CSV
  • olm-status-descriptors-test: Every field in the CRD’s status section has a descriptor listed in CSV

Tests are defined in config/scorecard/bases as stages and executed on pods created on the cluster. At each stage, the tests are executed parallelly or sequentially. The result file is generated in JSON/XML/Text format.

scorecard subcommand is used to trigger test execution

1

operator-sdk scorecard docker.io/bovem/memcached-operator-bundle:v0.0.1

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Operator Bundle
What is the Manager
What is Memcached
Kubebuilder
What’s in a basic project?
Go Operator Tutorial
operator-sdk init
operator-sdk create api
operator-sdk olm
operator-sdk run
operator-sdk bundle validate
operator-sdk scorecard

OperatorHub

The OperatorHub is an interface for searching and installing operators. It has the following categories of operators:

• Red Hat Operators: Operators developed and supported by Red Hat. Example: Red Hat Quay Operator
• Certified Operators: Operators listed by Red Hat’s Independent Software Vendors (ISVs). Example: CockroachDB Operator
• Red Hat Marketplace Operators: Applications purchased from Red Hat Marketplace available as Operators. Example: Dynatrace Operator
• Community Operators: Default catalog of Operators maintained by their communities. Example: Infispan Operator

OperatorHub fetches the catalog data from an operator installed by default on all clusters called Marketplace Operator.

The value of the field disableAllDefaultSources needs to be false in the CR called cluster to view the operator’s catalog.

Installing an Operator from OperatorHub

OperatorHub is accessible from Operators -> OperatorHub on the web UI of the OpenShift cluster.

OperatorHub UI

To install an operator you can just click the Install button and it will present options to select an update channel (version of the operator), installation mode (specific or all namespaces), and update approval (automatic or manual).

Operator Installation

Once installed the operator should be visible in Operators -> Installed Operators.

Installed Operators

Operator Framework

Operator Framework is a set of developer tools for operators, it includes:

• Operator Lifecycle Manager (OLM): Handles installation, update, and management of operators on the cluster.
• Operator SDK: SDK for building operators using Ansible, Helm Chart, or Go.
• Operator Registry: Registry of operators that could be hosted on an OpenShift cluster.

To explore other projects under Operator Framework you can check their GitHub.

Operator Lifecycle Manager (OLM)

The Operator Lifecycle Manager (OLM) is installed by default on OpenShift clusters. It ensures:

• The dependencies of an operator (including other operators) are satisfied before its installation.
• Installed operators are up-to-date.
• Availability of the operators to the users of the cluster.
• There are no conflicts between multiple operator installations.
• Installed operators are accessible through UI, API, and CLI.

The OLM itself consists of two operators: OLM Operator and Catalog Operator.

OLM Operator

The OLM Operator deploys the resources defined in ClusterServiceVersion present on the namespace.

ClusterServiceVersion

The ClusterServiceVersion or CSV represents a specific version of an operator. It contains operator metadata such as its name, description, version, details of the maintainer, installation strategy, and APIs provided by the operator.

OperatorGroup

OLM has cluster-admin privileges. Operators could request some of these cluster-admin permissions in their CSV. By creating an OperatorGroup the cluster administrator can take control of the permissions granted to the operators or limit them to one or more namespaces.

OperatorCondition

Operators can modify the OLM’s management strategy by stating their conditions in OperatorCondition. For example, an operator could set the status for the Upgradable property to False if its installation has to be frozen on a specific version.

Catalog Operator

Catalog Operator installs the operator based on InstallPlan. It also updates the operator if a new version is available on its Subscription.

CatalogSource

CatalogSource is a metadata store for discovering and installing operators.

Subscription

Subscription specifies the CatalogSource to be referenced by OLM during installation and updates. The channel (alpha/beta/stable) of the operator could also be specified in the Subscription.

InstallPlan

An InstallPlan defines the operator’s custom resources to be installed.

Installing a Specific Version of the Operator

By default, OpenShift installs the latest available version of an operator in the CatalogSource. If you want to change this behavior and install a specific version of the operator you have to edit its Subscription and specify the version to be installed in startingCSV and set installPlanApproval to Manual. Note that if the required version is not available on the default channel then you have to change it as well.

Example of a Subscription for installing v1.9.0 of OpenShift Pipelines Operator

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-pipelines-operator-rh
  namespace: openshift-operators
spec:
  channel: pipelines-1.9
  installPlanApproval: Manual
  name: openshift-pipelines-operator-rh
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: openshift-pipelines-operator-rh.v1.9.0

If an update is available in the future the operator’s InstallPlan has to be approved manually, this will prevent automatic updates.

Uninstalling an Operator

The option to uninstall an operator is present in its Actions on UI.

To uninstall an operator from CLI, you have to remove its Subscription as well as its ClusterServiceVersion using the delete subcommand on the specific resources.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Listing available subscriptions
$ oc get subscriptions -n openshift-operators
NAME                              PACKAGE                           SOURCE
openshift-pipelines-operator-rh   openshift-pipelines-operator-rh   redhat-operators

# Deleting subscription for OpenShift Pipelines Operator
$ oc delete subscription/openshift-pipelines-operator-rh -n openshift-operators

# Listing available clusterserviceversions
$ oc get clusterserviceversions -n openshift-operators
NAME                                      DISPLAY                       VERSION
openshift-pipelines-operator-rh.v1.10.0   Red Hat OpenShift Pipelines   1.10.0

# Deleting clusterserviceversion for OpenShift Pipelines Operator
$ oc delete csv/openshift-pipelines-operator-rh.v1.10.0 -n openshift-operators

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

OperatorHub
Operator Lifecycle Manager (OLM)
ClusterServiceVersion
CatalogSource
OperatorCondition
Subscription
InstallPlan
OperatorGroup
OLM Architecture
OLM Workflow

An Operator consists of:

• Custom Resources (CRs) required by the application
• Custom controller for managing these CRs

Control Loop

A control loop is an infinite loop for monitoring the state of a system. If the desired state of the system is different from its current state then the control loop makes changes to the system until it reaches its desired state.

An example of a control loop is a thermostat that monitors the current room temperature on frequent time intervals and if the desired temperature is different from the current temperature then it increases or decreases the cooling.

Kubernetes Controller

The Kubernetes controller functions as a control loop for the resources on the cluster. It performs necessary changes to the resources using the Kubernetes API such that they reach their desired state.

• If a deployment needs to be scaled up or down (due to a manual instruction from the cluster administrator or an increase/decrease in its resource utilization) then the controller will deploy/destroy pods in the deployment.
• If an application has to be updated then the controller will make sure that eventually all the running containers are replaced with the latest version of the application (based on the rollout strategy defined in deployment).
• If updated pods aren’t healthy then the controller will roll back the changes until the deployment is stable (based on deployment and rollback strategies).

The Kubernetes Controller monitors and maintains the state of resources on the cluster

Reconcile Loop and Reconciliation

The loop that constantly monitors the state of resources on the cluster is called reconcile loop and the process of making changes to the cluster to match the desired state is called reconciliation.

Human Operator

Applications using CRs with developer-defined strategies for scaling, updates, rollback, etc. like a database application that maintains two instances: read-only and read-write could not be managed by the default Kubernetes Controller. Such applications will require a human operator to administer their deployment on the cluster. They will be responsible for:

• Installation and configuration of the application based on the cluster’s environment.
• Scaling strategies for the CRs based on current resource utilization, application’s business logic, and other constraints.
• Rollout and rollback of updates to the application without affecting the end users.
• Fixing application issues during runtime.

Kubernetes Operators

A Kubernetes Operator aims to automate the functions of a Human Operator automating processes like

• installation
• configuration and reconfiguration
• update
• backup
• recovery

By packaging an application as an Operator its developer could ensure that

• The deployment process is standardized in all environments.
• The application could be scaled up or down based on the logic defined by the developer.
• The application could fix itself upon encountering any known issues after deployment. Even rolling back to a previous state if required.
• New changes are rolled out strategically without breaking existing functionality.

Capability Levels

Based on the level of maturity or capability of automation provided by an Operator it could be classified into 5 levels:

• LEVEL 1: Automated installation and initial configuration of the application.
• LEVEL 2: Patching and upgrading the application.
• LEVEL 3: Manage the complete application lifecycle i.e. backup, failure, and recovery.
• LEVEL 4: Provide application metrics, logs, and workload analysis
• LEVEL 5: Scaling (Horizontal and Vertical), reconfiguration of application, tuning, and abnormality detection.

A Level 3 operator could manage the whole application cycle while also performing Level 1 (installation & configuration) and Level 2 (patches & upgrades) automation.

Just like the Kubernetes controller, a Kubernetes Operator monitors and maintains the state of Custom Resources of an applications

Custom Resources and Controller

A Custom Controller created by the Operator manages the state of the cluster’s resources including Custom Resources (CRs) using Kubernetes API just like the default Kubernetes Controller.

It could be limited to a single namespace or deployed cluster-wide. It is recommended to have a single custom controller for a CR as multiple controllers can make changes simultaneously overwriting each other.

Example of a Kubernetes Operator: MariaDB Operator

MariaDB Operator performs the following changes on the Kubernetes Cluster automates:

• Deployment of MariaDB server on Kubernetes with default/configured version.
• Creation of PersistentVolumeClaims (PVCs) and a custom database with a credential set.
• Future upgrades for MariaDB without affecting current services.
• Database backups
• Metrics for database

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Operator Pattern
Kubernetes Operators
What is a Kubernetes Operator
Operator Capability Levels

The build steps are written in a plaintext file called Containerfile and parsed by container engines (or buildah) during the build process.

1
2
3
4
5
6
7
8

# Containerfile
FROM node:18-alpine
LABEL version="1.0" 
WORKDIR /app
COPY . .
RUN yarn install --production
CMD ["node", "src/index.js"]
EXPOSE 3000

Containerfile Instructions

Steps inside the containerfile are defined using instructions such as FROM, RUN, ADD, COPY, etc. Container Engines go through the containerfile line-by-line and perform each step, stacking a new image layer on top of the previous one.

FROM

FROM instruction specifies the base layer container image. The base layer image could be chosen based on OS preference (like Fedora, Kali, etc.) or programming language preference (like gcc, python, etc.), or any other preference. It is mandatory to have a FROM instruction in the containerfile.

ARG

Variables for FROM instruction are defined using ARG.

1
2

ARG IMAGE_VERSION=3.7
FROM python:${IMAGE_VERSION}

After FROM instruction is executed, the value for the variable declared in ARG becomes inaccessible.

ENV

The Environment variables (used by other instructions or container at runtime) are declared using ENV.

1
2

ENV DEPLOYMENT_FOLDER=src
WORKDIR ${DEPLOYMENT_FOLDER}

The = operator between the variable name and its value is optional.

LABEL

LABEL is used to add metadata for container images in the form of key-value pairs. Metadata usually contains the author’s name, author’s email, version of the container image, etc.

ADD

ADD instruction will copy files/directories from the source (host OS/URL/tarball) to the destination (container image). ADD allows permissions to be omitted on the copied files/directories with the --chown flag.

1

ADD --chown=user:group src/ dest

Builds are performed inside a build context directory on the host. ADD will fail if the source file/directory is outside the build context.

COPY

COPY is similar to ADD but it is limited to sources on the host, it doesn’t support URLs or tarball. COPY is sufficient for most use cases, so it is preferred over ADD.

WORKDIR

WORKDIR changes the current working directory (inside the image) for the next instructions.

RUN

RUN instruction executes the shell command passed as an argument.

A temporary container is created to execute the shell command and the changes on the file system (after command execution) are layered on top of existing image layers. By default, the instruction RUN apt-get update -y will be executed as /bin/bash -c apt-get update -y.

SHELL instruction is used in containerfile to change the default shell for RUN, for example, SHELL ["/bin/zsh", "-c"] will change the default shell from bash to zsh.

EXPOSE

EXPOSE specifies the network ports to be used by the container on runtime. The ports still have to be published when the container is provisioned.

CMD

CMD adds a shell command that will be executed on container startup. CMD instructions could be written as CMD param1 param2 or CMD ["/bin/executable", "param1", "param2"].

Adding a CMD instruction can define the behavior of the container on startup, for example, a webserver container should start the webserver automatically on startup by running the invocation command defined in CMD.

ENTRYPOINT

If the executable on container startup is expected to be the same in most conditions then it is preferable to add ENTRYPOINT over CMD. Parameters for the executable could be passed from CMD instruction.

1
2

ENTRYPOINT /bin/executable
CMD ["param1", "param2"]

VOLUME

To create a mount point inside the container, VOLUME instruction can be used. During runtime, the --volume flag is used to map a directory on the host to the mount point declared by VOLUME instruction. The data stored on the mount point will persist after container execution.

ONBUILD

The command added in the ONBUILD instruction is executed when the built image is used as the base container image.

If container image container-image-b uses container-image-a as a base image, then the ONBUILD instruction defined in container-image-a will be triggered after FROM instruction is executed in container-image-b.

Execution of ONBUILD instruction

Using ONBUILD, the developers can ensure that any new image built upon their container image has the latest source code or that packages are up-to-date.

1

ONBUILD RUN git clone source-code-repo-link.git

USER

USER instruction sets UID or GID during the build process.

Building container images using Docker

Docker and Podman provide build subcommand for building container images from containerfile.

1

docker build .

. will be substituted with the current working directory on the host.

By default Docker expects Dockerfile (instead of Containerfile) in the build context, --file flag could be used to pass the path to containerfile.

1

docker build --tag node-application:latest . --file ./Containerfile

Container Engines cache the built layers to save time and storage. Cached layers are reused in other image builds to disable this behavior --no-cache flag could be used with the build subcommand.

--rm flag ensures that the intermediate container images are removed once the build process is successful.

Adding files with sensitive data to .dockerignore will make sure that they aren’t included in the container image.

Building container images using buildah

buildah provides a bud subcommand for building container images from containerfile.

1

buildah bud --tag "webserver:latest" .

Images built by buildah could then be executed using Podman and Docker or any other OCI image-compatible container engine. Podman also has a build subcommand for building container images and it uses the same code as buildah.

buildah provides finer control over image building process that allows users to

• Create base images using the scratch image, an image with a minimal footprint.
• Mount a container’s root filesystem, useful for debugging container-related issues.
• Create a container image from a working container, helpful if Containerfile isn’t accessible.
• Execution of containerfile instructions using subcommands. For example, RUN instruction could be executed on a container image using buildah run. Similarly for COPY and ADD instructions.
• Debug image building process by executing instructions manually with buildah.
• Making ad-hoc changes to existing container images.
• Automating build process.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

About the Open Container Initiative
OCI Image Manifest Specification
Image Layer Filesystem Changeset
OCI Image Configuration
About storage drivers
Dockerfile reference
docker build
buildah
scratch container image
Buildah Tutorial 1
Buildah and Podman relationship

Open Container Initiative (OCI) Specification

Open Container Initiative was established by The Linux Foundation in 2015 to provide

• Runtime specification
• Image specification
• Distribution specification for container images.

A container image created from OCI Image specification should have

• Layered Filesystem: Stores the packaged contents of the container.
• Image Index and Manifest: Contains a manifest list that stores the container image metadata for multiple platforms.
• Image Configuration: Arguments and environment variables for the applications inside the container.

Images created from Docker, Podman, and buildah follow OCI Image Specification.

Layered filesystems

Each filesystem layer stores changes performed on top of the previous layer. They are differentiated using their SHA256 digest.

Base and Image Layers

To build a container image the developer has to use another container image as a base layer. The base layer container image could be chosen based on OS preference (like Fedora, Kali, etc.) or programming language preference (like gcc, python, etc.), or any other preference. On top of the base layer developers can add changes (like installing packages and dependencies, performing configuration changes, etc.) by creating stacked image layers.

The layers are cached on the host to reduce the build time and storage space required by the image. Cached layers could also be reused in the build process of different images, for example, two images (on the same host) using base layer ubuntu:22.0 will refer to the same cached layer.

When a container image is provisioned as a container, the base and image layers are locked as read-only.

Read-Write layer added on top of Image Layers

A data state of image layers is generated and a writable container layer is added on top. It stores runtime changes (like file creation, configuration changes, etc.) and deleted with the container.

Image Index and Manifest

The Image Index of a container image contains the metadata related to the filesystem layers, the command to be executed once the container is provisioned and other information required by the container runtime.

To view the index of a container image, the manifest subcommand with inspect option could be used.

1

docker manifest inspect docker.io/library/httpd:latest

The output will be a JSON document with a manifest array containing references to multiple image variations based on the OS and CPU architecture of the host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:d866e5c91f31fc6a122aaf37149cc67ba2ca0de68ae73ab206747a190937967e",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:32588e5c7552750100ad3110a64d163b1881ce92216d67e828c42d3322c439d1",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v5"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:67586a7e127abd9b362884172b575a43b3342725ae310c339f4f7d5e5bdba918",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:bc5f484630b50cec12a50035d22ed717d980c52c9871105e91e276ebcbee69a2",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            }
        }
    ]
}

An image could be built and distributed for different platforms with singular name and tag. The container engine verifies from the manifest if a compatible image is available for the host.

Image Configuration

The changes to be implemented on the filesystem during the creation of the container are stored in the image configuration . It also stores parameters and environment variables for applications inside the container.

docker image inspect image-name could be used to view the configuration of a container image in JSON format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

[
    {
        "Id": "sha256:daab1fa13f8608841399e8552ab7833e90307543509ced13cd40b3f7411632a3",
        "RepoTags": [
            "httpd:latest"
        ],
        "RepoDigests": [
            "httpd@sha256:76618ddd53f315a1436a56dc84ad57032e1b2123f2f6489ce9c575c4b280c4f4"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2023-03-07T20:24:12.062824222Z",
        "Container": "30df81c7e4e9fcec6990ff7a5aee09f0b8d765a65d575599d92fa3d2c2da6048",
        "DockerVersion": "20.10.23",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HTTPD_PREFIX=/usr/local/apache2",
                "HTTPD_VERSION=2.4.56",
                "HTTPD_SHA256=d8d45f1398ba84edd05bb33ca7593ac2989b17cb9c7a0cafe5442d41afdb2d7c",
                "HTTPD_PATCHES="
            ],
            "Cmd": [
                "httpd-foreground"
            ],
            "Image": "sha256:a5930d7e9b3c7fa530cff2806db329c228e08a75d9db73314b3b5724c0858b60",
            "Volumes": null,
            "WorkingDir": "/usr/local/apache2",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null,
            "StopSignal": "SIGWINCH"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 145140075,
        "VirtualSize": 145140075,
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:650abce4b096b06ac8bec2046d821d66d801af34f1f1d4c5e272ad030c7873db",
                "sha256:2309cdaf4afb9ce407a51c6d3ae54fb915bfbcc480e596e72417206902b4c51f",
                "sha256:849b101b0e3b3f03fad010462b6ab14f1372d449a1fb803750182547e7df3d28",
                "sha256:a30707f342ec97e05e57d91804d8ae3d5f93d5e6c9ef2c4e8a3d805b70b0d53e",
                "sha256:087e3023406cdd4c0765ee68d24c88da0c0335ac2ce82d991bd336a648c033c6"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

Distributing container images

Tarfile

Exporting containers

Containers could be exported as tar archives using the export subcommand.

1

docker export httpd-container > httpd.tar

To import this tarfile as a container image, the import subcommand is used.

1

docker import httpd.tar

import also supports URL for tarfile as an argument.

Exporting container images

To export container images as tarfile we can use the save subcommand. Output from the command needs to be redirected using > operator into a *.tar file.

1

docker save httpd:latest > httpd.tar

This tarfile could be loaded as a container image using the load subcommand and < operator.

1

docker load < httpd.tar

Container Registry

Distributing images using Container Registry

Container Registry is a service that allows the distribution of container images. It also provides features such as

• Access control to images
• Vulnerability scanning
• Control over distribution
• High availability

Some of the commonly used container registries are Quay.io, DockerHub, and Google Container Registry.

A registry account is necessary to distribute an image. The credentials from the registry will be used to log in from the container engine.

1

docker login quay.io

The image to be pushed should be tagged/renamed in format <registry-name>/<username>/<image-name>:<image-tag>, for example, quay.io/username/webserver:latest.

1

docker tag webserver:latest quay.io/username/webserver:latest

push subcommand is used to upload the tagged image to the registry.

1

docker push quay.io/username/webserver:latest

Creating images from existing containers

Commit running containers to container image

To create a container image from a running container, the commit subcommand could be used.

1

docker commit webserver-container quay.io/username/webserver:latest

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

About the Open Container Initiative
OCI Image Manifest Specification
Image Layer Filesystem Changeset
OCI Image Configuration
About storage drivers
docker export
docker import
docker save
docker load
docker commit
docker push

We can install Podman on your system by following the steps in Podman Installation Instructions. Podman also provides a graphical interface for managing containers, images, and other resources called Podman Desktop.

Accessing Container Images

To create a container first, we have to download its image from a registry. A container image is a file that contains all the dependencies and executable code for the container.

Images could be fetched from container registries such as DockerHub or Quay.io.

Login to Container Image Registry

A container image registry is a service that handles the storage and distribution of container images. It is necessary to log in to the container registry to access private images.

1

podman login docker.io

After executing this command a prompt will ask for your username and password, for that we need to have a container registry account. Images used in this article are public so we don’t need to create an account or log in.

Once login is successful, Podman stores the encrypted user information in file ${XDG_RUNTIME_DIR}/containers/auth.json. XDG_RUNTIME_DIR is an environment variable set by systemd that stores the path of the directory containing user-specific runtime files.

Pulling Container Images

Pulling container images from registries

Public registries provide an interface for searching images or Podman’s subcommand search could also be used.

1

podman search httpd

Once we have the image name we can pull it on our machine using the pull subcommand

1

podman pull docker.io/library/httpd 

or using the Podman Desktop app you can go to Images -> Pull an Image.

Pulling container image using Podman Desktop

Different versions/variations of container images are managed by Tags. The image tag latest is pulled by default. You can view the list of available tags on the registry’s webpage or use skopeo list-tags. Skopeo requires the transport (docker in our case) to be mentioned explicitly with the image name.

1

skopeo list-tags docker://docker.io/library/httpd

Managing Container Lifecycle

Once we have access to the container image we can easily create one or more containers.

Creating a Container

Creating a container from the container image

We use the run subcommand for creating containers. If the image specified in the command is not present in the system then Podman will attempt to pull it from the registry.

1

podman run --name httpd-test docker.io/library/httpd

After executing this command a container named httpd-test will be created on your system and its output will be attached to your terminal. To exit you can use the shortcut Ctrl+C. To run the container in the background --detach flag could be used.

podman ps command will list running containers on the system.

On Podman Desktop, the Containers section provides utilities for managing the state of containers.

Creating container using Podman Desktop

Executing Commands inside a Container

To run a container and access its shell we have to use options --interactive & --tty.

--interactive will allow us to provide inputs to the process running inside the container and --tty will attach a pseudo-terminal to the container.

Entrypoint specifies a command that is executed during container initialization. It could be defined by default in the image, but you can override it with the --entrypoint flag.

1
2
3

podman run -it --name httpd-test \
               --entrypoint="/bin/bash" \
               docker.io/library/httpd

To execute a command or script inside the container we use Podman’s exec subcommand.

1

podman exec httpd-test echo "Hello, world"

Attaching Volumes

Directories from the host could be mounted on a container. It is useful when scripts are edited outside the container (an IDE/Code Editor on the host) or the files have to be shared between multiple containers.

To mount a directory from host to container flag --volume could be used with run subcommand and values are passed as <HOSTDIR>:<CONTAINERDIR>.

1
2
3
4

podman run -it --name python-test --detach \
               --entrypoint="/bin/bash" \
               --volume $(pwd):/app:Z \
               docker.io/library/python:3.10.6-slim-bullseye

This command starts a python container with contents of the present working directory (fetched from the pwd command) mounted on the /app directory inside the container. Both directory paths have to be absolute.

Before mounting the host directory we have to change its SELinux context to container_file_t otherwise we could encounter permission issues while accessing directory contents. To change the SELinux context you can append either :z (if multiple containers need read-write access) or :Z (if only the current container needs read-write access).

Accessing volumes on Podman Desktop

Port Forwarding

To communicate with the services running inside the container, port forwarding has to be established from the host using the --publish flag in the run subcommand. The port specified for the host and container could be different for example the following command forwards the output from the httpd-test container’s port 80 to port 8080 on the host.

1
2
3

podman run --name httpd-test --detach \
           --publish 8080:80 \
           docker.io/library/httpd           

To view the output we can visit localhost:8080 in our browser. It is advised that only the necessary network ports should be forwarded from container to host.

Pausing an Executing Container

Pausing running container

To demonstrate a paused container we will take the help of the following Python script

1
2
3
4
5
6
7
8
9

# This script runs an infinite loop 
# with the time interval of 2 seconds to slow down the output

import time
i=0
while True:
    print("Value of i: {}".format(i))
    time.sleep(2)
    i+=1

Assuming that this script is saved in the current directory with name counting.py, we run a container created from image docker.io/library/python:3.10.6-slim-bullseye in detached mode.

1
2
3
4

podman run -it --name python-test --detach \
               --entrypoint="/bin/bash" \
               -v $(pwd):/app:Z \
               docker.io/library/python:3.10.6-slim-bullseye

Then we execute the script inside the container in interactive mode

1

podman exec -it python-test python3 /app/counting.py

An infinite loop will be initiated and the output should be visible on the terminal. Now we open a new terminal window and pause this container.

1

podman pause python-test

If we go back to the first terminal we can observe that the script execution is paused. When unpaused the container will restart right where it left.

1

podman unpause python-test

Stopping an Executing Container

Stop running containers

While pausing a container stops its execution in its current state, stopping a container kills the executing process.

1

podman stop python-test

Upon restarting container will run the entrypoint command, starting a new process.

1

podman start python-test

Deleting a Container

Deleting containers

Before we delete a running container we have to stop it or we can use the --force flag along with the rm subcommand.

1

podman rm python-test

Metrics and Logging

Listing Containers

By default, the ps subcommand fetches the list of running containers but to view all the containers regardless of their state we can use the --all flag.

1

podman ps --all

Resource Utilization Statistics

stats subcommand provides live statistics of resource utilization by containers.

1

podman stats

Accessing Logs from a Container

To access the logs of a running/completed container we use the logs subcommand. To view the live output from a running container --follow flag is added.

1

podman logs --follow httpd-test

To access the container logs from Podman Desktop we can click on the container name in the Containers section.

Accessing container logs from Podman Desktop

We can also attach a terminal to the container from the Terminal tab.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Podman Installation Instructions
Podman Desktop
podman-login
XDG Base Directory Specification
httpd container image
python container image
podman-search
podman-pull
podman-run
podman-exec
podman-pause
podman-stop
podman-stats
podman-logs

• Namespaces
• Control Groups
• Secure Computing
• Security-Enhanced Linux

Namespaces

Namespaces are created to limit the reach of a container to its host’s resources. It helps with security and well as limits resources available to the container.

Linux command lsns could be used for listing details of namespaces.

The namespaces essential for containers are User, Mount, Unix Timesharing System, Process ID, Network, and Inter-Process Communication.

User

The users and groups created inside a container are different from its host. Processes running inside the container as a root user could be mapped to a non-root user on the host.

Using the id command you can verify that the containers are present on a different user namespace than other processes on your host.

Running id on host:

1
2

$ id
uid=1000(username) gid=1000(username) groups=1000(username),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Running id inside a container created from docker.io/library/httpd:latest container image:

Process ID (PID)

Each process running on the host has a unique Process ID (PID) assigned to it. The PIDs of processes running inside container are separate from PIDs assigned by the host. Due to process ID isolation, a container can’t access the details of processes running on its host.

To fetch the list of PID namespaces you can use the command:

1
2
3
4
5

$ lsns -t pid
        NS TYPE NPROCS   PID USER  COMMAND
4026531836 pid     128  4996 user /usr/lib/systemd/systemd --user
4026533251 pid      16  4525 user nginx: worker process
...

Just like other processes, containers also have PIDs assigned to them by the host. You can fetch the PID of a running container using the following command:

1
2

$  docker inspect -f '{{.State.Pid}}' deploy-hugo-server-1
7172

ps aux command could be used to list the running processes on the system along with their details.

1
2

$ ps aux | grep 7172
root        7172  0.0  0.4 759596 68860 ?        Ssl  Jan22   0:32 /usr/lib/hugo/hugo server --buildFuture --bind=0.0.0.0

Mount (mnt)

By using different mount namespaces for different processes we can ensure that they won’t be able to access each other’s files.

You can use df command to view the filesystems mounted on your system.

1
2
3
4
5
6

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
devtmpfs            4096         0      4096   0% /dev
tmpfs            7849324     42384   7806940   1% /dev/shm
tmpfs            3139732      2444   3137288   1% /run
...

A container has its separate file system hierarchy which could be viewed by using df command on its shell.

1
2
3
4
5
6

root@14ed72afd62e:/usr/local/apache2# df
Filesystem     1K-blocks      Used Available Use% Mounted on
overlay        498443264 308159584 185270656  63% /
tmpfs              65536         0     65536   0% /dev
tmpfs            1569860       276   1569584   1% /etc/hosts
...

It could also be viewed by the host in the file /proc/<CONTAINER_PID>/mounts.

1
2
3
4
5
6
7
8

$ docker inspect -f '{{.State.Pid}}' deploy-hugo-server-1
7172

$ cat /proc/7172/mounts
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,seclabel,nosuid,size=65536k,mode=755,inode64 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
...

Unix Timesharing System (UTS)

Unix Time System (UTS) namespace allows containers to have hostnames. We can verify this with the hostname command.

On the host:

1
2

[username@host-1 ~]$ hostname
host-1

Inside a container:

1
2

root@14ed72afd62e:/usr/local/apache2# hostname
14ed72afd62e

Network

Each container has a IP address and network ports assigned to it by its network namespace. It allows the developer to run multiple processes inside the container and expose them over different network ports.

To access or communicate with a process inside the container, port forwarding should be established from the host.

Inter-Process Communication (IPC)

Processes in the same IPC namespace can share the resources such as memory, semaphores, and message queues. Keeping separate IPC namespaces ensures that the processes inside a container cannot access the resources used by the host’s processes.

Time

Time namespaces are available since the release of Linux Kernel 5.6.
Maybe in the future containers can have a different time than their host.

Control Groups (Cgroups)

A control group is created to effectively allocate resources of host its processes. These Cgroups are hierarchical i.e. a child Cgroup could be spawned from the parent and it will inherit its certain attributes.

By creating a Cgroup a process in it could be prioritized, paused, removed, or resumed based on the resources allocated to it. It also helps in monitoring the resources used by particular processes.

If you are using an OS with systemd init system (to verify this you can use the command ps -p 1 -o comm=) then you can use the command systemctl list-units to list all the Cgroups. It will open a table containing the Cgroup name, state, and description. The names of the Cgroup will be in the form <parent-cgroup>.<child-cgroup> like sys-devices-platform-serial8250-tty-ttyS0.device.

To view the hierarchy of Cgroups you can use the command systemd-cgls. It presents cgroups as a tree structure.

1
2
3
4
5
6
7
8
9

Control group /:
-.slice
├─user.slice (#1309)
│ └─user-1000.slice (#10010)
│   ├─user@1000.service (#10106)
│   │ ├─session.slice (#10394)
│   │ │ ├─dbus-broker.service (#10442)
│   │ │ │ ├─ 5088 /usr/bin/dbus-broker-launch --scope user
...

Secure Computing (Seccomp)

Using Secure Computing (or seccomp) you can disable the system calls your process can make to the host’s kernel.

A seccomp profile is a definition with a set of restricted and allowed system calls stored in a file. Default seccomp profile used by Docker: default.json.

Docker allows you to define your seccomp profile for a container in JSON format.

1

docker run --rm -it --security-opt seccomp=/path/to/seccomp/profile.json hello-world

Security-Enhanced Linux (SELinux)

SELinux is a security architecture for GNU/Linux-based OS that defines access to files and processes. It is enforced on users or processes to restrict their access to the resources.

SELinux checks the SELinux context of the file or process to make decisions related to its access control. To view the SELinux context of a file use command ls -Z <FILENAME> and to view it for a process using the command ps -eZ | grep <PROCESS_NAME>.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Cgroups, namespaces, and beyond: what are containers made from?
4 Linux technologies fundamental to containers
systemd(1) — Linux manual page
The 7 most used Linux namespaces
Inter-process communication in Linux: Shared storage
DKER-EE-001250 - The Docker Enterprise hosts IPC namespace must not be shared.
It’s Finally Time: The Time Namespace Support Has Been Added To The Linux 5.6 Kernel
Obtaining Information about Control Groups
seccomp(2) — Linux manual page
What is SELinux?
Podman volumes and SELinux

If the environment for the deployment of this application is different then the whole process of dependency installation has to repeat. You might have to some other changes, like setting up environment variables and making changes to the configuration files.

This problem becomes more challenging once you start collaborating with other people. The complete installation process has to be repeated on every development machine. Even if complete documentation of the setup process is available the developers could have a different environment. It is also possible that the developers want to work on two different projects simultaneously and there could be a conflict between their configurations (like different versions of the framework).

When an application is deployed directly on the OS running on the physical hardware it is called baremetal deployment.

Baremetal Deployment

Virtual Machines (VMs)

A virtual machine as the name suggests is a machine running upon the OS installed on the physical hardware. It has its virtual CPU, memory, networking interfaces, and other resources that are provided by a hypervisor.

A hypervisor is a program that is used to create and manage virtual machines. Some common hypervisors are:

• Oracle VM VirtualBox
• Hyper-V

Advantages of using VMs for deploying applications

Virtualized Deployment

So, what advantage does development over VMs provide over development on the OS deployed on the physical hardware?

1. The environment is persistent across the machines.
   If you create your application’s development environment inside a VM, then you can share it with anyone else who is interested to run it themselves.
   The environment will be the same irrespective of the OS installed on the physical hardware.
2. Load balancing is easier in the production environment.
   You can run multiple instances of the same application such that the load could be distributed effectively. If one of the instances is down (due to updates or outage) the traffic to it could be re-routed to another instance.
3. Effective distribution of physical hardware resources available.
   The hypervisor creates a pool of resources available to it from the OS. Then these resources could be effectively distributed among the VMs created. A deployment VM could use fewer resources compared to a development VM.
4. Provides an option to run multiple OS on a single hardware.

Containers

A container is a set of one or more isolated processes running on an OS, it could be on a VM or directly on the physical hardware. Usually, it requires fewer resources than a virtual machine and provides similar performance and benefits depending on your use case.

The workloads running inside a container have very limited exposure to the resources of its host i.e its CPUs, memory, network, and storage. It uses the same kernel as its host, that’s why you can’t spin up a Linux container on a machine running Windows (without provisioning the VM through Windows Subsystem for Linux or any other hypervisor).

Advantages of using Containers over Virtual Machines

Containerized Deployment

1. Relatively lighter compared to VMs
   Containers don’t need virtual resources. The size of the container could be shrunk down to the point where only the dependencies required for the application are present on it.
2. Faster deployment
   Due to their smaller size compared to VMs they could be deployed faster.
3. More scalable
   It is possible to execute multiple containers in the same amount of resources required by a VM for an application.

Depending on your use case the deployment could be a mix of both virtualization and containerization. Like a service comprising of multiple containers could be deployed on a VM.

Managing Containers

Container Runtime

The container is executed through a container runtime. It sets up the namespaces for the container.

Examples of container runtimes:

• containerd: Developed by Docker and donated to Cloud Native Computing Foundation (CNCF). Used in Docker Engine.
• cri-o: Container Runtime commonly used in Kubernetes.

Container Images

A container image is a file that contains all the dependencies and executable code for the container. It is a way of storing and sharing containers.

Container Image Registries

A container image registry is a service that handles the storage and distribution of container images.

Some of the common container registries are: DockerHub, Quay.io, etc.

Container Engine

When a container runtime is expanded with CLI and/or GUI utility for

• Creation of containers
• Changing the state of container (starting, stopping, resuming, deleting)
• Managing container images
• APIs for developers to create layered products on top of it

it becomes a container engine. Some examples of container engines are:

• Docker
• Podman

Open Container Initiative (OCI)

Open Container Initiative was established in 2015 for providing open specifications for:

• Container Runtime
• Container Image
• Container Image Distribution

The objective is to establish a standardized format for the containers such that an OCI container image could be used between different container engines without any issues.

Docker

One of the most commonly used container engines. It is developed by Docker, Inc. It uses a daemon (dockerd) that provides all the management services for the container. This daemon runs with root privileges.

Podman

The acronym for Pod Manager tool is a daemonless container engine originally developed by Red Hat.

Unlike docker, it doesn’t use any daemon for interactions with containers. It also has other security-focused features like the containers don’t run as root by default. This reduces the attack surface as the host system cannot be accessed from the inside of the container.

Skopeo

skopeo is a CLI utility used alongside podman for the management, inspection, and transfer of container images.

Buildah

buildah is another CLI utility used alongside podman and skopeo for building OCI container images.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Oracle VM VirtualBox
Hyper-V
containerd
cri-o
DockerHub
Quay.io
Open Container Initiative
Docker
Podman
Skopeo
Buildah