A Container Storage Interface (CSI) plugin is implemented by the storage providers (Amazon, Google, IBM) as an interface to provision and mount volumes for workloads when requested by container orchestrators. The CSI plugin provisions the volume, procures it from the node hosting the container, and mounts it to the requesting container. It standardizes the process of allocating storage for containers between different orchestrators.

Volume Plugin

Before CSIs, the Volume Plugins were used to link the storage solutions and container orchestrators.

Limitations of Volume Plugins

• Volume Plugins were a part of Kubernetes core. Thus, their development and release were tightly coupled with Kubernetes.
• Instability and reliability issues with volume plugins could lead to the failure of core Kubernetes components.
• Every container orchestrator required a different implementation of the plugin.
• Orchestrators had to communicate with the storage solution throughout the lifecycle of the volume to avoid failures.
• Volume Plugins had to be open source.

Lifecycle of a Volume with Volume Plugins

Volume Plugins

CSI Specification

The CSI specification defines interfaces for implementing a plugin

• The plugin should work seamlessly across all container orchestrators.
• Ability to dynamically provision and unprovisioned volumes.
• Mounting, unmounting, attaching, and detaching volumes on the node.
• Ability to work with block and mountable volumes.
• Performing snapshots of volumes, that could be used for provisioning a new volume.

Architecture of a CSI

The orchestrator interacts with the CSI plugin through gRPCs, a cross-platform Remote Procedure Call framework introduced by Google.

Remote Procedure Calls (RPCs) are used in distributed systems to execute a procedure (subroutine) on a remote system. The CSI’s RPCs are idempotent i.e. a call should not change the result beyond its initial application.

RPC endpoints

There are three sets of gRPCs provided by the CSI Plugin: Identity, Controller, and Node service.

Identity Service RPCs

The Identity Service provides gPRC endpoints that allow an orchestrator to query the capabilities of the plugin, its health, and the metadata associated with the plugin.

gRPC endpoints provided by the identity service:

• GetPluginInfo: Returns the plugin metadata
• GetPluginCapabilities: Returns the plugin capabilities (fetched from controller service)
• Probe: Returns the health of the plugin

Controller Service RPCs

Controller Service groups all the gRPC endpoints responsible for

• Creating and deleting the volume
• Publishing and unpublishing the volume to the nodes
• Creating snapshots of the volumes

gRPC endpoints provided by the controller service:

• CreateVolume: Creates the volumes on the storage solution. The volume could be created as an empty new volume or from an existing snapshot or from an existing volume.
• DeleteVolume: Deletes a provisioned volume on the storage solution. Although the volume is deleted its snapshots are functional and available as sources for new volumes.
• ControllerPublishVolume: Performs the necessary operations to make the volume available to the requesting node.
• ControllerUnpublishVolume: Reverts the operations performed in ContainerPublishVolume.
• ValidateVolumeCapabilities: Verifies the provisioned volume for capabilities requested by the orchestrator.
• ListVolumes: Returns information about all the volumes.
• ControllerGetVolume: Called by the orchestrator to fetch the current information associated with a specific volume.
• GetCapacity: Queries the storage pool for its current capacity.
• ControllerGetCapabilities: Returns the details of the capabilities of the controller service.
• CreateSnapshot: Creates a snapshot of an existing volume.
• DeleteSnapshot: Deletes the volume snapshot.
• ListSnapshots: Returns details of all snapshots.
• ControllerExpandVolume: Using this RPC the orchestrator can request for expansion in the size of a volume.

Node Service RPCs

These RPCs are called from the node where the volume is requested. They are responsible for the actions performed on the volume after it is mounted on the node.

gRPC endpoints provided by the controller service:

• NodeStageVolume: After the workload (container) is scheduled on the orchestrator it calls NodeStageVolume on the volume provided by ContainerPublishVolume. The volume is mounted on the node on stage_path temporarily.
• NodeUnstageVolume: Reverts the actions performed by NodeStageVolume.
• NodePublishVolume: After NodeStageVolume is called the NodePublishVolume mounts the volume on target_path to be consumed by the workload.
• NodeUnpublishVolume: Reverts the actions performed by NodeUnpublishVolume.
• NodeGetVolumeStats: Returns the capacity statistics for a volume.
• NodeGetCapabilities: Called by the orchestrator to check the capabilities of the node services.
• NodeGetInfo: Returns the information for the node where the volume has to be mounted.
• NodeExpandVolume: Expands the volume mounted on the node. This call should be performed after NodeStageVolume and NodePublishVolume.

Implementation of a CSI

CSI specification provides examples of different implementations for plugins.

Centralized Controller Plugin

Centralized Controller CSI Plugin

The controller service RPCs could be packaged in a Controller Plugin located on the master node of the orchestrators. The node service RPCs are bundled as Node Plugin located on each worker node (or any node that could request volumes).

Headless Plugin

Headless CSI Plugin

The Controller and Node Plugin could be placed together inside the worker node.

Controller Plugin

Controller CSI Plugin

CSI could be deployed with just the Controller plugin on the worker node that provides both controller and node services.

Node Plugin

Node CSI Plugin

CSI could also be deployed with just a Node plugin on the worker node providing just the node services.

Volume lifecycle with CSI

Lifecycle of a Volume managed by CSI Plugin

Mounting a Volume on a Container

1. A Workload is scheduled by the container orchestrator.
2. The orchestrator calls CreateVolume RPC to provision a volume on the storage solution.
3. Once the volume is provisioned on the storage solution the orchestrator calls ContainerPublishVolume to make it available to the node.
4. After the volume is available to the node, the orchestrator calls NodeStageVolume to mount the volume on the node at stage_path.
5. Finally, the orchestrator calls NodePublishVolume to mount the volume on target_path. Thus making it available for the container.

Deleting a Volume

1. The volume to be deleted is unpublished by the orchestrator by calling NodeUnpublishVolume RPC.
2. Once unpublished from the container the volume is unstaged by NodeUnstageVolume.
3. The unstaged volume is then unpublished from the node by ContainerUnpublishVolume.
4. Finally, the unpublished volume is deleted by DeleteVolume RPC.

Example of a CSI: Intel Optane Persistent Memory PMEM-CSI Storage Driver

Intel Optane is a tiered persistent memory technology. It is used as a high-performance storage for desktop (improving system boot time) and enterprise (CAD, Modeling, Media editing) workloads.

The PMEM-CSI storage driver is provided by Intel to configure the Optane Persistent Memory as a storage solution for Kubernetes. The persistent volumes could be provisioned dynamically on the Optane memory and mounted on the containers present on the cluster.

Using this driver the developers can take advantage of Optane memory in their cache data store like Memcached Operator.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Container storage 101: What is CSI and how does it work? | Computer Weekly
Understanding the Container Storage Interface (CSI) | by anoop vijayan maniankara
Container Storage Interface: Present and Future - Jie Yu, Mesosphere, Inc.
Container Storage Interface (CSI) for Kubernetes
Introduction - Kubernetes CSI Developer Documentation
Container Storage Interface (CSI) for Kubernetes GA
container-storage-interface/spec
Persistent Memory
intel/pmem-csi
Persistent Memory (PMEM) CSI
Redis-pmem operator
memcached with PMEM

The Container Network Interface (CNI) project provides specifications and libraries for implementing a plugin-based solution for managing network interfaces for containers. The runtime executes the CNI plugins provided as binary executable files.

A network configuration is passed to the runtime as a JSON file. It contains the details of the CNI plugins and the network interfaces to be configured.

Example of a network configuration file from CNI specification

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

{
  "cniVersion": "1.0.0",
  "name": "dbnet",
  "plugins": [
    {
      "type": "bridge",
      // plugin specific parameters
      "bridge": "cni0",
      "keyA": ["some more", "plugin specific", "configuration"],
      
      "ipam": {
        "type": "host-local",
        // ipam specific
        "subnet": "10.1.0.0/16",
        "gateway": "10.1.0.1",
        "routes": [
            {"dst": "0.0.0.0/0"}
        ]
      },
      "dns": {
        "nameservers": [ "10.1.0.1" ]
      }
    },
    {
      "type": "tuning",
      "capabilities": {
        "mac": true
      },
      "sysctl": {
        "net.core.somaxconn": "500"
      }
    },
    {
        "type": "portmap",
        "capabilities": {"portMappings": true}
    }
  ]
}

Runtime executes the CNI plugins with the following environment variables

• CNI_COMMAND: Operation for CNI (ADD/DELETE/CHECK/VERSION)
• CNI_ARGS: Arguments for CNI
• CNI_PATH: Path to CNI binaries.
• CNI_IFNAME: Name of the network interface to be configured
• CNI_NETNS: Path to the container’s network namespace
• CNI_CONTAINERID: Container ID assigned by the runtime

CNI Operations

According to the CNI specification, a CNI plugin should have ADD, DELETE, CHECK, and VERSION operations.

ADD

If the environment variable passed to the runtime by the CNI plugin is CNI_COMMAND=ADD then it will create or configure the network interfaces specified in the configuration.

Execution flow of ADD operation

The interfaces will be created or configured in the order specified in the plugins key inside the network configuration. The result from the previous plugin execution will be passed to the next plugin and the final result will be stored with runtime.

So from the example above the bridge plugin will be executed first, followed by the tuning plugin, and finally the portmap plugin.

DELETE

The DELETE operation will undo operations performed by the ADD, deleting the created interfaces and reverting the configurations. It executes CNI plugins in the reverse order of their specification in network configuration. So the changes performed by the portmap plugin will be reverted first then the tuning plugin and lastly the bridge plugin.

Execution flow of DELETE operation

It has to be preceded by an ADD operation because it takes the final result stored by the last ADD operation as input from container runtime.

CHECK

CHECK operation probes the container and network interfaces for their health and current status. The container runtime will go through the changes performed by each CNI plugin and validate its functionality. Similar to the DELETE operation it also uses the result from the previous ADD operation as input from container runtime.

VERSION

The VERSION operation returns the version of the CNI plugin.

Reference CNI Plugins

Network Interface Plugins

• bridge: Creates a virtual switch that connects all the containers on a host. An IP address is assigned to each container by the bridge.
• macvlan: Creates a sub-interface from the host interface. Each virtual interface connected to macvlan has its own MAC address.

• ipvlan: Similar to macvlan but all connected devices share the same MAC address. Before transmitting a packet the kernel driver inspects the IP address of the destination virtual interface.
• ptp: Creates a point-to-point link between the container and its host by creating a Virtual Ethernet (veth) pair where one end resides inside the container and the other on the host.

• host-device: An existing device could be moved from the host’s network namespace to the container’s network namespace.
• vlan: Creates a Virtual Local Area Network (VLAN) interface between the host and the container.

Windows Plugins

These plugins are created specifically for Microsoft Windows containers.

• win-bridge: All containers will be connected to a bridge network that has an endpoint on the host’s network namespace.
• win-overlay: An Overlay network allows containers from different hosts to act as if they are on the same host. Using the win-overlay plugin all containers on a host could be connected to an Overlay network.

IP Address Allocation Plugin

• dhcp: The IP leased by the DHCP server has to be renewed periodically. The dhcp plugin creates a daemon separate from the container to perform these requests.
• host-local: Allocates IPv4 and IPv6 addresses out of a provided range of addresses.
• static: Allocates static IPv4 and IPv6 addresses to the containers.

Other Plugins

• tuning: Used with other plugins to change system controls or interface attributes (like MTU and MAC address)
• portmap: Forwards traffic from the host’s network ports to the container’s network ports.
• bandwidth: Configures traffic bucket filter (tbf) to limit bandwidth on ingress and egress traffic.
• sbr: Configures source based routing (sbr) where the application (source) decides the correct network interface for outgoing traffic.
• firewall: Creates firewall rules to allow traffic to the container’s IP address.

Example of 3rd Party CNI Plugin: Weave

The Weave is a CNI plugin created by Weaveworks that creates an overlay mesh network that allows Dockers containers running on different hosts to connect.

Features of Weave:

• Containers from different hosts and data centers will act as if they are on the same network without any additional configuration.
• Application containers (running on any host) could be exposed to the outside network.
• It can create networks between legacy systems and containers.
• Traffic could be encrypted.
• Supports Amazon ECS, Apache Mesos, Kubernetes, etc.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Kubernetes CNI Explained
Introduction to CNI, the Container Network Interface Project - Bryan Boreham & Dan Williams
Deep Dive: CNI - Bryan Boreham, Weaveworks & Dan Williams, Red Hat
Kubernetes Networking: The Complete Guide | Tigera
Container Networking: What You Should Know
cni/SPEC.md at main · containernetworking/cni · GitHub
Kubernetes Networking: How to Write a CNI Plugin From Scratch - Eran Yanay, Twistlock
A brief overview of the Container Network Interface (CNI) in Kubernetes | Enable Sysadmin
CNI Plugins Overview
Network Plugins | Kubernetes
containernetworking/plugins
Introduction to Weave Net

The build steps are written in a plaintext file called Containerfile and parsed by container engines (or buildah) during the build process.

1
2
3
4
5
6
7
8

# Containerfile
FROM node:18-alpine
LABEL version="1.0" 
WORKDIR /app
COPY . .
RUN yarn install --production
CMD ["node", "src/index.js"]
EXPOSE 3000

Containerfile Instructions

Steps inside the containerfile are defined using instructions such as FROM, RUN, ADD, COPY, etc. Container Engines go through the containerfile line-by-line and perform each step, stacking a new image layer on top of the previous one.

FROM

FROM instruction specifies the base layer container image. The base layer image could be chosen based on OS preference (like Fedora, Kali, etc.) or programming language preference (like gcc, python, etc.), or any other preference. It is mandatory to have a FROM instruction in the containerfile.

ARG

Variables for FROM instruction are defined using ARG.

1
2

ARG IMAGE_VERSION=3.7
FROM python:${IMAGE_VERSION}

After FROM instruction is executed, the value for the variable declared in ARG becomes inaccessible.

ENV

The Environment variables (used by other instructions or container at runtime) are declared using ENV.

1
2

ENV DEPLOYMENT_FOLDER=src
WORKDIR ${DEPLOYMENT_FOLDER}

The = operator between the variable name and its value is optional.

LABEL

LABEL is used to add metadata for container images in the form of key-value pairs. Metadata usually contains the author’s name, author’s email, version of the container image, etc.

ADD

ADD instruction will copy files/directories from the source (host OS/URL/tarball) to the destination (container image). ADD allows permissions to be omitted on the copied files/directories with the --chown flag.

1

ADD --chown=user:group src/ dest

Builds are performed inside a build context directory on the host. ADD will fail if the source file/directory is outside the build context.

COPY

COPY is similar to ADD but it is limited to sources on the host, it doesn’t support URLs or tarball. COPY is sufficient for most use cases, so it is preferred over ADD.

WORKDIR

WORKDIR changes the current working directory (inside the image) for the next instructions.

RUN

RUN instruction executes the shell command passed as an argument.

A temporary container is created to execute the shell command and the changes on the file system (after command execution) are layered on top of existing image layers. By default, the instruction RUN apt-get update -y will be executed as /bin/bash -c apt-get update -y.

SHELL instruction is used in containerfile to change the default shell for RUN, for example, SHELL ["/bin/zsh", "-c"] will change the default shell from bash to zsh.

EXPOSE

EXPOSE specifies the network ports to be used by the container on runtime. The ports still have to be published when the container is provisioned.

CMD

CMD adds a shell command that will be executed on container startup. CMD instructions could be written as CMD param1 param2 or CMD ["/bin/executable", "param1", "param2"].

Adding a CMD instruction can define the behavior of the container on startup, for example, a webserver container should start the webserver automatically on startup by running the invocation command defined in CMD.

ENTRYPOINT

If the executable on container startup is expected to be the same in most conditions then it is preferable to add ENTRYPOINT over CMD. Parameters for the executable could be passed from CMD instruction.

1
2

ENTRYPOINT /bin/executable
CMD ["param1", "param2"]

VOLUME

To create a mount point inside the container, VOLUME instruction can be used. During runtime, the --volume flag is used to map a directory on the host to the mount point declared by VOLUME instruction. The data stored on the mount point will persist after container execution.

ONBUILD

The command added in the ONBUILD instruction is executed when the built image is used as the base container image.

If container image container-image-b uses container-image-a as a base image, then the ONBUILD instruction defined in container-image-a will be triggered after FROM instruction is executed in container-image-b.

Execution of ONBUILD instruction

Using ONBUILD, the developers can ensure that any new image built upon their container image has the latest source code or that packages are up-to-date.

1

ONBUILD RUN git clone source-code-repo-link.git

USER

USER instruction sets UID or GID during the build process.

Building container images using Docker

Docker and Podman provide build subcommand for building container images from containerfile.

1

docker build .

. will be substituted with the current working directory on the host.

By default Docker expects Dockerfile (instead of Containerfile) in the build context, --file flag could be used to pass the path to containerfile.

1

docker build --tag node-application:latest . --file ./Containerfile

Container Engines cache the built layers to save time and storage. Cached layers are reused in other image builds to disable this behavior --no-cache flag could be used with the build subcommand.

--rm flag ensures that the intermediate container images are removed once the build process is successful.

Adding files with sensitive data to .dockerignore will make sure that they aren’t included in the container image.

Building container images using buildah

buildah provides a bud subcommand for building container images from containerfile.

1

buildah bud --tag "webserver:latest" .

Images built by buildah could then be executed using Podman and Docker or any other OCI image-compatible container engine. Podman also has a build subcommand for building container images and it uses the same code as buildah.

buildah provides finer control over image building process that allows users to

• Create base images using the scratch image, an image with a minimal footprint.
• Mount a container’s root filesystem, useful for debugging container-related issues.
• Create a container image from a working container, helpful if Containerfile isn’t accessible.
• Execution of containerfile instructions using subcommands. For example, RUN instruction could be executed on a container image using buildah run. Similarly for COPY and ADD instructions.
• Debug image building process by executing instructions manually with buildah.
• Making ad-hoc changes to existing container images.
• Automating build process.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

About the Open Container Initiative
OCI Image Manifest Specification
Image Layer Filesystem Changeset
OCI Image Configuration
About storage drivers
Dockerfile reference
docker build
buildah
scratch container image
Buildah Tutorial 1
Buildah and Podman relationship

Open Container Initiative (OCI) Specification

Open Container Initiative was established by The Linux Foundation in 2015 to provide

• Runtime specification
• Image specification
• Distribution specification for container images.

A container image created from OCI Image specification should have

• Layered Filesystem: Stores the packaged contents of the container.
• Image Index and Manifest: Contains a manifest list that stores the container image metadata for multiple platforms.
• Image Configuration: Arguments and environment variables for the applications inside the container.

Images created from Docker, Podman, and buildah follow OCI Image Specification.

Layered filesystems

Each filesystem layer stores changes performed on top of the previous layer. They are differentiated using their SHA256 digest.

Base and Image Layers

To build a container image the developer has to use another container image as a base layer. The base layer container image could be chosen based on OS preference (like Fedora, Kali, etc.) or programming language preference (like gcc, python, etc.), or any other preference. On top of the base layer developers can add changes (like installing packages and dependencies, performing configuration changes, etc.) by creating stacked image layers.

The layers are cached on the host to reduce the build time and storage space required by the image. Cached layers could also be reused in the build process of different images, for example, two images (on the same host) using base layer ubuntu:22.0 will refer to the same cached layer.

When a container image is provisioned as a container, the base and image layers are locked as read-only.

Read-Write layer added on top of Image Layers

A data state of image layers is generated and a writable container layer is added on top. It stores runtime changes (like file creation, configuration changes, etc.) and deleted with the container.

Image Index and Manifest

The Image Index of a container image contains the metadata related to the filesystem layers, the command to be executed once the container is provisioned and other information required by the container runtime.

To view the index of a container image, the manifest subcommand with inspect option could be used.

1

docker manifest inspect docker.io/library/httpd:latest

The output will be a JSON document with a manifest array containing references to multiple image variations based on the OS and CPU architecture of the host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:d866e5c91f31fc6a122aaf37149cc67ba2ca0de68ae73ab206747a190937967e",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:32588e5c7552750100ad3110a64d163b1881ce92216d67e828c42d3322c439d1",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v5"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:67586a7e127abd9b362884172b575a43b3342725ae310c339f4f7d5e5bdba918",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1366,
            "digest": "sha256:bc5f484630b50cec12a50035d22ed717d980c52c9871105e91e276ebcbee69a2",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            }
        }
    ]
}

An image could be built and distributed for different platforms with singular name and tag. The container engine verifies from the manifest if a compatible image is available for the host.

Image Configuration

The changes to be implemented on the filesystem during the creation of the container are stored in the image configuration . It also stores parameters and environment variables for applications inside the container.

docker image inspect image-name could be used to view the configuration of a container image in JSON format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

[
    {
        "Id": "sha256:daab1fa13f8608841399e8552ab7833e90307543509ced13cd40b3f7411632a3",
        "RepoTags": [
            "httpd:latest"
        ],
        "RepoDigests": [
            "httpd@sha256:76618ddd53f315a1436a56dc84ad57032e1b2123f2f6489ce9c575c4b280c4f4"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2023-03-07T20:24:12.062824222Z",
        "Container": "30df81c7e4e9fcec6990ff7a5aee09f0b8d765a65d575599d92fa3d2c2da6048",
        "DockerVersion": "20.10.23",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HTTPD_PREFIX=/usr/local/apache2",
                "HTTPD_VERSION=2.4.56",
                "HTTPD_SHA256=d8d45f1398ba84edd05bb33ca7593ac2989b17cb9c7a0cafe5442d41afdb2d7c",
                "HTTPD_PATCHES="
            ],
            "Cmd": [
                "httpd-foreground"
            ],
            "Image": "sha256:a5930d7e9b3c7fa530cff2806db329c228e08a75d9db73314b3b5724c0858b60",
            "Volumes": null,
            "WorkingDir": "/usr/local/apache2",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null,
            "StopSignal": "SIGWINCH"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 145140075,
        "VirtualSize": 145140075,
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:650abce4b096b06ac8bec2046d821d66d801af34f1f1d4c5e272ad030c7873db",
                "sha256:2309cdaf4afb9ce407a51c6d3ae54fb915bfbcc480e596e72417206902b4c51f",
                "sha256:849b101b0e3b3f03fad010462b6ab14f1372d449a1fb803750182547e7df3d28",
                "sha256:a30707f342ec97e05e57d91804d8ae3d5f93d5e6c9ef2c4e8a3d805b70b0d53e",
                "sha256:087e3023406cdd4c0765ee68d24c88da0c0335ac2ce82d991bd336a648c033c6"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

Distributing container images

Tarfile

Exporting containers

Containers could be exported as tar archives using the export subcommand.

1

docker export httpd-container > httpd.tar

To import this tarfile as a container image, the import subcommand is used.

1

docker import httpd.tar

import also supports URL for tarfile as an argument.

Exporting container images

To export container images as tarfile we can use the save subcommand. Output from the command needs to be redirected using > operator into a *.tar file.

1

docker save httpd:latest > httpd.tar

This tarfile could be loaded as a container image using the load subcommand and < operator.

1

docker load < httpd.tar

Container Registry

Distributing images using Container Registry

Container Registry is a service that allows the distribution of container images. It also provides features such as

• Access control to images
• Vulnerability scanning
• Control over distribution
• High availability

Some of the commonly used container registries are Quay.io, DockerHub, and Google Container Registry.

A registry account is necessary to distribute an image. The credentials from the registry will be used to log in from the container engine.

1

docker login quay.io

The image to be pushed should be tagged/renamed in format <registry-name>/<username>/<image-name>:<image-tag>, for example, quay.io/username/webserver:latest.

1

docker tag webserver:latest quay.io/username/webserver:latest

push subcommand is used to upload the tagged image to the registry.

1

docker push quay.io/username/webserver:latest

Creating images from existing containers

Commit running containers to container image

To create a container image from a running container, the commit subcommand could be used.

1

docker commit webserver-container quay.io/username/webserver:latest

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

About the Open Container Initiative
OCI Image Manifest Specification
Image Layer Filesystem Changeset
OCI Image Configuration
About storage drivers
docker export
docker import
docker save
docker load
docker commit
docker push

We can install Podman on your system by following the steps in Podman Installation Instructions. Podman also provides a graphical interface for managing containers, images, and other resources called Podman Desktop.

Accessing Container Images

To create a container first, we have to download its image from a registry. A container image is a file that contains all the dependencies and executable code for the container.

Images could be fetched from container registries such as DockerHub or Quay.io.

Login to Container Image Registry

A container image registry is a service that handles the storage and distribution of container images. It is necessary to log in to the container registry to access private images.

1

podman login docker.io

After executing this command a prompt will ask for your username and password, for that we need to have a container registry account. Images used in this article are public so we don’t need to create an account or log in.

Once login is successful, Podman stores the encrypted user information in file ${XDG_RUNTIME_DIR}/containers/auth.json. XDG_RUNTIME_DIR is an environment variable set by systemd that stores the path of the directory containing user-specific runtime files.

Pulling Container Images

Pulling container images from registries

Public registries provide an interface for searching images or Podman’s subcommand search could also be used.

1

podman search httpd

Once we have the image name we can pull it on our machine using the pull subcommand

1

podman pull docker.io/library/httpd 

or using the Podman Desktop app you can go to Images -> Pull an Image.

Pulling container image using Podman Desktop

Different versions/variations of container images are managed by Tags. The image tag latest is pulled by default. You can view the list of available tags on the registry’s webpage or use skopeo list-tags. Skopeo requires the transport (docker in our case) to be mentioned explicitly with the image name.

1

skopeo list-tags docker://docker.io/library/httpd

Managing Container Lifecycle

Once we have access to the container image we can easily create one or more containers.

Creating a Container

Creating a container from the container image

We use the run subcommand for creating containers. If the image specified in the command is not present in the system then Podman will attempt to pull it from the registry.

1

podman run --name httpd-test docker.io/library/httpd

After executing this command a container named httpd-test will be created on your system and its output will be attached to your terminal. To exit you can use the shortcut Ctrl+C. To run the container in the background --detach flag could be used.

podman ps command will list running containers on the system.

On Podman Desktop, the Containers section provides utilities for managing the state of containers.

Creating container using Podman Desktop

Executing Commands inside a Container

To run a container and access its shell we have to use options --interactive & --tty.

--interactive will allow us to provide inputs to the process running inside the container and --tty will attach a pseudo-terminal to the container.

Entrypoint specifies a command that is executed during container initialization. It could be defined by default in the image, but you can override it with the --entrypoint flag.

1
2
3

podman run -it --name httpd-test \
               --entrypoint="/bin/bash" \
               docker.io/library/httpd

To execute a command or script inside the container we use Podman’s exec subcommand.

1

podman exec httpd-test echo "Hello, world"

Attaching Volumes

Directories from the host could be mounted on a container. It is useful when scripts are edited outside the container (an IDE/Code Editor on the host) or the files have to be shared between multiple containers.

To mount a directory from host to container flag --volume could be used with run subcommand and values are passed as <HOSTDIR>:<CONTAINERDIR>.

1
2
3
4

podman run -it --name python-test --detach \
               --entrypoint="/bin/bash" \
               --volume $(pwd):/app:Z \
               docker.io/library/python:3.10.6-slim-bullseye

This command starts a python container with contents of the present working directory (fetched from the pwd command) mounted on the /app directory inside the container. Both directory paths have to be absolute.

Before mounting the host directory we have to change its SELinux context to container_file_t otherwise we could encounter permission issues while accessing directory contents. To change the SELinux context you can append either :z (if multiple containers need read-write access) or :Z (if only the current container needs read-write access).

Accessing volumes on Podman Desktop

Port Forwarding

To communicate with the services running inside the container, port forwarding has to be established from the host using the --publish flag in the run subcommand. The port specified for the host and container could be different for example the following command forwards the output from the httpd-test container’s port 80 to port 8080 on the host.

1
2
3

podman run --name httpd-test --detach \
           --publish 8080:80 \
           docker.io/library/httpd           

To view the output we can visit localhost:8080 in our browser. It is advised that only the necessary network ports should be forwarded from container to host.

Pausing an Executing Container

Pausing running container

To demonstrate a paused container we will take the help of the following Python script

1
2
3
4
5
6
7
8
9

# This script runs an infinite loop 
# with the time interval of 2 seconds to slow down the output

import time
i=0
while True:
    print("Value of i: {}".format(i))
    time.sleep(2)
    i+=1

Assuming that this script is saved in the current directory with name counting.py, we run a container created from image docker.io/library/python:3.10.6-slim-bullseye in detached mode.

1
2
3
4

podman run -it --name python-test --detach \
               --entrypoint="/bin/bash" \
               -v $(pwd):/app:Z \
               docker.io/library/python:3.10.6-slim-bullseye

Then we execute the script inside the container in interactive mode

1

podman exec -it python-test python3 /app/counting.py

An infinite loop will be initiated and the output should be visible on the terminal. Now we open a new terminal window and pause this container.

1

podman pause python-test

If we go back to the first terminal we can observe that the script execution is paused. When unpaused the container will restart right where it left.

1

podman unpause python-test

Stopping an Executing Container

Stop running containers

While pausing a container stops its execution in its current state, stopping a container kills the executing process.

1

podman stop python-test

Upon restarting container will run the entrypoint command, starting a new process.

1

podman start python-test

Deleting a Container

Deleting containers

Before we delete a running container we have to stop it or we can use the --force flag along with the rm subcommand.

1

podman rm python-test

Metrics and Logging

Listing Containers

By default, the ps subcommand fetches the list of running containers but to view all the containers regardless of their state we can use the --all flag.

1

podman ps --all

Resource Utilization Statistics

stats subcommand provides live statistics of resource utilization by containers.

1

podman stats

Accessing Logs from a Container

To access the logs of a running/completed container we use the logs subcommand. To view the live output from a running container --follow flag is added.

1

podman logs --follow httpd-test

To access the container logs from Podman Desktop we can click on the container name in the Containers section.

Accessing container logs from Podman Desktop

We can also attach a terminal to the container from the Terminal tab.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Podman Installation Instructions
Podman Desktop
podman-login
XDG Base Directory Specification
httpd container image
python container image
podman-search
podman-pull
podman-run
podman-exec
podman-pause
podman-stop
podman-stats
podman-logs

• Namespaces
• Control Groups
• Secure Computing
• Security-Enhanced Linux

Namespaces

Namespaces are created to limit the reach of a container to its host’s resources. It helps with security and well as limits resources available to the container.

Linux command lsns could be used for listing details of namespaces.

The namespaces essential for containers are User, Mount, Unix Timesharing System, Process ID, Network, and Inter-Process Communication.

User

The users and groups created inside a container are different from its host. Processes running inside the container as a root user could be mapped to a non-root user on the host.

Using the id command you can verify that the containers are present on a different user namespace than other processes on your host.

Running id on host:

1
2

$ id
uid=1000(username) gid=1000(username) groups=1000(username),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Running id inside a container created from docker.io/library/httpd:latest container image:

Process ID (PID)

Each process running on the host has a unique Process ID (PID) assigned to it. The PIDs of processes running inside container are separate from PIDs assigned by the host. Due to process ID isolation, a container can’t access the details of processes running on its host.

To fetch the list of PID namespaces you can use the command:

1
2
3
4
5

$ lsns -t pid
        NS TYPE NPROCS   PID USER  COMMAND
4026531836 pid     128  4996 user /usr/lib/systemd/systemd --user
4026533251 pid      16  4525 user nginx: worker process
...

Just like other processes, containers also have PIDs assigned to them by the host. You can fetch the PID of a running container using the following command:

1
2

$  docker inspect -f '{{.State.Pid}}' deploy-hugo-server-1
7172

ps aux command could be used to list the running processes on the system along with their details.

1
2

$ ps aux | grep 7172
root        7172  0.0  0.4 759596 68860 ?        Ssl  Jan22   0:32 /usr/lib/hugo/hugo server --buildFuture --bind=0.0.0.0

Mount (mnt)

By using different mount namespaces for different processes we can ensure that they won’t be able to access each other’s files.

You can use df command to view the filesystems mounted on your system.

1
2
3
4
5
6

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
devtmpfs            4096         0      4096   0% /dev
tmpfs            7849324     42384   7806940   1% /dev/shm
tmpfs            3139732      2444   3137288   1% /run
...

A container has its separate file system hierarchy which could be viewed by using df command on its shell.

1
2
3
4
5
6

root@14ed72afd62e:/usr/local/apache2# df
Filesystem     1K-blocks      Used Available Use% Mounted on
overlay        498443264 308159584 185270656  63% /
tmpfs              65536         0     65536   0% /dev
tmpfs            1569860       276   1569584   1% /etc/hosts
...

It could also be viewed by the host in the file /proc/<CONTAINER_PID>/mounts.

1
2
3
4
5
6
7
8

$ docker inspect -f '{{.State.Pid}}' deploy-hugo-server-1
7172

$ cat /proc/7172/mounts
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,seclabel,nosuid,size=65536k,mode=755,inode64 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
...

Unix Timesharing System (UTS)

Unix Time System (UTS) namespace allows containers to have hostnames. We can verify this with the hostname command.

On the host:

1
2

[username@host-1 ~]$ hostname
host-1

Inside a container:

1
2

root@14ed72afd62e:/usr/local/apache2# hostname
14ed72afd62e

Network

Each container has a IP address and network ports assigned to it by its network namespace. It allows the developer to run multiple processes inside the container and expose them over different network ports.

To access or communicate with a process inside the container, port forwarding should be established from the host.

Inter-Process Communication (IPC)

Processes in the same IPC namespace can share the resources such as memory, semaphores, and message queues. Keeping separate IPC namespaces ensures that the processes inside a container cannot access the resources used by the host’s processes.

Time

Time namespaces are available since the release of Linux Kernel 5.6.
Maybe in the future containers can have a different time than their host.

Control Groups (Cgroups)

A control group is created to effectively allocate resources of host its processes. These Cgroups are hierarchical i.e. a child Cgroup could be spawned from the parent and it will inherit its certain attributes.

By creating a Cgroup a process in it could be prioritized, paused, removed, or resumed based on the resources allocated to it. It also helps in monitoring the resources used by particular processes.

If you are using an OS with systemd init system (to verify this you can use the command ps -p 1 -o comm=) then you can use the command systemctl list-units to list all the Cgroups. It will open a table containing the Cgroup name, state, and description. The names of the Cgroup will be in the form <parent-cgroup>.<child-cgroup> like sys-devices-platform-serial8250-tty-ttyS0.device.

To view the hierarchy of Cgroups you can use the command systemd-cgls. It presents cgroups as a tree structure.

1
2
3
4
5
6
7
8
9

Control group /:
-.slice
├─user.slice (#1309)
│ └─user-1000.slice (#10010)
│   ├─user@1000.service (#10106)
│   │ ├─session.slice (#10394)
│   │ │ ├─dbus-broker.service (#10442)
│   │ │ │ ├─ 5088 /usr/bin/dbus-broker-launch --scope user
...

Secure Computing (Seccomp)

Using Secure Computing (or seccomp) you can disable the system calls your process can make to the host’s kernel.

A seccomp profile is a definition with a set of restricted and allowed system calls stored in a file. Default seccomp profile used by Docker: default.json.

Docker allows you to define your seccomp profile for a container in JSON format.

1

docker run --rm -it --security-opt seccomp=/path/to/seccomp/profile.json hello-world

Security-Enhanced Linux (SELinux)

SELinux is a security architecture for GNU/Linux-based OS that defines access to files and processes. It is enforced on users or processes to restrict their access to the resources.

SELinux checks the SELinux context of the file or process to make decisions related to its access control. To view the SELinux context of a file use command ls -Z <FILENAME> and to view it for a process using the command ps -eZ | grep <PROCESS_NAME>.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Cgroups, namespaces, and beyond: what are containers made from?
4 Linux technologies fundamental to containers
systemd(1) — Linux manual page
The 7 most used Linux namespaces
Inter-process communication in Linux: Shared storage
DKER-EE-001250 - The Docker Enterprise hosts IPC namespace must not be shared.
It’s Finally Time: The Time Namespace Support Has Been Added To The Linux 5.6 Kernel
Obtaining Information about Control Groups
seccomp(2) — Linux manual page
What is SELinux?
Podman volumes and SELinux

If the environment for the deployment of this application is different then the whole process of dependency installation has to repeat. You might have to some other changes, like setting up environment variables and making changes to the configuration files.

This problem becomes more challenging once you start collaborating with other people. The complete installation process has to be repeated on every development machine. Even if complete documentation of the setup process is available the developers could have a different environment. It is also possible that the developers want to work on two different projects simultaneously and there could be a conflict between their configurations (like different versions of the framework).

When an application is deployed directly on the OS running on the physical hardware it is called baremetal deployment.

Baremetal Deployment

Virtual Machines (VMs)

A virtual machine as the name suggests is a machine running upon the OS installed on the physical hardware. It has its virtual CPU, memory, networking interfaces, and other resources that are provided by a hypervisor.

A hypervisor is a program that is used to create and manage virtual machines. Some common hypervisors are:

• Oracle VM VirtualBox
• Hyper-V

Advantages of using VMs for deploying applications

Virtualized Deployment

So, what advantage does development over VMs provide over development on the OS deployed on the physical hardware?

1. The environment is persistent across the machines.
   If you create your application’s development environment inside a VM, then you can share it with anyone else who is interested to run it themselves.
   The environment will be the same irrespective of the OS installed on the physical hardware.
2. Load balancing is easier in the production environment.
   You can run multiple instances of the same application such that the load could be distributed effectively. If one of the instances is down (due to updates or outage) the traffic to it could be re-routed to another instance.
3. Effective distribution of physical hardware resources available.
   The hypervisor creates a pool of resources available to it from the OS. Then these resources could be effectively distributed among the VMs created. A deployment VM could use fewer resources compared to a development VM.
4. Provides an option to run multiple OS on a single hardware.

Containers

A container is a set of one or more isolated processes running on an OS, it could be on a VM or directly on the physical hardware. Usually, it requires fewer resources than a virtual machine and provides similar performance and benefits depending on your use case.

The workloads running inside a container have very limited exposure to the resources of its host i.e its CPUs, memory, network, and storage. It uses the same kernel as its host, that’s why you can’t spin up a Linux container on a machine running Windows (without provisioning the VM through Windows Subsystem for Linux or any other hypervisor).

Advantages of using Containers over Virtual Machines

Containerized Deployment

1. Relatively lighter compared to VMs
   Containers don’t need virtual resources. The size of the container could be shrunk down to the point where only the dependencies required for the application are present on it.
2. Faster deployment
   Due to their smaller size compared to VMs they could be deployed faster.
3. More scalable
   It is possible to execute multiple containers in the same amount of resources required by a VM for an application.

Depending on your use case the deployment could be a mix of both virtualization and containerization. Like a service comprising of multiple containers could be deployed on a VM.

Managing Containers

Container Runtime

The container is executed through a container runtime. It sets up the namespaces for the container.

Examples of container runtimes:

• containerd: Developed by Docker and donated to Cloud Native Computing Foundation (CNCF). Used in Docker Engine.
• cri-o: Container Runtime commonly used in Kubernetes.

Container Images

A container image is a file that contains all the dependencies and executable code for the container. It is a way of storing and sharing containers.

Container Image Registries

A container image registry is a service that handles the storage and distribution of container images.

Some of the common container registries are: DockerHub, Quay.io, etc.

Container Engine

When a container runtime is expanded with CLI and/or GUI utility for

• Creation of containers
• Changing the state of container (starting, stopping, resuming, deleting)
• Managing container images
• APIs for developers to create layered products on top of it

it becomes a container engine. Some examples of container engines are:

• Docker
• Podman

Open Container Initiative (OCI)

Open Container Initiative was established in 2015 for providing open specifications for:

• Container Runtime
• Container Image
• Container Image Distribution

The objective is to establish a standardized format for the containers such that an OCI container image could be used between different container engines without any issues.

Docker

One of the most commonly used container engines. It is developed by Docker, Inc. It uses a daemon (dockerd) that provides all the management services for the container. This daemon runs with root privileges.

Podman

The acronym for Pod Manager tool is a daemonless container engine originally developed by Red Hat.

Unlike docker, it doesn’t use any daemon for interactions with containers. It also has other security-focused features like the containers don’t run as root by default. This reduces the attack surface as the host system cannot be accessed from the inside of the container.

Skopeo

skopeo is a CLI utility used alongside podman for the management, inspection, and transfer of container images.

Buildah

buildah is another CLI utility used alongside podman and skopeo for building OCI container images.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Oracle VM VirtualBox
Hyper-V
containerd
cri-o
DockerHub
Quay.io
Open Container Initiative
Docker
Podman
Skopeo
Buildah