What is Docker Swarm? A Comprehensive Guide

Posted on by

Docker Swarm Architecture

Nodes in a Swarm: Managers and Workers

A Docker Swarm consists of two types of nodes: manager nodes and worker nodes. These nodes work together to maintain the desired state of your application. Understanding their roles is crucial for effective container orchestration using Docker Swarm.

Manager Nodes: Responsibilities

Manager nodes are the brains of the swarm. They are responsible for maintaining the swarm’s state, scheduling services, and handling orchestration tasks. Think of them as the control plane. Key responsibilities include:

  • Cluster Management: Maintaining the overall health and configuration of the Docker Swarm cluster.
  • Service Orchestration: Scheduling tasks (containers) across worker nodes to fulfill service deployments.
  • Raft Consensus: Using the Raft consensus algorithm to ensure fault tolerance and high availability of the swarm’s state. This means that even if some manager nodes fail, the swarm can continue operating.
  • API Endpoint: Exposing the Docker API, allowing you to interact with the swarm and manage your applications.

For production environments, it’s recommended to have multiple manager nodes (typically 3 or 5) to ensure high availability. If one manager fails, the others can take over its responsibilities. This decentralized design for high availability is a key advantage of Docker Swarm.

Worker Nodes: Responsibilities

Worker nodes are the workhorses of the swarm. They execute the tasks assigned to them by the manager nodes. Key responsibilities include:

  • Task Execution: Running containerized applications (tasks) as instructed by the manager nodes.
  • Reporting Status: Reporting the status of running tasks back to the manager nodes.

Worker nodes don’t participate in the Raft consensus or make scheduling decisions. They simply execute the tasks assigned to them. You can add or remove worker nodes as needed to scale your application based on demand. This scalability is essential for efficient docker clustering.

Swarm Services and Tasks

In Docker Swarm, applications are deployed as services. A service defines the desired state of your application, such as the number of replicas (containers) to run, the network configuration, and the resources to allocate. Tasks are the individual units of work that implement the service. Each task represents a single container running an instance of your application.

Defining Services

Services are defined using the docker service create command or, more commonly, with a Docker Compose file (version 3 or higher). The Compose file allows you to define multi-container applications and deploy them as a single service. This integration with Docker Compose simplifies the deployment process.

Here’s a simple example of creating a service using the command line:

docker service create --name my-web-app -p 80:80 nginx

This command creates a service named my-web-app that runs the nginx image and maps port 80 on the host to port 80 in the container. Let’s break down the options:

  • docker service create: This is the base command to create a new service in the Docker Swarm.
  • --name my-web-app: This assigns the name “my-web-app” to the service. This name is used to manage and refer to the service later.
  • -p 80:80: This option publishes the service’s port. In this case, it maps port 80 on the host (the Swarm node) to port 80 inside the container. This allows you to access the web application running in the container through the host’s port 80.
  • nginx: This specifies the Docker image to use for the service. In this case, it’s the official Nginx image from Docker Hub. Nginx is a popular web server.

Tasks Execution

When you create a service, the manager nodes determine where to run the tasks (containers) based on resource availability and other constraints. The manager nodes distribute tasks across the worker nodes. Docker Swarm automatically handles task placement and ensures that the desired number of replicas is running. If a task fails, the manager node will automatically reschedule it on another worker node. This ensures the service maintains the desired state, contributing to a robust and self-healing system.

Overlay Networks for Service Communication

Docker Swarm uses overlay networks to enable communication between services running on different nodes. An overlay network is a virtual network that spans across multiple Docker hosts. This allows containers running on different nodes to communicate with each other as if they were on the same network. This is fundamental for service discovery and load balancing in a distributed environment.

When you create a service, you can attach it to one or more overlay networks. Docker Swarm provides built-in service discovery, allowing containers to find each other by service name. The swarm also provides load balancing, distributing traffic across the available tasks (containers) within a service. This ensures that no single container is overloaded and that the application remains responsive.

Setting Up a Docker Swarm

Prerequisites

Before diving into the setup, let’s ensure you have everything you need to create your Docker Swarm. The following are essential prerequisites:

Docker Installation

First and foremost, Docker must be installed on all machines that will participate in the swarm (manager and worker nodes). Ensure you have the latest stable version installed. You can follow the official Docker documentation for installation instructions specific to your operating system. This usually involves downloading and running an installation package or using a package manager like apt or yum. Remember to configure Docker to start on boot to ensure your swarm nodes automatically rejoin the swarm after a reboot.

Basic Docker Knowledge

A fundamental understanding of Docker concepts is also crucial. You should be familiar with:

  • Images: Understanding how to pull and manage Docker images.
  • Containers: Knowing how to run and manage containers.
  • Docker Compose (Optional but Recommended): Familiarity with Docker Compose for defining and managing multi-container applications. This will make deploying complex applications to your swarm much easier.

If you’re new to Docker, consider going through a basic Docker tutorial before proceeding. There are many excellent resources available online, including the official Docker documentation and interactive tutorials.

Initializing a Swarm

The first step in creating a Docker Swarm is to initialize it on one of the nodes. This node will become the first manager node in the swarm. It’s important to choose a machine with reliable network connectivity to serve as the initial manager.

docker swarm init Command

To initialize a swarm, use the following command:

docker swarm init

This command will initialize a new swarm on the current node and print a docker swarm join command that you can use to add worker nodes to the swarm. Keep this command handy, as you’ll need it later.

Options Explained

The docker swarm init command has several useful options. Here are a few of the most important:

  • --advertise-addr <ip-address>: This option specifies the address that other nodes will use to connect to the manager node. It’s crucial to set this correctly, especially if your manager node has multiple network interfaces. For example:
    docker swarm init --advertise-addr 192.168.1.100
    This tells other nodes to connect to the manager via the IP address 192.168.1.100.
  • --listen-addr <ip-address:port>: This option specifies the address and port on which the manager node will listen for incoming connections. The default port is 2377. You typically don’t need to change this unless you have a specific reason to do so.
  • --force-new-cluster: Use this option if you want to create a new cluster from an existing swarm that is in a failed or inconsistent state. Use this with caution, as it can potentially lead to data loss if not used correctly.

Adding Worker Nodes to the Swarm

Once the swarm is initialized, you can add worker nodes. Worker nodes execute the tasks assigned to them by the manager nodes. Adding worker nodes increases the swarm’s capacity to run containers. Make sure Docker is installed on the worker nodes before attempting to join them to the swarm.

Joining Tokens

Docker Swarm uses tokens to authenticate nodes joining the swarm. There are separate tokens for worker and manager nodes. The docker swarm init command generates a join token for worker nodes. To get the manager join token, run the following command on a manager node:

docker swarm join-token manager

Keep these tokens secure, as anyone with a valid token can join the swarm.

docker swarm join Command

To add a worker node to the swarm, use the docker swarm join command that was printed when you initialized the swarm, or construct it manually using the manager’s IP address and the worker join token. The general format is:

docker swarm join --token <worker-token> <manager-ip>:2377

Replace <worker-token> with the actual worker token and <manager-ip> with the IP address of the manager node. For example:

docker swarm join --token SWMTKN-1-59wb0jgj9cz9nq6s6c6lz7jdx0x4kw4ffj96r3yd6s8v0w5nr-39cr00o9wn0v9jfjf3upn8mqd 192.168.1.100:2377

Run this command on each worker node that you want to add to the swarm.

Verifying the Swarm Setup

After adding worker nodes, it’s essential to verify that the swarm is set up correctly and that all nodes are participating as expected. This involves checking the status of the nodes and ensuring they are in the Ready state.

Checking Node Status

You can check the status of the nodes in the swarm using the docker node ls command. This command must be run on a manager node.

docker node ls Command

Run the following command on a manager node:

docker node ls

This command will display a list of all nodes in the swarm, along with their ID, hostname, status, availability, and role (manager or worker). The output will look similar to this:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
8rroua98r4vwuh8s8505llx3s *   manager1            Ready               Active              Leader              19.03.1
ba130wtwzh0bzj338vsv0j9q6     worker1             Ready               Active                                  19.03.1
c63x7pjy3pwo2z8lfyssjwpj3     worker2             Ready               Active                                  19.03.1

In this example, manager1 is the leader manager node, and worker1 and worker2 are worker nodes. The STATUS column indicates the current status of the node. A status of Ready means the node is healthy and participating in the swarm. The AVAILABILITY column indicates whether the node is available to accept tasks. A status of Active means the node is available.
docker swarm tutorial

Creating a Docker Service

In Docker Swarm, applications are deployed as services. The docker service create command is used to define and deploy these services. Key options include --name (service name), --image (Docker image), -p (port mapping), --replicas (number of replicas), --network (network attachment), --mount (volume mounting), --env (environment variables), --limit-memory (memory limit), and --reserve-memory (memory reservation). For example: 

docker service create --name my-web-app --image nginx -p 80:80 --replicas 3

Scaling Services


The docker service scale command is used to adjust the number of replicas for a service. For example: 

docker service scale my-web-app=5

Docker Swarm will automatically adjust the number of containers to match the desired scale.

Updating Services

The docker service update command is used to update a service’s configuration. Docker Swarm supports rolling updates for zero-downtime deployments. Options include --update-delay (delay between updates), --update-parallelism (number of containers to update simultaneously), and --update-failure-action (action on update failure). For example: 

docker service update --image nginx:latest --update-delay 10s my-web-app

Monitoring Services

Monitoring is crucial for service health. The docker service ps command displays the status of tasks. You can check service logs using docker logs. Docker Swarm Visualizer provides a visual representation of the swarm cluster. The adoption of microservice architecture for cloud-native applications emphasizes the importance of visualizing deployments.

Docker Swarm Commands and Usage

Essential Swarm Commands

Docker Swarm provides a set of commands to manage the swarm, nodes, services, and stacks. Understanding these commands is essential for effective container orchestration.

docker swarm

The docker swarm command is the root command for managing the swarm itself. It’s used to initialize, join, leave, and update the swarm. Key subcommands include:

  • init: Initializes a new swarm. (covered in previous chapter)
  • join: Joins an existing swarm as a worker or manager. (covered in previous chapter)
  • leave: Leaves the current swarm. Use docker swarm leave --force to force a node to leave if it’s having trouble.
  • update: Updates the swarm configuration, such as the Raft settings or autolock key.
  • inspect: Displays detailed information about the swarm. This is useful for troubleshooting and verifying the swarm’s configuration.

docker node

The docker node command is used to manage the nodes in the swarm. Key subcommands include:

  • ls: Lists the nodes in the swarm and their status. (covered in previous chapter)
  • inspect: Displays detailed information about a specific node. This includes the node’s ID, hostname, status, and resources.
  • rm: Removes a node from the swarm. You can only remove nodes that are in the Down state.
  • update: Updates the node’s configuration, such as its availability (Active, Pause, Drain) or role (manager or worker). Setting a node to Drain will prevent new tasks from being scheduled on it, allowing you to gracefully remove it.

docker service

The docker service command is used to manage services running in the swarm. Key subcommands include:

  • create: Creates a new service. (covered in previous chapter)
  • ls: Lists the services running in the swarm.
  • inspect: Displays detailed information about a specific service, including its configuration, tasks, and endpoints.
  • update: Updates the configuration of a service. (covered in previous chapter)
  • scale: Scales a service by adjusting the number of replicas. (covered in previous chapter)
  • rm: Removes a service from the swarm.
  • ps: Lists the tasks (containers) associated with a service and their status. (covered in previous chapter)
  • logs: Fetches the logs of a service or specific task.

docker stack

The docker stack command is used to deploy and manage multi-service applications defined in a Docker Compose file. This is a convenient way to deploy complex applications to a swarm. Key subcommands include:

  • deploy: Deploys a stack to the swarm. You specify the Compose file using the -c option. For example: docker stack deploy -c docker-compose.yml my-stack. The --prune option removes services that are no longer defined in the Compose file.
  • ls: Lists the stacks deployed in the swarm.
  • inspect: Displays detailed information about a specific stack.
  • rm: Removes a stack from the swarm.
  • services: Lists the services associated with a stack.

Examples of Real Use Cases

Let’s explore some real-world examples of how you can use Docker Swarm to deploy and manage applications.

Deploying a Web Application

To deploy a simple web application, you can use the docker service create command with the appropriate image and port mappings. For example, to deploy an Nginx web server:

docker service create --name web-app --image nginx:latest -p 80:80 --replicas 2

This command creates a service named web-app that runs two replicas of the Nginx image and maps port 80 on the host to port 80 in the containers. You can then access the web application by browsing to the IP address of any of the swarm nodes.

Deploying a Database

To deploy a database, such as MySQL or PostgreSQL, you can use the docker service create command with the appropriate image and environment variables. For example, to deploy a MySQL database:

docker service create --name mysql-db --image mysql:5.7 -e MYSQL_ROOT_PASSWORD=secret -p 3306:3306 --replicas 1 --mount type=volume,source=mysql-data,target=/var/lib/mysql

This command creates a service named mysql-db that runs one replica of the MySQL 5.7 image, sets the root password to secret, maps port 3306 on the host to port 3306 in the container, and mounts a volume named mysql-data to persist the database data. docker swarm tutorial. Remember to choose a strong password for production deployments!

Deploying a Microservices Architecture

Docker Swarm is well-suited for deploying microservices architectures. You can define each microservice as a separate service in a Docker Compose file and then deploy the entire application using the docker stack deploy command. This allows you to manage all of the microservices as a single unit. For example, you might have services for user management, product catalog, and order processing, all defined in a single Compose file and deployed as a stack.

Advanced Docker Swarm Concepts

Docker Stack Files

Defining Applications with YAML

Docker stack files use the Docker Compose file format (version 3 or higher) to define multi-service applications. These files are written in YAML and describe the services, networks, volumes, and other resources that make up your application. This declarative approach simplifies the deployment and management of complex applications.

Here’s a simple example of a Docker Compose file:

version: "3.7"
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    networks:
      - my-network
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: secret
    volumes:
      - db-data:/var/lib/mysql
    networks:
      - my-network
volumes:
  db-data:
networks:
  my-network:

This Compose file defines two services: web (an Nginx web server) and db (a MySQL database). They are connected to a common network called my-network, and the database uses a volume named db-data for persistent storage.

As described in the Docker documentation, a Docker image is a read-only template used to build containers. Compose files specify which images to use for each service.

Deploying Stacks

Once you have a Docker Compose file, you can deploy it to a Docker Swarm using the docker stack deploy command.

docker stack deploy Command

To deploy a stack, use the following command:

docker stack deploy -c <compose-file.yml> <stack-name>

Replace <compose-file.yml> with the path to your Docker Compose file and <stack-name> with the desired name for the stack. For example:

docker stack deploy -c docker-compose.yml my-app

This command will deploy the application defined in docker-compose.yml to the swarm and name it my-app. Docker creates a standardized, encapsulated environment that runs applications.

Secrets Management

Storing Sensitive Data

When deploying applications, you often need to manage sensitive data such as passwords, API keys, and certificates. Storing this data directly in your Docker images or Compose files is not a secure practice. Docker Swarm provides a built-in secrets management feature to securely store and manage sensitive data.

Using Docker Secrets

Docker secrets are encrypted and stored in the swarm’s Raft log, ensuring that only authorized services can access them. To create a secret, use the docker secret create command:

docker secret create <secret-name> <secret-file>

Replace <secret-name> with the desired name for the secret and <secret-file> with the path to the file containing the secret data. For example:

docker secret create db_password db_password.txt

To use a secret in a service, add a secrets section to the service definition in your Compose file:

version: "3.7"
services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
secrets:
  db_password:
    external: true

This tells Docker to mount the db_password secret into the container at /run/secrets/db_password. The application can then read the secret from this file. The use of the _FILE suffix in the environment variable is a common convention for indicating that the value should be read from a file.

Configs Management

Managing Configuration Files

Similar to secrets, Docker Swarm provides a way to manage configuration files using configs. Configs allow you to store non-sensitive configuration data separately from your application code.

Using Docker Configs

To create a config, use the docker config create command:

docker config create <config-name> <config-file>

Replace <config-name> with the desired name for the config and <config-file> with the path to the file containing the configuration data. For example:

docker config create my_config my_config.txt

To use a config in a service, add a configs section to the service definition in your Compose file:

version: "3.7"
services:
  web:
    image: nginx:latest
    configs:
      - source: my_config
        target: /etc/nginx/conf.d/my_config.conf

This tells Docker to mount the my_config config into the container at /etc/nginx/conf.d/my_config.conf. The application can then read the configuration from this file.

Troubleshooting Docker Swarm

Common Issues and Solutions

Node Unavailability

One of the most common issues in Docker Swarm is node unavailability. A node might become unavailable due to hardware failures, network connectivity problems, or Docker daemon crashes. When a node becomes unavailable, the services running on that node will be affected. To troubleshoot node unavailability, start by checking the node’s status using the docker node ls command. This command will show you the current status of each node in the swarm. If a node is in the Down state, it means that the manager nodes are unable to communicate with it.

Solutions:

  • Check Network Connectivity: Verify that the node can communicate with the other nodes in the swarm, especially the manager nodes. Use tools like ping and traceroute to diagnose network connectivity problems. Ensure that there are no firewall rules blocking communication between the nodes.
  • Restart Docker Daemon: Try restarting the Docker daemon on the unavailable node. This can often resolve temporary issues that are preventing the node from communicating with the swarm. Use the command sudo systemctl restart docker to restart the Docker daemon.
  • Inspect Docker Daemon Logs: Check the Docker daemon logs on the unavailable node for any error messages or warnings that might indicate the cause of the problem. The logs are typically located in /var/log/docker.log.
  • Rejoin the Swarm: If the node is still unavailable after trying the above steps, you might need to remove it from the swarm and rejoin it. First, remove the node from the swarm using the docker node rm command on a manager node. Then, rejoin the node to the swarm using the docker swarm join command, as described in a previous chapter.

Service Deployment Failures

Another common issue is service deployment failures. A service might fail to deploy due to various reasons, such as invalid image names, incorrect port mappings, or resource constraints. When a service fails to deploy, Docker Swarm will attempt to reschedule the tasks on other available nodes.

Solutions:

  • Verify Image Name and Availability: Double-check the image name specified in the service definition to ensure that it is correct and that the image is available in the Docker registry. If the image is not available, Docker Swarm will be unable to pull it and deploy the service.
  • Check Port Mappings: Ensure that the port mappings specified in the service definition are correct and that there are no port conflicts. If two services are trying to use the same port on the same node, one of the services will fail to deploy.
  • Inspect Service Logs: Check the service logs for any error messages or warnings that might indicate the cause of the deployment failure. Use the docker service logs command to view the logs for a specific service.
  • Check Resource Constraints: Verify that the nodes in the swarm have sufficient resources (CPU, memory) to run the service. If the nodes are running out of resources, the service might fail to deploy. You can use the docker stats command to monitor the resource usage of the nodes.
  • Review Service Definition: Carefully review the service definition in the Docker Compose file or the docker service create command to ensure that all the options are correctly configured. Look for any typos or errors that might be causing the deployment failure.

Network Connectivity Problems

Network connectivity problems can also cause issues in Docker Swarm. Services might be unable to communicate with each other or with external resources due to network configuration errors or firewall rules. Docker Swarm uses overlay networks to enable communication between services running on different nodes.

Solutions:

  • Verify Overlay Network Configuration: Ensure that the overlay networks are correctly configured and that the services are attached to the correct networks. You can use the docker network ls command to list the networks in the swarm and the docker network inspect command to inspect the configuration of a specific network.
  • Check DNS Resolution: Verify that the services can resolve the DNS names of other services and external resources. If DNS resolution is not working correctly, the services will be unable to communicate with each other.
  • Inspect Container Network Configuration: Check the network configuration of the containers running the services to ensure that they have the correct IP addresses and routing rules. You can use the docker inspect command to inspect the network configuration of a container.
  • Firewall Rules: Ensure that there are no firewall rules blocking communication between the services. Check the firewall rules on the nodes and on any external firewalls that might be affecting network connectivity.

Debugging Techniques

Checking Logs

Checking logs is a crucial debugging technique for Docker Swarm. Docker provides several commands for viewing logs:

  • docker service logs <service-name>: This command displays the logs for all tasks (containers) associated with a service.
  • docker container logs <container-id>: This command displays the logs for a specific container. You can obtain the container ID using the docker service ps command.
  • docker system events: This command shows real-time events within Docker, including service updates, node status changes, and container creation/deletion.

Inspecting Nodes and Services

Inspecting nodes and services provides detailed information about their configuration and status. Use these commands:

  • docker node inspect <node-id>: This command displays detailed information about a specific node, including its ID, hostname, status, and resources.
  • docker service inspect <service-name>: This command displays detailed information about a specific service, including its configuration, tasks, and endpoints.

By carefully examining the logs and inspecting the configuration of nodes and services, you can effectively troubleshoot many common issues in Docker Swarm. Remember to use the docker swarm, docker node and docker service commands effectively when performing docker clustering to resolve issues related to container orchestration and leverage the benefits of swarm mode.

Conclusion

Docker Swarm offers a powerful and relatively simple solution for container orchestration, especially if you’re already invested in the Docker ecosystem. Its ease of use, integration with Docker Compose, and decentralized design make it a compelling choice for many use cases. While it might not possess all the advanced features of Kubernetes, Docker Swarm provides a solid foundation for deploying and managing containerized applications at scale. Whether you’re deploying a simple web application or a complex microservices architecture, Docker Swarm empowers you to efficiently manage your resources and ensure high availability.

As a container orchestration tool, it simplifies docker clustering and leverages swarm mode. Understanding docker swarm vs kubernetes is important to choosing the right tool, but Swarm is a great option, especially with this docker swarm tutorial. With its native clustering and orchestration for Docker, decentralized design for high availability, and easy to use CLI for managing swarm clusters, you’ll find it a valuable asset. You should feel empowered to start leveraging its capabilities.

Reference

Docker official site