DevOps GitLab CI/CD + Ansible Molecule Pipeline

8 February 2024 / Comments

Introduction

Streamlining the development, testing, and deployment of infrastructure and applications is a cornerstone of modern DevOps practices. GitLab CI/CD provides a robust platform for automating pipelines, and Ansible Molecule offers a powerful framework for testing Ansible roles prior to deployment. In this article, I'll demonstrate how to marry these tools to create a cohesive GitLab DevOps pipeline, including integration testing and provisioning.

You can find the source code for this project in my GitLab repository here:
Testing Ansible with Molecule in Docker

Key Components

GitLab CI/CD: The central orchestration platform for our automated build, test, and deploy processes.
Ansible: The configuration management and IT automation engine, used to define our infrastructure as code.
Ansible Roles: Modular and reusable Ansible code blocks to structure infrastructure configuration.
Molecule: A framework specifically designed for testing the logic and execution of Ansible roles.

The Pipeline

Our GitLab DevOps pipeline will consist of three primary stages:

Container Build:
- A Dockerfile defines a container image containing Ansible, Molecule, and necessary dependencies.
- GitLab CI/CD automates the image build and pushes it to a container registry (GitLab Container Registry).
Ansible Role Testing with Molecule:
- Molecule test scenarios are defined, ensuring our Ansible roles function as intended in the various architectures and OS.
- Tests are executed within the built container, providing isolation and consistency.
Provisioning:
- An Ansible playbook utilizes the tested roles to configure target infrastructure.
- GitLab CI/CD executes the playbook against designated environments (dev, staging, production).

Code example (.gitlab-ci.yml)

Build multi-arch Docker images rootless with Kaniko

Defined in .gitlab-ci.yml the CI the stages:

The GitLab Runners build the containers, for 2 architectures, arm64 and amd64, in separated instances.
Create Docker manifest for the 2 architectures and push the containers to GitLab Registry

Understanding the Structure in depth

Stages: The pipeline is divided into two stages
- build: This is where the Docker image is built for different architectures.
- multiarch-manifest: This creates a multi-architecture manifest to seamlessly support various architectures.
Jobs: We have two build jobs (kaniko-amd64, kaniko-arm64) and one manifest job (multiarch-manifest).

1. `kaniko-amd64` & `kaniko-arm64` Jobs

Purpose: Build Docker images for AMD64 (kaniko-amd64) and ARM64 (kaniko-arm64) architectures.
Image: Use the Kaniko executor image (gcr.io/kaniko-project/executor:debug) to build Docker images without Docker-in-Docker.
Variables
- KANIKO_ARGS: Additional flags for the Kaniko executor.
- KANIKO_BUILD_CONTEXT: The directory where the Dockerfile is located.
Runner Tags: associate the workload to specific runners
- kaniko-arm64 is specifically tagged to run on an ARM64, in this case a private runner.
- kaniko-amd64 is tagged to run on an AMD64 runner.

kaniko-amd64:

  variables:
    # Additional options for Kaniko executor.
    # For more details see https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#additional-flags
    KANIKO_ARGS: ""
    KANIKO_BUILD_CONTEXT: $CI_PROJECT_DIR

  stage: build
  image:
    # For latest releases see https://github.com/GoogleContainerTools/kaniko/releases
    # Only debug/*-debug versions of the Kaniko image are known to work within Gitlab CI
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]

  script:
    # if the user provide IMAGE_TAG then use it, else build the image tag using the default logic.
    # Default logic
    # Compose docker tag name
    # Git Branch/Tag to Docker Image Tag Mapping
    #   * Default Branch: main -> latest
    #   * Branch: feature/my-feature -> branch-feature-my-feature
    #   * Tag: v1.0.0/beta2 -> v1.0.0-beta2
    - |
      if [ -z ${IMAGE_TAG+x} ]; then
        if [ "$CI_COMMIT_REF_NAME" = $CI_DEFAULT_BRANCH ]; then
            VERSION="latest"
          elif [ -n "$CI_COMMIT_TAG" ];then
            NOSLASH=$(echo "$CI_COMMIT_TAG" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9.-]/}"
            VERSION="$SANITIZED"
          else \
            NOSLASH=$(echo "$CI_COMMIT_REF_NAME" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9-]/}"
            VERSION="branch-$SANITIZED"
          fi
        export IMAGE_TAG=$CI_REGISTRY_IMAGE:$VERSION-amd64
      fi
    - echo $IMAGE_TAG
    - mkdir -p /kaniko/.docker
    # Write credentials to access Gitlab Container Registry within the runner/ci
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    # Build and push the container. To disable push add --no-push
    - DOCKERFILE_PATH=${DOCKERFILE_PATH:-"$KANIKO_BUILD_CONTEXT/Dockerfile"}
    - /kaniko/executor --context $KANIKO_BUILD_CONTEXT --dockerfile $DOCKERFILE_PATH --destination $IMAGE_TAG $KANIKO_ARGS

  # Run this job in a branch/tag where a Dockerfile exists
  rules:
    - exists:
        - Dockerfile
    # only tags
    - if: "$CI_COMMIT_TAG"
    # custom Dockerfile path
    - if: $DOCKERFILE_PATH
    # custom build context without an explicit Dockerfile path
    - if: $KANIKO_BUILD_CONTEXT != $CI_PROJECT_DIR

2. `multiarch-manifest` Job

Purpose: Creates a multi-architecture manifest, enabling Docker to pull the appropriate image based on the user's system architecture.
Image: Uses a standard docker:latest image, as this task relies on the Docker CLI.
Services:
- docker:dind: Leverages Docker-in-Docker to provide Docker containers inside the GitLab runner.

stage: multiarch-manifest

  image:
    # For latest releases see https://github.com/GoogleContainerTools/kaniko/releases
    # Only debug/*-debug versions of the Kaniko image are known to work within Gitlab CI
    name: docker:latest
    entrypoint: [""]

  services:
    - docker:dind

  before_script:
    # Connect to GitLab Docker in Docker DIND
    - mkdir -pv ~/.docker
    - cp -v $DOCKER_TLS_CERTDIR/client/* ~/.docker   # Copy default docker certs
    - docker info   # Show connection to docker daemon
    # Login... if necessary
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY

  script:
    # The Docker script use another tag format `:7-8-5`, use Kaniko style for coerence
    - |
      if [ -z ${IMAGE_TAG+x} ]; then
        if [ "$CI_COMMIT_REF_NAME" = $CI_DEFAULT_BRANCH ]; then
            VERSION="latest"
          elif [ -n "$CI_COMMIT_TAG" ];then
            NOSLASH=$(echo "$CI_COMMIT_TAG" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9.-]/}"
            VERSION="$SANITIZED"
          else \
            NOSLASH=$(echo "$CI_COMMIT_REF_NAME" | tr -s / - )
            SANITIZED="${NOSLASH//[^a-zA-Z0-9-]/}"
            VERSION="branch-$SANITIZED"
          fi
        export IMAGE_TAG=$CI_REGISTRY_IMAGE:$VERSION
      fi
    # Example Kaniko tag: registry.gitlab.com/aleliga/docker-ansible-runner:7.8.5
    # Pull images
    - docker pull $IMAGE_TAG-arm64
    - docker pull $IMAGE_TAG-amd64
    # Fill variables for `docker manifest`
    - export MANIFEST_LIST=$IMAGE_TAG && export MANIFEST_BASE=$MANIFEST_LIST
    - echo "Debug vars $MANIFEST_LIST $MANIFEST_BASE"
    # Create docker manifest
    - echo "docker manifest create $MANIFEST_LIST $MANIFEST_BASE-arm64 $MANIFEST_BASE-amd64"
    - docker manifest create $MANIFEST_LIST $MANIFEST_BASE-arm64 $MANIFEST_BASE-amd64
    # Annotate arch
    - echo "docker manifest annotate --os linux --arch arm64 $MANIFEST_LIST $MANIFEST_BASE-arm64"
    - docker manifest annotate --os linux --arch arm64 $MANIFEST_LIST $MANIFEST_BASE-arm64
    - echo "docker manifest annotate --os linux --arch amd64 $MANIFEST_LIST $MANIFEST_BASE-amd64"
    - docker manifest annotate --os linux --arch amd64 $MANIFEST_LIST $MANIFEST_BASE-amd64
    # Push manifest
    - echo "docker manifest push $MANIFEST_LIST"
    - docker manifest push $MANIFEST_LIST

  rules:
    - exists:
        - Dockerfile
    # only tags
    - if: "$CI_COMMIT_TAG"
    # custom Dockerfile path
    - if: $DOCKERFILE_PATH
    # custom build context without an explicit Dockerfile path
    - if: $KANIKO_BUILD_CONTEXT != $CI_PROJECT_DIR

Run a Shell inside the Container

Useful for test, we can run an interactive shell inside the container, with mounted our Ansible code as a volume.

docker run -it --rm \
  -v $(pwd)/ansible:/ansible \
  registry.gitlab.com/aleliga/docker-ansible-molecule \
  bash

Add Collections to the Container

The container can be used as base image to add collections and roles

FROM registry.gitlab.com/aleliga/docker-ansible-molecule:latest
RUN python3 -m venv venv
ENV PATH="/venv/bin:$PATH"

# Add a collection
RUN ansible-galaxy collection install community.general

Run Ansible Molecule in GitLab CI/CD

A secondary CI pipeline, is dedicated to the integrations tests of Ansible Playbooks and Roles with molecule, using the previous built container

Molecule initialization script

This script is executed inside the docker-ansible-molecule container during the CI. Set up the environment (symbolic link, virtual environment activation), and then execute the provided Molecule subcommand on the project's Ansible directory.

#!/usr/bin/env bash
MOLECULE_COMMAND="$1"

# Create link to recreate path as in the container
ln -s $CI_PROJECT_DIR/ansible /ansible

# Enter ansible directory
PWD="/ansible" ; cd /ansible

# Add single variables to molecule
MOLECULE="true" \
    molecule $1

GitLab CI configuration

The .gitlab-ci.yaml define the image, run a forced cleanup, and molecule test

stages:
  - molecule-test

variables:
  CONTAINER_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_TAG"
  SHELL: /bin/bash
  CI_REGISTRY: registry.gitlab.com

before_script:
  - echo "Before script"
  - mkdir -p ~/.docker/
  - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" >> ~/.docker/config.json

molecule:
  image: registry.gitlab.com/aleliga/docker-ansible-molecule:latest
  stage: molecule-test
  services:
    - docker:dind
  tags:
    - private-runner
    - molecule
  script:
    - echo "Run molecule destroy"
    - "$CI_PROJECT_DIR/bin/molecule-ci-direct.sh destroy"
    - echo "Remove temp files"
    - rm -fr ~/.cache/molecule/
    - echo "Run molecule test"
    - "$CI_PROJECT_DIR/bin/molecule-ci-direct.sh test"

Deploy with Ansible in CD

The last pipeline, deploy the Ansible playbooks from GitLab Runner, using DinD (Docker in Docker)

variables:
  # Deploy repository
  PRIVATE_REPO: "gitlab.com/USER/PRIVATE_REPO.git"
  # Docker DIND vars
  DOCKER_HOST: "tcp://docker:2376"
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: 1

ansible-deploy:
  image: registry.gitlab.com/aleliga/docker-ansible-molecule:latest
  stage: deploy
  services:
    - docker:dind
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo -e $CI_SSH_PRIVATE_KEY
    - echo -e "$CI_SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - cd ~/
    - mkdir -p $PERSONAL_REPO_DIR
    - apt-get update && apt-get install git -qqy
  script:
    - git clone https://"$ANSIBLE_USER":"$ANSIBLE_DEPLOY_TOKEN"@"$PRIVATE_REPO" $PERSONAL_REPO_DIR
  # Run Ansible "upgrade-hosts" to deploy
    - cd $PERSONAL_REPO_DIR
    - echo "Deploy to servers"
    - bin/gitlabci-upgrade-hosts.sh

Benefits

Enhanced Reliability: Automated testing with Molecule increases confidence in Ansible roles, preventing errors from slipping into production environments.
Faster Iteration: A well-defined CI/CD pipeline enables faster development and changes with a higher degree of safety.
Infrastructure as Code: Ansible ensures predictable and repeatable deployments across environments.

Considerations

Secure Variable Handling: Use GitLab CI/CD Secret Variables and Vault solutions for storing sensitive data like credentials.
Environment Management: Consider separating pipeline configurations for development, staging, and production environments.

References

Update September 2024

From the release of GitLab 17, the cloud DevOps infrastructure, provide shared ARM64 SaaS Runners, can be used defining the correct tags in kaniko-arm64 and other jobs, but as of September 2024 are only for Premium and Ultimate customers.

To use the GitLab ARM64 hosted runners, we can set the appropriate tags in the CI stage, for example:

  tags:
   -  saas-linux-medium-arm64

Source: GitLab Hosted runners on Linux

Conclusion

Integrating GitLab CI and Ansible Molecule empowers a solid DevOps workflow. It ensures reliable infrastructure changes, accelerates delivery time, and reduces the risk of deployment errors.

Feel free to explore the repository, contribute, or use it as a reference. Let me know if you have any feedback or suggestions!

If you need an expert integration for your Ansible pipelines, try our DevOps consultancy for free

DevOps

Linux

Orario

DevOps GitLab CI/CD + Ansible Molecule Pipeline

Introduction

Key Components

The Pipeline

Code example (.gitlab-ci.yml)

Build multi-arch Docker images rootless with Kaniko

Understanding the Structure in depth

1. `kaniko-amd64` & `kaniko-arm64` Jobs

2. `multiarch-manifest` Job

Run a Shell inside the Container

Add Collections to the Container

Run Ansible Molecule in GitLab CI/CD

Molecule initialization script

GitLab CI configuration

Deploy with Ansible in CD

Benefits

Considerations

References

Update September 2024

Conclusion

Categorie Post: Tags Post

Post: Post recenti (Sidebar)

Orario

Portfolio

DevOps GitLab CI/CD + Ansible Molecule Pipeline

Introduction

Key Components

The Pipeline

Code example (.gitlab-ci.yml)

Build multi-arch Docker images rootless with Kaniko

Understanding the Structure in depth

1. kaniko-amd64 & kaniko-arm64 Jobs

2. multiarch-manifest Job

Run a Shell inside the Container

Add Collections to the Container

Run Ansible Molecule in GitLab CI/CD

Molecule initialization script

GitLab CI configuration

Deploy with Ansible in CD

Benefits

Considerations

References

Update September 2024

Conclusion

Categorie Post: Tags Post

Post: Post recenti (Sidebar)

1. `kaniko-amd64` & `kaniko-arm64` Jobs

2. `multiarch-manifest` Job