Tag: Packer

Posted on by

Building a Template Image using Packer and Ansible

In this guide, we will walk through the process of building template images using Packer and Ansible. We will be using a pre-configured repository that contains all the necessary files and configurations to streamline the build process. By the end of this guide, you will have a solid understanding of how to build images for different providers using Packer and Ansible.

Prerequisites

Before we begin, ensure that you have the following software and requirements met:

sudo apt install zfsutils-linux

Note: For Bhyve Images, This setup could also work for other operating systems that support ZFS, VirtualBox, Packer, and Ansible.

Step 1: Clone the Repository

  1. Open a terminal or command prompt.
  2. Navigate to the directory where you want to clone the repository.
  3. Run the following command to clone the repository with submodules:
    git clone --recursive https://github.com/STARTcloud/vagrant_box_template_creator
  4. Change into the cloned repository directory:
    cd vagrant_box_template_creator/builder

Step 2: Configuration

  1. Navigate to the definitions folder:
    cd definitions
  2. Create a cloud-credentials.json file based on the cloud-credentials-example.json file:
    cp cloud-credentials-example.json cloud-credentials.json
  3. Open the cloud-credentials.json file in a text editor and fill in the necessary secrets for pushing images to cloud repositories. Replace the placeholder values with your actual credentials.
  4. Review the vendor.json file and update it with your organization-specific details, such as product URL, vendor URL, vendor name, and vendor domain.
  5. Choose the desired operating system template from the definitions/templates folder. In this example, we’ll use debian12-server.json for Debian 12.

Step 3: Building the Base Image

  1. Open a terminal and navigate to the root directory of the cloned repo:
    cd vagrant_box_template_creator/builder
  2. Run the following command to build the base image using Packer and Ansible-Local:

    packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' tasks/build-ansible-local.json

    This command uses the build-ansible-local.json file as the main Packer build file and incorporates variables from debian12-server.json, vendor.json, and cloud-credentials.json.
  3. Packer will start the build process and use VirtualBox to create the base image. The build process will take some time, and you can monitor the progress in the terminal.
  4. Once the build is complete, the base image will be stored as an OVA file in the temp directory.

Step 4: Creating Provider-Specific Images and Uploading to Vagrant Cloud

After building the base image, you can convert it to other formats for different providers and upload them to BoxVault (or Vagrant Cloud) using the following commands:

VirtualBox

packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' providers/virtualbox/publish.json

This command will create a VirtualBox-compatible image using the publish.json file in the providers/virtualbox folder. The resulting image will be stored in the providers/virtualbox/boxes folder and uploaded to Vagrant Cloud.

Zone (Bhyve)

packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' providers/zones/publish.json

This command will create a Zone (Bhyve)-compatible image using the publish.json file in the providers/zones folder. The resulting image will be stored in the providers/zones/boxes folder and uploaded to Vagrant Cloud.

AMI (Amazon Machine Image)

packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' providers/ami/publish.json

This command will create an AMI (Amazon Machine Image) using the publish.json file in the providers/ami folder. The resulting image will be stored in the providers/ami/boxes folder and uploaded to Vagrant Cloud.

Docker

packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' providers/docker/publish.json

This command will create a Docker image using the publish.json file in the providers/docker folder. The resulting image will be stored in the providers/docker/boxes folder and uploaded to Vagrant Cloud.

Accessing Images on Vagrant Cloud

Once the images are uploaded, you can find them under your respective organization on Vagrant Cloud. For example, the Debian 12 server image can be accessed at:

https://portal.cloud.hashicorp.com/vagrant/discover/STARTcloud/debian12-server

Each of these commands uses the respective publish.json file located in the providers folder to build, publish, and upload the image for the specific provider. The publish.json files contain the necessary configuration and provisioning steps for each provider, including the upload to Vagrant Cloud.

Step 5: Customizing the Build Process

The build process can be customized and extended to fit your specific requirements. Here are a few key areas you can explore:

Ansible Playbooks

  • The Ansible playbooks used for provisioning the image are located in the provisioners/ansible/playbooks folder.
  • The main playbook for building the image with Ansible-Local is build-ansible-local-playbook.yml.
  • You can customize the playbook and roles to add additional provisioning steps or modify the existing configuration.

Preseed and Shell Scripts

  • Preseed files for different operating systems and types (server/desktop) are located in the provisioners/preseed folder.
  • Shell scripts for various provisioning tasks are located in the provisioners/shell folder.
  • You can modify or add new scripts to perform additional provisioning tasks specific to your needs.

Temporary Files and Output

  • During the build process, temporary files and output images are stored in the temp folder.
  • The final built images for each provider can be found in their respective boxes folders under the providers folder.

Step 6: Cleaning Up

After the build process is complete and you have obtained the desired images, you can clean up the temporary files and artifacts by running the following command:

packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' tasks/cleanup.json

This command uses the cleanup.json file in the tasks folder to remove the temporary files and artifacts generated during the build process.

Conclusion

Congratulations! You have now learned how to build Packer images using Packer and Ansible based on the provided setup. You can use this knowledge to create custom images for different providers and automate the provisioning process.

Remember to review and update the configuration files, credentials, and templates according to your specific needs. Feel free to explore the different folders and files to gain a deeper understanding of the build process and make any necessary modifications.

If you encounter any issues or have further questions, refer to the official documentation of Packer and Ansible for more information and troubleshooting steps.

Happy building!

Posted on by

How to Automate building a docker container using Packer and Ansible from a templated image

Introduction

This guide will help you build a Docker container for the Moonshine-dev application, which is written in Haxe with a Gradle REST API. The container will consist of three layers:

  1. Base Layer: A Debian 12-based image prepared using Packer and Ansible, as referenced here: Building a Template Image using Packer and Ansible
  2. Intermediate Layer: Common provisioning steps to speed up future builds.
    moonshine-dev/base-latest
  3. Application Layer: Installation and configuration of the Moonshine-dev application.
    moonshine-dev/latest

See more here about Image Layers in Docker

Prerequisites

Ensure you have the following software and requirements met:

  • Super.Human.Installer (SHI) Instance: Use a SHI instance, which is a GUI wrapper around Vagrant. This instance is based on a STARTcloud template and already has Ansible installed.
  • Docker: Install Docker on the SHI instance. You can follow the official Docker installation guide for your operating system. or add the role startcloud.startcloud_roles.docker

Step 1: Clone the Repository

  1. Open a terminal in your SHI instance.
  2. Navigate to the directory where you want to clone the repository.
  3. Run the following command to clone the repository with submodules:
    git clone --recursive https://github.com/STARTcloud/vagrant_box_template_creator
  1. Change into the cloned repository directory:
    cd vagrant_box_template_creator

Step 2: Build the Base Layer

The base layer is a Debian 12-based image that serves as a foundation for other applications. It is built using Packer and Ansible.

Base Playbook

The base playbook for all STARTcloud images is as follows, this was referenced in the other article: Building a Template image with Packer and Ansible:

---
- name: "This Playbook Creates the Base Template via Ansible-Local"
  become: true
  gather_facts: true
  hosts: all
  collections:
    - startcloud.startcloud_roles
  roles:
    - role: startcloud.startcloud_roles.dependencies
    - role: startcloud.startcloud_roles.serial
    - role: startcloud.startcloud_roles.cockpit
    - role: startcloud.startcloud_roles.nfs
      vars:
        nfs_exports: []
        nfs_rpcbind_enabled: true
        nfs_rpcbind_state: started
    - role: startcloud.startcloud_roles.ntp
      vars:
        ntp_area: ""
        ntp_cron_handler_enabled: false
        ntp_enabled: true
        ntp_manage_config: false
        ntp_restrict:
          - "127.0.0.1"
          - "::1"
        ntp_servers:
          - "ntp1.prominic.net iburst"
          - "ntp2.prominic.net iburst"
        ntp_timezone: America/Chicago
        ntp_tinker_panic: false
    - role: startcloud.startcloud_roles.motd
      vars:
        add_footer: false
        add_update: true
        remove_default_config: true
        restore_default_config: false
        sysadmins_email: [email protected]
        sysadmins_signature: "STARTCloud Contact Email"
    - role: startcloud.startcloud_roles.cleanup

Build the Base Image

Run the following command to build the base image:
packer build -var-file='definitions/cloud-credentials.json' -var-file='definitions/vendor.json' -var-file='definitions/templates/x64/debian12-server.json' tasks/build-ansible-local.json

This command uses the build-ansible-local.json file to create a VirtualBox VDI file, which is then converted to a Docker image and pushed to both Vagrant Cloud and Docker Hub.

Step 3: Build the Intermediate Layer

The intermediate layer includes common provisioning steps to speed up future builds. It is built using the same Packer script (deploy.json) as the application layer.

Intermediate Layer Playbook

The playbook for the intermediate layer is as follows:

---
- name: "Setup Intermediate Layer"
  become: true
  gather_facts: true
  hosts: all
  roles:
    - name: startcloud.startcloud_roles.setup
    - name: startcloud.startcloud_roles.hostname
    - name: startcloud.startcloud_roles.dependencies
    - name: startcloud.startcloud_roles.service_user
    - name: startcloud.startcloud_roles.sdkman_install
    - name: startcloud.startcloud_roles.sdkman_java
    - name: startcloud.startcloud_roles.sdkman_gradle
    - name: startcloud.startcloud_roles.ssl
    - name: startcloud.startcloud_roles.supervisord

Build the Intermediate Layer

Run the following command to build the intermediate layer:

VERSION=0.0.1 sudo -E packer build -on-error=abort \
  -var-file='definitions/cloud-credentials.json' \
  -var-file='definitions/vendor.json' \
  -var-file='definitions/templates/x64/debian12-server.json' \
  -var "docker_hub_template_repo_name=debian12-server" \
  -var "docker_hub_template_repo_tag=0.0.8" \
  -var "playbook_file=provisioners/ansible/ansible_collections/moonshine/moonshine_roles/playbooks/main-base-playbook.yml" \
  -var "repo_version=base-latest" \
  -var 'repo_name=moonshine-dev' \
  providers/docker/deploy.json

Step 4: Build the Application Layer

The application layer installs and configures the Moonshine-dev application. It uses the intermediate layer as its base.

Application Layer Playbook

The playbook for the application layer is as follows:

---
-
  name: "Generating Playbook"
  become: true
  gather_facts: true
  hosts: all
  vars:
    core_provisioner_version: 0.0.1
    provisioner_name: PackerImageBuilder
    provisioner_version: 0.0.1
    settings:
      hostname: app
      domain: moonshine.dev
      server_id: 500001
      vagrant_user_pass: 'XaVuzq2vRV4fTk'
    debug_all: true
    selfsigned_enabled: true
    haproxy_ssl_redirect: true
    letsencrypt_enabled: false
    service_user: java_user
    service_group: java_group
    service_home_dir: /local/notesjava
    cert_dir: /secure
    installer_dir: /vagrant/installers
    completed_dir: /vagrant/completed
    domino_organization: STARTcloud
    domino_install_dir: /opt/hcl/domino/notes/latest/linux

  collections:
    - startcloud.startcloud_roles
    - startcloud.hcl_roles
    - moonshine.moonshine_roles

  roles:
    - name: startcloud.hcl_roles.domino_vagrant_rest_api
    - name: startcloud.startcloud_roles.haxe
    - name: moonshine.moonshine_roles.moonshinedev_deploy
    - name: startcloud.startcloud_roles.haproxy
      vars:
        haproxy_cfg: /vagrant/ansible/ansible_collections/moonshine/moonshine_roles/roles/moonshinedev_deploy/templates/moonshinedev-haproxy.cfg.j2

Build the Application Layer

Run the following command to build the application layer, note that the repo_name and repo_version would be based off your intermediary image/layer above:

VERSION=0.0.1 sudo -E packer build -on-error=abort \
  -var-file='definitions/cloud-credentials.json' \
  -var-file='definitions/vendor.json' \
  -var-file='definitions/templates/x64/debian12-server.json' \
  -var "docker_hub_template_repo_name=moonshine-dev" \
  -var "docker_hub_template_repo_tag=base-latest" \
  -var "playbook_file=provisioners/ansible/ansible_collections/moonshine/moonshine_roles/playbooks/main-application-playbook.yml" \
  -var "repo_version=latest" \
  -var 'repo_name=moonshine-dev' \
  providers/docker/deploy.json

Step 5: Testing the Image

After building the Docker image, you can test it by running the following command:

docker run -p 80:80/tcp -p 443:443/tcp -p 8080:8080/tcp -i -t startcloud/moonshinedev:latest /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf

or

docker run -p 80:80/tcp -p 443:443/tcp -p 8080:8080/tcp -i -t startcloud/moonshinedev:latest

This command will start the Moonshine-dev application in a Docker container, allowing you to test its functionality.

If you need to debug the docker container, you can access it with the following commands
1. list the docker container and grab its name or id:

sudo docker container ls

2. The you can use exec to access the container:

sudo docker container exec -it <CONTAINER_ID_OR_NAME> /bin/bash

Conclusion

By following this guide, you have successfully built a multi-layer Docker container for the Moonshine-dev application. Each layer serves a specific purpose, from providing a base Debian 12 image to installing and configuring the application itself. This modular approach allows for efficient and flexible image management, making it easier to update and maintain the application over time.