Golden Images and Proxmox Templates with Packer

Table of Contents

Intro #

Hashicorp Packer is an open-source tool that lets you build consistent and reproducible virtual machines and containers from a configuration file. It supports various platforms and provides a wide range of features for building custom images. It is highly extensible, with broad community and official support for major cloud providers, local hypervisors and virtualization software via builder plugins.

A configuration file consist of several top-level code blocks - source, variable and build, with each block containing sets of key-value pairs. The source block configures a builder plugin, typically defining machine attributes, e.g. vcpus, and platform access, e.g. API tokens. You can set these values using variables that are defined in variable blocks with default values and HCL types. Lastly, the build block is used to combine builders and provisioners to create a final image.

For this tutorial, we’ll use the Proxmox plugin and it’s proxmox-iso builder to create a Proxmox template based on Ubuntu server 22.04. We’ll craft the template so that meets the following criteria:

All packages are up-to-date and cloud-init, openssh-server and qemu-guest-agent are pre-installed.
There are no pre-existing users or credentials on the image.
The image is reset to a clean state, so that it can be fully configured using cloud-init.

To achieve this, we’ll use several tricks in the autoinstall configuration file and build the image using only the root user.

For a universal Packer configuration to create Debian, CentOS, Fedora and Ubuntu templates, see the companion repo: Packer Proxmox Templates.

Grant Packer Access to Proxmox #

Packer configures the VM template by making API request to a Proxmox Virtual Environment (PVE) endpoint. To grant Packer access, let’s use the Proxmox CLI to create a new role, group with permissions, and user; then generate an access token for the packer user. For additional configuration options see Proxmox Wiki: User Management or pveum help.

From the PVE CLI enter the following commands:

# create role
pveum role add PackerUser --privs "Datastore.AllocateSpace \
  Datastore.AllocateTemplate Datastore.Audit Sys.Audit Sys.Modify \
  VM.Allocate VM.Audit VM.Clone VM.Config.CDROM
  VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk \
  VM.Config.HWType VM.Config.Memory VM.Config.Network \
  VM.Config.Options VM.Console VM.Monitor VM.PowerMgmt"

# additional permissions for PVE 8
pveum role modify PackerUser --privs "SDN.Use" --append

# create group
pveum group add packer-users

# add permissions
pveum acl modify / -group packer-users -role PackerUser

# create user 'packer'
pveum useradd packer@pve -groups packer-users

# generate a token
pveum user token add packer@pve token -privsep 0

The last command will output a token value similar to the following, which we will use to validate Packer with Proxmox.

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ packer@pve!token                     │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ 782a7700-4010-4802-8f4d-820f1b226850 │
└──────────────┴──────────────────────────────────────┘

Packer: Configure the Build #

In this tutorial, we will create several files: pve-image.pkr.hcl, pve-vars.pkr.hcl, meta-data and user-data. The following sections discuss the logic behind each file and Packer code using snippets with complete code examples provided at the end. Code examples primarily use hard-coded values for simplicity, however, consider replacing these values with variables, using pve-vars.pkr.hcl as a reference.

Source Block: Writing a Reusable Top-Level Source Block #

Start by creating a top-level source block that uses the proxmox-iso builder with a simple but unique identifier, image. This block will be used as a common source configuration for all of the builds, with the goal of defining stable common values such as: 1) Proxmox login credentials; 2) SSH login credentials for Packer; and 3) PVE template parameters. For enhanced readability, group similar values together and use comment headers to demarcate each group. Later, we will create a build-level source block that uses distribution specific configuration values such as the ISO image and template name.

All of the templates will use cloud-init to configure the machine image, so we’ll attach a cloud-init drive and add a single boot disk to install the base ISO onto. Additionally, we will install qemu-guest-agent using cloud-init to enable PVE management of VMs cloned from a template, so enable qemu_agent in the Packer configuration. Furthermore, we will add a dynamic description that includes the creation date using the built-in timestamp function.

Packer variable blocks do not support using functions in the default value, so it’s best to hard code template_description. If we used this value in multiple locations, using a local that supports functions would be more appropriate.

To mask sensitive values from Packer’s output and pass via environment variables, we’ll use variables for PVE and SSH login credentials, e.g. var.pve_token. In the next section , we’ll create the corresponding variable blocks.

# pve-image.pkr.hcl
source "proxmox-iso" "image" {
  // PVE login
  proxmox_url              = var.pve_api_url
  username                 = var.pve_username
  token                    = var.pve_token
  node                     = var.pve_node
  insecure_skip_tls_verify = true

  // SSH
  ssh_username              = var.ssh_username
  ssh_keypair_name          = var.ssh_keypair_name
  ssh_private_key_file      = var.ssh_private_key_file
  ssh_clear_authorized_keys = true
  ssh_timeout               = "20m"

  // ISO
  ...
  template_description = "Packer generated template image on ${timestamp()}"

  // System
  ...
  qemu_agent = true

  // Disks
  disks {
    type         = "scsi"
    storage_pool = "local-lvm"
    disk_size    = "10G"
    cache_mode   = "writeback"
    format       = "raw"
  }

  // Cloud-init
  cloud_init              = true
  cloud_init_storage_pool = "local-lvm"

  // CPU & Memory
  ...

  // Network
  ...
}

Variable Blocks: Masking Sensitive Data & Validating Inputs #

Next, let’s create several variables using variable blocks to validate inputs, set reasonable defaults and mask sensitive data. This will make it easier to pass in sensitive and non-sensitive values via environment variables and command line arguments, respectively.

Packer expects all variables passed via the CLI to be strings, but you can specify the HCL type for each variable by setting type, ensuring that type constraints and transformations are properly applied. You can also validate a input further using validation rules within a validation block. In the example below, we’ll validate that the pve_api_url input begins with http(s):// and ends with /api2/json using the built-in regex function. The can function converts the regex output into a boolean for the validation rule to use as a conditional.

Setting a default value for a variable ensures that Packer will always pass a value when building an image, turning it into an optional parameter for the end-user. Conversely, excluding the default key will cause Packer to fail if a value is not passed via a variable file or CLI. Setting default = null is a special use-case, in which Packer skips checking for an input and assumes the plugin builder key using the variable is an optional parameter and handles null values.

To validate a variable configuration file against the schema, pass dummy values for undefined variables to packer validate:

packer validate \
  -var 'pve_token=TOKEN' \
  -var 'pve_username=packer@pve!token' \
  -var 'pve_api_url=http://pve.example.com/api2/json' \
  pve-vars.pkr.hcl

Next, let’s set all PVE login and SSH variables to sensitive = true to mask these values from Packer’s logs.

# pve-vars.pkr.hcl
// PVE Variables
...

variable "pve_api_url" {
  description = "Proxmox API Endpoint, e.g. 'https://pve.example.com/api2/json'"
  type        = string
  sensitive   = true
  validation {
    condition     = can(regex("(?i)^http[s]?://.*/api2/json$", var.pve_api_url))
    error_message = "Proxmox API Endpoint Invalid. Check URL - Scheme and Path required."
  }
}

// SSH Variables
variable "ssh_private_key_file" {
  description = "Private SSH Key for VM"
  default     = "~/.ssh/packer_id_ed25519"
  type        = string
  sensitive   = true
}
...

Build Block: Extending Top-Level Source Block Configurations #

In this last part, we’ll create a build block and extend the top-level source block configuration from above with distribution specific boot commands and cloud-init configuration files. Additionally, we’ll use a built-in provisioner to run several shell commands to prepare the image for provisioning.

To extend a top-level source block, we’ll create a build-level source block that references the builder and identifier, proxmox-iso.image, of the top-level block and give it a unique name, i.e. ubuntu22. This will allow us to reference this specific build configuration in the CLI using the only or except flags, e.g. packer build -only=proxmox-iso.ubuntu22. This name is not to be confused with the template_name, which is the template name in Proxmox.

Here we’ll also define variables that are specific to a given distribution, iso_url and iso_checksum, and set the template ID to 9000. I’ll cover the additional parts and provider block over the next few sections.

# pve-image.pkr.hcl
...

build {
  source "proxmox-iso.image" {
    name          = "ubuntu22" # Packer Reference Name
    template_name = "ubuntu22" # Proxmox Template Name
    vm_id         = 9000
    iso_url       = "https://releases.ubuntu.com/22.04/ubuntu-22.04.2-live-server-amd64.iso"
    iso_checksum  = "file:https://releases.ubuntu.com/22.04/SHA256SUMS"
    boot_wait     = "5s"
    boot_command = [
      "c",
      "linux /casper/vmlinuz --- autoinstall 'ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/' ",
      "<enter><wait>",
      "initrd /casper/initrd",
      "<enter><wait>",
      "boot<enter>"
    ]
    http_content = {
      "/meta-data" = file("configs/meta-data")
      "/user-data" = templatefile("configs/user-data",
        {
          var            = var,
          ssh_public_key = chomp(file(var.ssh_public_key_file))
      })
    }
  }

  provisioner "shell" {
    inline = [
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
      // clean image identifiers
      "cloud-init clean --machine-id --seed",
      "rm /etc/hostname /etc/ssh/ssh_host_* /var/lib/systemd/random-seed",
      "truncate -s 0 /root/.ssh/authorized_keys",
      // disable SSH password authentication and root access
      "sed -i 's/^#PasswordAuthentication\\ yes/PasswordAuthentication\\ no/' /etc/ssh/sshd_config",
      "sed -i 's/^#PermitRootLogin\\ prohibit-password/PermitRootLogin\\ no/' /etc/ssh/sshd_config"
    ]
  }
}

Distribution Specific Boot Commands #

The proxmox-iso builder boot command provides a set of special keys to interact with an image’s boot loader. Also, it can generate an HTTP server and serve files defined in http_content, which is useful for providing configuration files. Here, we will escape GRUB’s automatic boot sequence, c, and instruct the kernel to use autoinstall to skip the guided menu prompts and set the datasource as ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/, which will instruct cloud-init to fetch the autoinstall configuration from Packer’s HTTP server.

# pve-image.pkr.hcl
build{
  source "proxmox-iso.image" {
    boot_wait      = "5s"
    boot_command = [
      "c",
      "linux /casper/vmlinuz --- autoinstall 'ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/' ",
      "<enter><wait>",
      "initrd /casper/initrd",
      "<enter><wait>",
      "boot<enter>"
    ]
    ...
  }
}

Autoinstall Configuration File #

For the subiquity autoinstall, we’ll create two files: meta-data and user-data. The meta-data file is a single line file that contains the word uninitialized. While the user-data file is a mixture of autoinstall directives and cloud-init directives. For the user-data file, we’ll create a Packer template that uses variables to define several key values. The ssh_public_key_file variable is transformed using built-in functions and assigned to a new key prior to being passed into the template; while all other variables are passed by including: var = var.

# pve-image.pkr.hcl
build{
  source "proxmox-iso.image" {
    ...
    http_content = {
      "/meta-data" = file("configs/meta-data")
      "/user-data" = templatefile("configs/user-data",
        {
          var            = var,
          ssh_public_key = chomp(file(var.ssh_public_key_file))
      })
    }
  }
}

For the user-data file, we’ll avoid creating a default user and instead use the root user during the build process. Autoinstall requires that either the identity or user-data section be present in the configuration. Using identity is a non-starter as it will create a new user with a password that is required for all calls to sudo, which complicates using commands in the shell provisioner. Instead we’ll use the user-data directive and skip creating a user by supplying an empty list, users: []; and enable root SSH access by setting disable_root: false. We’ll also disable password authentication for all users with ssh.allow-pw: false and add a SSH key for Packer to use during the build. Both the root SSH key and access will be removed at the end of the build with the shell provisioner (next section).

By using the root user to build the image, we avoid creating the standard default user, ubuntu, or another one that will remain on the image after the build - even if you set a different name for the default user in cloud-init. This approach does require that a user is created during provisioning with a password and/or SSH key, but this is easily configured using the PVE GUI or a provisioning tool like Terraform (discussed later).

One of the advantages of using a Packer template is the ability to conditionally apply key-values. In the example below, if values are supplied for apt_proxy_http and/or apt_proxy_https, then an APT proxy configuration file is created at /etc/apt/apt.conf.d/90curtin-aptproxy; otherwise, these keys are never generated in the configuration file.

#cloud-config
autoinstall:
  version: 1
  user-data:
    disable_root: false   # allow SSH for root
    users: []             # skip creating default users
  locale: en_US.UTF-8
  packages:
    - qemu-guest-agent
  refresh-installer:
    update: true
  shutdown: reboot
  ssh:
    install-server: true  # enable SSH server
    allow-pw: false       # disable password authentication
    authorized-keys:
      - ${ssh_public_key} # add SSH key to root user
  timezone: UTC
  updates: all            # update all packages
  apt:
    %{ if var.apt_proxy_http != "" }
    http_proxy: ${var.apt_proxy_http}
    %{ endif }
    %{ if var.apt_proxy_https != "" }
    https_proxy: ${var.apt_proxy_https}
    %{ endif }

Additional top-level keys can be found on the autoinstall configuration site.

Preparing the Image for Provisioning #

Finally, to prepare the image for provisioning we’ll use the shell provisioner to reset cloud-init; remove the hostname; and re-configure SSH access. The while ... line is a common practice to ensure that cloud-init finishes prior to shutting down the instance or running additional commands.

To remove the /var/lib/cloud/ directory and clear /etc/machine-id, add cloud-init clean --machine-id --seed. Additionally, we’ll clear /var/lib/systemd/random-seed to avoid clones from starting with the same random pool. To secure SSH access, we’ll clear the SSH keys generated during the build and used by Packer for the root user; and pre-configure the OpenSSH server to disable password authentication and root access.

# pve-image.pkr.hcl
build{
  ...
  provisioner "shell" {
    inline = [
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
      // clean image identifiers
      "cloud-init clean --machine-id --seed",
      "rm /etc/hostname /etc/ssh/ssh_host_* /var/lib/systemd/random-seed",
      "truncate -s 0 /root/.ssh/authorized_keys",
      // disable SSH password authentication and root access
      "sed -i 's/^#PasswordAuthentication\\ yes/PasswordAuthentication\\ no/' /etc/ssh/sshd_config",
      "sed -i 's/^#PermitRootLogin\\ prohibit-password/PermitRootLogin\\ no/' /etc/ssh/sshd_config"
    ]
  }
}

For additional information on cleaning images prior to provisioning, see systemd: Building Images.

Packer: Build the Image #

With the completed configuration files, we can now build our image and PVE template. Packer allows you to set variables using: 1) environment variables, prefixing the variable name with PKR_VAR_; 2) CLI args, using the -var= flag; and/or 3) variable files, having the extension *.pkrvars.hcl. In the following example we’ll set two environment variables for our Proxmox API token, then pass the Proxmox URL via the CLI.

export PKR_VAR_pve_username='packer@pve!token'
export PKR_VAR_pve_token='782a7700-4010-4802-8f4d-820f1b226850'

# build packer image
packer build \
  -var='pve_api_url=https://pve.example.com/api2/json' \
  .

After the image is built, you should see a new template in your Proxmox GUI. Test it out by cloning the template, then adding a default user in the cloud-init section. You must add a password for console access and/or a SSH key for remote access. Leaving the user field blank will default to creating a user named ubuntu. Also, you must configure the ‘IP Config’ section.

Proxmox GUI showing the virtual machine cloud-init settings panel

Recap #

Packer provides a streamlined approach for building Proxmox VM templates. This post explored the steps involved in creating a custom Proxmox template using Packer, covering builder plugin configurations, variable definitions, and image sanitation. By leveraging Packer, you can automate the process of building and managing your Proxmox infrastructure, ensuring consistency and efficiency in your VM deployments.

By using a common top-level source block, you can easily extend this the build block to create multiple templates from different linux distributions. Furthermore, by building your images using only the root user, you can ensure that your images are free of any user level configurations and ready for cloud-init.

For ideas on how to extend this template for other operating systems, please refer to the companion repo: Packer Proxmox Templates.

Limitations #

Avoid adding a cloud-init network-config file to the template, as you will want to configure network settings using Proxmox’s cloud-init integration. Also the network configuration is cleared at the end of the template creation via cloud-init clean.
The Proxmox plugin does not provide a way to configure cloud-init settings that are available in the Proxmox cloud-init GUI, Proxmox plugin - issue #7. This only leaves the option of manually configuring the values via Proxmox or third-party provisioner, e.g. Terraform. Note: With plugin version 1.1.3, you can use the “clone” builder to add cloud-init networking information, however, the information is removed during template finalization. Also the “clone” builder is only for creating template clones - not to be confused with “clones” in Proxmox, which is a way to provision VMs from Proxmox templates.
You are unable to use *.qcow2 cloud images as a base OS, see Proxmox plugin - issue #29.

Building the Image With a Non-Root User #

If you decide create a new user using the identity or user-data in the autoinstall configuration, you will need to provide a password for sudo in the shell provisioner. This can be done by piping the password into sudo -S from stdin as follows:

# pve-image.pkr.hcl
  provisioner "shell" {
    env = {
      USER_PASSWORD = var.ssh_password
    }
    inline = [
      ...
      // clean image identifiers
      "echo $USER_PASSWORD | sudo -S cloud-init clean --machine-id --seed",
      ...
    ]
  }

Change and add the following variables:

# pve-vars.pkr.hcl
variable "ssh_username" {
  description = "SSH Username"
  type        = string
  default     = "packer"
}

variable "ssh_password" {
  description = "SSH User Password"
  type        = string
  default     = "password"
  sensitive   = true
}

Update the autoinstall configuration:

#cloud-config
autoinstall:
  version: 1
  user-data:
    users:
      - name: ${var.ssh_username}
        groups: users, admin
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        ssh_authorized_keys:
          - ${ssh_public_key}
  ...
  ssh:
    install-server: true
  timezone: UTC

See cloud-init module ‘users’, for more configuration options.

Complete Code Examples #

pve-image.pkr.hcl #

packer {
  required_plugins {
    proxmox = {
      version = ">=1.1.2"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

source "proxmox-iso" "image" {
  // PVE login
  proxmox_url              = var.pve_api_url
  username                 = var.pve_username
  token                    = var.pve_token
  node                     = var.pve_node
  insecure_skip_tls_verify = true

  // SSH
  ssh_username              = var.ssh_username
  ssh_keypair_name          = var.ssh_keypair_name
  ssh_private_key_file      = var.ssh_private_key_file
  ssh_clear_authorized_keys = true
  ssh_timeout               = "20m"

  // ISO
  iso_download_pve     = true # added in v1.1.2
  iso_storage_pool     = "local"
  unmount_iso          = true
  os                   = "l26"
  template_description = "Packer generated template image on ${timestamp()}"

  // System
  machine    = "q35"
  bios       = "seabios"
  qemu_agent = true

  // Disks
  scsi_controller = "virtio-scsi-pci"
  disks {
    type         = "scsi"
    storage_pool = "local-lvm"
    // storage_pool_type = "lvm" # depreciated in v1.1.2
    disk_size    = "10G"
    cache_mode   = "writeback"
    format       = "raw"
    io_thread    = false
  }

  // Cloud-init
  cloud_init              = true
  cloud_init_storage_pool = "local-lvm"

  // CPU & Memory
  sockets  = 1
  cores    = 2
  cpu_type = "host"
  memory   = 2048

  // Network
  network_adapters {
    bridge   = "vmbr0"
    model    = "virtio"
    vlan_tag = "1"
    firewall = false
  }
}

build {
  source "proxmox-iso.image" {
    name          = "ubuntu22"
    template_name = "ubuntu22"
    vm_id         = 9000
    iso_url       = "https://releases.ubuntu.com/22.04/ubuntu-22.04.2-live-server-amd64.iso"
    iso_checksum  = "file:https://releases.ubuntu.com/22.04/SHA256SUMS"
    boot_wait     = "5s"
    boot_command = [
      "c",
      "linux /casper/vmlinuz --- autoinstall 'ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/' ",
      "<enter><wait>",
      "initrd /casper/initrd",
      "<enter><wait>",
      "boot<enter>"
    ]
    http_content = {
      "/meta-data" = file("configs/meta-data")
      "/user-data" = templatefile("configs/user-data",
        {
          var            = var,
          ssh_public_key = chomp(file(var.ssh_public_key_file))
      })
    }
  }

  provisioner "shell" {
    inline = [
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done",
      // clean image identifiers
      "cloud-init clean --machine-id --seed",
      "rm /etc/hostname /etc/ssh/ssh_host_* /var/lib/systemd/random-seed",
      "truncate -s 0 /root/.ssh/authorized_keys",
      // disable SSH password authentication and root access
      "sed -i 's/^#PasswordAuthentication\\ yes/PasswordAuthentication\\ no/' /etc/ssh/sshd_config",
      "sed -i 's/^#PermitRootLogin\\ prohibit-password/PermitRootLogin\\ no/' /etc/ssh/sshd_config"
    ]
  }
}

pve-vars.pkr.hcl #

// PVE Variables //
// Pass Sensitive Variables CLI Args or Env Vars
variable "pve_token" {
  description = "Proxmox API Token, e.g. '782a7700-4010-4802-8f4d-820f1b226850'."
  type        = string
  sensitive   = true
}

variable "pve_username" {
  description = "Username when authenticating to Proxmox, e.g. 'packer@pve!token'."
  type        = string
  sensitive   = true
}

variable "pve_api_url" {
  description = "Proxmox API Endpoint, e.g. 'https://pve.example.com/api2/json'."
  type        = string
  sensitive   = true
  validation {
    condition     = can(regex("(?i)^http[s]?://.*/api2/json$", var.pve_api_url))
    error_message = "Proxmox API Endpoint Invalid. Check URL - Scheme and Path required."
  }
}

variable "pve_node" {
  type    = string
  default = "pve"
}

// SSH Variables //
variable "ssh_username" {
  description = "Image SSH Username"
  type        = string
  default     = "root"
}

variable "ssh_keypair_name" {
  default = "packer_id_ed25519"
  type    = string
}

variable "ssh_private_key_file" {
  description = "Private SSH Key for VM"
  default     = "~/.ssh/packer_id_ed25519"
  type        = string
  sensitive   = true
}

variable "ssh_public_key_file" {
  description = "Public SSH Key for VM"
  default     = "~/.ssh/packer_id_ed25519.pub"
  type        = string
  sensitive   = true
}

// Config Template Variables
variable "apt_proxy_http" {
  description = <<EOT
  APT proxy URL for Ubuntu, format: 'http://[[user][:pass]@]host[:port]/'.
  Default 'null' skips setting proxy.
  EOT
  type        = string
  default     = ""
}

variable "apt_proxy_https" {
  description = <<EOT
  APT proxy URL for Ubuntu, format: 'https://[[user][:pass]@]host[:port]/'.
  Default 'null' skips setting proxy.
  EOT
  type        = string
  default     = ""
}

meta-data #

uninitialized

user-data #

#cloud-config
autoinstall:
  version: 1
  user-data:
    disable_root: false   # allow SSH for root
    users: []             # skip creating default users
  locale: en_US.UTF-8
  packages:
    - qemu-guest-agent
  refresh-installer:
    update: true
  shutdown: reboot
  ssh:
    install-server: true  # enable SSH server
    allow-pw: false       # disable password authentication
    authorized-keys:
      - ${ssh_public_key} # add SSH key to root user
  timezone: UTC
  updates: all            # update all packages
  apt:
    %{ if var.apt_proxy_http != "" }
    http_proxy: ${var.apt_proxy_http}
    %{ endif }
    %{ if var.apt_proxy_https != "" }
    https_proxy: ${var.apt_proxy_https}
    %{ endif }

References #

Packer:

Proxmox: