Provisioning Proxmox 8 VMs with Terraform and BPG

Table of Contents

Intro #

Since the post Provisioning Proxmox VMs with Terraform, I switched to using the BPG Proxmox provider. The schema is different but easily adaptable to the prior code base. This provider is feature pack and provisioning is faster than other providers. Two features I am excited for is the ability to:

Download LXC and VM cloud images directly onto the Proxmox server.
Build VM templates from cloud images using Terraform.

I will be covering this provider across several post, but this one covers the basics using pre-existing templates.

In a prior post we created a Proxmox template that was configured using cloud-init vendor data file. This included several shell scripts for downloading cloud images and building templates. In this post, we’ll use those scripts to download and create an Ubuntu 24.04 template, then test out the BPG provider. You’ll need to have Terraform installed on your local machine and access to a Proxmox VE (PVE) server with the shell scripts installed.

BPG module is available on GitHub (link)

Create a Template #

To start, create a minimal cloud-init vendor data file for the template on your Proxmox server. Note: For multi-node PVE clusters, the vendor data file and template must be available on each node.

# SSH into Proxmox
user@desktop:~$ ssh root@pve

Create the vendor-data.yaml file in /var/lib/vz/snippets/ with the following content:

cat << EOF > /var/lib/vz/snippets/vendor-data.yaml
#cloud-config
packages:
  - qemu-guest-agent
package_update: true
power_state:
  mode: reboot
  timeout: 30
EOF

Next, download the latest Ubuntu 24.04 release and create template using the shell scripts:

# download/update Ubuntu 24.04
image-update -d ubuntu -r 24

# create a template
build-template --id 9024 --name ubuntu24 --img /var/lib/vz/template/iso/ubuntu-24.04-server-cloudimg-amd64.img --bios ovmf

This template will already have QEMU Guest Agent enabled, along with an EFI disk, cloud-init drive, and boot disk based on the Ubuntu 24.04 cloud image. Additionally, it will use the cloud-init vendor file we created earlier to install qemu-guest-agent, update all packages, then reboot the VM to ensure that Proxmox registers the guest agent. Because the cloud-init configuration file is imported as vendor data, you can still configure cloud-init user data using the Proxmox GUI - for details on why this works, see this blog post.

Grant Terraform Access to Proxmox #

Next, let’s create a Terraform user, group and token to access the Proxmox API. We’ll limit the group’s access to only the Proxmox resources it requires. On your Proxmox server, run the following as a privileged user:

# create role in PVE 8
pveum role add TerraformUser -privs "Datastore.Allocate \
  Datastore.AllocateSpace Datastore.AllocateTemplate \
  Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify \
  SDN.Use VM.Allocate VM.Audit VM.Clone VM.Config.CDROM \
  VM.Config.Cloudinit VM.Config.CPU VM.Config.Disk VM.Config.HWType \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.Migrate \
  VM.Monitor VM.PowerMgmt User.Modify"

# create group
pveum group add terraform-users

# add permissions
pveum acl modify /storage -group terraform-users -role TerraformUser

pveum acl modify /vms -group terraform-users -role TerraformUser

pveum acl modify /sdn/zones -group terraform-users -role TerraformUser

# create user 'terraform'
pveum useradd terraform@pve -groups terraform-users

# generate a token
pveum user token add terraform@pve token -privsep 0

The last command will output a token value similar to the following, save this information as we’ll pass it via environment variables to Terraform at the end.

┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ terraform@pve!token                  │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ 782a7700-4010-4802-8f4d-820f1b226850 │
└──────────────┴──────────────────────────────────────┘

The Provider notes that root SSH access is required for several functions. However, for this post token access is sufficient. In the next post we’ll upload images and config files that will require root SSH access.

BGP Provider #

Next, let’s create the several Terraform files that will utilize the BGP provider. For simplicity, we’ll hard-code the majority of attributes, however, let’s create a few variables to mask sensitive information and provide more flexibility to the end-user. Start by creating a new folder called bgp-example in your working directory and add the following files:

main.tf #

Within the main.tf file, create a terraform block and provider block that uses the BPG Proxmox provider. The provider block attributes for the PVE host address, endpoint, and the API token, api_token, are variables that we’ll define later in variable.tf. Additionally, the api_token is a concatenation of the PVE token ID and value, <TOKEN_ID>=<TOKEN_VALUE>, that we’ll want to avoid exposing in logs.

Notably, this provider uses custom blocks, e.g. efi_disk, to configure many of the VM settings. To add flexibility to the Terraform configuration, we’ll conditionally use a dynamic block to create an EFI disk when the bios is set to ovmf; otherwise, a null list is passed to the for_each statement and a EFI disk is not created.

The cloud-init disk is configured using the initialization block, within this block we’ll keep the cloud-init vendor data file attached and set the datasource, type, to NoCloud. This will ensure that cloud-init pulls the configuration from the attached config drive. Lastly, in order to login into the VM using the default user, ubuntu, we’ll add a public ssh key to the cloud-init config, keys, and use DHCP to configure the network interface.

# bgp-example/main.tf
terraform {
  required_version = ">=1.5.0"
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = ">=0.53.1"
    }
  }
}

provider "proxmox" {
  endpoint  = var.pve_api_url
  api_token = "${var.pve_token_id}=${var.pve_token_secret}"
  insecure  = true
}

resource "proxmox_virtual_environment_vm" "vm" {
  node_name   = "pve"
  vm_id       = 100
  name        = "vm-example"
  description = "Managed by Terraform"
  tags        = ["terraform", "ubuntu"]
  bios        = var.bios

  # clone from the Ubuntu 24.04 template we created earlier
  clone {
    vm_id = 9024
    full  = true
  }

  # keep the first disk as boot disk
  disk {
    datastore_id = "local-lvm"
    interface    = "scsi0"
    size         = 8
    file_format  = "raw"
    cache        = "writeback"
    iothread     = false
    ssd          = true
    discard      = "on"
  }

  # create an EFI disk when the bios is set to ovmf
  dynamic "efi_disk" {
    for_each = (var.bios == "ovmf" ? [1] : [])
    content {
      datastore_id      = "local-lvm"
      file_format       = "raw"
      type              = "4m"
      pre_enrolled_keys = true
    }
  }

  network_device {
    bridge  = "vmbr0"
    vlan_id = "1"
  }

  # cloud-init config
  initialization {
    interface           = "ide2"
    type                = "nocloud"
    vendor_data_file_id = "local:snippets/vendor-data.yaml"
    upgrade             = true

    # add SSH key to cloud-init default user
    user_account {
      keys     = [file("${var.ci_ssh_key}")]
    }

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }

  # cloud-init SSH keys will cause a forced replacement, this is expected
  # behavior see https://github.com/bpg/terraform-provider-proxmox/issues/373
  lifecycle {
    ignore_changes = [initialization["user_account"], ]
  }
}

For a full list of available resources and their attributes, please refer to the BPG provider documentation.

output.tf #

Given the network interface is using DHCP, let’s create an output.tf file with a single variable that will return the IPv4 address:

# bgp-example/outputs.tf
output "public_ipv4" {
  description = "Instance Public IPv4 Address"
  value       = flatten(proxmox_virtual_environment_vm.vm.ipv4_addresses[1])
}

variables.tf #

Now, create a variables.tf file and define the variables that will be used throughout the configuration:

# bgp-example/variables.tf
## Provider Login Variables
variable "pve_token_id" {
  description = "Proxmox API Token Name."
  sensitive   = true
}

variable "pve_token_secret" {
  description = "Proxmox API Token Value."
  sensitive   = true
}

variable "pve_api_url" {
  description = "Proxmox API Endpoint, e.g. 'https://pve.example.com/api2/json'"
  type        = string
  sensitive   = true
  validation {
    condition     = can(regex("(?i)^http[s]?://.*/api2/json$", var.pve_api_url))
    error_message = "Proxmox API Endpoint Invalid. Check URL - Scheme and Path required."
  }
}

## VM Variables
variable "bios" {
  description = "VM bios, setting to `ovmf` will automatically create a EFI disk."
  type        = string
  default     = "seabios"
  validation {
    condition     = contains(["seabios", "ovmf"], var.bios)
    error_message = "Invalid bios setting: ${var.bios}. Valid options: 'seabios' or 'ovmf'."
  }
}

variable "ci_ssh_key" {
  description = "File path to SSH key for 'default' user, e.g. `~/.ssh/id_ed25519.pub`."
  type        = string
  default     = null
}

proxmox.tfvars #

Finally, to make use of the above variables and reduce the number of variables to pass via the command line, create a proxmox.tfvars file and set the pve_api_url and ci_ssh_key to your PVE server IP address or FQDN and SSH file path:

# bgp-example/proxmox.tfvars
pve_api_url      = "https://pve.example.com/api2/json"
bios             = "ovmf"
ci_ssh_key       = "~/.ssh/id_ed25519.pub"

Run Terraform #

Now let’s test out the configuration, Terraform allows you to set variables using: 1) environment variables, prefixing the variable name with TF_VAR_; 2) CLI args, using the -var= flag; and/or 3) variable files, having the extension *.tfvars and using the -var-file flag. The following commands will initialize and apply the terraform configuration:

# use environment variables to pass the token id and secret
export TF_VAR_pve_token_id="terraform@pve!token"
export TF_VAR_pve_token_secret="782a7700-4010-4802-8f4d-820f1b226850"

# initialize the provider
terraform init

# create a terraform plan & apply it
terraform plan -var-file proxmox.tfvars -out tfplan
terraform apply tfplan

Once the plan is applied successfully, you should see output similar to this:

...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

public_ipv4 = [
  "192.168.1.16",
]

Test out the VM by SSHing into it and running uname -a:

user@desktop:~$ ssh ubuntu@192.168.1.16
ubuntu@vm-example:~$ uname -a

To cleanup the environment, run the following command:

# remove the vm
terraform destroy -var-file proxmox.tfvars

Recap #

So far we created a simple root module to deploy a VM from a template. This is a great starting point for creating more complex modules, especially given the BPG Proxmox provider consist of multiple resources that combine together to encapsulate complex Proxmox deployments. If you are interested I have a BPG module (link) that can: download VM and LXC images; create VM templates; and deploy LXC containers and VMs from templates. Feel free to use it and thanks for reading!

References #

Proxmox:

Terraform:

Terraform Providers / Plugins:

Other:

Cloud-init Documentation