Operating the Cluster — Participant Handout

Ansible for Proxmox VE · Module 05

One API, one cluster

Once the nodes form a cluster, almost everything you configure is cluster-wide. You send an API request to any node, and pmxcfs replicates the result to all of them. Storage definitions, users, permissions, pools and backup jobs all work this way; only guests and node-local settings are tied to a specific node.

For your playbooks this means: one play against one API endpoint, not a loop over three nodes.

module_defaults: write the credentials once

Every community.proxmox module takes the same four or five API parameters. Repeating them in every task is noise and a source of copy-paste errors.

playbooks/30-operations.yml

---
- name: Configure the cluster
  hosts: localhost
  connection: local
  gather_facts: false
  module_defaults:
    group/community.proxmox.proxmox:
      api_host: "{{ hostvars['pve01'].ansible_host }}"
      api_user: root@pam
      api_password: "{{ hostvars['pve01'].pve_root_password }}"
      validate_certs: false
  tasks:
    - name: ...

group/community.proxmox.proxmox is an action group: the collection declares which modules belong to it, and the defaults apply to all of them. Individual tasks can still override any value.

Because this play talks only to the API, it runs on localhost. There is no need to iterate over hosts at all.

Storage

Adding directory storage illustrates the separation of concerns: the directory path on the filesystem is created via SSH, while the cluster-wide storage definition is registered via the Proxmox API.

- name: Ensure the backup directory exists on every node
  hosts: pve
  tasks:
    - name: Create the directory
      ansible.builtin.file:
        path: "{{ pve_backup_path }}"
        state: directory
        owner: root
        group: root
        mode: "0755"

- name: Register the directory as cluster storage
  community.proxmox.proxmox_storage:
    name: pve-backup
    type: dir
    dir_options:
      path: "{{ pve_backup_path }}"
    content: [backup]
    nodes: "{{ groups['pve'] }}"
    state: present

content decides what the storage may hold: backup, images, rootdir, iso, vztmpl, snippets. A storage that does not allow backup will refuse backups with an error that does not obviously say so.

The same module handles nfs, cifs, iscsi, zfspool, rbd, cephfs and pbs, each with its own <type>_options dictionary:

- name: Attach the shared NFS library
  community.proxmox.proxmox_storage:
    name: "{{ pve_nfs_storage }}"
    type: nfs
    nfs_options:
      server: "{{ pve_nfs_server }}"
      export: "{{ pve_nfs_export }}"
    content: [import, iso, vztmpl]
    nodes: "{{ groups['pve'] }}"
    state: present

One API call and the share is mounted on every node, below /mnt/pve/<storage name>. Note what content does not contain: without images or rootdir, Proxmox VE will never allocate a guest disk there, so the shared library stays a library. That is the difference between the local backup directory from the previous section, which each node owns, and this one, which everybody reads.

The same call with curl

TOKEN='PVEAPIToken=root@pam!ansible=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
PVE=https://192.168.0.1:8006/api2/json

# list the storages   (Ansible: community.proxmox.proxmox_storage_info)
curl -sk -H "Authorization: $TOKEN" "$PVE/storage" | jq '.data[] | {storage, type, content}'

# create a directory storage   (Ansible: community.proxmox.proxmox_storage)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/storage" \
     -d storage=pve-backup \
     -d type=dir \
     -d path=/var/lib/vz-backup \
     -d content=backup \
     -d nodes=pve01,pve02,pve03 | jq

API reference: /storage.

NoteWhere that token comes from

Nothing in this workshop creates a token for root@pam, so make one on a node before you try these calls:

pveum user token add root@pam ansible --privsep 0

The secret is printed once. If you use the automation@pve token from module 05 instead, the cluster-wide endpoints come back empty rather than failing: that account holds PVEVMAdmin, which does not include Sys.Audit. An empty result from the Proxmox VE API usually means missing privileges, not missing data.

The module’s dir_options.path is simply the path parameter; nfs_options, pbs_options and the others map to that type’s parameters in the same way. When a module is missing an option, this page tells you whether the API has it at all.

Users, roles and permissions

Proxmox VE separates authentication realms from authorisation:

Realm	Meaning
@pam	a Linux user that exists on the node (root@pam)
@pve	a user that exists only inside Proxmox VE
LDAP / AD / OIDC	configured realms, for real environments

Permissions are ACL entries: a path (/, /vms/9001, /storage/pve-backup), a subject (user, group or token) and a role (a named set of privileges).

- name: Ensure the automation group exists
  community.proxmox.proxmox_group:
    name: automation
    comment: Service accounts

- name: Ensure the service account exists
  community.proxmox.proxmox_user:
    name: automation@pve
    groups: [automation]
    comment: Ansible service account
    enable: true

- name: Grant the group permissions on the whole datacenter
  community.proxmox.proxmox_access_acl:
    path: /
    type: group
    ugid: automation
    roleid: PVEVMAdmin
    propagate: 1

Built-in roles include PVEVMAdmin, PVEVMUser, PVEDatastoreAdmin, PVEAuditor and Administrator. If none fits, define your own:

- name: A role that may only run backups
  community.proxmox.proxmox_role:
    roleid: BackupOperator
    privs:
      - VM.Backup
      - Datastore.AllocateSpace
      - VM.Audit

Tip

Grant on the narrowest path that works. / with propagate: 1 is convenient in a lab and almost always too much in production. /vms, /pool/web or /storage/pve-backup are usually what you actually meant.

API tokens

A token is a second credential belonging to a user. It can be revoked individually, restricted by its own ACL, and it does not unlock an SSH login.

- name: Ensure the service account and its token exist
  community.proxmox.proxmox_user:
    name: automation@pve
    groups: [automation]
    tokens:
      - tokenid: ansible
        comment: Created by Ansible
        privsep: false
  register: automation_user

The module returns the secrets it created in automation_user.secrets, keyed by token id. The API returns a token secret only at creation time. If you lose it, you delete the token and create a new one.

Privilege separation

A token is not just a second password for the user. It is a separate identity, user@realm!tokenid, and it can carry its own ACL entries. privsep decides whether it does:

privsep	Proxmox VE calls it	Effective permissions
true	separated privileges	intersection of the user’s permissions and the token’s own ACL
false	full privileges	identical to the user’s

true is the default, both in the API and in proxmox_user. Our playbook sets false explicitly, so the token inherits what automation@pve may do and nothing else has to be granted.

Two consequences follow from the intersection rule, and both cost people time:

A separated token starts with no permissions at all. It does not matter what the user may do: until the token has an ACL entry of its own, the intersection is empty. Grant it like any other subject, with the full token id as ugid:

- name: Let the token manage guests, and nothing else
  community.proxmox.proxmox_access_acl:
    path: /vms
    type: token
    ugid: "automation@pve!ansible"
    roleid: PVEVMAdmin
    propagate: 1

A token can never exceed its user. Granting the token Administrator on / changes nothing if the user does not hold it. That is the point of the intersection: the token is a way to narrow an account for one integration, never to widen it.

How the failure looks

Missing token privileges rarely announce themselves. A write is refused with a permission error, which is at least honest, but a read comes back empty:

# token without Sys.Audit
curl -sk -H "Authorization: $TOKEN" "$PVE/cluster/status"
# {"data":null}

null or [] where you expected data usually means the token lacks the privilege, not that the cluster has nothing to report. Check with pveum acl list before you start debugging the playbook.

Which mode to use

• privsep: false for a service account that exists only for this automation. The account itself is already the scope, and a second layer of ACLs on top buys nothing but confusion. That is our case.
• privsep: true when the token hangs off an account that a human also uses, or when one account holds several tokens that should differ in scope — one for inventory, one for guest management.

Either way, set expire (Unix epoch seconds) when the integration is temporary. A token with no expiry outlives the reason it was created for.

WarningHandling the secret

Do not print it with debug in a shared screen session, and do not commit it in clear text. In this lab it goes into a file with mode 0600; in production it belongs in Ansible Vault or your CI/CD secret store.

Authenticating with the token afterwards:

  module_defaults:
    group/community.proxmox.proxmox:
      api_host: "{{ hostvars['pve01'].ansible_host }}"
      api_user: automation@pve
      api_token_id: ansible
      api_token_secret: "{{ pve_api_token_secret }}"
      validate_certs: false

The same calls with curl

# create a group           (Ansible: proxmox_group)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/access/groups" \
     -d groupid=automation -d comment="Service accounts"

# create a user            (Ansible: proxmox_user)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/access/users" \
     -d userid=automation@pve -d groups=automation -d enable=1

# create an API token      (Ansible: proxmox_user, tokens:)
curl -sk -H "Authorization: $TOKEN" \
     -X POST "$PVE/access/users/automation@pve/token/ansible" \
     -d privsep=0 | jq
# -> {"data": {"full-tokenid": "automation@pve!ansible", "value": "xxxxxxxx-..."}}

# grant a permission       (Ansible: proxmox_access_acl)   note: PUT, not POST
curl -sk -H "Authorization: $TOKEN" -X PUT "$PVE/access/acl" \
     -d path=/ -d groups=automation -d roles=PVEVMAdmin -d propagate=1

API reference: /access/users, /access/groups, /access/acl, /access/roles.

The token response is the one place where the API returns a secret, and it returns it exactly once. That is a property of the API, not a limitation of the Ansible module.

Guests

Virtual machines, the minimal case

This creates an empty guest: a disk, some RAM, a network card, no operating system. Useful to see what proxmox_kvm does with nothing else in the way; the section after it builds a guest you can actually log into.

- name: Ensure an empty VM exists
  community.proxmox.proxmox_kvm:
    node: pve02
    vmid: 9001
    name: web01
    cores: 1
    memory: 512
    ostype: l26
    scsihw: virtio-scsi-single
    scsi:
      scsi0: "{{ pve_vm_storage }}:1"
    net:
      net0: virtio,bridge=vmbr0
    state: present

state accepts present, started, stopped, restarted, absent and a few more. The module identifies the guest by vmid, so a second run reports ok.

Cloning from a template is the common production path, because it gives you a prepared image instead of an empty shell:

- name: Clone the golden image
  community.proxmox.proxmox_kvm:
    node: pve02
    clone: debian12-template
    newid: 9010
    name: web02
    storage: "{{ pve_vm_storage }}"
    full: true
    timeout: 300

With a cloud-init enabled template you also set ciuser, sshkeys and ipconfig0, and the guest configures itself on first boot. Template plus cloud-init plus Ansible is what most Proxmox VE automation looks like in practice.

Installing from a cloud image

Creating an empty VM and then installing an operating system into it is the slow path. A cloud image is a disk that is already installed and carries cloud-init, which configures the guest on first boot from data the hypervisor hands it.

- name: Ensure the guest exists, installed from the cloud image
  community.proxmox.proxmox_kvm:
    node: "{{ pve_vm_node }}"
    vmid: "{{ pve_vm_id }}"
    name: "{{ pve_vm_name }}"
    cores: 1
    memory: 1024
    ostype: l26
    scsihw: virtio-scsi-single
    scsi:
      scsi0: "{{ pve_vm_storage }}:0,import-from={{ pve_cloud_image_path }}"
    ide:
      ide2: "{{ pve_vm_storage }}:cloudinit"
    net:
      net0: "virtio,bridge=vmbr0"
    boot: "order=scsi0"
    serial:
      serial0: socket
    ciuser: "{{ pve_cloud_user }}"
    sshkeys: "{{ lookup('ansible.builtin.file', '~/.ssh/id_ed25519.pub') }}"
    ipconfig:
      ipconfig0: "ip={{ pve_cloud_vm_ip }},gw={{ pve_cloud_vm_gw }}"
    timeout: 300
    state: present

WarningAttach and import in one run: two traps

The storage is mounted lazily. Proxmox VE mounts an NFS storage on first activation, not when you attach it. Importing from it in the same playbook run fails with can't find file on a path that exists as soon as you look at it by hand, which makes the error thoroughly confusing. Activating the storage first solves it:

- name: Activate the NFS library on the target node and find the image
  community.proxmox.proxmox_storage_contents_info:
    node: "{{ pve_vm_node }}"
    storage: "{{ pve_nfs_storage }}"
    content: import
  register: library
  until: library.proxmox_storage_content | default([])
         | selectattr('volid', 'search', pve_cloud_image) | list | count > 0
  retries: 12
  delay: 5

Waiting for the file to appear with wait_for does not work, because nothing has asked the node to mount the share.

The import is slower than the default timeout. proxmox_kvm waits 30 seconds; reading a few hundred megabytes over NFS and converting it takes longer, especially when multiple nodes import concurrently in a classroom. Set timeout: 300 on the task.

Four parts of that task are worth reading slowly:

scsi0: "<storage>:0,import-from=<path>". The 0 is the size, and 0 means “take it from the source image”. import-from points at the qcow2 on the NFS library, which every node can read because the storage from the previous section mounted it everywhere. Proxmox VE copies and converts it into a real disk on local-lvm; on this lab that takes a few seconds.

ide2: "<storage>:cloudinit". A small generated ISO holding the configuration below. Without it the guest boots the image unchanged and ignores everything you set.

ciuser and sshkeys. The user cloud-init creates, and the key it authorises. cipassword exists too, but on its own it will not get you in: the Debian cloud image ships with PasswordAuthentication no, so a password-only guest is one you can reach on the console and nowhere else.

ipconfig0. Static addressing handed to the guest. ip=dhcp works as well when the network has a DHCP server.

- name: Start it
  community.proxmox.proxmox_kvm:
    node: "{{ pve_vm_node }}"
    vmid: "{{ pve_vm_id }}"
    state: started

- name: Wait until cloud-init has brought up SSH
  ansible.builtin.wait_for:
    host: "{{ pve_cloud_vm_ip.split('/')[0] }}"
    port: 22
    timeout: 300
  delegate_to: localhost

Import, boot and cloud-init together stay under a minute here. The same guest built from an ISO and an installer takes minutes at best, and needs an answer file that nobody maintains.

NoteWhere the image comes from

The image is a file in the import/ directory of the NFS library, and it is staged there before the workshop. To fetch one yourself:

curl -O https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2

genericcloud is the variant without cloud-provider specifics. There are equivalents for every distribution that matters; the mechanics above do not change.

Containers

- name: Download the container template
  community.proxmox.proxmox_template:
    node: pve02
    storage: local
    content_type: vztmpl
    template: "{{ pve_ct_template }}"

- name: Ensure the container exists
  community.proxmox.proxmox:
    vmid: 9002
    node: pve02
    hostname: ct01.example.internal
    ostemplate: "local:vztmpl/{{ pve_ct_template }}"
    storage: "{{ pve_vm_storage }}"
    disk: 4
    cores: 1
    memory: 512
    password: "{{ ct_root_password }}"
    state: present

Note the module name: containers are handled by community.proxmox.proxmox, without a suffix. It is a historical artefact and it does confuse people.

Do not hard-code the template file name; it carries a version and changes. Ask the node what is on offer and put the answer in a variable:

pveam update
pveam available --section system | grep debian-13
# system  debian-13-standard_13.x-y_amd64.tar.zst
pveam list local                      # what is already downloaded

Your lab runs Proxmox VE 9, so a Debian 13 template is the natural match for a new container, but any of the listed templates works.

The same calls with curl

# create a VM              (Ansible: proxmox_kvm)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/nodes/pve02/qemu" \
     -d vmid=9001 -d name=web01 -d cores=1 -d memory=512 \
     -d ostype=l26 -d scsihw=virtio-scsi-single \
     -d scsi0=local-lvm:1 \
     -d net0=virtio,bridge=vmbr0 | jq

# read its configuration   (Ansible: proxmox_vm_info)
curl -sk -H "Authorization: $TOKEN" "$PVE/nodes/pve02/qemu/9001/config" | jq

# start it                 (Ansible: proxmox_kvm, state: started)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/nodes/pve02/qemu/9001/status/start"

# clone a template         (Ansible: proxmox_kvm, clone:)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/nodes/pve02/qemu/9000/clone" \
     -d newid=9010 -d name=web02 -d full=1

API reference: /nodes/{node}/qemu for VMs and /nodes/{node}/lxc for containers. Every parameter of proxmox_kvm has the same name here, which makes this page the fastest way to look up what a VM option is actually called.

Each of these returns a UPID. The Ansible module waits for the task; with curl you poll /nodes/{node}/tasks/{upid}/status yourself.

Editing a file on the node directly

Every automation project reaches a point where no module fits: an option the module does not model, a file no collection knows about, a workaround that has to go in today. Doing it over SSH is legitimate. It is a different trade-off, and one you should make consciously.

A second bridge, written straight into /etc/network/interfaces:

- name: Add a second bridge the manual way
  hosts: pve03
  gather_facts: false

  handlers:
    - name: Reload network configuration
      ansible.builtin.command: ifreload -a

  tasks:
    - name: Keep a pristine copy of the original interface configuration
      ansible.builtin.copy:
        src: /etc/network/interfaces
        dest: /etc/network/interfaces.orig
        remote_src: true
        force: false
        mode: "0644"

    - name: Define vmbr1 in /etc/network/interfaces
      ansible.builtin.blockinfile:
        path: /etc/network/interfaces
        marker: "# {mark} ANSIBLE MANAGED - vmbr1"
        block: |
          auto vmbr1
          iface vmbr1 inet manual
              bridge-ports none
              bridge-stp off
              bridge-fd 0
        validate: "grep -q 'iface vmbr1' %s"
      notify: Reload network configuration

Three safety measures are doing real work here:

• force: false on the copy keeps the original from being overwritten on the second run. A safety net that silently updates itself to the broken state is not a safety net.
• validate: runs a command against a temporary copy before the real file is replaced; %s is the temporary path. The check above is minimal. In production, use dedicated syntax validators: visudo -cf %s, nginx -t -c %s, named-checkconf %s.
• hosts: pve03 limits the blast radius to one node.

Warningifreload -a applies immediately

An error affecting vmbr0 takes the node off the network, including your own SSH session. Never run a networking change against all nodes at once, always keep a console (VDI, IPMI, KVM) available, and test with --check --diff first. This is not paranoia. It is a common way to lose a Proxmox VE node remotely.

The same change through the API

- name: The same bridge, through the API
  community.proxmox.proxmox_node_network:
    node: pve03
    state: present
    iface: vmbr2
    iface_type: bridge
    bridge_ports: ""
    autostart: true

The module writes to /etc/network/interfaces.new, the same pending changes mechanism the GUI uses. The change becomes active on Apply Configuration, on reboot, or with ifreload -a. So the API route gives you a review step that the direct edit does not.

	direct edit over SSH	proxmox_node_network
Idempotent	only because blockinfile is	yes
Reviewable before it applies	no, it is live after ifreload	yes, as pending changes
Works without the collection	yes	no
Covers exotic configuration	anything you can write	what the module models
Visible in the GUI	as normal configuration	as pending changes

The rule that follows: use the module when it covers your case. When it does not, edit the file, but limit the scope, keep a copy, validate, and have a console open.

The same calls with curl

# list the interfaces      (Ansible: proxmox_node_network_info)
curl -sk -H "Authorization: $TOKEN" "$PVE/nodes/pve03/network" | jq '.data[] | {iface, type, active}'

# create a bridge          (Ansible: proxmox_node_network)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/nodes/pve03/network" \
     -d iface=vmbr2 -d type=bridge -d bridge_ports="" -d autostart=1

# apply the pending changes  (the GUI button "Apply Configuration")
curl -sk -H "Authorization: $TOKEN" -X PUT "$PVE/nodes/pve03/network"

# discard the pending changes
curl -sk -H "Authorization: $TOKEN" -X DELETE "$PVE/nodes/pve03/network"

API reference: /nodes/{node}/network.

Note that POST only writes /etc/network/interfaces.new; the separate PUT is what applies it, and DELETE throws the pending changes away. That two-step design is the safety net the direct file edit does not have, and the reason the module alone does not activate your change.

Backup

- name: Back up the new VM
  community.proxmox.proxmox_backup:
    mode: include
    vmids: [9001]
    storage: pve-backup
    compress: zstd
    wait: true
    wait_timeout: 600

mode is include, exclude or all. With wait: true the task returns when the backup has finished; without it, the task returns as soon as the API has accepted the job, which is rarely what you want in a pipeline.

To add or remove guests from an existing scheduled job:

- name: Ensure the VM is part of the nightly job
  community.proxmox.proxmox_backup_schedule:
    vm_id: 9001
    backup_id: backup-b2adffdc-316e
    state: present

When no module exists

Creating a scheduled backup job itself is not covered by a module at the time of writing. The fallback is to call the REST API directly:

- name: Create a nightly backup job
  ansible.builtin.uri:
    url: "https://{{ pve_api_host }}:8006/api2/json/cluster/backup"
    method: POST
    validate_certs: false
    headers:
      Authorization: "PVEAPIToken={{ pve_api_user }}!{{ pve_api_token_id }}={{ pve_api_token_secret }}"
    body_format: form-urlencoded
    body:
      schedule: "02:30"
      storage: pve-backup
      mode: snapshot
      all: 1
    status_code: [200]

You lose idempotency the moment you leave the modules behind, so you add the guard yourself: check for the job first, and run the creation only when it is missing. This is the same discipline as with command, one layer up.

The same calls with curl

# run a backup now         (Ansible: proxmox_backup)
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/nodes/pve02/vzdump" \
     -d vmid=9001 -d storage=pve-backup -d compress=zstd -d mode=snapshot | jq
# -> "UPID:pve02:00001234:...:vzdump::root@pam:"

# follow the task
curl -sk -H "Authorization: $TOKEN" "$PVE/nodes/pve02/tasks/$UPID/status" | jq '.data.status'

# list the backup jobs     (no Ansible module - this is the uri fallback)
curl -sk -H "Authorization: $TOKEN" "$PVE/cluster/backup" | jq

# create a scheduled job
curl -sk -H "Authorization: $TOKEN" -X POST "$PVE/cluster/backup" \
     -d schedule="02:30" -d storage=pve-backup -d mode=snapshot -d all=1

API reference: /nodes/{node}/vzdump and /cluster/backup.

wait: true in the Ansible module is exactly the polling loop shown above. Without it, “the backup ran” means “the API accepted the request”.

Verifying what you built

ansible pve01 -m ansible.builtin.command -a 'pvesm status'
ansible pve01 -m ansible.builtin.command -a 'pveum user list'
ansible pve01 -m ansible.builtin.command -a 'pveum acl list'
ansible pve01 -m ansible.builtin.command -a 'qm list'
ansible pve01 -m ansible.builtin.command -a 'ls -lh /var/lib/vz-backup/dump/'

Every one of these has a matching *_info module if you want the result as data rather than as text: proxmox_storage_info, proxmox_user_info, proxmox_vm_info, proxmox_backup_info.

Further reading

The collection: the modules used here

• community.proxmox collection index (start here and browse; the module list is short enough to read once)
• proxmox_storage
• proxmox_user, proxmox_group, proxmox_role, proxmox_access_acl
• proxmox_kvm (virtual machines), proxmox (containers), proxmox_template
• proxmox_backup, proxmox_backup_schedule
• proxmox_node_network
• Proxmox API authentication guide (the collection’s own page on passwords versus tokens)

Ansible concepts from this module

• Module defaults (including action groups such as group/community.proxmox.proxmox)
• ansible.builtin.uri (the fallback when no module exists)
• ansible.builtin.blockinfile and its validate: option

Proxmox VE

• Storage: which storage type can hold which content, and why dir behaves differently from lvmthin
• User Management: realms, roles, privileges, API tokens, and the full privilege list for custom roles
• Backup and Restore
• Qemu/KVM Virtual Machines and Linux Containers
• Network Configuration (bridges, bonds, VLANs, and how pending changes work)
• API viewer: when a module lacks an option, this tells you whether the API has it at all

Offline

ansible-doc -l community.proxmox           # the full module list
ansible-doc community.proxmox.proxmox_kvm  # every option of the VM module
man pvesm ; man pveum ; man vzdump ; man qm