Operating the Cluster

Ansible for Proxmox VE · Module 05

credativ GmbH

Learning goals

• Stop repeating API credentials with module_defaults
• Add storage to the cluster
• Create users, roles, permissions and an API token, then use it
• Create a VM and back it up, all from the same playbook
• See what it costs to edit a config file on the node instead

40 minutes, the second-largest block. This is where participants recognise their own daily work, so let questions run a little longer here than elsewhere.

The pattern is always the same

One API call, one cluster-wide change. You address any node; pmxcfs distributes the result.

Look it up: the API browser

Every node serves the full API reference, for exactly the version you are running:

https://192.168.0.1:8006/pve-docs/api-viewer/

• searchable tree of every endpoint, with every parameter and its type
• the public copy: https://pve.proxmox.com/pve-docs/api-viewer/

Use it when a module lacks an option. The page tells you whether the API has it at all — which decides between “file a feature request” and “use ansible.builtin.uri”.

Module option, API parameter, same name

community.proxmox.proxmox_kvm:
  node: pve02
  vmid: 9001
  cores: 1
  memory: 512
  scsihw: virtio-scsi-single

POST /nodes/{node}/qemu in the API browser lists vmid, cores, memory, scsihw — the same names. That correspondence is what makes the browser the fastest way to find out what a guest option is actually called.

Stop repeating yourself: module_defaults

- name: Configure the cluster
  hosts: localhost
  connection: local
  gather_facts: false
  module_defaults:
    group/community.proxmox.proxmox:
      api_host: "{{ hostvars['pve01'].ansible_host }}"
      api_user: root@pam
      api_password: "{{ hostvars['pve01'].pve_root_password }}"
      validate_certs: false
  tasks:
    ...

Every community.proxmox module in this play inherits those four lines.

Storage

Two halves of one job

# 1 - on the nodes, over SSH
- name: Ensure the backup directory exists
  hosts: pve
  tasks:
    - ansible.builtin.file:
        path: /var/lib/vz-backup
        state: directory
        mode: "0755"

The other half, over the API

# 2 - on the cluster, over the API
- name: Register the directory as storage
  community.proxmox.proxmox_storage:
    name: pve-backup
    type: dir
    dir_options:
      path: /var/lib/vz-backup
    content: [backup]
    nodes: "{{ groups['pve'] }}"
    state: present

The node is a Debian machine and a Proxmox VE node.

Most of this workshop comes down to knowing which of the two you are talking to. Say it here, where both halves are on the same slide.

A second storage, a shared one

The backup directory is local to each node. An NFS library is shared, and it is where images and templates live in a real environment:

- community.proxmox.proxmox_storage:
    name: nfs-shared
    type: nfs
    nfs_options:
      server: "{{ pve_nfs_server }}"
      export: "{{ pve_nfs_export }}"
    content: [import, iso, vztmpl]
    nodes: "{{ groups['pve'] }}"
    state: present

• one API call, mounted on every node under /mnt/pve/nfs-shared
• content decides what may live there; no images, so nothing gets allocated on it
• this is where the cloud image for the next slide comes from

Users and permissions

Group, user, permission

- community.proxmox.proxmox_group:
    name: automation
    comment: Service accounts

- community.proxmox.proxmox_user:
    name: automation@pve
    groups: [automation]
    comment: Ansible service account
    enable: true

- community.proxmox.proxmox_access_acl:
    path: /
    type: group
    ugid: automation
    roleid: PVEVMAdmin
    propagate: 1

Realms matter: @pam are Linux users on the node, @pve exist only in Proxmox VE.

API tokens instead of passwords

- community.proxmox.proxmox_user:
    name: automation@pve
    groups: [automation]
    tokens:
      - tokenid: ansible
        comment: Created by Ansible
        privsep: false
  register: automation_user

• The secret is returned once, at creation, in automation_user.secrets
• privsep: false gives the token the user’s permissions; true restricts it further with its own ACL

Never print a secret with debug in a shared session. Write it somewhere safe, ideally straight into an encrypted file.

Privilege separation is worth two minutes, because it is where people lose an afternoon later.

A token is a separate identity, user@realm!tokenid, and it can hold its own ACL entries. With privsep: true — the default, in the API and in the module — the effective permissions are the intersection of the user’s and the token’s own. Two things follow. First, a fresh separated token can do nothing at all, no matter what the user may do, until you grant the token itself something. Second, a token can never exceed its user: granting it Administrator changes nothing if the user does not have it. Separation narrows an account, it never widens it.

We set false here because automation@pve exists only for this automation. The account is already the scope; a second layer of ACLs on top would only add confusion. Use true when the token hangs off an account a human also uses, or when one account needs several tokens with different scopes.

Point out how the failure looks, because it does not look like a permission problem: writes are refused with a proper error, but reads come back empty. A null from /cluster/status usually means the token lacks Sys.Audit, not that the cluster has nothing to say. If there is time, set privsep: true, re-run the VM task, and read the error out loud.

Mention expire as well: a token with no expiry outlives the reason it was created.

Then authenticate with the token

  module_defaults:
    group/community.proxmox.proxmox:
      api_host: "{{ hostvars['pve01'].ansible_host }}"
      api_user: automation@pve
      api_token_id: ansible
      api_token_secret: "{{ pve_api_token_secret }}"
      validate_certs: false

A token can be revoked on its own, is scoped by ACL, and never unlocks an SSH session. The root password can do none of those things.

Guests

A guest from a cloud image

- community.proxmox.proxmox_kvm:
    node: "{{ pve_vm_node }}"
    vmid: "{{ pve_vm_id }}"
    name: "{{ pve_vm_name }}"
    cores: 1
    memory: 1024
    ostype: l26
    scsihw: virtio-scsi-single
    scsi:
      scsi0: "{{ pve_vm_storage }}:0,import-from={{ pve_cloud_image_path }}"
    ide:
      ide2: "{{ pve_vm_storage }}:cloudinit"
    net:
      net0: virtio,bridge=vmbr0
    boot: "order=scsi0"
    timeout: 300          # NFS import is slower than the 30 s default
    state: present

• :0 means “size taken from the source image”
• import-from reads the qcow2 straight off the NFS library
• ide2: …:cloudinit adds the cloud-init drive

Lazy mounts bite here

- community.proxmox.proxmox_storage_contents_info:
    node: "{{ pve_vm_node }}"
    storage: "{{ pve_nfs_storage }}"
    content: import
  register: library
  until: library.proxmox_storage_content | default([])
         | selectattr('volid', 'search', pve_cloud_image) | list | count > 0
  retries: 12
  delay: 5

• Proxmox VE mounts an NFS storage on first activation, not when you attach it
• so importing in the same run fails with can't find file
• listing the contents activates it and confirms the image

Waiting for the path does not help: nothing has asked the node to mount it yet.

cloud-init does the rest

    ciuser: "{{ pve_cloud_user }}"
    sshkeys: "{{ lookup('ansible.builtin.file', '~/.ssh/id_ed25519.pub') }}"
    ipconfig:
      ipconfig0: "ip={{ pve_cloud_vm_ip }},gw={{ pve_cloud_vm_gw }}"

The guest configures its user, its SSH key and its address on first boot. No installer, no answer file, no template maintenance.

cipassword alone will not get you in: the Debian cloud image ships with PasswordAuthentication no. Set sshkeys.

From nothing to reachable in 30 seconds

- community.proxmox.proxmox_kvm:
    node: "{{ pve_vm_node }}"
    vmid: "{{ pve_vm_id }}"
    state: started

- ansible.builtin.wait_for:
    host: "{{ pve_cloud_vm_ip.split('/')[0] }}"
    port: 22
    timeout: 300
  delegate_to: localhost

Import, boot and cloud-init together take well under a minute on this lab.

This is the argument for cloud images over installer automation. Let the timing make it rather than the slide.

Containers are one module away

- community.proxmox.proxmox:
    vmid: 9002
    node: pve02
    hostname: ct01.example.internal
    ostemplate: "local:vztmpl/{{ pve_ct_template }}"
    storage: "{{ pve_vm_storage }}"
    disk: 4
    cores: 1
    memory: 512
    password: "{{ ct_root_password }}"
    state: present

The template has to be on the storage first; proxmox_template downloads it. Never hard-code the file name; ask the node which ones exist:

pveam update && pveam available --section system | grep debian-13

The other route

Sometimes you really do edit the file

Not everything has a module, and some things are simply faster to write directly. A second bridge, over SSH:

- name: Add a second bridge
  hosts: pve
  tasks:
    - name: Define vmbr1 in /etc/network/interfaces
      ansible.builtin.blockinfile:
        path: /etc/network/interfaces
        marker: "# {mark} ANSIBLE MANAGED - vmbr1"
        block: |
          auto vmbr1
          iface vmbr1 inet manual
              bridge-ports none
              bridge-stp off
              bridge-fd 0
      notify: Reload network configuration

Applying it is then your problem

  handlers:
    - name: Reload network configuration
      ansible.builtin.command: ifreload -a

ifreload -a applies the change immediately. A mistake in the file for vmbr0 takes the node off the network, including your own SSH session. Always keep a console (VDI, IPMI) available, and never touch the management interface of all three nodes in one run.

The same thing, through the API

- community.proxmox.proxmox_node_network:
    node: pve01
    state: present
    iface: vmbr1
    iface_type: bridge
    bridge_ports: ""
    autostart: true

• writes to /etc/network/interfaces.new, the same pending state the GUI shows
• becomes active on Apply Configuration, on reboot, or via ifreload -a

Both routes end in the same file. Only one of them is idempotent and reviewable before it takes effect.

Which route when

	blockinfile over SSH	proxmox_node_network
Idempotent	you build the guard	yes
Visible in the GUI as pending	no, it is already in the live file	yes
Works without the collection	yes	no
Risk of locking yourself out	high	lower, but real

Use the module when it covers your case. Reach for the file when it does not, and then go carefully: one node at a time, --limit, console at hand.

Backup

Run a backup

- community.proxmox.proxmox_backup:
    mode: include
    vmids: [9001]
    storage: pve-backup
    compress: zstd
    wait: true
    wait_timeout: 600

wait: true makes the task finish when the backup finishes, not when the API has accepted the request.

What has no module yet

Creating a scheduled backup job is not covered by a module today. proxmox_backup_schedule only adds guests to a job that already exists.

- name: Create a nightly backup job
  ansible.builtin.uri:
    url: "https://{{ api_host }}:8006/api2/json/cluster/backup"
    method: POST
    ...

ansible.builtin.uri against the REST API works for anything. You lose idempotency, so you add the guard yourself, the same way you do with command.

Exercise 05 — 20 minutes

Storage, a service account with a token, a VM, a backup. One playbook.

→ Exercise sheet: Operating the cluster

Trainer notes

• Timing: ~20 min presentation, 20 min lab. If time is short, drop the container slide and the uri fallback slide; they are in the handout.
• Demonstrate in the GUI after every step. Storage appears under Datacenter → Storage, the user under Permissions, the VM in the tree. Beginners need the feedback loop between YAML and something they recognise.
• module_defaults is the part of this module people will use first when they write their own playbooks. Say so.
• Common pitfalls:
  • pve_vm_storage not matching the lab (e.g. no local-lvm on a ZFS install). Check the environment beforehand and set the variable accordingly.
  • Token secret lost because the task was re-run. The API returns it only at creation; recovery means deleting and recreating the token.
  • privsep: true (the default everywhere) plus no ACL on the token. Writes fail with a permission error, reads come back empty. Speaker notes on the token slide cover the intersection rule.
  • Backup failing because the target storage does not allow backup content.
• Security framing: this is the natural place to say that the root password we used in module 03 would not survive a production review, and what replaces it: scoped API tokens.

Exercise 05 · Solutions

1 — Check the ground

ansible pve01 -m ansible.builtin.command -a 'pvesm status'

Name             Type     Status
local             dir     active
local-lvm     lvmthin     active

pve_vm_storage: local-lvm on this lab. A ZFS install would say local-zfs, and the guest task fails late and confusingly if it is wrong.

2 — Storage, both halves

- ansible.builtin.file:          # over SSH, on every node
    path: "{{ pve_backup_path }}"
    state: directory

- community.proxmox.proxmox_storage:   # one API call, whole cluster
    name: "{{ pve_backup_storage }}"
    type: dir
    dir_options: {path: "{{ pve_backup_path }}"}
    content: [backup]
    nodes: "{{ groups['pve'] }}"

3 — The token trap

privsep: false gives the token its user’s permissions.

With privsep: true the token needs its own ACL as well, and the effective rights are the intersection of both. A token that works for one call and returns an empty list for the next is almost always this.

4 — Why the contents call comes first

Proxmox VE mounts NFS on first activation, not when you attach it. Importing in the same run fails with can't find file, and wait_for on the path does not help — nothing has asked the node to mount it yet.

- community.proxmox.proxmox_storage_contents_info:
    node: "{{ pve_vm_node }}"
    storage: "{{ pve_nfs_storage }}"
    content: import
  register: library
  until: library.proxmox_storage_content | default([])
         | selectattr('volid', 'search', pve_cloud_image) | list | count > 0
  retries: 12
  delay: 5

5 — The run

localhost : ok=11  changed=9  unreachable=0  failed=0
pve01     : ok=1   changed=1  unreachable=0  failed=0
pve02     : ok=1   changed=1  unreachable=0  failed=0
pve03     : ok=4   changed=4  unreachable=0  failed=0

Most tasks run on localhost: the API modules talk HTTPS and are delegated there.

Second run: localhost : ok=10 changed=1 skipped=1. The one changed is Back up the VM — a backup is an action, not a state. The skipped task writes the token secret, which the API returns only at creation.