PebbleRack

Guide 23 of 25 · intermediate

Track configuration drift and keep a lab change log

Know what changed, why it changed, and how to undo it.

Source-verified · Not lab-testedOfficial sources checked 2026-09-29. No PebbleRack hardware compatibility claim.

Outcome

You will have a private baseline and a lightweight change log for the settings that can break your lab: host version, network, storage, VM definitions, users, firewall rules, and key services. A change log helps you distinguish a planned update from unexplained drift. It is not a complete backup, and diffing configuration files cannot prove that a service works. This is a suggested practice; it has not been run on PebbleRack hardware.

Before you start

Choose a private repository or encrypted directory that only lab administrators can access. Establish what must never be copied there: passwords, private keys, session tokens, full environment files, customer data, and raw logs. Have a tested backup of the host and guests before changing settings. Note your Proxmox VE and Linux versions because file locations and interfaces can differ.

Steps

  1. Create a one-page inventory with node name, OS/version, management address, storage names, VM/container IDs and purpose, backup destination, and owner. Use placeholders or a private location for sensitive addresses. Record the date and who observed each value. Do not infer an intended architecture from a running configuration.
  2. Capture a baseline using supported read-only interfaces. In Proxmox, use the GUI to record Datacenter, node, storage, network, firewall, and selected guest configuration. For automation, use the documented Proxmox API or CLI read operations with a minimally privileged account, and save sanitized output. On Ubuntu guests, use package inventory, systemctl list-unit-files, and a curated list of service configuration files. Avoid recursively copying /etc into Git, because it often contains secrets.
  3. Create a change entry template: date/time and time zone; change ID; service and host; reason; planned command or UI action; expected impact; backup/snapshot reference; approver if needed; verification; rollback; actual result. Separate “planned” and “observed.” A proposed change is not a tested outcome.
  4. Before a routine change, write the proposed diff and its rollback. Apply one bounded change, then verify both the intended service and adjacent dependencies. For example, changing a VM network setting should be followed by checks from another device, not only a local console ping. Record exact version numbers and task IDs where available.
  5. Schedule a weekly drift review. Compare current read-only exports with the prior approved baseline. Classify differences as authorized, unknown, or emergency changes awaiting documentation. Investigate unknown admin users, firewall openings, and disabled backup jobs promptly. Avoid automatically reverting drift: a well-intended automation could remove an emergency repair or break remote access.
  6. Update the baseline only after a change has been verified. Keep an append-only incident or change history, and link it to the backup and monitoring records. Add a quarterly review of stale users, unused services, and old remote-access devices.

Check it worked

Choose one known harmless setting or package update. A second reader should be able to locate its reason, before/after state, test result, and rollback. A review identifies the known change without exposing a secret in the diff. Unknown differences remain flagged until someone resolves them; do not relabel them as approved merely to clear a dashboard.

If it fails / rollback

If the export includes a secret, remove it from the working copy and repository history using an appropriate private process, rotate the affected credential, and narrow the export. If a change breaks access, use the documented local console path and reverse the specific setting. Do not restore an entire old host configuration without checking what legitimate later changes it would overwrite.

Safety and data notes

Configuration histories are sensitive maps of the lab. Encrypt and restrict them, and keep secrets in a dedicated secret store. Git is useful for text history but does not encrypt content. A baseline is a snapshot of observed settings, not evidence that the settings are correct or secure.

Sources

Official sources checked 2026-09-29: Proxmox VE administration guide and API references; Ubuntu systemd service management; NIST configuration management guidance.

Next guide

Continue with a bounded automatic restart.