Back up and restore PE

Use the Puppet Enterprise (PE) backup tool to create regular backups of your installation. Then, if you migrate your primary server to a new operating system or replace your primary server hardware, you can restore your installation. The backup and restore process can also be useful for troubleshooting or for recovering your installation after a system failure.

The PE backup tool is designed for backing up and restoring standard or large PE installations. For extra-large installations, consider using an alternative backup or snapshot process.

If you have a standard or large PE installation, you can implement the following backup and restore process:
  1. Use the puppet-backup create command to back up the primary server.
  2. When necessary, use the puppet-backup restore command to restore the primary server and your PE infrastructure.
    Disaster recovery consideration
    If your PE installation includes disaster recovery, then after restoring the primary server, you must remove the existing replica and provision a new one.
Important: When upgrading your operating system to a new major version, you must:
  1. Back up PE on your existing primary server.
  2. Install a new PE primary server on a node with the upgraded operating system.
  3. Restore the PE backup on the new primary server.
  4. For installations with compilers:
    • Remove existing compilers and install new compilers on the upgraded operating system.
  5. For installations with disaster recovery:
    • Remove the existing replica and provision a new replica on the upgraded operating system.
Restriction: You cannot use the puppet-backup command to back up or restore the following components:
  • The replica of your primary server
  • Compilers
  • Secret keys

Customize scope of backup and restore

You can use the --scope option to customize what data is backed up or restored.

By default, the puppet-backup create command backs up the following data, and the puppet-backup restore command restores the same data:
  • Your PE configuration, including license, classification, and RBAC settings. However, the configuration backup data does not include Puppet gems or Puppet Server gems.
  • PE CA certificates and the full SSL directory.
  • The Puppet code deployed to your code directory at the time of the backup.
  • PuppetDB data, including facts, catalogs and historical reports.
CAUTION: The puppet-backup command does not include secret keys. You must back up this data separately and securely.

If you want to have discrete backup files, or if you want to back up some parts of your infrastructure more often than others, you can use --scope command line option to limit the scope of a backup or restore. The --scope option accepts one or more of certs, code, config, or puppetdb. If unspecified, the default value is all. For details about what is and isn't included in each scope, refer to Directories and data in backups.

For example, if you have frequent code changes, you might back up your Puppet code more often than you back up the rest of your infrastructure. When you limit the backup scope, the backup file contains only the parts of your infrastructure that you specify. Be sure to identify the scope in your backup file's name so you know what each file contains.

When you restore your primary server, you must restore your Puppet configuration, certificates, code, and PuppetDB data. However, you can restore each aspect from different files, either by using backup files that have limited scopes or by limiting the restore scope. For example, by limiting the scope when you run the puppet-backup restore command, you could restore Puppet code, configuration, and certificates from one backup file, and then restore PuppetDB from a different backup file.

Important: When you restore your primary server, you must restore all four data sets: configuration, certificates, code, and PuppetDB. However, you can't restore data that isn't contained in the backup file you're restoring from. For example, a backup file that only contains PuppetDB data can only be used to restore PuppetDB data. In this case you'll need to run the puppet-backup restore command multiple times, restoring a different file each time, until you have restored all four data sets (configuration, certificates, code, and PuppetDB).

Back up your infrastructure

The backup process creates a copy of your primary server, including configuration, certificates, code, and PuppetDB. Backup can take several hours depending on the size of PuppetDB.

Before you begin
If you want to encrypt your backup, you must import your GPG public key to your primary server.
To create a complete set of backup data, you need to backup your infrastructure's secret keys and use the puppet-backup command to backup your PE configuration, PE certificates, Puppet code, and PuppetDB data. For details about the data included in backup files, refer to Customize scope of backup and restore.
  1. To ensure that pg_repack doesn't run during the backup process, stop the pe_databases module timers: 
    systemctl stop pe_databases-*.timer
  2. Run the puppet-backup command on your primary server. The default command is: 
    sudo puppet-backup create --dir=<BACKUP_DIRECTORY>
    You can customize your backup by specifying the following optional parameters:
    • --dir=<BACKUP_DIRECTORY>: Specify a separate a secure location for your backup.
    • --name=<BACKUP_NAME>: Specify the backup file's name. The default name is pe_backup with a timestamp indicating when the backup file was created, such as: pe_backup-<TIMESTAMP>.tgz.
    • --pe-environment=<ENVIRONMENT>: Specify an environment to back up. To ensure the configuration is recovered correctly, this must be the environment where your primary server is located. The default value is production.
    • --scope=<SCOPE_LIST>: Specify the data you want the backup file to contain. This is used for Customize scope of backup and restore. The default scope is all. To backup specific data, limit the scope by using one or more of the following values: certs, code, config, or puppetdb.
      Tip: If you specify --scope, specify a --name that describes the file's scope.
    • --gpgkey=<KEY_ID>: Specify a GPG key ID to use to encrypt the backup file.
    • --force: Specify this parameter if you want to bypass validation checks and ignore warnings.
  3. Back up the secret keys directory and, if applicable, the secret key for the LDAP service.
    CAUTION:

    The puppet-backup command does not include secret keys. You must back up this data separately.

    Secure the keys as you would any sensitive information.

    • The secret keys directory is located at: /etc/puppetlabs/orchestration-services/conf.d/secrets/
    • (If applicable) The LDAP service key is located at: /etc/puppetlabs/console-services/conf.d/secrets/keys.json
  4. Restart the pe_databases module timers: 
    systemctl start pe_databases-catalogs.timer pe_databases-facts.timer pe_databases-other.timer
Results

Each time you use puppet-backup to create a new backup, PE creates a single backup file containing everything you're backing up (defined by the --scope). PE writes backup files to /var/puppetlabs/backups, unless you specify a different location in the puppet-backup command. The file name follows the default naming convention (pe_backup-<TIMESTAMP>.tgz), unless you specified a different name in the puppet-backup command.

Restore your infrastructure

Use the restore process when you migrate your primary server to a new operating system or to a new host. You can also use the restore process to recover your installation after a system failure.

Before you begin

You must have created backup files, as described in Back up your infrastructure.

You must import the GPG key pair (both the public and private keys) that you used for encryption to your new primary server. The GPG keys are required to decrypt an encrypted backup.

  1. To ensure that the pg_repack extension doesn't run, stop the pe_databases module timers: 
    systemctl stop pe_databases-*.timer
  2. If you are restoring the primary server to a node that is currently hosting an active primary server, you must first purge the existing installation and reinstall PE. To do this:
    1. On the primary server, run this command to uninstall PE: 
      sudo /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d
    2. Ensure that these directories are no longer present on the system: /opt/puppetlabs/ and /etc/puppetlabs/
  3. Install PE on the primary server you want to restore to. You must install the same PE version used to create your backup files.
    1. If you don't have the PE installer script on the machine you want to restore to, download the installer tarball and unpack it by running: tar -xf <TARBALL_FILENAME>
    2. Navigate to the directory containing the install script. Normally, the installer script is located in the PE directory that is created when you unpacked the tarball.
    3. To install PE, run: sudo ./puppet-enterprise-installer
  4. On your primary server, run the puppet-backup restore command to restore your PE infrastructure. The default command is: 
    sudo puppet-backup restore <BACKUP-FILENAME>
    You must use the following parameter to specify the backup file you want to restore from:
    You can customize your restore by specifying the following optional parameters:
    • --pe-environment=<ENVIRONMENT>: Specify an environment to restore. The default value is production.
    • --scope=<SCOPE_LIST>: Specify the data you want to restore. This is used for Customize scope of backup and restore. The default scope is all. If you only want to restore specific data, specify one or more of: certs, code, config, or puppetdb.
    • --force: Specify this parameter if you want to bypass validation checks and ignore warnings.
    Important: You might have to run the puppet-backup restore command again if either the backup file or your restore command had limited --scope. For more information, refer to Customize scope of backup and restore.
  5. Restore your secret key backups. These keys are used to encrypt and decrypt sensitive data in the inventory service, orchestrator, and the LDAP service (if enabled).
    • The location of the Inventory service and orchestrator keys directory is: /etc/puppetlabs/orchestration-services/conf.d/secrets/
    • The location of the LDAP service key file is: /etc/puppetlabs/console-services/conf.d/secrets/keys.json
    CAUTION: The puppet-backup restore command does not include secret keys. You must restore this data separately.
  6. Make sure the inventory service's secret key ownership is configured as: pe-orchestration-services:pe-orchestration-services
  7. If the LDAP service is enabled, make sure the LDAP service's secret key ownership is configured as: pe-console-services:pe-console-services
  8. Restart pe-orchestration-services and pe-console-services.
  9. If you restored PE onto a primary server with a different hostname than the original installation, and you have not configured the dns_alt_names setting in the pe.conf file, you need to redirect your agents to the new primary server. One way to do this is by running a task with the Bolt task runner:
    1. If Bolt isn't already installed, download and install Bolt.
    2. To update the puppet.conf file to point all agents at the new primary server, run the following commands: 
      bolt task run puppet_conf action=set section=agent setting=server value=<RESTORE_HOSTNAME> --targets <COMMA-SEPARATED_LIST_OF_NODES>
bolt task run puppet_conf action=set section=main setting=server value=<RESTORE_HOSTNAME> --targets <COMMA-SEPARATED_LIST_OF-NODES>
    3. On the newly restored primary server, run puppet agent -t --no-use_cached_catalog to apply the changes. Then run the same command again to restart services.
      Important: You must run the command twice.
    4. To test the connection to the new primary server, on each agent node, run the following command:
      puppet agent -t --no-use_cached_catalog
  10. If Code Manager was enabled when you created your backup file, complete the following steps on the newly restored primary server:
    1. Run puppet-access login.
    2. For each code environment you want to deploy, run puppet code deploy --wait <ENVIRONMENT_NAME>.
      Tip: If you want to deploy all of your PE code environments, you can run puppet code deploy --wait --all.
    3. Run the following: 
      puppet agent -t --no-use_cached_catalog
puppet agent -t --no-use_cached_catalog
      Important: You must run the command twice.
  11. If your installation includes compilers and you are restoring your primary server on the same operating system that was in use when the backup was created, run the following command on all compilers: 
    puppet agent -t --server_list <PRIMARY_CERTNAME>
  12. If your installation includes compilers and you are restoring your primary server after changing or upgrading your operating system, you must complete the following steps:
    1. On the primary server, run the following command for each compiler: 
      puppet node purge <COMPILER_CERTNAME>
    2. If any of the compilers included in your installation were legacy compilers (compilers without the PuppetDB service), you must unpin them from the PE Master node group.
    3. Install and configure new compilers.
      Important: Compilers must run the same OS major version, platform, and architecture as the primary server.
  13. If your installation includes a primary server replica, you must complete the following steps:
    1. On the primary server, run the following command: 
      puppet infrastructure forget <REPLICA_CERTNAME>
    2. Provision and enable a new replica.

Directories and data in backups

These directories and data are included in PE backups.

A default puppet-backup command captures all scopes, meaning all directories and data described in the table below. However, you can use the --scope option to limit the contents of backup files, or to restore data from multiple backup files, as described in Customize scope of backup and restore. In this case, the --scope option indicates which directories and data to back up or restore.

Scope Directories and databases
certs (PE certificates)
  • PE CA certificates
  • /etc/puppetlabs/puppet/ssl/
code (Puppet code) This scope captures the Puppet code deployed to your code directory at the time of the backup. Specifically:
  • /etc/puppetlabs/code/
  • /etc/puppetlabs/code-staging/
  • /opt/puppetlabs/server/data/puppetserver/filesync/storage/
  • /opt/puppetlabs/server/data/orchestration-services/data-dir
  • /opt/puppetlabs/server/data/orchestration-services/code
config (PE configuration) This scope captures your PE configuration, including license, classification, and RBAC settings. Some directories and data are excluded, such as Puppet gems, Puppet Server gems, and directories captured in other scopes. Specifically, it includes:
  • The orchestrator, RBAC, and classifier databases
  • The contents of /etc/puppetlabs/, except:
    • The /code and /code-staging directories, which are included in the code scope.
    • The /puppet/ssl directory, which is included in the certs scope.
  • The contents of /opt/puppetlabs/, except:
    • /puppet
    • /server/pe_build
    • /server/data/packages
    • /server/apps
    • /server/data/postgresql
    • /server/data/enterprise/modules
    • /server/data/puppetserver/vendored-jruby-gems
    • /bin
    • /client-tools
    • /server/share
    • /server/data/puppetserver/filesync/storage
    • /server/data/puppetserver/filesync/client
    • /server/data/orchestration-services/data-dir and /server/data/orchestration-services/code, which are included in the code scope.
puppetdb (PuppetDB)
  • PuppetDB data, including facts, catalogs, and historical reports
  • The contents of /opt/puppetlabs/server/data/puppetdb, except the /stockpile directory.