Configuration v2

Note

The configuration file format v2 is in preview. It has been available since v0.17.0: mixins were added in v0.18.0 and group schedules in v0.29.0.

The format v1 remains the default and keeps working unchanged.

Introduction

The original file format (v1) was designed when resticprofile only supported toml. Every profile is a section at the top level of the file, next to the reserved sections global, groups and includes.

The format v2 gives each kind of declaration its own section. This makes room for features that cannot be expressed in v1:

  • Groups with their own configuration: description, error handling and schedules
  • Mixins to share pieces of configuration between profiles
  • Predictable inheritance of lists, with a syntax to append or prepend values
  • Safer defaults for backup and retention

Selecting the version

The format is selected by the version property at the top level of the configuration file:

versionFormat used
not setv1
"1"v1
"2"v2
​
version: "2"
version = "2"
{
  "version": "2"
}

A value that is not a number selects the format v1.

The version can be written as a number (version: 2), but the JSON schema only accepts a string (version: "2").

File formats

The format v2 is available for:

  • TOML
  • YAML
  • JSON

It is not available for HCL. A HCL file that declares version = 2 fails to load with the error:

HCL format is not supported in version 2, please use version 1 or another file format

HCL files can still be used with the format v1.

Structure

A v2 configuration file contains up to six top level properties:

PropertyPurpose
versionSelects the file format, must be "2"
globalSettings that apply to all profiles
includesOther configuration files to merge into this one
mixinsNamed pieces of configuration that profiles can use
groupsNamed groups of profiles
profilesNamed profiles
​
version: "2"

global:
  default-command: snapshots

includes:
  - "profiles.d/*.yaml"

mixins:
  name-of-mixin:
    password-file: key

groups:
  name-of-group:
    profiles:
      - name-of-profile

profiles:
  name-of-profile:
    use: name-of-mixin
    repository: /backup
    backup:
      source: /home
version = "2"

includes = ["profiles.d/*.toml"]

[global]
  default-command = "snapshots"

[mixins.name-of-mixin]
  password-file = "key"

[groups.name-of-group]
  profiles = ["name-of-profile"]

[profiles.name-of-profile]
  use = "name-of-mixin"
  repository = "/backup"

  [profiles.name-of-profile.backup]
    source = "/home"
{
  "version": "2",
  "global": {
    "default-command": "snapshots"
  },
  "includes": [
    "profiles.d/*.json"
  ],
  "mixins": {
    "name-of-mixin": {
      "password-file": "key"
    }
  },
  "groups": {
    "name-of-group": {
      "profiles": [
        "name-of-profile"
      ]
    }
  },
  "profiles": {
    "name-of-profile": {
      "use": "name-of-mixin",
      "repository": "/backup",
      "backup": {
        "source": "/home"
      }
    }
  }
}
Warning

Any other property at the top level is ignored. A profile that is declared at the top level, as in the format v1, is not loaded and no error is reported.

Differences with v1

v1v2
versionOptionalRequired: "2"
File formatsTOML, YAML, JSON, HCLTOML, YAML, JSON
ProfilesSections at the top levelNested in the profiles section
GroupsA list of profile namesAn object: profiles, description, continue-on-error, schedules
Group schedulesNot availableAvailable
MixinsNot availablemixins section and use property
Lists in derived profilesMerged with the list of the parent, index by indexReplace the list of the parent, or append and prepend to it
host in backupNot setDefaults to the host name of the system
after-backup in retentionNot setEnabled when a keep-* flag is set
tag and host in retentionNot setDefault to the values used by backup

Everything else is identical in both formats: the global section, the content of a profile, variables, templates, includes and schedules of a profile.

Sections

global

The global section is the same as in the format v1. See the reference for the list of properties.

version: "2"

global:
  default-command: snapshots
  initialize: false
  priority: low

profiles

All profiles are nested in the profiles section. The content of a profile is the same as in the format v1: see the reference for the list of properties.

​
version: "2"

profiles:
  default:
    env:
      tmp: /tmp
    password-file: key
    repository: /backup

  documents:
    inherit: default
    backup:
      source: ~/Documents
    snapshots:
      tag:
        - documents
version = "2"

[profiles.default]
  password-file = "key"
  repository = "/backup"

  [profiles.default.env]
    tmp = "/tmp"

[profiles.documents]
  inherit = "default"

  [profiles.documents.backup]
    source = "~/Documents"

  [profiles.documents.snapshots]
    tag = ["documents"]
{
  "version": "2",
  "profiles": {
    "default": {
      "env": {
        "tmp": "/tmp"
      },
      "password-file": "key",
      "repository": "/backup"
    },
    "documents": {
      "inherit": "default",
      "backup": {
        "source": "~/Documents"
      },
      "snapshots": {
        "tag": [
          "documents"
        ]
      }
    }
  }
}

Since profiles have their own section, a profile can use any name, including global, groups, includes, mixins and version.

groups

A group is a list of profiles that run one after the other, in the order they are listed.

In the format v1 a group is a list of profile names. In the format v2 a group is an object, and the list of profile names moves to its profiles property:

PropertyTypeDefaultPurpose
profileslist of profile namesProfiles that belong to the group, in the order they run
descriptionstringDescribes the group, displayed by resticprofile profiles
continue-on-errortrue, false, autoautoKeep running the next profiles after a profile failed
schedulesobjectSchedules of the group, see scheduling a group
​
version: "2"

groups:
  full-backup:
    description: Backup of the system and the documents
    continue-on-error: true
    profiles:
      - root
      - documents

profiles:
  root:
    repository: /backup/root
    password-file: key
    backup:
      source: /

  documents:
    repository: /backup/documents
    password-file: key
    backup:
      source: ~/Documents
version = "2"

[groups.full-backup]
  description = "Backup of the system and the documents"
  continue-on-error = true
  profiles = ["root", "documents"]

[profiles.root]
  repository = "/backup/root"
  password-file = "key"

  [profiles.root.backup]
    source = "/"

[profiles.documents]
  repository = "/backup/documents"
  password-file = "key"

  [profiles.documents.backup]
    source = "~/Documents"
{
  "version": "2",
  "groups": {
    "full-backup": {
      "description": "Backup of the system and the documents",
      "continue-on-error": true,
      "profiles": [
        "root",
        "documents"
      ]
    }
  },
  "profiles": {
    "root": {
      "repository": "/backup/root",
      "password-file": "key",
      "backup": {
        "source": "/"
      }
    },
    "documents": {
      "repository": "/backup/documents",
      "password-file": "key",
      "backup": {
        "source": "~/Documents"
      }
    }
  }
}

A group is used like a profile: its name is given to the --name flag, or as a prefix of the command.

resticprofile --name full-backup backup
resticprofile full-backup.backup

Both commands run backup on the profile root, then on the profile documents.

resticprofile profiles lists the groups with their profiles and description, and resticprofile --name full-backup show displays the configuration of a group, including its schedules.

Warning

Profiles and groups share the same names on the command line. Use a name only once: when a profile and a group have the same name, the profile is selected and resticprofile reports Duplicate profile and group names found.

Error handling

By default, a group stops at the first profile that fails and the remaining profiles do not run.

continue-on-error changes this for one group:

  • true: a failure is logged and the group continues with the next profile
  • false: the group stops at the first failure
  • auto (or not set): the group follows group-continue-on-error from the global section, which is false by default

global.group-continue-on-error is also available in the format v1. Setting it per group requires the format v2.

Scheduling a group

A group can be scheduled as a whole. When the schedule triggers, the scheduled command runs on each profile of the group in sequence, following the error handling of the group.

The schedules property of a group contains one schedule per command. The commands that can be scheduled are backup, check, forget, prune and copy.

​
version: "2"

groups:
  full-backup:
    profiles:
      - root
      - documents
    schedules:
      backup:
        at: "daily"
        permission: system
        lock-wait: 15m
      check:
        at: "Sun 03:00"
        permission: system
        log: check.log

profiles:
  root:
    repository: /backup/root
    password-file: key
    backup:
      source: /

  documents:
    repository: /backup/documents
    password-file: key
    backup:
      source: ~/Documents
version = "2"

[groups.full-backup]
  profiles = ["root", "documents"]

  [groups.full-backup.schedules.backup]
    at = "daily"
    permission = "system"
    lock-wait = "15m"

  [groups.full-backup.schedules.check]
    at = "Sun 03:00"
    permission = "system"
    log = "check.log"

[profiles.root]
  repository = "/backup/root"
  password-file = "key"

  [profiles.root.backup]
    source = "/"

[profiles.documents]
  repository = "/backup/documents"
  password-file = "key"

  [profiles.documents.backup]
    source = "~/Documents"
{
  "version": "2",
  "groups": {
    "full-backup": {
      "profiles": [
        "root",
        "documents"
      ],
      "schedules": {
        "backup": {
          "at": "daily",
          "permission": "system",
          "lock-wait": "15m"
        },
        "check": {
          "at": "Sun 03:00",
          "permission": "system",
          "log": "check.log"
        }
      }
    }
  },
  "profiles": {
    "root": {
      "repository": "/backup/root",
      "password-file": "key",
      "backup": {
        "source": "/"
      }
    },
    "documents": {
      "repository": "/backup/documents",
      "password-file": "key",
      "backup": {
        "source": "~/Documents"
      }
    }
  }
}

Each schedule is an object. Only at is required: a schedule without it is ignored.

The properties are the schedule-* properties of a profile, without the schedule- prefix. They are described in schedule configuration:

Property of a group scheduleEquivalent in a profile
atschedule
permissionschedule-permission
run-levelschedule-run-level
logschedule-log
priorityschedule-priority
lock-modeschedule-lock-mode
lock-waitschedule-lock-wait
capture-environmentschedule-capture-environment
ignore-on-batteryschedule-ignore-on-battery
ignore-on-battery-less-thanschedule-ignore-on-battery-less-than
after-network-onlineschedule-after-network-online
hide-windowschedule-hide-window
start-when-availableschedule-start-when-available

A group schedule also accepts command-output and systemd-drop-in-files. All the properties are listed in the reference.

A property that is not set takes its value from schedule-defaults in the global section.

The schedule of a group is installed, removed and checked with the schedule commands and the name of the group:

resticprofile --name full-backup schedule
resticprofile --name full-backup status
resticprofile --name full-backup unschedule

resticprofile schedule --all installs the schedules of all profiles and all groups.

Note

The schedules of a group are independent of the schedules of its profiles. If the group schedules backup and one of its profiles also schedules backup, both schedules are installed and the profile is backed up by both.

mixins

Mixins are named pieces of configuration, declared in the mixins section. A profile merges a mixin into its configuration with the use property, at any level of the profile. The use property is also accepted in a group and in the global section.

​
version: "2"

mixins:
  retain-last:
    default-vars:
      LAST: 30
    retention:
      keep-last: $LAST

  exclude-hidden:
    exclude...:
      - ".*"

profiles:
  documents:
    repository: /backup/documents
    password-file: key
    use:
      - name: retain-last
        LAST: 60
    backup:
      use: exclude-hidden
      source: ~/Documents
version = "2"

[mixins.retain-last]
  default-vars = { LAST = 30 }

  [mixins.retain-last.retention]
    keep-last = "$LAST"

[mixins.exclude-hidden]
  exclude__APPEND = [".*"]

[profiles.documents]
  repository = "/backup/documents"
  password-file = "key"
  use = [{ name = "retain-last", LAST = 60 }]

  [profiles.documents.backup]
    use = "exclude-hidden"
    source = "~/Documents"
{
  "version": "2",
  "mixins": {
    "retain-last": {
      "default-vars": {
        "LAST": 30
      },
      "retention": {
        "keep-last": "$LAST"
      }
    },
    "exclude-hidden": {
      "exclude...": [
        ".*"
      ]
    }
  },
  "profiles": {
    "documents": {
      "repository": "/backup/documents",
      "password-file": "key",
      "use": [
        {
          "name": "retain-last",
          "LAST": 60
        }
      ],
      "backup": {
        "use": "exclude-hidden",
        "source": "~/Documents"
      }
    }
  }
}

Mixins are described in detail in inheritance.

includes

includes works as in the format v1, see includes. Two rules apply to the version:

  • All the files must use the same version. Loading fails with cannot include different versions of the configuration file, all files must use the same version when the main file and an included file declare a different version.
  • An included file that does not declare a version is read with the version of the main file. Declaring the version in every file is recommended, and the JSON schema of the format v2 requires it.

An included v2 file follows the same structure as the main file: its profiles are in the profiles section. Files in TOML, YAML and JSON can be mixed, and a mixin declared in one file can be used in any other file.

Changes in profile behaviour

The content of a profile is the same in both formats, but a v2 profile is not always resolved to the same configuration as its v1 equivalent.

Tip

Use resticprofile --name <profile-name> show to display the resulting configuration of a profile, and resticprofile --dry-run --name <profile-name> backup to display the restic commands it generates.

Inheritance of lists

When a profile inherits from another profile and both define the same list (source, exclude, tag, run-before…):

  • In the format v1, the two lists are merged index by index. With source: [/a, /b] in the parent and source: [/c] in the derived profile, the result is [/c, /b].
  • In the format v2, the list of the derived profile replaces the list of the parent. With the same example, the result is [/c].

To add values to the list of the parent instead of replacing it, the format v2 adds a syntax to append or prepend:

PropertyEffect
<list-property>... or <list-property>__APPENDAppends to the list of the parent
...<list-property> or <list-property>__PREPENDPrepends to the list of the parent
​
version: "2"

profiles:
  default:
    repository: /backup
    password-file: key
    backup:
      exclude:
        - ".*"
        - "~*"

  documents:
    inherit: default
    backup:
      exclude...: ".git"
      source: ~/Documents
version = "2"

[profiles.default]
  repository = "/backup"
  password-file = "key"

  [profiles.default.backup]
    exclude = [".*", "~*"]

[profiles.documents]
  inherit = "default"

  [profiles.documents.backup]
    exclude__APPEND = ".git"
    source = "~/Documents"
{
  "version": "2",
  "profiles": {
    "default": {
      "repository": "/backup",
      "password-file": "key",
      "backup": {
        "exclude": [
          ".*",
          "~*"
        ]
      }
    },
    "documents": {
      "inherit": "default",
      "backup": {
        "exclude...": ".git",
        "source": "~/Documents"
      }
    }
  }
}

The value of exclude in the profile documents is [".*", "~*", ".git"].

The same syntax is available in mixins.

Defaults for backup and retention

The format v2 sets a few defaults so that a backup and the retention that follows it select the same snapshots. These defaults are not applied in the format v1.

In the backup section:

  • host defaults to true, which is replaced by the host name of the system. Set host in the backup section to use another name.

In the retention section:

  • after-backup defaults to true when at least one keep-* flag is set and neither after-backup nor before-backup is set. Declaring a retention policy is enough to apply it after each backup.
  • tag defaults to true when tag is set in the backup section. The retention only considers the snapshots that have the tags of the backup.
  • host defaults to the host of the backup section. The retention only considers the snapshots of the same host.

Setting any of these properties explicitly in the retention section overrides its default.

For example, this profile:

version: "2"

profiles:
  documents:
    repository: /backup/documents
    password-file: key
    backup:
      source: /home/user/Documents
      tag: documents
    retention:
      keep-last: 30

is resolved as:

profiles:
  documents:
    repository: /backup/documents
    password-file: key
    backup:
      source: /home/user/Documents
      tag: documents
      host: true
    retention:
      keep-last: 30
      after-backup: true
      path: /home/user/Documents
      tag: documents
      host: true

The same profile in the format v1 only sets path in the retention section: the retention does not run after the backup, and forget is not restricted to a host or to the tags of the backup.

JSON schema

The format v2 has its own JSON schema:

The schema of the format v2 requires the version property, as a string.

A schema can also be generated locally:

resticprofile generate --json-schema v2

Migrating from v1

A v1 configuration is converted in a few steps:

  1. Convert the file to TOML, YAML or JSON if it is a HCL file.
  2. Add version: "2" at the top of the file, and of every included file.
  3. Move all the profiles into a profiles section. Leave global, groups and includes at the top level.
  4. Change each group from a list of profile names to an object with a profiles property.
  5. Review the lists in the profiles that use inherit: a list now replaces the list of the parent. Rename the property with a ... suffix where values were meant to be added.
  6. Review the retention sections: a section with a keep-* flag now runs after each backup, and is restricted to the host and the tags of the backup. Set after-backup: false to keep a retention policy that does not run after a backup.
  7. Check the result with resticprofile profiles, which lists all the profiles and groups, then with show and --dry-run on each profile.
​
version: "1"

global:
  default-command: snapshots

groups:
  full-backup:
    - root
    - documents

default:
  repository: /backup
  password-file: key
  backup:
    exclude:
      - ".*"

root:
  inherit: default
  backup:
    source: /

documents:
  inherit: default
  backup:
    source: ~/Documents
version: "2"

global:
  default-command: snapshots

groups:
  full-backup:
    profiles:
      - root
      - documents

profiles:
  default:
    repository: /backup
    password-file: key
    backup:
      exclude:
        - ".*"

  root:
    inherit: default
    backup:
      source: /

  documents:
    inherit: default
    backup:
      source: ~/Documents

The names of the profiles and groups do not change: the schedules that are already installed keep working as long as the configuration file keeps its name.

Complete example

​
version: "2"

global:
  default-command: snapshots
  priority: low
  schedule-defaults:
    permission: system
    log: /var/log/resticprofile.log

mixins:
  retain:
    default-vars:
      DAYS: 14
    retention:
      keep-daily: $DAYS
      keep-weekly: 8

  exclude-caches:
    exclude...:
      - "**/.cache"
      - "**/node_modules"

groups:
  all:
    description: All the backups of this machine
    continue-on-error: true
    profiles:
      - system
      - home
    schedules:
      backup:
        at: "02:30"
        lock-wait: 30m
      check:
        at: "Sun 05:00"

profiles:
  default:
    description: Common settings of all the profiles
    repository: /mnt/backup/{{ .Profile.Name }}
    password-file: key
    initialize: true
    backup:
      exclude-caches: true
      exclude:
        - "*.tmp"

  system:
    description: System files
    inherit: default
    use: retain
    backup:
      source:
        - /etc
        - /var/lib

  home:
    description: Home directories
    inherit: default
    use:
      - name: retain
        DAYS: 60
    backup:
      use: exclude-caches
      source: /home
      tag: home
version = "2"

[global]
  default-command = "snapshots"
  priority = "low"

  [global.schedule-defaults]
    permission = "system"
    log = "/var/log/resticprofile.log"

[mixins.retain]
  default-vars = { DAYS = 14 }

  [mixins.retain.retention]
    keep-daily = "$DAYS"
    keep-weekly = 8

[mixins.exclude-caches]
  exclude__APPEND = ["**/.cache", "**/node_modules"]

[groups.all]
  description = "All the backups of this machine"
  continue-on-error = true
  profiles = ["system", "home"]

  [groups.all.schedules.backup]
    at = "02:30"
    lock-wait = "30m"

  [groups.all.schedules.check]
    at = "Sun 05:00"

[profiles.default]
  description = "Common settings of all the profiles"
  repository = "/mnt/backup/{{ .Profile.Name }}"
  password-file = "key"
  initialize = true

  [profiles.default.backup]
    exclude-caches = true
    exclude = ["*.tmp"]

[profiles.system]
  description = "System files"
  inherit = "default"
  use = "retain"

  [profiles.system.backup]
    source = ["/etc", "/var/lib"]

[profiles.home]
  description = "Home directories"
  inherit = "default"
  use = [{ name = "retain", DAYS = 60 }]

  [profiles.home.backup]
    use = "exclude-caches"
    source = "/home"
    tag = "home"
{
  "version": "2",
  "global": {
    "default-command": "snapshots",
    "priority": "low",
    "schedule-defaults": {
      "permission": "system",
      "log": "/var/log/resticprofile.log"
    }
  },
  "mixins": {
    "retain": {
      "default-vars": {
        "DAYS": 14
      },
      "retention": {
        "keep-daily": "$DAYS",
        "keep-weekly": 8
      }
    },
    "exclude-caches": {
      "exclude...": [
        "**/.cache",
        "**/node_modules"
      ]
    }
  },
  "groups": {
    "all": {
      "description": "All the backups of this machine",
      "continue-on-error": true,
      "profiles": [
        "system",
        "home"
      ],
      "schedules": {
        "backup": {
          "at": "02:30",
          "lock-wait": "30m"
        },
        "check": {
          "at": "Sun 05:00"
        }
      }
    }
  },
  "profiles": {
    "default": {
      "description": "Common settings of all the profiles",
      "repository": "/mnt/backup/{{ .Profile.Name }}",
      "password-file": "key",
      "initialize": true,
      "backup": {
        "exclude-caches": true,
        "exclude": [
          "*.tmp"
        ]
      }
    },
    "system": {
      "description": "System files",
      "inherit": "default",
      "use": "retain",
      "backup": {
        "source": [
          "/etc",
          "/var/lib"
        ]
      }
    },
    "home": {
      "description": "Home directories",
      "inherit": "default",
      "use": [
        {
          "name": "retain",
          "DAYS": 60
        }
      ],
      "backup": {
        "use": "exclude-caches",
        "source": "/home",
        "tag": "home"
      }
    }
  }
}

With this configuration:

  • resticprofile --name home backup backs up /home, excluding *.tmp, **/.cache and **/node_modules, then applies the retention policy: 60 daily and 8 weekly snapshots are kept.
  • resticprofile --name all backup backs up system then home, and backs up home even if system failed.
  • resticprofile --name all schedule installs two schedules: backup every day at 02:30 and check every Sunday at 05:00. Both run on system then home, with the permission and the log file of schedule-defaults.
Tip

The format v2 is discussed in this issue.