mirror of https://github.com/jimsalterjrs/sanoid
Merge branch 'master' into direct-connection
This commit is contained in:
Christoph Klaffl
commit
9e04911945
38
CHANGELIST
38
CHANGELIST
|
|
@ -1,9 +1,35 @@
|
|||
2.1.0 [overall] documentation updates, small fixes (@HavardLine, @croadfeldt, @jimsalterjrs, @jim-perkins, @kr4z33, @phreaker0)
|
||||
[syncoid] do not require user to be specified for syncoid (@aerusso)
|
||||
[syncoid] implemented option for keeping sync snaps (@phreaker0)
|
||||
[syncoid] use sudo if necessary for checking pool capabilities regarding resumable send (@phreaker0)
|
||||
[syncoid] catch another case were the resume state isn't available anymore (@phreaker0)
|
||||
[syncoid] check for an invalid argument combination (@phreaker0)
|
||||
[syncoid] fix iszfsbusy check for similar dataset names (@phreaker0)
|
||||
[syncoid] append timezone offset to the syncoid snapshot name to fix DST collisions (@phreaker0)
|
||||
[packaging] post install script for debian package to remove old unused snapshot cache file (@phreaker0)
|
||||
[syncoid] implemented fallback for listing snapshots on solaris (@phreaker0)
|
||||
[sanoid] remove invalid locks (@phreaker0)
|
||||
[packaging] removed debian dependency for systemd (@phreaker0)
|
||||
[sanoid] move sanoid cache and lock files to subdirectories (@lopsided98)
|
||||
[sanoid] remove 's in monitoring messages (@dlangille)
|
||||
[findoid] reworked argument parsing and error out if file path is not provided (@phreaker0)
|
||||
[findoid] also show current file version if available (@phreaker0)
|
||||
[findoid] handle FileNotFound errors properly (@phreaker0)
|
||||
[findoid] don't use hardcoded paths (@phreaker0)
|
||||
[findoid] improve dataset detection by only including mounted datasets (@phreaker0)
|
||||
[sanoid] pass more information to pre/post/prune scripts and execute them only once per dataset (@tiedotguy, @phreaker0)
|
||||
[syncoid] implemented option for preserving recordsizes on initial replications (@phreaker0)
|
||||
[syncoid] fixed send size estimation for latest FreeBSD versions (@phreaker0)
|
||||
[syncoid] add ability to configure pv (@gdevenyi)
|
||||
[sanoid] don't use hardcoded paths (@phreaker0)
|
||||
[syncoid] gracefully handle error when source dataset disappeared (@mschout)
|
||||
|
||||
2.0.3 [sanoid] reverted DST handling and improved it as quickfix (@phreaker0)
|
||||
|
||||
2.0.2 [overall] documentation updates, new dependencies, small fixes, more warnings (@benyanke, @matveevandrey, @RulerOf, @klemens-u, @johnramsden, @danielewood, @g-a-c, @hartzell, @fryfrog, @phreaker0)
|
||||
[sanoid] changed and simplified DST handling (@shodanshok)
|
||||
[syncoid] reset partially resume state automatically (@phreaker0)
|
||||
[syncoid] handle some zfs erros automatically by parsing the stderr outputs (@phreaker0)
|
||||
[syncoid] handle some zfs errors automatically by parsing the stderr outputs (@phreaker0)
|
||||
[syncoid] fixed ordering of snapshots with the same creation timestamp (@phreaker0)
|
||||
[syncoid] don't use hardcoded paths (@phreaker0)
|
||||
[syncoid] fix for special setup with listsnapshots=on (@phreaker0)
|
||||
|
|
@ -20,7 +46,7 @@
|
|||
[syncoid] option to change mbuffer size (@TerraTech)
|
||||
[tests] fixes for FreeBSD (@phreaker0)
|
||||
[sanoid] support for zfs recursion (@jMichaelA, @phreaker0)
|
||||
[syncoid] fixed bookmark handling for volumens (@ppcontrib)
|
||||
[syncoid] fixed bookmark handling for volumes (@ppcontrib)
|
||||
[sanoid] allow time units for monitoring warn/crit values (@phreaker0)
|
||||
|
||||
2.0.1 [sanoid] fixed broken monthly warn/critical monitoring values in default template (@jimsalterjrs)
|
||||
|
|
@ -58,7 +84,7 @@
|
|||
[sanoid] implemented monitor-capacity flag for checking zpool capacity limits (@phreaker0)
|
||||
[syncoid] Added support for ZStandard compression.(@danielewood)
|
||||
[syncoid] implemented support for excluding datasets from replication with regular expressions (@phreaker0)
|
||||
[syncoid] correctly parse zfs column output, fixes resumeable send with datasets containing spaces (@phreaker0)
|
||||
[syncoid] correctly parse zfs column output, fixes resumable send with datasets containing spaces (@phreaker0)
|
||||
[syncoid] added option for using extra identification in the snapshot name for replication to multiple targets (@phreaker0)
|
||||
[syncoid] added option for skipping the parent dataset in recursive replication (@phreaker0)
|
||||
[syncoid] typos (@UnlawfulMonad, @jsavikko, @phreaker0)
|
||||
|
|
@ -92,12 +118,12 @@
|
|||
replicating to target/parent/child2. This could still use some cleanup TBH; syncoid SHOULD exit 3
|
||||
if any of these errors happen (to assist detection of errors in scripting) but now would exit 0.
|
||||
|
||||
1.4.12 Sanoid now strips trailing whitespace in template definitions in sanoid.conf, per Github #61
|
||||
1.4.12 Sanoid now strips trailing whitespace in template definitions in sanoid.conf, per GitHub #61
|
||||
|
||||
1.4.11 enhanced Syncoid to use zfs `guid` property rather than `creation` property to ensure snapshots on source
|
||||
and target actually match. This immediately prevents conflicts due to timezone differences on source and target,
|
||||
and also paves the way in the future for Syncoid to find matching snapshots even after `zfs rename` on source
|
||||
or target. Thank you Github user @mailinglists35 for the idea!
|
||||
or target. Thank you GitHub user @mailinglists35 for the idea!
|
||||
|
||||
1.4.10 added --compress=pigz-fast and --compress=pigz-slow. On a Xeon E3-1231v3, pigz-fast is equivalent compression
|
||||
to --compress=gzip but with compressed throughput of 75.2 MiB/s instead of 18.1 MiB/s. pigz-slow is around 5%
|
||||
|
|
@ -215,4 +241,4 @@
|
|||
|
||||
1.0.1 ported slightly modified iszfsbusy sub from syncoid to sanoid (to keep from thinning snapshots during replications)
|
||||
|
||||
1.0.0 initial commit to Github
|
||||
1.0.0 initial commit to GitHub
|
||||
|
|
|
|||
|
|
@ -1,4 +1,10 @@
|
|||
FreeBSD users will need to change the Perl shebangs at the top of the executables from #!/usr/bin/perl
|
||||
Syncoid assumes a bourne style shell on remote hosts. Using (t)csh (the default for root under FreeBSD)
|
||||
will cause syncoid to fail cryptically due to 2>&1 output redirects.
|
||||
|
||||
To use syncoid successfully with FreeBSD targets, you must use the chsh command to change the root shell:
|
||||
root@bsd:~# chsh -s /bin/sh
|
||||
|
||||
FreeBSD users will also need to change the Perl shebangs at the top of the executables from #!/usr/bin/perl
|
||||
to #!/usr/local/bin/perl in most cases.
|
||||
|
||||
Sorry folks, but if I set this with #!/usr/bin/env perl as suggested, then nothing works properly
|
||||
|
|
@ -11,14 +17,3 @@ If you don't want to have to change the shebangs, your other option is to drop a
|
|||
root@bsd:~# ln -s /usr/local/bin/perl /usr/bin/perl
|
||||
|
||||
After putting this symlink in place, ANY perl script shebanged for Linux will work on your system too.
|
||||
|
||||
Syncoid assumes a bourne style shell on remote hosts. Using (t)csh (the default for root under FreeBSD)
|
||||
has some known issues:
|
||||
|
||||
* If mbuffer is present, syncoid will fail with an "Ambiguous output redirect." error. So if you:
|
||||
root@bsd:~# ln -s /usr/local/bin/mbuffer /usr/bin/mbuffer
|
||||
make sure the remote user is using an sh compatible shell.
|
||||
|
||||
To change to a compatible shell, use the chsh command:
|
||||
|
||||
root@bsd:~# chsh -s /bin/sh
|
||||
|
|
|
|||
107
INSTALL.md
107
INSTALL.md
|
|
@ -8,6 +8,7 @@
|
|||
- [Debian/Ubuntu](#debianubuntu)
|
||||
- [CentOS](#centos)
|
||||
- [FreeBSD](#freebsd)
|
||||
- [Alpine Linux / busybox](#alpine-Linux-busybox-based-distributions)
|
||||
- [Other OSes](#other-oses)
|
||||
- [Configuration](#configuration)
|
||||
- [Sanoid](#sanoid)
|
||||
|
|
@ -21,39 +22,52 @@ Install prerequisite software:
|
|||
|
||||
```bash
|
||||
|
||||
apt install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer
|
||||
apt install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuffer build-essential git
|
||||
|
||||
```
|
||||
|
||||
Clone this repo, build the debian package and install it (alternatively you can skip the package and do it manually like described below for CentOS):
|
||||
|
||||
```bash
|
||||
# Download the repo as root to avoid changing permissions later
|
||||
sudo git clone https://github.com/jimsalterjrs/sanoid.git
|
||||
git clone https://github.com/jimsalterjrs/sanoid.git
|
||||
cd sanoid
|
||||
# checkout latest stable release or stay on master for bleeding edge stuff (but expect bugs!)
|
||||
git checkout $(git tag | grep "^v" | tail -n 1)
|
||||
ln -s packages/debian .
|
||||
dpkg-buildpackage -uc -us
|
||||
apt install ../sanoid_*_all.deb
|
||||
sudo apt install ../sanoid_*_all.deb
|
||||
```
|
||||
|
||||
Enable sanoid timer:
|
||||
```bash
|
||||
# enable and start the sanoid timer
|
||||
sudo systemctl enable sanoid.timer
|
||||
sudo systemctl start sanoid.timer
|
||||
sudo systemctl enable --now sanoid.timer
|
||||
```
|
||||
|
||||
## CentOS
|
||||
## CentOS/RHEL
|
||||
|
||||
Install prerequisite software:
|
||||
|
||||
```bash
|
||||
# Install and enable epel if we don't already have it, and git too
|
||||
# Install and enable EPEL if we don't already have it, and git too:
|
||||
# (Note that on RHEL we cannot enable EPEL with the epel-release
|
||||
# package, so you should follow the instructions on the main EPEL site.)
|
||||
sudo yum install -y epel-release git
|
||||
# On CentOS, we also need to enable the PowerTools repo:
|
||||
sudo yum config-manager --set-enabled powertools
|
||||
# For Centos 8 you need to enable the PowerTools repo to make all the needed Perl modules available (Recommended)
|
||||
sudo dnf config-manager --set-enabled powertools
|
||||
# On RHEL, instead of PowerTools, we need to enable the CodeReady Builder repo:
|
||||
sudo subscription-manager repos --enable=codeready-builder-for-rhel-8-x86_64-rpms
|
||||
# Install the packages that Sanoid depends on:
|
||||
sudo yum install -y perl-Config-IniFiles perl-Data-Dumper perl-Capture-Tiny lzop mbuffer mhash pv
|
||||
sudo yum install -y perl-Config-IniFiles perl-Data-Dumper perl-Capture-Tiny perl-Getopt-Long lzop mbuffer mhash pv
|
||||
# The repositories above should contain all the relevant Perl modules, but if you
|
||||
# still cannot find them then you can install them from CPAN manually:
|
||||
sudo dnf install perl-CPAN perl-CPAN
|
||||
cpan # answer the questions and paste the following lines:
|
||||
# install Capture::Tiny
|
||||
# install Config::IniFiles
|
||||
# install Getopt::Long
|
||||
```
|
||||
|
||||
Clone this repo, then put the executables and config files into the appropriate directories:
|
||||
|
|
@ -136,8 +150,7 @@ sudo systemctl daemon-reload
|
|||
# Enable sanoid-prune.service to allow it to be triggered by sanoid.service
|
||||
sudo systemctl enable sanoid-prune.service
|
||||
# Enable and start the Sanoid timer
|
||||
sudo systemctl enable sanoid.timer
|
||||
sudo systemctl start sanoid.timer
|
||||
sudo systemctl enable --now sanoid.timer
|
||||
```
|
||||
|
||||
Now, proceed to configure [**Sanoid**](#configuration)
|
||||
|
|
@ -158,6 +171,58 @@ pkg install p5-Config-Inifiles p5-Capture-Tiny pv mbuffer lzop
|
|||
|
||||
* See note about mbuffer and other things in FREEBSD.readme
|
||||
|
||||
## Alpine Linux / busybox based distributions
|
||||
|
||||
The busybox implementation of ps is lacking needed arguments so a proper ps program needs to be installed.
|
||||
For Alpine Linux this can be done with:
|
||||
|
||||
`apk --no-cache add procps`
|
||||
|
||||
## MacOS
|
||||
|
||||
Install prerequisite software:
|
||||
|
||||
```
|
||||
perl -MCPAN -e install Config::IniFiles
|
||||
```
|
||||
|
||||
The crontab can be used as on a normal unix. To use launchd instead, this example config file can be use can be used. Modify it for your needs. In particular, adjust the sanoid path.
|
||||
It will start sanoid once per hour, at minute 51. Missed invocations due to standby will be merged into a single invocation at the next wakeup.
|
||||
|
||||
```bash
|
||||
cat << "EOF" | sudo tee /Library/LaunchDaemons/net.openoid.Sanoid.plist
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>net.openoid.Sanoid</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/usr/local/sanoid/sanoid</string>
|
||||
<string>--cron</string>
|
||||
</array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>TZ</key>
|
||||
<string>UTC</string>
|
||||
<key>PATH</key>
|
||||
<string>/usr/local/zfs/bin:$PATH:/usr/local/bin</string>
|
||||
</dict>
|
||||
<key>StartCalendarInterval</key>
|
||||
<array>
|
||||
<dict>
|
||||
<key>Minute</key>
|
||||
<integer>51</integer>
|
||||
</dict>
|
||||
</array>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
|
||||
sudo launchctl load /Library/LaunchDaemons/net.openoid.Sanoid.plist
|
||||
```
|
||||
|
||||
## Other OSes
|
||||
|
||||
**Sanoid** depends on the Perl module Config::IniFiles and will not operate without it. Config::IniFiles may be installed from CPAN, though the project strongly recommends using your distribution's repositories instead.
|
||||
|
|
@ -171,6 +236,17 @@ pkg install p5-Config-Inifiles p5-Capture-Tiny pv mbuffer lzop
|
|||
3. Create the config directory `/etc/sanoid` and put `sanoid.defaults.conf` in there, and create `sanoid.conf` in it too
|
||||
4. Create a cron job or a systemd timer that runs `sanoid --cron` once per minute
|
||||
|
||||
## cron
|
||||
|
||||
If you use cron there is the need to ensure that only one instance of sanoid is run at any time (or else there will be funny error messages about missing snapshots, ...). It's also good practice to separate the snapshot taking and pruning so the later won't block the former in case of long running pruning operations. Following is the recommend setup for a standard install:
|
||||
|
||||
```
|
||||
*/15 * * * * root flock -n /var/run/sanoid/cron-take.lock -c "TZ=UTC sanoid --take-snapshots"
|
||||
*/15 * * * * root flock -n /var/run/sanoid/cron-prune.lock -c "sanoid --prune-snapshots"
|
||||
```
|
||||
|
||||
Adapt the timer interval to the lowest configured snapshot interval.
|
||||
|
||||
# Configuration
|
||||
|
||||
**Sanoid** won't do anything useful unless you tell it how to handle your ZFS datasets in `/etc/sanoid/sanoid.conf`.
|
||||
|
|
@ -182,3 +258,12 @@ pkg install p5-Config-Inifiles p5-Capture-Tiny pv mbuffer lzop
|
|||
Take a look at the files `sanoid.defaults.conf` and `sanoid.conf` for all possible configuration options.
|
||||
|
||||
Also have a look at the README.md for a simpler suggestion for `sanoid.conf`.
|
||||
|
||||
## Syncoid
|
||||
If you are pushing or pulling from a remote host, create a user with privileges to `ssh` as well as `sudo`. To ensure that `zfs send/receive` can execute, adjust the privileges of the user to execute `sudo` **without** a password for only the `zfs` binary (run `which zfs` to find the path of the `zfs` binary). Modify `/etc/sudoers` by running `# visudo`. Add the following line for your user.
|
||||
|
||||
```
|
||||
...
|
||||
<user> ALL=NOPASSWD: <path of zfs binary>
|
||||
...
|
||||
```
|
||||
|
|
|
|||
138
README.md
138
README.md
|
|
@ -1,15 +1,15 @@
|
|||
<p align="center"><img src="http://www.openoid.net/wp-content/themes/openoid/images/sanoid_logo.png" alt="sanoid logo" title="sanoid logo"></p>
|
||||
|
||||
<img src="http://openoid.net/gplv3-127x51.png" width=127 height=51 align="right">Sanoid is a policy-driven snapshot management tool for ZFS filesystems. When combined with the Linux KVM hypervisor, you can use it to make your systems <a href="http://openoid.net/transcend" target="_blank">functionally immortal</a>.
|
||||
<img src="http://openoid.net/gplv3-127x51.png" width=127 height=51 align="right">Sanoid is a policy-driven snapshot management tool for ZFS filesystems. When combined with the Linux KVM hypervisor, you can use it to make your systems <a href="http://openoid.net/transcend" target="_blank">functionally immortal</a>.
|
||||
|
||||
<p align="center"><a href="https://youtu.be/ZgowLNBsu00" target="_blank"><img src="http://www.openoid.net/sanoid_video_launcher.png" alt="sanoid rollback demo" title="sanoid rollback demo"></a><br clear="all"><sup>(Real time demo: rolling back a full-scale cryptomalware infection in seconds!)</sup></p>
|
||||
|
||||
More prosaically, you can use Sanoid to create, automatically thin, and monitor snapshots and pool health from a single eminently human-readable TOML config file at /etc/sanoid/sanoid.conf. (Sanoid also requires a "defaults" file located at /etc/sanoid/sanoid.defaults.conf, which is not user-editable.) A typical Sanoid system would have a single cron job:
|
||||
More prosaically, you can use Sanoid to create, automatically thin, and monitor snapshots and pool health from a single eminently human-readable TOML config file at /etc/sanoid/sanoid.conf. (Sanoid also requires a "defaults" file located at /etc/sanoid/sanoid.defaults.conf, which is not user-editable.) A typical Sanoid system would have a single cron job but see INSTALL.md for more details:
|
||||
```
|
||||
* * * * * TZ=UTC /usr/local/bin/sanoid --cron
|
||||
```
|
||||
|
||||
`Note`: Using UTC as timezone is recommend to prevent problems with daylight saving times
|
||||
`Note`: Using UTC as timezone is recommended to prevent problems with daylight saving times
|
||||
|
||||
And its /etc/sanoid/sanoid.conf might look something like this:
|
||||
|
||||
|
|
@ -39,6 +39,10 @@ And its /etc/sanoid/sanoid.conf might look something like this:
|
|||
|
||||
Which would be enough to tell sanoid to take and keep 36 hourly snapshots, 30 dailies, 3 monthlies, and no yearlies for all datasets under data/images (but not data/images itself, since process_children_only is set). Except in the case of data/images/win7, which follows the same template (since it's a child of data/images) but only keeps 4 hourlies for whatever reason.
|
||||
|
||||
For more full details on sanoid.conf settings see [Wiki page](https://github.com/jimsalterjrs/sanoid/wiki/Sanoid#options).
|
||||
|
||||
**Note**: Be aware that if you don't specify some interval options the defaults will be used (from /etc/sanoid/sanoid.defaults.conf)
|
||||
|
||||
##### Sanoid Command Line Options
|
||||
|
||||
+ --cron
|
||||
|
|
@ -91,7 +95,7 @@ Which would be enough to tell sanoid to take and keep 36 hourly snapshots, 30 da
|
|||
|
||||
+ --quiet
|
||||
|
||||
Supress non-error output.
|
||||
Suppress non-error output.
|
||||
|
||||
+ --verbose
|
||||
|
||||
|
|
@ -99,7 +103,7 @@ Which would be enough to tell sanoid to take and keep 36 hourly snapshots, 30 da
|
|||
|
||||
+ --debug
|
||||
|
||||
This prints out quite alot of additional information during a sanoid run, and is normally not needed.
|
||||
This prints out quite a lot of additional information during a sanoid run, and is normally not needed.
|
||||
|
||||
+ --readonly
|
||||
|
||||
|
|
@ -109,6 +113,100 @@ Which would be enough to tell sanoid to take and keep 36 hourly snapshots, 30 da
|
|||
|
||||
Show help message.
|
||||
|
||||
### Sanoid script hooks
|
||||
|
||||
There are three script types which can optionally be executed at various stages in the lifecycle of a snapshot:
|
||||
|
||||
#### `pre_snapshot_script`
|
||||
|
||||
Will be executed before the snapshot(s) of a single dataset are taken. The following environment variables are passed:
|
||||
|
||||
| Env vars | Description |
|
||||
| ----------------- | ----------- |
|
||||
| `SANOID_SCRIPT` | The type of script being executed, one of `pre`, `post`, or `prune`. Allows for one script to be used for multiple tasks |
|
||||
| `SANOID_TARGET` | **DEPRECATED** The dataset about to be snapshot (only the first dataset will be provided) |
|
||||
| `SANOID_TARGETS` | Comma separated list of all datasets to be snapshotted (currently only a single dataset, multiple datasets will be possible later with atomic groups) |
|
||||
| `SANOID_SNAPNAME` | **DEPRECATED** The name of the snapshot that will be taken (only the first name will be provided, does not include the dataset name) |
|
||||
| `SANOID_SNAPNAMES` | Comma separated list of all snapshot names that will be taken (does not include the dataset name) |
|
||||
| `SANOID_TYPES` | Comma separated list of all snapshot types to be taken (yearly, monthly, weekly, daily, hourly, frequently) |
|
||||
|
||||
If the script returns a non-zero exit code, the snapshot(s) will not be taken unless `no_inconsistent_snapshot` is false.
|
||||
|
||||
#### `post_snapshot_script`
|
||||
|
||||
Will be executed when:
|
||||
|
||||
- The pre-snapshot script succeeded or
|
||||
- The pre-snapshot script failed and `force_post_snapshot_script` is true.
|
||||
|
||||
| Env vars | Description |
|
||||
| -------------------- | ----------- |
|
||||
| `SANOID_SCRIPT` | as above |
|
||||
| `SANOID_TARGET` | **DEPRECATED** as above |
|
||||
| `SANOID_TARGETS` | as above |
|
||||
| `SANOID_SNAPNAME` | **DEPRECATED** as above |
|
||||
| `SANOID_SNAPNAMES` | as above |
|
||||
| `SANOID_TYPES` | as above |
|
||||
| `SANOID_PRE_FAILURE` | This will indicate if the pre-snapshot script failed |
|
||||
|
||||
#### `pruning_script`
|
||||
|
||||
Will be executed after a snapshot is successfully deleted. The following environment variables will be passed:
|
||||
|
||||
| Env vars | Description |
|
||||
| ----------------- | ----------- |
|
||||
| `SANOID_SCRIPT` | as above |
|
||||
| `SANOID_TARGET` | as above |
|
||||
| `SANOID_SNAPNAME` | as above |
|
||||
|
||||
|
||||
#### example
|
||||
|
||||
**sanoid.conf**:
|
||||
```
|
||||
...
|
||||
[sanoid-test-0]
|
||||
use_template = production
|
||||
recursive = yes
|
||||
pre_snapshot_script = /tmp/debug.sh
|
||||
post_snapshot_script = /tmp/debug.sh
|
||||
pruning_script = /tmp/debug.sh
|
||||
...
|
||||
```
|
||||
|
||||
**verbose sanoid output**:
|
||||
```
|
||||
...
|
||||
executing pre_snapshot_script '/tmp/debug.sh' on dataset 'sanoid-test-0'
|
||||
taking snapshot sanoid-test-0@autosnap_2020-02-12_14:49:33_yearly
|
||||
taking snapshot sanoid-test-0@autosnap_2020-02-12_14:49:33_monthly
|
||||
taking snapshot sanoid-test-0@autosnap_2020-02-12_14:49:33_daily
|
||||
taking snapshot sanoid-test-0@autosnap_2020-02-12_14:49:33_hourly
|
||||
executing post_snapshot_script '/tmp/debug.sh' on dataset 'sanoid-test-0'
|
||||
...
|
||||
```
|
||||
|
||||
**pre script env variables**:
|
||||
```
|
||||
SANOID_SCRIPT=pre
|
||||
SANOID_TARGET=sanoid-test-0/b/bb
|
||||
SANOID_TARGETS=sanoid-test-0/b/bb
|
||||
SANOID_SNAPNAME=autosnap_2020-02-12_14:49:32_yearly
|
||||
SANOID_SNAPNAMES=autosnap_2020-02-12_14:49:32_yearly,autosnap_2020-02-12_14:49:32_monthly,autosnap_2020-02-12_14:49:32_daily,autosnap_2020-02-12_14:49:32_hourly
|
||||
SANOID_TYPES=yearly,monthly,daily,hourly
|
||||
```
|
||||
|
||||
**post script env variables**:
|
||||
```
|
||||
SANOID_SCRIPT=post
|
||||
SANOID_TARGET=sanoid-test-0/b/bb
|
||||
SANOID_TARGETS=sanoid-test-0/b/bb
|
||||
SANOID_SNAPNAME=autosnap_2020-02-12_14:49:32_yearly
|
||||
SANOID_SNAPNAMES=autosnap_2020-02-12_14:49:32_yearly,autosnap_2020-02-12_14:49:32_monthly,autosnap_2020-02-12_14:49:32_daily,autosnap_2020-02-12_14:49:32_hourly
|
||||
SANOID_TYPES=yearly,monthly,daily,hourly
|
||||
SANOID_PRE_FAILURE=0
|
||||
```
|
||||
|
||||
----------
|
||||
|
||||
# Syncoid
|
||||
|
|
@ -134,7 +232,7 @@ syncoid root@remotehost:data/images/vm backup/images/vm
|
|||
Which would pull-replicate the filesystem from the remote host to the local system over an SSH tunnel.
|
||||
|
||||
Syncoid supports recursive replication (replication of a dataset and all its child datasets) and uses mbuffer buffering, lzop compression, and pv progress bars if the utilities are available on the systems used.
|
||||
If ZFS supports resumeable send/receive streams on both the source and target those will be enabled as default.
|
||||
If ZFS supports resumable send/receive streams on both the source and target those will be enabled as default.
|
||||
|
||||
As of 1.4.18, syncoid also automatically supports and enables resume of interrupted replication when both source and target support this feature.
|
||||
|
||||
|
|
@ -188,15 +286,15 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
+ --compress <compression type>
|
||||
|
||||
Currently accepted options: gzip, pigz-fast, pigz-slow, zstd-fast, zstd-slow, lz4, xz, lzo (default) & none. If the selected compression method is unavailable on the source and destination, no compression will be used.
|
||||
Compression method to use for network transfer. Currently accepted options: gzip, pigz-fast, pigz-slow, zstd-fast, zstd-slow, lz4, xz, lzo (default) & none. If the selected compression method is unavailable on the source and destination, no compression will be used.
|
||||
|
||||
+ --source-bwlimit <limit t|g|m|k>
|
||||
|
||||
This is the bandwidth limit in bytes (kbytes, mbytes, etc) per second imposed upon the source. This is mainly used if the target does not have mbuffer installed, but bandwidth limits are desired.
|
||||
This is the bandwidth limit in bytes (kbytes, mbytes, etc) per second imposed upon the source. This is mainly used if the target does not have mbuffer installed, but bandwidth limits are desired.
|
||||
|
||||
+ --target-bw-limit <limit t|g|m|k>
|
||||
+ --target-bwlimit <limit t|g|m|k>
|
||||
|
||||
This is the bandwidth limit in bytes (kbytes, mbytesm etc) per second imposed upon the target. This is mainly used if the source does not have mbuffer installed, but bandwidth limits are desired.
|
||||
This is the bandwidth limit in bytes (kbytes, mbytes, etc) per second imposed upon the target. This is mainly used if the source does not have mbuffer installed, but bandwidth limits are desired.
|
||||
|
||||
+ --no-command-checks
|
||||
|
||||
|
|
@ -210,9 +308,17 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
This argument tells syncoid to restrict itself to existing snapshots, instead of creating a semi-ephemeral syncoid snapshot at execution time. Especially useful in multi-target (A->B, A->C) replication schemes, where you might otherwise accumulate a large number of foreign syncoid snapshots.
|
||||
|
||||
+ --keep-sync-snap
|
||||
|
||||
This argument tells syncoid to skip pruning old snapshots created and used by syncoid for replication if '--no-sync-snap' isn't specified.
|
||||
|
||||
+ --create-bookmark
|
||||
|
||||
This argument tells syncoid to create a zfs bookmark for the newest snapshot after it got replicated successfully. The bookmark name will be equal to the snapshot name. Only works in combination with the --no-sync-snap option. This can be very useful for irregular replication where the last matching snapshot on the source was already deleted but the bookmark remains so a replication is still possible.
|
||||
This argument tells syncoid to create a zfs bookmark for the newest snapshot after it got replicated successfully. The bookmark name will be equal to the snapshot name. Only works in combination with the --no-sync-snap option. This can be very useful for irregular replication where the last matching snapshot on the source was already deleted but the bookmark remains so a replication is still possible.
|
||||
|
||||
+ --preserve-recordsize
|
||||
|
||||
This argument tells syncoid to set the recordsize on the target before writing any data to it matching the one set on the replication src. This only applies to initial sends.
|
||||
|
||||
+ --no-clone-rollback
|
||||
|
||||
|
|
@ -228,15 +334,15 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
+ --no-resume
|
||||
|
||||
This argument tells syncoid to not use resumeable zfs send/receive streams.
|
||||
This argument tells syncoid to not use resumable zfs send/receive streams.
|
||||
|
||||
+ --force-delete
|
||||
|
||||
Remove target datasets recursively (WARNING: this will also affect child datasets with matching snapshots/bookmarks), if there are no matching snapshots/bookmarks.
|
||||
Remove target datasets recursively (WARNING: this will also affect child datasets with matching snapshots/bookmarks), if there are no matching snapshots/bookmarks. Also removes conflicting snapshots if the replication would fail because of a snapshot which has the same name between source and target but different contents.
|
||||
|
||||
+ --no-clone-handling
|
||||
|
||||
This argument tells syncoid to not recreate clones on the targe on initial sync and doing a normal replication instead.
|
||||
This argument tells syncoid to not recreate clones on the target on initial sync, and do a normal replication instead.
|
||||
|
||||
+ --dumpsnaps
|
||||
|
||||
|
|
@ -269,11 +375,11 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
+ --quiet
|
||||
|
||||
Supress non-error output.
|
||||
Suppress non-error output.
|
||||
|
||||
+ --debug
|
||||
|
||||
This prints out quite alot of additional information during a sanoid run, and is normally not needed.
|
||||
This prints out quite a lot of additional information during a sanoid run, and is normally not needed.
|
||||
|
||||
+ --help
|
||||
|
||||
|
|
|
|||
89
findoid
89
findoid
|
|
@ -4,16 +4,26 @@
|
|||
# from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17. A copy should also be available in this
|
||||
# project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
|
||||
|
||||
$::VERSION = '2.1.0';
|
||||
|
||||
use strict;
|
||||
use warnings;
|
||||
use Getopt::Long qw(:config auto_version auto_help);
|
||||
use Pod::Usage;
|
||||
|
||||
my $zfs = 'zfs';
|
||||
my %args = getargs(@ARGV);
|
||||
my %args = ('path' => '');
|
||||
GetOptions(\%args, "path=s") or pod2usage(2);
|
||||
|
||||
my $progversion = '1.4.7';
|
||||
|
||||
if ($args{'version'}) { print "$progversion\n"; exit 0; }
|
||||
if ($args{'path'} eq '') {
|
||||
if (scalar(@ARGV) < 1) {
|
||||
warn "file path missing!\n";
|
||||
pod2usage(2);
|
||||
exit 127;
|
||||
} else {
|
||||
$args{'path'} = $ARGV[0];
|
||||
}
|
||||
}
|
||||
|
||||
my $dataset = getdataset($args{'path'});
|
||||
|
||||
|
|
@ -136,62 +146,21 @@ sub getdataset {
|
|||
return $bestmatch;
|
||||
}
|
||||
|
||||
sub getargs {
|
||||
my @args = @_;
|
||||
my %args;
|
||||
__END__
|
||||
|
||||
my %novaluearg;
|
||||
my %validarg;
|
||||
push my @validargs, ('debug','version');
|
||||
foreach my $item (@validargs) { $validarg{$item} = 1; }
|
||||
push my @novalueargs, ('debug','version');
|
||||
foreach my $item (@novalueargs) { $novaluearg{$item} = 1; }
|
||||
=head1 NAME
|
||||
|
||||
while (my $rawarg = shift(@args)) {
|
||||
my $arg = $rawarg;
|
||||
my $argvalue;
|
||||
if ($rawarg =~ /=/) {
|
||||
# user specified the value for a CLI argument with =
|
||||
# instead of with blank space. separate appropriately.
|
||||
$argvalue = $arg;
|
||||
$arg =~ s/=.*$//;
|
||||
$argvalue =~ s/^.*=//;
|
||||
}
|
||||
if ($rawarg =~ /^--/) {
|
||||
# doubledash arg
|
||||
$arg =~ s/^--//;
|
||||
if (! $validarg{$arg}) { die "ERROR: don't understand argument $rawarg.\n"; }
|
||||
if ($novaluearg{$arg}) {
|
||||
$args{$arg} = 1;
|
||||
} else {
|
||||
# if this CLI arg takes a user-specified value and
|
||||
# we don't already have it, then the user must have
|
||||
# specified with a space, so pull in the next value
|
||||
# from the array as this value rather than as the
|
||||
# next argument.
|
||||
if ($argvalue eq '') { $argvalue = shift(@args); }
|
||||
$args{$arg} = $argvalue;
|
||||
}
|
||||
} elsif ($arg =~ /^-/) {
|
||||
# singledash arg
|
||||
$arg =~ s/^-//;
|
||||
if (! $validarg{$arg}) { die "ERROR: don't understand argument $rawarg.\n"; }
|
||||
if ($novaluearg{$arg}) {
|
||||
$args{$arg} = 1;
|
||||
} else {
|
||||
# if this CLI arg takes a user-specified value and
|
||||
# we don't already have it, then the user must have
|
||||
# specified with a space, so pull in the next value
|
||||
# from the array as this value rather than as the
|
||||
# next argument.
|
||||
if ($argvalue eq '') { $argvalue = shift(@args); }
|
||||
$args{$arg} = $argvalue;
|
||||
}
|
||||
} else {
|
||||
# bare arg
|
||||
$args{'path'} = $arg;
|
||||
}
|
||||
}
|
||||
findoid - ZFS file version listing tool
|
||||
|
||||
return %args;
|
||||
}
|
||||
=head1 SYNOPSIS
|
||||
|
||||
findoid [options] FILE
|
||||
|
||||
FILE local path to file for version listing
|
||||
|
||||
Options:
|
||||
|
||||
--path=FILE alternative to specify file path to list versions for
|
||||
|
||||
--help Prints this helptext
|
||||
--version Prints the version number
|
||||
|
|
|
|||
|
|
@ -1,3 +1,33 @@
|
|||
sanoid (2.1.0) unstable; urgency=medium
|
||||
|
||||
[overall] documentation updates, small fixes (@HavardLine, @croadfeldt, @jimsalterjrs, @jim-perkins, @kr4z33, @phreaker0)
|
||||
[syncoid] do not require user to be specified for syncoid (@aerusso)
|
||||
[syncoid] implemented option for keeping sync snaps (@phreaker0)
|
||||
[syncoid] use sudo if necessary for checking pool capabilities regarding resumable send (@phreaker0)
|
||||
[syncoid] catch another case were the resume state isn't available anymore (@phreaker0)
|
||||
[syncoid] check for an invalid argument combination (@phreaker0)
|
||||
[syncoid] fix iszfsbusy check for similar dataset names (@phreaker0)
|
||||
[syncoid] append timezone offset to the syncoid snapshot name to fix DST collisions (@phreaker0)
|
||||
[packaging] post install script for debian package to remove old unused snapshot cache file (@phreaker0)
|
||||
[syncoid] implemented fallback for listing snapshots on solaris (@phreaker0)
|
||||
[sanoid] remove invalid locks (@phreaker0)
|
||||
[packaging] removed debian dependency for systemd (@phreaker0)
|
||||
[sanoid] move sanoid cache and lock files to subdirectories (@lopsided98)
|
||||
[sanoid] remove 's in monitoring messages (@dlangille)
|
||||
[findoid] reworked argument parsing and error out if file path is not provided (@phreaker0)
|
||||
[findoid] also show current file version if available (@phreaker0)
|
||||
[findoid] handle FileNotFound errors properly (@phreaker0)
|
||||
[findoid] don't use hardcoded paths (@phreaker0)
|
||||
[findoid] improve dataset detection by only including mounted datasets (@phreaker0)
|
||||
[sanoid] pass more information to pre/post/prune scripts and execute them only once per dataset (@tiedotguy, @phreaker0)
|
||||
[syncoid] implemented option for preserving recordsizes on initial replications (@phreaker0)
|
||||
[syncoid] fixed send size estimation for latest FreeBSD versions (@phreaker0)
|
||||
[syncoid] add ability to configure pv (@gdevenyi)
|
||||
[sanoid] don't use hardcoded paths (@phreaker0)
|
||||
[syncoid] gracefully handle error when source dataset disappeared (@mschout)
|
||||
|
||||
-- Jim Salter <github@jrs-s.net> Tue, 24 Nov 2020 11:47:00 +0100
|
||||
|
||||
sanoid (2.0.3) unstable; urgency=medium
|
||||
|
||||
[sanoid] reverted DST handling and improved it as quickfix (@phreaker0)
|
||||
|
|
@ -9,7 +39,7 @@ sanoid (2.0.2) unstable; urgency=medium
|
|||
[overall] documentation updates, new dependencies, small fixes, more warnings (@benyanke, @matveevandrey, @RulerOf, @klemens-u, @johnramsden, @danielewood, @g-a-c, @hartzell, @fryfrog, @phreaker0)
|
||||
[syncoid] changed and simplified DST handling (@shodanshok)
|
||||
[syncoid] reset partially resume state automatically (@phreaker0)
|
||||
[syncoid] handle some zfs erros automatically by parsing the stderr outputs (@phreaker0)
|
||||
[syncoid] handle some zfs errors automatically by parsing the stderr outputs (@phreaker0)
|
||||
[syncoid] fixed ordering of snapshots with the same creation timestamp (@phreaker0)
|
||||
[syncoid] don't use hardcoded paths (@phreaker0)
|
||||
[syncoid] fix for special setup with listsnapshots=on (@phreaker0)
|
||||
|
|
@ -72,7 +102,7 @@ sanoid (2.0.0) unstable; urgency=medium
|
|||
[sanoid] implemented monitor-capacity flag for checking zpool capacity limits (@phreaker0)
|
||||
[syncoid] Added support for ZStandard compression.(@danielewood)
|
||||
[syncoid] implemented support for excluding datasets from replication with regular expressions (@phreaker0)
|
||||
[syncoid] correctly parse zfs column output, fixes resumeable send with datasets containing spaces (@phreaker0)
|
||||
[syncoid] correctly parse zfs column output, fixes resumable send with datasets containing spaces (@phreaker0)
|
||||
[syncoid] added option for using extra identification in the snapshot name for replication to multiple targets (@phreaker0)
|
||||
[syncoid] added option for skipping the parent dataset in recursive replication (@phreaker0)
|
||||
[syncoid] typos (@UnlawfulMonad, @jsavikko, @phreaker0)
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -1,4 +1,4 @@
|
|||
%global version 2.0.3
|
||||
%global version 2.1.0
|
||||
%global git_tag v%{version}
|
||||
|
||||
# Enable with systemctl "enable sanoid.timer"
|
||||
|
|
@ -111,11 +111,13 @@ echo "* * * * * root %{_sbindir}/sanoid --cron" > %{buildroot}%{_docdir}/%{name}
|
|||
%endif
|
||||
|
||||
%changelog
|
||||
* Tue Nov 24 2020 Christoph Klaffl <christoph@phreaker.eu> - 2.1.0
|
||||
- Bump to 2.1.0
|
||||
* Wed Oct 02 2019 Christoph Klaffl <christoph@phreaker.eu> - 2.0.3
|
||||
- Bump to 2.0.3
|
||||
* Wed Sep 25 2019 Christoph Klaffl <christoph@phreaker.eu> - 2.0.2
|
||||
- Bump to 2.0.2
|
||||
* Wed Dec 04 2018 Christoph Klaffl <christoph@phreaker.eu> - 2.0.0
|
||||
* Tue Dec 04 2018 Christoph Klaffl <christoph@phreaker.eu> - 2.0.0
|
||||
- Bump to 2.0.0
|
||||
* Sat Apr 28 2018 Dominic Robinson <github@dcrdev.com> - 1.4.18-1
|
||||
- Bump to 1.4.18
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
cf0ec23c310d2f9416ebabe48f5edb73 sanoid-1.4.18.tar.gz
|
||||
222
sanoid
222
sanoid
|
|
@ -4,7 +4,7 @@
|
|||
# from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17. A copy should also be available in this
|
||||
# project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
|
||||
|
||||
$::VERSION = '2.0.3';
|
||||
$::VERSION = '2.1.0';
|
||||
my $MINIMUM_DEFAULTS_VERSION = 2;
|
||||
|
||||
use strict;
|
||||
|
|
@ -35,10 +35,13 @@ if (keys %args < 2) {
|
|||
$args{'verbose'} = 1;
|
||||
}
|
||||
|
||||
my $pscmd = '/bin/ps';
|
||||
# for compatibility reasons, older versions used hardcoded command paths
|
||||
$ENV{'PATH'} = $ENV{'PATH'} . ":/bin:/sbin";
|
||||
|
||||
my $zfs = '/sbin/zfs';
|
||||
my $zpool = '/sbin/zpool';
|
||||
my $pscmd = 'ps';
|
||||
|
||||
my $zfs = 'zfs';
|
||||
my $zpool = 'zpool';
|
||||
|
||||
my $conf_file = "$args{'configdir'}/sanoid.conf";
|
||||
my $default_conf_file = "$args{'configdir'}/sanoid.defaults.conf";
|
||||
|
|
@ -127,7 +130,7 @@ sub monitor_snapshots {
|
|||
|
||||
my ($config, $snaps, $snapsbytype, $snapsbypath) = @_;
|
||||
my %datestamp = get_date();
|
||||
my $errorlevel = 0;
|
||||
my $errlevel = 0;
|
||||
my $msg;
|
||||
my @msgs;
|
||||
my @paths;
|
||||
|
|
@ -166,7 +169,7 @@ sub monitor_snapshots {
|
|||
my $dispcrit = displaytime($crit);
|
||||
if ( $elapsed > $crit || $elapsed == -1) {
|
||||
if ($crit > 0) {
|
||||
if (! $config{$section}{'monitor_dont_crit'}) { $errorlevel = 2; }
|
||||
if (! $config{$section}{'monitor_dont_crit'}) { $errlevel = 2; }
|
||||
if ($elapsed == -1) {
|
||||
push @msgs, "CRIT: $path has no $type snapshots at all!";
|
||||
} else {
|
||||
|
|
@ -175,7 +178,7 @@ sub monitor_snapshots {
|
|||
}
|
||||
} elsif ($elapsed > $warn) {
|
||||
if ($warn > 0) {
|
||||
if (! $config{$section}{'monitor_dont_warn'} && ($errorlevel < 2) ) { $errorlevel = 1; }
|
||||
if (! $config{$section}{'monitor_dont_warn'} && ($errlevel < 2) ) { $errlevel = 1; }
|
||||
push @msgs, "WARN: $path newest $type snapshot is $dispelapsed old (should be < $dispwarn)";
|
||||
}
|
||||
} else {
|
||||
|
|
@ -193,7 +196,7 @@ sub monitor_snapshots {
|
|||
if ($msg eq '') { $msg = "OK: all monitored datasets \($paths\) have fresh snapshots"; }
|
||||
|
||||
print "$msg\n";
|
||||
exit $errorlevel;
|
||||
exit $errlevel;
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -316,6 +319,25 @@ sub prune_snapshots {
|
|||
if (checklock('sanoid_pruning')) {
|
||||
writelock('sanoid_pruning');
|
||||
foreach my $snap( @prunesnaps ){
|
||||
my $dataset = (split '@', $snap)[0];
|
||||
my $snapname = (split '@', $snap)[1];
|
||||
|
||||
if (! $args{'readonly'} && $config{$dataset}{'pre_pruning_script'}) {
|
||||
$ENV{'SANOID_TARGET'} = $dataset;
|
||||
$ENV{'SANOID_SNAPNAME'} = $snapname;
|
||||
if ($args{'verbose'}) { print "executing pre_pruning_script '".$config{$dataset}{'pre_pruning_script'}."' on dataset '$dataset'\n"; }
|
||||
my $ret = runscript('pre_pruning_script', $dataset);
|
||||
|
||||
delete $ENV{'SANOID_TARGET'};
|
||||
delete $ENV{'SANOID_SNAPNAME'};
|
||||
|
||||
if ($ret != 0) {
|
||||
# warning was already thrown by runscript function
|
||||
# skip pruning if pre snapshot script returns non zero exit code
|
||||
next;
|
||||
}
|
||||
}
|
||||
|
||||
if ($args{'verbose'}) { print "INFO: pruning $snap ... \n"; }
|
||||
if (!$args{'force-prune'} && iszfsbusy($path)) {
|
||||
if ($args{'verbose'}) { print "INFO: deferring pruning of $snap - $path is currently in zfs send or receive.\n"; }
|
||||
|
|
@ -323,16 +345,16 @@ sub prune_snapshots {
|
|||
if (! $args{'readonly'}) {
|
||||
if (system($zfs, "destroy", $snap) == 0) {
|
||||
$pruned{$snap} = 1;
|
||||
my $dataset = (split '@', $snap)[0];
|
||||
my $snapname = (split '@', $snap)[1];
|
||||
if ($config{$dataset}{'pruning_script'}) {
|
||||
$ENV{'SANOID_TARGET'} = $dataset;
|
||||
$ENV{'SANOID_SNAPNAME'} = $snapname;
|
||||
$ENV{'SANOID_SCRIPT'} = 'prune';
|
||||
if ($args{'verbose'}) { print "executing pruning_script '".$config{$dataset}{'pruning_script'}."' on dataset '$dataset'\n"; }
|
||||
my $ret = runscript('pruning_script',$dataset);
|
||||
|
||||
delete $ENV{'SANOID_TARGET'};
|
||||
delete $ENV{'SANOID_SNAPNAME'};
|
||||
delete $ENV{'SANOID_SCRIPT'};
|
||||
}
|
||||
} else {
|
||||
warn "could not remove $snap : $?";
|
||||
|
|
@ -367,7 +389,7 @@ sub take_snapshots {
|
|||
my %datestamp = get_date();
|
||||
my $forcecacheupdate = 0;
|
||||
|
||||
my @newsnaps;
|
||||
my %newsnapsgroup;
|
||||
|
||||
# get utc timestamp of the current day for DST check
|
||||
my $daystartUtc = timelocal(0, 0, 0, $datestamp{'mday'}, ($datestamp{'mon'}-1), $datestamp{'year'});
|
||||
|
|
@ -389,9 +411,9 @@ sub take_snapshots {
|
|||
if ($config{$section}{'process_children_only'}) { next; }
|
||||
|
||||
my $path = $config{$section}{'path'};
|
||||
my @types = ('yearly','monthly','weekly','daily','hourly','frequently');
|
||||
my @types = ('yearly','monthly','weekly','daily','hourly','frequently');
|
||||
|
||||
foreach my $type (@types) {
|
||||
foreach my $type (@types) {
|
||||
if ($config{$section}{$type} > 0) {
|
||||
|
||||
my $newestage; # in seconds
|
||||
|
|
@ -511,55 +533,58 @@ sub take_snapshots {
|
|||
my $maxage = time()-$lastpreferred;
|
||||
|
||||
if ( $newestage > $maxage ) {
|
||||
# update to most current possible datestamp
|
||||
%datestamp = get_date();
|
||||
# print "we should have had a $type snapshot of $path $maxage seconds ago; most recent is $newestage seconds old.\n";
|
||||
|
||||
my $flags = "";
|
||||
# use zfs (atomic) recursion if specified in config
|
||||
if ($config{$section}{'zfs_recursion'}) {
|
||||
$flags .= "r";
|
||||
}
|
||||
if ($handleDst) {
|
||||
$flags .= "d";
|
||||
if (!exists $newsnapsgroup{$path}) {
|
||||
$newsnapsgroup{$path} = {
|
||||
'recursive' => $config{$section}{'zfs_recursion'},
|
||||
'handleDst' => $handleDst,
|
||||
'datasets' => [$path], # for later atomic grouping, currently only a one element array
|
||||
'types' => []
|
||||
};
|
||||
}
|
||||
|
||||
if ($flags ne "") {
|
||||
push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}_$type\@$flags");
|
||||
} else {
|
||||
push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}_$type");
|
||||
}
|
||||
push(@{$newsnapsgroup{$path}{'types'}}, $type);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if ( (scalar(@newsnaps)) > 0) {
|
||||
foreach my $snap ( @newsnaps ) {
|
||||
my $extraMessage = "";
|
||||
my @split = split '@', $snap, -1;
|
||||
my $recursiveFlag = 0;
|
||||
my $dstHandling = 0;
|
||||
if (scalar(@split) == 3) {
|
||||
my $flags = $split[2];
|
||||
if (index($flags, "r") != -1) {
|
||||
$recursiveFlag = 1;
|
||||
$extraMessage = " (zfs recursive)";
|
||||
chop $snap;
|
||||
}
|
||||
if (index($flags, "d") != -1) {
|
||||
$dstHandling = 1;
|
||||
chop $snap;
|
||||
}
|
||||
chop $snap;
|
||||
if (%newsnapsgroup) {
|
||||
while ((my $path, my $snapData) = each(%newsnapsgroup)) {
|
||||
my $recursiveFlag = $snapData->{recursive};
|
||||
my $dstHandling = $snapData->{handleDst};
|
||||
|
||||
my @datasets = @{$snapData->{datasets}};
|
||||
my $dataset = $datasets[0];
|
||||
my @types = @{$snapData->{types}};
|
||||
|
||||
# same timestamp for all snapshots types (daily, hourly, ...)
|
||||
my %datestamp = get_date();
|
||||
my @snapshots;
|
||||
|
||||
foreach my $type (@types) {
|
||||
my $snapname = "autosnap_$datestamp{'sortable'}_$type";
|
||||
push(@snapshots, $snapname);
|
||||
}
|
||||
my $dataset = $split[0];
|
||||
my $snapname = $split[1];
|
||||
|
||||
my $datasetString = join(",", @datasets);
|
||||
my $typeString = join(",", @types);
|
||||
my $snapshotString = join(",", @snapshots);
|
||||
|
||||
my $extraMessage = "";
|
||||
if ($recursiveFlag) {
|
||||
$extraMessage = " (zfs recursive)";
|
||||
}
|
||||
|
||||
my $presnapshotfailure = 0;
|
||||
my $ret = 0;
|
||||
if ($config{$dataset}{'pre_snapshot_script'}) {
|
||||
$ENV{'SANOID_TARGET'} = $dataset;
|
||||
$ENV{'SANOID_SNAPNAME'} = $snapname;
|
||||
$ENV{'SANOID_TARGETS'} = $datasetString;
|
||||
$ENV{'SANOID_SNAPNAME'} = $snapshots[0];
|
||||
$ENV{'SANOID_SNAPNAMES'} = $snapshotString;
|
||||
$ENV{'SANOID_TYPES'} = $typeString;
|
||||
$ENV{'SANOID_SCRIPT'} = 'pre';
|
||||
if ($args{'verbose'}) { print "executing pre_snapshot_script '".$config{$dataset}{'pre_snapshot_script'}."' on dataset '$dataset'\n"; }
|
||||
|
||||
if (!$args{'readonly'}) {
|
||||
|
|
@ -567,7 +592,11 @@ sub take_snapshots {
|
|||
}
|
||||
|
||||
delete $ENV{'SANOID_TARGET'};
|
||||
delete $ENV{'SANOID_TARGETS'};
|
||||
delete $ENV{'SANOID_SNAPNAME'};
|
||||
delete $ENV{'SANOID_SNAPNAMES'};
|
||||
delete $ENV{'SANOID_TYPES'};
|
||||
delete $ENV{'SANOID_SCRIPT'};
|
||||
|
||||
if ($ret != 0) {
|
||||
# warning was already thrown by runscript function
|
||||
|
|
@ -575,47 +604,58 @@ sub take_snapshots {
|
|||
$presnapshotfailure = 1;
|
||||
}
|
||||
}
|
||||
if ($args{'verbose'}) { print "taking snapshot $snap$extraMessage\n"; }
|
||||
if (!$args{'readonly'}) {
|
||||
my $stderr;
|
||||
my $exit;
|
||||
($stderr, $exit) = tee_stderr {
|
||||
if ($recursiveFlag) {
|
||||
system($zfs, "snapshot", "-r", "$snap");
|
||||
} else {
|
||||
system($zfs, "snapshot", "$snap");
|
||||
}
|
||||
};
|
||||
|
||||
$exit == 0 or do {
|
||||
if ($dstHandling) {
|
||||
if ($stderr =~ /already exists/) {
|
||||
$exit = 0;
|
||||
$snap =~ s/_([a-z]+)$/dst_$1/g;
|
||||
if ($args{'verbose'}) { print "taking dst snapshot $snap$extraMessage\n"; }
|
||||
if ($recursiveFlag) {
|
||||
system($zfs, "snapshot", "-r", "$snap") == 0
|
||||
or warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
|
||||
} else {
|
||||
system($zfs, "snapshot", "$snap") == 0
|
||||
or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
|
||||
foreach my $snap (@snapshots) {
|
||||
$snap = "$dataset\@$snap";
|
||||
if ($args{'verbose'}) { print "taking snapshot $snap$extraMessage\n"; }
|
||||
|
||||
if (!$args{'readonly'}) {
|
||||
my $stderr;
|
||||
my $exit;
|
||||
($stderr, $exit) = tee_stderr {
|
||||
if ($recursiveFlag) {
|
||||
system($zfs, "snapshot", "-r", "$snap");
|
||||
} else {
|
||||
system($zfs, "snapshot", "$snap");
|
||||
}
|
||||
};
|
||||
|
||||
$exit == 0 or do {
|
||||
if ($dstHandling) {
|
||||
if ($stderr =~ /already exists/) {
|
||||
$exit = 0;
|
||||
$snap =~ s/_([a-z]+)$/dst_$1/g;
|
||||
if ($args{'verbose'}) { print "taking dst snapshot $snap$extraMessage\n"; }
|
||||
if ($recursiveFlag) {
|
||||
system($zfs, "snapshot", "-r", "$snap") == 0
|
||||
or warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
|
||||
} else {
|
||||
system($zfs, "snapshot", "$snap") == 0
|
||||
or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
};
|
||||
|
||||
$exit == 0 or do {
|
||||
if ($recursiveFlag) {
|
||||
warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
|
||||
} else {
|
||||
warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
|
||||
}
|
||||
};
|
||||
$exit == 0 or do {
|
||||
if ($recursiveFlag) {
|
||||
warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
|
||||
} else {
|
||||
warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if ($config{$dataset}{'post_snapshot_script'}) {
|
||||
if (!$presnapshotfailure or $config{$dataset}{'force_post_snapshot_script'}) {
|
||||
$ENV{'SANOID_TARGET'} = $dataset;
|
||||
$ENV{'SANOID_SNAPNAME'} = $snapname;
|
||||
$ENV{'SANOID_TARGETS'} = $datasetString;
|
||||
$ENV{'SANOID_SNAPNAME'} = $snapshots[0];
|
||||
$ENV{'SANOID_SNAPNAMES'} = $snapshotString;
|
||||
$ENV{'SANOID_TYPES'} = $typeString;
|
||||
$ENV{'SANOID_SCRIPT'} = 'post';
|
||||
$ENV{'SANOID_PRE_FAILURE'} = $presnapshotfailure;
|
||||
if ($args{'verbose'}) { print "executing post_snapshot_script '".$config{$dataset}{'post_snapshot_script'}."' on dataset '$dataset'\n"; }
|
||||
|
||||
if (!$args{'readonly'}) {
|
||||
|
|
@ -623,7 +663,12 @@ sub take_snapshots {
|
|||
}
|
||||
|
||||
delete $ENV{'SANOID_TARGET'};
|
||||
delete $ENV{'SANOID_TARGETS'};
|
||||
delete $ENV{'SANOID_SNAPNAME'};
|
||||
delete $ENV{'SANOID_SNAPNAMES'};
|
||||
delete $ENV{'SANOID_TYPES'};
|
||||
delete $ENV{'SANOID_SCRIPT'};
|
||||
delete $ENV{'SANOID_PRE_FAILURE'};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -853,6 +898,13 @@ sub init {
|
|||
if (! defined ($defaults{'template_default'}{$key})) {
|
||||
die "FATAL ERROR: I don't understand the setting $key you've set in \[$section\] in $conf_file.\n";
|
||||
}
|
||||
|
||||
# in case of duplicate lines we will end up with an array of all values
|
||||
my $value = $ini{$section}{$key};
|
||||
if (ref($value) eq 'ARRAY') {
|
||||
warn "duplicate key '$key' in section '$section', using the value from the first occurence and ignoring the others.\n";
|
||||
$ini{$section}{$key} = $value->[0];
|
||||
}
|
||||
}
|
||||
|
||||
if ($section =~ /^template_/) { next; } # don't process templates directly
|
||||
|
|
@ -861,7 +913,7 @@ sub init {
|
|||
# for sections directly when they've already been defined recursively, without starting them over from scratch.
|
||||
if (! defined ($config{$section}{'initialized'})) {
|
||||
if ($args{'debug'}) { print "DEBUG: initializing \$config\{$section\} with default values from $default_conf_file.\n"; }
|
||||
# set default values from %defaults, which can then be overriden by template
|
||||
# set default values from %defaults, which can then be overridden by template
|
||||
# and/or local settings within the module.
|
||||
foreach my $key (keys %{$defaults{'template_default'}}) {
|
||||
if (! ($key =~ /template|recursive|children_only/)) {
|
||||
|
|
@ -1109,7 +1161,7 @@ sub check_zpool() {
|
|||
}
|
||||
}
|
||||
|
||||
# Tony: Debuging
|
||||
# Tony: Debugging
|
||||
# print "Size: $size \t Used: $used \t Avai: $avail \t Cap: $cap \t Health: $health\n";
|
||||
|
||||
close(STAT);
|
||||
|
|
@ -1211,7 +1263,7 @@ sub check_zpool() {
|
|||
## no display for verbose level 1
|
||||
next if ($verbose==1);
|
||||
## don't display working devices for verbose level 2
|
||||
if ($verbose==2 && ($state eq "OK" || $sta eq "ONLINE" || $sta eq "AVAIL" || $sta eq "INUSE")) {
|
||||
if ($verbose==2 && ($state eq "OK" || $sta eq "ONLINE" || $sta eq "AVAIL")) {
|
||||
# check for io/checksum errors
|
||||
|
||||
my @vdeverr = ();
|
||||
|
|
|
|||
53
sanoid.conf
53
sanoid.conf
|
|
@ -3,30 +3,35 @@
|
|||
# It should go in /etc/sanoid. #
|
||||
######################################
|
||||
|
||||
# name your backup modules with the path to their ZFS dataset - no leading slash.
|
||||
[zpoolname/datasetname]
|
||||
# pick one or more templates - they're defined (and editable) below. Comma separated, processed in order.
|
||||
# in this example, template_demo's daily value overrides template_production's daily value.
|
||||
use_template = production,demo
|
||||
## name your backup modules with the path to their ZFS dataset - no leading slash.
|
||||
#[zpoolname/datasetname]
|
||||
# # pick one or more templates - they're defined (and editable) below. Comma separated, processed in order.
|
||||
# # in this example, template_demo's daily value overrides template_production's daily value.
|
||||
# use_template = production,demo
|
||||
#
|
||||
# # if you want to, you can override settings in the template directly inside module definitions like this.
|
||||
# # in this example, we override the template to only keep 12 hourly and 1 monthly snapshot for this dataset.
|
||||
# hourly = 12
|
||||
# monthly = 1
|
||||
#
|
||||
## you can also handle datasets recursively.
|
||||
#[zpoolname/parent]
|
||||
# use_template = production
|
||||
# recursive = yes
|
||||
# # if you want sanoid to manage the child datasets but leave this one alone, set process_children_only.
|
||||
# process_children_only = yes
|
||||
#
|
||||
## you can selectively override settings for child datasets which already fall under a recursive definition.
|
||||
#[zpoolname/parent/child]
|
||||
# # child datasets already initialized won't be wiped out, so if you use a new template, it will
|
||||
# # only override the values already set by the parent template, not replace it completely.
|
||||
# use_template = demo
|
||||
|
||||
# if you want to, you can override settings in the template directly inside module definitions like this.
|
||||
# in this example, we override the template to only keep 12 hourly and 1 monthly snapshot for this dataset.
|
||||
hourly = 12
|
||||
monthly = 1
|
||||
|
||||
# you can also handle datasets recursively.
|
||||
[zpoolname/parent]
|
||||
# you can also handle datasets recursively in an atomic way without the possibility to override settings for child datasets.
|
||||
[zpoolname/parent2]
|
||||
use_template = production
|
||||
recursive = yes
|
||||
# if you want sanoid to manage the child datasets but leave this one alone, set process_children_only.
|
||||
process_children_only = yes
|
||||
|
||||
# you can selectively override settings for child datasets which already fall under a recursive definition.
|
||||
[zpoolname/parent/child]
|
||||
# child datasets already initialized won't be wiped out, so if you use a new template, it will
|
||||
# only override the values already set by the parent template, not replace it completely.
|
||||
use_template = demo
|
||||
|
||||
recursive = zfs
|
||||
|
||||
|
||||
|
||||
|
|
@ -91,12 +96,14 @@
|
|||
daily_crit = 4d
|
||||
|
||||
[template_scripts]
|
||||
### dataset and snapshot name will be supplied as environment variables
|
||||
### for all pre/post/prune scripts ($SANOID_TARGET, $SANOID_SNAPNAME)
|
||||
### information about the snapshot will be supplied as environment variables,
|
||||
### see the README.md file for details about what is passed when.
|
||||
### run script before snapshot
|
||||
pre_snapshot_script = /path/to/script.sh
|
||||
### run script after snapshot
|
||||
post_snapshot_script = /path/to/script.sh
|
||||
### run script before pruning snapshot
|
||||
pre_pruning_script = /path/to/script.sh
|
||||
### run script after pruning snapshot
|
||||
pruning_script = /path/to/script.sh
|
||||
### don't take an inconsistent snapshot (skip if pre script fails)
|
||||
|
|
|
|||
|
|
@ -19,8 +19,10 @@ use_template =
|
|||
process_children_only =
|
||||
skip_children =
|
||||
|
||||
# See "Sanoid script hooks" in README.md for information about scripts.
|
||||
pre_snapshot_script =
|
||||
post_snapshot_script =
|
||||
pre_pruning_script =
|
||||
pruning_script =
|
||||
script_timeout = 5
|
||||
no_inconsistent_snapshot =
|
||||
|
|
@ -108,5 +110,6 @@ yearly_warn = 0
|
|||
yearly_crit = 0
|
||||
|
||||
# default limits for capacity checks (if set to 0, limit will not be checked)
|
||||
# for overriding these values one needs to specify them in a root pool section! ([tank]\n ...)
|
||||
capacity_warn = 80
|
||||
capacity_crit = 95
|
||||
|
|
|
|||
193
syncoid
193
syncoid
|
|
@ -4,7 +4,7 @@
|
|||
# from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17. A copy should also be available in this
|
||||
# project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
|
||||
|
||||
$::VERSION = '2.0.3';
|
||||
$::VERSION = '2.1.0';
|
||||
|
||||
use strict;
|
||||
use warnings;
|
||||
|
|
@ -20,13 +20,14 @@ my $pvoptions = "-p -t -e -r -b";
|
|||
|
||||
# Blank defaults to use ssh client's default
|
||||
# TODO: Merge into a single "sshflags" option?
|
||||
my %args = ('sshkey' => '', 'sshport' => '', 'sshcipher' => '', 'sshoption' => [], 'target-bwlimit' => '', 'source-bwlimit' => '');
|
||||
my %args = ('sshconfig' => '', 'sshkey' => '', 'sshport' => '', 'sshcipher' => '', 'sshoption' => [], 'target-bwlimit' => '', 'source-bwlimit' => '');
|
||||
GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsnaps", "recursive|r", "sendoptions=s", "recvoptions=s",
|
||||
"source-bwlimit=s", "target-bwlimit=s", "sshkey=s", "sshport=i", "sshcipher|c=s", "sshoption|o=s@",
|
||||
"source-bwlimit=s", "target-bwlimit=s", "sshconfig=s", "sshkey=s", "sshport=i", "sshcipher|c=s", "sshoption|o=s@",
|
||||
"debug", "quiet", "no-stream", "no-sync-snap", "no-resume", "exclude=s@", "skip-parent", "identifier=s",
|
||||
"no-clone-handling", "no-privilege-elevation", "force-delete", "no-clone-rollback", "no-rollback",
|
||||
"create-bookmark", "pv-options=s" => \$pvoptions, "insecure-direct-connection=s",
|
||||
"mbuffer-size=s" => \$mbuffer_size) or pod2usage(2);
|
||||
"no-clone-handling", "no-privilege-elevation", "force-delete", "no-rollback", "create-bookmark",
|
||||
"pv-options=s" => \$pvoptions, "keep-sync-snap", "preserve-recordsize", "mbuffer-size=s" => \$mbuffer_size,
|
||||
"insecure-direct-connection=s")
|
||||
or pod2usage(2);
|
||||
|
||||
my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be done with GetOptions arg, as default still needs to be set
|
||||
|
||||
|
|
@ -38,6 +39,16 @@ if (length $args{'sendoptions'}) {
|
|||
pod2usage(2);
|
||||
exit 127;
|
||||
}
|
||||
|
||||
if (defined $args{'recursive'}) {
|
||||
foreach my $option(@sendoptions) {
|
||||
if ($option->{option} eq 'R') {
|
||||
warn "invalid argument combination, zfs send -R and --recursive aren't compatible!";
|
||||
pod2usage(2);
|
||||
exit 127;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
my @recvoptions = ();
|
||||
|
|
@ -99,6 +110,9 @@ if (length $args{'sshcipher'}) {
|
|||
if (length $args{'sshport'}) {
|
||||
$args{'sshport'} = "-p $args{'sshport'}";
|
||||
}
|
||||
if (length $args{'sshconfig'}) {
|
||||
$args{'sshconfig'} = "-F $args{'sshconfig'}";
|
||||
}
|
||||
if (length $args{'sshkey'}) {
|
||||
$args{'sshkey'} = "-i $args{'sshkey'}";
|
||||
}
|
||||
|
|
@ -116,7 +130,7 @@ if (length $args{'identifier'}) {
|
|||
}
|
||||
|
||||
# figure out if source and/or target are remote.
|
||||
$sshcmd = "$sshcmd $args{'sshcipher'} $sshoptions $args{'sshport'} $args{'sshkey'}";
|
||||
$sshcmd = "$sshcmd $args{'sshconfig'} $args{'sshcipher'} $sshoptions $args{'sshport'} $args{'sshkey'}";
|
||||
if ($debug) { print "DEBUG: SSHCMD: $sshcmd\n"; }
|
||||
my ($sourcehost,$sourcefs,$sourceisroot) = getssh($rawsourcefs);
|
||||
my ($targethost,$targetfs,$targetisroot) = getssh($rawtargetfs);
|
||||
|
|
@ -172,6 +186,8 @@ my %avail = checkcommands();
|
|||
my %snaps;
|
||||
my $exitcode = 0;
|
||||
|
||||
my $replicationCount = 0;
|
||||
|
||||
## break here to call replication individually so that we ##
|
||||
## can loop across children separately, for recursive ##
|
||||
## replication ##
|
||||
|
|
@ -331,7 +347,7 @@ sub syncdataset {
|
|||
|
||||
if (!defined $sync) {
|
||||
# zfs already printed the corresponding error
|
||||
if ($error =~ /\bdataset does not exist\b/) {
|
||||
if ($error =~ /\bdataset does not exist\b/ && $replicationCount > 0) {
|
||||
if (!$quiet) { print "WARN Skipping dataset (dataset no longer exists): $sourcefs...\n"; }
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -371,6 +387,7 @@ sub syncdataset {
|
|||
|
||||
my $receiveextraargs = "";
|
||||
my $receivetoken;
|
||||
|
||||
if ($resume) {
|
||||
# save state of interrupted receive stream
|
||||
$receiveextraargs = "-s";
|
||||
|
|
@ -469,6 +486,14 @@ sub syncdataset {
|
|||
}
|
||||
my $oldestsnapescaped = escapeshellparam($oldestsnap);
|
||||
|
||||
if (defined $args{'preserve-recordsize'}) {
|
||||
my $type = getzfsvalue($sourcehost,$sourcefs,$sourceisroot,'type');
|
||||
if ($type eq "filesystem") {
|
||||
my $recordsize = getzfsvalue($sourcehost,$sourcefs,$sourceisroot,'recordsize');
|
||||
$recvoptions .= "-o recordsize=$recordsize"
|
||||
}
|
||||
}
|
||||
|
||||
my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $sourcefsescaped\@$oldestsnapescaped";
|
||||
my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
|
||||
|
||||
|
|
@ -591,7 +616,10 @@ sub syncdataset {
|
|||
}
|
||||
|
||||
$exit == 0 or do {
|
||||
if ($stdout =~ /\Qused in the initial send no longer exists\E/) {
|
||||
if (
|
||||
$stdout =~ /\Qused in the initial send no longer exists\E/ ||
|
||||
$stdout =~ /incremental source [0-9xa-f]+ no longer exists/
|
||||
) {
|
||||
if (!$quiet) { print "WARN: resetting partially receive state because the snapshot source no longer exists\n"; }
|
||||
resetreceivestate($targethost,$targetfs,$targetisroot);
|
||||
# do an normal sync cycle
|
||||
|
|
@ -701,21 +729,6 @@ sub syncdataset {
|
|||
return 0;
|
||||
} else {
|
||||
my $matchingsnapescaped = escapeshellparam($matchingsnap);
|
||||
# rollback target to matchingsnap
|
||||
if (!defined $args{'no-rollback'}) {
|
||||
my $rollbacktype = "-R";
|
||||
if (defined $args{'no-clone-rollback'}) {
|
||||
$rollbacktype = "-r";
|
||||
}
|
||||
if ($debug) { print "DEBUG: rolling back target to $targetfs\@$matchingsnap...\n"; }
|
||||
if ($targethost ne '') {
|
||||
if ($debug) { print "$sshcmd $targethost $targetsudocmd $zfscmd rollback $rollbacktype $targetfsescaped\@$matchingsnapescaped\n"; }
|
||||
system ("$sshcmd $targethost " . escapeshellparam("$targetsudocmd $zfscmd rollback $rollbacktype $targetfsescaped\@$matchingsnapescaped"));
|
||||
} else {
|
||||
if ($debug) { print "$targetsudocmd $zfscmd rollback $rollbacktype $targetfsescaped\@$matchingsnapescaped\n"; }
|
||||
system ("$targetsudocmd $zfscmd rollback $rollbacktype $targetfsescaped\@$matchingsnapescaped");
|
||||
}
|
||||
}
|
||||
|
||||
my $nextsnapshot = 0;
|
||||
|
||||
|
|
@ -833,6 +846,36 @@ sub syncdataset {
|
|||
if ($exitcode < 2) { $exitcode = 2; }
|
||||
return 0;
|
||||
}
|
||||
} elsif ($args{'force-delete'} && $stdout =~ /\Qdestination already exists\E/) {
|
||||
(my $existing) = $stdout =~ m/^cannot restore to ([^:]*): destination already exists$/g;
|
||||
if ($existing eq "") {
|
||||
warn "CRITICAL ERROR: $synccmd failed: $?";
|
||||
if ($exitcode < 2) { $exitcode = 2; }
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!$quiet) { print "WARN: removing existing destination: $existing\n"; }
|
||||
my $rcommand = '';
|
||||
my $mysudocmd = '';
|
||||
my $existingescaped = escapeshellparam($existing);
|
||||
|
||||
if ($targethost ne '') { $rcommand = "$sshcmd $targethost"; }
|
||||
if (!$targetisroot) { $mysudocmd = $sudocmd; }
|
||||
|
||||
my $prunecmd = "$mysudocmd $zfscmd destroy $existingescaped; ";
|
||||
if ($targethost ne '') {
|
||||
$prunecmd = escapeshellparam($prunecmd);
|
||||
}
|
||||
|
||||
my $ret = system("$rcommand $prunecmd");
|
||||
if ($ret != 0) {
|
||||
warn "CRITICAL ERROR: $rcommand $prunecmd failed: $?";
|
||||
if ($exitcode < 2) { $exitcode = 2; }
|
||||
return 0;
|
||||
} else {
|
||||
# redo sync and skip snapshot creation (already taken)
|
||||
return syncdataset($sourcehost, $sourcefs, $targethost, $targetfs, undef, 1);
|
||||
}
|
||||
} else {
|
||||
warn "CRITICAL ERROR: $synccmd failed: $?";
|
||||
if ($exitcode < 2) { $exitcode = 2; }
|
||||
|
|
@ -848,6 +891,8 @@ sub syncdataset {
|
|||
}
|
||||
}
|
||||
|
||||
$replicationCount++;
|
||||
|
||||
if (defined $args{'no-sync-snap'}) {
|
||||
if (defined $args{'create-bookmark'}) {
|
||||
my $bookmarkcmd;
|
||||
|
|
@ -858,7 +903,7 @@ sub syncdataset {
|
|||
}
|
||||
if ($debug) { print "DEBUG: $bookmarkcmd\n"; }
|
||||
system($bookmarkcmd) == 0 or do {
|
||||
# fallback: assume nameing conflict and try again with guid based suffix
|
||||
# fallback: assume naming conflict and try again with guid based suffix
|
||||
my $guid = $snaps{'source'}{$newsyncsnap}{'guid'};
|
||||
$guid = substr($guid, 0, 6);
|
||||
|
||||
|
|
@ -878,9 +923,11 @@ sub syncdataset {
|
|||
};
|
||||
}
|
||||
} else {
|
||||
# prune obsolete sync snaps on source and target (only if this run created ones).
|
||||
pruneoldsyncsnaps($sourcehost,$sourcefs,$newsyncsnap,$sourceisroot,keys %{ $snaps{'source'}});
|
||||
pruneoldsyncsnaps($targethost,$targetfs,$newsyncsnap,$targetisroot,keys %{ $snaps{'target'}});
|
||||
if (!defined $args{'keep-sync-snap'}) {
|
||||
# prune obsolete sync snaps on source and target (only if this run created ones).
|
||||
pruneoldsyncsnaps($sourcehost,$sourcefs,$newsyncsnap,$sourceisroot,keys %{ $snaps{'source'}});
|
||||
pruneoldsyncsnaps($targethost,$targetfs,$newsyncsnap,$targetisroot,keys %{ $snaps{'target'}});
|
||||
}
|
||||
}
|
||||
|
||||
} # end syncdataset()
|
||||
|
|
@ -1124,11 +1171,11 @@ sub checkcommands {
|
|||
my $resumechkcmd = "$zpoolcmd get -o value -H feature\@extensible_dataset";
|
||||
|
||||
if ($debug) { print "DEBUG: checking availability of zfs resume feature on source...\n"; }
|
||||
$avail{'sourceresume'} = system("$sourcessh $resumechkcmd $srcpool 2>/dev/null | grep '\\(active\\|enabled\\)' >/dev/null 2>&1");
|
||||
$avail{'sourceresume'} = system("$sourcessh $sourcesudocmd $resumechkcmd $srcpool 2>/dev/null | grep '\\(active\\|enabled\\)' >/dev/null 2>&1");
|
||||
$avail{'sourceresume'} = $avail{'sourceresume'} == 0 ? 1 : 0;
|
||||
|
||||
if ($debug) { print "DEBUG: checking availability of zfs resume feature on target...\n"; }
|
||||
$avail{'targetresume'} = system("$targetssh $resumechkcmd $dstpool 2>/dev/null | grep '\\(active\\|enabled\\)' >/dev/null 2>&1");
|
||||
$avail{'targetresume'} = system("$targetssh $targetsudocmd $resumechkcmd $dstpool 2>/dev/null | grep '\\(active\\|enabled\\)' >/dev/null 2>&1");
|
||||
$avail{'targetresume'} = $avail{'targetresume'} == 0 ? 1 : 0;
|
||||
|
||||
if ($avail{'sourceresume'} == 0 || $avail{'targetresume'} == 0) {
|
||||
|
|
@ -1164,7 +1211,7 @@ sub iszfsbusy {
|
|||
|
||||
foreach my $process (@processes) {
|
||||
# if ($debug) { print "DEBUG: checking process $process...\n"; }
|
||||
if ($process =~ /zfs *(receive|recv).*\Q$fs\E/) {
|
||||
if ($process =~ /zfs *(receive|recv)[^\/]*\Q$fs\E\Z/) {
|
||||
# there's already a zfs receive process for our target filesystem - return true
|
||||
if ($debug) { print "DEBUG: process $process matches target $fs!\n"; }
|
||||
return 1;
|
||||
|
|
@ -1495,20 +1542,53 @@ sub targetexists {
|
|||
sub getssh {
|
||||
my $fs = shift;
|
||||
|
||||
my $rhost;
|
||||
my $rhost = "";
|
||||
my $isroot;
|
||||
my $socket;
|
||||
my $remoteuser = "";
|
||||
|
||||
# if we got passed something with an @ in it, we assume it's an ssh connection, eg root@myotherbox
|
||||
if ($fs =~ /\@/) {
|
||||
$rhost = $fs;
|
||||
$fs =~ s/^\S*\@\S*://;
|
||||
$fs =~ s/^[^\@:]*\@[^\@:]*://;
|
||||
$rhost =~ s/:\Q$fs\E$//;
|
||||
my $remoteuser = $rhost;
|
||||
$remoteuser =~ s/\@.*$//;
|
||||
$remoteuser = $rhost;
|
||||
$remoteuser =~ s/\@.*$//;
|
||||
# do not require a username to be specified
|
||||
$rhost =~ s/^@//;
|
||||
} elsif ($fs =~ m{^[^/]*:}) {
|
||||
# if we got passed something with an : in it, BEFORE any forward slash
|
||||
# (i.e., not in a dataset name) it MAY be an ssh connection
|
||||
# but we need to check if there is a pool with that name
|
||||
my $pool = $fs;
|
||||
$pool =~ s%/.*$%%;
|
||||
my ($pools, $error, $exit) = capture {
|
||||
system("$zfscmd list -d0 -H -oname");
|
||||
};
|
||||
$rhost = $fs;
|
||||
if ($exit != 0) {
|
||||
warn "Unable to enumerate pools (is zfs available?)";
|
||||
} else {
|
||||
foreach (split(/\n/,$pools)) {
|
||||
if ($_ eq $pool) {
|
||||
# there's a pool with this name.
|
||||
$rhost = "";
|
||||
last;
|
||||
}
|
||||
}
|
||||
}
|
||||
if ($rhost ne "") {
|
||||
# there's no pool that might conflict with this
|
||||
$rhost =~ s/:.*$//;
|
||||
$fs =~ s/\Q$rhost\E://;
|
||||
}
|
||||
}
|
||||
|
||||
if ($rhost ne "") {
|
||||
if ($remoteuser eq 'root' || $args{'no-privilege-elevation'}) { $isroot = 1; } else { $isroot = 0; }
|
||||
# now we need to establish a persistent master SSH connection
|
||||
$socket = "/tmp/syncoid-$remoteuser-$rhost-" . time();
|
||||
$socket = "/tmp/syncoid-$rhost-" . time() . "-" . int(rand(10000));
|
||||
|
||||
open FH, "$sshcmd -M -S $socket -o ControlPersist=1m $args{'sshport'} $rhost exit |";
|
||||
close FH;
|
||||
|
||||
|
|
@ -1539,6 +1619,8 @@ sub getsnaps() {
|
|||
my $fsescaped = escapeshellparam($fs);
|
||||
if ($isroot) { $mysudocmd = ''; } else { $mysudocmd = $sudocmd; }
|
||||
|
||||
my $rhostOriginal = $rhost;
|
||||
|
||||
if ($rhost ne '') {
|
||||
$rhost = "$sshcmd $rhost";
|
||||
# double escaping needed
|
||||
|
|
@ -1556,7 +1638,7 @@ sub getsnaps() {
|
|||
my @rawsnaps = <FH>;
|
||||
close FH or do {
|
||||
# fallback (solaris for example doesn't support the -t option)
|
||||
return getsnapsfallback($type,$rhost,$fs,$isroot,%snaps);
|
||||
return getsnapsfallback($type,$rhostOriginal,$fs,$isroot,%snaps);
|
||||
};
|
||||
|
||||
# this is a little obnoxious. get guid,creation returns guid,creation on two separate lines
|
||||
|
|
@ -1739,7 +1821,7 @@ sub getbookmarks() {
|
|||
$creation =~ s/^.*\tcreation\t*(\d*).*/$1/;
|
||||
my $bookmark = $line;
|
||||
$bookmark =~ s/^.*\#(.*)\tcreation.*$/$1/;
|
||||
$bookmarks{$lastguid}{'creation'}=$creation;
|
||||
$bookmarks{$lastguid}{'creation'}=$creation . "000";
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1823,7 +1905,19 @@ sub getsendsize {
|
|||
}
|
||||
|
||||
sub getdate {
|
||||
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
|
||||
my @time = localtime(time);
|
||||
|
||||
# get timezone info
|
||||
my $offset = timegm(@time) - timelocal(@time);
|
||||
my $sign = ''; # + is not allowed in a snapshot name
|
||||
if ($offset < 0) {
|
||||
$sign = '-';
|
||||
$offset = abs($offset);
|
||||
}
|
||||
my $hours = int($offset / 3600);
|
||||
my $minutes = int($offset / 60) - $hours * 60;
|
||||
|
||||
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = @time;
|
||||
$year += 1900;
|
||||
my %date;
|
||||
$date{'unix'} = (((((((($year - 1971) * 365) + $yday) * 24) + $hour) * 60) + $min) * 60) + $sec;
|
||||
|
|
@ -1833,7 +1927,8 @@ sub getdate {
|
|||
$date{'hour'} = sprintf ("%02u", $hour);
|
||||
$date{'mday'} = sprintf ("%02u", $mday);
|
||||
$date{'mon'} = sprintf ("%02u", ($mon + 1));
|
||||
$date{'stamp'} = "$date{'year'}-$date{'mon'}-$date{'mday'}:$date{'hour'}:$date{'min'}:$date{'sec'}";
|
||||
$date{'tzoffset'} = sprintf ("GMT%s%02d:%02u", $sign, $hours, $minutes);
|
||||
$date{'stamp'} = "$date{'year'}-$date{'mon'}-$date{'mday'}:$date{'hour'}:$date{'min'}:$date{'sec'}-$date{'tzoffset'}";
|
||||
return %date;
|
||||
}
|
||||
|
||||
|
|
@ -1957,9 +2052,9 @@ syncoid - ZFS snapshot replication tool
|
|||
=head1 SYNOPSIS
|
||||
|
||||
syncoid [options]... SOURCE TARGET
|
||||
or syncoid [options]... SOURCE USER@HOST:TARGET
|
||||
or syncoid [options]... USER@HOST:SOURCE TARGET
|
||||
or syncoid [options]... USER@HOST:SOURCE USER@HOST:TARGET
|
||||
or syncoid [options]... SOURCE [[USER]@]HOST:TARGET
|
||||
or syncoid [options]... [[USER]@]HOST:SOURCE TARGET
|
||||
or syncoid [options]... [[USER]@]HOST:SOURCE [[USER]@]HOST:TARGET
|
||||
|
||||
SOURCE Source ZFS dataset. Can be either local or remote
|
||||
TARGET Target ZFS dataset. Can be either local or remote
|
||||
|
|
@ -1976,12 +2071,14 @@ Options:
|
|||
--pv-options=OPTIONS Configure how pv displays the progress bar, default '-p -t -e -r -b'
|
||||
--no-stream Replicates using newest snapshot instead of intermediates
|
||||
--no-sync-snap Does not create new snapshot, only transfers existing
|
||||
--keep-sync-snap Don't destroy created sync snapshots
|
||||
--create-bookmark Creates a zfs bookmark for the newest snapshot on the source after replication succeeds (only works with --no-sync-snap)
|
||||
--no-clone-rollback Does not rollback clones on target
|
||||
--no-rollback Does not rollback clones or snapshots on target (it probably requires a readonly target)
|
||||
--preserve-recordsize Preserves the recordsize on initial sends to the target
|
||||
--no-rollback Does not rollback snapshots on target (it probably requires a readonly target)
|
||||
--exclude=REGEX Exclude specific datasets which match the given regular expression. Can be specified multiple times
|
||||
--sendoptions=OPTIONS Use advanced options for zfs send (the arguments are filterd as needed), e.g. syncoid --sendoptions="Lc e" sets zfs send -L -c -e ...
|
||||
--recvoptions=OPTIONS Use advanced options for zfs receive (the arguments are filterd as needed), e.g. syncoid --recvoptions="ux recordsize o compression=lz4" sets zfs receive -u -x recordsize -o compression=lz4 ...
|
||||
--sendoptions=OPTIONS Use advanced options for zfs send (the arguments are filtered as needed), e.g. syncoid --sendoptions="Lc e" sets zfs send -L -c -e ...
|
||||
--recvoptions=OPTIONS Use advanced options for zfs receive (the arguments are filtered as needed), e.g. syncoid --recvoptions="ux recordsize o compression=lz4" sets zfs receive -u -x recordsize -o compression=lz4 ...
|
||||
--sshconfig=FILE Specifies an ssh_config(5) file to be used
|
||||
--sshkey=FILE Specifies a ssh key to use to connect
|
||||
--sshport=PORT Connects to remote on a particular port
|
||||
--sshcipher|c=CIPHER Passes CIPHER to ssh to use a particular cipher set
|
||||
|
|
@ -1999,4 +2096,4 @@ Options:
|
|||
--no-clone-handling Don't try to recreate clones on target
|
||||
--no-privilege-elevation Bypass the root check, for use with ZFS permission delegation
|
||||
|
||||
--force-delete Remove target datasets recursively, if there are no matching snapshots/bookmarks
|
||||
--force-delete Remove target datasets recursively, if there are no matching snapshots/bookmarks (also overwrites conflicting named snapshots)
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ function checkEnvironment {
|
|||
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
|
||||
echo "you should be running this test in a"
|
||||
echo "dedicated vm, as it will mess with your system!"
|
||||
echo "Are you sure you wan't to continue? (y)"
|
||||
echo "Are you sure you want to continue? (y)"
|
||||
echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
|
||||
set -x
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,48 @@
|
|||
#!/bin/bash
|
||||
|
||||
# test preserving the recordsize from the src filesystem to the target one
|
||||
|
||||
set -x
|
||||
set -e
|
||||
|
||||
. ../../common/lib.sh
|
||||
|
||||
POOL_IMAGE="/tmp/syncoid-test-7.zpool"
|
||||
MOUNT_TARGET="/tmp/syncoid-test-7.mount"
|
||||
POOL_SIZE="1000M"
|
||||
POOL_NAME="syncoid-test-7"
|
||||
|
||||
truncate -s "${POOL_SIZE}" "${POOL_IMAGE}"
|
||||
|
||||
zpool create -m none -f "${POOL_NAME}" "${POOL_IMAGE}"
|
||||
|
||||
function cleanUp {
|
||||
zpool export "${POOL_NAME}"
|
||||
}
|
||||
|
||||
# export pool in any case
|
||||
trap cleanUp EXIT
|
||||
|
||||
zfs create "${POOL_NAME}"/src
|
||||
zfs create -V 100M -o volblocksize=4k "${POOL_NAME}"/src/zvol4
|
||||
zfs create -V 100M -o volblocksize=16k "${POOL_NAME}"/src/zvol16
|
||||
zfs create -V 100M -o volblocksize=64k "${POOL_NAME}"/src/zvol64
|
||||
zfs create -o recordsize=16k "${POOL_NAME}"/src/16
|
||||
zfs create -o recordsize=32k "${POOL_NAME}"/src/32
|
||||
zfs create -o recordsize=128k "${POOL_NAME}"/src/128
|
||||
../../../syncoid --preserve-recordsize --recursive --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
|
||||
zfs get recordsize -t filesystem -r "${POOL_NAME}"/dst
|
||||
zfs get volblocksize -t volume -r "${POOL_NAME}"/dst
|
||||
|
||||
if [ "$(zfs get recordsize -H -o value -t filesystem "${POOL_NAME}"/dst/16)" != "16K" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get recordsize -H -o value -t filesystem "${POOL_NAME}"/dst/32)" != "32K" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get recordsize -H -o value -t filesystem "${POOL_NAME}"/dst/128)" != "128K" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
#!/bin/bash
|
||||
|
||||
# test replication with deletion of conflicting snapshot on target
|
||||
|
||||
set -x
|
||||
set -e
|
||||
|
||||
. ../../common/lib.sh
|
||||
|
||||
POOL_IMAGE="/tmp/syncoid-test-8.zpool"
|
||||
POOL_SIZE="200M"
|
||||
POOL_NAME="syncoid-test-8"
|
||||
TARGET_CHECKSUM="ee439200c9fa54fc33ce301ef64d4240a6c5587766bfeb651c5cf358e11ec89d -"
|
||||
|
||||
truncate -s "${POOL_SIZE}" "${POOL_IMAGE}"
|
||||
|
||||
zpool create -m none -f "${POOL_NAME}" "${POOL_IMAGE}"
|
||||
|
||||
function cleanUp {
|
||||
zpool export "${POOL_NAME}"
|
||||
}
|
||||
|
||||
# export pool in any case
|
||||
trap cleanUp EXIT
|
||||
|
||||
zfs create "${POOL_NAME}"/src
|
||||
zfs snapshot "${POOL_NAME}"/src@duplicate
|
||||
|
||||
# initial replication
|
||||
../../../syncoid -r --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
# recreate snapshot with the same name on src
|
||||
zfs destroy "${POOL_NAME}"/src@duplicate
|
||||
zfs snapshot "${POOL_NAME}"/src@duplicate
|
||||
sleep 1
|
||||
../../../syncoid -r --force-delete --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst || exit 1
|
||||
|
||||
# verify
|
||||
output1=$(zfs list -t snapshot -r -H -o guid,name "${POOL_NAME}"/src | sed 's/@syncoid_.*$'/@syncoid_/)
|
||||
checksum1=$(echo "${output1}" | shasum -a 256)
|
||||
|
||||
output2=$(zfs list -t snapshot -r -H -o guid,name "${POOL_NAME}"/dst | sed 's/@syncoid_.*$'/@syncoid_/ | sed 's/dst/src/')
|
||||
checksum2=$(echo "${output2}" | shasum -a 256)
|
||||
|
||||
if [ "${checksum1}" != "${checksum2}" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
Loading…
Reference in New Issue