From 429da3a13158f61cb0d9fddb97cd9fc791675640 Mon Sep 17 00:00:00 2001
From: Andrew Bobulsky <andrew.bobulsky@buyerquest.com>
Date: Mon, 26 Mar 2018 03:44:00 -0400
Subject: [PATCH 01/62] Add comprehensive installation instructions

Converts the old INSTALL file to a markdown-formatted INSTALL.md file
that contains comprehensive installation instructions for supported
operating systems, and an outline that can be used on others where
Sanoid/Syncoid ought to work.

The Installation section is complete for CentOS/RHEL, and has been
"dressed up" for FreeBSD/Ubuntu. Those ones still need to be completed.
I don't cover the topic of running Syncoid periodically or how to do it
because I don't know. It's probably out of scope for the Installation
guide anyway.

The Configuration sections is intended to walk through some common
setups for sanoid.conf. I suggest just cleaning up the existing conf
file examples and putting them into that section.
---
 INSTALL    |  31 -------------
 INSTALL.md | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+), 31 deletions(-)
 delete mode 100644 INSTALL
 create mode 100644 INSTALL.md

diff --git a/INSTALL b/INSTALL
deleted file mode 100644
index 1492546..0000000
--- a/INSTALL
+++ /dev/null
@@ -1,31 +0,0 @@
-SYNCOID
--------
-Syncoid depends on ssh, pv, gzip, lzop, and mbuffer.  It can run with reduced
-functionality in the absence of any or all of the above. SSH is only required
-for remote synchronization.  On newer FreeBSD and Ubuntu Xenial
-chacha20-poly1305@openssh.com, on other distributions arcfour crypto is the
-default for SSH transport since v1.4.6. Syncoid runs will fail if one of them
-is not available on either end of the transport.
-
-On Ubuntu: apt install pv lzop mbuffer
-On CentOS: yum install lzo pv mbuffer lzop
-On FreeBSD: pkg install pv lzop
-
-FreeBSD notes: FreeBSD may place pv and lzop in somewhere other than
-               /usr/bin ; syncoid currently does not check path.
-
-               Simplest path workaround is symlinks, eg:
-               root@bsd:~# ln -s /usr/local/bin/lzop /usr/bin/lzop
-               or similar, as appropriate, to create links in /usr/bin
-               to wherever the utilities actually are on your system.
-
-
-SANOID
-------
-Sanoid depends on the Perl module Config::IniFiles and will not operate
-without it.  Config::IniFiles may be installed from CPAN, though the project
-strongly recommends using your distribution's repositories instead.
-
-On Ubuntu: apt install libconfig-inifiles-perl
-On CentOS: yum install perl-Config-IniFiles
-On FreeBSD: pkg install p5-Config-Inifiles
diff --git a/INSTALL.md b/INSTALL.md
new file mode 100644
index 0000000..491afd2
--- /dev/null
+++ b/INSTALL.md
@@ -0,0 +1,131 @@
+# Installation
+
+**Sanoid** and **Syncoid** are complementary but separate pieces of software. To install and configure them, follow the guide below for your operating system. Everything in `code blocks` should be copy-pasteable. If your OS isn't listed, a set of general instructions is at the end of the list and you can perform the process manually.
+
+<!-- TOC depthFrom:1 depthTo:6 withLinks:1 updateOnSave:0 orderedList:0 -->
+
+- [Installation](#installation)
+	- [Ubuntu](#ubuntu)
+	- [CentOS](#centos)
+	- [FreeBSD](#freebsd)
+	- [Other OSes](#other-oses)
+- [Configuration](#configuration)
+	- [sanoid.conf](#sanoidconf)
+
+<!-- /TOC -->
+
+
+## Ubuntu
+
+Install prerequisite software:
+
+```bash
+apt install libconfig-inifiles-perl pv lzop mbuffer
+```
+
+## CentOS
+
+Install prerequisite software:
+
+```bash
+# Install and enable epel if we don't already have it, and git too
+sudo yum install -y epel-release git
+# Install the packages that Sanoid depends on:
+sudo yum install -y perl-Config-IniFiles perl-Data-Dumper lzop mbuffer mhash pv
+```
+
+Clone this repo, then put the executables and config files into the appropriate directories:
+
+```bash
+# Download the repo as root to avoid changing permissions later
+sudo git clone https://github.com/jimsalterjrs/sanoid.git
+cd sanoid
+# Install the executables
+sudo cp sanoid syncoid findoid sleepymutex /usr/local/sbin
+# Create the config directory
+sudo mkdir /etc/sanoid
+# Install default config
+sudo cp sanoid.defaults.conf /etc/sanoid
+# Create a blank config file
+sudo touch /etc/sanoid/sanoid.conf
+# Place the sample config in the conf directory for reference
+sudo cp sanoid.conf /etc/sanoid/sanoid.example.conf
+```
+
+Create a systemd service:
+
+```bash
+cat << "EOF" | sudo tee /etc/systemd/system/sanoid.service
+[Unit]
+Description=Snapshot ZFS Pool
+Requires=zfs.target
+After=zfs.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/sbin/sanoid --cron
+EOF
+```
+
+And a systemd timer that will execute **Sanoid** once per minute:
+
+```bash
+cat << "EOF" | sudo tee /etc/systemd/system/sanoid.timer
+[Unit]
+Description=Run Sanoid Every Minute
+Requires=sanoid.service
+
+[Timer]
+OnCalendar=*:0/1
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+EOF
+```
+
+Reload systemd and start our timer:
+```bash
+# Tell systemd about our new service definitions
+sudo systemctl daemon-reload
+# Enable and start the Sanoid timer
+sudo systemctl enable sanoid.timer
+sudo systemctl start sanoid.timer
+```
+
+Now, proceed to configure [**Sanoid**](#configuration)
+
+## FreeBSD
+
+Install prerequisite software:
+
+```bash
+pkg install p5-Config-Inifiles pv lzop
+```
+
+**Additional notes:**
+
+*   FreeBSD may place pv and lzop in somewhere other than /usr/bin — syncoid currently does not check path.
+
+*   Simplest path workaround is symlinks, eg `ln -s /usr/local/bin/lzop /usr/bin/lzop` or similar, as appropriate to create links in **/usr/bin** to wherever the utilities actually are on your system.
+
+## Other OSes
+
+**Sanoid** depends on the Perl module Config::IniFiles and will not operate without it. Config::IniFiles may be installed from CPAN, though the project strongly recommends using your distribution's repositories instead.
+
+**Syncoid** depends on ssh, pv, gzip, lzop, and mbuffer. It can run with reduced functionality in the absence of any or all of the above. SSH is only required for remote synchronization. On newer FreeBSD and Ubuntu Xenial chacha20-poly1305@openssh.com, on other distributions arcfour crypto is the default for SSH transport since v1.4.6. Syncoid runs will fail if one of them is not available on either end of the transport.
+
+### General outline for installation
+
+1.  Install prerequisites: Perl module Config::IniFiles, ssh, pv, gzip, lzop, and mbuffer
+2.  Download the **Sanoid** repo
+3.  Create the config directory `/etc/sanoid` and put `sanoid.defaults.conf` in there, and create `sanoid.conf` in it too
+4.  Create a cron job or a systemd timer that runs `sanoid --cron` once per minute
+
+# Configuration
+
+**Sanoid** won't do anything useful unless you tell it how to handle your ZFS datasets in `/etc/sanoid/sanoid.conf`. **Syncoid** is a command line utility that doesn't require any configuration, with all of its switches set at runtime.
+
+## sanoid.conf
+
+Instructions on how to set up `sanoid.conf`.  Maybe just copy/paste the example `sanoid.conf` file in here but clean it up a little bit.

From 0f669a304a4416761a2aa239f54c8372c61fb7b8 Mon Sep 17 00:00:00 2001
From: Andrew Bobulsky <andrew.bobulsky@buyerquest.com>
Date: Mon, 26 Mar 2018 03:54:28 -0400
Subject: [PATCH 02/62] Cleaned up Configuration section to match the rest of
 the document

---
 INSTALL.md | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/INSTALL.md b/INSTALL.md
index 491afd2..94a9f2e 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -10,7 +10,7 @@
 	- [FreeBSD](#freebsd)
 	- [Other OSes](#other-oses)
 - [Configuration](#configuration)
-	- [sanoid.conf](#sanoidconf)
+	- [Sanoid](#sanoid)
 
 <!-- /TOC -->
 
@@ -124,8 +124,10 @@ pkg install p5-Config-Inifiles pv lzop
 
 # Configuration
 
-**Sanoid** won't do anything useful unless you tell it how to handle your ZFS datasets in `/etc/sanoid/sanoid.conf`. **Syncoid** is a command line utility that doesn't require any configuration, with all of its switches set at runtime.
+**Sanoid** won't do anything useful unless you tell it how to handle your ZFS datasets in `/etc/sanoid/sanoid.conf`.
 
-## sanoid.conf
+**Syncoid** is a command line utility that doesn't require any configuration, with all of its switches set at runtime.
+
+## Sanoid
 
 Instructions on how to set up `sanoid.conf`.  Maybe just copy/paste the example `sanoid.conf` file in here but clean it up a little bit.

From 9b2c2f29dc126c8629d2d2612331b07b2e71c30d Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 12 Sep 2018 10:00:09 +0200
Subject: [PATCH 03/62] don't use hardcoded paths

---
 syncoid | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git a/syncoid b/syncoid
index 30aef49..257e652 100755
--- a/syncoid
+++ b/syncoid
@@ -48,17 +48,20 @@ my $debug = $args{'debug'};
 my $quiet = $args{'quiet'};
 my $resume = !$args{'no-resume'};
 
-my $zfscmd = '/sbin/zfs';
-my $sshcmd = '/usr/bin/ssh';
-my $pscmd = '/bin/ps';
+# for compatibility reasons, older versions used hardcoded command paths
+$ENV{'PATH'} = $ENV{'PATH'} . ":/bin:/usr/bin:/sbin";
 
-my $pvcmd = '/usr/bin/pv';
-my $mbuffercmd = '/usr/bin/mbuffer';
-my $sudocmd = '/usr/bin/sudo';
+my $zfscmd = 'zfs';
+my $sshcmd = 'ssh';
+my $pscmd = 'ps';
+
+my $pvcmd = 'pv';
+my $mbuffercmd = 'mbuffer';
+my $sudocmd = 'sudo';
 my $mbufferoptions = '-q -s 128k -m 16M 2>/dev/null';
-# currently using ls to check for file existence because we aren't depending on perl
+# currently using which to check for command existence because we aren't depending on perl
 # being present on remote machines.
-my $lscmd = '/bin/ls';
+my $whichcmd = 'which';
 
 if (length $args{'sshcipher'}) {
 	$args{'sshcipher'} = "-c $args{'sshcipher'}";
@@ -564,11 +567,11 @@ sub checkcommands {
 		if ($debug) { print "DEBUG: compression forced off from command line arguments.\n"; }
 	} else {
 		if ($debug) { print "DEBUG: checking availability of $compressargs{'rawcmd'} on source...\n"; }
-		$avail{'sourcecompress'} = `$sourcessh $lscmd $compressargs{'rawcmd'} 2>/dev/null`;
+		$avail{'sourcecompress'} = `$sourcessh $whichcmd $compressargs{'rawcmd'} 2>/dev/null`;
 		if ($debug) { print "DEBUG: checking availability of $compressargs{'rawcmd'} on target...\n"; }
-		$avail{'targetcompress'} = `$targetssh $lscmd $compressargs{'rawcmd'} 2>/dev/null`;
+		$avail{'targetcompress'} = `$targetssh $whichcmd $compressargs{'rawcmd'} 2>/dev/null`;
 		if ($debug) { print "DEBUG: checking availability of $compressargs{'rawcmd'} on local machine...\n"; }
-		$avail{'localcompress'} = `$lscmd $compressargs{'rawcmd'} 2>/dev/null`;
+		$avail{'localcompress'} = `$whichcmd $compressargs{'rawcmd'} 2>/dev/null`;
 	}
 
 	my ($s,$t);
@@ -620,7 +623,7 @@ sub checkcommands {
 	}
 
 	if ($debug) { print "DEBUG: checking availability of $mbuffercmd on source...\n"; }
-	$avail{'sourcembuffer'} = `$sourcessh $lscmd $mbuffercmd 2>/dev/null`;
+	$avail{'sourcembuffer'} = `$sourcessh $whichcmd $mbuffercmd 2>/dev/null`;
 	if ($avail{'sourcembuffer'} eq '') {
 		if (!$quiet) { print "WARN: $mbuffercmd not available on source $s - sync will continue without source buffering.\n"; }
 		$avail{'sourcembuffer'} = 0;
@@ -629,7 +632,7 @@ sub checkcommands {
 	}
 
 	if ($debug) { print "DEBUG: checking availability of $mbuffercmd on target...\n"; }
-	$avail{'targetmbuffer'} = `$targetssh $lscmd $mbuffercmd 2>/dev/null`;
+	$avail{'targetmbuffer'} = `$targetssh $whichcmd $mbuffercmd 2>/dev/null`;
 	if ($avail{'targetmbuffer'} eq '') {
 		if (!$quiet) { print "WARN: $mbuffercmd not available on target $t - sync will continue without target buffering.\n"; }
 		$avail{'targetmbuffer'} = 0;
@@ -640,7 +643,7 @@ sub checkcommands {
 	# if we're doing remote source AND remote target, check for local mbuffer as well
 	if ($sourcehost ne '' && $targethost ne '') {
 		if ($debug) { print "DEBUG: checking availability of $mbuffercmd on local machine...\n"; }
-		$avail{'localmbuffer'} = `$lscmd $mbuffercmd 2>/dev/null`;
+		$avail{'localmbuffer'} = `$whichcmd $mbuffercmd 2>/dev/null`;
 		if ($avail{'localmbuffer'} eq '') {
 			$avail{'localmbuffer'} = 0;
 			if (!$quiet) { print "WARN: $mbuffercmd not available on local machine - sync will continue without local buffering.\n"; }
@@ -648,7 +651,7 @@ sub checkcommands {
 	}
 
 	if ($debug) { print "DEBUG: checking availability of $pvcmd on local machine...\n"; }
-	$avail{'localpv'} = `$lscmd $pvcmd 2>/dev/null`;
+	$avail{'localpv'} = `$whichcmd $pvcmd 2>/dev/null`;
 	if ($avail{'localpv'} eq '') {
 		if (!$quiet) { print "WARN: $pvcmd not available on local machine - sync will continue without progress bar.\n"; }
 		$avail{'localpv'} = 0;

From be50481374be1265ddbaee0decd65a6b39634366 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Fri, 7 Dec 2018 17:03:49 +0100
Subject: [PATCH 04/62] use more portable/POSIX compatible ways to check for
 command existence

---
 syncoid | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/syncoid b/syncoid
index e618eb8..cb57c28 100755
--- a/syncoid
+++ b/syncoid
@@ -62,7 +62,7 @@ my $sudocmd = 'sudo';
 my $mbufferoptions = '-q -s 128k -m 16M 2>/dev/null';
 # currently using which to check for command existence because we aren't depending on perl
 # being present on remote machines.
-my $whichcmd = 'which';
+my $checkcmd = 'command -v';
 
 if (length $args{'sshcipher'}) {
 	$args{'sshcipher'} = "-c $args{'sshcipher'}";
@@ -769,11 +769,11 @@ sub checkcommands {
 		if ($debug) { print "DEBUG: compression forced off from command line arguments.\n"; }
 	} else {
 		if ($debug) { print "DEBUG: checking availability of $compressargs{'rawcmd'} on source...\n"; }
-		$avail{'sourcecompress'} = `$sourcessh $whichcmd $compressargs{'rawcmd'} 2>/dev/null`;
+		$avail{'sourcecompress'} = `$sourcessh $checkcmd $compressargs{'rawcmd'} 2>/dev/null`;
 		if ($debug) { print "DEBUG: checking availability of $compressargs{'rawcmd'} on target...\n"; }
-		$avail{'targetcompress'} = `$targetssh $whichcmd $compressargs{'rawcmd'} 2>/dev/null`;
+		$avail{'targetcompress'} = `$targetssh $checkcmd $compressargs{'rawcmd'} 2>/dev/null`;
 		if ($debug) { print "DEBUG: checking availability of $compressargs{'rawcmd'} on local machine...\n"; }
-		$avail{'localcompress'} = `$whichcmd $compressargs{'rawcmd'} 2>/dev/null`;
+		$avail{'localcompress'} = `$checkcmd $compressargs{'rawcmd'} 2>/dev/null`;
 	}
 
 	my ($s,$t);
@@ -825,7 +825,7 @@ sub checkcommands {
 	}
 
 	if ($debug) { print "DEBUG: checking availability of $mbuffercmd on source...\n"; }
-	$avail{'sourcembuffer'} = `$sourcessh $whichcmd $mbuffercmd 2>/dev/null`;
+	$avail{'sourcembuffer'} = `$sourcessh $checkcmd $mbuffercmd 2>/dev/null`;
 	if ($avail{'sourcembuffer'} eq '') {
 		if (!$quiet) { print "WARN: $mbuffercmd not available on source $s - sync will continue without source buffering.\n"; }
 		$avail{'sourcembuffer'} = 0;
@@ -834,7 +834,7 @@ sub checkcommands {
 	}
 
 	if ($debug) { print "DEBUG: checking availability of $mbuffercmd on target...\n"; }
-	$avail{'targetmbuffer'} = `$targetssh $whichcmd $mbuffercmd 2>/dev/null`;
+	$avail{'targetmbuffer'} = `$targetssh $checkcmd $mbuffercmd 2>/dev/null`;
 	if ($avail{'targetmbuffer'} eq '') {
 		if (!$quiet) { print "WARN: $mbuffercmd not available on target $t - sync will continue without target buffering.\n"; }
 		$avail{'targetmbuffer'} = 0;
@@ -845,7 +845,7 @@ sub checkcommands {
 	# if we're doing remote source AND remote target, check for local mbuffer as well
 	if ($sourcehost ne '' && $targethost ne '') {
 		if ($debug) { print "DEBUG: checking availability of $mbuffercmd on local machine...\n"; }
-		$avail{'localmbuffer'} = `$whichcmd $mbuffercmd 2>/dev/null`;
+		$avail{'localmbuffer'} = `$checkcmd $mbuffercmd 2>/dev/null`;
 		if ($avail{'localmbuffer'} eq '') {
 			$avail{'localmbuffer'} = 0;
 			if (!$quiet) { print "WARN: $mbuffercmd not available on local machine - sync will continue without local buffering.\n"; }
@@ -853,7 +853,7 @@ sub checkcommands {
 	}
 
 	if ($debug) { print "DEBUG: checking availability of $pvcmd on local machine...\n"; }
-	$avail{'localpv'} = `$whichcmd $pvcmd 2>/dev/null`;
+	$avail{'localpv'} = `$checkcmd $pvcmd 2>/dev/null`;
 	if ($avail{'localpv'} eq '') {
 		if (!$quiet) { print "WARN: $pvcmd not available on local machine - sync will continue without progress bar.\n"; }
 		$avail{'localpv'} = 0;

From 941a770c12ab318ebc366dd24b7a6cd5a504d15a Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Fri, 7 Dec 2018 17:05:53 +0100
Subject: [PATCH 05/62] updated comment regarding command existence check

---
 syncoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index cb57c28..9530b00 100755
--- a/syncoid
+++ b/syncoid
@@ -60,7 +60,7 @@ my $pvcmd = 'pv';
 my $mbuffercmd = 'mbuffer';
 my $sudocmd = 'sudo';
 my $mbufferoptions = '-q -s 128k -m 16M 2>/dev/null';
-# currently using which to check for command existence because we aren't depending on perl
+# currently using POSIX compatible command to check for program existence because we aren't depending on perl
 # being present on remote machines.
 my $checkcmd = 'command -v';
 

From 89f1d4e9a69fe088f391034e72e2c55120986783 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Sat, 8 Dec 2018 14:13:54 +0100
Subject: [PATCH 06/62] run compress commands bare too

---
 syncoid | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)

diff --git a/syncoid b/syncoid
index 9530b00..495f712 100755
--- a/syncoid
+++ b/syncoid
@@ -673,51 +673,51 @@ sub compressargset {
 			decomargs   => '',
 		},
 		'gzip' => {
-			rawcmd      => '/bin/gzip',
+			rawcmd      => 'gzip',
 			args        => '-3',
-			decomrawcmd => '/bin/zcat',
+			decomrawcmd => 'zcat',
 			decomargs   => '',
 		},
 		'pigz-fast' => {
-			rawcmd      => '/usr/bin/pigz',
+			rawcmd      => 'pigz',
 			args        => '-3',
-			decomrawcmd => '/usr/bin/pigz',
+			decomrawcmd => 'pigz',
 			decomargs   => '-dc',
 		},
 		'pigz-slow' => {
-			rawcmd      => '/usr/bin/pigz',
+			rawcmd      => 'pigz',
 			args        => '-9',
-			decomrawcmd => '/usr/bin/pigz',
+			decomrawcmd => 'pigz',
 			decomargs   => '-dc',
 		},
 		'zstd-fast' => {
-			rawcmd      => '/usr/bin/zstd',
+			rawcmd      => 'zstd',
 			args        => '-3',
-			decomrawcmd => '/usr/bin/zstd',
+			decomrawcmd => 'zstd',
 			decomargs   => '-dc',
 		},
 		'zstd-slow' => {
-			rawcmd      => '/usr/bin/zstd',
+			rawcmd      => 'zstd',
 			args        => '-19',
-			decomrawcmd => '/usr/bin/zstd',
+			decomrawcmd => 'zstd',
 			decomargs   => '-dc',
 		},
 		'xz' => {
-			rawcmd      => '/usr/bin/xz',
+			rawcmd      => 'xz',
 			args        => '',
-			decomrawcmd => '/usr/bin/xz',
+			decomrawcmd => 'xz',
 			decomargs   => '-d',
 		},
 		'lzo' => {
-			rawcmd      => '/usr/bin/lzop',
+			rawcmd      => 'lzop',
 			args        => '',
-			decomrawcmd => '/usr/bin/lzop',
+			decomrawcmd => 'lzop',
 			decomargs   => '-dfc',
 		},
 		'lz4' => {
-			rawcmd      => '/usr/bin/lz4',
+			rawcmd      => 'lz4',
 			args        => '',
-			decomrawcmd => '/usr/bin/lz4',
+			decomrawcmd => 'lz4',
 			decomargs   => '-dc',
 		},
 	);

From ac80a753157dc26005c976675336f1e3fe3e1ad9 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 19 Dec 2018 00:36:10 +0100
Subject: [PATCH 07/62] made time units case insensitive and removed monthlies

---
 sanoid               | 15 ++++++---------
 sanoid.defaults.conf |  7 ++++---
 2 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/sanoid b/sanoid
index 3aa57a3..ce8207c 100755
--- a/sanoid
+++ b/sanoid
@@ -1519,25 +1519,22 @@ sub convertTimePeriod {
 	my $value=shift;
 	my $period=shift;
 
-	if ($value =~ /^\d+Y$/) {
+	if ($value =~ /^\d+[yY]$/) {
 		$period = 60*60*24*31*365;
 		chop $value;
-	} elsif ($value =~ /^\d+M$/) {
-		$period = 60*60*24*31;
-		chop $value;
-	} elsif ($value =~ /^\d+W$/) {
+	} elsif ($value =~ /^\d+[wW]$/) {
 		$period = 60*60*24*7;
 		chop $value;
-	} elsif ($value =~ /^\d+D$/) {
+	} elsif ($value =~ /^\d+[dD]$/) {
 		$period = 60*60*24;
 		chop $value;
-	} elsif ($value =~ /^\d+h$/) {
+	} elsif ($value =~ /^\d+[hH]$/) {
 		$period = 60*60;
 		chop $value;
-	} elsif ($value =~ /^\d+m$/) {
+	} elsif ($value =~ /^\d+[mM]$/) {
 		$period = 60;
 		chop $value;
-	} elsif ($value =~ /^\d+s$/) {
+	} elsif ($value =~ /^\d+[sS]$/) {
 		$period = 1;
 		chop $value;
 	} elsif ($value =~ /^\d+$/) {
diff --git a/sanoid.defaults.conf b/sanoid.defaults.conf
index 6649c2e..a9ca382 100644
--- a/sanoid.defaults.conf
+++ b/sanoid.defaults.conf
@@ -84,7 +84,8 @@ yearly_min = 0
 # example hourly_warn = 90 means issue WARNING if most recent hourly snapshot is not less than 90 minutes old,
 # daily_crit = 36 means issue CRITICAL if most recent daily snapshot is not less than 36 hours old,
 # monthly_warn = 5 means issue WARNING if most recent monthly snapshot is not less than 5 weeks old... etc.
-# the following time suffixes can also be used: Y = years, M = months, W = weeks, D = days, h = hours, m = minutes, s = seconds
+# the following time case insensitive suffixes can also be used:
+# y = years, w = weeks, d = days, h = hours, m = minutes, s = seconds
 #
 # monitor_dont_warn = yes will cause the monitoring service to report warnings as text, but with status OK.
 # monitor_dont_crit = yes will cause the monitoring service to report criticals as text, but with status OK.
@@ -101,8 +102,8 @@ daily_warn = 28h
 daily_crit = 32h
 weekly_warn = 0
 weekly_crit = 0
-monthly_warn = 32D
-monthly_crit = 40D
+monthly_warn = 32d
+monthly_crit = 40d
 yearly_warn = 0
 yearly_crit = 0
 

From 8d88c4743e9e0a5bcb66a2530f9c2b1950f3fc65 Mon Sep 17 00:00:00 2001
From: Ben Yanke <ben@benyanke.com>
Date: Wed, 3 Oct 2018 16:20:58 -0500
Subject: [PATCH 08/62] improved documentation on --no-command-checks

---
 README.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index b833dec..99d8db1 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,8 @@
 
 <img src="http://openoid.net/gplv3-127x51.png" width=127 height=51 align="right">Sanoid is a policy-driven snapshot management tool for ZFS filesystems.  When combined with the Linux KVM hypervisor, you can use it to make your systems <a href="http://openoid.net/transcend" target="_blank">functionally immortal</a>.  
 
-<p align="center"><a href="https://youtu.be/ZgowLNBsu00" target="_blank"><img src="http://www.openoid.net/sanoid_video_launcher.png" alt="sanoid rollback demo" title="sanoid rollback demo"></a><br clear="all"><sup>(Real time demo: rolling back a full-scale cryptomalware infection in seconds!)</sup></p>
+<p align="center"><a href="https://youtu.be/ZgowLNBsu00" target="_blank"><img src="http://www.
+	openoid.net/sanoid_video_launcher.png" alt="sanoid rollback demo" title="sanoid rollback demo"></a><br clear="all"><sup>(Real time demo: rolling back a full-scale cryptomalware infection in seconds!)</sup></p>
 
 More prosaically, you can use Sanoid to create, automatically thin, and monitor snapshots and pool health from a single eminently human-readable TOML config file at /etc/sanoid/sanoid.conf.  (Sanoid also requires a "defaults" file located at /etc/sanoid/sanoid.defaults.conf, which is not user-editable.)  A typical Sanoid system would have a single cron job:
 ```
@@ -180,7 +181,7 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
 
 + --no-command-checks
 
-	Do not check the existance of commands before attempting the transfer. It assumes all programs are available. This should never be used.
+	Does not check the existence of commands before attempting the transfer, providing administrators a way to run the tool with minimal overhead and maximum speed, at risk of potentially failed replication, or other possible edge cases. It assumes all programs are available, and should not be used in most situations. This is an not an officially supported run mode.
 
 + --no-stream
 

From 1605a60c624b586243afa6a5d803b746a83b3b17 Mon Sep 17 00:00:00 2001
From: Sam Allred <sam.allred@usurf.usu.edu>
Date: Tue, 19 Dec 2017 15:47:40 -0700
Subject: [PATCH 09/62] CLIMATE-1151: added config option to use zfs recursion

---
 sanoid | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/sanoid b/sanoid
index ce8207c..30363c5 100755
--- a/sanoid
+++ b/sanoid
@@ -502,7 +502,14 @@ sub take_snapshots {
 					# update to most current possible datestamp
 					%datestamp = get_date();
 					# print "we should have had a $type snapshot of $path $maxage seconds ago; most recent is $newestage seconds old.\n";
-					push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type");
+
+					# use zfs recursion if specified in config
+					if ($config{$section}{'zfs_recursion'}) {
+						push(@newsnaps, "-r $path\@autosnap_$datestamp{'sortable'}_$type");
+
+					}else{
+						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}_$type");
+					}
 				}
 			}
 		}
@@ -530,7 +537,7 @@ sub take_snapshots {
 			}
 			if ($args{'verbose'}) { print "taking snapshot $snap\n"; }
 			if (!$args{'readonly'}) {
-				system($zfs, "snapshot", "$snap") == 0
+				system("$zfs snapshot $snap") == 0
 					or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
 			}
 			if ($config{$dataset}{'post_snapshot_script'} and !$args{'readonly'}) {
@@ -846,7 +853,10 @@ sub init {
 		my $recursive = $ini{$section}{'recursive'} && grep( /^$ini{$section}{'recursive'}$/, @istrue );
 		my $skipChildren = $ini{$section}{'skip_children'} && grep( /^$ini{$section}{'skip_children'}$/, @istrue );
 		my @datasets;
-		if ($recursive || $skipChildren) {
+
+		if($ini{$section}{'recursive'} =~ /zfs/i) {
+			$config{$section}{'zfs_recursion'} = 1;
+		}elsif ($ini{$section}{'recursive'}) {
 			@datasets = getchilddatasets($config{$section}{'path'});
 			DATASETS: foreach my $dataset(@datasets) {
 				chomp $dataset;

From 4daafca9cf5386ac33bb0cec6e29c2c0e9ab2c38 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Thu, 20 Dec 2018 01:33:35 +0100
Subject: [PATCH 10/62] prevent problems with shell expansion and codestyle

---
 sanoid | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)

diff --git a/sanoid b/sanoid
index 30363c5..9fcbb88 100755
--- a/sanoid
+++ b/sanoid
@@ -503,12 +503,11 @@ sub take_snapshots {
 					%datestamp = get_date();
 					# print "we should have had a $type snapshot of $path $maxage seconds ago; most recent is $newestage seconds old.\n";
 
-					# use zfs recursion if specified in config
+					# use zfs (atomic) recursion if specified in config
 					if ($config{$section}{'zfs_recursion'}) {
-						push(@newsnaps, "-r $path\@autosnap_$datestamp{'sortable'}_$type");
-
-					}else{
-						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}_$type");
+						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type\@");
+					} else {
+						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type");
 					}
 				}
 			}
@@ -517,8 +516,14 @@ sub take_snapshots {
 
 	if ( (scalar(@newsnaps)) > 0) {
 		foreach my $snap ( @newsnaps ) {
-			my $dataset = (split '@', $snap)[0];
-			my $snapname = (split '@', $snap)[1];
+			my @split = split '@', $snap, -1;
+			my $recursiveFlag = 0;
+			if (scalar(@split) == 3) {
+				$recursiveFlag = 1;
+				chop $snap;
+			}
+			my $dataset = $split[0];
+			my $snapname = $split[1];
 			my $presnapshotfailure = 0;
 			if ($config{$dataset}{'pre_snapshot_script'} and !$args{'readonly'}) {
 				$ENV{'SANOID_TARGET'} = $dataset;
@@ -537,8 +542,13 @@ sub take_snapshots {
 			}
 			if ($args{'verbose'}) { print "taking snapshot $snap\n"; }
 			if (!$args{'readonly'}) {
-				system("$zfs snapshot $snap") == 0
-					or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
+				if ($recursiveFlag) {
+					system($zfs, "snapshot", "-r", "$snap") == 0
+						or warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
+				} else {
+					system($zfs, "snapshot", "$snap") == 0
+						or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
+				}
 			}
 			if ($config{$dataset}{'post_snapshot_script'} and !$args{'readonly'}) {
 				if (!$presnapshotfailure or $config{$dataset}{'force_post_snapshot_script'}) {
@@ -853,10 +863,9 @@ sub init {
 		my $recursive = $ini{$section}{'recursive'} && grep( /^$ini{$section}{'recursive'}$/, @istrue );
 		my $skipChildren = $ini{$section}{'skip_children'} && grep( /^$ini{$section}{'skip_children'}$/, @istrue );
 		my @datasets;
-
-		if($ini{$section}{'recursive'} =~ /zfs/i) {
+		if ($ini{$section}{'recursive'} =~ /zfs/i) {
 			$config{$section}{'zfs_recursion'} = 1;
-		}elsif ($ini{$section}{'recursive'}) {
+		} elsif ($recursive || $skipChildren) {
 			@datasets = getchilddatasets($config{$section}{'path'});
 			DATASETS: foreach my $dataset(@datasets) {
 				chomp $dataset;

From 7cf64be7029a985bd8dbde04688bb947861a15ef Mon Sep 17 00:00:00 2001
From: matveevandrey <matveev.as@gmail.com>
Date: Thu, 20 Dec 2018 22:53:57 +0300
Subject: [PATCH 11/62] pre/post snapshot script calls should be also dumped in
 --readonly mode

---
 sanoid | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/sanoid b/sanoid
index ce8207c..03aefc5 100755
--- a/sanoid
+++ b/sanoid
@@ -513,11 +513,15 @@ sub take_snapshots {
 			my $dataset = (split '@', $snap)[0];
 			my $snapname = (split '@', $snap)[1];
 			my $presnapshotfailure = 0;
-			if ($config{$dataset}{'pre_snapshot_script'} and !$args{'readonly'}) {
+			my $ret = 0;
+			if ($config{$dataset}{'pre_snapshot_script'}) {
 				$ENV{'SANOID_TARGET'} = $dataset;
 				$ENV{'SANOID_SNAPNAME'} = $snapname;
 				if ($args{'verbose'}) { print "executing pre_snapshot_script '".$config{$dataset}{'pre_snapshot_script'}."' on dataset '$dataset'\n"; }
-				my $ret = runscript('pre_snapshot_script',$dataset);
+
+				if (!$args{'readonly'}) {
+					$ret = runscript('pre_snapshot_script',$dataset);
+				}
 
 				delete $ENV{'SANOID_TARGET'};
 				delete $ENV{'SANOID_SNAPNAME'};
@@ -533,12 +537,15 @@ sub take_snapshots {
 				system($zfs, "snapshot", "$snap") == 0
 					or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
 			}
-			if ($config{$dataset}{'post_snapshot_script'} and !$args{'readonly'}) {
+			if ($config{$dataset}{'post_snapshot_script'}) {
 				if (!$presnapshotfailure or $config{$dataset}{'force_post_snapshot_script'}) {
 					$ENV{'SANOID_TARGET'} = $dataset;
 					$ENV{'SANOID_SNAPNAME'} = $snapname;
 					if ($args{'verbose'}) { print "executing post_snapshot_script '".$config{$dataset}{'post_snapshot_script'}."' on dataset '$dataset'\n"; }
-					runscript('post_snapshot_script',$dataset);
+	
+					if (!$args{'readonly'}) {
+						runscript('post_snapshot_script',$dataset);
+					}
 
 					delete $ENV{'SANOID_TARGET'};
 					delete $ENV{'SANOID_SNAPNAME'};

From f8e0e006ab92344b62a640988c668631c243107b Mon Sep 17 00:00:00 2001
From: matveevandrey <matveev.as@gmail.com>
Date: Thu, 20 Dec 2018 23:52:40 +0300
Subject: [PATCH 12/62] preserve taking snapshots order always from yearly to
 frequently

very simple patch (not best way in terms of programming but minimal change requirements achieved)
---
 sanoid | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/sanoid b/sanoid
index 03aefc5..d30dcba 100755
--- a/sanoid
+++ b/sanoid
@@ -377,9 +377,12 @@ sub take_snapshots {
 		if ($config{$section}{'process_children_only'}) { next; }
 
 		my $path = $config{$section}{'path'};
+                my @types = ('yearly','monthly','weekly','daily','hourly','frequently');
 
-		foreach my $type (keys %{ $config{$section} }){
-			unless ($type =~ /ly$/) { next; }
+                foreach my $type (@types) {
+
+		foreach my $_type (keys %{ $config{$section} }){
+			unless ($type eq $_type) { next; }
 			if ($config{$section}{$type} > 0) {
 
 				my $newestage; # in seconds
@@ -506,6 +509,7 @@ sub take_snapshots {
 				}
 			}
 		}
+		}
 	}
 
 	if ( (scalar(@newsnaps)) > 0) {

From c35c953e548a922e57edfaa046313a5fafe271a6 Mon Sep 17 00:00:00 2001
From: matveevandrey <matveev.as@gmail.com>
Date: Fri, 21 Dec 2018 12:56:10 +0300
Subject: [PATCH 13/62] remove redundant loop

---
 sanoid | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/sanoid b/sanoid
index d30dcba..82cfce9 100755
--- a/sanoid
+++ b/sanoid
@@ -380,9 +380,6 @@ sub take_snapshots {
                 my @types = ('yearly','monthly','weekly','daily','hourly','frequently');
 
                 foreach my $type (@types) {
-
-		foreach my $_type (keys %{ $config{$section} }){
-			unless ($type eq $_type) { next; }
 			if ($config{$section}{$type} > 0) {
 
 				my $newestage; # in seconds
@@ -509,7 +506,6 @@ sub take_snapshots {
 				}
 			}
 		}
-		}
 	}
 
 	if ( (scalar(@newsnaps)) > 0) {

From ca76f4268b27f8330f91b1e1421853765c326047 Mon Sep 17 00:00:00 2001
From: root <TerraTech@users.noreply.github.com>
Date: Sun, 23 Dec 2018 09:18:49 -0500
Subject: [PATCH 14/62] syncoid: add '--mbuffer-size' option

---
 syncoid | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/syncoid b/syncoid
index d3e678f..1085e36 100755
--- a/syncoid
+++ b/syncoid
@@ -14,13 +14,16 @@ use Pod::Usage;
 use Time::Local;
 use Sys::Hostname;
 
+my $mbuffer_size = "16M";
+
 # Blank defaults to use ssh client's default
 # TODO: Merge into a single "sshflags" option?
 my %args = ('sshkey' => '', 'sshport' => '', 'sshcipher' => '', 'sshoption' => [], 'target-bwlimit' => '', 'source-bwlimit' => '');
 GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsnaps", "recursive|r",
                    "source-bwlimit=s", "target-bwlimit=s", "sshkey=s", "sshport=i", "sshcipher|c=s", "sshoption|o=s@",
                    "debug", "quiet", "no-stream", "no-sync-snap", "no-resume", "exclude=s@", "skip-parent", "identifier=s",
-                   "no-clone-handling", "no-privilege-elevation", "force-delete", "no-clone-rollback", "no-rollback") or pod2usage(2);
+                   "no-clone-handling", "no-privilege-elevation", "force-delete", "no-clone-rollback", "no-rollback",
+                   "mbuffer-size=s" => \$mbuffer_size) or pod2usage(2);
 
 my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be done with GetOptions arg, as default still needs to be set
 
@@ -56,7 +59,7 @@ my $pscmd = '/bin/ps';
 my $pvcmd = '/usr/bin/pv';
 my $mbuffercmd = '/usr/bin/mbuffer';
 my $sudocmd = '/usr/bin/sudo';
-my $mbufferoptions = '-q -s 128k -m 16M 2>/dev/null';
+my $mbufferoptions = "-q -s 128k -m $mbuffer_size 2>/dev/null";
 # currently using ls to check for file existence because we aren't depending on perl
 # being present on remote machines.
 my $lscmd = '/bin/ls';
@@ -1489,6 +1492,7 @@ Options:
   --skip-parent         Skips syncing of the parent dataset. Does nothing without '--recursive' option.
   --source-bwlimit=<limit k|m|g|t>  Bandwidth limit on the source transfer
   --target-bwlimit=<limit k|m|g|t>  Bandwidth limit on the target transfer
+  --mbuffer-size=VALUE  Specify the mbuffer size (default: 16M), please refer to mbuffer(1) manual page.
   --no-stream           Replicates using newest snapshot instead of intermediates
   --no-sync-snap        Does not create new snapshot, only transfers existing
   --no-clone-rollback   Does not rollback clones on target

From bd4eb491d8d101333381340382d1f1ab3a88d4d5 Mon Sep 17 00:00:00 2001
From: Matthew Swabey <matthew@swabey.org>
Date: Wed, 12 Sep 2018 22:22:03 -0400
Subject: [PATCH 15/62] Added in --sendoptions=OPTIONS and
 --recvoptions=OPTIONS to inject OPTIONS enabling things like syncoid
 --sendoptions=-Lcep and syncoid --recvoptions="-x property"

Co-authored-by: Clint Armstrong <clint@clintarmstrong.net>
---
 syncoid | 41 +++++++++++++++++++++++++++--------------
 1 file changed, 27 insertions(+), 14 deletions(-)

diff --git a/syncoid b/syncoid
index d3e678f..009f10e 100755
--- a/syncoid
+++ b/syncoid
@@ -17,13 +17,24 @@ use Sys::Hostname;
 # Blank defaults to use ssh client's default
 # TODO: Merge into a single "sshflags" option?
 my %args = ('sshkey' => '', 'sshport' => '', 'sshcipher' => '', 'sshoption' => [], 'target-bwlimit' => '', 'source-bwlimit' => '');
-GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsnaps", "recursive|r",
+GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsnaps", "recursive|r", "sendoptions=s", "recvoptions=s",
                    "source-bwlimit=s", "target-bwlimit=s", "sshkey=s", "sshport=i", "sshcipher|c=s", "sshoption|o=s@",
                    "debug", "quiet", "no-stream", "no-sync-snap", "no-resume", "exclude=s@", "skip-parent", "identifier=s",
                    "no-clone-handling", "no-privilege-elevation", "force-delete", "no-clone-rollback", "no-rollback") or pod2usage(2);
 
 my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be done with GetOptions arg, as default still needs to be set
 
+my $sendoptions = '';
+if (length $args{'sendoptions'}) {
+	$sendoptions = $args{'sendoptions'}
+}
+
+my $recvoptions = '';
+if (length $args{'recvoptions'}) {
+	$recvoptions = $args{'recvoptions'}
+}
+
+
 # TODO Expand to accept multiple sources?
 if (scalar(@ARGV) != 2) {
 	print("Source or target not found!\n");
@@ -365,13 +376,13 @@ sub syncdataset {
 		if (defined $args{'no-stream'}) { $oldestsnap = getnewestsnapshot(\%snaps); }
 		my $oldestsnapescaped = escapeshellparam($oldestsnap);
 
-		my $sendcmd = "$sourcesudocmd $zfscmd send $sourcefsescaped\@$oldestsnapescaped";
-		my $recvcmd = "$targetsudocmd $zfscmd receive $receiveextraargs $forcedrecv $targetfsescaped";
+		my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $sourcefsescaped\@$oldestsnapescaped";
+		my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 
 		my $pvsize;
 		if (defined $origin) {
 			my $originescaped = escapeshellparam($origin);
-			$sendcmd = "$sourcesudocmd $zfscmd send -i $originescaped $sourcefsescaped\@$oldestsnapescaped";
+			$sendcmd = "$sourcesudocmd $zfscmd send $sendoptions -i $originescaped $sourcefsescaped\@$oldestsnapescaped";
 			my $streamargBackup = $args{'streamarg'};
 			$args{'streamarg'} = "-i";
 			$pvsize = getsendsize($sourcehost,$origin,"$sourcefs\@$oldestsnap",$sourceisroot);
@@ -419,7 +430,7 @@ sub syncdataset {
 			# $originaltargetreadonly = getzfsvalue($targethost,$targetfs,$targetisroot,'readonly');
 			# setzfsvalue($targethost,$targetfs,$targetisroot,'readonly','on');
 
-			$sendcmd = "$sourcesudocmd $zfscmd send $args{'streamarg'} $sourcefsescaped\@$oldestsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
+			$sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $args{'streamarg'} $sourcefsescaped\@$oldestsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
 			$pvsize = getsendsize($sourcehost,"$sourcefs\@$oldestsnap","$sourcefs\@$newsyncsnap",$sourceisroot);
 			$disp_pvsize = readablebytes($pvsize);
 			if ($pvsize == 0) { $disp_pvsize = "UNKNOWN"; }
@@ -456,8 +467,8 @@ sub syncdataset {
 		# and because this will ony resume the receive to the next
 		# snapshot, do a normal sync after that
 		if (defined($receivetoken)) {
-		    my $sendcmd = "$sourcesudocmd $zfscmd send -t $receivetoken";
-		    my $recvcmd = "$targetsudocmd $zfscmd receive $receiveextraargs $forcedrecv $targetfsescaped";
+		    my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions -t $receivetoken";
+		    my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 		    my $pvsize = getsendsize($sourcehost,"","",$sourceisroot,$receivetoken);
 		    my $disp_pvsize = readablebytes($pvsize);
 		    if ($pvsize == 0) { $disp_pvsize = "UNKNOWN"; }
@@ -608,8 +619,8 @@ sub syncdataset {
 				if ($nextsnapshot) {
 					my $nextsnapshotescaped = escapeshellparam($nextsnapshot);
 
-					my $sendcmd = "$sourcesudocmd $zfscmd send -i $sourcefsescaped#$bookmarkescaped $sourcefsescaped\@$nextsnapshotescaped";
-					my $recvcmd = "$targetsudocmd $zfscmd receive $receiveextraargs $forcedrecv $targetfsescaped";
+					my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions -i $sourcefsescaped#$bookmarkescaped $sourcefsescaped\@$nextsnapshotescaped";
+					my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 					my $synccmd = buildsynccmd($sendcmd,$recvcmd,$pvsize,$sourceisroot,$targetisroot);
 
 					if (!$quiet) { print "Sending incremental $sourcefs#$bookmarkescaped ... $nextsnapshot (~ $disp_pvsize):\n"; }
@@ -623,8 +634,8 @@ sub syncdataset {
 					$matchingsnap = $nextsnapshot;
 					$matchingsnapescaped = escapeshellparam($matchingsnap);
 				} else {
-					my $sendcmd = "$sourcesudocmd $zfscmd send -i $sourcefsescaped#$bookmarkescaped $sourcefsescaped\@$newsyncsnapescaped";
-					my $recvcmd = "$targetsudocmd $zfscmd receive $receiveextraargs $forcedrecv $targetfsescaped";
+					my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions -i $sourcefsescaped#$bookmarkescaped $sourcefsescaped\@$newsyncsnapescaped";
+					my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 					my $synccmd = buildsynccmd($sendcmd,$recvcmd,$pvsize,$sourceisroot,$targetisroot);
 
 					if (!$quiet) { print "Sending incremental $sourcefs#$bookmarkescaped ... $newsyncsnap (~ $disp_pvsize):\n"; }
@@ -640,8 +651,8 @@ sub syncdataset {
 			# do a normal replication if bookmarks aren't used or if previous
 			# bookmark replication was only done to the next oldest snapshot
 			if (!$bookmark || $nextsnapshot) {
-				my $sendcmd = "$sourcesudocmd $zfscmd send $args{'streamarg'} $sourcefsescaped\@$matchingsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
-				my $recvcmd = "$targetsudocmd $zfscmd receive $receiveextraargs $forcedrecv $targetfsescaped";
+				my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $args{'streamarg'} $sourcefsescaped\@$matchingsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
+				my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 				my $pvsize = getsendsize($sourcehost,"$sourcefs\@$matchingsnap","$sourcefs\@$newsyncsnap",$sourceisroot);
 				my $disp_pvsize = readablebytes($pvsize);
 				if ($pvsize == 0) { $disp_pvsize = "UNKNOWN"; }
@@ -1384,7 +1395,7 @@ sub getsendsize {
 		$snaps = "-t $receivetoken";
 	}
 
-	my $getsendsizecmd = "$sourcessh $mysudocmd $zfscmd send -nP $snaps";
+	my $getsendsizecmd = "$sourcessh $mysudocmd $zfscmd send $sendoptions -nP $snaps";
 	if ($debug) { print "DEBUG: getting estimated transfer size from source $sourcehost using \"$getsendsizecmd 2>&1 |\"...\n"; }
 
 	open FH, "$getsendsizecmd 2>&1 |";
@@ -1494,6 +1505,8 @@ Options:
   --no-clone-rollback   Does not rollback clones on target
   --no-rollback         Does not rollback clones or snapshots on target (it probably requires a readonly target)
   --exclude=REGEX       Exclude specific datasets which match the given regular expression. Can be specified multiple times
+  --sendoptions=OPTIONS  DANGER: Inject OPTIONS into zfs send, e.g. syncoid --sendoptions="-Lce" sets zfs send -Lce ...
+  --recvoptions=OPTIONS  DANGER: Inject OPTIONS into zfs received, e.g. syncoid --recvoptions="-x property" sets zfs receive -x property ...
   --sshkey=FILE         Specifies a ssh public key to use to connect
   --sshport=PORT        Connects to remote on a particular port
   --sshcipher|c=CIPHER  Passes CIPHER to ssh to use a particular cipher set

From f941d7f78244f2be1f9b3f7a3d826feccae993b4 Mon Sep 17 00:00:00 2001
From: WG Dev <dev@wangate.com>
Date: Sun, 6 Jan 2019 10:46:24 +0100
Subject: [PATCH 16/62] fix for #316 - CRITICAL ERROR: bookmarks couldn't be
 listed for <volume>

---
 syncoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index d3e678f..b0a8ec8 100755
--- a/syncoid
+++ b/syncoid
@@ -1316,7 +1316,7 @@ sub getbookmarks() {
 	close FH or $error = 1;
 
 	if ($error == 1) {
-		if ($rawbookmarks[0] =~ /invalid type/) {
+		if ($rawbookmarks[0] =~ /invalid type/ or $rawbookmarks[0] =~ /operation not applicable to datasets of this type/) {
 			# no support for zfs bookmarks, return empty hash
 			return %bookmarks;
 		}

From 165faee70600bd866ea037c6c4fd123a6fd227d2 Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Sun, 6 Jan 2019 15:29:29 -0500
Subject: [PATCH 17/62] remove spurious line break from HTML

---
 README.md | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 99d8db1..4a57c32 100644
--- a/README.md
+++ b/README.md
@@ -2,8 +2,7 @@
 
 <img src="http://openoid.net/gplv3-127x51.png" width=127 height=51 align="right">Sanoid is a policy-driven snapshot management tool for ZFS filesystems.  When combined with the Linux KVM hypervisor, you can use it to make your systems <a href="http://openoid.net/transcend" target="_blank">functionally immortal</a>.  
 
-<p align="center"><a href="https://youtu.be/ZgowLNBsu00" target="_blank"><img src="http://www.
-	openoid.net/sanoid_video_launcher.png" alt="sanoid rollback demo" title="sanoid rollback demo"></a><br clear="all"><sup>(Real time demo: rolling back a full-scale cryptomalware infection in seconds!)</sup></p>
+<p align="center"><a href="https://youtu.be/ZgowLNBsu00" target="_blank"><img src="http://www.openoid.net/sanoid_video_launcher.png" alt="sanoid rollback demo" title="sanoid rollback demo"></a><br clear="all"><sup>(Real time demo: rolling back a full-scale cryptomalware infection in seconds!)</sup></p>
 
 More prosaically, you can use Sanoid to create, automatically thin, and monitor snapshots and pool health from a single eminently human-readable TOML config file at /etc/sanoid/sanoid.conf.  (Sanoid also requires a "defaults" file located at /etc/sanoid/sanoid.defaults.conf, which is not user-editable.)  A typical Sanoid system would have a single cron job:
 ```

From 50a237402156f816b7dbfba50f29a4e44e66dfd1 Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Sun, 6 Jan 2019 15:41:18 -0500
Subject: [PATCH 18/62] Revert "Zfs Recursion"

---
 sanoid | 31 ++++++-------------------------
 1 file changed, 6 insertions(+), 25 deletions(-)

diff --git a/sanoid b/sanoid
index 9fcbb88..ce8207c 100755
--- a/sanoid
+++ b/sanoid
@@ -502,13 +502,7 @@ sub take_snapshots {
 					# update to most current possible datestamp
 					%datestamp = get_date();
 					# print "we should have had a $type snapshot of $path $maxage seconds ago; most recent is $newestage seconds old.\n";
-
-					# use zfs (atomic) recursion if specified in config
-					if ($config{$section}{'zfs_recursion'}) {
-						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type\@");
-					} else {
-						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type");
-					}
+					push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type");
 				}
 			}
 		}
@@ -516,14 +510,8 @@ sub take_snapshots {
 
 	if ( (scalar(@newsnaps)) > 0) {
 		foreach my $snap ( @newsnaps ) {
-			my @split = split '@', $snap, -1;
-			my $recursiveFlag = 0;
-			if (scalar(@split) == 3) {
-				$recursiveFlag = 1;
-				chop $snap;
-			}
-			my $dataset = $split[0];
-			my $snapname = $split[1];
+			my $dataset = (split '@', $snap)[0];
+			my $snapname = (split '@', $snap)[1];
 			my $presnapshotfailure = 0;
 			if ($config{$dataset}{'pre_snapshot_script'} and !$args{'readonly'}) {
 				$ENV{'SANOID_TARGET'} = $dataset;
@@ -542,13 +530,8 @@ sub take_snapshots {
 			}
 			if ($args{'verbose'}) { print "taking snapshot $snap\n"; }
 			if (!$args{'readonly'}) {
-				if ($recursiveFlag) {
-					system($zfs, "snapshot", "-r", "$snap") == 0
-						or warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
-				} else {
-					system($zfs, "snapshot", "$snap") == 0
-						or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
-				}
+				system($zfs, "snapshot", "$snap") == 0
+					or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
 			}
 			if ($config{$dataset}{'post_snapshot_script'} and !$args{'readonly'}) {
 				if (!$presnapshotfailure or $config{$dataset}{'force_post_snapshot_script'}) {
@@ -863,9 +846,7 @@ sub init {
 		my $recursive = $ini{$section}{'recursive'} && grep( /^$ini{$section}{'recursive'}$/, @istrue );
 		my $skipChildren = $ini{$section}{'skip_children'} && grep( /^$ini{$section}{'skip_children'}$/, @istrue );
 		my @datasets;
-		if ($ini{$section}{'recursive'} =~ /zfs/i) {
-			$config{$section}{'zfs_recursion'} = 1;
-		} elsif ($recursive || $skipChildren) {
+		if ($recursive || $skipChildren) {
 			@datasets = getchilddatasets($config{$section}{'path'});
 			DATASETS: foreach my $dataset(@datasets) {
 				chomp $dataset;

From e4e90659a10734ff4b38ca6de2aacdd7f4641000 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 7 Jan 2019 12:08:55 +0100
Subject: [PATCH 19/62] removed debug output from excludes loop

---
 syncoid | 1 -
 1 file changed, 1 deletion(-)

diff --git a/syncoid b/syncoid
index b0a8ec8..9c87376 100755
--- a/syncoid
+++ b/syncoid
@@ -213,7 +213,6 @@ sub getchilddatasets {
 		if (defined $args{'exclude'}) {
 			my $excludes = $args{'exclude'};
 			foreach (@$excludes) {
-				print("$dataset\n");
 				if ($dataset =~ /$_/) {
 					if ($debug) { print "DEBUG: excluded $dataset because of $_\n"; }
 					next DATASETS;

From 271ede7116c7d57bf0ea1d48487d9190fe527b4d Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 7 Jan 2019 19:59:31 +0100
Subject: [PATCH 20/62] Revert "Revert "Zfs Recursion""

This reverts commit 50a237402156f816b7dbfba50f29a4e44e66dfd1.
---
 sanoid | 31 +++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)

diff --git a/sanoid b/sanoid
index ce8207c..9fcbb88 100755
--- a/sanoid
+++ b/sanoid
@@ -502,7 +502,13 @@ sub take_snapshots {
 					# update to most current possible datestamp
 					%datestamp = get_date();
 					# print "we should have had a $type snapshot of $path $maxage seconds ago; most recent is $newestage seconds old.\n";
-					push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type");
+
+					# use zfs (atomic) recursion if specified in config
+					if ($config{$section}{'zfs_recursion'}) {
+						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type\@");
+					} else {
+						push(@newsnaps, "$path\@autosnap_$datestamp{'sortable'}${dateSuffix}_$type");
+					}
 				}
 			}
 		}
@@ -510,8 +516,14 @@ sub take_snapshots {
 
 	if ( (scalar(@newsnaps)) > 0) {
 		foreach my $snap ( @newsnaps ) {
-			my $dataset = (split '@', $snap)[0];
-			my $snapname = (split '@', $snap)[1];
+			my @split = split '@', $snap, -1;
+			my $recursiveFlag = 0;
+			if (scalar(@split) == 3) {
+				$recursiveFlag = 1;
+				chop $snap;
+			}
+			my $dataset = $split[0];
+			my $snapname = $split[1];
 			my $presnapshotfailure = 0;
 			if ($config{$dataset}{'pre_snapshot_script'} and !$args{'readonly'}) {
 				$ENV{'SANOID_TARGET'} = $dataset;
@@ -530,8 +542,13 @@ sub take_snapshots {
 			}
 			if ($args{'verbose'}) { print "taking snapshot $snap\n"; }
 			if (!$args{'readonly'}) {
-				system($zfs, "snapshot", "$snap") == 0
-					or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
+				if ($recursiveFlag) {
+					system($zfs, "snapshot", "-r", "$snap") == 0
+						or warn "CRITICAL ERROR: $zfs snapshot -r $snap failed, $?";
+				} else {
+					system($zfs, "snapshot", "$snap") == 0
+						or warn "CRITICAL ERROR: $zfs snapshot $snap failed, $?";
+				}
 			}
 			if ($config{$dataset}{'post_snapshot_script'} and !$args{'readonly'}) {
 				if (!$presnapshotfailure or $config{$dataset}{'force_post_snapshot_script'}) {
@@ -846,7 +863,9 @@ sub init {
 		my $recursive = $ini{$section}{'recursive'} && grep( /^$ini{$section}{'recursive'}$/, @istrue );
 		my $skipChildren = $ini{$section}{'skip_children'} && grep( /^$ini{$section}{'skip_children'}$/, @istrue );
 		my @datasets;
-		if ($recursive || $skipChildren) {
+		if ($ini{$section}{'recursive'} =~ /zfs/i) {
+			$config{$section}{'zfs_recursion'} = 1;
+		} elsif ($recursive || $skipChildren) {
 			@datasets = getchilddatasets($config{$section}{'path'});
 			DATASETS: foreach my $dataset(@datasets) {
 				chomp $dataset;

From 0fc16b4903b98ef03a120f6c1f46a39ed050005e Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 7 Jan 2019 09:48:04 +0100
Subject: [PATCH 21/62] fixed uninitialzed variable warning and clarified
 recursive value

---
 sanoid | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sanoid b/sanoid
index 9fcbb88..db5376b 100755
--- a/sanoid
+++ b/sanoid
@@ -765,6 +765,8 @@ sub init {
 
 	# we'll use these later to normalize potentially true and false values on any toggle keys
 	my @toggles = ('autosnap','autoprune','monitor_dont_warn','monitor_dont_crit','monitor','recursive','process_children_only','skip_children','no_inconsistent_snapshot','force_post_snapshot_script');
+	# recursive is defined as toggle but can also have the special value "zfs", it is kept to be backward compatible
+
 	my @istrue=(1,"true","True","TRUE","yes","Yes","YES","on","On","ON");
 	my @isfalse=(0,"false","False","FALSE","no","No","NO","off","Off","OFF");
 
@@ -861,9 +863,10 @@ sub init {
 
 		# how 'bout some recursion? =)
 		my $recursive = $ini{$section}{'recursive'} && grep( /^$ini{$section}{'recursive'}$/, @istrue );
+		my $zfsRecursive = $ini{$section}{'recursive'} && $ini{$section}{'recursive'} =~ /zfs/i;
 		my $skipChildren = $ini{$section}{'skip_children'} && grep( /^$ini{$section}{'skip_children'}$/, @istrue );
 		my @datasets;
-		if ($ini{$section}{'recursive'} =~ /zfs/i) {
+		if ($zfsRecursive) {
 			$config{$section}{'zfs_recursion'} = 1;
 		} elsif ($recursive || $skipChildren) {
 			@datasets = getchilddatasets($config{$section}{'path'});

From ab12540a96633ce92abfa11c099a0cfa4ef83950 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 7 Jan 2019 18:22:54 +0100
Subject: [PATCH 22/62] added message if zfs recursion is used and fixed
 ordinary recursion in combination with zfs recursion

---
 sanoid | 48 +++++++++++++++++++++++++++++++++++-------------
 1 file changed, 35 insertions(+), 13 deletions(-)

diff --git a/sanoid b/sanoid
index db5376b..304f5f6 100755
--- a/sanoid
+++ b/sanoid
@@ -516,10 +516,12 @@ sub take_snapshots {
 
 	if ( (scalar(@newsnaps)) > 0) {
 		foreach my $snap ( @newsnaps ) {
+			my $extraMessage = "";
 			my @split = split '@', $snap, -1;
 			my $recursiveFlag = 0;
 			if (scalar(@split) == 3) {
 				$recursiveFlag = 1;
+				$extraMessage = " (zfs recursive)";
 				chop $snap;
 			}
 			my $dataset = $split[0];
@@ -540,7 +542,7 @@ sub take_snapshots {
 					$presnapshotfailure = 1;
 				}
 			}
-			if ($args{'verbose'}) { print "taking snapshot $snap\n"; }
+			if ($args{'verbose'}) { print "taking snapshot $snap$extraMessage\n"; }
 			if (!$args{'readonly'}) {
 				if ($recursiveFlag) {
 					system($zfs, "snapshot", "-r", "$snap") == 0
@@ -862,29 +864,49 @@ sub init {
 		}
 
 		# how 'bout some recursion? =)
+		if ($config{$section}{'zfs_recursion'} && $config{$section}{'zfs_recursion'} == 1 && $config{$section}{'autosnap'} == 1) {
+			warn "ignored autosnap configuration for '$section' because it's part of a zfs recursion.\n";
+			$config{$section}{'autosnap'} = 0;
+		}
+
 		my $recursive = $ini{$section}{'recursive'} && grep( /^$ini{$section}{'recursive'}$/, @istrue );
 		my $zfsRecursive = $ini{$section}{'recursive'} && $ini{$section}{'recursive'} =~ /zfs/i;
 		my $skipChildren = $ini{$section}{'skip_children'} && grep( /^$ini{$section}{'skip_children'}$/, @istrue );
 		my @datasets;
-		if ($zfsRecursive) {
-			$config{$section}{'zfs_recursion'} = 1;
-		} elsif ($recursive || $skipChildren) {
+		if ($zfsRecursive || $recursive || $skipChildren) {
+			if ($zfsRecursive) {
+				$config{$section}{'zfs_recursion'} = 1;
+			}
+
 			@datasets = getchilddatasets($config{$section}{'path'});
 			DATASETS: foreach my $dataset(@datasets) {
 				chomp $dataset;
 
-				if ($skipChildren) {
-					if ($args{'debug'}) { print "DEBUG: ignoring $dataset.\n"; }
-					delete $config{$dataset};
-					next DATASETS;
-				}
+				if ($zfsRecursive) {
+					# don't try to take the snapshot ourself, recursive zfs snapshot will take care of that
+					$config{$dataset}{'autosnap'} = 0;
 
-				foreach my $key (keys %{$config{$section}} ) {
-					if (! ($key =~ /template|recursive|children_only/)) {
-						if ($args{'debug'}) { print "DEBUG: recursively setting $key from $section to $dataset.\n"; }
-						$config{$dataset}{$key} = $config{$section}{$key};
+					foreach my $key (keys %{$config{$section}} ) {
+						if (! ($key =~ /template|recursive|children_only|autosnap/)) {
+							if ($args{'debug'}) { print "DEBUG: recursively setting $key from $section to $dataset.\n"; }
+							$config{$dataset}{$key} = $config{$section}{$key};
+						}
+					}
+				} else {
+					if ($skipChildren) {
+						if ($args{'debug'}) { print "DEBUG: ignoring $dataset.\n"; }
+						delete $config{$dataset};
+						next DATASETS;
+					}
+
+					foreach my $key (keys %{$config{$section}} ) {
+						if (! ($key =~ /template|recursive|children_only/)) {
+							if ($args{'debug'}) { print "DEBUG: recursively setting $key from $section to $dataset.\n"; }
+							$config{$dataset}{$key} = $config{$section}{$key};
+						}
 					}
 				}
+
 				$config{$dataset}{'path'} = $dataset;
 				$config{$dataset}{'initialized'} = 1;
 			}

From ea482ce7b6d7f662701874c2ef889054831f36d5 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Tue, 8 Jan 2019 20:10:38 +0100
Subject: [PATCH 23/62] make tests work on FreeBSD

---
 tests/1_one_year/run.sh                            |  2 +-
 tests/2_dst_handling/run.sh                        |  2 +-
 tests/common/lib.sh                                | 14 +++++++++++++-
 tests/run-tests.sh                                 |  2 +-
 .../1_bookmark_replication_intermediate/run.sh     |  4 ++--
 .../2_bookmark_replication_no_intermediate/run.sh  |  4 ++--
 tests/syncoid/3_force_delete/run.sh                |  4 ++--
 7 files changed, 22 insertions(+), 10 deletions(-)

diff --git a/tests/1_one_year/run.sh b/tests/1_one_year/run.sh
index 1cae7b4..88100da 100755
--- a/tests/1_one_year/run.sh
+++ b/tests/1_one_year/run.sh
@@ -39,7 +39,7 @@ function cleanUp {
 trap cleanUp EXIT
 
 while [ $timestamp -le $END ]; do
-    date --utc --set @$timestamp; date; "${SANOID}" --cron --verbose
+    setdate $timestamp; date; "${SANOID}" --cron --verbose
     timestamp=$((timestamp+3600))
 done
 
diff --git a/tests/2_dst_handling/run.sh b/tests/2_dst_handling/run.sh
index eba21ed..7d7774e 100755
--- a/tests/2_dst_handling/run.sh
+++ b/tests/2_dst_handling/run.sh
@@ -42,7 +42,7 @@ function cleanUp {
 trap cleanUp EXIT
 
 while [ $timestamp -le $END ]; do
-    date --utc --set @$timestamp; date; "${SANOID}" --cron --verbose
+    setdate $timestamp; date; "${SANOID}" --cron --verbose
     timestamp=$((timestamp+900))
 done
 
diff --git a/tests/common/lib.sh b/tests/common/lib.sh
index 78f128b..3aee40d 100644
--- a/tests/common/lib.sh
+++ b/tests/common/lib.sh
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+unamestr="$(uname)"
+
 function setup {
     export LANG=C
     export LANGUAGE=C
@@ -90,7 +92,7 @@ function verifySnapshotList {
         message="${message}monthly snapshot count is wrong: ${monthly_count}\n"
     fi
 
-    checksum=$(sha256sum "${RESULT}" | cut -d' ' -f1)
+    checksum=$(shasum -a 256 "${RESULT}" | cut -d' ' -f1)
     if [ "${checksum}" != "${CHECKSUM}" ]; then
         failed=1
         message="${message}result checksum mismatch\n"
@@ -105,3 +107,13 @@ function verifySnapshotList {
 
     exit 1
 }
+
+function setdate {
+	TIMESTAMP="$1"
+
+	if [ "$unamestr" == 'FreeBSD' ]; then
+		date -u -f '%s' "${TIMESTAMP}"
+	else
+		date --utc --set "@${TIMESTAMP}"
+	fi
+}
diff --git a/tests/run-tests.sh b/tests/run-tests.sh
index a8469e9..38054b0 100755
--- a/tests/run-tests.sh
+++ b/tests/run-tests.sh
@@ -15,7 +15,7 @@ for test in */; do
 
     echo -n "Running test ${testName} ... "
     cd "${test}"
-    echo | bash run.sh > "${LOGFILE}" 2>&1
+    echo -n y | bash run.sh > "${LOGFILE}" 2>&1
 
     if [ $? -eq 0 ]; then
         echo "[PASS]"
diff --git a/tests/syncoid/1_bookmark_replication_intermediate/run.sh b/tests/syncoid/1_bookmark_replication_intermediate/run.sh
index 11edb04..66af442 100755
--- a/tests/syncoid/1_bookmark_replication_intermediate/run.sh
+++ b/tests/syncoid/1_bookmark_replication_intermediate/run.sh
@@ -46,8 +46,8 @@ zfs snapshot "${POOL_NAME}"/src@snap5
 ../../../syncoid --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst || exit 1
 
 # verify
-output=$(zfs list -t snapshot -r "${POOL_NAME}" -H -o name)
-checksum=$(echo "${output}" | grep -v syncoid_ | sha256sum)
+output=$(zfs list -t snapshot -r -H -o name "${POOL_NAME}")
+checksum=$(echo "${output}" | grep -v syncoid_ | shasum -a 256)
 
 if [ "${checksum}" != "${TARGET_CHECKSUM}" ]; then
 	exit 1
diff --git a/tests/syncoid/2_bookmark_replication_no_intermediate/run.sh b/tests/syncoid/2_bookmark_replication_no_intermediate/run.sh
index 94ac690..f6c1755 100755
--- a/tests/syncoid/2_bookmark_replication_no_intermediate/run.sh
+++ b/tests/syncoid/2_bookmark_replication_no_intermediate/run.sh
@@ -46,8 +46,8 @@ zfs snapshot "${POOL_NAME}"/src@snap5
 ../../../syncoid --no-stream --no-sync-snap --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst || exit 1
 
 # verify
-output=$(zfs list -t snapshot -r "${POOL_NAME}" -H -o name)
-checksum=$(echo "${output}" | sha256sum)
+output=$(zfs list -t snapshot -r -H -o name "${POOL_NAME}")
+checksum=$(echo "${output}" | shasum -a 256)
 
 if [ "${checksum}" != "${TARGET_CHECKSUM}" ]; then
 	exit 1
diff --git a/tests/syncoid/3_force_delete/run.sh b/tests/syncoid/3_force_delete/run.sh
index 03ad9fa..25044cb 100755
--- a/tests/syncoid/3_force_delete/run.sh
+++ b/tests/syncoid/3_force_delete/run.sh
@@ -37,8 +37,8 @@ sleep 1
 ../../../syncoid -r --force-delete --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst || exit 1
 
 # verify
-output=$(zfs list -t snapshot -r "${POOL_NAME}" -H -o name | sed 's/@syncoid_.*$'/@syncoid_/)
-checksum=$(echo "${output}" | sha256sum)
+output=$(zfs list -t snapshot -r -H -o name "${POOL_NAME}" | sed 's/@syncoid_.*$'/@syncoid_/)
+checksum=$(echo "${output}" | shasum -a 256)
 
 if [ "${checksum}" != "${TARGET_CHECKSUM}" ]; then
 	exit 1

From c3e20fdefcd0a61976155831c6b2b23ac7aa035f Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 9 Jan 2019 17:41:16 +0100
Subject: [PATCH 24/62] FreeBSD sed needs a different syntax

---
 tests/common/lib.sh | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/tests/common/lib.sh b/tests/common/lib.sh
index 3aee40d..b070da2 100644
--- a/tests/common/lib.sh
+++ b/tests/common/lib.sh
@@ -60,7 +60,11 @@ function saveSnapshotList {
     zfs list -t snapshot -o name -Hr "${POOL_NAME}" | sort > "${RESULT}"
 
     # clear the seconds for comparing
-    sed -i 's/\(autosnap_[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]_[0-9][0-9]:[0-9][0-9]:\)[0-9][0-9]_/\100_/g' "${RESULT}"
+    if [ "$unamestr" == 'FreeBSD' ]; then
+        sed -i '' 's/\(autosnap_[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]_[0-9][0-9]:[0-9][0-9]:\)[0-9][0-9]_/\100_/g' "${RESULT}"
+    else
+        sed -i 's/\(autosnap_[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]_[0-9][0-9]:[0-9][0-9]:\)[0-9][0-9]_/\100_/g' "${RESULT}"
+    fi
 }
 
 function verifySnapshotList {
@@ -109,11 +113,11 @@ function verifySnapshotList {
 }
 
 function setdate {
-	TIMESTAMP="$1"
+    TIMESTAMP="$1"
 
-	if [ "$unamestr" == 'FreeBSD' ]; then
-		date -u -f '%s' "${TIMESTAMP}"
-	else
-		date --utc --set "@${TIMESTAMP}"
-	fi
+    if [ "$unamestr" == 'FreeBSD' ]; then
+        date -u -f '%s' "${TIMESTAMP}"
+    else
+        date --utc --set "@${TIMESTAMP}"
+    fi
 }

From 7ef435c5e31c00d47c8b4442334eed5df4ca175c Mon Sep 17 00:00:00 2001
From: Jim Salter <jim@jrs-s.net>
Date: Fri, 25 Jan 2019 14:48:17 -0500
Subject: [PATCH 25/62] add "hotspare" template (for local, hourly replication
 targets)

---
 sanoid.conf | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/sanoid.conf b/sanoid.conf
index 9f13105..6bd5c62 100644
--- a/sanoid.conf
+++ b/sanoid.conf
@@ -69,6 +69,27 @@
 	daily_warn = 48
 	daily_crit = 60
 
+[template_hotspare]
+	autoprune = yes
+	frequently = 0
+	hourly = 30
+	daily = 90
+	monthly = 3
+	yearly = 0
+
+	### don't take new snapshots - snapshots on backup
+	### datasets are replicated in from source, not
+	### generated locally
+	autosnap = no
+
+	### monitor hourlies and dailies, but don't warn or
+	### crit until they're over 4h old, since replication
+	### is typically hourly only
+	hourly_warn = 4h
+	hourly_crit = 6h
+	daily_warn = 2d
+	daily_crit = 4d
+
 [template_scripts]
 	### dataset and snapshot name will be supplied as environment variables
 	### for all pre/post/prune scripts ($SANOID_TARGET, $SANOID_SNAPNAME)

From c12b6d716bee8070c111832ff87a970985ec12ec Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Fri, 25 Jan 2019 23:25:22 +0100
Subject: [PATCH 26/62] check pool capability for resumeable replication
 instead of checking for support in the zfs module

---
 syncoid | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

diff --git a/syncoid b/syncoid
index 3f6f9ba..b12f059 100755
--- a/syncoid
+++ b/syncoid
@@ -64,6 +64,7 @@ my $quiet = $args{'quiet'};
 my $resume = !$args{'no-resume'};
 
 my $zfscmd = '/sbin/zfs';
+my $zpoolcmd = '/sbin/zpool';
 my $sshcmd = '/usr/bin/ssh';
 my $pscmd = '/bin/ps';
 
@@ -886,14 +887,32 @@ sub checkcommands {
 
 	# check for ZFS resume feature support
 	if ($resume) {
-		my $resumechkcmd = "$zfscmd get -d 0 receive_resume_token";
+		my @parts = split ('/', $args{'source'});
+		my $srcpool = $parts[0];
+		@parts = split ('/', $args{'target'});
+		my $dstpool = $parts[0];
+
+		$srcpool = escapeshellparam($srcpool);
+		$dstpool = escapeshellparam($dstpool);
+
+		if ($sourcehost ne '') {
+			# double escaping needed
+			$srcpool = escapeshellparam($srcpool);
+		}
+
+		if ($targethost ne '') {
+			# double escaping needed
+			$dstpool = escapeshellparam($dstpool);
+		}
+
+		my $resumechkcmd = "$zpoolcmd get -o value -H feature\@extensible_dataset";
 
 		if ($debug) { print "DEBUG: checking availability of zfs resume feature on source...\n"; }
-		$avail{'sourceresume'} = system("$sourcessh $resumechkcmd >/dev/null 2>&1");
+		$avail{'sourceresume'} = system("$sourcessh $resumechkcmd $srcpool 2>/dev/null | grep '\\(active\\|enabled\\)' >/dev/null 2>&1");
 		$avail{'sourceresume'} = $avail{'sourceresume'} == 0 ? 1 : 0;
 
 		if ($debug) { print "DEBUG: checking availability of zfs resume feature on target...\n"; }
-		$avail{'targetresume'} = system("$targetssh $resumechkcmd >/dev/null 2>&1");
+		$avail{'targetresume'} = system("$targetssh $resumechkcmd $dstpool 2>/dev/null | grep '\\(active\\|enabled\\)' >/dev/null 2>&1");
 		$avail{'targetresume'} = $avail{'targetresume'} == 0 ? 1 : 0;
 
 		if ($avail{'sourceresume'} == 0 || $avail{'targetresume'} == 0) {

From 85561ddcda0da820d62b205d69346c8f28ae7839 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Fri, 25 Jan 2019 23:46:58 +0100
Subject: [PATCH 27/62] handle the case of mistyped dataset name nicely

---
 syncoid | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index 3f6f9ba..6040110 100755
--- a/syncoid
+++ b/syncoid
@@ -263,6 +263,12 @@ sub syncdataset {
 
 	my $sync = getzfsvalue($sourcehost,$sourcefs,$sourceisroot,'syncoid:sync');
 
+	if (!defined $sync) {
+		# zfs already printed the corresponding error
+		if ($exitcode < 2) { $exitcode = 2; }
+		return 0;
+	}
+
 	if ($sync eq 'true' || $sync eq '-' || $sync eq '') {
 		# empty is handled the same as unset (aka: '-')
 		# definitely sync this dataset - if a host is called 'true' or '-', then you're special
@@ -978,6 +984,11 @@ sub getzfsvalue {
 	open FH, "$rhost $mysudocmd $zfscmd get -H $property $fsescaped |";
 	my $value = <FH>;
 	close FH;
+
+	if (!defined $value) {
+		return undef;
+	}
+
 	my @values = split(/\t/,$value);
 	$value = $values[2];
 	return $value;
@@ -1468,7 +1479,7 @@ sub getreceivetoken() {
 	my ($rhost,$fs,$isroot) = @_;
 	my $token = getzfsvalue($rhost,$fs,$isroot,"receive_resume_token");
 
-	if ($token ne '-' && $token ne '') {
+	if (defined $token && $token ne '-' && $token ne '') {
 		return $token;
 	}
 

From 132c765e28316116735d1682bed38b609efd58a1 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Fri, 25 Jan 2019 23:58:12 +0100
Subject: [PATCH 28/62] error out if no datasets are processed with recursive
 arg

---
 syncoid | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/syncoid b/syncoid
index 6040110..8f17c13 100755
--- a/syncoid
+++ b/syncoid
@@ -124,6 +124,12 @@ if (!defined $args{'recursive'}) {
 	if ($debug) { print "DEBUG: recursive sync of $sourcefs.\n"; }
 	my @datasets = getchilddatasets($sourcehost, $sourcefs, $sourceisroot);
 
+	if (!@datasets) {
+		warn "CRITICAL ERROR: no datasets found";
+		@datasets = ();
+		$exitcode = 2;
+	}
+
 	my @deferred;
 
 	foreach my $datasetProperties(@datasets) {

From ea9e989fed298c89b269c47cb00ece83c601ec74 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 30 Jan 2019 23:26:07 +0100
Subject: [PATCH 29/62] only list filesystems and volumes

---
 sanoid | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sanoid b/sanoid
index 82cfce9..3cb1bad 100755
--- a/sanoid
+++ b/sanoid
@@ -542,7 +542,7 @@ sub take_snapshots {
 					$ENV{'SANOID_TARGET'} = $dataset;
 					$ENV{'SANOID_SNAPNAME'} = $snapname;
 					if ($args{'verbose'}) { print "executing post_snapshot_script '".$config{$dataset}{'post_snapshot_script'}."' on dataset '$dataset'\n"; }
-	
+
 					if (!$args{'readonly'}) {
 						runscript('post_snapshot_script',$dataset);
 					}
@@ -1422,7 +1422,7 @@ sub getchilddatasets {
 	my $fs = shift;
 	my $mysudocmd = '';
 
-	my $getchildrencmd = "$mysudocmd $zfs list -o name -Hr $fs |";
+	my $getchildrencmd = "$mysudocmd $zfs list -o name -t filesystem,volume -Hr $fs |";
 	 if ($args{'debug'}) { print "DEBUG: getting list of child datasets on $fs using $getchildrencmd...\n"; }
 	open FH, $getchildrencmd;
 	my @children = <FH>;

From 7141eab594ff8a362f2e4f3a8565eab699ce1f7e Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 30 Jan 2019 23:49:38 +0100
Subject: [PATCH 30/62] fix bookmark edge case where replication was already
 done to the latest snapshot (+ test)

---
 syncoid                                       |  5 ++
 .../4_bookmark_replication_edge_case/run.sh   | 46 +++++++++++++++++++
 2 files changed, 51 insertions(+)
 create mode 100755 tests/syncoid/4_bookmark_replication_edge_case/run.sh

diff --git a/syncoid b/syncoid
index 3f6f9ba..812c733 100755
--- a/syncoid
+++ b/syncoid
@@ -654,6 +654,11 @@ sub syncdataset {
 			# do a normal replication if bookmarks aren't used or if previous
 			# bookmark replication was only done to the next oldest snapshot
 			if (!$bookmark || $nextsnapshot) {
+				if ($matchingsnap eq $newsyncsnap) {
+					# edge case: bookmark replication used the latest snapshot
+					return 0;
+				}
+
 				my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $args{'streamarg'} $sourcefsescaped\@$matchingsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
 				my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 				my $pvsize = getsendsize($sourcehost,"$sourcefs\@$matchingsnap","$sourcefs\@$newsyncsnap",$sourceisroot);
diff --git a/tests/syncoid/4_bookmark_replication_edge_case/run.sh b/tests/syncoid/4_bookmark_replication_edge_case/run.sh
new file mode 100755
index 0000000..2a93ce4
--- /dev/null
+++ b/tests/syncoid/4_bookmark_replication_edge_case/run.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+
+# test replication edge cases with bookmarks
+
+set -x
+set -e
+
+. ../../common/lib.sh
+
+POOL_IMAGE="/tmp/syncoid-test-4.zpool"
+POOL_SIZE="200M"
+POOL_NAME="syncoid-test-4"
+TARGET_CHECKSUM="ad383b157b01635ddcf13612ac55577ad9c8dcf3fbfc9eb91792e27ec8db739b  -"
+
+truncate -s "${POOL_SIZE}" "${POOL_IMAGE}"
+
+zpool create -m none -f "${POOL_NAME}" "${POOL_IMAGE}"
+
+function cleanUp {
+  zpool export "${POOL_NAME}"
+}
+
+# export pool in any case
+trap cleanUp EXIT
+
+zfs create "${POOL_NAME}"/src
+zfs snapshot "${POOL_NAME}"/src@snap1
+zfs bookmark "${POOL_NAME}"/src@snap1 "${POOL_NAME}"/src#snap1
+# initial replication
+../../../syncoid --no-sync-snap --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst
+# destroy last common snapshot on source
+zfs destroy "${POOL_NAME}"/src@snap1
+zfs snapshot "${POOL_NAME}"/src@snap2
+
+# replicate which should fallback to bookmarks and stop because it's already on the latest snapshot
+../../../syncoid --no-sync-snap --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst || exit 1
+
+# verify
+output=$(zfs list -t snapshot -r -H -o name "${POOL_NAME}")
+checksum=$(echo "${output}" | grep -v syncoid_ | shasum -a 256)
+
+if [ "${checksum}" != "${TARGET_CHECKSUM}" ]; then
+	exit 1
+fi
+
+exit 0

From d68d7b4afa0ac0d8d126a952c5a5f8730941754b Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Thu, 31 Jan 2019 00:45:12 +0100
Subject: [PATCH 31/62] implemented creation of zfs bookmarks

---
 README.md |  4 ++++
 syncoid   | 19 ++++++++++++++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index a8cdfc7..fe4b1a9 100644
--- a/README.md
+++ b/README.md
@@ -202,6 +202,10 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
 
 	This argument tells syncoid to restrict itself to existing snapshots, instead of creating a semi-ephemeral syncoid snapshot at execution time. Especially useful in multi-target (A->B, A->C) replication schemes, where you might otherwise accumulate a large number of foreign syncoid snapshots.
 
++ --create-bookmark
+
+	This argument tells syncoid to create a zfs bookmark for the newest snapshot after it got replicated successfully. The bookmark name will be equal to the snapshot name. Only works in combination with the --no-sync-snap option. This can be very useful for irregular replication where the last matching snapshot on the source was already deleted but the bookmark remains so a replication is still possible. 
+
 + --no-clone-rollback
 
 	Do not rollback clones on target
diff --git a/syncoid b/syncoid
index 3f6f9ba..5aba87f 100755
--- a/syncoid
+++ b/syncoid
@@ -23,6 +23,7 @@ GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsn
                    "source-bwlimit=s", "target-bwlimit=s", "sshkey=s", "sshport=i", "sshcipher|c=s", "sshoption|o=s@",
                    "debug", "quiet", "no-stream", "no-sync-snap", "no-resume", "exclude=s@", "skip-parent", "identifier=s",
                    "no-clone-handling", "no-privilege-elevation", "force-delete", "no-clone-rollback", "no-rollback",
+                   "create-bookmark",
                    "mbuffer-size=s" => \$mbuffer_size) or pod2usage(2);
 
 my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be done with GetOptions arg, as default still needs to be set
@@ -677,7 +678,22 @@ sub syncdataset {
 		}
 	}
 
-	if (!defined $args{'no-sync-snap'}) {
+	if (defined $args{'no-sync-snap'}) {
+		if (defined $args{'create-bookmark'}) {
+			my $bookmarkcmd;
+			if ($sourcehost ne '') {
+				$bookmarkcmd = "$sshcmd $sourcehost " . escapeshellparam("$zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped");
+			} else {
+				$bookmarkcmd = "$sourcesudocmd $zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped";
+			}
+			if ($debug) { print "DEBUG: $bookmarkcmd\n"; }
+			system($bookmarkcmd) == 0 or do {
+				warn "CRITICAL ERROR: $bookmarkcmd failed: $?";
+				if ($exitcode < 2) { $exitcode = 2; }
+				return 0;
+			};
+		}
+	} else {
 		# prune obsolete sync snaps on source and target (only if this run created ones).
 		pruneoldsyncsnaps($sourcehost,$sourcefs,$newsyncsnap,$sourceisroot,keys %{ $snaps{'source'}});
 		pruneoldsyncsnaps($targethost,$targetfs,$newsyncsnap,$targetisroot,keys %{ $snaps{'target'}});
@@ -1506,6 +1522,7 @@ Options:
   --mbuffer-size=VALUE  Specify the mbuffer size (default: 16M), please refer to mbuffer(1) manual page.
   --no-stream           Replicates using newest snapshot instead of intermediates
   --no-sync-snap        Does not create new snapshot, only transfers existing
+  --create-bookmark		Creates a zfs bookmark for the newest snapshot on the source after replication succeeds (only works with --no-sync-snap)
   --no-clone-rollback   Does not rollback clones on target
   --no-rollback         Does not rollback clones or snapshots on target (it probably requires a readonly target)
   --exclude=REGEX       Exclude specific datasets which match the given regular expression. Can be specified multiple times

From 5107380cb3a68790324aa0c30c42ebea5687ec35 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Thu, 31 Jan 2019 00:47:13 +0100
Subject: [PATCH 32/62] codestyle

---
 syncoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index 5aba87f..eee2d44 100755
--- a/syncoid
+++ b/syncoid
@@ -1522,7 +1522,7 @@ Options:
   --mbuffer-size=VALUE  Specify the mbuffer size (default: 16M), please refer to mbuffer(1) manual page.
   --no-stream           Replicates using newest snapshot instead of intermediates
   --no-sync-snap        Does not create new snapshot, only transfers existing
-  --create-bookmark		Creates a zfs bookmark for the newest snapshot on the source after replication succeeds (only works with --no-sync-snap)
+  --create-bookmark     Creates a zfs bookmark for the newest snapshot on the source after replication succeeds (only works with --no-sync-snap)
   --no-clone-rollback   Does not rollback clones on target
   --no-rollback         Does not rollback clones or snapshots on target (it probably requires a readonly target)
   --exclude=REGEX       Exclude specific datasets which match the given regular expression. Can be specified multiple times

From c5eebbe81dcc9d205c474873035a3e42202eba5f Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Thu, 31 Jan 2019 18:14:35 +0100
Subject: [PATCH 33/62] implemented parsing of provided zfs send/recv options
 and whitelisting for each use case as needed

---
 syncoid | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 82 insertions(+), 7 deletions(-)

diff --git a/syncoid b/syncoid
index 3f6f9ba..b0a93af 100755
--- a/syncoid
+++ b/syncoid
@@ -27,14 +27,14 @@ GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsn
 
 my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be done with GetOptions arg, as default still needs to be set
 
-my $sendoptions = '';
+my @sendoptions = ();
 if (length $args{'sendoptions'}) {
-	$sendoptions = $args{'sendoptions'}
+	@sendoptions = parsespecialoptions($args{'sendoptions'});
 }
 
-my $recvoptions = '';
+my @recvoptions = ();
 if (length $args{'recvoptions'}) {
-	$recvoptions = $args{'recvoptions'}
+	@recvoptions = parsespecialoptions($args{'recvoptions'});
 }
 
 
@@ -352,6 +352,9 @@ sub syncdataset {
 		# with ZFS on Linux (possibly OpenZFS in general) when setting/unsetting readonly.
 	#my $originaltargetreadonly;
 
+	my $sendoptions = getoptionsline(\@sendoptions, ('D','L','P','R','c','e','p','v','w'));
+	my $recvoptions = getoptionsline(\@recvoptions, ('o','x','u','v'));
+
 	# sync 'em up.
 	if (! $targetexists) {
 		# do an initial sync from the oldest source snapshot
@@ -470,6 +473,7 @@ sub syncdataset {
 		# and because this will ony resume the receive to the next
 		# snapshot, do a normal sync after that
 		if (defined($receivetoken)) {
+		    $sendoptions = getoptionsline(\@sendoptions, ('P','e','v','w'));
 		    my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions -t $receivetoken";
 		    my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 		    my $pvsize = getsendsize($sourcehost,"","",$sourceisroot,$receivetoken);
@@ -619,9 +623,9 @@ sub syncdataset {
 				my $pvsize = 0;
 				my $disp_pvsize = "UNKNOWN";
 
+				$sendoptions = getoptionsline(\@sendoptions, ('L','c','e','w'));
 				if ($nextsnapshot) {
 					my $nextsnapshotescaped = escapeshellparam($nextsnapshot);
-
 					my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions -i $sourcefsescaped#$bookmarkescaped $sourcefsescaped\@$nextsnapshotescaped";
 					my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 					my $synccmd = buildsynccmd($sendcmd,$recvcmd,$pvsize,$sourceisroot,$targetisroot);
@@ -654,6 +658,7 @@ sub syncdataset {
 			# do a normal replication if bookmarks aren't used or if previous
 			# bookmark replication was only done to the next oldest snapshot
 			if (!$bookmark || $nextsnapshot) {
+				$sendoptions = getoptionsline(\@sendoptions, ('D','L','P','R','c','e','p','v','w'));
 				my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $args{'streamarg'} $sourcefsescaped\@$matchingsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
 				my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 				my $pvsize = getsendsize($sourcehost,"$sourcefs\@$matchingsnap","$sourcefs\@$newsyncsnap",$sourceisroot);
@@ -1398,6 +1403,12 @@ sub getsendsize {
 		$snaps = "-t $receivetoken";
 	}
 
+	my $sendoptions;
+	if (defined($receivetoken)) {
+		$sendoptions = getoptionsline(\@sendoptions, ('e', 'w'));
+	} else {
+		$sendoptions = getoptionsline(\@sendoptions, ('D','L','R','c','e','p','v','w'));
+	}
 	my $getsendsizecmd = "$sourcessh $mysudocmd $zfscmd send $sendoptions -nP $snaps";
 	if ($debug) { print "DEBUG: getting estimated transfer size from source $sourcehost using \"$getsendsizecmd 2>&1 |\"...\n"; }
 
@@ -1479,6 +1490,70 @@ sub getreceivetoken() {
 	return
 }
 
+sub parsespecialoptions {
+	my ($line) = @_;
+
+	print("$line\n");
+
+	my @options = ();
+
+	my @values = split(/ /, $line);
+
+	my $optionValue = 0;
+	my $lastOption;
+
+	foreach my $value (@values) {
+		if ($optionValue ne 0) {
+			my %item = (
+				"option"  => $lastOption,
+				"line" => "-$lastOption $value",
+			);
+
+			push @options, \%item;
+			$optionValue = 0;
+			next;
+		}
+
+		for my $char (split //, $value) {
+			if ($optionValue ne 0) {
+				return undef;
+			}
+
+			if ($char eq 'o' || $char eq 'x') {
+				$lastOption = $char;
+				$optionValue = 1;
+			} else {
+				my %item = (
+					"option"  => $char,
+					"line" => "-$char",
+				);
+
+				push @options, \%item;
+			}
+		}
+	}
+
+	return @options;
+}
+
+sub getoptionsline {
+	my ($options_ref, @allowed) = @_;
+
+	my $line = '';
+
+	foreach my $value (@{ $options_ref }) {
+		if (@allowed) {
+			if (!grep( /^$$value{'option'}$/, @allowed) ) {
+				next;
+			}
+		}
+
+		$line = "$line$$value{'line'} ";
+	}
+
+	return $line;
+}
+
 __END__
 
 =head1 NAME
@@ -1509,8 +1584,8 @@ Options:
   --no-clone-rollback   Does not rollback clones on target
   --no-rollback         Does not rollback clones or snapshots on target (it probably requires a readonly target)
   --exclude=REGEX       Exclude specific datasets which match the given regular expression. Can be specified multiple times
-  --sendoptions=OPTIONS  DANGER: Inject OPTIONS into zfs send, e.g. syncoid --sendoptions="-Lce" sets zfs send -Lce ...
-  --recvoptions=OPTIONS  DANGER: Inject OPTIONS into zfs received, e.g. syncoid --recvoptions="-x property" sets zfs receive -x property ...
+  --sendoptions=OPTIONS Use advanced options for zfs send (the arguments are filterd as needed), e.g. syncoid --sendoptions="Lc e" sets zfs send -L -c -e ...
+  --recvoptions=OPTIONS Use advanced options for zfs receive (the arguments are filterd as needed), e.g. syncoid --recvoptions="ux recordsize o compression=lz4" sets zfs receive -u -x recordsize -o compression=lz4 ...
   --sshkey=FILE         Specifies a ssh public key to use to connect
   --sshport=PORT        Connects to remote on a particular port
   --sshcipher|c=CIPHER  Passes CIPHER to ssh to use a particular cipher set

From f0fec5348e62dae3b8de70d49a312763a952294f Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Thu, 31 Jan 2019 18:25:41 +0100
Subject: [PATCH 34/62] fix root pool parsing for remote hosts

---
 syncoid | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/syncoid b/syncoid
index b12f059..e6710bb 100755
--- a/syncoid
+++ b/syncoid
@@ -887,9 +887,9 @@ sub checkcommands {
 
 	# check for ZFS resume feature support
 	if ($resume) {
-		my @parts = split ('/', $args{'source'});
+		my @parts = split ('/', $sourcefs);
 		my $srcpool = $parts[0];
-		@parts = split ('/', $args{'target'});
+		@parts = split ('/', $targetfs);
 		my $dstpool = $parts[0];
 
 		$srcpool = escapeshellparam($srcpool);

From 1053e9370e362575ed360261f3f55474c97cf9b7 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Fri, 1 Feb 2019 17:43:04 +0100
Subject: [PATCH 35/62] missed sudo command for bookmark creation if source
 host is remote

---
 syncoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index eee2d44..d6b5f1f 100755
--- a/syncoid
+++ b/syncoid
@@ -682,7 +682,7 @@ sub syncdataset {
 		if (defined $args{'create-bookmark'}) {
 			my $bookmarkcmd;
 			if ($sourcehost ne '') {
-				$bookmarkcmd = "$sshcmd $sourcehost " . escapeshellparam("$zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped");
+				$bookmarkcmd = "$sshcmd $sourcehost " . escapeshellparam("$sourcesudocmd $zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped");
 			} else {
 				$bookmarkcmd = "$sourcesudocmd $zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped";
 			}

From 7d522fa5901037f0fe780e07c8056cd87399eb52 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 4 Feb 2019 17:56:27 +0100
Subject: [PATCH 36/62] fix unitialized variable

---
 syncoid | 1 +
 1 file changed, 1 insertion(+)

diff --git a/syncoid b/syncoid
index 3f6f9ba..c0c1a3d 100755
--- a/syncoid
+++ b/syncoid
@@ -817,6 +817,7 @@ sub checkcommands {
 
 	if (!defined $avail{'sourcecompress'}) { $avail{'sourcecompress'} = ''; }
 	if (!defined $avail{'targetcompress'}) { $avail{'targetcompress'} = ''; }
+	if (!defined $avail{'localcompress'}) { $avail{'localcompress'} = ''; }
 	if (!defined $avail{'sourcembuffer'}) { $avail{'sourcembuffer'} = ''; }
 	if (!defined $avail{'targetmbuffer'}) { $avail{'targetmbuffer'} = ''; }
 

From 10b2012401bf8a3ae81007dc70f71ef6eef0f278 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 4 Feb 2019 18:08:01 +0100
Subject: [PATCH 37/62] removed debug output

---
 syncoid | 2 --
 1 file changed, 2 deletions(-)

diff --git a/syncoid b/syncoid
index b0a93af..50788e7 100755
--- a/syncoid
+++ b/syncoid
@@ -1493,8 +1493,6 @@ sub getreceivetoken() {
 sub parsespecialoptions {
 	my ($line) = @_;
 
-	print("$line\n");
-
 	my @options = ();
 
 	my @values = split(/ /, $line);

From 5812b57349c32b4068e1735a1f12b81382b2ff46 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 4 Feb 2019 18:19:44 +0100
Subject: [PATCH 38/62] check for invalid send/receive option syntax

---
 syncoid | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/syncoid b/syncoid
index 50788e7..df1fc11 100755
--- a/syncoid
+++ b/syncoid
@@ -30,11 +30,21 @@ my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be
 my @sendoptions = ();
 if (length $args{'sendoptions'}) {
 	@sendoptions = parsespecialoptions($args{'sendoptions'});
+	if (! defined($sendoptions[0])) {
+		warn "invalid send options!";
+		pod2usage(2);
+		exit 127;
+	}
 }
 
 my @recvoptions = ();
 if (length $args{'recvoptions'}) {
 	@recvoptions = parsespecialoptions($args{'recvoptions'});
+	if (! defined($recvoptions[0])) {
+		warn "invalid receive options!";
+		pod2usage(2);
+		exit 127;
+	}
 }
 
 

From 99b439e3833b7b56524efdc9c72ec72ba0e2191f Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 4 Feb 2019 18:24:19 +0100
Subject: [PATCH 39/62] return if there is nothing to do

---
 syncoid | 1 +
 1 file changed, 1 insertion(+)

diff --git a/syncoid b/syncoid
index d6b5f1f..147dc80 100755
--- a/syncoid
+++ b/syncoid
@@ -581,6 +581,7 @@ sub syncdataset {
 		if ($matchingsnap eq $newsyncsnap) {
 			# barf some text but don't touch the filesystem
 			if (!$quiet) { print "INFO: no snapshots on source newer than $newsyncsnap on target. Nothing to do, not syncing.\n"; }
+			return 0;
 		} else {
 			my $matchingsnapescaped = escapeshellparam($matchingsnap);
 			# rollback target to matchingsnap

From a748b41f09cbd24684fb2351ae0239cb476e960a Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Mon, 4 Feb 2019 18:38:09 +0100
Subject: [PATCH 40/62] if bookmark creation fails use guid bases suffix based
 fallback name

---
 syncoid | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/syncoid b/syncoid
index 147dc80..c8c5262 100755
--- a/syncoid
+++ b/syncoid
@@ -689,9 +689,23 @@ sub syncdataset {
 			}
 			if ($debug) { print "DEBUG: $bookmarkcmd\n"; }
 			system($bookmarkcmd) == 0 or do {
-				warn "CRITICAL ERROR: $bookmarkcmd failed: $?";
-				if ($exitcode < 2) { $exitcode = 2; }
-				return 0;
+				# fallback: assume nameing conflict and try again with guid based suffix
+				my $guid = $snaps{'source'}{$newsyncsnap}{'guid'};
+				$guid = substr($guid, 0, 6);
+
+				if (!$quiet) { print "INFO: bookmark creation failed, retrying with guid based suffix ($guid)...\n"; }
+
+				if ($sourcehost ne '') {
+					$bookmarkcmd = "$sshcmd $sourcehost " . escapeshellparam("$sourcesudocmd $zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped$guid");
+				} else {
+					$bookmarkcmd = "$sourcesudocmd $zfscmd bookmark $sourcefsescaped\@$newsyncsnapescaped $sourcefsescaped\#$newsyncsnapescaped$guid";
+				}
+				if ($debug) { print "DEBUG: $bookmarkcmd\n"; }
+				system($bookmarkcmd) == 0 or do {
+					warn "CRITICAL ERROR: $bookmarkcmd failed: $?";
+					if ($exitcode < 2) { $exitcode = 2; }
+					return 0;
+				}
 			};
 		}
 	} else {

From d3f19c839782a112c03bed4d18cb13eff66d91a4 Mon Sep 17 00:00:00 2001
From: Simon Alman <haven@thehavennet.org.uk>
Date: Sun, 17 Feb 2019 07:17:03 +0000
Subject: [PATCH 41/62] Sanoid ebuild for Gentoo

---
 packages/gentoo/sys-fs/Manifest            |  4 +++
 packages/gentoo/sys-fs/files/sanoid.cron   |  1 +
 packages/gentoo/sys-fs/sanoid-2.0.1.ebuild | 36 ++++++++++++++++++++
 packages/gentoo/sys-fs/sanoid-9999.ebuild  | 38 ++++++++++++++++++++++
 4 files changed, 79 insertions(+)
 create mode 100644 packages/gentoo/sys-fs/Manifest
 create mode 100644 packages/gentoo/sys-fs/files/sanoid.cron
 create mode 100644 packages/gentoo/sys-fs/sanoid-2.0.1.ebuild
 create mode 100644 packages/gentoo/sys-fs/sanoid-9999.ebuild

diff --git a/packages/gentoo/sys-fs/Manifest b/packages/gentoo/sys-fs/Manifest
new file mode 100644
index 0000000..4c629f9
--- /dev/null
+++ b/packages/gentoo/sys-fs/Manifest
@@ -0,0 +1,4 @@
+AUX sanoid.cron 45 BLAKE2B 3f6294bbbf485dc21a565cd2c8da05a42fb21cdaabdf872a21500f1a7338786c60d4a1fd188bbf81ce85f06a376db16998740996f47c049707a5109bdf02c052 SHA512 7676b32f21e517e8c84a097c7934b54097cf2122852098ea756093ece242125da3f6ca756a6fbb82fc348f84b94bfd61639e86e0bfa4bbe7abf94a8a4c551419
+DIST sanoid-2.0.1.tar.gz 106981 BLAKE2B 824b7271266ac9f9bf1fef5374a442215c20a4f139081f77d5d8db2ec7db9b8b349d9d0394c76f9d421a957853af64ff069097243f69e7e4b83a804f5ba992a6 SHA512 9d999b0f071bc3c3ca956df11e1501fd72a842f7d3315ede3ab3b5e0a36351100b6edbab8448bba65a2e187e4e8f77ff24671ed33b28f2fca9bb6ad0801aba9d
+EBUILD sanoid-2.0.1.ebuild 772 BLAKE2B befbc479b5c79faa88ae21649ed31d1af70dbecb60416e8c879fffd9a3cdf9f3f508e12d8edc9f4e0afbf0e6ab0491a36fdae2af995a1984072dc5bffd63fe1d SHA512 d90a8b8ae40634e2f2e1fa11ba787cfcb461b75fa65b19c0d9a34eb458f07f510bbb1992f4a0e7a0e4aa5f55a5acdc064779c9a4f993b30eb5cbf39037f97858
+EBUILD sanoid-9999.ebuild 752 BLAKE2B 073533436c6f5c47b9e8410c898bf86b605d61c9b16a08b57253f5a87ad583e00d935ae9ea90f98b42c20dc1fbda0b9f1a8a7bf5be1cf3daf20afc640f1428ca SHA512 40ad34230fdb538bbdcda2d8149f37eac2a0e2accce5f79f7ba77d8e62e3fd78e997d8143baa0e050f548f90ce1cb6827e50b536b5e3acc444c6032f170251be
diff --git a/packages/gentoo/sys-fs/files/sanoid.cron b/packages/gentoo/sys-fs/files/sanoid.cron
new file mode 100644
index 0000000..09169ad
--- /dev/null
+++ b/packages/gentoo/sys-fs/files/sanoid.cron
@@ -0,0 +1 @@
+* * * * * root TZ=UTC /usr/bin/sanoid --cron
diff --git a/packages/gentoo/sys-fs/sanoid-2.0.1.ebuild b/packages/gentoo/sys-fs/sanoid-2.0.1.ebuild
new file mode 100644
index 0000000..5a8d67e
--- /dev/null
+++ b/packages/gentoo/sys-fs/sanoid-2.0.1.ebuild
@@ -0,0 +1,36 @@
+# Copyright 2019 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=7
+
+DESCRIPTION="Policy-driven snapshot management and replication tools for ZFS"
+HOMEPAGE="https://github.com/jimsalterjrs/sanoid"
+SRC_URI="https://github.com/jimsalterjrs/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
+
+LICENSE="GPL-3.0"
+SLOT="0"
+KEYWORDS="~x86 ~amd64"
+IUSE=""
+
+DEPEND="app-arch/lzop
+		dev-perl/Config-IniFiles
+		sys-apps/pv
+		sys-block/mbuffer
+		virtual/perl-Data-Dumper"
+RDEPEND="${DEPEND}"
+BDEPEND=""
+
+DOCS=( README.md )
+
+src_install() {
+	dobin findoid
+	dobin sanoid
+	dobin sleepymutex
+	dobin syncoid
+	keepdir /etc/${PN}
+	insinto /etc/${PN}
+	doins sanoid.conf
+	doins sanoid.defaults.conf
+	insinto /etc/cron.d
+	newins "${FILESDIR}/${PN}.cron" ${PN}
+}
diff --git a/packages/gentoo/sys-fs/sanoid-9999.ebuild b/packages/gentoo/sys-fs/sanoid-9999.ebuild
new file mode 100644
index 0000000..7eaf509
--- /dev/null
+++ b/packages/gentoo/sys-fs/sanoid-9999.ebuild
@@ -0,0 +1,38 @@
+# Copyright 2019 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=7
+
+EGIT_REPO_URI="https://github.com/jimsalterjrs/${PN}.git"
+inherit git-r3
+
+DESCRIPTION="Policy-driven snapshot management and replication tools for ZFS"
+HOMEPAGE="https://github.com/jimsalterjrs/sanoid"
+
+LICENSE="GPL-3.0"
+SLOT="0"
+KEYWORDS="**"
+IUSE=""
+
+DEPEND="app-arch/lzop
+		dev-perl/Config-IniFiles
+		sys-apps/pv
+		sys-block/mbuffer
+		virtual/perl-Data-Dumper"
+RDEPEND="${DEPEND}"
+BDEPEND=""
+
+DOCS=( README.md )
+
+src_install() {
+	dobin findoid
+	dobin sanoid
+	dobin sleepymutex
+	dobin syncoid
+	keepdir /etc/${PN}
+	insinto /etc/${PN}
+	doins sanoid.conf
+	doins sanoid.defaults.conf
+	insinto /etc/cron.d
+	newins "${FILESDIR}/${PN}.cron" ${PN}
+}

From c543f9aa0c13c037d402d890c6ac67b5d21d2983 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Tue, 19 Feb 2019 16:00:46 +0100
Subject: [PATCH 42/62] use ordinary replication if clone recreation fails

---
 syncoid | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/syncoid b/syncoid
index 3f6f9ba..d974f15 100755
--- a/syncoid
+++ b/syncoid
@@ -416,6 +416,12 @@ sub syncdataset {
 			return 0;
 		}
 		system($synccmd) == 0 or do {
+			if (defined $origin) {
+				print "INFO: clone creation failed, trying ordinary replication as fallback\n";
+				syncdataset($sourcehost, $sourcefs, $targethost, $targetfs, undef, 1);
+				return 0;
+			}
+
 			warn "CRITICAL ERROR: $synccmd failed: $?";
 			if ($exitcode < 2) { $exitcode = 2; }
 			return 0;

From 25c5e2e24231c1b963e36f96c75375551cac5e12 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Tue, 19 Feb 2019 23:27:52 +0100
Subject: [PATCH 43/62] allow new zfs send/recv hold parameter (introduced in
 recent master) and drop invalid raw parameter for send size estimation for
 resumeable send

---
 syncoid | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/syncoid b/syncoid
index df1fc11..c20825e 100755
--- a/syncoid
+++ b/syncoid
@@ -362,8 +362,8 @@ sub syncdataset {
 		# with ZFS on Linux (possibly OpenZFS in general) when setting/unsetting readonly.
 	#my $originaltargetreadonly;
 
-	my $sendoptions = getoptionsline(\@sendoptions, ('D','L','P','R','c','e','p','v','w'));
-	my $recvoptions = getoptionsline(\@recvoptions, ('o','x','u','v'));
+	my $sendoptions = getoptionsline(\@sendoptions, ('D','L','P','R','c','e','h','p','v','w'));
+	my $recvoptions = getoptionsline(\@recvoptions, ('h','o','x','u','v'));
 
 	# sync 'em up.
 	if (! $targetexists) {
@@ -668,7 +668,7 @@ sub syncdataset {
 			# do a normal replication if bookmarks aren't used or if previous
 			# bookmark replication was only done to the next oldest snapshot
 			if (!$bookmark || $nextsnapshot) {
-				$sendoptions = getoptionsline(\@sendoptions, ('D','L','P','R','c','e','p','v','w'));
+				$sendoptions = getoptionsline(\@sendoptions, ('D','L','P','R','c','e','h','p','v','w'));
 				my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $args{'streamarg'} $sourcefsescaped\@$matchingsnapescaped $sourcefsescaped\@$newsyncsnapescaped";
 				my $recvcmd = "$targetsudocmd $zfscmd receive $recvoptions $receiveextraargs $forcedrecv $targetfsescaped";
 				my $pvsize = getsendsize($sourcehost,"$sourcefs\@$matchingsnap","$sourcefs\@$newsyncsnap",$sourceisroot);
@@ -1415,9 +1415,9 @@ sub getsendsize {
 
 	my $sendoptions;
 	if (defined($receivetoken)) {
-		$sendoptions = getoptionsline(\@sendoptions, ('e', 'w'));
+		$sendoptions = getoptionsline(\@sendoptions, ('e'));
 	} else {
-		$sendoptions = getoptionsline(\@sendoptions, ('D','L','R','c','e','p','v','w'));
+		$sendoptions = getoptionsline(\@sendoptions, ('D','L','R','c','e','h','p','v','w'));
 	}
 	my $getsendsizecmd = "$sourcessh $mysudocmd $zfscmd send $sendoptions -nP $snaps";
 	if ($debug) { print "DEBUG: getting estimated transfer size from source $sourcehost using \"$getsendsizecmd 2>&1 |\"...\n"; }

From eefc659c0396a90b6afa5524c3b903092771e29c Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 20 Feb 2019 08:15:45 +0100
Subject: [PATCH 44/62] update the install instruction with recent changes

---
 INSTALL.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 47 insertions(+), 8 deletions(-)

diff --git a/INSTALL.md b/INSTALL.md
index 94a9f2e..3a941a6 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -5,7 +5,7 @@
 <!-- TOC depthFrom:1 depthTo:6 withLinks:1 updateOnSave:0 orderedList:0 -->
 
 - [Installation](#installation)
-	- [Ubuntu](#ubuntu)
+	- [Debian/Ubuntu](#debianubuntu)
 	- [CentOS](#centos)
 	- [FreeBSD](#freebsd)
 	- [Other OSes](#other-oses)
@@ -15,7 +15,7 @@
 <!-- /TOC -->
 
 
-## Ubuntu
+## Debian/Ubuntu
 
 Install prerequisite software:
 
@@ -23,6 +23,24 @@ Install prerequisite software:
 apt install libconfig-inifiles-perl pv lzop mbuffer
 ```
 
+Clone this repo, build the debian package and install it (alternatively you can skip the package and do it manually like described below for CentOS):
+
+```bash
+# Download the repo as root to avoid changing permissions later
+sudo git clone https://github.com/jimsalterjrs/sanoid.git
+cd sanoid
+ln -s packages/debian .
+dpkg-buildpackage -uc -us
+apt install ../sanoid_*_all.deb
+```
+
+Enable sanoid timer:
+```bash
+# enable and start the sanoid timer
+sudo systemctl enable sanoid.timer
+sudo systemctl start sanoid.timer
+```
+
 ## CentOS
 
 Install prerequisite software:
@@ -60,23 +78,42 @@ cat << "EOF" | sudo tee /etc/systemd/system/sanoid.service
 Description=Snapshot ZFS Pool
 Requires=zfs.target
 After=zfs.target
+ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 
 [Service]
+Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/sbin/sanoid --cron
+ExecStart=/usr/sbin/sanoid --take-snapshots
+EOF
+
+cat << "EOF" | sudo tee /etc/systemd/system/sanoid-prune.service
+[Unit]
+Description=Cleanup ZFS Pool
+Requires=zfs.target
+After=zfs.target sanoid.service
+ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
+
+[Service]
+Environment=TZ=UTC
+Type=oneshot
+ExecStart=/usr/sbin/sanoid --prune-snapshots
+
+[Install]
+WantedBy=sanoid.service
 EOF
 ```
 
-And a systemd timer that will execute **Sanoid** once per minute:
+And a systemd timer that will execute **Sanoid** once per quarter hour
+(Decrease the interval as suitable for configuration):
 
 ```bash
 cat << "EOF" | sudo tee /etc/systemd/system/sanoid.timer
 [Unit]
-Description=Run Sanoid Every Minute
+Description=Run Sanoid Every 15 Minutes
 Requires=sanoid.service
 
 [Timer]
-OnCalendar=*:0/1
+OnCalendar=*:0/15
 Persistent=true
 
 [Install]
@@ -100,7 +137,7 @@ Now, proceed to configure [**Sanoid**](#configuration)
 Install prerequisite software:
 
 ```bash
-pkg install p5-Config-Inifiles pv lzop
+pkg install p5-Config-Inifiles pv mbuffer lzop
 ```
 
 **Additional notes:**
@@ -109,6 +146,8 @@ pkg install p5-Config-Inifiles pv lzop
 
 *   Simplest path workaround is symlinks, eg `ln -s /usr/local/bin/lzop /usr/bin/lzop` or similar, as appropriate to create links in **/usr/bin** to wherever the utilities actually are on your system.
 
+*   See note about mbuffer and other things in FREEBSD.readme
+
 ## Other OSes
 
 **Sanoid** depends on the Perl module Config::IniFiles and will not operate without it. Config::IniFiles may be installed from CPAN, though the project strongly recommends using your distribution's repositories instead.
@@ -130,4 +169,4 @@ pkg install p5-Config-Inifiles pv lzop
 
 ## Sanoid
 
-Instructions on how to set up `sanoid.conf`.  Maybe just copy/paste the example `sanoid.conf` file in here but clean it up a little bit.
+Take a look at the files `sanoid.defaults.conf` and` sanoid.conf.example` for all possible configuration options. Also have a look at the README.md

From e014fd79423129b7913d71b48a5f40ccab030edb Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 20 Feb 2019 18:06:04 +0100
Subject: [PATCH 45/62] check if ssh connection works

---
 syncoid | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/syncoid b/syncoid
index 3f6f9ba..6c98ff8 100755
--- a/syncoid
+++ b/syncoid
@@ -1247,6 +1247,13 @@ sub getssh {
 		$socket = "/tmp/syncoid-$remoteuser-$rhost-" . time();
 		open FH, "$sshcmd -M -S $socket -o ControlPersist=1m $args{'sshport'} $rhost exit |";
 		close FH;
+
+		system("$sshcmd -S $socket $rhost echo -n") == 0 or do {
+			my $code = $? >> 8;
+			warn "CRITICAL ERROR: ssh connection echo test failed for $rhost with exit code $code";
+			exit(2);
+		};
+
 		$rhost = "-S $socket $rhost";
 	} else {
 		my $localuid = $<;

From a97da7d0d2c0b12869b492423b748e653d7a3e98 Mon Sep 17 00:00:00 2001
From: Klemens Ullmann-Marx <klemens@ull.at>
Date: Thu, 28 Feb 2019 09:07:03 +0100
Subject: [PATCH 46/62] Added bwlimit units

---
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index a8cdfc7..70ded8f 100644
--- a/README.md
+++ b/README.md
@@ -184,11 +184,11 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
 
 + --source-bwlimit <limit t|g|m|k>
 
-	This is the bandwidth limit imposed upon the source. This is mainly used if the target does not have mbuffer installed, but bandwidth limites are desired.
+	This is the bandwidth limit in bytes per second imposed upon the source. This is mainly used if the target does not have mbuffer installed, but bandwidth limites are desired. 
 
 + --target-bw-limit <limit t|g|m|k>
 
-	This is the bandwidth limit imposed upon the target. This is mainly used if the source does not have mbuffer installed, but bandwidth limites are desired.
+	This is the bandwidth limit in bytes per second imposed upon the target. This is mainly used if the source does not have mbuffer installed, but bandwidth limites are desired.
 
 + --no-command-checks
 

From e0e7b0d2aed1ae33dc9b1b5ee42975a6647348d2 Mon Sep 17 00:00:00 2001
From: Klemens Ullmann-Marx <klemens@ull.at>
Date: Thu, 28 Feb 2019 09:09:24 +0100
Subject: [PATCH 47/62] added bwlimit units

---
 syncoid | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/syncoid b/syncoid
index 3f6f9ba..e4c81db 100755
--- a/syncoid
+++ b/syncoid
@@ -1501,8 +1501,8 @@ Options:
   --identifier=EXTRA    Extra identifier which is included in the snapshot name. Can be used for replicating to multiple targets.
   --recursive|r         Also transfers child datasets
   --skip-parent         Skips syncing of the parent dataset. Does nothing without '--recursive' option.
-  --source-bwlimit=<limit k|m|g|t>  Bandwidth limit on the source transfer
-  --target-bwlimit=<limit k|m|g|t>  Bandwidth limit on the target transfer
+  --source-bwlimit=<limit k|m|g|t>  Bandwidth limit in bytes per second on the source transfer
+  --target-bwlimit=<limit k|m|g|t>  Bandwidth limit in bytes per second on the target transfer
   --mbuffer-size=VALUE  Specify the mbuffer size (default: 16M), please refer to mbuffer(1) manual page.
   --no-stream           Replicates using newest snapshot instead of intermediates
   --no-sync-snap        Does not create new snapshot, only transfers existing

From 736d0319792fe9d1b55ff51aad920368b97af686 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Thu, 14 Mar 2019 17:35:58 +0100
Subject: [PATCH 48/62] initial replication without intermediate snapshots
 should use the created sync snapshot

---
 syncoid | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index 3f6f9ba..f01a1c4 100755
--- a/syncoid
+++ b/syncoid
@@ -376,7 +376,13 @@ sub syncdataset {
 		}
 
 		# if --no-stream is specified, our full needs to be the newest snapshot, not the oldest.
-		if (defined $args{'no-stream'}) { $oldestsnap = getnewestsnapshot(\%snaps); }
+		if (defined $args{'no-stream'}) {
+			if (defined ($args{'no-sync-snap'}) ) {
+				$oldestsnap = getnewestsnapshot(\%snaps);
+			} else {
+				$oldestsnap = $newsyncsnap;
+			}
+		}
 		my $oldestsnapescaped = escapeshellparam($oldestsnap);
 
 		my $sendcmd = "$sourcesudocmd $zfscmd send $sendoptions $sourcefsescaped\@$oldestsnapescaped";
@@ -1199,6 +1205,7 @@ sub newsyncsnap {
 	my %date = getdate();
 	my $snapname = "syncoid\_$identifier$hostid\_$date{'stamp'}";
 	my $snapcmd = "$rhost $mysudocmd $zfscmd snapshot $fsescaped\@$snapname\n";
+	if ($debug) { print "DEBUG: creating sync snapshot using \"$snapcmd\"...\n"; }
 	system($snapcmd) == 0 or do {
 		warn "CRITICAL ERROR: $snapcmd failed: $?";
 		if ($exitcode < 2) { $exitcode = 2; }

From df1e344678e9879af33068a2ed2a5f21b20b74a6 Mon Sep 17 00:00:00 2001
From: Simon Alman <haven@thehavennet.org.uk>
Date: Fri, 29 Mar 2019 10:27:21 +0000
Subject: [PATCH 49/62] Fix gentoo package snafu - need a sanoid directory to
 store the files

---
 packages/gentoo/sys-fs/sanoid/Manifest        |  4 ++
 .../gentoo/sys-fs/sanoid/files/sanoid.cron    |  1 +
 .../gentoo/sys-fs/sanoid/sanoid-2.0.1.ebuild  | 36 ++++++++++++++++++
 .../gentoo/sys-fs/sanoid/sanoid-9999.ebuild   | 38 +++++++++++++++++++
 4 files changed, 79 insertions(+)
 create mode 100644 packages/gentoo/sys-fs/sanoid/Manifest
 create mode 100644 packages/gentoo/sys-fs/sanoid/files/sanoid.cron
 create mode 100644 packages/gentoo/sys-fs/sanoid/sanoid-2.0.1.ebuild
 create mode 100644 packages/gentoo/sys-fs/sanoid/sanoid-9999.ebuild

diff --git a/packages/gentoo/sys-fs/sanoid/Manifest b/packages/gentoo/sys-fs/sanoid/Manifest
new file mode 100644
index 0000000..4c629f9
--- /dev/null
+++ b/packages/gentoo/sys-fs/sanoid/Manifest
@@ -0,0 +1,4 @@
+AUX sanoid.cron 45 BLAKE2B 3f6294bbbf485dc21a565cd2c8da05a42fb21cdaabdf872a21500f1a7338786c60d4a1fd188bbf81ce85f06a376db16998740996f47c049707a5109bdf02c052 SHA512 7676b32f21e517e8c84a097c7934b54097cf2122852098ea756093ece242125da3f6ca756a6fbb82fc348f84b94bfd61639e86e0bfa4bbe7abf94a8a4c551419
+DIST sanoid-2.0.1.tar.gz 106981 BLAKE2B 824b7271266ac9f9bf1fef5374a442215c20a4f139081f77d5d8db2ec7db9b8b349d9d0394c76f9d421a957853af64ff069097243f69e7e4b83a804f5ba992a6 SHA512 9d999b0f071bc3c3ca956df11e1501fd72a842f7d3315ede3ab3b5e0a36351100b6edbab8448bba65a2e187e4e8f77ff24671ed33b28f2fca9bb6ad0801aba9d
+EBUILD sanoid-2.0.1.ebuild 772 BLAKE2B befbc479b5c79faa88ae21649ed31d1af70dbecb60416e8c879fffd9a3cdf9f3f508e12d8edc9f4e0afbf0e6ab0491a36fdae2af995a1984072dc5bffd63fe1d SHA512 d90a8b8ae40634e2f2e1fa11ba787cfcb461b75fa65b19c0d9a34eb458f07f510bbb1992f4a0e7a0e4aa5f55a5acdc064779c9a4f993b30eb5cbf39037f97858
+EBUILD sanoid-9999.ebuild 752 BLAKE2B 073533436c6f5c47b9e8410c898bf86b605d61c9b16a08b57253f5a87ad583e00d935ae9ea90f98b42c20dc1fbda0b9f1a8a7bf5be1cf3daf20afc640f1428ca SHA512 40ad34230fdb538bbdcda2d8149f37eac2a0e2accce5f79f7ba77d8e62e3fd78e997d8143baa0e050f548f90ce1cb6827e50b536b5e3acc444c6032f170251be
diff --git a/packages/gentoo/sys-fs/sanoid/files/sanoid.cron b/packages/gentoo/sys-fs/sanoid/files/sanoid.cron
new file mode 100644
index 0000000..09169ad
--- /dev/null
+++ b/packages/gentoo/sys-fs/sanoid/files/sanoid.cron
@@ -0,0 +1 @@
+* * * * * root TZ=UTC /usr/bin/sanoid --cron
diff --git a/packages/gentoo/sys-fs/sanoid/sanoid-2.0.1.ebuild b/packages/gentoo/sys-fs/sanoid/sanoid-2.0.1.ebuild
new file mode 100644
index 0000000..5a8d67e
--- /dev/null
+++ b/packages/gentoo/sys-fs/sanoid/sanoid-2.0.1.ebuild
@@ -0,0 +1,36 @@
+# Copyright 2019 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=7
+
+DESCRIPTION="Policy-driven snapshot management and replication tools for ZFS"
+HOMEPAGE="https://github.com/jimsalterjrs/sanoid"
+SRC_URI="https://github.com/jimsalterjrs/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
+
+LICENSE="GPL-3.0"
+SLOT="0"
+KEYWORDS="~x86 ~amd64"
+IUSE=""
+
+DEPEND="app-arch/lzop
+		dev-perl/Config-IniFiles
+		sys-apps/pv
+		sys-block/mbuffer
+		virtual/perl-Data-Dumper"
+RDEPEND="${DEPEND}"
+BDEPEND=""
+
+DOCS=( README.md )
+
+src_install() {
+	dobin findoid
+	dobin sanoid
+	dobin sleepymutex
+	dobin syncoid
+	keepdir /etc/${PN}
+	insinto /etc/${PN}
+	doins sanoid.conf
+	doins sanoid.defaults.conf
+	insinto /etc/cron.d
+	newins "${FILESDIR}/${PN}.cron" ${PN}
+}
diff --git a/packages/gentoo/sys-fs/sanoid/sanoid-9999.ebuild b/packages/gentoo/sys-fs/sanoid/sanoid-9999.ebuild
new file mode 100644
index 0000000..7eaf509
--- /dev/null
+++ b/packages/gentoo/sys-fs/sanoid/sanoid-9999.ebuild
@@ -0,0 +1,38 @@
+# Copyright 2019 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=7
+
+EGIT_REPO_URI="https://github.com/jimsalterjrs/${PN}.git"
+inherit git-r3
+
+DESCRIPTION="Policy-driven snapshot management and replication tools for ZFS"
+HOMEPAGE="https://github.com/jimsalterjrs/sanoid"
+
+LICENSE="GPL-3.0"
+SLOT="0"
+KEYWORDS="**"
+IUSE=""
+
+DEPEND="app-arch/lzop
+		dev-perl/Config-IniFiles
+		sys-apps/pv
+		sys-block/mbuffer
+		virtual/perl-Data-Dumper"
+RDEPEND="${DEPEND}"
+BDEPEND=""
+
+DOCS=( README.md )
+
+src_install() {
+	dobin findoid
+	dobin sanoid
+	dobin sleepymutex
+	dobin syncoid
+	keepdir /etc/${PN}
+	insinto /etc/${PN}
+	doins sanoid.conf
+	doins sanoid.defaults.conf
+	insinto /etc/cron.d
+	newins "${FILESDIR}/${PN}.cron" ${PN}
+}

From ffa6ad32e5ec3c6841dfbc86ee3b305132bab605 Mon Sep 17 00:00:00 2001
From: Simon Alman <haven@thehavennet.org.uk>
Date: Fri, 29 Mar 2019 10:28:17 +0000
Subject: [PATCH 50/62] Fix gentoo package snafu - need a sanoid directory to
 store the files

---
 packages/gentoo/sys-fs/Manifest            |  4 ---
 packages/gentoo/sys-fs/files/sanoid.cron   |  1 -
 packages/gentoo/sys-fs/sanoid-2.0.1.ebuild | 36 --------------------
 packages/gentoo/sys-fs/sanoid-9999.ebuild  | 38 ----------------------
 4 files changed, 79 deletions(-)
 delete mode 100644 packages/gentoo/sys-fs/Manifest
 delete mode 100644 packages/gentoo/sys-fs/files/sanoid.cron
 delete mode 100644 packages/gentoo/sys-fs/sanoid-2.0.1.ebuild
 delete mode 100644 packages/gentoo/sys-fs/sanoid-9999.ebuild

diff --git a/packages/gentoo/sys-fs/Manifest b/packages/gentoo/sys-fs/Manifest
deleted file mode 100644
index 4c629f9..0000000
--- a/packages/gentoo/sys-fs/Manifest
+++ /dev/null
@@ -1,4 +0,0 @@
-AUX sanoid.cron 45 BLAKE2B 3f6294bbbf485dc21a565cd2c8da05a42fb21cdaabdf872a21500f1a7338786c60d4a1fd188bbf81ce85f06a376db16998740996f47c049707a5109bdf02c052 SHA512 7676b32f21e517e8c84a097c7934b54097cf2122852098ea756093ece242125da3f6ca756a6fbb82fc348f84b94bfd61639e86e0bfa4bbe7abf94a8a4c551419
-DIST sanoid-2.0.1.tar.gz 106981 BLAKE2B 824b7271266ac9f9bf1fef5374a442215c20a4f139081f77d5d8db2ec7db9b8b349d9d0394c76f9d421a957853af64ff069097243f69e7e4b83a804f5ba992a6 SHA512 9d999b0f071bc3c3ca956df11e1501fd72a842f7d3315ede3ab3b5e0a36351100b6edbab8448bba65a2e187e4e8f77ff24671ed33b28f2fca9bb6ad0801aba9d
-EBUILD sanoid-2.0.1.ebuild 772 BLAKE2B befbc479b5c79faa88ae21649ed31d1af70dbecb60416e8c879fffd9a3cdf9f3f508e12d8edc9f4e0afbf0e6ab0491a36fdae2af995a1984072dc5bffd63fe1d SHA512 d90a8b8ae40634e2f2e1fa11ba787cfcb461b75fa65b19c0d9a34eb458f07f510bbb1992f4a0e7a0e4aa5f55a5acdc064779c9a4f993b30eb5cbf39037f97858
-EBUILD sanoid-9999.ebuild 752 BLAKE2B 073533436c6f5c47b9e8410c898bf86b605d61c9b16a08b57253f5a87ad583e00d935ae9ea90f98b42c20dc1fbda0b9f1a8a7bf5be1cf3daf20afc640f1428ca SHA512 40ad34230fdb538bbdcda2d8149f37eac2a0e2accce5f79f7ba77d8e62e3fd78e997d8143baa0e050f548f90ce1cb6827e50b536b5e3acc444c6032f170251be
diff --git a/packages/gentoo/sys-fs/files/sanoid.cron b/packages/gentoo/sys-fs/files/sanoid.cron
deleted file mode 100644
index 09169ad..0000000
--- a/packages/gentoo/sys-fs/files/sanoid.cron
+++ /dev/null
@@ -1 +0,0 @@
-* * * * * root TZ=UTC /usr/bin/sanoid --cron
diff --git a/packages/gentoo/sys-fs/sanoid-2.0.1.ebuild b/packages/gentoo/sys-fs/sanoid-2.0.1.ebuild
deleted file mode 100644
index 5a8d67e..0000000
--- a/packages/gentoo/sys-fs/sanoid-2.0.1.ebuild
+++ /dev/null
@@ -1,36 +0,0 @@
-# Copyright 2019 Gentoo Authors
-# Distributed under the terms of the GNU General Public License v2
-
-EAPI=7
-
-DESCRIPTION="Policy-driven snapshot management and replication tools for ZFS"
-HOMEPAGE="https://github.com/jimsalterjrs/sanoid"
-SRC_URI="https://github.com/jimsalterjrs/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
-
-LICENSE="GPL-3.0"
-SLOT="0"
-KEYWORDS="~x86 ~amd64"
-IUSE=""
-
-DEPEND="app-arch/lzop
-		dev-perl/Config-IniFiles
-		sys-apps/pv
-		sys-block/mbuffer
-		virtual/perl-Data-Dumper"
-RDEPEND="${DEPEND}"
-BDEPEND=""
-
-DOCS=( README.md )
-
-src_install() {
-	dobin findoid
-	dobin sanoid
-	dobin sleepymutex
-	dobin syncoid
-	keepdir /etc/${PN}
-	insinto /etc/${PN}
-	doins sanoid.conf
-	doins sanoid.defaults.conf
-	insinto /etc/cron.d
-	newins "${FILESDIR}/${PN}.cron" ${PN}
-}
diff --git a/packages/gentoo/sys-fs/sanoid-9999.ebuild b/packages/gentoo/sys-fs/sanoid-9999.ebuild
deleted file mode 100644
index 7eaf509..0000000
--- a/packages/gentoo/sys-fs/sanoid-9999.ebuild
+++ /dev/null
@@ -1,38 +0,0 @@
-# Copyright 2019 Gentoo Authors
-# Distributed under the terms of the GNU General Public License v2
-
-EAPI=7
-
-EGIT_REPO_URI="https://github.com/jimsalterjrs/${PN}.git"
-inherit git-r3
-
-DESCRIPTION="Policy-driven snapshot management and replication tools for ZFS"
-HOMEPAGE="https://github.com/jimsalterjrs/sanoid"
-
-LICENSE="GPL-3.0"
-SLOT="0"
-KEYWORDS="**"
-IUSE=""
-
-DEPEND="app-arch/lzop
-		dev-perl/Config-IniFiles
-		sys-apps/pv
-		sys-block/mbuffer
-		virtual/perl-Data-Dumper"
-RDEPEND="${DEPEND}"
-BDEPEND=""
-
-DOCS=( README.md )
-
-src_install() {
-	dobin findoid
-	dobin sanoid
-	dobin sleepymutex
-	dobin syncoid
-	keepdir /etc/${PN}
-	insinto /etc/${PN}
-	doins sanoid.conf
-	doins sanoid.defaults.conf
-	insinto /etc/cron.d
-	newins "${FILESDIR}/${PN}.cron" ${PN}
-}

From 681aa615c1f72a5a71f65fc88836c3ceb8b265a0 Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Fri, 29 Mar 2019 11:45:38 -0400
Subject: [PATCH 51/62] clarify bandwidth limit units

---
 syncoid | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/syncoid b/syncoid
index e4c81db..156c694 100755
--- a/syncoid
+++ b/syncoid
@@ -1501,8 +1501,8 @@ Options:
   --identifier=EXTRA    Extra identifier which is included in the snapshot name. Can be used for replicating to multiple targets.
   --recursive|r         Also transfers child datasets
   --skip-parent         Skips syncing of the parent dataset. Does nothing without '--recursive' option.
-  --source-bwlimit=<limit k|m|g|t>  Bandwidth limit in bytes per second on the source transfer
-  --target-bwlimit=<limit k|m|g|t>  Bandwidth limit in bytes per second on the target transfer
+  --source-bwlimit=<limit k|m|g|t>  Bandwidth limit in bytes/kbytes/etc per second on the source transfer
+  --target-bwlimit=<limit k|m|g|t>  Bandwidth limit in bytes/kbytes/etc per second on the target transfer
   --mbuffer-size=VALUE  Specify the mbuffer size (default: 16M), please refer to mbuffer(1) manual page.
   --no-stream           Replicates using newest snapshot instead of intermediates
   --no-sync-snap        Does not create new snapshot, only transfers existing

From 56d7c59bd9748758092dde1cccd5545d9c9447eb Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Fri, 29 Mar 2019 11:47:24 -0400
Subject: [PATCH 52/62] clarify --bwlimit units

---
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index 70ded8f..1c6c9c5 100644
--- a/README.md
+++ b/README.md
@@ -184,11 +184,11 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
 
 + --source-bwlimit <limit t|g|m|k>
 
-	This is the bandwidth limit in bytes per second imposed upon the source. This is mainly used if the target does not have mbuffer installed, but bandwidth limites are desired. 
+	This is the bandwidth limit in bytes (kbytes, mbytes, etc) per second imposed upon the source. This is mainly used if the target does not have mbuffer installed, but bandwidth limits are desired. 
 
 + --target-bw-limit <limit t|g|m|k>
 
-	This is the bandwidth limit in bytes per second imposed upon the target. This is mainly used if the source does not have mbuffer installed, but bandwidth limites are desired.
+	This is the bandwidth limit in bytes (kbytes, mbytesm etc) per second imposed upon the target. This is mainly used if the source does not have mbuffer installed, but bandwidth limits are desired.
 
 + --no-command-checks
 

From 68cc59a41cb2a89632fd74eb5ca2ace2244c281c Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Fri, 29 Mar 2019 11:53:56 -0400
Subject: [PATCH 53/62] Update INSTALL.md

---
 INSTALL.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/INSTALL.md b/INSTALL.md
index 3a941a6..c4063a4 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -83,7 +83,7 @@ ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 [Service]
 Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/sbin/sanoid --take-snapshots
+ExecStart=/usr/local/sbin/sanoid --take-snapshots
 EOF
 
 cat << "EOF" | sudo tee /etc/systemd/system/sanoid-prune.service
@@ -96,7 +96,7 @@ ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 [Service]
 Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/sbin/sanoid --prune-snapshots
+ExecStart=/usr/local/sbin/sanoid --prune-snapshots
 
 [Install]
 WantedBy=sanoid.service

From 17e15c2f54843e0f2c1ecff234e2e83f89c1c77f Mon Sep 17 00:00:00 2001
From: Daniel Wood <daniel@woods.us>
Date: Fri, 29 Mar 2019 09:37:28 -0700
Subject: [PATCH 54/62] Fix systemd service definitions.

`sanoid-prune.service` is not referenced by a timer, so I have removed it.

I also modified the behavior of sanoid.service so it will take and prune snapshots according to the config file:
 - `--cron` switch is required for it to take action as a service
 - `--verbose` is there so that useful logs appear under `journalctl -t sanoid` (I personally run `--debug`)
---
 INSTALL.md | 20 ++------------------
 1 file changed, 2 insertions(+), 18 deletions(-)

diff --git a/INSTALL.md b/INSTALL.md
index c4063a4..8b9b31d 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -75,7 +75,7 @@ Create a systemd service:
 ```bash
 cat << "EOF" | sudo tee /etc/systemd/system/sanoid.service
 [Unit]
-Description=Snapshot ZFS Pool
+Description=Sanoid ZFS Snapshot Manager
 Requires=zfs.target
 After=zfs.target
 ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
@@ -83,23 +83,7 @@ ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 [Service]
 Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/local/sbin/sanoid --take-snapshots
-EOF
-
-cat << "EOF" | sudo tee /etc/systemd/system/sanoid-prune.service
-[Unit]
-Description=Cleanup ZFS Pool
-Requires=zfs.target
-After=zfs.target sanoid.service
-ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
-
-[Service]
-Environment=TZ=UTC
-Type=oneshot
-ExecStart=/usr/local/sbin/sanoid --prune-snapshots
-
-[Install]
-WantedBy=sanoid.service
+ExecStart=/usr/local/sbin/sanoid --cron --verbose
 EOF
 ```
 

From 90b8c92ec656000af5bb9d1605a940f32ed3f7db Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Sun, 31 Mar 2019 15:21:19 +0200
Subject: [PATCH 55/62] sanoid-prune.service is indeed referenced and run after
 the sanoid.service

Revert "Fix systemd service definitions."

This reverts commit 17e15c2f54843e0f2c1ecff234e2e83f89c1c77f.
---
 INSTALL.md | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/INSTALL.md b/INSTALL.md
index 8b9b31d..c4063a4 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -75,7 +75,7 @@ Create a systemd service:
 ```bash
 cat << "EOF" | sudo tee /etc/systemd/system/sanoid.service
 [Unit]
-Description=Sanoid ZFS Snapshot Manager
+Description=Snapshot ZFS Pool
 Requires=zfs.target
 After=zfs.target
 ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
@@ -83,7 +83,23 @@ ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 [Service]
 Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/local/sbin/sanoid --cron --verbose
+ExecStart=/usr/local/sbin/sanoid --take-snapshots
+EOF
+
+cat << "EOF" | sudo tee /etc/systemd/system/sanoid-prune.service
+[Unit]
+Description=Cleanup ZFS Pool
+Requires=zfs.target
+After=zfs.target sanoid.service
+ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
+
+[Service]
+Environment=TZ=UTC
+Type=oneshot
+ExecStart=/usr/local/sbin/sanoid --prune-snapshots
+
+[Install]
+WantedBy=sanoid.service
 EOF
 ```
 

From 1900ecaf32252d274ebae6d5502b5f3de89e7d12 Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Sun, 31 Mar 2019 15:25:30 +0200
Subject: [PATCH 56/62] added verbose cmd line flags for journald logging

---
 INSTALL.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/INSTALL.md b/INSTALL.md
index c4063a4..88435d0 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -83,7 +83,7 @@ ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 [Service]
 Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/local/sbin/sanoid --take-snapshots
+ExecStart=/usr/local/sbin/sanoid --take-snapshots --verbose
 EOF
 
 cat << "EOF" | sudo tee /etc/systemd/system/sanoid-prune.service
@@ -96,7 +96,7 @@ ConditionFileNotEmpty=/etc/sanoid/sanoid.conf
 [Service]
 Environment=TZ=UTC
 Type=oneshot
-ExecStart=/usr/local/sbin/sanoid --prune-snapshots
+ExecStart=/usr/local/sbin/sanoid --prune-snapshots --verbose
 
 [Install]
 WantedBy=sanoid.service

From 607aad0a91c0e59c450e0c616fe6dfb899c9bd63 Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Sun, 7 Apr 2019 19:48:25 -0400
Subject: [PATCH 57/62] Add -r to zfs get -Hpt snap for zfs 8.1

---
 sanoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sanoid b/sanoid
index daa81a1..ffc5323 100755
--- a/sanoid
+++ b/sanoid
@@ -721,7 +721,7 @@ sub getsnaps {
 					print "INFO: cache expired - updating from zfs list.\n";
 				}
 			}
-			open FH, "$zfs get -Hpt snapshot creation |";
+			open FH, "$zfs get -Hrpt snapshot creation |";
 			@rawsnaps = <FH>;
 			close FH;
 

From 8d8df7e533b7d72c8f99683dd1f87dc2b758b2ad Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Sun, 7 Apr 2019 19:51:32 -0400
Subject: [PATCH 58/62] Add -r to zfs get -t snap for zfs 8.1

---
 syncoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index c2a0649..39010cd 100755
--- a/syncoid
+++ b/syncoid
@@ -1384,7 +1384,7 @@ sub getsnaps() {
 		$fsescaped = escapeshellparam($fsescaped);
 	}
 
-	my $getsnapcmd = "$rhost $mysudocmd $zfscmd get -Hpd 1 -t snapshot guid,creation $fsescaped |";
+	my $getsnapcmd = "$rhost $mysudocmd $zfscmd get -Hpd 1 -rt snapshot guid,creation $fsescaped |";
 	if ($debug) { print "DEBUG: getting list of snapshots on $fs using $getsnapcmd...\n"; }
 	open FH, $getsnapcmd;
 	my @rawsnaps = <FH>;

From ea90108eab3f6db7cc78395e80ac7aa27e130d6f Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Tue, 9 Apr 2019 10:50:29 -0400
Subject: [PATCH 59/62] revert -r in zfs get (not needed here)

---
 syncoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index 39010cd..c2a0649 100755
--- a/syncoid
+++ b/syncoid
@@ -1384,7 +1384,7 @@ sub getsnaps() {
 		$fsescaped = escapeshellparam($fsescaped);
 	}
 
-	my $getsnapcmd = "$rhost $mysudocmd $zfscmd get -Hpd 1 -rt snapshot guid,creation $fsescaped |";
+	my $getsnapcmd = "$rhost $mysudocmd $zfscmd get -Hpd 1 -t snapshot guid,creation $fsescaped |";
 	if ($debug) { print "DEBUG: getting list of snapshots on $fs using $getsnapcmd...\n"; }
 	open FH, $getsnapcmd;
 	my @rawsnaps = <FH>;

From 483ba0c3ee3223c7be0d2ba22c78f831dac65b73 Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Sat, 20 Apr 2019 16:48:21 -0400
Subject: [PATCH 60/62] version number update, oops

---
 sanoid | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sanoid b/sanoid
index ffc5323..e54800f 100755
--- a/sanoid
+++ b/sanoid
@@ -4,7 +4,7 @@
 # from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17.  A copy should also be available in this
 # project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
 
-$::VERSION = '2.0.0';
+$::VERSION = '2.0.1';
 my $MINIMUM_DEFAULTS_VERSION = 2;
 
 use strict;

From dd47dd536886dbd77c66ca77e26b2ac4b9a7f759 Mon Sep 17 00:00:00 2001
From: Jim Salter <github@jrs-s.net>
Date: Sat, 20 Apr 2019 16:48:49 -0400
Subject: [PATCH 61/62] version number update, oops

---
 VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/VERSION b/VERSION
index 227cea2..38f77a6 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-2.0.0
+2.0.1

From 522bdecd53d221d9c937986c5de76b80d287d8bc Mon Sep 17 00:00:00 2001
From: Christoph Klaffl <christoph@phreaker.eu>
Date: Wed, 15 May 2019 16:47:51 +0200
Subject: [PATCH 62/62] fixed ordering of snapshots with the same creation
 timestamp

---
 syncoid | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/syncoid b/syncoid
index 39010cd..fc5854b 100755
--- a/syncoid
+++ b/syncoid
@@ -1393,6 +1393,8 @@ sub getsnaps() {
 	# this is a little obnoxious. get guid,creation returns guid,creation on two separate lines
 	# as though each were an entirely separate get command.
 
+	my %creationtimes=();
+
 	foreach my $line (@rawsnaps) {
 		# only import snap guids from the specified filesystem
 		if ($line =~ /\Q$fs\E\@.*guid/) {
@@ -1413,7 +1415,24 @@ sub getsnaps() {
 			$creation =~ s/^.*\tcreation\t*(\d*).*/$1/;
 			my $snap = $line;
 			$snap =~ s/^.*\@(.*)\tcreation.*$/$1/;
-			$snaps{$type}{$snap}{'creation'}=$creation;
+
+			# the accuracy of the creation timestamp is only for a second, but
+			# snapshots in the same second are highly likely. The list command
+			# has an ordered output so we append another three digit running number
+			# to the creation timestamp and make sure those are ordered correctly
+			# for snapshot with the same creation timestamp
+			my $counter = 0;
+			my $creationsuffix;
+			while ($counter < 999) {
+				$creationsuffix = sprintf("%s%03d", $creation, $counter);
+				if (!defined $creationtimes{$creationsuffix}) {
+					$creationtimes{$creationsuffix} = 1;
+					last;
+				}
+				$counter += 1;
+			}
+
+			$snaps{$type}{$snap}{'creation'}=$creationsuffix;
 		}
 	}