2.0.2 9791094 some FreeBSD related fixes 56f97b5 capture the stderr of zfs send command in the case of a resumed replication where the send size can't be estimated (in which case pv is not working anyway, so we can break the interactivitiy) 22fe41c redirect zfs receive error to stdout and only capture stdout for parsing of error codes, this way stderr is left alone and pv can work interactively 5d7ee40 fix centos package name 83c5e2c added new dependency to all packages and install instructions 0537de5 test case for reseting the resume state if it's invalid 6eb1be3 handle resume states where the source snapshot was already deleted 16fceba moved common used variables to top of function 0ef2bd2 added test to verify the last changes to handle partially received replication streams 1046f7f use capture::tiny for teeing commands eef598c Update rules 67ff4fe Update INSTALL.md 522bdec fixed ordering of snapshots with the same creation timestamp ce93d89 Declare that sanoid service 'Wants' sanoid-prune 2.0.1 ea90108 revert -r in zfs get (not needed here) 8d8df7e Add -r to zfs get -t snap for zfs 8.1 607aad0 Add -r to zfs get -Hpt snap for zfs 8.1 faf268b Added line to enable sanoid-prune.service 1900eca added verbose cmd line flags for journald logging 90b8c92 sanoid-prune.service is indeed referenced and run after the sanoid.service 17e15c2 Fix systemd service definitions. 68cc59a Update INSTALL.md 56d7c59 clarify --bwlimit units 681aa61 clarify bandwidth limit units ffa6ad3 Fix gentoo package snafu - need a sanoid directory to store the files df1e344 Fix gentoo package snafu - need a sanoid directory to store the files cf496dc Changed (and simplified) DST handling (see here: https://github.com/jimsalterjrs/sanoid/issues/155) 736d031 initial replication without intermediate snapshots should use the created sync snapshot e0e7b0d added bwlimit units a97da7d Added bwlimit units e014fd7 check if ssh connection works eefc659 update the install instruction with recent changes 25c5e2e allow new zfs send/recv hold parameter (introduced in recent master) and drop invalid raw parameter for send size estimation for resumeable send c543f9a use ordinary replication if clone recreation fails d3f19c8 Sanoid ebuild for Gentoo a748b41 if bookmark creation fails use guid bases suffix based fallback name 99b439e return if there is nothing to do 5812b57 check for invalid send/receive option syntax 10b2012 removed debug output 7d522fa fix unitialized variable 1053e93 missed sudo command for bookmark creation if source host is remote f0fec53 fix root pool parsing for remote hosts c5eebbe implemented parsing of provided zfs send/recv options and whitelisting for each use case as needed 5107380 codestyle d68d7b4 implemented creation of zfs bookmarks 7141eab fix bookmark edge case where replication was already done to the latest snapshot (+ test) ea9e989 only list filesystems and volumes 132c765 error out if no datasets are processed with recursive arg 85561dd handle the case of mistyped dataset name nicely c12b6d7 check pool capability for resumeable replication instead of checking for support in the zfs module 7ef435c add "hotspare" template (for local, hourly replication targets) c3e20fd FreeBSD sed needs a different syntax ea482ce make tests work on FreeBSD ab12540 added message if zfs recursion is used and fixed ordinary recursion in combination with zfs recursion 0fc16b4 fixed uninitialzed variable warning and clarified recursive value 271ede7 Revert "Revert "Zfs Recursion"" e4e9065 removed debug output from excludes loop 50a2374 Revert "Zfs Recursion" 165faee remove spurious line break from HTML f941d7f fix for #316 - CRITICAL ERROR: bookmarks couldn't be listed for bd4eb49 Added in --sendoptions=OPTIONS and --recvoptions=OPTIONS to inject OPTIONS enabling things like syncoid --sendoptions=-Lcep and syncoid --recvoptions="-x property" ca76f42 syncoid: add '--mbuffer-size' option c35c953 remove redundant loop f8e0e00 preserve taking snapshots order always from yearly to frequently 7cf64be pre/post snapshot script calls should be also dumped in --readonly mode 4daafca prevent problems with shell expansion and codestyle 1605a60 CLIMATE-1151: added config option to use zfs recursion 8d88c47 improved documentation on --no-command-checks ac80a75 made time units case insensitive and removed monthlies cfab4ea added/fixed documentation 89f1d4e run compress commands bare too 941a770 updated comment regarding command existence check be50481 use more portable/POSIX compatible ways to check for command existence 9b2c2f2 don't use hardcoded paths abf3d45 check for partial resume state for non resume style transfers, if so reset state and retry 0f669a3 Cleaned up Configuration section to match the rest of the document 429da3a Add comprehensive installation instructions 2.0.0 [overall] documentation updates, small fixes, more warnings (@sparky3387, @ljwobker, @phreaker0) [syncoid] added force delete flag (@phreaker0) [sanoid] removed sleeping between snapshot taking (@phreaker0) [syncoid] added '--no-privilege-elevation' option to bypass root check (@lopsided98) [sanoid] implemented weekly period (@phreaker0) [syncoid] implemented support for zfs bookmarks as fallback (@phreaker0) [sanoid] support for pre, post and prune snapshot scripts (@jouir, @darkbasic, @phreaker0) [sanoid] ignore snapshots types that are set to 0 (@muff1nman) [packaging] split snapshot taking/pruning into separate systemd units for debian package (@phreaker0) [syncoid] replicate clones (@phreaker0) [syncoid] added compression algorithms: lz4, xz (@spheenik, @phreaker0) [sanoid] added option to defer pruning based on the available pool capacity (@phreaker0) [sanoid] implemented frequent snapshots with configurable period (@phreaker0) [syncoid] prevent a perl warning on systems which doesn't output estimated send size information (@phreaker0) [packaging] dependency fixes (@rodgerd, mabushey) [syncoid] implemented support for excluding children of a specific dataset (@phreaker0) [sanoid] monitor-health command additionally checks vdev members for io and checksum errors (@phreaker0) [syncoid] added ability to skip datasets by a custom dataset property 'syncoid:no-sync' (@attie) [syncoid] don't die on some critical replication errors, but continue with the remaining datasets (@phreaker0) [syncoid] return a non zero exit code if there was a problem replicating datasets (@phreaker0) [syncoid] make local source bwlimit work (@phreaker0) [syncoid] fix 'resume support' detection on FreeBSD (@pit3k) [sanoid] updated INSTALL with missing dependency [sanoid] fixed monitor-health command for pools containing cache and log devices (@phreaker0) [sanoid] quiet flag suppresses all info output (@martinvw) [sanoid] check for empty lockfile which lead to sanoid failing on start (@jasonblewis) [sanoid] added dst handling to prevent multiple invalid snapshots on time shift (@phreaker0) [sanoid] cache improvements, makes sanoid much faster with a huge amount of datasets/snapshots (@phreaker0) [sanoid] implemented monitor-capacity flag for checking zpool capacity limits (@phreaker0) [syncoid] Added support for ZStandard compression.(@danielewood) [syncoid] implemented support for excluding datasets from replication with regular expressions (@phreaker0) [syncoid] correctly parse zfs column output, fixes resumeable send with datasets containing spaces (@phreaker0) [syncoid] added option for using extra identification in the snapshot name for replication to multiple targets (@phreaker0) [syncoid] added option for skipping the parent dataset in recursive replication (@phreaker0) [syncoid] typos (@UnlawfulMonad, @jsavikko, @phreaker0) [sanoid] use UTC by default in unit template and documentation (@phreaker0) [syncoid] don't prune snapshots if instructed to not create them either (@phreaker0) [syncoid] documented compatibility issues with (t)csh shells (@ecoutu) 1.4.18 implemented special character handling and support of ZFS resume/receive tokens by default in syncoid, thank you @phreaker0! 1.4.17 changed die to warn when unexpectedly unable to remove a snapshot - this allows sanoid to continue taking/removing other snapshots not affected by whatever lock prevented the first from being taken or removed 1.4.16 merged @hrast01's extended fix to support -o option1=val,option2=val passthrough to SSH. merged @JakobR's off-by-one fix to stop unnecessary extra snapshots being taken under certain conditions. merged @stardude900's update to INSTALL for FreeBSD users re:symlinks. Implemented @LordAro's update to change DIE to WARN when encountering a dataset with no snapshots and --no-sync-snap set during recursive replication. Implemented @LordAro's update to sanoid.conf to add an ignore template which does not snap, prune, or monitor. 1.4.15 merged @hrast01's -o option to pass ssh CLI options through. Currently only supports a single -o=option argument - in the near future, need to add some simple parsing to expand -o=option1,option2 on the CLI to -o option1 -o option2 as passed to SSH. 1.4.14 fixed significant regression in syncoid - now pulls creation AND guid on each snap; sorts by creation and matches by guid. regression reported in #112 by @da-me, thank you! 1.4.13 Syncoid will now continue trying to replicate other child datasets after one dataset fails replication when called recursively. Eg syncoid -r source/parent target/parent when source/parent/child1 has been deleted and replaced with an imposter will no longer prevent source/parent/child2 from successfully replicating to target/parent/child2. This could still use some cleanup TBH; syncoid SHOULD exit 3 if any of these errors happen (to assist detection of errors in scripting) but now would exit 0. 1.4.12 Sanoid now strips trailing whitespace in template definitions in sanoid.conf, per Github #61 1.4.11 enhanced Syncoid to use zfs `guid` property rather than `creation` property to ensure snapshots on source and target actually match. This immediately prevents conflicts due to timezone differences on source and target, and also paves the way in the future for Syncoid to find matching snapshots even after `zfs rename` on source or target. Thank you Github user @mailinglists35 for the idea! 1.4.10 added --compress=pigz-fast and --compress=pigz-slow. On a Xeon E3-1231v3, pigz-fast is equivalent compression to --compress=gzip but with compressed throughput of 75.2 MiB/s instead of 18.1 MiB/s. pigz-slow is around 5% better compression than compress=gzip with roughly equivalent compressed throughput. Note that pigz-fast produces a whopping 20+% better compression on the test data (a linux boot drive) than lzop does, while still being fast enough to saturate or nearly saturate a real-world gigabit LAN link. The down side: pigz chews through 100% util of all available system threads, if not bottlenecked by the network link speed. Default compression remains lzop for SSH transport, with compression automatically set to none if there's no transport (ie syncoid replication from dataset to dataset on the local machine only). 1.4.9 added -c option to manually specify the SSH cipher used. Must use a cipher supported by both source and target! Thanks Tamas Papp. 1.4.8 added --no-stream argument to syncoid: allows use of -i incrementals (do not replicate a full snapshot stream, only a direct incremental update from oldest to most recent snapshot) instead of the normal -I incrementals which include all intermediate snapshots. added --no-sync-snap, which has syncoid replicate using only the newest PRE-EXISTING snapshot on source, instead of default behavior in which syncoid creates a new, ephemeral syncoid snapshot. 1.4.7a (syncoid only) added standard invocation output when called without source or target as per @rriley and @fajarnugraha suggestions 1.4.7 reverted Perl shebangs to #!/usr/bin/perl - sorry FreeBSD folks, shebanged to /usr/bin/env perl bare calls to syncoid or sanoid (without explicit calls to Perl) don't work on EITHER of our systems. I'm not OK with that, this is going to be an OS localization issue that can either be addressed with BSD-specific packaging, or you can individually address it by editing the shebangs on your own systems OR by doing a one-time ln -s /usr/local/bin/perl to /usr/bin/perl, which will fix the issue for this particular script AND all other Perl scripts developed on non-BSD systems. also temporarily dyked out the set readonly functionality in syncoid - it was causing more problems than it prevented, and using the -F argument with receive prevents incautious writes (including just cd'ing into mounted datasets, if atimes are on) from interrupting syncoid runs anyway. 1.4.6c merged @gusson's pull request to add -sshport argument 1.4.6b updated default cipherlist for syncoid to chacha20-poly1305@openssh.com,arcfour - arcfour isn't supported on newer SSH (in Ubuntu Xenial and FreeBSD), chacha20 isn't supported on some older SSH versions (Ubuntu Precise< I think?) 1.4.6a due to bug in ZFS on Linux which frequently causes errors to return from `zfs set readonly`, changed ==0 or die in setzfsvalue() to ==0 or [complain] - it's not worth causing replication to fail while this ZFS on Linux bug exists. 1.4.6 added a mollyguard to syncoid to help newbies who try to zfs create a new target dataset before doing an initial replication, instead of letting the replication itself create the target. added "==0 or die" to all system() calls in sanoid and syncoid that didn't already have them. 1.4.5 altered shebang to '#!/usr/bin/env perl' for enhanced FreeBSD compatibility 1.4.4 merged pull requests from jjlawren for OmniOS compatibility, added --configdir=/path/to/configs CLI option to sanoid at jjlawrens' request presumably for same 1.4.3 added SSH persistence to syncoid - using socket speeds up SSH overhead 300%! =) one extra commit to get rid of the "Exit request sent." SSH noise at the end. 1.4.2 removed -r flag for zfs destroy of pruned snapshots in sanoid, which unintentionally caused same-name child snapshots to be deleted - thank you Lenz Weber! 1.4.1 updated check_zpool() in sanoid to parse zpool list properly both pre- and post- ZoL v0.6.4 1.4.0 added findoid tool - find and list all versions of a given file in all available ZFS snapshots. use: findoid /path/to/file 1.3.1 whoops - prevent process_children_only from getting set from blank value in defaults 1.3.0 changed monitor_children_only to process_children_only. which keeps sanoid from messing around with empty parent datasets at all. also more thoroughly documented features in default config files. 1.2.0 added monitor_children_only parameter to sanoid.conf for use with recursive definitions - in cases where container dataset is kept empty 1.1.0 woooo - working recursive definitions in Sanoid! Also intelligent config errors in Sanoid; will die with errors if unknown config value is set. 1.0.20 greatly cleaned up config parsing in sanoid, got rid of 'hardcoded defaults' in favor of /etc/sanoid/sanoid.defaults.conf 1.0.19 working recursive sync (sync specified dataset and all child datasets, ie pool/ds, pool/ds/1, pool, ds/1/a, pool/ds/2 ...) with --recursive or -r in syncoid! 1.0.18 updated syncoid to break sync out of main routine and into syncdataset(). this will allow doing recursive sync, in next update :) 1.0.17 updated syncoid to use sudo when necessary if it isn't already root - working user needs NOPASSWD for /sbin/zfs in /etc/sudoers updated syncoid to throw errors on unknown arguments 1.0.16 updated sanoid to use VASTLY improved CLI argument parsing, throw errors on unknown arguments, etc 1.0.15 updated syncoid to accept compression engine flags - --compress=lzo|gzip|none 1.0.14 updated syncoid to reduce output when fetching snapshot list - thank you github user @0xFate. 1.0.13 removed monitor_version again - sorry for the feature instability, forgot I removed it in the first place because I didn't like pulling in so many dependencies for such a trivial feature 1.0.12 patched default sanoid.conf provided to set yearly_mon and yearly_mday correctly 1.0.11 patched bug in yearly snapshots - thank you @stevenolen for the bug report! 1.0.10 added --monitor-version to check installed version against current version in trunk on GitHub 1.0.9 added CR/LF after --monitor-health message output to make CLI usage cleaner looking 1.0.8 added --version checking in sanoid and syncoid 1.0.7 got rid of unnecessary sudo commands in sanoid's --monitor-health 1.0.6 added 'or die' to make sure /var/cache/sanoidsnapshots.txt is writeable 1.0.5 fixed another ps BSD-ism bug in sanoid 1.0.4 updated ps usage in sanoid to comply with BSDisms 1.0.3 nerfed references to $debug (not implemented yet) in sanoid 1.0.2 implemented iszfsbusy check to the thinning routine in sanoid 1.0.1 ported slightly modified iszfsbusy sub from syncoid to sanoid (to keep from thinning snapshots during replications) 1.0.0 initial commit to Github