cyteen
/
sanoid
mirror of https://github.com/jimsalterjrs/sanoid
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
916 Commits 2 Branches 20 Tags 4.3 MiB
Commit Graph

362 Commits

Author SHA1 Message Date
Christoph Klaffl 91d96a4c87
support bookmarks which are taken in the same second 2023-03-24 09:00:07 +01:00
Christoph Klaffl 0b27059133
Merge remote-tracking branch 'upstream/master' into filter-snaps 2023-03-21 16:35:20 +01:00
Christoph Klaffl f3152d6f6c
Merge remote-tracking branch 'upstream/master' into syncoid-hold 2023-03-21 15:44:04 +01:00
Christoph Klaffl b3d32d9394
Merge branch 'pr788' into pr-merger 2023-03-20 22:31:19 +01:00
Christoph Klaffl f942341f8d
Merge branch 'pr636' into pr-merger 2023-03-20 21:49:00 +01:00
Christoph Klaffl 42ed026837
Merge branch 'pr538' into pr-merger 2023-03-20 21:20:26 +01:00
Christoph Klaffl 9e87e8c95a
Merge branch 'pr769' into pr-merger 2023-03-20 21:14:43 +01:00
Christoph Klaffl 487fba24e5
Merge branch 'pr768' into pr-merger 2023-03-20 21:14:34 +01:00
Christoph Klaffl b488611ae7
Merge branch 'pr767' into pr-merger 2023-03-20 21:14:29 +01:00
Christoph Klaffl c2f5367d3d
Merge branch 'pr766' into pr-merger 2023-03-20 21:14:25 +01:00
Christoph Klaffl 59544de134
Merge branch 'pr605' into pr-merger 2023-03-20 21:14:02 +01:00
Josh Soref 9421892402 spelling: naming 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2023-01-01 14:43:35 -05:00
Vinnie Okada 0c577fc735 Deprecate the --exclude option 
Add a new option, --exclude-datasets, to replace --exclude. This makes
the naming more consistent now that there are options to filter both
snapshots and datasets.

Also add more information to the README about the distinction between
--exclude-datasets and --(in|ex)clude-snaps.
 2022-12-24 12:59:59 -07:00
Vinnie Okada 14ed85163a Filter snapshots in getsnapsfallback() 2022-12-24 12:55:17 -07:00
Vinnie Okada 9a067729a9 Implement include-snaps and exclude-snaps 
Add --include-snaps and --exclude-snaps options to filter the snapshots
that syncoid uses.
 2022-12-24 12:55:08 -07:00
Vinnie Okada 603c286b50 Don't iterate over snaps twice 
Process snapshots in one pass rather than looping separately for both
guid and create time.
 2022-12-24 12:52:06 -07:00
Vinnie Okada 09b42d6ade Refactor system calls 
Build the zfs send and receive commands in a new subroutine, and
implement other subroutines that can be called instead of building a zfs
command and running it with system();
 2022-12-24 12:51:58 -07:00
Vinnie Okada c4e7028022 Refactor terminal output 
Replace `print` and `warn` statements with a logging function.
 2022-12-24 12:48:54 -07:00
Christoph Klaffl 0808575874
syncoid should exit with an error if the specified src dataset doesn't exist (and only skip child datasets which vanished) 2022-09-13 17:13:01 +02:00
Christoph Klaffl a5d7896c0f
improve accuracy of zfs receive check 2022-09-13 00:16:49 +02:00
Christoph Klaffl 4f150ec6ec
fix snapshot listing fallback 2022-09-13 00:02:03 +02:00
Christoph Klaffl fa67fbe845
bring back no-rollback option 2022-09-12 23:44:52 +02:00
Mark Scholes 59c59e2b00 Removed unneeded 2>/dev/null from mbufferoptions 2022-05-19 11:54:16 +01:00
Endre Szabo 21931bd8a6
Merge branch 'jimsalterjrs:master' into ssh_config 2022-04-25 09:05:58 +02:00
Jim Salter ef601eb96b Merge branch 'master' of https://github.com/jimsalterjrs/sanoid 2022-04-08 21:34:58 +00:00
Jim Salter ee8c884460 put local resume token support back in 2022-04-08 21:34:01 +00:00
Jim Salter b0a71f763a
Merge pull request #734 from Gryd3/patch-1 
append a rand int to the socket name.
 2022-04-08 17:18:23 -04:00
Jim Salter ef2b4f887a no more rolling targets back, receive -F only 2022-04-08 21:09:51 +00:00
Jim Salter ec7858c1c7 don't use resume tokens for purely local replication 2022-04-08 20:57:53 +00:00
Gryd3 b1c2684c63
append a rand int to the socket name. 
Running more than one 'syncoid' at the same time to the same host resulted in two jobs referencing the same socket.
Often results in "already exists, disabling multiplexing" but has in more than one occasion failed with the following:

ControlSocket /tmp/syncoid-zfsbackup-zfsbackup@10.0.0.1-1649107066 already exists, disabling multiplexing
lzop: Inappropriate ioctl for device: <stdin>
CRITICAL ERROR: ssh     -S /tmp/syncoid-zfsbackup-zfsbackup@10.0.0.1-1649107066 zfsbackup@10.0.0.1 ' zfs send  -I '"'"'pool/office'"'"'@'"'"'autosnap_2022-04-04_21:00:00_frequently'"'"' '"'"'pool/office'"'"'@'"'"'autosnap_2022-04-04_21:15:00_frequently'"'"' | lzop  | mbuffer -R 5m -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -s 18356312 |  zfs receive  -s -F 'zfs-pool/vault/office' 2>&1 failed: 256 at /usr/sbin/syncoid line 786.

Sample use-case: 
Using Monit, Cron, or some other scheduler to trigger more than syncoid to the same host to sync two datasets.
Stagger the sync so that no two jobs get started at the same time, or add some form of randomization to the socket name so that two jobs may start at the same time.
 2022-04-04 15:00:31 -07:00
Juergen Heiles 8cf0144c02
Merge branch 'jimsalterjrs:master' into syncoid-hold 2021-11-13 21:09:48 +01:00
git@end.re c151703837 Implementing support for ssh_config(5) files 2021-04-08 15:01:45 +02:00
rbike 21eee41fdb Final revision 
comments from @phreaker0 implemented
 2021-04-01 09:23:48 +02:00
rbike 2858fbe907 Docs update + error handling 
updated in-program docs and readme,
check if matchingsnap exists before releasing hold
 2021-03-03 13:49:13 +01:00
rbike 3bae6baf3d syncoid hold 
Added hold feature to syncoid.
Parameter "--use-hold" sets hold for newsyncsmap and removes hold from matchingsnap both on source and target. Hold name is "syncoid" + identiifer + host name
 2021-03-02 16:03:16 +01:00
Christoph Klaffl 309c0866fa
implemented removal of conflicting snapshots with force-delete option 2020-12-07 22:40:28 +01:00
Christoph Klaffl 77d0135a60
prepare v2.1.0 release 2020-11-24 11:52:20 +01:00
Christoph Klaffl 2da9ba2095
only preserve recordsize for datasets of type "filesystem" 2020-11-03 18:05:13 +01:00
Christoph Klaffl bdd0dfb733
Merge branch 'master' into preserve-recordsize 2020-11-03 17:50:50 +01:00
Jim Salter 59e8949efe
Merge pull request #498 from phreaker0/syncoid-dst-fix 
append timezone offset to the syncoid snapshot name to fix DST collis…
 2020-11-01 17:46:32 -05:00
Jim Salter 14e6c62d0a
Merge pull request #508 from phreaker0/check-arguments 
check for an invalid argument combination
 2020-11-01 17:24:29 -05:00
Jim Salter 42ea8dffce
Merge pull request #507 from phreaker0/fix-busy-detection 
fix iszfsbusy check for similar dataset names
 2020-11-01 17:24:20 -05:00
Jim Salter 345e7e58b5
Merge pull request #374 from aerusso/pulls/syncoid-dont-require-user 
Do not require user to be specified for syncoid
 2020-11-01 17:22:57 -05:00
Jim Salter f187e072b8
Merge pull request #529 from phreaker0/root-resume-check 
use sudo if neccessary for checking pool capabilities regarding resum…
 2020-11-01 17:19:50 -05:00
Jim Salter c21b72a376
Merge pull request #510 from phreaker0/reset-resume-2 
catch another case in which the resume state isn't availabe anymore
 2020-11-01 17:19:22 -05:00
Jim Salter 0ac10750a7
Merge pull request #536 from phreaker0/keep-sync-snap 
implemented option for keeping sync snaps
 2020-11-01 17:19:00 -05:00
Antonio Russo f397687968
Do not require user to be specified for syncoid 
Extends syncoid remote capabilities to match that of ssh as closely as
possible: allow a remote dataset to be specified without a username.

 - Detect if a remote reference is possible by looking for a : before
   any / characters.
 - Check if there are any pool names that might conflict with this
   name. E.g., 'weird:symbol/ds' might refer to the pool "symbol" on
   host "weird", and dataset ds. OR it might refer to the local pool
   "weird:symbol" and dataset ds.
 - Prefer local pools, matching existing behavior. No preexisting
   functioning configurations will break.
 - The name of the control socket is changed slightly.
 - A bug in the handling of remote datasets with colons in the name
   is addressed.

Signed-off-by: Antonio Russo <aerusso@aerusso.net>
 2020-11-01 10:55:50 -07:00
Jim Perkins 5092ff1000
typo 'target -bwlimit' 
Use of uninitialized value in concatenation (.) or string at ./syncoid line 1317.
just an asuumption on my part considering the rest of the variables with same name dont have the space in that location
 2020-08-05 11:37:48 -05:00
jimsalterjrs 9544ea7e6a typo in syncoid help 2020-05-23 16:27:29 -04:00
James Bunton 6bc210deb3 fix stream sync from bookmark 
syncoid fails in the following case:
- performing a stream sync (default)
- there are no matching snapshots between source and target
- there is a bookmark on the source matching a snapshot on the target
- there is an unrelated snapshot on the source older than this bookmark

This happens because the units for snapshots and bookmarks were
different. syncoid tells `zfs send` to generate an incremental stream
starting from the bookmark and going to the unrelated older snapshot.
ZFS refuses to do this because it doesn't make sense.

This fix is a bit of hack. Ideally we should share the counter code
between getsnaps and getbookmarks.
 2020-05-03 23:45:29 +10:00
Christoph Klaffl 9ab57e9bb4
added documentation regarding --keep-sync-snap option 2020-05-01 17:26:54 +02:00
Christoph Klaffl d77cfdf4c7
implemented option for keeping sync snaps 2020-05-01 17:20:07 +02:00
Christoph Klaffl 0e807386c4
use sudo if neccessary for checking pool capabilities regarding resumeable send 2020-04-20 19:20:28 +02:00
Christoph Klaffl e1a6507455
direct connection will default to busybox nc again but can be switched to mbuffer 2020-04-02 21:37:06 +02:00
Christoph Klaffl 807f6fa1bb
check if socat and mbuffer is available for insecure direct connection 2020-03-31 09:27:17 +02:00
Christoph Klaffl f0dfd46c9f
use mbuffer tcp listen socket instead of busybox nc 2020-03-30 20:15:09 +02:00
Christoph Klaffl 2e6abddb5e
fixed uninitialized value warning 2020-03-30 19:01:54 +02:00
Christoph Klaffl 84bc3164f0
implemented option for direct connection via socat and busybox nc 2020-02-21 20:34:28 +01:00
Christoph Klaffl b808a74e2e
catch another case were the resume state isn't availabe anymore 2020-02-14 18:38:38 +01:00
Christoph Klaffl 4ebdc9d173
check for an invalid argument combination 2020-02-13 18:31:56 +01:00
Christoph Klaffl 13ba2b43fa
fix iszfsbusy check for similar dataset names 2020-02-13 01:37:54 +01:00
Christoph Klaffl 1bad3cd25b
append timezone offset to the syncoid snapshot name to fix DST collisions 2020-02-12 18:24:08 +01:00
croadfeldt bb30496d19
Fixed Typo in arguments. 
Removed errant A in zfs list command line arguments.
 2020-01-18 01:52:12 -06:00
Christoph Klaffl d0ef3ec2d5
Merge branch 'master' into preserve-recordsize 2020-01-16 21:23:07 +01:00
Jim Salter e5c998c71d
Merge pull request #408 from gdevenyi/pvoptions 
add ability to configure pv in syncoid
 2020-01-16 13:57:10 -05:00
Jim Salter e031976cde
Merge pull request #415 from phreaker0/send-size-adaption 
added needed verbose flag for send size estimation (at least for late…
 2020-01-16 13:50:57 -05:00
Jim Salter 69d6ae21d7
Merge pull request #383 from mschout/feature/ignore-missing-filesystems 
Gracefully handle error when source dataset disappeared
 2020-01-16 13:47:09 -05:00
Christoph Klaffl 28ef311ba5
only print stderr output of failed listing command with --debug flag 2020-01-16 18:10:20 +01:00
Christoph Klaffl 3b18948f29
typo 2020-01-16 17:49:43 +01:00
Christoph Klaffl 77fd555a84
fixed state reset 2020-01-16 01:26:44 +01:00
Christoph Klaffl 30fb5aabeb
implemented fallback for listing snapshots on solaris 2020-01-13 19:25:22 +01:00
Christoph Klaffl 96a48efd92
prepare v2.0.3 bugfix release 2019-10-02 16:42:38 +02:00
Christoph Klaffl b04bd2f02d
proper changelog for the last two version and packaging updates 2019-09-25 22:22:00 +02:00
Christoph Klaffl 381560a320
documented preserve-recordsize flag 2019-09-25 18:33:55 +02:00
Christoph Klaffl a9043b5301
implemented flag for preserving recordsize on inital sends 2019-09-25 18:24:40 +02:00
Christoph Klaffl a85ad93a18
added needed verbose flag for send size estimation (at least for latest FreeBSD codebase) 2019-06-18 08:32:11 +02:00
Gabriel A. Devenyi dee9d1817f Add ability to configure pv 2019-06-14 11:50:34 -04:00
Jim Salter a90eec33e3
Merge pull request #405 from hartzell/patch-1 
Documentation buglet, --sshkey specifies the *key*
 2019-06-14 10:11:19 -04:00
Christoph Klaffl 9791094816
some FreeBSD related fixes 2019-06-14 15:45:44 +02:00
George Hartzell cb94fa1d16
Documentation buglet, --sshkey specifies the *key* 
The `--sshkey` option specifies the ssh key, *not* the ssh public key.  Empirically, it fails if you point it at a public key and works if you point it at the private part.  Make sense since it's being used as the arg to `-i` in the `ssh` command line.
 2019-06-13 14:34:08 -07:00
Christoph Klaffl 56f97b5bd2
capture the stderr of zfs send command in the case of a resumed replication where the send size can't be estimated (in which case pv is not working anyway, so we can break the interactivitiy) 2019-06-06 08:09:42 +02:00
Christoph Klaffl 22fe41cb11
redirect zfs receive error to stdout and only capture stdout for parsing of error codes, this way stderr is left alone and pv can work interactively 2019-06-06 08:04:00 +02:00
Michael Schout 3892d73594 Capture output in getzfsvalue and handle datasets that disappeared 
Change getzfsvalue() so that if called in array context, returns the
value and the error as a two element list.  This allows the caller to
recieve the error from the zfs command.

If the dataset went away before fetching the sanoid:sync property, just
issue a warning and skip it.
 2019-05-29 17:13:31 -05:00
Michael Schout 784efe2d85 Remove debug warning 2019-05-25 12:01:54 -05:00
Michael Schout fd71e794ad Gracefully handle error when source dataset disappeared 
If the source dataset dissappeared before we were able to get the
sanoid:sync property, do not set an error exit code.  Handle this
gracefully as this should not be considered an error.

Fixes #380
 2019-05-25 12:01:54 -05:00
Christoph Klaffl 6eb1be3ef7
handle resume states where the source snapshot was already deleted 2019-05-20 18:18:21 +02:00
Christoph Klaffl 16fceba54c
moved common used variables to top of function 2019-05-20 18:01:29 +02:00
Christoph Klaffl 1046f7f58e
use capture::tiny for teeing commands 2019-05-16 18:42:26 +02:00
Christoph Klaffl f7bd267a6b
Merge branch 'master' into reset-resume-state 2019-05-16 18:15:02 +02:00
Jim Salter e320462c69
Merge pull request #382 from phreaker0/fix-snapshot-ordering 
fixed ordering of snapshots with the same creation timestamp
 2019-05-15 12:41:32 -04:00
Christoph Klaffl 522bdecd53
fixed ordering of snapshots with the same creation timestamp 2019-05-15 16:47:51 +02:00
Jim Salter ea90108eab
revert -r in zfs get (not needed here) 2019-04-09 10:50:29 -04:00
Jim Salter 8d8df7e533
Add -r to zfs get -t snap for zfs 8.1 2019-04-07 19:51:32 -04:00
Christoph Klaffl a78faed88c
Merge branch 'master' into path 2019-03-31 20:52:10 +02:00
Christoph Klaffl 111d93a166
Merge branch 'master' into fix-bookmark-edge-cases 2019-03-31 15:39:56 +02:00
Jim Salter e376eadc97
Merge pull request #342 from phreaker0/create-bookmark 
implemented creation of zfs bookmarks
 2019-03-29 12:00:33 -04:00
Jim Salter 12198180d8
Merge pull request #352 from phreaker0/clone-fallback 
use ordinary replication if clone recreation fails
 2019-03-29 12:00:00 -04:00
Jim Salter 11b3af65fd
Merge pull request #334 from phreaker0/check-pool-resume 
check pool capability for resumeable replication instead of checking …
 2019-03-29 11:59:22 -04:00
Jim Salter fd2823926d
Merge pull request #319 from phreaker0/silence-excludes 
removed debug output from excludes loop
 2019-03-29 11:58:50 -04:00
Jim Salter 31e46da29b
Merge pull request #343 from phreaker0/whitelist-send-recv-options 
implemented parsing of provided zfs send/recv options and whitelisting for each use case as needed
 2019-03-29 11:57:13 -04:00