Compare commits

...

6 Commits

Author SHA1 Message Date
crpb 8791076a37
Merge ef178f8cd8 into 6beef5fee6 2025-02-13 22:34:45 +00:00
Jim Salter 6beef5fee6
Update INSTALL.md
add cd /tmp to Debian and Ubuntu package build instructions, to avoid newbies getting confused by limited permissions under eg /root

add reference to dependency on Capture::Tiny in sanoid and syncoid
2025-02-11 19:54:53 -05:00
cb ef178f8cd8 Merge branch 'syncoid.service' of https://github.com/crpb/sanoid into syncoid.service 2024-04-04 19:32:36 +02:00
cb 70f8bff190 syncoid.timer: typos 2024-04-04 19:31:42 +02:00
cb 047b0dcdda syncoid.timer: typos 2024-02-11 16:07:24 +01:00
cb 4ab023b63d syncoid: systemd.unit + debian TODOs 2024-02-11 07:39:46 +01:00
14 changed files with 316 additions and 27 deletions

1
.gitignore vendored Normal file
View File

@ -0,0 +1 @@
/debian

View File

@ -26,9 +26,10 @@ apt install debhelper libcapture-tiny-perl libconfig-inifiles-perl pv lzop mbuff
``` ```
Clone this repo, build the debian package and install it (alternatively you can skip the package and do it manually like described below for CentOS): Clone this repo under /tmp (to make sure the apt user has access to the unpacked clone), build the debian package and install it (alternatively you can skip the package and do it manually like described below for CentOS):
```bash ```bash
cd /tmp
git clone https://github.com/jimsalterjrs/sanoid.git git clone https://github.com/jimsalterjrs/sanoid.git
cd sanoid cd sanoid
# checkout latest stable release or stay on master for bleeding edge stuff (but expect bugs!) # checkout latest stable release or stay on master for bleeding edge stuff (but expect bugs!)
@ -73,6 +74,7 @@ cpan # answer the questions and paste the following lines:
Clone this repo, then put the executables and config files into the appropriate directories: Clone this repo, then put the executables and config files into the appropriate directories:
```bash ```bash
cd /tmp
# Download the repo as root to avoid changing permissions later # Download the repo as root to avoid changing permissions later
sudo git clone https://github.com/jimsalterjrs/sanoid.git sudo git clone https://github.com/jimsalterjrs/sanoid.git
cd sanoid cd sanoid
@ -225,9 +227,9 @@ sudo launchctl load /Library/LaunchDaemons/net.openoid.Sanoid.plist
## Other OSes ## Other OSes
**Sanoid** depends on the Perl module Config::IniFiles and will not operate without it. Config::IniFiles may be installed from CPAN, though the project strongly recommends using your distribution's repositories instead. **Sanoid** depends on the Perl modules Config::IniFiles and Capture::Tiny and will not operate without them. These modules may be installed from CPAN, though the project strongly recommends using your distribution's repositories instead.
**Syncoid** depends on ssh, pv, gzip, lzop, and mbuffer. It can run with reduced functionality in the absence of any or all of the above. SSH is only required for remote synchronization. On newer FreeBSD and Ubuntu Xenial chacha20-poly1305@openssh.com, on other distributions arcfour crypto is the default for SSH transport since v1.4.6. Syncoid runs will fail if one of them is not available on either end of the transport. **Syncoid** depends on ssh, pv, gzip, lzop, and mbuffer as well as sharing sanoid's dependency on Capture::Tiny. Capture::Tiny is mandatory, but syncoid can function with reduced functionality without any or all of the command-line dependencies. SSH is only required for remote synchronization. On newer FreeBSD and Ubuntu Xenial chacha20-poly1305@openssh.com, on other distributions arcfour crypto is the default for SSH transport since v1.4.6. Syncoid runs will fail if one of them is not available on either end of the transport.
### General outline for installation ### General outline for installation

View File

@ -1,3 +1,9 @@
sanoid (2.2.1) unstable; urgency=medium
[syncoid] add simple systemd.units
-- Christopher Bock <christopher@bocki.com> Sat, 10 Feb 2023 23:15:00 +0200
sanoid (2.2.0) unstable; urgency=medium sanoid (2.2.0) unstable; urgency=medium
[overall] documentation updates, small fixes (@azmodude, @deviantintegral, @jimsalterjrs, @alexhaydock, @cbreak-black, @kd8bny, @JavaScriptDude, @veeableful, @rsheasby, @Topslakr, @mavhc, @adam-stamand, @joelishness, @jsoref, @dodexahedron, @phreaker0) [overall] documentation updates, small fixes (@azmodude, @deviantintegral, @jimsalterjrs, @alexhaydock, @cbreak-black, @kd8bny, @JavaScriptDude, @veeableful, @rsheasby, @Topslakr, @mavhc, @adam-stamand, @joelishness, @jsoref, @dodexahedron, @phreaker0)

View File

@ -20,4 +20,11 @@ Recommends: gzip,
mbuffer, mbuffer,
openssh-client | ssh-client, openssh-client | ssh-client,
pv pv
Description: Policy-driven snapshot management and replication tools Description: Policy-driven ZFS snapshot management and replication tool
Sanoid is a policy-driven snapshot management and replication tool for ZFS
filesystems.
.
More prosaically, you can use Sanoid to create, automatically thin, and
monitor snapshots and pool health from a single eminently human-readable
TOML config file. Sanoid also includes a replication tool, syncoid, which
facilitates the asynchronous incremental replication of ZFS filesystems.

24
packages/debian/findoid.8 Normal file
View File

@ -0,0 +1,24 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.3.
.TH FINDOID "8" "February 2024" "findoid version 2.2.0" "System Administration Utilities"
.SH NAME
findoid \- manual page for findoid version 2.2.0
.SH DESCRIPTION
.SS "Usage:"
.IP
findoid [options] FILE
.TP
FILE
local path to file for version listing
.IP
Options:
.TP
\fB\-\-path\fR=\fI\,FILE\/\fR
alternative to specify file path to list versions for
.TP
\fB\-\-help\fR
Prints this helptext
.TP
\fB\-\-version\fR
Prints the version number
.PP
(Getopt::Long::GetOptions version 2.52; Perl version 5.36.0)

View File

@ -7,30 +7,8 @@
%: %:
dh $@ dh $@
DESTDIR = $(CURDIR)/debian/sanoid DESTDIR = $(CURDIR)/debian/sanoid
override_dh_auto_install: override_dh_auto_install:
install -d $(DESTDIR)/etc/sanoid install -d $(DESTDIR)/etc/sanoid
install -m 664 sanoid.defaults.conf $(DESTDIR)/etc/sanoid install -m 664 sanoid.defaults.conf $(DESTDIR)/etc/sanoid
install -d $(DESTDIR)/lib/systemd/system
install -m 664 debian/sanoid-prune.service debian/sanoid.timer \
$(DESTDIR)/lib/systemd/system
install -d $(DESTDIR)/usr/sbin
install -m 775 \
findoid sanoid sleepymutex syncoid \
$(DESTDIR)/usr/sbin
install -d $(DESTDIR)/usr/share/doc/sanoid
install -m 664 sanoid.conf \
$(DESTDIR)/usr/share/doc/sanoid/sanoid.conf.example
override_dh_installinit:
dh_installinit --noscripts
override_dh_systemd_enable:
dh_systemd_enable sanoid.timer
dh_systemd_enable sanoid-prune.service
override_dh_systemd_start:
dh_systemd_start sanoid.timer

66
packages/debian/sanoid.8 Normal file
View File

@ -0,0 +1,66 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.3.
.TH SANOID "8" "February 2024" "sanoid version 2.2.0" "System Administration Utilities"
.SH NAME
sanoid \- manual page for sanoid version 2.2.0
.SH DESCRIPTION
.SS "Usage:"
.IP
sanoid [options]
.IP
Assumes \fB\-\-cron\fR \fB\-\-verbose\fR if no other arguments (other than configdir)
are specified
.IP
Options:
.TP
\fB\-\-configdir\fR=\fI\,DIR\/\fR
Specify a directory to find config file sanoid.conf
.TP
\fB\-\-cache\-dir\fR=\fI\,DIR\/\fR
Specify a directory to store the zfs snapshot cache
.TP
\fB\-\-run\-dir\fR=\fI\,DIR\/\fR
Specify a directory for temporary files such as lock files
.TP
\fB\-\-cron\fR
Creates snapshots and purges expired snapshots
.TP
\fB\-\-verbose\fR
Prints out additional information during a sanoid run
.TP
\fB\-\-readonly\fR
Simulates creation/deletion of snapshots
.TP
\fB\-\-quiet\fR
Suppresses non\-error output
.TP
\fB\-\-force\-update\fR
Clears out sanoid's zfs snapshot cache
.TP
\fB\-\-monitor\-health\fR
Reports on zpool "health", in a Nagios compatible format
.TP
\fB\-\-monitor\-capacity\fR
Reports on zpool capacity, in a Nagios compatible format
.TP
\fB\-\-monitor\-snapshots\fR
Reports on snapshot "health", in a Nagios compatible format
.TP
\fB\-\-take\-snapshots\fR
Creates snapshots as specified in sanoid.conf
.TP
\fB\-\-prune\-snapshots\fR
Purges expired snapshots as specified in sanoid.conf
.TP
\fB\-\-force\-prune\fR
Purges expired snapshots even if a send/recv is in progress
.TP
\fB\-\-help\fR
Prints this helptext
.TP
\fB\-\-version\fR
Prints the version number
.TP
\fB\-\-debug\fR
Prints out a lot of additional information during a sanoid run
.PP
(Getopt::Long::GetOptions version 2.52; Perl version 5.36.0)

View File

@ -0,0 +1,4 @@
sanoid.conf
debian/syncoid.conf
debian/syncoid.timer
debian/syncoid.service

View File

@ -0,0 +1,7 @@
syncoid /usr/sbin/
sanoid /usr/sbin/
findoid /usr/sbin/
sleepymutex /usr/sbin/
sanoid.defaults.conf /usr/share/sanoid/
debian/sanoid-prune.service /lib/systemd/system
CHANGELIST /usr/share/doc/sanoid/changelog

View File

@ -0,0 +1,3 @@
debian/findoid.8
debian/sanoid.8
debian/syncoid.8

133
packages/debian/syncoid.8 Normal file
View File

@ -0,0 +1,133 @@
.\" DO NOT MODIFY THIS FILE! It was generated by help2man 1.49.3.
.TH SYNCOID "8" "February 2024" "syncoid version 2.2.0" "System Administration Utilities"
.SH NAME
syncoid \- manual page for syncoid version 2.2.0
.SH DESCRIPTION
.SS "Usage:"
.IP
syncoid [options]... SOURCE TARGET
or syncoid [options]... SOURCE [[USER]@]HOST:TARGET
or syncoid [options]... [[USER]@]HOST:SOURCE TARGET
or syncoid [options]... [[USER]@]HOST:SOURCE [[USER]@]HOST:TARGET
.TP
SOURCE
Source ZFS dataset. Can be either local or remote
.TP
TARGET
Target ZFS dataset. Can be either local or remote
.IP
Options:
.TP
\fB\-\-compress\fR=\fI\,FORMAT\/\fR
Compresses data during transfer. Currently accepted options are gzip, pigz\-fast, pigz\-slow, zstd\-fast, zstd\-slow, lz4, xz, lzo (default) & none
.TP
\fB\-\-identifier\fR=\fI\,EXTRA\/\fR
Extra identifier which is included in the snapshot name. Can be used for replicating to multiple targets.
.TP
\fB\-\-recursive\fR|r
Also transfers child datasets
.TP
\fB\-\-skip\-parent\fR
Skips syncing of the parent dataset. Does nothing without '\-\-recursive' option.
.TP
\fB\-\-source\-bwlimit=\fR<limit k|m|g|t>
Bandwidth limit in bytes/kbytes/etc per second on the source transfer
.TP
\fB\-\-target\-bwlimit=\fR<limit k|m|g|t>
Bandwidth limit in bytes/kbytes/etc per second on the target transfer
.TP
\fB\-\-mbuffer\-size\fR=\fI\,VALUE\/\fR
Specify the mbuffer size (default: 16M), please refer to mbuffer(1) manual page.
.TP
\fB\-\-pv\-options\fR=\fI\,OPTIONS\/\fR
Configure how pv displays the progress bar, default '\-p \fB\-t\fR \fB\-e\fR \fB\-r\fR \fB\-b\fR'
.TP
\fB\-\-no\-stream\fR
Replicates using newest snapshot instead of intermediates
.TP
\fB\-\-no\-sync\-snap\fR
Does not create new snapshot, only transfers existing
.TP
\fB\-\-keep\-sync\-snap\fR
Don't destroy created sync snapshots
.TP
\fB\-\-create\-bookmark\fR
Creates a zfs bookmark for the newest snapshot on the source after replication succeeds (only works with \fB\-\-no\-sync\-snap\fR)
.TP
\fB\-\-use\-hold\fR
Adds a hold to the newest snapshot on the source and target after replication succeeds and removes the hold after the next successful replication. The hold name includes the identifier if set. This allows for separate holds in case of multiple targets
.HP
\fB\-\-preserve\-recordsize\fR Preserves the recordsize on initial sends to the target
.HP
\fB\-\-preserve\-properties\fR Preserves locally set dataset properties similar to the zfs send \fB\-p\fR flag but this one will also work for encrypted datasets in non raw sends
.TP
\fB\-\-no\-rollback\fR
Does not rollback snapshots on target (it probably requires a readonly target)
.HP
\fB\-\-delete\-target\-snapshots\fR With this argument snapshots which are missing on the source will be destroyed on the target. Use this if you only want to handle snapshots on the source.
.TP
\fB\-\-exclude\fR=\fI\,REGEX\/\fR
DEPRECATED. Equivalent to \fB\-\-exclude\-datasets\fR, but will be removed in a future release. Ignored if \fB\-\-exclude\-datasets\fR is also provided.
.HP
\fB\-\-exclude\-datasets\fR=\fI\,REGEX\/\fR Exclude specific datasets which match the given regular expression. Can be specified multiple times
.HP
\fB\-\-exclude\-snaps\fR=\fI\,REGEX\/\fR Exclude specific snapshots that match the given regular expression. Can be specified multiple times. If a snapshot matches both the exclude\-snaps and include\-snaps patterns, then it will be excluded.
.HP
\fB\-\-include\-snaps\fR=\fI\,REGEX\/\fR Only include snapshots that match the given regular expression. Can be specified multiple times. If a snapshot matches both the exclude\-snaps and include\-snaps patterns, then it will be excluded.
.HP
\fB\-\-sendoptions\fR=\fI\,OPTIONS\/\fR Use advanced options for zfs send (the arguments are filtered as needed), e.g. syncoid \fB\-\-sendoptions=\fR"Lc e" sets zfs send \fB\-L\fR \fB\-c\fR \fB\-e\fR ...
.HP
\fB\-\-recvoptions\fR=\fI\,OPTIONS\/\fR Use advanced options for zfs receive (the arguments are filtered as needed), e.g. syncoid \fB\-\-recvoptions=\fR"ux recordsize o compression=lz4" sets zfs receive \fB\-u\fR \fB\-x\fR recordsize \fB\-o\fR compression=lz4 ...
.TP
\fB\-\-sshconfig\fR=\fI\,FILE\/\fR
Specifies an ssh_config(5) file to be used
.TP
\fB\-\-sshkey\fR=\fI\,FILE\/\fR
Specifies a ssh key to use to connect
.TP
\fB\-\-sshport\fR=\fI\,PORT\/\fR
Connects to remote on a particular port
.TP
\fB\-\-sshcipher\fR|c=CIPHER
Passes CIPHER to ssh to use a particular cipher set
.TP
\fB\-\-sshoption\fR|o=OPTION
Passes OPTION to ssh for remote usage. Can be specified multiple times
.TP
\fB\-\-insecure\-direct\-connection\fR=\fI\,IP\/\fR:PORT[,IP:PORT]
WARNING: DATA IS NOT ENCRYPTED. First address pair is for connecting to the target and the second for listening at the target
.TP
\fB\-\-help\fR
Prints this helptext
.TP
\fB\-\-version\fR
Prints the version number
.TP
\fB\-\-debug\fR
Prints out a lot of additional information during a syncoid run
.TP
\fB\-\-monitor\-version\fR
Currently does nothing
.TP
\fB\-\-quiet\fR
Suppresses non\-error output
.TP
\fB\-\-dumpsnaps\fR
Dumps a list of snapshots during the run
.TP
\fB\-\-no\-command\-checks\fR
Do not check command existence before attempting transfer. Not recommended
.TP
\fB\-\-no\-resume\fR
Don't use the ZFS resume feature if available
.TP
\fB\-\-no\-clone\-handling\fR
Don't try to recreate clones on target
.TP
\fB\-\-no\-privilege\-elevation\fR
Bypass the root check, for use with ZFS permission delegation
.TP
\fB\-\-force\-delete\fR
Remove target datasets recursively, if there are no matching snapshots/bookmarks (also overwrites conflicting named snapshots)
.PP
(Getopt::Long::GetOptions version 2.52; Perl version 5.36.0)

View File

@ -0,0 +1,32 @@
# syncoid config example for simple one target/source synchronization
#
# syntax: VARIABLE='foo' # don't forget the ''
# keep in mind that the syncoid.service will call
# ExecStart=/usr/sbin/syncoid $DEBUG $OPTS $EXCLUDES $INCLUDES $SSHOPTS $SOURCE $TARGET
# there are so many variables so we don't have to write an gigantic line
# and as i couldn't use e.g. --exclude-datasets="" it can't be used otherwise
#
# TODO: write syncoid-generator and probably a template syncoid@.service which
# is driven by configuration files in /etc/sanoid/syncoid/$JOB.conf?
# hint: systemd.path(5) PathChanged= for automatic config change detection
# this will get rid of those disgusting variables
#
# all variables should be set, so don't comment them out!
# DEBUG='--debug'
DEBUG=''
# OPTS="--compress=zstd-fast --delete-target-snapshots --force-delete --no-privilege-elevation --recursive"
OPTS=''
# SOURCE='rpool/home'
# SOURCE='user@host:pool/data/set'
SOURCE=''
# TARGET='pool1/data/set'
# TARGET='user@host:pool'
TARGET=''
# SSHOPTS='--sshkey=/path/to/.ssh/id_rsa --sshport=22'
SSHOPTS=''
# EXCLUDES="--exclude-datasets='.*nosnap.*'"
EXCLUDES=''
# INCLUDES="--include-datasets=
INCLUDES=''
# mbuffer behaves icky because env HOME isn't available
MBUFFERRC='/etc/mbuffer.rc'

View File

@ -0,0 +1,11 @@
[Unit]
Description=Transfer ZFS Snapshots
Documentation=man:syncoid(8)
Requires=local-fs.target
Wants=sanoid.service
ConditionFileNotEmpty=/etc/sanoid/syncoid.conf
[Service]
EnvironmentFile=/etc/sanoid/syncoid.conf
Type=oneshot
ExecStart=/usr/sbin/syncoid $DEBUG $OPTS $EXCLUDES $INCLUDES $SSHOPTS $SOURCE $TARGET

View File

@ -0,0 +1,15 @@
[Unit]
Description=Transfer ZFS Snapshots
Documentation=man:syncoid(8)
Documentation=man:systemd.time(7)
[Install]
WantedBy=timers.target
[Timer]
RandomizedDelaySec=60
OnBootSec=5m
#OnUnitActiveSec=20m
OnCalendar=00/2:30
Persistent=true
Unit=syncoid.service