Deduplicating archiver with compression and authenticated encryption.

Test-wrapper is far from full functionality, but right now it works as a fakeroot replacement (to fix #2482).

• ~Xattr: Linux and OSX only (no *BSD yet)~
• Keeping track of file descriptors is a bit slow, but I'm not sure this can be avoided.
• ~No error injection yet.~
• If you're wondering, I'm pretty sure Cargo.lock should stay in git for projects building a binary (like ours).
• ~The dependency libc is currently set to the git version as we need some constants I added in, but the constants haven't made it into a release yet. After their next release we should fix this.~
• I don't have an OSX machine so I'm relying on Travis to test some OSX only xattr stuff, but I checked that it builds fine. Edit: OSX not working due to being unable to intercept unlink.
• ~Does anyone know why -e $TOXENV was passed to tox on OSX? I've removed it, but if Travis fails I guess I'll add it back.~ NVM I think I see why. I'm passing it to all platforms now, I think that'll be fine.

do practical testing with master branch code (or some alpha release).

report anything you find on the issue tracker (as a separate issue), check first if the issue already has been reported. you can post "works for me" in this ticket here.

see end of this ticket for current betas / rc (or directly check the github releases page).

do not run master branch code or borg pre-releases against production repos, always use fresh repos made by borg init!

This issue tracks ongoing work to package borg in various distributions.

Done items are documented in the install docs and should be edited there directly when new packages are published.

Non-exhaustive list of packages needing work:

• Redhat derivatives:
  • [ ] CentOS / RHEL - not in "core" yet, but available from EPEL
  • [x] Fedora https://apps.fedoraproject.org/packages/borgbackup
• OS X
  • [x] brew cask
  • [ ] darwin ports or eventually a DMG, once we get a GUI, could be nice
  • [ ] standalone binary releases + regular testing, see #913
• Windows
  • [ ] Native - in progress, see #983 for roadmap
  • [ ] Cygwin - in progress, see #440, likely to be abandoned for the native port above

Note: there are binary release packages for Linux, FreeBSD and OS X on the github releases page.

Idea of GUI client

Create a simple desktop application that executes borg behind the scenes and runs the very basics of backups.

Goal

Aimed at users with no understanding of backup tools, directory structures, ssh, backup algorithms, etc. Allow them to run their favorite backup tool on any platform.

Proposed tech stack

• PyQt 5 (Qt framework)
• QML for UI
• borg executed as subprocess underneath

Upd (8/24/2017):

• require borg >= 1.1.0rc1 (and later >= 1.1.0)
• use --json(-lines)

Basic requirements

• Backup files to external hard drive
• Run on every major platform: ~~Windows~~, MacOS X, Linux, FreeBSD?

Available target platforms for Qt

Note: Qt is not targeting FreeBSD as of latest 5.9 version

| Platform | |--| | Windows 7, 8.1, 10 (32/64-bit) | | openSUSE 42.1 (64-bit) |   | Red Hat Enterprise Linux 6.6, 7.2 (64-bit) | | Ubuntu 16.04 (64-bit) | | (Linux 32/64-bit) | | macOS 10.10, 10.11, 10.12 | | Embedded Linux | | QNX 6.6.0, 7.0 (armv7le and x86) | | INTEGRITY 11.4.x | | Universal Windows Platform (UWP) (x86, x86_64, armv7) | | Android (API Level: 16) |

Development plan

I'd like to start on this project and create a separate repository for the Desktop GUI. I suppose "borgbackup" can then fork my repository.

Looking forward to any ideas and comments about requirements for such desktop client.

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes.

Is this a BUG / ISSUE report or a QUESTION?

Issue.

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

Currently 1.1.10 on client and server, using the official binary, but the issue persists all the way from 1.1.4. Always used official binaries from this page.

Operating system (distribution) and version.

The issues persists from Debian 8 (Jessie) to Debian 10 (Buster), on client and/or server.

Hardware / network configuration, and filesystems used.

Various Dell PowerEdges, with ECC RAM, disks in RAID-10 (mdadm), gigabit network, ext4 filesystems.

How much data is handled by borg?

We backup ~1TB with Borg on total, but using several smaller repositories by VM/data type.

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create -vsx --files-cache=mtime,size --show-rc --one-file-system --list --filter AME -C zstd [email protected]:postgresql::{now} .

Describe the problem you're observing.

We use Borg for backups for almost two years. Most data are backup-ed from an LVM snapshot. Most are backup-ed every hour, but some data (ex: databases) are backed up every 10 minutes. Some VMs (qcow2 disks) are backup-ed up daily (they are shutdown off-hours for the backup).

With this setup, since we started using Borg, we usually experience 1 or 2 problems per month with Borg repository corruption. In every case, we could fix the problem with borg check --repair.

Every backup is done using SSH.

We had this issue backing-up from VMs and physical servers.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

We could never reproduce the issue.

We checked every disk from every server with a complete S.M.A.R.T. test, with no error detected.

We checked RAM with memtest86 for hours and nothing were found.

We checked that every backup BEFORE the backup job that broke finished successfully.

Destination server never got short on RAM or disk space.

Include any warning/errors/backtraces from the system logs

 + borg create -vsx --files-cache=mtime,size --show-rc --one-file-system --list --filter AME -C zstd [email protected]:postgresql::{now} .
Creating archive at "[email protected]:postgresql::{now}"
M 9.6/main/base/141585/171760
M 9.6/main/base/16507/156896
M 9.6/main/base/16507/1259
M 9.6/main/base/16507/2619
M 9.6/main/base/16507/16551
M 9.6/main/base/16507/16577
M 9.6/main/base/16507/16582
M 9.6/main/base/16507/16606
M 9.6/main/base/16507/16731
M 9.6/main/base/16507/16606_fsm
M 9.6/main/base/16507/17268
M 9.6/main/base/16507/17274
M 9.6/main/base/16507/17326
M 9.6/main/base/16507/17327
M 9.6/main/base/16507/16606_vm
M 9.6/main/pg_clog/0003
A 9.6/main/pg_xlog/000000010000001200000078
M 9.6/main/global/pg_control
Traceback (most recent call last):

  File "borg/remote.py", line 247, in serve

  File "borg/repository.py", line 442, in commit

  File "borg/repository.py", line 789, in compact_segments

AssertionError: Corrupted segment reference count - corrupted index or hints

Borg server: Platform: Linux vm-borg-1 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64
Borg server: Linux: debian 9.9
Borg server: Borg: 1.1.10  Python: CPython 3.5.7 msgpack: 0.5.6
Borg server: PID: 15222  CWD: /srv/borg/root@vm-postgres-1
Borg server: sys.argv: ['borg', 'serve', '--restrict-to-path', '/srv/borg/[email protected]']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --info'
Platform: Linux vm-postgres-1 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64
Linux: debian 9.9
Borg: 1.1.10  Python: CPython 3.5.7 msgpack: 0.5.6
PID: 3953  CWD: /mnt
sys.argv: ['borg', 'create', '-vsx', '--files-cache=mtime,size', '--show-rc', '--one-file-system', '--list', '--filter', 'AME', '-C', 'zstd', '[email protected]:postgresql::{now}', '.']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2

this is a FAQ (by people who have firewalls or want it for other reasons) and some people are evaluating setups with ssh -R (see some posts in #36).

this issue is to collect such setups and if evaluated successfully, add it to the documentation.

note: the debian/ubuntu package description says borg only supports push, maybe that can be removed after this ticket is closed.

so, if you successfully run a pull-like setup, the best thing you can do is to make a pull request that closes this ticket.

:moneybag: there is a bounty for this

Note: to collect the bounty you need to run a reliable pull-like setup, do a pull request for our documentation, documenting the pull-related parts of the setup.

This changes the collision handling strategy in _hashindex.c to robin-hood hashing. Incidentally this should mutually be compatible with the old version.

This hasn't been properly tested yet, except by the unittests, but I'm sharing this early to get some feedback. Some testing and a before and after performance measurement will follow.

Also please note this temporarily disables a couple of tests that hard coded hashes that now no longer match.

If borg could run directly on a normal Windows installation (via cmd, not requiring cygwin or other major runtime support), it could be useful for windows machines that do not have cygwin installed.

Notes:

Doing a native windows port is likely quite some work over a longer time, due to all the differences between windows and UNIX-like operating systems. Whether it can be done in a pretty (regarding to source code changes) and good working (regarding to practical use) way has to be seen.

Because of that, it will have to live in a separate branch of the repository at first (which should be regularly updated from master branch, so it is kept up-to-date and without potential merge conflicts).

At some time in the future, that branch might or might not get merged into master.

:moneybag: there is a bounty for this

I opened a (first) bounty for this - if you would like to see this happen, feel free to support this or a future, related bounty.

The goal and scope of this bounty is:

• have a borg codebase that is able to run on native windows (without cygwin or similar runtime requirements)
• working basic borg commands: init, create, list, extract, delete
• working unit tests (failures in some of them that test stuff not in scope of the bounty are acceptable)
• initial documentation for developers and users (how to set up dev environment, how to install on a user system)
• not required: encryption, special windows file attributes support (like ACLs, xattrs), VSS support, installer

Here is a list of all open issues in attic, acquired with this dirty python script . I suggest we go through all of them and tick them when fixed.

• [ ] init: missing parameter to avoid error if repository already initialized
• [ ] separable repository format
• [ ] ssh:// after network re-connection and SIGINT issues
• [ ] Suggestion: Allow multiple servers to back up to the same repo at the same time
• [ ] Add metadata search index for more responsive FUSE
• [ ] Force-include the specified paths in 'create'

Done / tracked here / invalid / out-of-scope:

• [x] Update installation instruction in README
• [x] Cannot build with pip because of EVP_CIPHER_CTX error
• [x] transaction recovery issue?
• [x] Attic stores hard link to inaccessible file
• [x] attic mount doesn't work
• [x] build on fedora
• [x] safe to interrupt backup?
• [x] Has anyone got working on windows
• [x] Cannot build with pip because of EVP_CIPHER_CTX error
• [x] Attic stores hard link to inaccessible file
• [x] File search Request
• [x] Allow 'list' while backup is in progress
• [x] --include switch
• [x] Expose External Crypto providers
• [x] New security changes broke remote access to repos
• [x] Incorrect directory permissions during extract
• [x] where shall parameters get stored?
• [x] Flexibility and cleanup
• [x] "extract" command has no destination switch?
• [x] Has anyone got working on windows
• [x] Repository clone command?
• [x] Parallel backups from different hosts crash attic
• [x] Cannot mount due to llfuse error
• [x] ZFS snapshot ingestion support
• [x] Let attic know it should wait before beginning
• [x] aes-gcm support
• [x] aes-gcm or aes-ocb?
• [x] exclusive=x flag and RemoteRepository
• [x] "Initializing cache..." does not save progress
• [x] I/O Error on CIFS Mounts?
• [x] undefined symbol: EVP_aes_256_ctr error when running attic
• [x] draw a picture from the internals documentation
• [x] How to list files corresponding to corrupted data segments?
• [x] attic-mount misses files that can be extracted through attic-extract
• [x] PermissionError
• [x] setup.py: seek /opt/pkg (pkgsrc OS X) for openssl
• [X] Unable to download precompiled binaries
• [X] Use environment variable to set the tmp directory
• [x] msgpack.exceptions.UnpackValueError
• [x] attic check with ATTIC_PASSPHRASE -- getpass.py/terminal issue
• [x] non-zero exit status -11
• [x] SSH and Ctrl-C leave attic serve instances alive and prevent further upload
• [x] Typo: fileystem -> filesystem
• [x] Parity check/recover considered?
• [x] Can't subscribe to mailing list
• [x] Installation problems on CentOS 6.7
• [x] Locally backup S3 buckets
• [x] Tests failing on Gentoo Linux
• [x] Debian Jessie installation instructions for pip3
• [x] acl_use_local_uid_gid has suspicious code
• [x] Cache location
• [x] Data integrity error after running out of disk space
• [x] OverflowError when attempting to create a backup
• [x] attic crashed with "malloc failed"
• [x] ValueError on attic check --repair
• [x] Error with UnicodeDecode
• [x] Conversion from rdiff-backup
• [x] // , Installation fails on CEntOS 7.1.
• [x] // , gpg key for attic-backup.org
• [x] Default repository
• [x] low-impact - lrucache code appears to leak fds without closing them
• [x] Optimization: asynchronous writeback
• [x] repository: two minor improvements
• [x] Tiny fixes: printf error messages and gitignore
• [x] attic remote backup ends in broken pipe
• [x] attic/_version.py from github releases change over time
• [x] Attic segfaults
• [x] Let attic take a remote path option for cases where attic cannot be installed in standard places (like rsync, git, etc)
• [x] Honor nodump flag
• [x] osx pip3 install Attic - command not found
• [x] Allow to ignore file by size
• [x] Make the script for automatic backup finding the hostname
• [x] --include feature
• [x] OverflowError: value too large to convert to int
• [x] Undocumented standalone binaries
• [x] Undocumented environmente variables
• [x] Restore remote data inplace with deduplication or rsync algorithm
• [x] Add --files-from and --files-from0 options
• [x] create flag to print backed up files
• [x] "Failed to acquire write lock" FUSE
• [X] int32 chunk reference counter, is it a problem?
• [x] merge multiple repos into one
• [x] Exit gracefully when output pipe is broken
• [x] Performance: attic create stalls while index is read back
• [x] typo in docs theme css
• [x] st_mtime_ns precision or rounding related test failure
• [x] memory usage is too high
• [x] Question: list files modified/added in archive
• [x] Exception: hashindex_set failed (malloc failed)
• [x] Exception raised during create
• [x] Trailing slash on "init" confuses attic
• [x] Add disk space preview command
• [x] speed / compressed tradeoff parameter?
• [x] Doc symlinks are not followed on attic create.
• [x] sigusr1 / sigint signal handler to print current file infos
• [x] implement --timestamp, iso8601-like utc timestamp string or reference file
• [x] simple sparse file support, fixes #256
• [x] crc32 computation partly redundant?
• [x] use posix_fadvise to avoid spoiling the OS cache
• [x] Documentation for 'attic serve'
• [x] attic serve is not documented on attic-backup.org
• [x] Question about subsequent backups, and about pruning
• [x] improve and simplify key modes
• [x] potential counter/IV uniqueness issue
• [x] use poly1305-AES for faster MACs
• [x] attic create -n/--dry-run
• [x] ACLs do not work when using attic mount
• [x] Data integrity error due to index mismatch for key
• [x] Specify the date and time of the backup
• [x] Remove a path prefix on backup and/or restore
• [x] Specify the "extract to" location
• [x] improve restores by posix_fallocate
• [x] attic does not backup UNIX domain sockets
• [x] upgrade versioneer so it is pep440 compliant
• [x] handle sparse files
• [x] Adds a note about the ACCDCC2D signing key to the installation documentation
• [x] Fix test setup for 32bit platforms, partial fix for #196
• [x] wrong code in /bin/attic script
• [x] make system more responsive by using the fadvise DONTNEED
• [x] implement attic rename repo::oldname newname
• [x] implemented create --progress
• [x] own lrucache?
• [x] catch all Exceptions, transmit traceback back from remote
• [x] fix some exception handlers
• [x] fix noatime mode, fixes #243
• [x] atime preservation code is broken
• [x] Refactor 'list' and 'prune' operations on remote repositories to run on the server side
• [x] fix misleading hint the fuse ImportError handler gave, fixes #237
• [x] misleading error msg when importing fuse support fails
• [x] give specific path to xattr.is_enabled(), disable symlink setattr call ...
• [x] Traceback when running 'check --repair' on a repository
• [x] check --last N
• [x] avoid defect python-msgpack releases, fixes #171, fixes #185
• [x] check unpacked data from RPC for tuple type and correct length, fixes #127
• [x] improve on ready-only targets, see issue #202
• [x] Delete repo
• [x] better attic create -v output
• [x] add global option --no-cache-files to lower memory consumption
• [x] fix traceback when trying to do unsupported passphrase change, fixes #189
• [x] datetime does not like the year 10.000, fixes issue #139
• [x] Implement mid-file checkpoints?
• [x] improve exception handling / diagnosis
• [x] helping out with merge reviews and issue triage
• [x] compatibility policy / support timeframe
• [x] cleanup crypto.pyx, make it easier to adapt to other modes
• [x] OpenSSL's EVP_* interface not used correctly?
• [x] version number creates wrong impression
• [x] hash performance sha256 vs. sha512
• [x] stdin and stdout support
• [x] Misc. Fixes / Improvements
• [x] Error while creating backup - 'Repository' object has no attribute 'segments'
• [x] Document source of python 3.4 libraries bundled with releases
• [x] More user friendly message if repository is on read-only storage
• [x] attic hangs when initializing repository on a cifs mount
• [x] Attic mount dies (Input/output error) when backgrounded
• [x] Win32
• [x] Extra debug information for 'fread failed'
• [x] attic create fails with "assert segments[segment] == 0"
• [x] How to delete or rename a repository
• [x] add more .c files to .gitigore
• [x] Retry on Input/Output error
• [x] Changing passphrase fails: AttributeError: 'PassphraseKey' object has no attribute 'change_passphrase'
• [x] Changing/removing keyfile password?
• [x] FUSE: reflect deduplication in allocated blocks
• [x] attic does not respect XDG_CACHE_HOME
• [x] Creating archive with a slash breaks mount option
• [x] Suggestion: attic check --last N
• [x] extended attributes in 0.14 / linux
• [x] KeyError: 'cache'
• [x] attic check --repository-only reports "Error reading segment ###" at different locations
• [x] Pull mode
• [x] Enhancement: Add archive diff command
• [x] Only allow whitelisted RPC calls in server mode
• [x] Some file operations not working correctly in mounted repo
• [x] Issue with modification dates of parent directories in mounted archives
• [x] Traffic needed for 'list' and 'prune' operations on a remote repository increases linearly with time
• [x] Integrity error: 300k files 100GB of data.
• [x] Restrict "serve" command to "create" operation only
• [x] Traceback on a frequent backup
• [x] list changed files in info
• [x] Attic leaks file descriptors during checkpoint
• [x] document that multiple clients can backup to a single remote
• [x] document Internals better
• [x] document better different dependencies
• [x] document that hardlinks and symlinks are supported
• [x] Normalize source/exclude paths before matching
• [x] Suggestion: attic info --latest
• [x] Suggestion: Add a comment string to an archive
• [x] Suggestion: Mark a specific archive as "never prune"
• [x] Error backing up a changing filesystem
• [x] Implement --exclude-if-present
• [x] Specifying date or ability to change it.
• [x] attic list bombs with "ValueError: year is out of range"
• [x] Mounting a entire repo is not working
• [x] Support for Amazon S3?
• [x] pip install on Ubuntu 14.04 fails
• [x] FUSE 2.8.0 required?
• [x] Document that hard links work
• [x] Prune help unclear about what it will delete
• [x] Attic doesn't handle connections issues when using remote repo
• [x] --exclude-if-present
• [x] TypeError when trying to connect to a remote repository
• [x] Cache is newer than repository, refusing to continue
• [x] UnicodeEncodeError
• [x] cifs destination exception: listxattr failed
• [x] attic freezes sometimes
• [x] Subkeys
• [x] Add a interface to report current progress
• [x] Include other compressors like XZ for smaller size and LZO for speed
• [x] The "self contained" builds are missing the llfuse dependency
• [x] Add bandwidth limit support
• [x] Slow backup speed (~12MB/s)
• [x] Support custom ssh binary / ssh options?
• [x] description of exclude wildcards
• [x] Backup speed getting worse after cleanup
• [x] Copy repository to secondary location
• [x] Fixed buffer size a waste of resources/overhead?
• [x] Speed up directory scanning by ordering by inode
• [x] options to not compare inode numbers, and disable acl / xattr
• [x] Re-encrypt a repository with a new key
• [x] Convert unencrypted repositories to encrypted repositories
• [x] Document at "Usage->Attic" that init remote need installation on remote
• [x] create directories for mount points
• [x] docs bounty: let's crowdfund the attic docs wrinting ( 100 usd is my personal pledge. )
• [x] Poor FUSE performance
• [x] Windows Support
• [x] Attic and large files
• [x] Documentation on how I used Attic Backup (full system backup)
• [x] print current status/filename on SIGUSR1 in non-verbose mode
• [x] More informative "attic create -v" output
• [x] Document exit code(s)
• [x] Use logging lib for messages
• [x] Document known file cache and add an option to disable it
• [x] Add option to disable file chunks cache to reduce memory usage
• [x] Document how repository encryption works
• [x] Document how deduplication works
• [x] Document repository format
• [x] BlockingIOError in verbose mode
• [x] Archive stdin or fifo
• [x] config file whis tasks

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG / ISSUE

System information. For client/server mode post info for both machines.

Using an external drive USB drive on a Raspberry Pi 4, the partition is btrfs formatted.

Your borg version (borg -V).

borg 1.1.9 (Raspbian distribution package)

Operating system (distribution) and version.

Raspbian 4.19.88-v8+ #1284 SMP PREEMPT Wed Dec 11 13:57:50 GMT 2019 aarch64 GNU/Linux

Hardware / network configuration, and filesystems used.

Raspberry Pi 4, 4 GB, plenty of free space on SD card and external USB drive.

How much data is handled by borg?

None so far, repo creation fails.

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg init --encryption=repokey </local/path>

Describe the problem you're observing.

Enter new passphrase: 
Enter same passphrase again: 
Do you want your passphrase to be displayed for verification? [yN]: 
Fatal Python error: Bus error

Current thread 0xf7e898e0 (most recent call first):
  File "/usr/lib/python3/dist-packages/borg/crypto/file_integrity.py", line 75 in write
  File "/usr/lib/python3/dist-packages/borg/crypto/file_integrity.py", line 29 in write
  File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 38 in pack
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 580 in write_index
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 443 in commit
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 275 in do_init
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 154 in wrapper
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387 in run
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455 in main
  File "/usr/bin/borg", line 11 in <module>
Bus error

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, it is fully reproducible. The same happens if I use a a Ubuntu client running Borg 1.1.5 accessing the Raspberry Pi via ssh.

Encryption argument does not change the observed behaviour.

Include any warning/errors/backtraces from the system logs

Nothing relevant in syslog etc. drive seems healthy.

Repo content:

$ ls -lR
.:
total 16
-rw------- 1 user user  73 Dec 22 02:41 README
-rw------- 1 user user 700 Dec 22 02:41 config
drwx------ 1 user user   2 Dec 22 02:41 data
-rw------- 1 user user   0 Dec 22 02:41 hints.1.tmp
drwx------ 1 user user  52 Dec 22 02:41 lock.exclusive
-rw------- 1 user user  49 Dec 22 02:41 lock.roster
-rw------- 1 user user  16 Dec 22 02:41 nonce

./data:
total 0
drwx------ 1 user user 4 Dec 22 02:41 0

./data/0:
total 8
-rw------- 1 user user 530 Dec 22 02:41 0
-rw------- 1 user user  17 Dec 22 02:41 1

./lock.exclusive:
total 0
-rw------- 1 user user 0 Dec 22 02:41 [email protected]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.1.16

Operating system (distribution) and version.

Alpine Linux "edge" (rolling release)

Hardware / network configuration, and filesystems used.

• Borgbackup metadata (cache and configuration) is in a btrfs partition on the host machine
• Data to back up (LXC/LXD container metadata and filesystems) is ext4 on Ceph RBD block devices

How much data is handled by borg?

Original data: ~ 500GB (filesystems and metadata of ~20 containers) After deduplication: ~10GB per backup run in 20 archives (1 archive per container) Total (all archives): ~3TB

Describe the problem you're observing.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I use borgbackup to take daily backups of LXC/LXD containers on a dedicated container host. Each backup run calls

borg create "$ARCHIVE_URL::$CONTAINER-{now:%Y-%m-%dT%H:%M:%S}" . --stats

in the toplevel directory of a snapshot of the container file system (snapshots are created outside of borgbackup using Ceph tools). $ARCHIVE_URL is the SSH URL of an archive on a remote borgbackup server, and $CONTAINER is the container ID.

The first call to borg create in this backup run fails with a segfault on about every second run (see below for the backtrace).

The subsequent calls to back up different containers in the same run have had no errors so far.

The segfault is not specific to one container: If I change the order of the containers for backup I get the same error for a different container - but always for the first borg create call of a backup run.

Include any warning/errors/backtraces from the system logs

The crash occurs always in the same line in python code:

Fatal Python error: Segmentation fault

Current thread 0x00007fe4c6126b48 (most recent call first):
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 740 in write_archive_index
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 736 in fetch_and_build_idx
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 824 in create_master_idx
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 870 in sync
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 474 in __init__
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 379 in local
  File "/usr/lib/python3.9/site-packages/borg/cache.py", line 387 in __new__
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 586 in do_create
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 177 in wrapper
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 4622 in run
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 4690 in main
  File "/usr/bin/borg", line 33 in <module>
Segmentation fault

Line 740 in /usr/lib/python3.9/site-packages/borg/cache.py is simply

compact_chunks_archive_saved_space += chunk_idx.compact()

where chunk_idx is an instance of ChunkIndex, which is implemeted in C as far as I can see. So I suspect a bug in chunk_idx.compact()

python's io.BufferedWriter sizes its buffer based on st_blksize

If the write fits in this buffer, then it's possible none (or only some) of the data from idx.write() has been written through to fileno()

Note:1.2-maint counterpart of #7251.

Note: I see that the contribution guidelines ask for PRs only against master (leaving backports up to you maintainers) but in this case I want to publicly post a commit based on 1.2-maint to this repo, so that @mweinelt can have the nixpkgs derivation cherry-pick that commit for testing. I think we understand what's going on, but the failure is being seen on nixos autobuilders that use ZFS for $TEMP, and I don't personally have anything with that configuration, or direct access to those build machines.

Fixes #7250

python's io.BufferedWriter sizes its buffer based on st_blksize

If the write fits in this buffer, then it's possible none (or only some) of the data from idx.write() has been written through to fileno()

Fixes #7250

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Bug/Regression

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

Operating system (distribution) and version.

NixOS Unstable

Hardware / network configuration, and filesystems used.

ZFS Root

How much data is handled by borg?

n/a

Full borg commandline that lead to the problem (leave away excludes and passwords)

n/a

Describe the problem you're observing.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Running the test suite on ZFS breaks on test_size_on_disk_accurate since https://github.com/borgbackup/borg/commit/8ddfd94f0904449175151faed67f9640710b6454.

ZFS has a commit interval of 5 seconds, so a manual flush might be required for the size to be reflected instantly.

Include any warning/errors/backtraces from the system logs

_______________ HashIndexSizeTestCase.test_size_on_disk_accurate _______________
[gw42] linux -- Python 3.10.9 /nix/store/ar76b3831ydkmfhf0r9v3wbma5i61iz2-python3-3.10.9/bin/python3.10
self = <borg.testsuite.hashindex.HashIndexSizeTestCase testMethod=test_size_on_disk_accurate>

    def test_size_on_disk_accurate(self):
        idx = ChunkIndex()
        for i in range(1234):
            idx[H(i)] = i, i**2, i**3
        with tempfile.NamedTemporaryFile() as file:
            idx.write(file)
            size = os.path.getsize(file.fileno())
>       assert idx.size() == size
E       assert 90350 == 0
E        +  where 90350 = <built-in method size of borg.hashindex.ChunkIndex object at 0xffffeea51b30>()
E        +    where <built-in method size of borg.hashindex.ChunkIndex object at 0xffffeea51b30> = <borg.hashindex.ChunkIndex object at 0xffffeea51b30>.size

/nix/store/0ppx77a4xk6nij2p1klgsnksmrfnx9lj-borgbackup-1.2.3/lib/python3.10/site-packages/borg/testsuite/hashindex.py:202: AssertionError

When processing a pipe/stdin, borg create uses:

• uid2user(0) as --stdin-user default
• gid2group(0) as --stdin-group default

This is only so it can be given to process_pipe (which then does the mapping backwards).

This is problematic if the platform does not have a uid 0 or gid 0, like cygwin.

Also, user and group are optional, so we do not really need them. Guess we could also make uid and gid optional in archived items.

Alternatively, we could use the current effective uid/gid maybe to make sure we have something that actually exists and can be used as "owning" user/group.

When mapping such an item into a filesystem (borg mount), we'll need something for the uid/gid (it could also fall back to self.default_uid / self.default_gid).

Have you checked borgbackup docs, FAQ, and open Github issues?

YES

Is this a BUG / ISSUE report or a QUESTION?

1 BUG 1 ENHANCEMENT-REQUEST

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.2.1

Operating system (distribution) and version.

Debian bullseye x64

Hardware / network configuration, and filesystems used.

AMD x64 processor, 8GB RAM, 1Gbps ethernet 3 x 8TB drives layered as LUKS, LVM, BTRFS RAID1

How much data is handled by borg?

Repo1: 22 TB (un-deduplicated), 2TB (deduplicated), 105 archives in repo (most files are hardlinked together).

Full borg commandline that lead to the problem (leave away excludes and passwords)

See bash script below.

Describe the problem you're observing.

1. BUG: 'borg diff' does not report changed mtime for files within a repository, even though 'borg list --json-lines' shows that the file metadata have changed. (Other metadata than mtime may have same problem...haven't checked).
2. ENHANCEMENT REQUEST: 'borg list --json-lines' sometimes renders attributes (eg size, mtime) in different order; this causes a 'diff' of the directory entry against another instance to to mis-match, even though the attribute values are identical. Proposed change: please render attributes in a stable order (alphabetic sort order is suggested, but what is important is that it be the same order across all archives within a single repository).

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes. See script which follows:

#!/bin/bash

# This script demonstrates that 'borg diff' (version 1.2.1, at least) does
# not report a difference in mtime of a file.
# This issue may also apply to atime and ctime as well, and also to
# directories, and to other metadata attributes (such as permissions),
# but isn't tested by this script.

dir=$(mktemp -d)
pushd "$dir" > /dev/null

borg init -e none repo

mkdir work
echo "hello" > work/hello
borg create repo::archive01 work
borg list --json-lines repo::archive01 > list1
sleep 4      
touch work/hello
borg create repo::archive02 work
borg list --json-lines repo::archive02 > list2

# Enhancement request: please emit attributes (eg, mtime, size) in same
# order each time, so that a diff of unchanged elements (eg, a directory
# entry) will match:
# Note if 'sleep 4' above is change to 'sleep 2', the 'borg list --json-lines'
# attributes are rendered in same order.  Why?
diff list1 list2

# Bug: borg diff does not report the change in mtime, even though it is
# visible in 'borg list'
borg diff repo::archive01 archive02

popd

Related to #7232 and #6151.

Guess we can rely on hostname being pure ascii, right?

But how about user and group names (for now, but also for past borg1 archives)?

• pure ascii? (guess ideally/usually yes, but we can not rely on it)
• pure unicode without surrogate-escape?
• unicode with surrogate-escapes to represent arbitrary bytestrings?