From 1121b9e07310ad935dccb36d2678b56e9fab777c Mon Sep 17 00:00:00 2001 From: Marc Vertes Date: Mon, 7 Oct 2024 22:59:29 +0200 Subject: write to external repo --- README.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 68d9a2b..cad55d3 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Incremental encrypted backup system -## design v0 +## Archive v0 1. cksum original (sha256) 2. compress (gzip) @@ -22,7 +22,7 @@ Problems: - dedup occurs only for append only files. The same chunk content will lead to a different hmac if located at a different offset. -## design v1 +## Archive v1 - chunk before compression - name chunks from checksum of uncompressed/unencrypted data (invariant => allow dedup). @@ -37,7 +37,7 @@ Problems: with a foreign key (different user), which would a user to download a block which he could not decrypt. -## design v2 +## Archive v2 Each user has a fixed unique id: random 96 bits (12 bytes). This id is added to the content of each block / file prior to compute the invariant checksum @@ -51,8 +51,22 @@ Problems: - in this design, and all previous ones, there is no way to disgard data in an archive. For example, tarsnap does not allow to suppress data. +## Repository + +A repo is associated with a single id/key tuple, ensuring a deduplification space, +i.e. a unique `chunks` directory. + +Each backup is denoted by its archive index $host:$dir:$date + +$dir is the rootdir of the backuped files. + +A repo entrypoint is its current index, containing the list of $host:$dir:$date. + ## Roadmap +- p2p storage +- chunker based on rolling hash (instead of fixed size) + disgarded: - encode checksums in base64 instead of hex. Wrong idea: incompatible with case insensitive filesystems (macos). @@ -68,6 +82,7 @@ disgarded: ## References - tarsnap: https://www.tarsnap.com https://github.com/tarsnap/tarsnap -- chunker: https://github.com/karinushka/chunker +- tarsnap chunker in Go: https://github.com/karinushka/chunker - borg: https://borgbackup.org - rclone: https://rclone.org +- restic: https://restic.readthedocs.io/en/v0.2.0/Design/ -- cgit v1.2.3