Hello community, here is the log from the commit of package duplicity for openSUSE:Factory checked in at 2020-09-30 19:54:58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/duplicity (Old) and /work/SRC/openSUSE:Factory/.duplicity.new.4249 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Package is "duplicity" Wed Sep 30 19:54:58 2020 rev:61 rq:838693 version:0.8.16 Changes: -------- --- /work/SRC/openSUSE:Factory/duplicity/duplicity.changes 2020-07-29 17:23:24.748710748 +0200 +++ /work/SRC/openSUSE:Factory/.duplicity.new.4249/duplicity.changes 2020-09-30 19:55:03.100820126 +0200 @@ -1,0 +2,10 @@ +Wed Sep 30 00:25:50 UTC 2020 - Michael Gorse <mgorse@suse.com> + +- Update to version 0.8.16: + + Merged in convert2md - Convert README's to markdown. + + Merged in s3-boto3-region-and-endpoint - after fixes. + + Merged in lazy init for Boto3 network connections. + + Merged in OutlawPlz:paramiko-progress. + + Merged in s3-unfreeze-all. + +------------------------------------------------------------------- Old: ---- duplicity-0.8.15.tar.gz New: ---- duplicity-0.8.16.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ duplicity.spec ++++++ --- /var/tmp/diff_new_pack.IHSTMy/_old 2020-09-30 19:55:03.836820784 +0200 +++ /var/tmp/diff_new_pack.IHSTMy/_new 2020-09-30 19:55:03.836820784 +0200 @@ -17,7 +17,7 @@ Name: duplicity -Version: 0.8.15 +Version: 0.8.16 Release: 0 Summary: Encrypted bandwidth-efficient backup using the rsync algorithm License: GPL-3.0-or-later @@ -71,7 +71,7 @@ %files %license COPYING -%doc CHANGELOG README +%doc CHANGELOG README.md %{_bindir}/duplicity %{_bindir}/rdiffdir %{python3_sitearch}/duplicity ++++++ duplicity-0.8.15.tar.gz -> duplicity-0.8.16.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/CHANGELOG new/duplicity-0.8.16/CHANGELOG --- old/duplicity-0.8.15/CHANGELOG 2020-07-27 17:03:47.000000000 +0200 +++ new/duplicity-0.8.16/CHANGELOG 2020-09-29 18:06:38.000000000 +0200 @@ -1,3 +1,14 @@ +New in v0.8.16 (2020/09/29) +--------------------------- +* Revert "Merge branch 's3-boto3-region-and-endpoint' into 'master'" + - It broke existing setups since defaults were not supplied. +* Merged in convert2md - Convert README's to markdown +* Merged in s3-boto3-region-and-endpoint - after fixes. +* Merged in lazy init for Boto3 network connections +* Merged in OutlawPlz:paramiko-progress +* Merged in s3-unfreeze-all + + New in v0.8.15 (2020/07/27) --------------------------- * Fix bug #1887689 with patch from Matthew Barry diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/Changelog.GNU new/duplicity-0.8.16/Changelog.GNU --- old/duplicity-0.8.15/Changelog.GNU 2020-07-27 17:04:50.000000000 +0200 +++ new/duplicity-0.8.16/Changelog.GNU 2020-09-29 18:08:17.000000000 +0200 @@ -1,3 +1,29 @@ +2020-09-29 Kenneth Loafman <kenneth@loafman.com> + + * Prep for 0.8.16 + +2020-09-27 Kenneth Loafman <kenneth@loafman.com> + + * Merged in s3-unfreeze-all + +2020-09-25 Kenneth Loafman <kenneth@loafman.com> + + * Merged in OutlawPlz:paramiko-progress + +2020-09-24 Kenneth Loafman <kenneth@loafman.com> + + * Merged in lazy init for Boto3 network connections + +2020-07-31 Kenneth Loafman <kenneth@loafman.com> + + * Merged in convert2md - Convert README's to markdown + * Merged in s3-boto3-region-and-endpoint - after fixes. + +2020-07-28 Kenneth Loafman <kenneth@loafman.com> + + * Revert "Merge branch 's3-boto3-region-and-endpoint' into 'master'" + - It broke existing setups since defaults were not supplied. + 2020-07-27 Kenneth Loafman <kenneth@loafman.com> * Prep for 0.8.15 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/PKG-INFO new/duplicity-0.8.16/PKG-INFO --- old/duplicity-0.8.15/PKG-INFO 2020-07-27 17:18:41.000000000 +0200 +++ new/duplicity-0.8.16/PKG-INFO 2020-09-29 18:28:41.000000000 +0200 @@ -1,6 +1,6 @@ Metadata-Version: 2.1 Name: duplicity -Version: 0.8.15 +Version: 0.8.16 Summary: Encrypted backup using rsync algorithm Home-page: http://duplicity.nongnu.org/index.html Author: Ben Escoto <ben@emrose.org> @@ -8,28 +8,33 @@ Maintainer: Kenneth Loafman <kenneth@loafman.com> Maintainer-email: kenneth@loafman.com License: UNKNOWN -Description: INSTALLATION: +Description: # INSTALLATION Thank you for trying duplicity. To install, run: - python setup.py install + ``` + python setup.py install + ``` The build process can be also be run separately: - python setup.py build + ``` + python setup.py build + ``` The default prefix is /usr, so files are put in /usr/bin, /usr/share/man/, etc. An alternate prefix can be specified using the --prefix=<prefix> option. For example: - python setup.py install --prefix=/usr/local - export PYTHONPATH='/usr/local/lib/python.x/site-packages/' - /usr/local/bin/duplicity -V + ``` + python setup.py install --prefix=/usr/local + export PYTHONPATH='/usr/local/lib/python.x/site-packages/' + /usr/local/bin/duplicity -V` + ``` + # REQUIREMENTS - REQUIREMENTS: - - * Python 2.7, 3.6, 3.7 + * Python 2.7, or 3.5 to 3.9 * librsync v0.9.6 or later * GnuPG for encryption * fasteners 0.14.1 or later for concurrency locking @@ -38,6 +43,7 @@ * Boto 2.0 or later for single-processing S3 or GCS access (default) * Boto 2.1.1 or later for multi-processing S3 access * Boto 2.7.0 or later for Glacier S3 access + * Boto3 0.3.1 or later If you install from the source package, you will also need: @@ -45,37 +51,12 @@ * librsync development files, normally found in module 'librsync-dev'. - A NOTE ON GnuPGInterface.py AND MULTIPLE GPG PROCESSES: - - GnuPGInterface is used to access GPG from duplicity. The original - works quite well and has no bugs, however, we have patched the one - used in duplicity. Why? Duplicity is not perfect, yet, and has a - problem when handling long chains of incremental backup or restore - operations. The problem is that the waitpid() call only happens - after all the iterations complete, and with a long chain, that can - be a long while. Unless the waitpid() call is made, the child process - remains active. Duplicity's GnuPGInterface is patched to start an - immediate threaded waitpid() for each GPG task, thus harvesting the - task and freeing it's resources in a timely manner. This does not - affect the operation of duplicity, merely frees resources on time. - - Why the note? Some package maintainers remove duplicity's GnuPGInterface - in error, obviously unknowing of this issue and patch duplicity to use - the old unmaintained unpatched GnuPGInterface interface again. - So, if you have the problem that lots of GPG tasks are hanging around, - check and see if this has been done in your distro, and if so, report this - matter as a bug to the distro or package maintainer. - - As of october 2012 we pull the handbrake and refactor our code and rename - the class to gpginterface in the hope that package maintainers will stumble - over it and stop this problematic behaviour for good. - - DEVELOPMENT + # DEVELOPMENT For more information on downloading duplicity's source code from the code repository and developing for duplicity, see README-REPO. - HELP: + # HELP For more information see the duplicity home page at: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README new/duplicity-0.8.16/README --- old/duplicity-0.8.15/README 2020-06-21 17:06:38.000000000 +0200 +++ new/duplicity-0.8.16/README 1970-01-01 01:00:00.000000000 +0100 @@ -1,76 +0,0 @@ -INSTALLATION: - -Thank you for trying duplicity. To install, run: - - python setup.py install - -The build process can be also be run separately: - - python setup.py build - -The default prefix is /usr, so files are put in /usr/bin, -/usr/share/man/, etc. An alternate prefix can be specified using the ---prefix=<prefix> option. For example: - - python setup.py install --prefix=/usr/local - export PYTHONPATH='/usr/local/lib/python.x/site-packages/' - /usr/local/bin/duplicity -V - - -REQUIREMENTS: - - * Python 2.7, 3.6, 3.7 - * librsync v0.9.6 or later - * GnuPG for encryption - * fasteners 0.14.1 or later for concurrency locking - * for scp/sftp -- python-paramiko - * for ftp -- lftp version 3.7.15 or later - * Boto 2.0 or later for single-processing S3 or GCS access (default) - * Boto 2.1.1 or later for multi-processing S3 access - * Boto 2.7.0 or later for Glacier S3 access - -If you install from the source package, you will also need: - - * Python development files, normally found in module 'python-dev'. - * librsync development files, normally found in module 'librsync-dev'. - - -A NOTE ON GnuPGInterface.py AND MULTIPLE GPG PROCESSES: - -GnuPGInterface is used to access GPG from duplicity. The original -works quite well and has no bugs, however, we have patched the one -used in duplicity. Why? Duplicity is not perfect, yet, and has a -problem when handling long chains of incremental backup or restore -operations. The problem is that the waitpid() call only happens -after all the iterations complete, and with a long chain, that can -be a long while. Unless the waitpid() call is made, the child process -remains active. Duplicity's GnuPGInterface is patched to start an -immediate threaded waitpid() for each GPG task, thus harvesting the -task and freeing it's resources in a timely manner. This does not -affect the operation of duplicity, merely frees resources on time. - -Why the note? Some package maintainers remove duplicity's GnuPGInterface -in error, obviously unknowing of this issue and patch duplicity to use -the old unmaintained unpatched GnuPGInterface interface again. -So, if you have the problem that lots of GPG tasks are hanging around, -check and see if this has been done in your distro, and if so, report this -matter as a bug to the distro or package maintainer. - -As of october 2012 we pull the handbrake and refactor our code and rename -the class to gpginterface in the hope that package maintainers will stumble -over it and stop this problematic behaviour for good. - -DEVELOPMENT - -For more information on downloading duplicity's source code from the -code repository and developing for duplicity, see README-REPO. - -HELP: - -For more information see the duplicity home page at: - - http://www.nongnu.org/duplicity - -or post to the mailing list at - - http://mail.nongnu.org/mailman/listinfo/duplicity-talk/ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README-LOG new/duplicity-0.8.16/README-LOG --- old/duplicity-0.8.15/README-LOG 2020-06-02 21:10:41.000000000 +0200 +++ new/duplicity-0.8.16/README-LOG 1970-01-01 01:00:00.000000000 +0100 @@ -1,28 +0,0 @@ -Duplicity's log output is meant as a means of reporting status and information -back to the caller. This makes the life of a frontend writer much easier. - -The format consists of a stream of stanzas, each starting with a keyword and -some arguments, an optional suggested user text (each line of which starts with -". ") and ending with an endline. Like so: - -KEYWORD 3\n -. Hello! All work and now play make Jack a...\n -. dull boy.\n -\n - -You can get this output by specifying either --log-fd or --log-file. - -Currently, duplicity writes out status messages like WARNING or ERROR followed -by a message number. Each message number uniquely identifies a particular -warning or error so the frontend can take special action. For example, an ERROR -of 2 is a command line syntax error. Each message type has its own namespace -(i.e. a WARNING of 2 means something different than an ERROR of 2). A number -of 1 is a generic, non-unique number for messages without their own code. - -For a list of current numbers, see log.py - -HINTS FOR CONSUMERS -Ignore any extra arguments on the keyword line. -Ignore any stanzas that have a keyword you don't recognize. -Ignore any lines in a stanza that start with a character you don't know. - diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README-LOG.md new/duplicity-0.8.16/README-LOG.md --- old/duplicity-0.8.15/README-LOG.md 1970-01-01 01:00:00.000000000 +0100 +++ new/duplicity-0.8.16/README-LOG.md 2020-07-31 22:21:08.000000000 +0200 @@ -0,0 +1,33 @@ +# duplicity's Log Output + +Duplicity's log output is meant as a means of reporting status and information +back to the caller. This makes the life of a frontend writer much easier. + +The format consists of a stream of stanzas, each starting with a keyword and +some arguments, an optional suggested user text (each line of which starts with +". ") and ending with an endline. Like so: + +>>> +KEYWORD 3\n +. Hello! All work and now play make Jack a...\n +. dull boy.\n +\n +>>> + +You can get this output by specifying either *--log-fd* or *--log-file*. + +Currently, duplicity writes out status messages like WARNING or ERROR followed +by a message number. Each message number uniquely identifies a particular +warning or error so the frontend can take special action. For example, an ERROR +of 2 is a command line syntax error. Each message type has its own namespace +(i.e. a WARNING of 2 means something different than an ERROR of 2). A number +of 1 is a generic, non-unique number for messages without their own code. + +For a list of current numbers, see log.py + +## HINTS FOR CONSUMERS + +1. Ignore any extra arguments on the keyword line. +2. Ignore any stanzas that have a keyword you don't recognize. +3. Ignore any lines in a stanza that start with a character you don't know. + diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README-REPO new/duplicity-0.8.16/README-REPO --- old/duplicity-0.8.15/README-REPO 2020-06-19 20:18:24.000000000 +0200 +++ new/duplicity-0.8.16/README-REPO 1970-01-01 01:00:00.000000000 +0100 @@ -1,28 +0,0 @@ -REPO README - Notes for people checking out of Launchpad (bzr) --------------------------------------------------------------- - -------------------------- -Getting duplicity to run: -------------------------- - -By the numbers: -1) Do the checkout to a location called $DUP_ROOT: - [for the stable branch] - bzr branch lp:duplicity $DUP_ROOT - or - [for another branch, replace X with series number] - bzr branch lp:~duplicity-team/duplicity/0.X-series $DUP_ROOT -2) cd $DUP_ROOT/duplicity -3) Run "setup.py build_ext" to create _librsync.so -4) cd .. -5) Run "PYTHONPATH=$DUP_ROOT bin/duplicity -V". You will see - "duplicity $version" instead of the normal version number. - Versioning comes during the release. - -Use PYTHONPATH to set the path each time that you use the binaries: - -PYTHONPATH=$DUP_ROOT bin/duplicity - -or - -PYTHONPATH=$DUP_ROOT bin/rdiffdir diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README-REPO.md new/duplicity-0.8.16/README-REPO.md --- old/duplicity-0.8.15/README-REPO.md 1970-01-01 01:00:00.000000000 +0100 +++ new/duplicity-0.8.16/README-REPO.md 2020-07-31 16:41:44.000000000 +0200 @@ -0,0 +1,23 @@ +# REPO README - Notes for people checking out of GitLab (git) + +## Getting duplicity to run: + +By the numbers: +1. Do the checkout to a location called $DUP_ROOT: + - `git clone git@gitlab.com:duplicity/duplicity.git $DUP_ROOT` or + - `git clone https://gitlab.com/duplicity/duplicity.git $DUP_ROOT` +2. Build the extension module + - `cd $DUP_ROOT/duplicity` + - `setup.py build_ext` + - `cd ..` +5. Run `PYTHONPATH=$DUP_ROOT bin/duplicity -V`. You will see + "duplicity $version" instead of the normal version number. + Versioning comes during the release. + +Use PYTHONPATH to set the path each time that you use the binaries: + +`PYTHONPATH=$DUP_ROOT bin/duplicity` + +or + +`PYTHONPATH=$DUP_ROOT bin/rdiffdir` diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README-TESTING new/duplicity-0.8.16/README-TESTING --- old/duplicity-0.8.15/README-TESTING 2020-06-02 21:10:41.000000000 +0200 +++ new/duplicity-0.8.16/README-TESTING 1970-01-01 01:00:00.000000000 +0100 @@ -1,102 +0,0 @@ -# Testing duplicity - -## Introduction -Duplicity's tests are unit tests contained in the /testing folder of the main repository. - -## Running tests on your branch -The recommended approach is to test duplicity using Docker, to ensure that you are running tests in a known-good -environment. This also creates additional containers to test backends. You can run tests on your branch as follows: -a. Install Docker -b. cd [BRANCH FOLDER]/testing/docker/ -c. ./build-duplicity_test.sh to build the required Docker containers and copy your branch into - the duplicity_test container. -d. ./setup.sh to start the containers and open a shell inside your branch folder in duplicity_test. -e. cd testing/ -f. ./run-tests -g. When you are finished, exit the Docker container and run ./teardown.sh to delete the containers. - -Please test your branch using this method and ensure all tests pass before submitting a merge request. - -The decorator @unittest.expectedFailure can be used to commit a known-failing test case without breaking the test suite, -for example to exhibit the behaviour in a bug report before it has been fixed. - -## Manual testing and running individual tests -As to see in the following sketch, there are several levels of testing duplicity and each can be used directly. - - ┌─────────────────────┐ - │ docker image │ - ├─────────────────────┴────┐ - │ │ - │ ┌──────────────────┐ │ - │ │ run-tests/tox │ │ - │ └──────────────────┘ │ - │ │ │ - │ ▼ │ - │ ┌──────────────────┐ │ - │ │ unittests │ │ - │ └──────────────────┘ │ - │ │ │ - │ ▼ │ - │ ┌──────────────────┐ │ - │ │ duplicity │ │ - │ └──────────────────┘ │ - │ │ - └──────────────────────────┘ - -1. Docker container -Even if you wish to run tests manually, we recommend that you do this inside the provided Docker container to ensure -that you have a clean and reproducible environment with all required dependencies for executing these. Please follow -steps a to d of the above section titled "Running tests on your branch". - -2. Using run-tests -The run-tests script mentioned above is a shortcut to running all tests with tox. This is the recommended approach to -run tests on your code. We recommend running this within the provided Docker container as mentioned above, but it is -possible to run this directly if you have installed all required dependencies (see "Dependencies for testing" below). - -3. Using tox -Tox is a generic virtualenv management and test command line tool that is used for checking your package installs -correctly with different Python versions and interpreters. It runs the tests in each of the environments that are -configured in the tox.ini file (see root folder of the repository). As mentioned, we recommend this is run inside the -provided Docker container. - -A tox run can be started simply by typing - -‘tox‘ - -from the main duplicity folder. - -You can run specific tests using: -‘tox -- [test filename][::TestClassName::test_method]‘ -For example: -‘tox -- testing/unit/test_selection.py‘ -or: -‘tox -- testing/unit/test_selection.py::MatchingTest::test_tuple_include‘ - -You can test against a single environment, e.g. -‘tox -e py27‘ - -Or stack these together, e.g. -‘tox -e py3 -- testing/unit/test_selection.py::MatchingTest::test_tuple_include‘ - -This is helpful, for example, if you are working on fixing a bug, but please do a full run-tests before submitting a -merge request. - -4. Testing directly using __setup.py__ -Assuming that your machine has all the required dependencies installed, you can start all the unit tests by simply typing - -‘setup.py test‘ - -## Dependencies for testing -If you should prefer to execute the tests locally without using Docker, see the Dockerfile in -testing/docker/duplicity_test/ -for requirements to correctly set up your environment. - -## Working with test coverage -Python makes it easy to determine how well the tests cover the source code. - -You first run the tests __under observation__ of the coverage script: -‘coverage run setup.py test‘ -After that, a report can be generated by the use of the command: -‘coverage html --omit="testing/*,/usr/*"‘ - -The report will be generated and stored in the folder htmlcov. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README-TESTING.md new/duplicity-0.8.16/README-TESTING.md --- old/duplicity-0.8.15/README-TESTING.md 1970-01-01 01:00:00.000000000 +0100 +++ new/duplicity-0.8.16/README-TESTING.md 2020-06-02 21:10:41.000000000 +0200 @@ -0,0 +1,102 @@ +# Testing duplicity + +## Introduction +Duplicity's tests are unit tests contained in the /testing folder of the main repository. + +## Running tests on your branch +The recommended approach is to test duplicity using Docker, to ensure that you are running tests in a known-good +environment. This also creates additional containers to test backends. You can run tests on your branch as follows: +a. Install Docker +b. cd [BRANCH FOLDER]/testing/docker/ +c. ./build-duplicity_test.sh to build the required Docker containers and copy your branch into + the duplicity_test container. +d. ./setup.sh to start the containers and open a shell inside your branch folder in duplicity_test. +e. cd testing/ +f. ./run-tests +g. When you are finished, exit the Docker container and run ./teardown.sh to delete the containers. + +Please test your branch using this method and ensure all tests pass before submitting a merge request. + +The decorator @unittest.expectedFailure can be used to commit a known-failing test case without breaking the test suite, +for example to exhibit the behaviour in a bug report before it has been fixed. + +## Manual testing and running individual tests +As to see in the following sketch, there are several levels of testing duplicity and each can be used directly. + + ┌─────────────────────┐ + │ docker image │ + ├─────────────────────┴────┐ + │ │ + │ ┌──────────────────┐ │ + │ │ run-tests/tox │ │ + │ └──────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌──────────────────┐ │ + │ │ unittests │ │ + │ └──────────────────┘ │ + │ │ │ + │ ▼ │ + │ ┌──────────────────┐ │ + │ │ duplicity │ │ + │ └──────────────────┘ │ + │ │ + └──────────────────────────┘ + +1. Docker container +Even if you wish to run tests manually, we recommend that you do this inside the provided Docker container to ensure +that you have a clean and reproducible environment with all required dependencies for executing these. Please follow +steps a to d of the above section titled "Running tests on your branch". + +2. Using run-tests +The run-tests script mentioned above is a shortcut to running all tests with tox. This is the recommended approach to +run tests on your code. We recommend running this within the provided Docker container as mentioned above, but it is +possible to run this directly if you have installed all required dependencies (see "Dependencies for testing" below). + +3. Using tox +Tox is a generic virtualenv management and test command line tool that is used for checking your package installs +correctly with different Python versions and interpreters. It runs the tests in each of the environments that are +configured in the tox.ini file (see root folder of the repository). As mentioned, we recommend this is run inside the +provided Docker container. + +A tox run can be started simply by typing + +‘tox‘ + +from the main duplicity folder. + +You can run specific tests using: +‘tox -- [test filename][::TestClassName::test_method]‘ +For example: +‘tox -- testing/unit/test_selection.py‘ +or: +‘tox -- testing/unit/test_selection.py::MatchingTest::test_tuple_include‘ + +You can test against a single environment, e.g. +‘tox -e py27‘ + +Or stack these together, e.g. +‘tox -e py3 -- testing/unit/test_selection.py::MatchingTest::test_tuple_include‘ + +This is helpful, for example, if you are working on fixing a bug, but please do a full run-tests before submitting a +merge request. + +4. Testing directly using __setup.py__ +Assuming that your machine has all the required dependencies installed, you can start all the unit tests by simply typing + +‘setup.py test‘ + +## Dependencies for testing +If you should prefer to execute the tests locally without using Docker, see the Dockerfile in +testing/docker/duplicity_test/ +for requirements to correctly set up your environment. + +## Working with test coverage +Python makes it easy to determine how well the tests cover the source code. + +You first run the tests __under observation__ of the coverage script: +‘coverage run setup.py test‘ +After that, a report can be generated by the use of the command: +‘coverage html --omit="testing/*,/usr/*"‘ + +The report will be generated and stored in the folder htmlcov. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/README.md new/duplicity-0.8.16/README.md --- old/duplicity-0.8.15/README.md 1970-01-01 01:00:00.000000000 +0100 +++ new/duplicity-0.8.16/README.md 2020-09-27 20:24:24.000000000 +0200 @@ -0,0 +1,57 @@ +# INSTALLATION + +Thank you for trying duplicity. To install, run: + +``` +python setup.py install +``` + +The build process can be also be run separately: + +``` +python setup.py build +``` + +The default prefix is /usr, so files are put in /usr/bin, +/usr/share/man/, etc. An alternate prefix can be specified using the +--prefix=<prefix> option. For example: + +``` +python setup.py install --prefix=/usr/local +export PYTHONPATH='/usr/local/lib/python.x/site-packages/' +/usr/local/bin/duplicity -V` +``` + +# REQUIREMENTS + + * Python 2.7, or 3.5 to 3.9 + * librsync v0.9.6 or later + * GnuPG for encryption + * fasteners 0.14.1 or later for concurrency locking + * for scp/sftp -- python-paramiko + * for ftp -- lftp version 3.7.15 or later + * Boto 2.0 or later for single-processing S3 or GCS access (default) + * Boto 2.1.1 or later for multi-processing S3 access + * Boto 2.7.0 or later for Glacier S3 access + * Boto3 0.3.1 or later + +If you install from the source package, you will also need: + + * Python development files, normally found in module 'python-dev'. + * librsync development files, normally found in module 'librsync-dev'. + + +# DEVELOPMENT + +For more information on downloading duplicity's source code from the +code repository and developing for duplicity, see README-REPO. + +# HELP + +For more information see the duplicity home page at: + + http://www.nongnu.org/duplicity + +or post to the mailing list at + + http://mail.nongnu.org/mailman/listinfo/duplicity-talk/ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/bin/duplicity.1 new/duplicity-0.8.16/bin/duplicity.1 --- old/duplicity-0.8.15/bin/duplicity.1 2020-07-27 17:18:42.000000000 +0200 +++ new/duplicity-0.8.16/bin/duplicity.1 2020-09-29 18:28:41.000000000 +0200 @@ -1,4 +1,4 @@ -.TH DUPLICITY 1 "July 27, 2020" "Version 0.8.15" "User Manuals" \" -*- nroff -*- +.TH DUPLICITY 1 "September 29, 2020" "Version 0.8.16" "User Manuals" \" -*- nroff -*- .\" disable justification (adjust text to left margin only) .\" command line examples stay readable through that .ad l diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/bin/rdiffdir.1 new/duplicity-0.8.16/bin/rdiffdir.1 --- old/duplicity-0.8.15/bin/rdiffdir.1 2020-07-27 17:18:42.000000000 +0200 +++ new/duplicity-0.8.16/bin/rdiffdir.1 2020-09-29 18:28:41.000000000 +0200 @@ -1,4 +1,4 @@ -.TH RDIFFDIR 1 "July 27, 2020" "Version 0.8.15" "User Manuals" \" -*- nroff -*- +.TH RDIFFDIR 1 "September 29, 2020" "Version 0.8.16" "User Manuals" \" -*- nroff -*- .\" disable justification (adjust text to left margin only) .\" command line examples stay readable through that .ad l diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/debian/rules new/duplicity-0.8.16/debian/rules --- old/duplicity-0.8.15/debian/rules 2020-06-02 21:10:41.000000000 +0200 +++ new/duplicity-0.8.16/debian/rules 2020-07-31 23:22:47.000000000 +0200 @@ -22,4 +22,4 @@ grep -Rl "\$$version" debian/duplicity | xargs sed -i "s/\$$version/$(UPSTREAM_VERSION)/g" override_dh_installdocs: - dh_installdocs README README-LOG + dh_installdocs README.md README-LOG.md README-REPO.md README-TESTING.md diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/__init__.py new/duplicity-0.8.16/duplicity/__init__.py --- old/duplicity-0.8.15/duplicity/__init__.py 2020-07-27 17:18:42.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/__init__.py 2020-09-29 18:28:41.000000000 +0200 @@ -22,7 +22,7 @@ import sys import gettext -__version__ = u'0.8.15' +__version__ = u'0.8.16' if sys.version_info.major >= 3: gettext.install(u'duplicity', names=[u'ngettext']) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/backends/_boto_single.py new/duplicity-0.8.16/duplicity/backends/_boto_single.py --- old/duplicity-0.8.15/duplicity/backends/_boto_single.py 2020-06-02 21:10:42.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/backends/_boto_single.py 2020-09-28 17:12:15.000000000 +0200 @@ -21,6 +21,7 @@ from __future__ import division from builtins import str +from concurrent.futures import ThreadPoolExecutor import os import time @@ -339,3 +340,12 @@ time.sleep(60) self.resetConnection() log.Info(u"File %s was successfully restored from Glacier" % remote_filename) + + def pre_process_download_batch(self, remote_filenames): + log.Info(u"Starting batch unfreezing from Glacier") + # Used primarily to move all necessary files in Glacier to S3 at once + with ThreadPoolExecutor(thread_name_prefix=u's3-unfreeze-glacier') as executor: + for remote_filename in remote_filenames: + remote_filename = util.fsdecode(remote_filename) + executor.submit(self.pre_process_download, remote_filename, False) + log.Info(u"Batch unfreezing from Glacier finished") diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/backends/s3_boto3_backend.py new/duplicity-0.8.16/duplicity/backends/s3_boto3_backend.py --- old/duplicity-0.8.15/duplicity/backends/s3_boto3_backend.py 2020-07-24 17:12:07.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/backends/s3_boto3_backend.py 2020-09-24 18:11:41.000000000 +0200 @@ -82,7 +82,6 @@ self.s3 = None self.bucket = None self.tracker = UploadProgressTracker() - self.reset_connection() def reset_connection(self): import boto3 # pylint: disable=import-error @@ -105,6 +104,9 @@ self.bucket = self.s3.Bucket(self.bucket_name) # only set if bucket is thought to exist. def _put(self, local_source_path, remote_filename): + if not self.s3: + self.reset_connection() + remote_filename = util.fsdecode(remote_filename) key = self.key_prefix + remote_filename @@ -147,11 +149,17 @@ ExtraArgs=extra_args) def _get(self, remote_filename, local_path): + if not self.s3: + self.reset_connection() + remote_filename = util.fsdecode(remote_filename) key = self.key_prefix + remote_filename self.s3.Object(self.bucket.name, key).download_file(local_path.uc_name) def _list(self): + if not self.s3: + self.reset_connection() + filename_list = [] for obj in self.bucket.objects.filter(Prefix=self.key_prefix): try: @@ -163,11 +171,17 @@ return filename_list def _delete(self, remote_filename): + if not self.s3: + self.reset_connection() + remote_filename = util.fsdecode(remote_filename) key = self.key_prefix + remote_filename self.s3.Object(self.bucket.name, key).delete() def _query(self, remote_filename): + if not self.s3: + self.reset_connection() + import botocore # pylint: disable=import-error remote_filename = util.fsdecode(remote_filename) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/backends/ssh_paramiko_backend.py new/duplicity-0.8.16/duplicity/backends/ssh_paramiko_backend.py --- old/duplicity-0.8.15/duplicity/backends/ssh_paramiko_backend.py 2020-07-16 17:16:59.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/backends/ssh_paramiko_backend.py 2020-09-26 17:44:53.000000000 +0200 @@ -5,8 +5,6 @@ # Copyright 2011 Alexander Zangerl <az@snafu.priv.at> # Copyright 2012 edso (ssh_config added) # -# $Id: sshbackend.py,v 1.2 2011/12/31 04:44:12 az Exp $ -# # This file is part of duplicity. # # Duplicity is free software; you can redistribute it and/or modify it @@ -39,6 +37,7 @@ from binascii import hexlify import duplicity.backend +from duplicity import progress from duplicity import config from duplicity import util from duplicity.errors import BackendException @@ -317,14 +316,20 @@ response = chan.recv(1) if (response != b"\0"): raise BackendException(b"scp remote error: %b" % chan.recv(-1)) - chan.sendall(f.read() + b'\0') + file_pos = 0 + file_size = fstat.st_size + while file_pos < file_size: + chan.sendall(f.read(16384)) + file_pos = f.tell() + progress.report_transfer(file_pos, file_size) + chan.sendall(b'\0') f.close() response = chan.recv(1) if (response != b"\0"): raise BackendException(u"scp remote error: %s" % chan.recv(-1)) chan.close() else: - self.sftp.put(source_path.name, remote_filename) + self.sftp.put(source_path.name, remote_filename, callback=progress.report_transfer) def _get(self, remote_filename, local_path): # remote_filename is a byte object, not str or unicode diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/config.py new/duplicity-0.8.16/duplicity/config.py --- old/duplicity-0.8.15/duplicity/config.py 2020-07-18 21:50:05.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/config.py 2020-08-10 16:18:24.000000000 +0200 @@ -43,8 +43,12 @@ # Prefix for sig files only file_prefix_signature = b"" -# The name of the current host, or None if it cannot be set -hostname = socket.getfqdn() +# The name of the current host +hostname = socket.gethostname() + +# For historical reasons also save the FQDN for comparing manifests, but +# we tend to prefer the hostname going forward. +fqdn = socket.getfqdn() # The main local path. For backing up the is the path to be backed # up. For restoring, this is the destination of the restored files. @@ -249,6 +253,10 @@ s3_kms_key_id = None s3_kms_grant = None +# region and endpoint of s3 +s3_region_name = None +s3_endpoint_url = None + # Which storage policy to use for Swift containers swift_storage_policy = u"" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/dup_main.py new/duplicity-0.8.16/duplicity/dup_main.py --- old/duplicity-0.8.15/duplicity/dup_main.py 2020-07-15 21:24:14.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/dup_main.py 2020-09-28 23:21:19.000000000 +0200 @@ -748,6 +748,10 @@ u"""Get file object iterator from backup_set contain given index""" manifest = backup_set.get_manifest() volumes = manifest.get_containing_volumes(index) + + if hasattr(backup_set.backend.backend, u'pre_process_download_batch'): + backup_set.backend.backend.pre_process_download_batch(backup_set.volume_name_dict.values()) + for vol_num in volumes: yield restore_get_enc_fileobj(backup_set.backend, backup_set.volume_name_dict[vol_num], diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/manifest.py new/duplicity-0.8.16/duplicity/manifest.py --- old/duplicity-0.8.15/duplicity/manifest.py 2020-07-15 20:42:23.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/manifest.py 2020-09-28 23:21:19.000000000 +0200 @@ -91,7 +91,11 @@ if config.allow_source_mismatch: return - if self.hostname and self.hostname != config.hostname: + # Check both hostname and fqdn (we used to write the fqdn into the + # manifest, so we want to keep comparing against that) + if (self.hostname and + self.hostname != config.hostname and + self.hostname != config.fqdn): errmsg = _(u"Fatal Error: Backup source host has changed.\n" u"Current hostname: %s\n" u"Previous hostname: %s") % (config.hostname, self.hostname) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/path.py new/duplicity-0.8.16/duplicity/path.py --- old/duplicity-0.8.15/duplicity/path.py 2020-07-18 21:50:05.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/path.py 2020-08-04 16:28:11.000000000 +0200 @@ -578,6 +578,14 @@ u"""Return true if path is a directory and is empty""" return self.isdir() and not self.listdir() + def contains(self, child): + u"""Return true if path is a directory and contains child""" + if isinstance(child, u"".__class__): + child = util.fsencode(child) + # We don't use append(child).exists() here because that requires exec + # permissions as well as read. listdir() just needs read permissions. + return self.isdir() and child in self.listdir() + def open(self, mode=u"rb"): u""" Return fileobj associated with self diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity/selection.py new/duplicity-0.8.16/duplicity/selection.py --- old/duplicity-0.8.15/duplicity/selection.py 2020-06-02 21:10:42.000000000 +0200 +++ new/duplicity-0.8.16/duplicity/selection.py 2020-08-04 16:28:11.000000000 +0200 @@ -453,7 +453,7 @@ log.WarningCode.cannot_read, util.escape(path.uc_name)) if diffdir.stats: diffdir.stats.Errors += 1 - elif path.append(filename).exists(): + elif path.contains(filename): return 0 else: return None diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity.egg-info/PKG-INFO new/duplicity-0.8.16/duplicity.egg-info/PKG-INFO --- old/duplicity-0.8.15/duplicity.egg-info/PKG-INFO 2020-07-27 17:18:37.000000000 +0200 +++ new/duplicity-0.8.16/duplicity.egg-info/PKG-INFO 2020-09-29 18:28:41.000000000 +0200 @@ -1,6 +1,6 @@ Metadata-Version: 2.1 Name: duplicity -Version: 0.8.15 +Version: 0.8.16 Summary: Encrypted backup using rsync algorithm Home-page: http://duplicity.nongnu.org/index.html Author: Ben Escoto <ben@emrose.org> @@ -8,28 +8,33 @@ Maintainer: Kenneth Loafman <kenneth@loafman.com> Maintainer-email: kenneth@loafman.com License: UNKNOWN -Description: INSTALLATION: +Description: # INSTALLATION Thank you for trying duplicity. To install, run: - python setup.py install + ``` + python setup.py install + ``` The build process can be also be run separately: - python setup.py build + ``` + python setup.py build + ``` The default prefix is /usr, so files are put in /usr/bin, /usr/share/man/, etc. An alternate prefix can be specified using the --prefix=<prefix> option. For example: - python setup.py install --prefix=/usr/local - export PYTHONPATH='/usr/local/lib/python.x/site-packages/' - /usr/local/bin/duplicity -V + ``` + python setup.py install --prefix=/usr/local + export PYTHONPATH='/usr/local/lib/python.x/site-packages/' + /usr/local/bin/duplicity -V` + ``` + # REQUIREMENTS - REQUIREMENTS: - - * Python 2.7, 3.6, 3.7 + * Python 2.7, or 3.5 to 3.9 * librsync v0.9.6 or later * GnuPG for encryption * fasteners 0.14.1 or later for concurrency locking @@ -38,6 +43,7 @@ * Boto 2.0 or later for single-processing S3 or GCS access (default) * Boto 2.1.1 or later for multi-processing S3 access * Boto 2.7.0 or later for Glacier S3 access + * Boto3 0.3.1 or later If you install from the source package, you will also need: @@ -45,37 +51,12 @@ * librsync development files, normally found in module 'librsync-dev'. - A NOTE ON GnuPGInterface.py AND MULTIPLE GPG PROCESSES: - - GnuPGInterface is used to access GPG from duplicity. The original - works quite well and has no bugs, however, we have patched the one - used in duplicity. Why? Duplicity is not perfect, yet, and has a - problem when handling long chains of incremental backup or restore - operations. The problem is that the waitpid() call only happens - after all the iterations complete, and with a long chain, that can - be a long while. Unless the waitpid() call is made, the child process - remains active. Duplicity's GnuPGInterface is patched to start an - immediate threaded waitpid() for each GPG task, thus harvesting the - task and freeing it's resources in a timely manner. This does not - affect the operation of duplicity, merely frees resources on time. - - Why the note? Some package maintainers remove duplicity's GnuPGInterface - in error, obviously unknowing of this issue and patch duplicity to use - the old unmaintained unpatched GnuPGInterface interface again. - So, if you have the problem that lots of GPG tasks are hanging around, - check and see if this has been done in your distro, and if so, report this - matter as a bug to the distro or package maintainer. - - As of october 2012 we pull the handbrake and refactor our code and rename - the class to gpginterface in the hope that package maintainers will stumble - over it and stop this problematic behaviour for good. - - DEVELOPMENT + # DEVELOPMENT For more information on downloading duplicity's source code from the code repository and developing for duplicity, see README-REPO. - HELP: + # HELP For more information see the duplicity home page at: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/duplicity.egg-info/SOURCES.txt new/duplicity-0.8.16/duplicity.egg-info/SOURCES.txt --- old/duplicity-0.8.15/duplicity.egg-info/SOURCES.txt 2020-07-27 17:18:37.000000000 +0200 +++ new/duplicity-0.8.16/duplicity.egg-info/SOURCES.txt 2020-09-29 18:28:41.000000000 +0200 @@ -4,10 +4,10 @@ CHANGELOG COPYING Changelog.GNU -README -README-LOG -README-REPO -README-TESTING +README-LOG.md +README-REPO.md +README-TESTING.md +README.md pylintrc readthedocs.yaml requirements.txt diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/setup.py new/duplicity-0.8.16/setup.py --- old/duplicity-0.8.15/setup.py 2020-07-27 17:06:48.000000000 +0200 +++ new/duplicity-0.8.16/setup.py 2020-09-29 18:16:14.000000000 +0200 @@ -52,7 +52,7 @@ from setuptools_scm import get_version Version = get_version(**scm_version_args) except Exception as e: - Version = u"0.8.15" + Version = u"0.8.16" print(u"Unable to get SCM version: defaulting to %s" % (Version,)) Reldate = time.strftime(u"%B %d, %Y", time.localtime()) @@ -99,10 +99,10 @@ u'CHANGELOG', u'Changelog.GNU', u'COPYING', - u'README', - u'README-LOG', - u'README-REPO', - u'README-TESTING', + u'README.md', + u'README-LOG.md', + u'README-REPO.md', + u'README-TESTING.md', ], ), ] @@ -286,7 +286,7 @@ os.chmod(file, newmode) -with open(u"README") as fh: +with open(u"README.md") as fh: long_description = fh.read() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/snap/snapcraft.yaml new/duplicity-0.8.16/snap/snapcraft.yaml --- old/duplicity-0.8.15/snap/snapcraft.yaml 2020-07-27 17:18:42.000000000 +0200 +++ new/duplicity-0.8.16/snap/snapcraft.yaml 2020-09-29 18:28:41.000000000 +0200 @@ -1,5 +1,5 @@ name: duplicity -version: 0.8.15 +version: 0.8.16 summary: Efficient, encrypted backup to local or remote hosts description: | Duplicity backs directories by producing encrypted tar-format volumes and uploading diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/duplicity-0.8.15/testing/unit/test_manifest.py new/duplicity-0.8.16/testing/unit/test_manifest.py --- old/duplicity-0.8.15/testing/unit/test_manifest.py 2020-06-26 22:03:28.000000000 +0200 +++ new/duplicity-0.8.16/testing/unit/test_manifest.py 2020-08-10 16:18:24.000000000 +0200 @@ -25,6 +25,7 @@ import re import unittest +from mock import patch from duplicity import config from duplicity import manifest @@ -140,5 +141,27 @@ m2 = manifest.Manifest().from_string(s2) assert hasattr(m2, u'corrupt_filelist') + def test_hostname_checks(self): + self.set_config(u'hostname', u'hostname') + self.set_config(u'fqdn', u'fqdn') + m = manifest.Manifest() + + # Matching hostname should work + m.hostname = u'hostname' + m.check_dirinfo() + + # Matching fqdn should also work for backwards compatibility + m.hostname = u'fqdn' + m.check_dirinfo() + + # Bad match should throw a fatal error and quit + m.hostname = u'foobar' + self.assertRaises(SystemExit, m.check_dirinfo) + + # But not if we tell the system to ignore it + self.set_config(u'allow_source_mismatch', True) + m.check_dirinfo() + + if __name__ == u"__main__": unittest.main()