On Fri, Aug 21, 2009 at 12:00 PM, David C. Rankin wrote:

On Thursday 20 August 2009 11:27:18 pm Jay Mistry wrote:

...

Is it possible to have a local cache/ "repository" of all -

1) Updates (Critical, Security, Bug-fixes), and 2) Additional installed rpm's (that were installed through Yast / PackageKit), e.g. Opera, Adobe Reader, VLC Player, etc. so that I do not have to download all those again (800 MB + D/L)

Yes,

       I have done that for the past 6 months and love it. Basically I have 6-8 boxes that I update from a central computer in my house. Since all of my boxes don't have the same package config, I created a "Generic" openSuSE repository for each release (11.0, 11.1) and hold all the common packages there. The directory structure I use for the repository is:

01:01 nirvana:~/linux/apps> l /home/backup/rpms/ total 32 drwxr-xr-x  8 david dcr 4096 2009-08-10 21:39 ./ drwxr-xr-x 28 david dcr 4096 2009-07-27 17:47 ../ drwxr-xr-x  6 david dcr 4096 2009-08-20 18:58 data/ drwxr-xr-x  3 david dcr 4096 2009-08-09 04:02 dups/ drwxr-xr-x 12 david dcr 4096 2009-08-20 18:58 openSUSE_11.0/ drwxr-xr-x  8 david dcr 4096 2009-04-07 19:05 openSUSE_11.1/ drwxr-xr-x  2 david dcr 4096 2009-04-08 00:33 openSUSE_11.2/

       Where you have the normal architecture divisions you would expect under 11.0, 11.1, etc. Like:

01:01 nirvana:~/linux/apps> l /home/backup/rpms/openSUSE_11.0 total 776 drwxr-xr-x 12 david dcr       4096 2009-08-20 18:58 ./ drwxr-xr-x  8 david dcr       4096 2009-08-10 21:39 ../ drwxr-xr-x  2 david dcr       4096 2009-08-07 14:14 compiz-new/ drwxr-xr-x  2 david dcr      77824 2009-08-09 04:02 delta/ drwxr-xr-x  2 david dcr       4096 2009-07-13 11:23 i386/ drwxr-xr-x  2 david skyline 155648 2009-08-19 18:56 i586/ drwxr-xr-x  2 david dcr      12288 2009-08-15 23:07 i686/ drwxr-xr-x  2 david dcr      69632 2009-08-20 18:58 noarch/ drwxr-xr-x  4 david dcr      12288 2009-07-27 16:44 openOffice/ drwxr-xr-x  2 root  root      4096 2009-08-20 18:58 repodata/ drwxr-xr-x  2 david dcr       4096 2009-08-08 02:01 src/ drwxr-xr-x  2 david skyline 425984 2009-08-20 18:58 x86_64/

       The additional compiz-new and openOffice directories above just hold the alternate version for each like I have compiz 7.8 in my repository, but have 8.2 avaiable. For openOffice, I have 2.4 in the repo and 3.0.1 in the extra openOffice directory.

       The key to making this work is to have all clients that do updates use a script to separate all the files by architecture and then upload to the server via rsync. That way the next box that gets updated, pulls the rpms from your server instead of from the internet.

       By doing it this way, you have one central repository, that manages duplicates, and that holds updated packages for rpms of all architectures. Further, I put the repository under /home so it gets the space off the / partition freeing up space there.

       The first thing you want to do is to is to set the "keeppackages" option for each of your repository definition definition files in /etc/zypp/repo.d/ and turn off the use of delta.rpms in your zypper.conf file. They may be smaller, but they are a hell of a lot slower to install.

       I have just a couple of scripts that do the management of the repo. The client-side and server-side scripts I use can be obtained at:

http://www.3111skyline.com/download/openSUSE/pkgmanage/

       After one of your computers has done an update, just run the zyppmerge.dev script on the client and it will gather all the newnly downloaded rpms and rsync them to the box you have chosen as the server (you just need to edit a few variables to set the appropriate hostname, etc. This client side script will then call the server-side script that will eliminate any duplicate packages, update the deltarpm file and create and sign the repository with the new files in it to update the repo before passing control back to the client.

       Give it a go and see if it meets you needs.

-- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

David, Thanks for the detailed note & procedure. However, I forgot to mention: I am a home PC user and this is for a single desktop home PC (my laptop runs Windows XP w/ Linux Mint 7). I am not sure I followed all the instructions since there is no server: only a single-user home PC. Will use of only the client-side script do the job ?

The first thing you want to do is to is to set the "keeppackages" option for each of your repository definition definition files in /etc/zypp/repo.d/ and turn off the use of delta.rpms in your zypper.conf file. They may be smaller, but they are a hell of a lot slower to install.

I got the 'keeppackages' option part ...

__editors.repo__  (from /etc/zypp/repos.d/editors.repo) [editors] name=editors enabled=0 autorefresh=0 baseurl=http://download.opensuse.org/repositories/editors/openSUSE_11.1/ type=rpm-md keeppackages=1                              ^

... and the deltarpm's part (I think).  This is the relevant section from /etc/zypp/zypp.conf . Do I need to remove the single # (hash) from , or will just changing the value from default of "true" to "false' suffice ?

## ## Whether to consider using a .delta.rpm when downloading a package ## ## Valid values: boolean ## Default value: true ## ## Using a delta rpm will decrease the download size for package updates ## since it does not contain all files of the package but only the binary ## diff of changed ones. Recreating the rpm package on the local machine ## is an expensive operation (memory,CPU). If your network connection is ## not too slow, you benefit from disabling .delta.rpm. ## # download.use_deltarpm = false                                                     ^^^^^

After doing this , I am not sure how to proceed. I found 2 relevant links through Google, but they refer to Fedora. http://www.labtestproject.com/using_linux/setup_to_install_software_package_... http://www.labtestproject.com/using_linux/install_software_from_fedora_insta... Thanks, Jay -- Linux User 483705 | openSUSE 11.1, Ubuntu 9.04  (i686) w/ Win XP -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Wed, Sep 2, 2009 at 7:08 AM, David C. Rankin wrote:

On Friday 21 August 2009 09:41:36 am Jay Mistry wrote:

...

David,

Thanks for the detailed note & procedure. However, I forgot to mention: I am a home PC user and this is for a single desktop home PC (my laptop runs Windows XP w/ Linux Mint 7). I am not sure I followed all the instructions since there is no server: only a single-user home PC. Will use of only the client-side script do the job ?

Well, yes it will work fine for a single PC, but what you are doing there is basically just collecting the current set of rpms you have installed to save to a backup somewhere in case your box crashes and you need to reload. Then you will be able to do your install from the set of rpms you have saved in the local repository and avoid the download.

In the zyppmerge.dev script look at line 97, 117, 203, etc. at the use of the LOCALHOST variable. The script is set up to handle the situation where your local repository is the same box as your update server. It will set the variable to "1" if hostname == ${REPOHOST%%.*} and manage your repository there.

...

...

The first thing you want to do is to is to set the "keeppackages" option for each of your repository definition definition files in /etc/zypp/repo.d/ and turn off the use of delta.rpms in your zypper.conf file. They may be smaller, but they are a hell of a lot slower to install.

I got the 'keeppackages' option part ...

...

__editors.repo__  (from /etc/zypp/repos.d/editors.repo) [editors] name=editors enabled=0 autorefresh=0 baseurl=http://download.opensuse.org/repositories/editors/openSUSE_11.1/ type=rpm-md keeppackages=1

                                 ^

... and the deltarpm's part (I think).  This is the relevant section from /etc/zypp/zypp.conf . Do I need to remove the single # (hash) from , or will just changing the value from default of "true" to "false' suffice ?

Your /etc/zypp/zypp.conf should look like:

[main] download.use_deltarpm = false

...

...

## ## Whether to consider using a .delta.rpm when downloading a package ## ## Valid values: boolean ## Default value: true ## ## Using a delta rpm will decrease the download size for package updates ## since it does not contain all files of the package but only the binary ## diff of changed ones. Recreating the rpm package on the local machine ## is an expensive operation (memory,CPU). If your network connection is ## not too slow, you benefit from disabling .delta.rpm. ## # download.use_deltarpm = false

                                                       ^^^^^

After doing this , I am not sure how to proceed.

I found 2 relevant links through Google, but they refer to Fedora.

(damn I'm getting old, I had forgotten I had already put together a web page that explains most of this some 8-9 months ago, see: http://www.3111skyline.com/linux/openSuSE-LocalUpdate.php )

       In a bit more detail, here are the nuts & bolts of the process.

       All you are doing in setting up a local repository is getting all the rpms out from under /var/cache/zypp/packages/[lots of sub-directories] and moving them to wherever you want your local repository to be. (I use /home/backup/rpms/openSUSE_11.0/architecture (or 11.1, etc..) This keeps you from having to create 10 separate local repositories under /var/cache/zypp/packages/(name of each repo)

       When openSuSE came out with the keeppackages option so you could save a local copy, they didn't really make them usable due to the spiderweb of directories the rpms get saved to under /var/cache/zypp/packages/reponame... It's just a huge mess to try to do anything with the rpms there. That's why I came up with the script to go through that jungle and move the rpms to some place I could easily make them available in a "single" repository for each openSuSE release.

Here is what happens:

(1) After setting keeppackages=1 in your /etc/zypp/repos.d/ files and after setting download.use_deltarpm = false in /etc/zypp/zypp.conf, every rpm you download as an update to your system will be saved under /var/cache/zypp/packages/reponame...

(2) the zyppmerge.dev script (I really need to rename it) scans all files in /var/cache/zypp/packages and

       (a) separates them by architecture (i586, noarch, x86_64, etc..) and in the case where you are working on the same machine you want to use as a server, simply copies the rpms that it has found to REPOPATH=/home/backup/rpms/openSUSE_11.0/{i586,x86_64,etc..} where it will create the actual rpm repository (instead of rsync'ing them to a separate server),

       (b) searches through files in the repository to move all older versions of rpms to a backup location by calling the fduprpm script

       (c) calls the xmlparse-srv script to update the list of delta-rpms available (since you will probably already have some),

       (d) creates the repository metadata,

       (e) signs the repository so it is ready for use, and

       (f) asks you if you want to delete the rpms from /var/cache/zypp/packages since you have just copied them to your repo location at /home/backup/rpms (by default) After you are satisfied that everything is working, answer "yes" here and it will free up several gigabytes of space on your root partition that is currently taken up by the cached packages.

       Now you have all your rpms separated by architecture under:

/home/backup/rpms/openSUSE_11.0/{i586,x86_64,etc..}

       that can easily be used as an update repository or simply backed up to a dvd or another box for later use. The scripts I wrote just keep you from having to do all the tedious stuff by hand. Good luck!

(I know I need to update some of the script links on the site -- on the agenda)

...

http://www.labtestproject.com/using_linux/setup_to_install_software_package _from_fedora_10_dvd  http://www.labtestproject.com/using_linux/install_software_from_fedora_ins tallation_dvd

Thanks,

Jay

-- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Thanks for this. I actually found you old write up on your website after you wrote this. I have a question to ask. Since i've been running my openSUSE 11.1 for some time now and have done several updates in the process, will the new local repository start with the first set of updates or continue from where my updates currently are? Thanks Jide -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org