-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 12:31, Anders Johansson wrote:

On Sunday 25 July 2010 12:19:58 Carlos E. R. wrote:

...

Hi,

How about someone making a daemon that shares via some p2p protocol the /var/cache/zypp/packages/* contents amongst all local networked computers, so that what one downloads from outside can be reused by any other opensuse computer in the same network, automatically?

Don't look at me, I have no idea how to do that ;-)

Well, one obvious way would be to mount that directory from some sort of network file system, perhaps NFS. Then you wouldn't even need a daemon, the machines would just pick it up automatically, you wouldn't have packages taking up space on all machines, and there wouldn't be masses of traffic between the machines

I've doing exactly that, for some time O:-) There a few problems. One, is it not automatic. You have to make sure that all repos save the packages they download (which is not the default setting), and you have to do that one by one on all machines. Another, is that the same repo must use the exact same directory for storing those packages, on all the machines. Sometimes the name is "random" and difficult to guess. In mine Packman got the name of the mirror; another repo got "http:longsomething" as directory name. This directory name is taken from the "alias" of the repository (not the name), which is not shown in the yast module that handles repository management; you have to use zypper for that, change all the "alias" coordinately on all your computers. And the directories do not show till you really try to install something from there. Another is that do you have to activate nfs, server in one, client on another. In my network, the yast module says there are no clients on my network, which is false, it works as soon as I configure it manually. However, there is a big bug in yast client module (11.2 at least) that it _wrecks_ completely your fstab file, so be warned. It does so even if it doesn't add any nfs share, just opening the firewall and searching for one does it. Then, there are procedural problems. You can not run updates on two machines at the same time. What happens if two are downloading the same file to the same nfs share? Or if one tries to install an rpm that hasn't been fully downloaded on another? Or... what happens if one runs "zypper clean"? I think it will clear the main shared repo storage... absolute havoc! Or, what happens if your run package management without first starting the local nfs client? No, it can not be running full time, a machine can be a laptop that is often off the local network. My proposal is different. It has to be automatic, and distributed. It doesn't mean that all packages are available on all computers. Rather, that when one is about to download a package, it asks first the local network if anybody has it, and uses it. If not, it downloads it and makes it available to the rest. Whether that copy is stored again on that computer is something for the devs to decide. Perhaps it should be stored on a percent of the available machines. Or a few machines with room to spare are defined as p2p-servers for the rest - meaning that the main nfs server doesn't have to be running always, as in my setup above. It could have utilities to copy everything available on the network to another computer, for example to create a DVD repo to updte an off-line machine. The YOU and zypper modules should be aware of the presence of the daemon, and warn if the main local storages do not appear to be online. If the go ahead is issued, whatever is downloaded has to be stored locally, and later forwarded to the storage when it appears. See? I have ideas ;-) Just that I'm no linux programmer and can't do this myself. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMG1AACgkQU92UU+smfQWTvgCfTOx5N+c3fV70MC3LmfKTq+v+ LJkAnj+VvGy+lnQPTVX8G+dVp9jgqSBX =dqK9 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 13:16, Bernhard Walle wrote:

Am 25.07.10 13:09, schrieb Carlos E. R.:

...

See? I have ideas ;-) Just that I'm no linux programmer and can't do this myself.

You can learn it. :-)

Naaaa. I take for my self the counselor aka beggar role >:-) - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMIeEACgkQU92UU+smfQXo8gCeLe3VJS0vcFCEPo4qQ9OLkPii qpgAn0tMnRDOLGwl98JzZZG7jBs3dLLh =hxjE -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 13:36, jdd wrote:

Le 25/07/2010 13:09, Carlos E. R. a écrit :

...

It could have utilities to copy everything available on the network to another computer, for example to create a DVD repo to updte an off-line machine.

is not a proxy exactly done for that? (squid?)

Well... it should, if you have one permanently running. I don't. I know little about proxies, but I see one problem. Recently, I tried to mirror some of the 11.0 repos before they disappear, with wget and other tools. The problem is that all tools resolved to the real destination the download redirector gives, so that instead of having an "oss" directory, I had several "oss" directories, each from a different mirror. I didn't let it finish. Thus, if you try to use a proxy, it will have the same problem, it will cache each of the mirrors the redirector resolves to, instead of "download.opensuse.org", which destroys the purpose of the proxy idea. You end by having several copies of the same postfix rpm, one per mirror it has used on each attempt from each machine of the network. For this to work, you have to configure the same mirror for all the network, which means that you lose resiliency. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMLaAACgkQU92UU+smfQUj8QCcCxdVHQ39sU5ffsnFGZxAzDBR rVAAni/1pm2hjywqH+RnnUs2LXPHrVuj =zbZo -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 15:09, jdd wrote:

Le 25/07/2010 14:27, Carlos E. R. a écrit :

...

...

is not a proxy exactly done for that? (squid?)

Well... it should, if you have one permanently running. I don't.

it could be simpler to configure one than to do what you try to do.

an other way is to use only one mirror, for example http://ftp3.gwdg.de/pub/opensuse/distribution/11.3/iso/openSUSE-11.3-NET-i58...

wich is known to be very robust :-)

It is the only way I know about, if you want to use a proxy.

I just notice, searching for this url than

http://software.opensuse.org/113/en

if you clic on "pick a mirror" (english page), gives a complete list of actives mirrors (you just have to remove the file part)

I know.

With the low per byte cost of modern HDD, mirrorring all the update repository is not that a problem.

There is a download cost, in my case: days downloading. But what I propose is quite different. It is using the already in use resources on a home or small office network, with or without a permanent server, to download each package (update or not) only once if possible, with as little work by the local people as feasible. All the solutions we use, ad hoc, are used precisely because we do not have a method designed for just this purpose. Another idea is having a central "proxy repo", fully automatic. Any local computer wanting a package from any repo ask first this designated computer. If it is already there, it is used instantly. It is not there, it is told to wait while it is downloaded. This is different from a web proxy, in that it must understand enough of the redirector, and enough of how zypper/yast works. And it has to be automatic as well. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMVYAACgkQU92UU+smfQW5OwCcC4lzXLxUofS8FuhCJYYzhjWn x68Ani3IpC5y4WkJZi1IVUolilEcYY2m =QMVY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Per Jessen wrote:

I'm not sure, but I think(!) I've seen something in squid that allows you to sort "map" mirrored copies to one. I'm really not sure though.

IntelligentMirror - I haven't tried it, I always get the same mirror for download.opensuse.org. -- Per Jessen, Zürich (18.1°C) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Sunday 25 July 2010 13:09:04 Carlos E. R. wrote:

Another is that do you have to activate nfs, server in one, client on another.

Well, even with your idea, there has to be some sort of manual intervention. You do not want a daemon that automatically opens a port in the firewall and connects to other machines that happen to be in the same LAN. Somehow you have to decide which machines are trusted, perhaps by looking at their host keys. So completely automatic it can never be. Security dictates otherwise. Apart from that it looks like you have an interesting idea. There are of course already server solutions where you set up one server and push packages out to clients from it. This is what I personally prefer in a network with many machines, partly because it gives me one point of administration, and if a package fails for some reason I don't have to look through every machine to find the failing package and clean it out. But if you really prefer this peer- to-peer idea, then I wish you luck with it Anders -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Sun, 25 Jul 2010 13:45:29 +0200 Anders Johansson <ajh@nitio.de> wrote:

Apart from that it looks like you have an interesting idea. There are of course already server solutions where you set up one server and push packages out to clients from it. This is what I personally prefer in a network with many machines, partly because it gives me one point of administration, and if a package fails for some reason I don't have to look through every machine to find the failing package and clean it out.

Can SMT be made to work with openSUSE? That would be a nice thing (even though I'm not sure that it has "only download packages that at least one of the 'clients' needs" or only "mirror full update repos") -- Stefan Seyfried "Any ideas, John?" "Well, surrounding them's out." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 16:37, Richard Creighton wrote:

On Sunday 25 July 2010 07:09:04 Carlos E. R. wrote:

Check out Dave Rankins contribution:

http://www.3111skyline.com/download/openSUSE/pkgmanage/pkgmanage.tar.bz2

Works great and using cron or other method, you can automate it completely. He put a lot of thought and effort into this package and it is well documented and allows all your issues to be addressed across multiple machines and versions including simultaneous updates. He deserves many thanks.

I think I have seen it. Creating a local repo? Well, this is a different idea. There are many ways to skin a cat, as the saying goes :-) - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMUjEACgkQU92UU+smfQX/5gCfZ6zKyvNxVgPda9uyP7P8ufuw GjsAniOR6v99ngAH4dzcWUa+Vn5UmHaf =lPn8 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 17:18, Richard Creighton wrote:

On Sunday 25 July 2010 11:03:13 Carlos E. R. wrote:

...

On 2010-07-25 16:37, Richard Creighton wrote:

...

On Sunday 25 July 2010 07:09:04 Carlos E. R. wrote:

Check out Dave Rankins contribution:

http://www.3111skyline.com/download/openSUSE/pkgmanage/pkgmanage.tar.bz2

Works great and using cron or other method, you can automate it completely. He put a lot of thought and effort into this package and it is well documented and allows all your issues to be addressed across multiple machines and versions including simultaneous updates. He deserves many thanks.

I think I have seen it. Creating a local repo?

Well, this is a different idea. There are many ways to skin a cat, as the saying goes :-)

Possibly the biggest advantage of this is that it contains every file you will ever need to reconstruct your system(s) even AFTER end of support from official repositories, like the recent demise of 11.0 and earlier, 10.3 which just 'dissapeared' leaving people with older hardware somewhat out in the cold if they lost a drive containing their OS and needed to rebuild it. Sure they could reinstall from the original DVD, but what about all the updates that no longer are available....having a local copy of the ones you had applied on hand is certainly handy, as is all of the stuff you had ever downloaded that was for your version of the OS and your machine. It also allows you to go back to an earlier version that might be out of date/support if the current version turns out to have a hidden uh-oh. Sure backups are important, but they rarely are 'bare-metal', usually concentrating on data not the OS itself. Dave's system saves space, is quick, saves time and is easy to back up and unllike a mirror of a repository, only contains what you actually need for your installation as you have installed it on any of your machines.

Replicating the oss, non-oss, and updates repos, even for only one architecture, takes days of downloading. I know because I did so. I know, it is a good solution. It just needs space and good internet bandwidth (which I don't have). What I'm proposing is a different solution, with different requirements, and advantages, and problems. Recently somebody asked for a solution to his similar problem. His bandwidth is even more limited than mine, because he pays per megabyte. What he and I want is just downloading once everything we need for perhaps two or three computers. And doing so with as little hassle as possible. For this scenario a full local repo is impossible. The shared directories via nfs are workable, with some care. It is also possible to build, picking the files already downloaded by each computer in the network, a local repo of whatever we have, even incomplete. But there is no way I know that YaST could pick packages from that incomplete repo and what is missing from a remote. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMWUkACgkQU92UU+smfQWKFACfZVDdgBJG1RDTzaXyBthvp56S seAAnjTt074LV0nc1uSlyQ3OFR6sXZJD =+mrQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Sun, Jul 25, 2010 at 11:18 AM, Richard Creighton <ricreig@gmail.com> wrote:

On Sunday 25 July 2010 11:03:13 Carlos E. R. wrote:

...

On 2010-07-25 16:37, Richard Creighton wrote:

...

On Sunday 25 July 2010 07:09:04 Carlos E. R. wrote:

Check out Dave Rankins contribution:

http://www.3111skyline.com/download/openSUSE/pkgmanage/pkgmanage.tar.bz2

Works great and using cron or other method, you can automate it completely. He put a lot of thought and effort into this package and it is well documented and allows all your issues to be addressed across multiple machines and versions including simultaneous updates.   He deserves many thanks.

I think I have seen it. Creating a local repo?

Well, this is a different idea. There are many ways to skin a cat, as the saying goes :-)

Possibly the biggest advantage of this is that it contains every file you will ever need to reconstruct your system(s) even AFTER end of support from official repositories, like the recent demise of 11.0 and earlier, 10.3 which just 'dissapeared' leaving people with older hardware somewhat out in the cold if they lost a drive containing their OS and needed to rebuild it.

Richard, are you familiar with: http://ftp5.gwdg.de/pub/opensuse/discontinued/distribution/ It goes back to 10.2, which I think has been out of support for a 2 or 3 years now. Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-25 21:58, Larry Finger wrote:

The script /etc/sysconfig/network/scripts/ifup-wireless has two kinds of problems.

Please don't hijack a thread, start your own. This is about: subject: Re: [opensuse-factory] [RFC] How about a p2p repo sharing module for Yast? - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxMwZkACgkQU92UU+smfQXfdACcCLbc/rSLI1cOpm88ebnCtYUg awwAn2P21RVWcr1LiCSfETsA+2+gTS71 =n0FC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

* Larry Finger <Larry.Finger@lwfinger.net> [07-25-10 19:19]:

What makes you think I hijacked your thread?

You did "hijack" the thread. You changed the "Subject:" of an existing message and posted your message. Message headers contain links tying them to previous and subsequent postings making a *Thread*.

Look at the subject of what you sent me. Do you see anything about p2p, or do you own the entire mailing list?

When you changed the "Subject:", you do not change the thread. You must remove the header line "In-Reply-To:" to break the thread. And this is probably more that you wanted to know or really care about, but *you* "hijacked" the thread!. -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

On Sun, Jul 25, 2010 at 7:30 PM, Carlos E. R. <carlos.e.r@opensuse.org> wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 2010-07-25 19:52, Greg Freemyer wrote:

...

On Sun, Jul 25, 2010 at 6:19 AM, Carlos E. R. <carlos.e.r@opensuse.org> wrote:

...

...

How about someone making a daemon that shares via some p2p protocol the /var/cache/zypp/packages/* contents amongst all local networked computers, so that what one downloads from outside can be reused by any other opensuse computer in the same network, automatically?

Don't look at me, I have no idea how to do that ;-)

Ignoring lots of back and forth and acknolowging I don't know what sort of loads the official repo servers currently see:

I agree with Carlos because:

Zypper dup / yast2 wagon have made the online upgrade method almost painless,  certainly easier than upgrading via DVD.

I still prefer using the DVD. Just now I'm upgrading an 11.0 to 11.2, and the damn thing says it can not find the "content" file on the oss repo - which is not true, I pinged the repo from the same machine and verified the directory on another. Online upgrade is very unreliable.

...

But I suspect there are a lot of users like myself.  At my office I have 15 or 20 opensuse installations.  Currently only 2 have been upgraded to 11.3, both without using the DVD.

But in order to cut down on load on the mirrors, as well as my Internet connection this very weekend I'm downloading the 11.3 DVD(s) so I can start upgrading the remaining machines.

And then you will find that packages not in the DVD will not be upgraded, but removed instead >:-)

...

If I could have simply (and I mean simply as in via YaST) have designated one of my upgraded machines as a repo forwarder, I could have skipped the DVD download.

Correct. I did not think of this idea, but it certainly could be done and is very interesting as well.

...

I envision it working similar to DNS.  You request a package from the designated repo machine.  If it doesn't have it, then it requests it from a bigger repo, etc.

If a tiered approach like that could be setup, then even more major download sites could share the load just as DNS load is heavily distributed.

Mmmmm! :-)

...

It's not peer to peer as Carlos described, but it is a big step forward and as zypper dup gets more and more popular it seems to be a solution that is needed, as opposed to just a nice to have.

As to staleness or malware issues, I think as Carlos proposed that key metadata such as MD5 should come only from the official repos, but the packages themselves could come from anywhere as long as the md5 matches.

Well, currently the metadata doesn't come from the mirrors, but from the central redirector itself. This is done so that the mirror network gets automatic security: if one is a rogue server, it is inmediately detected. Each oS trying to install something from there will detect it and fail - unless it defines that mirror as _the_ download server.

So that doesn't need to change, and we get the same security with the tiered or p2p layout :-)

...

fyi: Their are no known cases of malware being created in such a way as to have the file have a predetermined md5.  It is theoretically possible, but no one has done it yet.  But if a solution like porposed above is pursued, possibly a more robust signature than md5 should be used.  ie. SHA-1 or SHA-256.

PGP is used already :-)

By the way: zypper/yast can currently use aria2c as a downloader, which I believe uses or can use metalink data for each package. And a metalink can give info on web mirror, ftp mirrors, but also p2p links. If I'm correct, we already have what is needed in zypper/yast to use a secure torrent network to get our updates!

It just need some modifications.

Is torrent a good solution for small packages? ie. With large files, torrent breaks the file into 2MB segments and gains speed by doing lots of chunks in parrellel, but my impression is it takes a few minutes to really get up to speed. Since a lot of packages are less than 2MB, I could imagine torrents being extremely inefficient because of high per package download overhead, but I'm guessing. (FTP suffers from this as well, but not as bad a torrents from what I've seen. FTP has a few seconds of overhead for every file download initiated, so downloading a couple thousand packages would add on a couple clock hours of time just due to the the ftp per file negotiation.) I still favor a tiered approach that can have an optimized package initiation protocol. When a repo refresh takes place, does the per package hash come down from the redirector, or is it done one package at time as the installs/upgrades are taking place? Greg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-07-26 09:42, Duncan Mac-Vicar P. wrote:

On 07/25/2010 12:19 PM, Carlos E. R. wrote:

...

Hi,

How about someone making a daemon that shares via some p2p protocol the /var/cache/zypp/packages/* contents amongst all local networked computers, so that what one downloads from outside can be reused by any other opensuse computer in the same network, automatically?

Don't look at me, I have no idea how to do that ;-)

http://www.novell.com/linux/smt

You need a SLES license to get it for free.

Well, it is similar, but it does more, and differently (like subscription control). My idea is not that. It is not a mandatory central repository, for instance, it is distributed (p2p or tiered (Greg Freemyer)), and requires access to the master download server at opensuse for each machine. But it is nice to see that the SLES people saw this need and made a software to fulfill it. It proves that my idea is not a useless thing >:-) - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkxNZqsACgkQU92UU+smfQWYgwCeKt3uVGke1YQs1bFtoS8RrAHM RrcAniigRiTs7fnT5YOsR+q05p66qYMU =hkYM -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org

Per Jessen wrote:

Carlos E. R. wrote:

...

How about someone making a daemon that shares via some p2p protocol the /var/cache/zypp/packages/* contents amongst all local networked computers, so that what one downloads from outside can be reused by any other opensuse computer in the same network, automatically?

Just run downloads via squid.

Sorry, I see I was a bit late with that. -- Per Jessen, Zürich (17.9°C) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org