Hi,
Are there any extra measures in place to prevent opensuse.org going down before the official 10.2 announcement, because of the hype?
Regards, apokryphos. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Saturday 02 December 2006 13:10, Francis Giannaros wrote:
Are there any extra measures in place to prevent opensuse.org going down before the official 10.2 announcement, because of the hype?
What hype? Do you mean rush?
Bye, Steve --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On 2006-12-02 08:48, Stephan Binner wrote:
On Saturday 02 December 2006 13:10, Francis Giannaros wrote:
Are there any extra measures in place to prevent opensuse.org going down before the official 10.2 announcement, because of the hype?
What hype? Do you mean rush?
I think he means, are you prepared for a possible DoS ahead of the release announcement, on account of all the noise that was generated in response to the deal with The Company From Redmond®.
Are there any extra measures in place to prevent opensuse.org going
down
before the official 10.2 announcement, because of the hype?
What hype? Do you mean rush?
I think he means, are you prepared for a possible DoS ahead of the release announcement, on account of all the noise that was generated in response to the deal with The Company From Redmond�.
This article at http://www.linux.com/article.pl?sid=06/11/01/1641247 deals with how Metalink users were still able to easily download Fedora Core 6 at release.
With the release of Fedora Core 6, their main website and yum update servers were unusable for days. I would guess this was because the main site was serving ISOs as well. I think it would be good to encourage people to download with methods that will lessen the strain on the main servers, at least for the first few days.
p2p is great for this, but we have to remember that p2p isn't available to everyone. It's blocked by some universities, businesses, and ISPs. Besides regular p2p/bittorrent, you can continue using Metalink to spread out the downloads between mirrors, either removing the listing for the main server or lowering its priority rating. Metalinks can list mirrors, along with torrents, ed2k, and magnet links. People can then download from local servers if they set their location, since the mirrors have location and priority information.
<url type="ftp" location="jp" preference="10"> ftp://ftp.jaist.ac.jp/pub/Linux/openSUSE/distribution/openSUSE-10.2-RC1/iso/openSUSE-10.2-RC1-i386-CD1.iso </url> <url type="http" location="jp" preference="10"> http://ftp.kddilabs.jp/Linux/packages/opensuse/distribution/openSUSE-10.2-RC... </url>
The Metalinks for RC1 are available at http://download.packages.ro/metalink/opensuse/ . Ones for the final release will be automatically generated once the ISOs are synced to the mirrors.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Saturday 02 December 2006 20:40, Anthony Bryan wrote:
p2p is great for this, but we have to remember that p2p isn't available to everyone. It's blocked by some universities, businesses, and ISPs. Besides regular p2p/bittorrent, you can continue using Metalink to spread out the downloads between mirrors, either removing the listing for the main server or lowering its priority rating. Metalinks can list mirrors, along with torrents, ed2k, and magnet links. People can then download from local servers if they set their location, since the mirrors have location and priority information.
<url type="ftp" location="jp" preference="10">ftp://ftp.jaist.ac.jp/pub/Linux/openSUSE/distribution/openSUSE-10.2-RC1/iso /openSUSE-10.2-RC1-i386-CD1.iso </url> <url type="http" location="jp" preference="10">
http://ftp.kddilabs.jp/Linux/packages/opensuse/distribution/openSUSE-10.2-R C1/iso/openSUSE-10.2-RC1-i386-CD1.iso </url>
The Metalinks for RC1 are available at http://download.packages.ro/metalink/opensuse/ . Ones for the final release will be automatically generated once the ISOs are synced to the mirrors.
That seems like a very nice idea; I investigated by downloading a few ISOs and they maxed out my connection (24mbit, but still). Are there any plans to advertise these metalinks specifically, at all? I guess part of the problem would be that people really don't know about the procedure or applications (despite being so easy to use), unlike torrents. Still, looks like something very good to explore. I've only been here for two previous releases, but the site was down both times and things weren't going so well at all. We get many new users in the channels asking questions, and while we'd like to give them the links to the very easy howtos on the wiki, it's not available :/. Try to improvise with Google cache, most of the time. Not ideal though. Regards, Francis. --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Saturday 02 December 2006 20:40, Anthony Bryan wrote:
p2p is great for this, but we have to remember that p2p isn't
available to
everyone. It's blocked by some universities, businesses, and ISPs.
Besides
regular p2p/bittorrent, you can continue using Metalink to spread out
the
downloads between mirrors, either removing the listing for the main
server
or lowering its priority rating. Metalinks can list mirrors, along
with
torrents, ed2k, and magnet links. People can then download from local servers if they set their location, since the mirrors have location
and
priority information.
<url type="ftp" location="jp" preference="10">ftp://ftp.jaist.ac.jp/pub/Linux/openSUSE/distribution/openSUSE-10.2-
RC1/iso
/openSUSE-10.2-RC1-i386-CD1.iso </url> <url type="http" location="jp" preference="10">
http://ftp.kddilabs.jp/Linux/packages/opensuse/distribution/openSUSE-
10.2-R
C1/iso/openSUSE-10.2-RC1-i386-CD1.iso </url>
The Metalinks for RC1 are available at http://download.packages.ro/metalink/opensuse/ . Ones for the final
release
will be automatically generated once the ISOs are synced to the
mirrors.
That seems like a very nice idea; I investigated by downloading a few ISOs and they maxed out my connection (24mbit, but still).
Are there any plans to advertise these metalinks specifically, at all? I guess part of the problem would be that people really don't know about the procedure or applications (despite being so easy to use), unlike torrents. Still, looks like something very good to explore.
I've only been here for two previous releases, but the site was down both times and things weren't going so well at all. We get many new users in the channels asking questions, and while we'd like to give them the links to the very easy howtos on the wiki, it's not available :/. Try to improvise with Google cache, most of the time. Not ideal though.
Hi Francis,
Thanks for taking the time to test metalink out. 24 mbit, nice! About 3 MB/sec? Unfortunately my home cable only gets about a third of that.
I don't know if metalinks will be specifically advertised. They are listed on http://en.opensuse.org/Released_Version#Metalink, but like you said when the sites down - it's down (kinda frustrating and). One nice thing about metalink, in these busy situations you don't have to manually try all sorts of mirrors to see if you can get through. The clients do that automatically. I always hated that. As I mentioned, it allowed people to get FC6 during the mad onslaught when it seemed impossible otherwise.
What I was proposing on the list (maybe not explicit enough) was that the site goes into crisis mode. Only list metalinks and torrents, and rely on p2p/mirrors to distribute the ISOs. Either stop the main site from ISO downloads, or severely limit it so the site can stay up. Only list the most important info.
Make sure people can get aria2 or wxDownload Fast easily, have em packaged for a few versions back of suse. Make it as simple as possible. 1) Install RPM 2) Open terminal and type 'aria2c URLtoMetalink' 3) Wait
curl and wget are adding support for metalink, but I think someone could whip something up quick which would allow either of them to use it now. It would just be a quick front end that reads the XML, and gets a chunk from a few servers, then runs md5sum on it when done.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
opensuse users, read ahead :-) thanks
Anthony Bryan a écrit :
What I was proposing on the list (maybe not explicit enough) was that the site goes into crisis mode. Only list metalinks and torrents, and rely on p2p/mirrors to distribute the ISOs. Either stop the main site from ISO downloads, or severely limit it so the site can stay up. Only list the most important info.
to be practical, I propose that you spread this very post. I already crosspost to opensuse list. Here are the instruction, feel freee to edit is I made any mistake.
If you have problems joining the opensuse.org site or any mirror during the first days of the 10.2 release, there is a solution.
The solution is to use metalink. This utility, in very short summary, make a balance between all the available ftp sources and allows you to download from the better one.
as long as the site is available you can look at http://en.opensuse.org/Released_Version#Metalink
The .metalink file is approximately like a torrent file, very small and easy to catch.
you need a metalink client, there are many for any system (for example aria2 or wxDownload).
then look at
ftp://ftp.jaist.ac.jp/pub/Linux/openSUSE/distribution/
with your browser. This ftp is _not_ a opensuse mirror, so should not be down
test your system with the 10.2 rc1, for example. GM and 10.2 are not yet functional, but will be asap (the dec 7)
enjoy :-)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Monday 2006-12-04 at 19:12 +0100, jdd sur free wrote:
...
The solution is to use metalink. This utility, in very short summary, make a balance between all the available ftp sources and allows you to download from the better one.
as long as the site is available you can look at http://en.opensuse.org/Released_Version#Metalink
...
you need a metalink client, there are many for any system (for example aria2 or wxDownload).
A warning: there are three "aria" projects at sourceforge. The "aria" one is:
ARIA (Accounting, Receiving and Inventory Administration) is a web based accounting and inventory management package based off the NOLA project. ARIA aim's to be platform and browser independent with multi-language support.
which obviously is not the one we want - I found out while trying to install it O:-) -. Another one is "Aria download manager RPM Project", which also isn't, but not so obvious (it is a "download manager", after all).
The correct one is "aria2", which is not found searching for "aria" in sourceforge: you have to search for "aria2" specifically.
It would be nice if the wiki (http://en.opensuse.org/Released_Version#Metalink) did not only mention the Metalink existence, but also provided a link to a readily compiled rpm for all currently SuSE versions in use (10.1, 10.1, 9.3, SLES, etc), if you want people to use metalink to download 10.2.
That means today ;-)
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Wednesday 2006-12-06 at 16:56 +0100, I wrote:
It would be nice if the wiki (http://en.opensuse.org/Released_Version#Metalink) did not only mention the Metalink existence, but also provided a link to a readily compiled rpm for all currently SuSE versions in use (10.1, 10.1, 9.3, SLES, etc), if you want people to use metalink to download 10.2.
That means today ;-)
I'll mention that is is not possible to compile aria2-0.9.0.tar.bz2 in suse 10.1, a dependency problem: it needs libxml 2.6.24, and 10.1 has 2.6.23. That support is necesary for the metalink part of the program.
I opened a thread about this in opensuse, please answer there. I just want to call your attention to that; me and many others will not be able to use metalink this run, I'm afraid.
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On the whole it was all quite successful -- the site stayed up throughout, and users got very fast and reliable downloads thanks to metalink. The torrents were originally very fast but the pace dropped down after the initial flood in. Metalinks went well throughout the day, so good stuff.
Regards, Francis.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Friday 2006-12-08 at 17:00 -0000, Francis Giannaros wrote:
On the whole it was all quite successful -- the site stayed up throughout, and users got very fast and reliable downloads thanks to metalink.
HA!
Don't make me laugh. Reliable my foot!
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Fri, Dec 08, 2006 at 05:00:52PM +0000, Francis Giannaros wrote:
On the whole it was all quite successful -- the site stayed up throughout, and users got very fast and reliable downloads thanks to metalink. The torrents were originally very fast but the pace dropped down after the initial flood in. Metalinks went well throughout the day, so good stuff.
It all depends on how many downloads there were compared to last times. It could b that people run away from Novell, due to the deal, or instead became curious and downloaded even more.
It would be interesting to see the numbers compared to other versions if those are available. I understand that you can't put a total on it, but Novell could compare the number of people who 'registerd'. In perecntage compared with the real downloads, that should give a good indication.
Or just some webstats comparing the different versions in another way.
I suspect that the download is less, because e.g. /. still has not mentioned it and my submited story is still pending and I doubt that I am the only one.
houghi
houghi houghi@houghi.org writes:
On Fri, Dec 08, 2006 at 05:00:52PM +0000, Francis Giannaros wrote:
On the whole it was all quite successful -- the site stayed up throughout, and users got very fast and reliable downloads thanks to metalink. The torrents were originally very fast but the pace dropped down after the initial flood in. Metalinks went well throughout the day, so good stuff.
It all depends on how many downloads there were compared to last times. It could b that people run away from Novell, due to the deal, or instead became curious and downloaded even more.
We were a bit better prepared than last time and added for example an extra mirror network where according to our admins "Yesterday we were sustained close to 8Gb/sec for a good chunk of the day". Thanks to all the mirrors helping us!
It would be interesting to see the numbers compared to other versions if those are available. I understand that you can't put a total on it, but Novell could compare the number of people who 'registerd'. In perecntage compared with the real downloads, that should give a good indication.
Or just some webstats comparing the different versions in another way.
I suspect that the download is less, because e.g. /. still has not mentioned it and my submited story is still pending and I doubt that I am the only one.
I'm interested as well ;-)
Andreas
A question on distro packaging, I am sure this has been asked in some way previously, But here goes. What are the negatives of packaging a "1-cd install" type first disk of new releases. To be more specific, let's say I know I want the KDE and not much else, why couldn't the first disk contain enough of the distro to get me to a networked KDE desktop with nothing else except for the software maintenance tools for access to the repo's (same for Gnome)? JT
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
Francis Giannaros a écrit :
The torrents were originally very fast but the pace dropped down after the initial flood
no drop here, on the contrary the number of clients/servers increase. the more are for the i386 dvd, of course. I have now 730 sources and 3303 clients for it (326/1144 for the x86_64 dvd)
and of course all the people installing the distro can't seed :-)
jdd
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Friday 2006-12-08 at 22:17 +0100, jdd sur free wrote:
The torrents were originally very fast but the pace dropped down after the initial flood
no drop here, on the contrary the number of clients/servers increase. the more are for the i386 dvd, of course. I have now 730 sources and 3303 clients for it (326/1144 for the x86_64 dvd)
No, it is slow: my torrent download is going slow: 58% in 22 hours... ie, less than 30 KB/S avg, a third of my bandwidth. This matchs what you say, that you see more clients. And remember, adsl is asymetric, we can not seed as fast as we download.
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Sat, 9 Dec 2006, Carlos E. R. wrote:
The Friday 2006-12-08 at 22:17 +0100, jdd sur free wrote:
The torrents were originally very fast but the pace dropped down after the initial flood
no drop here, on the contrary the number of clients/servers increase. the more are for the i386 dvd, of course. I have now 730 sources and 3303 clients for it (326/1144 for the x86_64 dvd)
No, it is slow: my torrent download is going slow: 58% in 22 hours... ie, less than 30 KB/S avg, a third of my bandwidth. This matchs what you say, that you see more clients. And remember, adsl is asymetric, we can not seed as fast as we download.
You can use metalink and use the stuff I added to the page. 30 people have verified that it works with rsync to correct problems. It is extremely fast repaiging them.
-- Boyd Gerber gerberb@zenez.com ZENEZ 1042 East Fort Union #135, Midvale Utah 84047 --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Friday 2006-12-08 at 17:13 -0700, Boyd Lynn Gerber wrote:
No, it is slow: my torrent download is going slow: 58% in 22 hours... ie, less than 30 KB/S avg, a third of my bandwidth. This matchs what you say, that you see more clients. And remember, adsl is asymetric, we can not seed as fast as we download.
You can use metalink and use the stuff I added to the page. 30 people have verified that it works with rsync to correct problems. It is extremely fast repaiging them.
I know they can be repaired. But, I had no chance to try because the client destroyed the downloaded thing! And of course, it doesn't support retaking the partial torrent download, so I can't try again.
If the client had asked before overwriting the existing file...
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Sat, 9 Dec 2006, Carlos E. R. wrote:
The Friday 2006-12-08 at 22:17 +0100, jdd sur free wrote:
The torrents were originally very fast but the pace dropped down
after the
initial flood
no drop here, on the contrary the number of clients/servers
increase. the more
are for the i386 dvd, of course. I have now 730 sources and 3303
clients for
it (326/1144 for the x86_64 dvd)
No, it is slow: my torrent download is going slow: 58% in 22 hours...
ie,
less than 30 KB/S avg, a third of my bandwidth. This matchs what you
say,
that you see more clients. And remember, adsl is asymetric, we can
not
seed as fast as we download.
You can use metalink and use the stuff I added to the page. 30 people have verified that it works with rsync to correct problems. It is extremely fast repaiging them.
From the point of view of metalink, this release seems to have gone pretty
well. I spent some time in #suse & would like to thank apokryphos, yaloki, bgerber, benjiman, and the others helping everyone out that was trying to get the ISOs. And Manuel Subredu, at the RoEduNet Iasi mirror who wrote the software generating the .metalinks.
It looks like there were twenty thousand .metalink downloads in the first day.
Obviously, most mirrors were stressed from the sudden onslaught and people were having trouble getting the ISOs or getting them very slowly. Some mirrors crawl, while others do better. I saw quite a few people learning about metalink for the first time. Everyone reported good download speeds (usually their max). A few reported checksum errors, which could not be avoided due to the situation, but were repaired with rsync. (Except for Carlos' problem). The chunk checksums should soon correct this, when they've been added to clients.
I always thought manually gathering URLs was annoying and I'm glad I could finally talk some people into trying metalink. Having the multiple redundant URLs and automating it is all much simpler.
It would be nice if there were official RPMs for aria2, hopefully enough people were exposed to it and had a good experience. KDE4's KGet should also support metalink so that should be much easier for average people.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
Anthony Bryan a écrit :
I always thought manually gathering URLs was annoying and I'm glad I could finally talk some people into trying metalink. Having the multiple redundant URLs and automating it is all much simpler.
metalinks would be even better if they included bittorrent right from the start (so there would be no need to two clients style :-)
jdd
On Sunday 10 December 2006 13:33, Anthony Bryan wrote: .....
I always thought manually gathering URLs was annoying and I'm glad I could finally talk some people into trying metalink. Having the multiple redundant URLs and automating it is all much simpler.
Hi Anthony,
The idea to automate mirror (download source) selection is great, but there is so many details on and around your web page that doesn't help me.
Let me explain.
1) Project web page http://www.metalinker.org gives promisses in the same marketing style as menu commercial sites. All is so great, no problems, only benefits, the life is good.
2) All rights reserved at the bottom, sounds so proprietary. No word about license.
3) Downloads are verified for enhanced reliability. -Nothing new for YaST users.
4) No Single Point of Failure (SPOF) like FTP or HTTP URLs. -It showed that implementation is not that perfect and single point of failure ruined some people experience. Anything that comes with a lot of declarative sentences and some inserted jokes, can't give me confidence. Jokes alone are not the problem.
5) More fault tolerant. -I don't need rsync to repair anything with classic methods like wget.
6) It's a neutral standard that doesn't favor any one program, Operating system, or group, and is easy to implement. -Who made it standard? I can't find references on web site.
7) OpenOffice.org uses Metalinks. http://distribution.openoffice.org/p2p/magnet.html I'm removing listed magnet clients from any computer and explain users that they should not install them again. Do you think that anything listed on the same page will gain my trust?
As I said, the idea is very attractive, but marketing isn't. The download speed and convenient automation of process are not the only factors when someone uses Internet as a software source.
Hi Rajko,
The idea to automate mirror (download source) selection is great, but there is so many details on and around your web page that doesn't help me.
Let me explain.
- Project web page http://www.metalinker.org
gives promisses in the same marketing style as menu commercial sites. All is so great, no problems, only benefits, the life is good.
The point of the site/text is to talk people into trying it out and giving it a chance. I've done my best to give info to most of the people interested. I'm not familiar with menu commercial sites.
- All rights reserved at the bottom, sounds so proprietary. No word
about license.
That's for the text of the site. What do you mean by license? I offer no software. Some of the software that uses Metalink is under the GPL, some is commercial.
- Downloads are verified for enhanced reliability.
-Nothing new for YaST users.
Cool. Not everyone uses or can use YaST in all situations :)
- No Single Point of Failure (SPOF) like FTP or HTTP URLs.
-It showed that implementation is not that perfect and single point of failure ruined some people experience. Anything that comes with a lot of declarative sentences and some inserted jokes, can't give me confidence. Jokes alone are not the problem.
That is a reference to having unorganized single links to an ISO for instance. If the server goes down or is hammered, it will be hard to download. A lot of people experienced this for the 10.2 release. Metalinks list many URLs, so the clients can automate the process of connecting and using them, if some mirrors are down it recovers gracefully, and for instance not using some mirrors if the download rate is below a certain limit, etc.
- More fault tolerant.
-I don't need rsync to repair anything with classic methods like wget.
? Of course you have to (or do something else), or you have to start over from scratch if there's an error in transfer. As far as I know, there is no difference in the basic transfer methods of Metalink clients and wget. Someone could quickly write a Metalink interface for wget.
- It's a neutral standard that doesn't favor any one program,
Operating system, or group, and is easy to implement. -Who made it standard? I can't find references on web site.
The clients that use it. No standards body has endorsed it. If you know of any that you think would be interested, let me know.
- OpenOffice.org uses Metalinks.
http://distribution.openoffice.org/p2p/magnet.html I'm removing listed magnet clients from any computer and explain users that they should not install them again. Do you think that anything listed on the same page will gain my trust?
I'm not sure what this means. Can you explain it more? Is there something untrustworthy about magnet clients? I wasn't aware of that. Metalink just happens to be listed on the same page by OpenOffice.org. If there is something wrong with them, I will try to get removed from that page.
As I said, the idea is very attractive, but marketing isn't. The download speed and convenient automation of process are not the only factors when someone uses Internet as a software source.
I'm sorry you see it as marketing, I know I'd be a pretty bad salesman. (it's just what I wrote to get the basic idea across, I don't think it's particularly good but it seemed ok to me). I think Metalink solves some problems and of course that's fine if you disagree :) I only hope that people try it out and decide for themselves.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On 12/11/06, Anthony Bryan albryan@comcast.net wrote:
- All rights reserved at the bottom, sounds so proprietary. No word
about license.
That's for the text of the site. What do you mean by license? I offer no software. Some of the software that uses Metalink is under the GPL, some is commercial.
The fact that there is no official implementation of Metalink software is actually the main issue I have with Metalink. It feels too much like there is snake oil involved, with claims and promises and what feels like marketing, and the delivery of actual results is left to other parties.
Take Bittorrent and its rise to fame, it had an official and basic client, which was proven to work, and was then implemented by countless clients. Similarly I would like to have seen Metalinks pushed by an implementor who wanted to share their creation.
I'm sorry you see it as marketing, I know I'd be a pretty bad salesman. (it's just what I wrote to get the basic idea across, I don't think it's particularly good but it seemed ok to me). I think Metalink solves some problems and of course that's fine if you disagree :) I only hope that people try it out and decide for themselves.
Most of us are used to two worlds, the opensource - where projects are generally started to scratch your own itch and the successful ones take on a life - and the commercial - where things are made for profit. Metalink.org and your motivations for pushing metalinks are not clear.
Don't get me wrong, I am not in any way trying to discourage you in your quest or others from trying clients with Metalinks, I am just sharing my thoughts how I and maybe others see it.
Pflodo Peter Flodin.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On 12/11/06, Anthony Bryan albryan@comcast.net wrote:
- All rights reserved at the bottom, sounds so proprietary. No
word
about license.
That's for the text of the site. What do you mean by license? I offer
no
software. Some of the software that uses Metalink is under the GPL,
some is
commercial.
The fact that there is no official implementation of Metalink software is actually the main issue I have with Metalink. It feels too much like there is snake oil involved, with claims and promises and what feels like marketing, and the delivery of actual results is left to other parties.
Take Bittorrent and its rise to fame, it had an official and basic client, which was proven to work, and was then implemented by countless clients. Similarly I would like to have seen Metalinks pushed by an implementor who wanted to share their creation.
I don't have the resources of BitTorrent Inc ($28 million and deals with the MPAA).
I am sharing my creation, there are multiple parties using it and I have been trying my best to get others to use it. I've been working on Metalink on and off for over a decade, and more than full time for over a year. I fund it all myself, with absolutely no income from it.
I'm sorry you see it as marketing, I know I'd be a pretty bad
salesman.
(it's just what I wrote to get the basic idea across, I don't think
it's
particularly good but it seemed ok to me). I think Metalink solves
some
problems and of course that's fine if you disagree :) I only hope
that
people try it out and decide for themselves.
Most of us are used to two worlds, the opensource - where projects are generally started to scratch your own itch and the successful ones take on a life - and the commercial - where things are made for profit. Metalink.org and your motivations for pushing metalinks are not clear.
Don't get me wrong, I am not in any way trying to discourage you in your quest or others from trying clients with Metalinks, I am just sharing my thoughts how I and maybe others see it.
I don't get it, it is pretty discouraging. Where are you seeing anything commercial with what I am doing? (Besides GetRight and Speed Download, which have pay versions, but that has nothing to do w/ me). There aren't even ads on my site (which is down now).
BitTorrent which you mentioned is commercial with a bit of opensource, now with closed source clients and patented extensions.
This is my itch. I'm scratching it. I'd like others to join in, and some have, but more would be nice.
just having a rotten day,
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
Anthony Bryan a écrit :
I don't have the resources of BitTorrent Inc
the real problem, here is in my opinions to know if metalink is a real progress.
I mean a complete system, with bittorrent and eventually rsync addition (may be a simple batch to do that).
and is it a real progress for Novell and openSUSE in the way of spreading they distribution.
If it is not, let alone...
if it is openSUSE have the man power to get a client uptodate (aeria2 or any other)... it's as simple as that.
We all know that openSUSE (and probably others distributions as well) have mirror problems, so is this a solution? obviously not for 10.2, but for 10.3
may be Anthony didn't know sufficiently well advertise it's product. If this product is really usefull, he may appreciate any help, I'm sure...
jdd
Anthony Bryan a écrit :
I don't have the resources of BitTorrent Inc
the real problem, here is in my opinions to know if metalink is a real progress.
I mean a complete system, with bittorrent and eventually rsync addition (may be a simple batch to do that).
Metalinks can list bittorrent and rsync resources. aria2 and GetRight support the bittorrent protocol. no metalink clients currently support the rsync protocol, but (if the full file checksum does not match) I assume it could be possible for some clients to 1) add rsync support (librsync?) and automatically use it to repair the file 2) at least list the rsync servers contained in the metalink and tell users the command to repair the file. The second option wouldn't be as nice and automated, but seems pretty simple. (But I might be wrong).
and is it a real progress for Novell and openSUSE in the way of spreading they distribution.
I think most Novell and openSUSE people have been quiet about metalink, I would be interested to know their opinions. Unless they hated it :)
There were 20,000 downloads of openSUSE 10.2 metalinks the first day of release, and about 60,000 total so far.
If it is not, let alone...
if it is openSUSE have the man power to get a client uptodate (aeria2 or any other)... it's as simple as that.
We all know that openSUSE (and probably others distributions as well) have mirror problems, so is this a solution? obviously not for 10.2, but for 10.3
may be Anthony didn't know sufficiently well advertise it's product. If this product is really usefull, he may appreciate any help, I'm sure...
Definitely, I would REALLY appreciate help.
I will try to reply to everyone once I get some time, swamped at the moment.
If anyone wants more info, there's this: http://en.wikipedia.org/wiki/Metalink
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Wednesday 13 December 2006 02:42, Anthony Bryan wrote:
I think most Novell and openSUSE people have been quiet about metalink, I would be interested to know their opinions. Unless they hated it :)
Metalink concept is not completely new. Bittorrent has similar ideas, it just doesn't involve FTP and HTTP servers.
For the engineering the point of better download protocol (method) is better usage of present resources (servers and bandwith) so if they find it interesting, they will come up.
Right now, from descriptions, I can see that metalinks client introduces some overhead by checking server speed from each client (60000 times), That can be avoided with central test servers in few parts of the world, or aggregating data from clients in different parts of the world in central database and sorting metalinks accordingly.
For instance the http://download.opensuse.org can be good place for that.
On Monday 11 December 2006 01:17, Anthony Bryan wrote:
Sorry for late answer, I've seen message yesterday, but it was too late to answer. I'm trying to be helpful and that takes time.
Hi Rajko,
The idea to automate mirror (download source) selection is great, but there is so many details on and around your web page that doesn't help me.
Let me explain.
- Project web page http://www.metalinker.org
gives promisses in the same marketing style as menu commercial sites. All is so great, no problems, only benefits, the life is good.
The point of the site/text is to talk people into trying it out and giving it a chance. I've done my best to give info to most of the people interested. I'm not familiar with menu commercial sites.
It should be "many" sorry for typo. I mentioned the style where many words are used to praise product and no word what are the limitations. More technical details about methods to achieve download will be more than welcome. Open talk about underlaying ideas will give everybody different feeling about your project.
The way you did it is OK for commercial software, where ideas how it works are hidden from public.
- All rights reserved at the bottom, sounds so proprietary. No word
about license.
That's for the text of the site. What do you mean by license? I offer no software. Some of the software that uses Metalink is under the GPL, some is commercial.
OK. The problem is that with previously mentioned ,missing talk about ideas, it gives feeling that whatever you offer is proprietary. If you would mention anywhere that described method is patent free and content of web site is protected with some kind of opensource license than there will be no reasons for questions.
- Downloads are verified for enhanced reliability.
-Nothing new for YaST users.
Cool. Not everyone uses or can use YaST in all situations :)
I agree. That is actually trivial and missing feature in many download products.
- No Single Point of Failure (SPOF) like FTP or HTTP URLs.
-It showed that implementation is not that perfect and single point of failure ruined some people experience. Anything that comes with a lot of declarative sentences and some inserted jokes, can't give me confidence. Jokes alone are not the problem.
This was explained above. More technical details will give me and many others more confidence that metalink is worth to try.
That is a reference to having unorganized single links to an ISO for instance. If the server goes down or is hammered, it will be hard to download. A lot of people experienced this for the 10.2 release. Metalinks list many URLs, so the clients can automate the process of connecting and using them, if some mirrors are down it recovers gracefully, and for instance not using some mirrors if the download rate is below a certain limit, etc.
- More fault tolerant.
-I don't need rsync to repair anything with classic methods like wget.
? Of course you have to (or do something else), or you have to start over from scratch if there's an error in transfer. As far as I know, there is no difference in the basic transfer methods of Metalink clients and wget. Someone could quickly write a Metalink interface for wget.
The problem is that TCP has error correction and it will request packages that are bad. The only thing that can brake download is timeout in transmission and that can happen in rush hours on new release. In that case wget -c will not start from scratch.
- It's a neutral standard that doesn't favor any one program,
Operating system, or group, and is easy to implement. -Who made it standard? I can't find references on web site.
The clients that use it. No standards body has endorsed it. If you know of any that you think would be interested, let me know.
This is new method and it is far from being even "de facto" standard because everybody is using it. If you would advertised it as a "new method that doesn't favor any one program, Operating system, or group, and is easy to implement." it will be exactly what it is and no one will complain. Well, I'm the only one that said this loud, but be sure that many think this way.
- OpenOffice.org uses Metalinks.
http://distribution.openoffice.org/p2p/magnet.html I'm removing listed magnet clients from any computer and explain users that they should not install them again. Do you think that anything listed on the same page will gain my trust?
I'm not sure what this means. Can you explain it more? Is there something untrustworthy about magnet clients? I wasn't aware of that. Metalink just happens to be listed on the same page by OpenOffice.org. If there is something wrong with them, I will try to get removed from that page.
Google returns links like this: http://www.wired.com/news/privacy/0,1848,49430,00.html http://news.com.com/2100-1038_3-5215028.html http://www.pestpatrol.com/spywarecenter/pest.aspx?id=453088059 (note versions) Could be also something like this: http://www.gnutellaforums.com/showthread.php?t=40161
So after some time spent in research I can understand why OP didn't understand my comment about magnet clients. It can be that it was just old version or hacked copies that gave me a lot of problems on few occasions.
As I said, the idea is very attractive, but marketing isn't. The download speed and convenient automation of process are not the only factors when someone uses Internet as a software source.
I'm sorry you see it as marketing, I know I'd be a pretty bad salesman. (it's just what I wrote to get the basic idea across, I don't think it's particularly good but it seemed ok to me). I think Metalink solves some problems and of course that's fine if you disagree :) I only hope that people try it out and decide for themselves.
I actually do not disagree with idea and I stated that more than one time, but your presentation of metalinks is missing details. I would like that you give some technical background how it works and to clear legal status. Is it proprietary technology or opensource?
I have no problem with any technology as long as it is explained, software works as described and the price is right :-) but I have problem with any black boxes that do something, but even author can't explain details. That is, looking to some other posts, not only my impression.
Present one (or few) clients in more details. You didn't make them, but they will make whole idea fly or fail, so you have to take time and present them with much more than one screenshot and few links. It will give to visitors feeling that whole idea is a serious technical work and that is what many are looking for before give a test drive.
Hi again Rajko,
It should be "many" sorry for typo. I mentioned the style where many words are used to praise product and no word what are the limitations. More technical details about methods to achieve download will be more than welcome. Open talk about underlaying ideas will give everybody different feeling about your project.
The way you did it is OK for commercial software, where ideas how it works are hidden from public.
Nothing at all is hidden from the public.
From the site:
Metalink is a simple XML (text) format. Download one and open it with a text editor, & most of it should be self explanatory. If you want to read the long, boring Metalink 3.0 specification, here you go:
http://www.metalinker.org/Metalink_3.0_Spec.odt http://www.metalinker.org/Metalink_3.0_Spec.pdf
(I think you are looking for section 3.2).
Here's a .metalink (minus most <url>s for brevity). The client can only use the information that's there. It's pretty straightforward.
<file name="openSUSE-10.2-GM-DVD-i386.iso">
<os>Linux-x86</os>
<size>3880814592</size> <verification> <hash type="md5">12b0f2fabb4d41586601f787d33ebc34</hash> </verification> <resources> <url type="http" location="jp" preference="10">
http://ftp.kddilabs.jp/Linux/packages/opensuse/distribution/openSUSE-current /iso/dvd/openSUSE-10.2-GM-DVD-i386.iso </url> <url type="ftp" location="tw" preference="10">
ftp://ftp.isu.edu.tw/Linux/SuSE/distribution/openSUSE-current/iso/dvd/openSU SE-10.2-GM-DVD-i386.iso </url> <url type="http" location="ch" preference="10">
http://mirror.switch.ch/ftp/mirror/opensuse/distribution/openSUSE-current/is o/dvd/openSUSE-10.2-GM-DVD-i386.iso </url> </resources> </file>
OK. The problem is that with previously mentioned ,missing talk about ideas, it gives feeling that whatever you offer is proprietary. If you would mention anywhere that described method is patent free and content of web site is protected with some kind of opensource license than there will be no reasons for questions.
Metalink is patent free, unlike BitTorrent. (BT is awesome and the best thing to happen to the net since the web).
The content of my website is under Creative Commons Attribution License. I believe this is the correct license for this type of thing, but I'm not very familiar w/ licenses. If someone knows of a more suitable one, please let me know.
- Downloads are verified for enhanced reliability.
-Nothing new for YaST users.
Cool. Not everyone uses or can use YaST in all situations :)
I agree. That is actually trivial and missing feature in many download products.
Yes, I am trying to encourage full file checksumming to be automated, as you can see. All the clients support this now. It may not be much, but it simplifies things for some people who aren't familiar with doing that.
The partial file/segment checksums are also available and I am encouraging that as well - this will allow the automatic error correction. Some of the clients are adding this now. I guess it isn't as simple as the full file checksums. I don't know how far along aria2 is if at all, but if someone wanted to submit a patch that might not hurt. (Might want to contact the author or wait until the next release).
- OpenOffice.org uses Metalinks.
http://distribution.openoffice.org/p2p/magnet.html I'm removing listed magnet clients from any computer and explain
users
that they should not install them again. Do you think that anything
listed
on the same page will gain my trust?
I'm not sure what this means. Can you explain it more? Is there
something
untrustworthy about magnet clients? I wasn't aware of that. Metalink
just
happens to be listed on the same page by OpenOffice.org. If there is something wrong with them, I will try to get removed from that page.
Google returns links like this: http://www.wired.com/news/privacy/0,1848,49430,00.html http://news.com.com/2100-1038_3-5215028.html http://www.pestpatrol.com/spywarecenter/pest.aspx?id=453088059 (note versions) Could be also something like this: http://www.gnutellaforums.com/showthread.php?t=40161
So after some time spent in research I can understand why OP didn't understand my comment about magnet clients. It can be that it was just old version or hacked copies that gave me a lot of problems on few occasions.
Ok, glad you understand that bad clients that may have used magnet links have no connection with metalink.
I actually do not disagree with idea and I stated that more than one time, but your presentation of metalinks is missing details. I would like that you give some technical background how it works and to clear legal status. Is it proprietary technology or opensource?
You are the first person to request details. I assume people really interested read the spec I mentioned above. Most people don't care about that much detail.
As I've mentioned before, metalink is not proprietary. I don't know how it could be open source, since I don't offer any actual programs or source. It's something developers have implemented in both proprietary and open source programs.
I have no problem with any technology as long as it is explained, software works as described and the price is right :-) but I have problem with any black boxes that do something, but even author can't explain details. That is, looking to some other posts, not only my impression.
If you're familiar with download managers and checksums then you already understand it.
Obviously, there are no black boxes in the open source clients and a plain text XML file!
The rsync should not be necessary. It is proof that either metalink
concept or
implementation is not good in all details. BTW, I an use rsync alone and skip any repair.
rsync was offered as a temporary solution until aria2 can use the partial file checksums for error correction.
From Francis: Anyhow, at the moment I think, considering all the flaws of the current implementation, it's still hands-down the top method I'd use. I explained why. Fastest ever download speeds, and if you use rsync it's always guaranteed. Again, not perfect, but still very very good for me. Considering I'd perhaps only ever have to do rsync 1 in 100 times.
The client can only go with what you give it. If it's downloading from ftp/http, that's all its downloading from. There is just the automated checksum at the end, which will tell you if there were errors. I suppose the larger the file, the larger the chance of errors. If you're downloading an ISO over 2, 4 or 8 hrs, chances are probably higher that you will have an error in there somewhere. I'm sure there are lots of factors, but the bottom line is that for most people errors and rare but unavoidable with ftp/http. I've downloaded over a thousand files with metalinks, testing various stuff, and have only had one checksum error. I guess that's a testament to the quality of my connection. And thankfully it was a small couple hundred k file, not a 3G monster ISO.
Once aria2 and other metalink clients support the partial file checksums, all this will be moot. I can't afford to pay people to work on this, so they will do it on their own time schedule. I'm sorry some people had problems, especially the file overwrite bug that someone ran into.
Some other neat features which are still unsupported are inclusion of PGP signatures and some other stuff.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Sunday 17 December 2006 15:51, Anthony Bryan wrote:
Hi again Rajko,
...
From the site: Metalink is a simple XML (text) format. Download one and open it with a text editor, & most of it should be self explanatory. If you want to read the long, boring Metalink 3.0 specification, here you go:
http://www.metalinker.org/Metalink_3.0_Spec.odt http://www.metalinker.org/Metalink_3.0_Spec.pdf
Thanks for this. ...
Metalink is patent free, unlike BitTorrent. (BT is awesome and the best thing to happen to the net since the web).
The content of my website is under Creative Commons Attribution License. I believe this is the correct license for this type of thing, but I'm not very familiar w/ licenses. If someone knows of a more suitable one, please let me know.
Creative Commons Attribution License is for me good as any other that doesn't limit flow of ideas. ...
The partial file/segment checksums are also available and I am encouraging that as well - this will allow the automatic error correction. Some of the clients are adding this now. I guess it isn't as simple as the full file checksums. I don't know how far along aria2 is if at all, but if someone wanted to submit a patch that might not hurt. (Might want to contact the author or wait until the next release).
I'm not developer, but I can try to break it, if possible, and than complain to developers about implementation :-) ...
my comment about magnet clients. It can be that it was just old version or hacked copies that gave me a lot of problems on few occasions.
Ok, glad you understand that bad clients that may have used magnet links have no connection with metalink.
But, one particular magnet client I can't recommend as my friends will not check for version and source, for sure. What I actually do is to recommend some subscription service like Rhapsody, not expensive and works for me, so it may work for them too.
I actually do not disagree with idea and I stated that more than one time, but your presentation of metalinks is missing details. I would like that you give some technical background how it works and to clear legal status. Is it proprietary technology or opensource?
You are the first person to request details. I assume people really interested read the spec I mentioned above. Most people don't care about that much detail.
The specification is enough details. The most of the people are using OS that drains pockets with add this, add that, upgrade software, upgrade hardware, and still they don't want to try financially better option finding million reasons why is better what they have. ...
The client can only go with what you give it. If it's downloading from ftp/http, that's all its downloading from. There is just the automated checksum at the end, which will tell you if there were errors. I suppose the larger the file, the larger the chance of errors. If you're downloading an ISO over 2, 4 or 8 hrs, chances are probably higher that you will have an error in there somewhere. I'm sure there are lots of factors, but the bottom line is that for most people errors and rare but unavoidable with ftp/http. I've downloaded over a thousand files with metalinks, testing various stuff, and have only had one checksum error. I guess that's a testament to the quality of my connection. And thankfully it was a small couple hundred k file, not a 3G monster ISO.
The http and ftp protocols are using built in error correction of TCP/IP and that will fail sometimes, under circumstances that occur more often in congestions like one on a release day. ...
Once aria2 and other metalink clients support the partial file checksums, all this will be moot.
Someone has to make them available, software maker or guys that create metalinks.
I can't afford to pay people to work on this, so they will do it on their own time schedule. I'm sorry some people had problems, especially the file overwrite bug that someone ran into.
It happens with software.
Some other neat features which are still unsupported are inclusion of PGP signatures and some other stuff.
That is what is necessary to verify source of files.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Sunday 2006-12-17 at 18:45 -0600, Rajko M. wrote:
...
Some other neat features which are still unsupported are inclusion of PGP signatures and some other stuff.
That is what is necessary to verify source of files.
IMO, it would be suficient to sign the xml metalink file itself. As it contains the md5sum check of the image, that would enough to certify that what you downloaded was the correct signed file.
Also, segment md5sums could be used to certify mirror sites: if a segment downloaded from a site doesn't check, and a retry fails again, that would mark that site as "bad" or bogus or whatever.
An alternative is to sign the image, but that would be better done by the image provider/maker.
Tricky problem! ;-)
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Monday 18 December 2006 09:30, Carlos E. R. wrote:
The Sunday 2006-12-17 at 18:45 -0600, Rajko M. wrote:
...
Some other neat features which are still unsupported are inclusion of PGP signatures and some other stuff.
That is what is necessary to verify source of files.
IMO, it would be suficient to sign the xml metalink file itself. As it contains the md5sum check of the image, that would enough to certify that what you downloaded was the correct signed file.
Also, segment md5sums could be used to certify mirror sites: if a segment downloaded from a site doesn't check, and a retry fails again, that would mark that site as "bad" or bogus or whatever.
An alternative is to sign the image, but that would be better done by the image provider/maker.
Tricky problem! ;-)
The links to spec files in one of other messages http://www.metalinker.org/Metalink_3.0_Spec.odt http://www.metalinker.org/Metalink_3.0_Spec.pdf
I haven't time to read specifications, so I can't say is verification included and if it is what method is used. Bryan mentioned that it will come in future versions of aria2.
The metalink might be good to distribute load on servers, but it needs improvements.
Now each client is testing servers independently. This produces some overhead that can be skipped if traffic data will be collected to central server from clients that start using metalink and after that metalink file will be changed to point to free resources ie. servers that show good performance.
The problem is not trivial, as change in metalink file will change distribution of the load and than reports will be changed. That means that we have feedback that has to be regulated to prevent system to go wild and lock everybody out.
Another problem is that each client should use minimum number of connections ie. servers to reach maximum download speed and than stop asking for more as each server has its maximum, so it will prevent others to download.
I have more questions than answers, and some reading is pending :-)
From Carlos:
The Sunday 2006-12-17 at 18:45 -0600, Rajko M. wrote:
Some other neat features which are still unsupported are inclusion
of PGP
signatures and some other stuff.
That is what is necessary to verify source of files.
IMO, it would be suficient to sign the xml metalink file itself. As it contains the md5sum check of the image, that would enough to certify that what you downloaded was the correct signed file.
That is definitely a possibility.
Also, segment md5sums could be used to certify mirror sites: if a segment downloaded from a site doesn't check, and a retry fails again, that would mark that site as "bad" or bogus or whatever.
Yes.
An alternative is to sign the image, but that would be better done by the image provider/maker.
Tricky problem! ;-)
Yes, hopefully whoever makes the ISO will sign them (if they're interested in this).
Here's the currently unimplemented idea for metalink, which could change if the clients find out it is clumsy or bad once they start putting it into practice.
For those unfamiliar, Linux Kernels (linux-2.6.19.1.tar.bz2) are signed with a separate file (linux-2.6.19.1.tar.bz2.sign) which looks like this:
</signature> </verification>
Or just listed as another file within the metalink
<resources>
<url>http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.19.1.tar.bz2.sign </url> </resources>
At the least, a client could recognize when a signature is included in the metalink and tell the client to verify it manually
gpg --verify linux-2.6.19.1.tar.bz2.sign linux-2.6.19.1.tar.bz2
But, I'm hoping it won't be too hard to automate this.
From Rajko:
The metalink might be good to distribute load on servers, but it needs improvements.
Now each client is testing servers independently. This produces some overhead that can be skipped if traffic data will be collected to central server from clients that start using metalink and after that metalink file will be changed to point to free resources ie. servers that show good performance.
The problem is not trivial, as change in metalink file will change distribution of the load and than reports will be changed. That means that we have feedback that has to be regulated to prevent system to go wild and lock everybody out.
Yes, there is currently no program that monitors the mirror servers and dynamically updates .metalink files according to mirror load.
I see this as being really useful at the busiest times, but maybe not as critical during normal times. But I might be wrong. I do think it would be really cool if someone was interested and would work on it tho.
Another problem is that each client should use minimum number of connections ie. servers to reach maximum download speed and than stop asking for more as each server has its maximum, so it will prevent others to download.
Yes, aria2 uses 15 connections to 15 different servers by default AFAIR. This can be changed with command line options. Or if a distribution shipped a binary, they could decide on a number they felt was better. I believe all the other clients go by whatever options the user has set, (I don't know the defaults off hand for all of them). It would be nice if clients could recognize they were downloading at the users maximum speed for their connection with 2 connections, so please don't open up 13 more.
It might be hard to fine tune what is most efficient. For dialup users, 1 is probably plenty. For me on cable, 4-6 is usually good in most conditions. I'd also like to find a number that is respectful to mirror server resources.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Tuesday 2006-12-19 at 01:15 -0500, Anthony Bryan wrote:
...
Here's the currently unimplemented idea for metalink, which could change if the clients find out it is clumsy or bad once they start putting it into practice.
For those unfamiliar, Linux Kernels (linux-2.6.19.1.tar.bz2) are signed with a separate file (linux-2.6.19.1.tar.bz2.sign) which looks like this:
...
This can be easily inside the Metalink, like this:
<verification> <signature type="pgp" file="linux-2.6.19.1.tar.bz2.sign"> -----BEGIN PGP SIGNATURE-----
...
That would be nice: if the file provider has signed it first, the metalink creator can include it. Otherwise, just sign the metalink file.
The "snag" is that checking both checksum and signature doubles checking time. I suppose that the user could select which one to use, same as rpm does, having both systems too, if I'm not mistaken.
...
From Rajko:
Yes, there is currently no program that monitors the mirror servers and dynamically updates .metalink files according to mirror load.
I see this as being really useful at the busiest times, but maybe not as critical during normal times. But I might be wrong. I do think it would be really cool if someone was interested and would work on it tho.
I suppose that if the client sees that a server is serving slowly it can switch to a different one automatically and dynamically. After all, it is connected to several and it can compare. Perhaps having a central server recommending which servers to use, or modifying the metalink file dynamically would be even better, but I think it works as it is.
Anyway... a server listing current mirror speeds or load would also be useful to other people using plain ftp/http. It could be useful for selecting an update server for Yast/YOU, too.
Another problem is that each client should use minimum number of connections ie. servers to reach maximum download speed and than stop asking for more as each server has its maximum, so it will prevent others to download.
Yes, aria2 uses 15 connections to 15 different servers by default AFAIR.
But not several connections to the same server: that would be considered aggressive, I suppose. I don't think it does that, AFAIK, so that's good.
This can be changed with command line options. Or if a distribution shipped a binary, they could decide on a number they felt was better. I believe all the other clients go by whatever options the user has set, (I don't know the defaults off hand for all of them). It would be nice if clients could recognize they were downloading at the users maximum speed for their connection with 2 connections, so please don't open up 13 more.
I saw aria2c starting at 15, and later dropping to 4 or 5 automatically, maxing throughput. That's working well, I think.
It might be hard to fine tune what is most efficient. For dialup users, 1 is probably plenty. For me on cable, 4-6 is usually good in most conditions. I'd also like to find a number that is respectful to mirror server resources.
I suppose the client can tune automatically to use the minimum number that maxes speed.
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Tuesday 19 December 2006 05:33, Carlos E. R. wrote:
I saw aria2c starting at 15, and later dropping to 4 or 5 automatically, maxing throughput. That's working well, I think.
That means it is implemented to use just enough to have maximum speed. Good.
On Tue, 19 Dec 2006, Carlos E. R. wrote:
The "snag" is that checking both checksum and signature doubles checking time. I suppose that the user could select which one to use, same as rpm does, having both systems too, if I'm not mistaken.
A cryptographic signature should detect data corruption as well as a checksum -- after all it needs to detect willful tampering, not just technical corruption. So that should suffice, shouldn't it?
Gerald --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Tue, 19 Dec 2006, Carlos E. R. wrote:
The "snag" is that checking both checksum and signature doubles
checking
time. I suppose that the user could select which one to use, same as
rpm
does, having both systems too, if I'm not mistaken.
A cryptographic signature should detect data corruption as well as a checksum -- after all it needs to detect willful tampering, not just technical corruption. So that should suffice, shouldn't it?
Definitely.
To put this in context, we were talking about metalink which is an XML list of mirrors, checksums, and signatures for easier downloading/file distribution. metalinks can contain:
partial file checksums for repairing a download full file checksums cryptographic signature
The problem is that there are multiple clients on multiple operating systems. Almost all clients support MD5/SHA-1 full file checksums now. None of them support partial file checksums or signatures so far. I doubt the Windows and Mac clients will support PGP signatures, but maybe. It seems like the Linux clients will be more likely to have GPG and integrate it. So, a client that supports signatures could use the partial file checksums for errors in transfer, then just use the signature and skip the full file checksum. Ones that don't support signatures can use the partial and full file checksums.
Support for metalink in KGet in KDE4 seems to be making progress.
(( Anthony Bryan )) Metalink [ http://www.metalinker.org ]
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Monday 2006-12-11 at 00:31 -0600, Rajko M. wrote:
- More fault tolerant.
-I don't need rsync to repair anything with classic methods like wget.
That's not completely true - neither affirmation.
Wget can fail, and does sometimes, although rarely. There is no filechecking in the ftp or http download protocol, not of the end result, that is (tcp has package error checking, I understand). Sometimes downloads do get errors. Even with a solid protocol as bittorrent, some people using ktorrent have complained of bad checksum (which other implementations of torrent can correct, by the way - ktorrent bug?)
Nor is metalink is more fault tolerant - in my brief experience and understanding -. What it does is a final checksum of the downloaded file, but it has no error correction method embedded. You are left to download again or repair - same as with any other protocol. It is just a convenience that it does an automated final check.
It would be more fault tolerant if it could, would, check the download as it progresses and re download the faulty segment - the segment could be big, but it would be better than re-downloading the whole thing. This would serve also to mark servers as "bad" if a second download from them fail again.
To be fair, the protocol allows for that. But! 1) the metalinks files we were provided did not contain segment checksum and 2) the available client (aria2c) doesn't support them, anyway (dunno about wxdfast, because it refused to download anything in my system).
I would be even nicer if metalink had support for rsync servers and use them to correct the end result automatically.
If the idea of the sentence "More fault tolerant" means that it will download from a different server if one fails, well, ok, but I think my above observations nullify that. Either the sentence is qualified, or it is false.
So... it _may_ be more fault tolerant, but it isn't, currently.
- OpenOffice.org uses Metalinks.
http://distribution.openoffice.org/p2p/magnet.html I'm removing listed magnet clients from any computer and explain users that they should not install them again. Do you think that anything listed on the same page will gain my trust?
I also don't understand that. :-?
- -- Cheers, Carlos E. R.
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
On Monday 11 December 2006 06:31, Rajko M. wrote:
- Project web page http://www.metalinker.org
gives promisses in the same marketing style as menu commercial sites. All is so great, no problems, only benefits, the life is good.
Don't be ridiculous, there's absolutely nothing wrong with shining up your project with a nice website page. In fact that's very good. When I see a new open source project, I'm interested in (i) what it does, (ii) how it helps me.
If you want a commercial-like website see: http://basket.kde.org/
...and what's wrong about such sites? Absolutely nothing. You're confusing traits that might be common to open source and commercial apps and then necessarily associating one of them solely with the other
- Downloads are verified for enhanced reliability.
-Nothing new for YaST users.
Something new for wget users.
- No Single Point of Failure (SPOF) like FTP or HTTP URLs.
-It showed that implementation is not that perfect and single point of failure ruined some people experience. Anything that comes with a lot of declarative sentences and some inserted jokes, can't give me confidence. Jokes alone are not the problem.
That's an issue for you to take on with yourself. Needless to say I wouldn't say this was a common trait.
- More fault tolerant.
-I don't need rsync to repair anything with classic methods like wget.
Rubbish. You can and do.
As I said, the idea is very attractive, but marketing isn't.
(i) there's nothing wrong with marketing. I'm not sure you really understand the term actually, since SUSE (and every other project practically) has marketing, but just different levels. There is absolutely nothing inherently wrong with marketing, but only the way in which it's delivered. Again, there's absolutely nothing striking about metalink's marketing.
The download speed and convenient automation of process are not the only factors when someone uses Internet as a software source.
It would be foolish to presuppose that they're not big, or rather BIG, factors. You think I'm going to wait around at release time for a slow torrent or a non-working main site like? Or, you think I'm going to hunt around through a long list of mirrors, when metalink can do it all for me?
At release time, now, if I wanted to get an image straight away, there's absolutely no way I'd go for any of the other methods. Why would you? Download is super fast, and if you actually have problems at the end of the download you can use rsync.
Needless to say, this point is moot since we've actually since the contrary in our very raw environment. Fedora had very good things to say about it, but there was plenty of first-hand experience for opensuse users in #suse. It was tested, it was proven that it can work very very well.
Regards, Francis.
On Tuesday 12 December 2006 15:39, Francis Giannaros wrote:
On Monday 11 December 2006 06:31, Rajko M. wrote:
- Project web page http://www.metalinker.org
gives promisses in the same marketing style as menu commercial sites. All is so great, no problems, only benefits, the life is good.
Don't be ridiculous, there's absolutely nothing wrong with shining up your project with a nice website page. In fact that's very good. When I see a new open source project, I'm interested in (i) what it does, (ii) how it helps me.
The problem is shiny doesn't go far enough. I need more details about method, before I can say that it will work for me.
If you want a commercial-like website see: http://basket.kde.org/
...and what's wrong about such sites? Absolutely nothing. You're confusing traits that might be common to open source and commercial apps and then necessarily associating one of them solely with the other
Commercial applications that offer empty shell wave with features and lack details. One (not so good) screenshot and few links to clients is not enough to present one technical idea.
- Downloads are verified for enhanced reliability.
-Nothing new for YaST users.
Something new for wget users.
- No Single Point of Failure (SPOF) like FTP or HTTP URLs.
-It showed that implementation is not that perfect and single point of failure ruined some people experience. Anything that comes with a lot of declarative sentences and some inserted jokes, can't give me confidence. Jokes alone are not the problem.
That's an issue for you to take on with yourself. Needless to say I wouldn't say this was a common trait.
One of main ideas of metalinks is to remove errors that are not covered with TCP/IP transfer, so if it happens to anyone that download fails because some server has problem, we are back to square one. The problem that you don't want to see is that the number of successes doesn't prove that implementation is good, and one failure proves opposite. It works the same as with software bugs.
- More fault tolerant.
-I don't need rsync to repair anything with classic methods like wget.
Rubbish. You can and do.
The wget works fine in normal traffic. It can fail if server (any on the road from source to destination) fails, and that should be covered with metalinks.
As I said, the idea is very attractive, but marketing isn't.
(i) there's nothing wrong with marketing. I'm not sure you really understand the term actually, since SUSE (and every other project practically) has marketing, but just different levels. There is absolutely nothing inherently wrong with marketing, but only the way in which it's delivered. Again, there's absolutely nothing striking about metalink's marketing.
I have no problem understanding marketing Francis, but there is marketing and "marketing". I would like to see metalins idea presented in the way that will satisfy not only people that click on anything clickable, but also those that really read the page before they click.
The download speed and convenient automation of process are not the only factors when someone uses Internet as a software source.
It would be foolish to presuppose that they're not big, or rather BIG, factors. You think I'm going to wait around at release time for a slow torrent or a non-working main site like? Or, you think I'm going to hunt around through a long list of mirrors, when metalink can do it all for me?
The speed and automation are important, but not only things that help me to accept new idea. For me is more important that metalinks clients really work the way they should, and for that I need to know how metalinks is designend. If you have any link to technical reference I would appreciate it.
At release time, now, if I wanted to get an image straight away, there's absolutely no way I'd go for any of the other methods. Why would you? Download is super fast, and if you actually have problems at the end of the download you can use rsync.
The rsync should not be necessary. It is proof that either metalink concept or implementation is not good in all details. BTW, I an use rsync alone and skip any repair.
Needless to say, this point is moot since we've actually since the contrary in our very raw environment. Fedora had very good things to say about it, but there was plenty of first-hand experience for opensuse users in #suse. It was tested, it was proven that it can work very very well.
Regards, Francis.
The Fedora, #suse and any size of positive comments will not help to convince any carefull reader. I can see only black box that sometimes fails on one of it's important features and no one can tell why.
I would like that metalinks works perfectly and to be what web site is promising, fine complement to present file transfer options that we have right now. The speed improvement is fine, but now developers have to see why it failed and how to skip rsync phase in the future.
On Wednesday 13 December 2006 05:14, Rajko M. wrote:
Commercial applications that offer empty shell wave with features and lack details. One (not so good) screenshot and few links to clients is not enough to present one technical idea.
The proposition that it offers empty shell wave features is completely unjustified. It quite clearly claims *what* it offers and how. Whether it delivers on all of the points is perhaps questionable for you, but it's emphatically clear that it does deliver on some of them.
The problem that you don't want to see is that the number of successes doesn't prove that implementation is good, and one failure proves opposite. It works the same as with software bugs.
Currently this is down to the incomplete implementation in aria2, granted. But your conclusion doesn't follow. If it works for hundreds of people then it is most certainly a decent implementation, but not a perfect one. So currently, we make do, and use rsync if there are any errors. You might not like it, hundreds of other people don't mind, if it's the way to go for the time-being.
The wget works fine in normal traffic. It can fail if server (any on the road from source to destination) fails, and that should be covered with metalinks.
That's the job of the client, and it's being implemented into aria2 as we speak I believe.
The rsync should not be necessary. It is proof that either metalink concept or implementation is not good in all details.
I completely agree, and it's the implementation isn't perfect yet. That doesn't mean it's not decent.
BTW, I an use rsync alone and skip any repair.
Which leaves you with (i) looking through some 51 mirrors to find a fast one, or (ii) dealing with an incredibly low download speed. Not a solution in my opinion, for release time.
The Fedora, #suse and any size of positive comments will not help to convince any carefull reader. I can see only black box that sometimes fails on one of it's important features and no one can tell why.
I'm not convinced that it's a black box: everything that's done is quite open. aria2 is free software, and metalinks is an idea quite freely available. There's also a difference between a "careful reader" and a "cautious reader". A careful reader will notice that the description doesn't provide enough information (if it doesn't), but it's the cautious reader that goes the extra step.
Anyhow, at the moment I think, considering all the flaws of the current implementation, it's still hands-down the top method I'd use. I explained why. Fastest ever download speeds, and if you use rsync it's always guaranteed. Again, not perfect, but still very very good for me. Considering I'd perhaps only ever have to do rsync 1 in 100 times.
I would like that metalinks works perfectly and to be what web site is promising, fine complement to present file transfer options that we have right now. The speed improvement is fine, but now developers have to see why it failed and how to skip rsync phase in the future.
Agreed.
Regards, Francis.
Carlos E. R. a écrit :
No, it is slow: my torrent download is going slow: 58% in 22 hours... ie, less than 30 KB/S avg, a third of my bandwidth. This matchs what you say, that you see more clients. And remember, adsl is asymetric, we can not seed as fast as we download.
I know (I have 10/1Mb), I mean only the numbers are encreasing, so there is more interest, but the ration dl/seed is not enough :-(, may be because one needs to install on his only computer (not anybody have 4 computers at home as I do :-()
jdd
sorry, bad link in previous mail :-(
opensuse users, read ahead :-) thanks
Anthony Bryan a écrit :
What I was proposing on the list (maybe not explicit enough) was that the site goes into crisis mode. Only list metalinks and torrents, and rely on p2p/mirrors to distribute the ISOs. Either stop the main site from ISO downloads, or severely limit it so the site can stay up. Only list the most important info.
to be practical, I propose that you spread this very post. I already crosspost to opensuse list. Here are the instruction, feel freee to edit is I made any mistake.
If you have problems joining the opensuse.org site or any mirror during the first days of the 10.2 release, there is a solution.
The solution is to use metalink. This utility, in very short summary, make a balance between all the available ftp sources and allows you to download from the better one.
as long as the site is available you can look at http://en.opensuse.org/Released_Version#Metalink
The .metalink file is approximately like a torrent file, very small and easy to catch.
you need a metalink client, there are many for any system (for example aria2 or wxDownload).
then look at
http://download.packages.ro/metalink/opensuse/
with your browser. This ftp is _not_ a opensuse mirror, so should not be down
test your system with the 10.2 rc1, for example. GM and 10.2 are not yet functional, but will be asap (the dec 7)
enjoy :-)
Hello,
Am Samstag, 2. Dezember 2006 13:10 schrieb Francis Giannaros:
Are there any extra measures in place to prevent opensuse.org going down before the official 10.2 announcement, because of the hype?
IMHO the proposal I made after the last downtime would still be a good idea: - wget -r all wikis - excluding "index.php?..." because page history etc. is not really needed if the server is more or less down - to avoid misunderstandings: to be done _once_ by SUSE team, not by everybody reading this mail ;-) - if the server load is too high, replace the "real" wiki with the static copy from wget.
The static copy will be read-only of course, but still has all information available. That's much better than the "we are DOSed by our own success" page you used last time ;-)
Regards,
Christian Boltz
Christian Boltz a écrit :
Hello,
Am Samstag, 2. Dezember 2006 13:10 schrieb Francis Giannaros:
Are there any extra measures in place to prevent opensuse.org going down before the official 10.2 announcement, because of the hype?
IMHO the proposal I made after the last downtime would still be a good idea:
- wget -r all wikis
- excluding "index.php?..." because page history etc. is not really needed if the server is more or less down
- to avoid misunderstandings: to be done _once_ by SUSE team, not by everybody reading this mail ;-)
- if the server load is too high, replace the "real" wiki with the static copy from wget.
The static copy will be read-only of course, but still has all information available. That's much better than the "we are DOSed by our own success" page you used last time ;-)
Regards,
Christian Boltz
very good idea jdd
On Sat, Dec 02, 2006 at 12:10:20PM +0000, Francis Giannaros wrote:
Are there any extra measures in place to prevent opensuse.org going down before the official 10.2 announcement, because of the hype?
You may rest assured that we will take appropriate measures to deal with the anticipated "hype" ;)
Best, Christoph --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-project+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-project+help@opensuse.org
-
Andreas Jaeger -
Anthony Bryan -
Boyd Lynn Gerber -
Carlos E. R. -
Christian Boltz -
Christoph Thiel -
Darryl Gregorash -
Francis Giannaros -
Gerald Pfeifer -
houghi -
James Tremblay -
jdd sur free -
Peter Flodin -
Rajko M. -
Stephan Binner