[opensuse] repository metatdata download times
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time. So is it using some kind of braindead download method or something? What can we do to improve it? Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Tue, 6 Dec 2016 14:23:05 +0100 "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully.
Exactly. My question is why? (Or WTF in modern parlance) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Op dinsdag 6 december 2016 19:50:26 CET schreef Dave Howorth:
On Tue, 6 Dec 2016 14:23:05 +0100
"Carlos E. R." <robin.listas@telefonica.net> wrote:
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully.
Exactly. My question is why? (Or WTF in modern parlance)
If it's up to me, stick to proper phrasing. Makes understanding others a bit less complicated IMNSHO. OT. 1 Byte = 8 bits, a factor 8. Which can make a difference in speed experience. Which repos are we talking about? The distro's repodata are > 100 MB AFAIK, but it's useless to refresh those repos, since they don't change after release. -- Gertjan Lettink, a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 06 Dec 2016 21:00:08 +0100 Knurpht - Gertjan Lettink <knurpht@opensuse.org> wrote:
Op dinsdag 6 december 2016 19:50:26 CET schreef Dave Howorth:
On Tue, 6 Dec 2016 14:23:05 +0100
"Carlos E. R." <robin.listas@telefonica.net> wrote:
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully.
Exactly. My question is why? (Or WTF in modern parlance)
If it's up to me, stick to proper phrasing. Makes understanding others a bit less complicated IMNSHO.
OT. 1 Byte = 8 bits, a factor 8. Which can make a difference in speed experience.
Which repos are we talking about? The distro's repodata are > 100 MB AFAIK, but it's useless to refresh those repos, since they don't change after release.
I don't know. YaST in it's wisdom doesn't let me copy text that it displays, and it anyway elides the name of the currently downloading repository with dots. It starts with download.[open]suse and it finishes with primary.xml.something and there's a big hex number in between among possibly other things. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-06 20:50, Dave Howorth wrote:
On Tue, 6 Dec 2016 14:23:05 +0100 "Carlos E. R." <> wrote:
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully.
Exactly. My question is why? (Or WTF in modern parlance)
How exactly would you download the changes? Please define the method and data structures that you propose in detail. Hint: it is not that simple. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Tue, 6 Dec 2016 21:20:08 +0100 "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 2016-12-06 20:50, Dave Howorth wrote:
On Tue, 6 Dec 2016 14:23:05 +0100 "Carlos E. R." <> wrote:
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully.
Exactly. My question is why? (Or WTF in modern parlance)
How exactly would you download the changes?
Please define the method and data structures that you propose in detail.
Hint: it is not that simple.
I appear to have more experience than you in data compression in conjunction with large data transfers since I know it is possible and various possible techniques. But I'm more interested in establishing first what is the problem to be solved than proposing hypothetical solutions to hypothetical problems. So I'd welcome a constructive dialogue. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-12-06 22:12, Dave Howorth wrote:
On Tue, 6 Dec 2016 21:20:08 +0100 "Carlos E. R." <> wrote:
I appear to have more experience than you in data compression in conjunction with large data transfers since I know it is possible and various possible techniques. But I'm more interested in establishing first what is the problem to be solved than proposing hypothetical solutions to hypothetical problems.
It is already compressed, thus you can not download a "modified chunk", it is all or nothing. You can have a look yourself at the metadata files on the server and find out what could be done. The only current possibility is to publish diff files, like what the clam antivirus does. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Wed, 7 Dec 2016 03:39:11 +0100 "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 2016-12-06 22:12, Dave Howorth wrote:
On Tue, 6 Dec 2016 21:20:08 +0100 "Carlos E. R." <> wrote:
I appear to have more experience than you in data compression in conjunction with large data transfers since I know it is possible and various possible techniques. But I'm more interested in establishing first what is the problem to be solved than proposing hypothetical solutions to hypothetical problems.
It is already compressed, thus you can not download a "modified chunk", it is all or nothing. You can have a look yourself at the metadata files on the server and find out what could be done.
The only current possibility is to publish diff files, like what the clam antivirus does.
Is anybody prepared to actually help me understand how the system currently works and think about how to improve it? Know-it-all it's-not-possible answers are not really what I'm looking for. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
08.12.2016 00:54, Dave Howorth пишет:
Is anybody prepared to actually help me understand how the system currently works and think about how to improve it?
You can look at /var/cache/zypp/raw/<repo> to see what zyppoer downloads and whether you can optimize it. For repositories on download.opensuse.org metadata would be under .../repodata directory. But as zypper supports different formats the answer is probably to browse libzypp source code.
Know-it-all it's-not-possible answers are not really what I'm looking for.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06.12.2016 21:20, Carlos E. R. wrote:
On 2016-12-06 20:50, Dave Howorth wrote:
On Tue, 6 Dec 2016 14:23:05 +0100 "Carlos E. R." <> wrote:
On 2016-12-06 00:54, Dave Howorth wrote:
When I run software update, it says it is updating the repositories and this takes some time (seemingly a long time). Tonight I noticed that it claims to be downloading at over 600 k(something)/second. That seems to me to be sufficiently fast that the changes in the repositories since the last time I donloaded them (yesterday) should take almost no time.
It doesn't download the "changes", it downloads some big files, fully.
Exactly. My question is why? (Or WTF in modern parlance)
How exactly would you download the changes?
Please define the method and data structures that you propose in detail.
- sort the metadata file by modification date (newest entry at the beginning of the file) on the server - when downloading, the client can optionally indicate the date of the last retrival - the server will cut the file at the date point the client indicated
Hint: it is not that simple.
What am I missing? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Andrei Borzenkov -
Carlos E. R. -
Christoph Feck -
Dave Howorth -
Knurpht - Gertjan Lettink