[opensuse-buildservice] linking to packages built with the build service
Hello! Currently there is no way to link to a package built on the build service because if we do so this link will break on every rebuild since the release number will change. Because of that I recommmend to automatically create symbolic links without the version number as we have in the update directories on ftp.suse.com. For example: zypper.rpm -> zypper-0.6.15-0.1.i586.rpm If we do so we can just link to http://software.opensuse.com/download/.../packagename.rpm. This link will never break due to an automatic rebuild. Robert -- Robert Schiele Dipl.-Wirtsch.informatiker mailto:rschiele@gmail.com "Quidquid latine dictum sit, altum sonatur."
On 2007-03-02 15:14:43 +0100, Robert Schiele wrote:
Currently there is no way to link to a package built on the build service because if we do so this link will break on every rebuild since the release number will change.
Because of that I recommmend to automatically create symbolic links without the version number as we have in the update directories on ftp.suse.com. For example:
zypper.rpm -> zypper-0.6.15-0.1.i586.rpm
If we do so we can just link to http://software.opensuse.com/download/.../packagename.rpm. This link will never break due to an automatic rebuild.
but it i will make the redirector harder. we aim for a redirector that doesnt need local file access anymore but works with a sql db as backend. to check what file is currently behind the symlink we would need to read the link and than lookup the path to the real file in the DB. of course we could transfer that symlink -> file mapping into the DB aswell. but that would complicate the query. and for performance reasons i would like to keep the query count per request low (ideally 1, most likely it will be 2) Furthermore i am more in favor of using package managers for the installation step. rpm -i should be a very rare exception. so it would be better to point to some doc, that explains to integrate the repo into your favorite package manager. just some thoughts darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tue, Mar 06, 2007 at 04:17:07PM +0100, Marcus Rueckert wrote:
but it i will make the redirector harder. we aim for a redirector that doesnt need local file access anymore but works with a sql db as backend. to check what file is currently behind the symlink we would need to read the link and than lookup the path to the real file in the DB. of course we could transfer that symlink -> file mapping into the DB aswell. but that would complicate the query. and for performance reasons i would like to keep the query count per request low (ideally 1, most likely it will be 2)
I agree that performance of the redirector might be an issue to consider here. But not having persistent URLs in my opinion is a usability problem that outweighs these performance issues here. Actually I would prefer not having a redirector over not having persistent links.
Furthermore i am more in favor of using package managers for the installation step. rpm -i should be a very rare exception. so it would be better to point to some doc, that explains to integrate the repo into your favorite package manager.
This sounds more clean from the perspective of maintaining exactly one type of systems but it makes projects websites that provide packages for various platforms extremely confusing if for each and every platform a link to different documentation is provided. It is just more simple for projects websites and their users just to _link_ to binaries for various systems than explaining all types of package management. I think most projects that want to provide binary RPMs for their tool actually want to provide binary RPMs for their tool and not write documentation for all sorts of package managers traveling around. And linking to generic documentation confuses most users providing a huge load of the project's help list. Thus if you make things to complicated for this scenario the result will just be that many project administrators might just stop using the build service again because from their perspective it just becomes a maintainance hell. BTW: You should not underestimate the number of people using the rpm command line because it just saves you a lot of time of your life in many situations. Robert -- Robert Schiele Dipl.-Wirtsch.informatiker mailto:rschiele@gmail.com "Quidquid latine dictum sit, altum sonatur."
On Tue, 6 Mar 2007, Robert Schiele wrote:
BTW: You should not underestimate the number of people using the rpm command line because it just saves you a lot of time of your life in many situations.
Yup. Especially as "add a new installation source" means (openSUSE 10.2) - Calling yast - Enter password - Call source manager - about 3-10 minutes waiting for inst-source updates - Getting URL (konqueror, firefox) - Entering URL in inst-source (C&P) - Retyping URL, as we forget directory or were a directory to deep - Wait again, lots of clicks - Go to installation - about 3-10 minutes waiting again - search the tool - select the tool - install the tool - 3-5 minutes waiting for SuSE config Compared to - wget URL:xxx.rpm - su -c "rpm -i xxx.rpm" - Enter password I only do the above for tools I want to be kept updated all the time. Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2007-03-06 18:53:41 +0100, Dirk Stoecker wrote:
Especially as "add a new installation source" means (openSUSE 10.2) - Calling yast - Enter password - Call source manager - about 3-10 minutes waiting for inst-source updates - Getting URL (konqueror, firefox) - Entering URL in inst-source (C&P) - Retyping URL, as we forget directory or were a directory to deep - Wait again, lots of clicks - Go to installation - about 3-10 minutes waiting again - search the tool - select the tool - install the tool - 3-5 minutes waiting for SuSE config
Compared to - wget URL:xxx.rpm - su -c "rpm -i xxx.rpm" - Enter password
and now compare to: downlaod xxx.rpm su -c "rpm -i xxx.rpm" missing libfoo.so.2 what package contains libfoo.so.2? and for all the user faults above you cant blame the tools either. last but not least most suseconfig scripts are optimized to only run the heavy stuff when needed. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tue, Mar 06, 2007 at 08:57:44PM +0100, Marcus Rueckert wrote:
and now compare to: downlaod xxx.rpm su -c "rpm -i xxx.rpm"
missing libfoo.so.2 what package contains libfoo.so.2?
If you have a huge amount of complicated dependencies this might become a problem but in _most_ cases it is just as simple as installing one package on an average system. I think we shouldn't optimize the process for the complicated cases completely ignoring the average case.
and for all the user faults above you cant blame the tools either.
That's right. But on the other side it is also a fact that for most cases installing with the command line rpm tool is much faster than using a complex installer. You can't just argue this away.
last but not least most suseconfig scripts are optimized to only run the heavy stuff when needed.
Sure. But they are often sill annoying if you just want to install one really stupid RPM package. This whole discussion reminds me somehow to the discussion we did about drpmsync when I implemented my own version that no longer relies on existence of a drpmsync server (which was often down for a long time). I was always told about some corner cases where my implementation might have problems. Since I implemented that I use it with Eberhard's reliable server without having any big problem since then. Now you are doing the same again: Although you seem to fear that the redirector might not stand the load you make it the base of the decissions. For the long term this will again create a single point of failure. Robert -- Robert Schiele Dipl.-Wirtsch.informatiker mailto:rschiele@gmail.com "Quidquid latine dictum sit, altum sonatur."
On Tue, Mar 06, 2007 at 06:53:41PM +0100, Dirk Stoecker wrote:
On Tue, 6 Mar 2007, Robert Schiele wrote:
BTW: You should not underestimate the number of people using the rpm command line because it just saves you a lot of time of your life in many situations.
Yup.
Especially as "add a new installation source" means (openSUSE 10.2) - Calling yast - Enter password - Call source manager - about 3-10 minutes waiting for inst-source updates - Getting URL (konqueror, firefox) - Entering URL in inst-source (C&P) - Retyping URL, as we forget directory or were a directory to deep - Wait again, lots of clicks - Go to installation - about 3-10 minutes waiting again - search the tool - select the tool - install the tool - 3-5 minutes waiting for SuSE config
Interesting workflow! For me, it looks much easier: cd /etc/yum.repos.d w3m download.opensuse.org/repositories # download desired foo.repo file yum install foo I'm hardly using the commandline rpm anymore. Of course occasionally, but yum is fast enough...
Compared to - wget URL:xxx.rpm - su -c "rpm -i xxx.rpm" - Enter password
I only do the above for tools I want to be kept updated all the time.
Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Sep 1 2007 15:36, Dr. Peter Poeml wrote:
BTW: You should not underestimate the number of people using the rpm command line because it just saves you a lot of time of your life in many situations.
Yup.
Especially as "add a new installation source" means (openSUSE 10.2) - Calling yast - Enter password - Call source manager - about 3-10 minutes waiting for inst-source updates - Getting URL (konqueror, firefox) - Entering URL in inst-source (C&P) - Retyping URL, as we forget directory or were a directory to deep - Wait again, lots of clicks - Go to installation - about 3-10 minutes waiting again - search the tool - select the tool - install the tool - 3-5 minutes waiting for SuSE config
Interesting workflow!
..which where yast needs severe improvements. That is, for people not versed enough to use command-line alternatives.
For me, it looks much easier:
cd /etc/yum.repos.d w3m download.opensuse.org/repositories # download desired foo.repo file yum install foo
I'm hardly using the commandline rpm anymore. Of course occasionally, but yum is fast enough...
Jan -- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Sat, 1 Sep 2007, Dr. Peter Poeml wrote:
On Tue, Mar 06, 2007 at 06:53:41PM +0100, Dirk Stoecker wrote:
On Tue, 6 Mar 2007, Robert Schiele wrote:
You came back from a very long ride, did you? Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tue, Mar 06, 2007 at 04:17:07PM +0100, Marcus Rueckert wrote:
On 2007-03-02 15:14:43 +0100, Robert Schiele wrote:
Currently there is no way to link to a package built on the build service because if we do so this link will break on every rebuild since the release number will change.
Because of that I recommmend to automatically create symbolic links without the version number as we have in the update directories on ftp.suse.com. For example:
zypper.rpm -> zypper-0.6.15-0.1.i586.rpm
If we do so we can just link to http://software.opensuse.com/download/.../packagename.rpm. This link will never break due to an automatic rebuild.
but it i will make the redirector harder.
The redirector works fine with symlinks, as far as I can see. The scanner we use to update the redirector database simply ignores symlinks. The redirector canonicalizes every path before the database lookup. Thus, the database doesn't care about symlinks. We currenty have a lot of them. Most are in a "full-names-something" directory of some released products. Which may be interesting or not. Then, they are used in some convenience places like linking foo-current to something else, and the like. They could be replaced with Apache redirects, in some cases, I guess, but it would make them "invisible", which be against their purpose. Now, symlinks also start to occur now through the update/10.3/rpm tree, where we traditionally offer those "unversioned" symlinks, in exactly the way Robert proposes.
we aim for a redirector that doesnt need local file access anymore but works with a sql db as backend. to check what file is currently behind the symlink we would
But hoo... what a mess of work that would be :-) Veto. I neither aim, nor do I agree that we _should_ aim at such a setup, because I know that it'd be substantial amount of work. Here are my thoughts behind this: It would be required to come up with a database representing all needed data, and to maintain it, in a way that it stays pretty consistent with the backend file storage -- otherwise it'll only make things harder. A number of new tools will be needed to handle tasks which are now done by existing tools (like rsync). Of course, the whole matter _would_ be much easier if we would deal with a different kind of files, which doesn't change at such a fast pace, being so sensitive against small inconsistencies, at the same time. Like released distributions. But we have buildservice repositories and security updates, which keep on coming much faster than mirrors are fed. Running without file storage would btw prevent useful things like: * serving files which are on no mirror. Factory snapshots, for instance... * serving certain stuff (metadata) directly (thus, never redirecting) to prevent issues with inconsistent repositories and similar. * serving small files directly, because it is faster for us and for the clients than querying the database and doing redirection for them * debug and understand the system by someone else, in a reasonable time :-) I would compare the effort to reimplement the required functionality from scratch with, for instance, the Apache's mod_proxy* and mod_cache* rewrite. I watch this project since years, and it is interesting to see that it literally takes years to mature. HTTP may seem simple, but ...
need to read the link and than lookup the path to the real file in the DB. of course we could transfer that symlink -> file mapping into the DB aswell. but that would complicate the query. and for performance reasons i would like to keep the query count per request low (ideally 1, most likely it will be 2)
Performance-wise, I am pretty sure that everything which increases the complexity on the database end is prone to become a problem. I'm glad when we survive with what we have ;) Alas, when _I_ think about symlinks ;-) I have a totally different level of optimization in mind. I would rather like to get rid of them to avoid the necessity of the FollowSymlinks option, which requires Apache to stat all directories above the file. I _don't_ even want to think about multiple queries to the database, and things like that. We get by with a single query now (plus queries counting packages downloaded from the buildservice repositories). IMO, development time can be well spent in things like improved handling of failure conditions, and a better redirection scheme. There is still no way to maintain the mirror database than with a mysql commandline client. Some poor idiot has to live with that: me... We have a lot to do there. And: Synchronise repo pushes with rsync pull runs, so they don't stop on each other's toes. Implement fallback mirror redirection, mirror preference by network prefix, memcache query lookups for most freqently requested objects. Count redirects so we can analyze what's going on. Really, I don't see a major rewrite of the redirector and database backend feasible in the near future. High effort, low win. What would be the advantage, anyway, other than saving a few bucks for disks? I don't see it. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Saturday 01 September 2007 17:29:24 wrote Dr. Peter Poeml:
On Tue, Mar 06, 2007 at 04:17:07PM +0100, Marcus Rueckert wrote:
On 2007-03-02 15:14:43 +0100, Robert Schiele wrote:
Currently there is no way to link to a package built on the build service because if we do so this link will break on every rebuild since the release number will change.
Because of that I recommmend to automatically create symbolic links without the version number as we have in the update directories on ftp.suse.com. For example:
zypper.rpm -> zypper-0.6.15-0.1.i586.rpm
If we do so we can just link to http://software.opensuse.com/download/.../packagename.rpm. This link will never break due to an automatic rebuild.
but it i will make the redirector harder.
The redirector works fine with symlinks, as far as I can see.
yes, but symlinks would double the amount of files which needs to be handled. Given that we do always have a very high load with rsyncd on our servers this is really a bad idea. Also, the symlinks can be broken during update runs, it is imho not a good idea to introduce another place which might break consistency. Additionally, it is also a bad idea to point to a rpm directly. An End-User would have to handle any dependency issues manually (and even search for missing repositories manually). What we need is a static URL for installing packages what takes care about repositories and dependencies. We have that already, if one creates a pattern file in the build service. But this should work of course also without the need to create manually patterns for each package. We have the on-the-fly ymp generation for this, but this does not yet work independend of the version. A bugreport is open regarding this. To conclude, I do not think this is a redirector task, this is something which happen on level on top of it. And Peter described extensivly further problems with this ;) bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Mon, Sep 03, 2007 at 11:04:36AM +0200, Adrian Schröter wrote:
If we do so we can just link to http://software.opensuse.com/download/.../packagename.rpm. This link will never break due to an automatic rebuild.
but it i will make the redirector harder.
The redirector works fine with symlinks, as far as I can see.
yes, but symlinks would double the amount of files which needs to be handled. Given that we do always have a very high load with rsyncd on our servers this is really a bad idea. Also, the symlinks can be broken during update runs, it is imho not a good idea to introduce another place which might break consistency.
Hm. We provide symlinks ourselves, in the update trees. Are those obsolete? Or are they different?
Additionally, it is also a bad idea to point to a rpm directly. An End-User would have to handle any dependency issues manually (and even search for missing repositories manually).
BTW, what about pointing to a changelog? At the moment, I find http://download.opensuse.org/repositories/openSUSE:/Tools/SUSE_Linux_10.1/re... which unfortunately also depending on the build number, but nothing else which is publicly visible. Do we have anything better than this to see the changes?
What we need is a static URL for installing packages what takes care about repositories and dependencies. We have that already, if one creates a pattern file in the build service. But this should work of course also without the need to create manually patterns for each package. We have the on-the-fly ymp generation for this, but this does not yet work independend of the version. A bugreport is open regarding this.
Something like that sounds cool! If the symlinks are only for humans, and used primarily by them, they are maybe not accessed too often so they could be "simulated" dynamically, by some script, maybe...
To conclude, I do not think this is a redirector task, this is something which happen on level on top of it.
And Peter described extensivly further problems with this ;)
Do I misunderstand something here? I feel wrongly quoted. I elaborated on the impracticability of a diskless redirector, which darix mentioned as a possible reason against symlinks. I don't think I said anything against symlinks. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Mon, Sep 03, 2007 at 03:14:33PM +0200, Dr. Peter Poeml wrote:
BTW, what about pointing to a changelog? At the moment, I find http://download.opensuse.org/repositories/openSUSE:/Tools/SUSE_Linux_10.1/re... which unfortunately also depending on the build number, but nothing else which is publicly visible.
These links suck pretty much, by the way. They clutter up Google with dead links.
Do we have anything better than this to see the changes?
Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Monday 03 September 2007 15:14:33 wrote Dr. Peter Poeml:
On Mon, Sep 03, 2007 at 11:04:36AM +0200, Adrian Schröter wrote:
If we do so we can just link to http://software.opensuse.com/download/.../packagename.rpm. This link will never break due to an automatic rebuild.
but it i will make the redirector harder.
The redirector works fine with symlinks, as far as I can see.
yes, but symlinks would double the amount of files which needs to be handled. Given that we do always have a very high load with rsyncd on our servers this is really a bad idea. Also, the symlinks can be broken during update runs, it is imho not a good idea to introduce another place which might break consistency.
Hm. We provide symlinks ourselves, in the update trees. Are those obsolete? Or are they different?
These are different in that way that they came from the "old" autobuild system. Because we do still build the distribution internally.
Additionally, it is also a bad idea to point to a rpm directly. An End-User would have to handle any dependency issues manually (and even search for missing repositories manually).
BTW, what about pointing to a changelog? At the moment, I find http://download.opensuse.org/repositories/openSUSE:/Tools/SUSE_Linux_10.1/r epodata/repoview/osc-0-0.98-3.1.html which unfortunately also depending on the build number, but nothing else which is publicly visible.
The problem is that there can be multiple versions of one package in the repo ...
Do we have anything better than this to see the changes?
What we need is a static URL for installing packages what takes care about repositories and dependencies. We have that already, if one creates a pattern file in the build service. But this should work of course also without the need to create manually patterns for each package. We have the on-the-fly ymp generation for this, but this does not yet work independend of the version. A bugreport is open regarding this.
Something like that sounds cool!
If the symlinks are only for humans, and used primarily by them, they are maybe not accessed too often so they could be "simulated" dynamically, by some script, maybe...
yes
To conclude, I do not think this is a redirector task, this is something which happen on level on top of it.
And Peter described extensivly further problems with this ;)
Do I misunderstand something here? I feel wrongly quoted. I elaborated on the impracticability of a diskless redirector, which darix mentioned as a possible reason against symlinks. I don't think I said anything against symlinks.
Yes, but you said that it is quite some work to support this in the redirector and listed some problems. Sorry, for being not excat here bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (6)
-
Adrian Schröter -
Dirk Stoecker -
Dr. Peter Poeml -
Jan Engelhardt -
Marcus Rueckert -
Robert Schiele