[opensuse-factory] RFC: forbid _service files
Hi, After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory. So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed. What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory. This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection. Please comment or I will go forward. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
2011/5/3 Stephan Kulow <coolo@novell.com>:
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
As a happy _service files user I would object... unless I would know more about "a series of broken packages". Sure, I had a problem recently. But it was a lot worse before, things have been improving a lot for me. So, why you see it as failed?
- instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
What do you mean with "verify"? If you mean verify the URL is not malformed, I don't think that would be really useful. If you mean use wget/curl on it, and check the submitted and downloaded tarballs match... well, that's what _service files do now. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 3. Mai 2011 schrieb Cristian Morales Vega:
2011/5/3 Stephan Kulow <coolo@novell.com>:
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
As a happy _service files user I would object... unless I would know more about "a series of broken packages". See Vincent's mail this morning - this was not even the last one.
Sure, I had a problem recently. But it was a lot worse before, things have been improving a lot for me. What packages in factory do you have with _service files and what do they use?
So, why you see it as failed?
- instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
What do you mean with "verify"? If you mean verify the URL is not malformed, I don't think that would be really useful. If you mean use wget/curl on it, and check the submitted and downloaded tarballs match... well, that's what _service files do now.
I mean to wget/curl them and check that it's the same as in the package and not something unrelated, but the source tar would stay in the package and wouldn't be regenerated/recommited on every source change. People clearly expect a osc commit after a working local build to also work on the server. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
2011/5/3 Stephan Kulow <coolo@novell.com>:
Am Dienstag, 3. Mai 2011 schrieb Cristian Morales Vega:
2011/5/3 Stephan Kulow <coolo@novell.com>:
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
As a happy _service files user I would object... unless I would know more about "a series of broken packages". See Vincent's mail this morning - this was not even the last one.
It looks like a minor problem, either it's a big download or you must be really fast to create a SR before the service has completed. And it should be simple enough to solve. In fact it's already solved in the client side. I don't look every day at Factory build errors, you know better than me the number and severity of the problems. But from my corner it seems right now _service files create nearly zero problems, I could be wrong...
Sure, I had a problem recently. But it was a lot worse before, things have been improving a lot for me. What packages in factory do you have with _service files and what do they use?
Which I consider mine: libgme, libebml, libmatroska and mkvtoolnix. All of them use download_url, verify_file and set_version. I don't remember having a problem with them ever. They are not the packages that could benefit the most from services, but it still makes life easier when someone submit requests me an upstream update to the devel project.
- instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
What do you mean with "verify"? If you mean verify the URL is not malformed, I don't think that would be really useful. If you mean use wget/curl on it, and check the submitted and downloaded tarballs match... well, that's what _service files do now. I mean to wget/curl them and check that it's the same as in the package and not something unrelated, but the source tar would stay in the package and wouldn't be regenerated/recommited on every source change. People clearly expect a osc commit after a working local build to also work on the server.
And this isn't exactly what download_url and verify_file are doing now? You want to change the current services for a different implementation that does exactly the same?? I will expect a new implementation to be buggier. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 3. Mai 2011, 19:24:35 schrieb Cristian Morales Vega:
2011/5/3 Stephan Kulow <coolo@novell.com>:
Am Dienstag, 3. Mai 2011 schrieb Cristian Morales Vega:
2011/5/3 Stephan Kulow <coolo@novell.com>:
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
As a happy _service files user I would object... unless I would know more about "a series of broken packages". See Vincent's mail this morning - this was not even the last one.
It looks like a minor problem, either it's a big download or you must be really fast to create a SR before the service has completed. And it should be simple enough to solve. In fact it's already solved in the client side. I don't look every day at Factory build errors, you know better than me the number and severity of the problems. But from my corner it seems right now _service files create nearly zero problems, I could be wrong...
I still want to solve open problems at least ;) However, the new way coolo suggested can be implemented via project wide source service. The download_url service can be replaced by a project wide download_files service. It would download the files according to spec file URL. combining this with new osc and "trylocal" option it would offer both ways, submitting the downloaded tar ball together with the submission and let the server just validated. Or trigger it to server side download when bandwidth problems exist for packagers with packages with large tar balls. However, I still see the need to allow to run tar_scm. At least manually created tar balls would be a step backward IMHO.
Sure, I had a problem recently. But it was a lot worse before, things have been improving a lot for me. What packages in factory do you have with _service files and what do they use?
Which I consider mine: libgme, libebml, libmatroska and mkvtoolnix. All of them use download_url, verify_file and set_version. I don't remember having a problem with them ever. They are not the packages that could benefit the most from services, but it still makes life easier when someone submit requests me an upstream update to the devel project.
- instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
What do you mean with "verify"? If you mean verify the URL is not malformed, I don't think that would be really useful. If you mean use wget/curl on it, and check the submitted and downloaded tarballs match... well, that's what _service files do now. I mean to wget/curl them and check that it's the same as in the package and not something unrelated, but the source tar would stay in the package and wouldn't be regenerated/recommited on every source change. People clearly expect a osc commit after a working local build to also work on the server.
And this isn't exactly what download_url and verify_file are doing now? You want to change the current services for a different implementation that does exactly the same?? I will expect a new implementation to be buggier.
it would be also a source service implementation but using the project wide services. I planned to write something more informative about this, so far there is only our book chapter: http://doc.opensuse.org/products/draft/OBS/obs-reference-guide_draft/cha.obs... -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 3. Mai 2011 schrieb Cristian Morales Vega:
And this isn't exactly what download_url and verify_file are doing now? You want to change the current services for a different implementation that does exactly the same?? I will expect a new implementation to be buggier.
The services are very similiar indeed, the only difference is that one actually downloads and commits to duplicate the information in the Url in a _service file and the other service only downloads and verifies without commit. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Tue, May 03, 2011 at 04:12:32PM +0200, Stephan Kulow wrote:
Hi,
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
Please comment or I will go forward.
Makes sense to me, please go forward with this. greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
* Stephan Kulow <coolo@novell.com> [2011-05-03 16:12]:
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
Please comment or I will go forward.
For all source files? How does that deal with snapshots from scm and custom local source files? What is the actual benefit (apart from knowing there is a file of the same name somewhere on the net)? -- Guido Berhoerster -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 3. Mai 2011 schrieb Guido Berhoerster:
* Stephan Kulow <coolo@novell.com> [2011-05-03 16:12]:
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
Please comment or I will go forward.
For all source files? How does that deal with snapshots from scm and custom local source files? What is the actual benefit (apart If there is no URL as Source, it's of course not possible to validate.
from knowing there is a file of the same name somewhere on the net)? We would validate more than the name - and that would make sure you can't pretend you upload linus kernel but instead upload something fishy.
Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 05/03/2011 04:12 PM, Stephan Kulow wrote:
Hi,
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
Please comment or I will go forward.
I don't have a problem with that. But only if there is a way to convert from _service to non-_service at the Factory gate. Or is it what will be done when "ban" is forced? The packages are much easier to update when _service file is used. thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 3. Mai 2011 schrieb Jiri Slaby:
I don't have a problem with that. But only if there is a way to convert from _service to non-_service at the Factory gate. Or is it what will be done when "ban" is forced?
The packages are much easier to update when _service file is used.
Why is it easier? Perhaps I'm seeing the use case differently. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 05/04/2011 09:31 AM, Stephan Kulow wrote:
Am Dienstag, 3. Mai 2011 schrieb Jiri Slaby:
I don't have a problem with that. But only if there is a way to convert from _service to non-_service at the Factory gate. Or is it what will be done when "ban" is forced?
The packages are much easier to update when _service file is used.
Why is it easier? Perhaps I'm seeing the use case differently.
Because one needs to change a single line to update a package from one version to another. Maybe BS can handle URLs from SOURCE line (I'm not sure about this) which would offer the same functionality. However it for example doesn't recompress the tar from big gz's to much smaller bz2's. And there are still people with low bandwidth like me where "osc co" size matters (e.g. llvm is packaged only by gzip -- llvm-gcc .gz ~ 52M, .bz2 ~ 39M). tar_scm is another time-saver. There are perpetual-beta projects with no releases at all (psi+ comes on my mind). The development occurs in a repository and one doesn't need to do snapshots manually or find time stamps of the last prepared snapshot. One "osc service remoterun" does that all. Really, isn't the obs/osc fixable to handle the _service files properly? I cannot tell, since I don't know what exact problems there are in Factory. I'm not fighting them every day. You do. thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 05/04/2011 09:31 AM, Stephan Kulow wrote:
Am Dienstag, 3. Mai 2011 schrieb Jiri Slaby:
I don't have a problem with that. But only if there is a way to convert from _service to non-_service at the Factory gate. Or is it what will be done when "ban" is forced?
The packages are much easier to update when _service file is used.
Why is it easier? Perhaps I'm seeing the use case differently.
Because one needs to change a single line to update a package from one version to another. Maybe BS can handle URLs from SOURCE line (I'm not sure about this) which would offer the same functionality. However it for example doesn't recompress the tar from big gz's to much smaller bz2's. And there are still people with low bandwidth like me where "osc co" size matters (e.g. llvm is packaged only by gzip -- llvm-gcc .gz ~ 52M, .bz2 ~ 39M). With _service files you download the original tar ball too. Very often even more often than once.
tar_scm is another time-saver. There are perpetual-beta projects with no releases at all (psi+ comes on my mind). The development occurs in a repository and one doesn't need to do snapshots manually or find time stamps of the last prepared snapshot. One "osc service remoterun" does that all. So what you're saying is that you want to update huge sources in the OBS without having them locally. All I want is basically that those sources are
Am Mittwoch, 4. Mai 2011 schrieb Jiri Slaby: then commited as fixed revision into the sourcetree without the service running at every source change, but only if you use e.g. remoterun. Perhaps we should not forbid _service files in general, but stop regenerating.
Really, isn't the obs/osc fixable to handle the _service files properly? I cannot tell, since I don't know what exact problems there are in Factory. I'm not fighting them every day. You do.
Well, I don't "fight" them either - but I reject many packages and people don't understand why. And the reason they don't understand it is that source services behave in unexepcted ways. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Wed, 4 May 2011, Stephan Kulow wrote:
On 05/04/2011 09:31 AM, Stephan Kulow wrote:
Am Dienstag, 3. Mai 2011 schrieb Jiri Slaby:
I don't have a problem with that. But only if there is a way to convert from _service to non-_service at the Factory gate. Or is it what will be done when "ban" is forced?
The packages are much easier to update when _service file is used.
Why is it easier? Perhaps I'm seeing the use case differently.
Because one needs to change a single line to update a package from one version to another. Maybe BS can handle URLs from SOURCE line (I'm not sure about this) which would offer the same functionality. However it for example doesn't recompress the tar from big gz's to much smaller bz2's. And there are still people with low bandwidth like me where "osc co" size matters (e.g. llvm is packaged only by gzip -- llvm-gcc .gz ~ 52M, .bz2 ~ 39M). With _service files you download the original tar ball too. Very often even more often than once.
tar_scm is another time-saver. There are perpetual-beta projects with no releases at all (psi+ comes on my mind). The development occurs in a repository and one doesn't need to do snapshots manually or find time stamps of the last prepared snapshot. One "osc service remoterun" does that all. So what you're saying is that you want to update huge sources in the OBS without having them locally. All I want is basically that those sources are
Am Mittwoch, 4. Mai 2011 schrieb Jiri Slaby: then commited as fixed revision into the sourcetree without the service running at every source change, but only if you use e.g. remoterun.
Perhaps we should not forbid _service files in general, but stop regenerating.
I'm not sure what the actual problem is but the time the service is processed bothered me as well. In some discussion with Adrian I suggested that it has to occur exactly once, atomically with the osc commit (so that when it fails the commit fails). I'm not sure where we are currently re-generating stuff. Richard. -- Richard Guenther <rguenther@suse.de> Novell / SUSE Labs SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer
Am Mittwoch, 4. Mai 2011, 10:56:20 schrieb Richard Guenther:
On Wed, 4 May 2011, Stephan Kulow wrote:
On 05/04/2011 09:31 AM, Stephan Kulow wrote:
Am Dienstag, 3. Mai 2011 schrieb Jiri Slaby:
I don't have a problem with that. But only if there is a way to convert from _service to non-_service at the Factory gate. Or is it what will be done when "ban" is forced?
The packages are much easier to update when _service file is used.
Why is it easier? Perhaps I'm seeing the use case differently.
Because one needs to change a single line to update a package from one version to another. Maybe BS can handle URLs from SOURCE line (I'm not sure about this) which would offer the same functionality. However it for example doesn't recompress the tar from big gz's to much smaller bz2's. And there are still people with low bandwidth like me where "osc co" size matters (e.g. llvm is packaged only by gzip -- llvm-gcc .gz ~ 52M, .bz2 ~ 39M). With _service files you download the original tar ball too. Very often even more often than once.
tar_scm is another time-saver. There are perpetual-beta projects with no releases at all (psi+ comes on my mind). The development occurs in a repository and one doesn't need to do snapshots manually or find time stamps of the last prepared snapshot. One "osc service remoterun" does that all. So what you're saying is that you want to update huge sources in the OBS without having them locally. All I want is basically that those sources are
Am Mittwoch, 4. Mai 2011 schrieb Jiri Slaby: then commited as fixed revision into the sourcetree without the service running at every source change, but only if you use e.g. remoterun.
Perhaps we should not forbid _service files in general, but stop regenerating.
I'm not sure what the actual problem is but the time the service is processed bothered me as well. In some discussion with Adrian I suggested that it has to occur exactly once, atomically with the osc commit (so that when it fails the commit fails). I'm not sure where we are currently re-generating stuff.
Yes, we want to reach this ideal state that the changes, like spec file formating happens already before committing. However, we do still want to enforce on the server side that it is really applied and that tar balls are really from the defined location. If that is the case no further commits happens. If not, another commit happens to show the difference between the change of the user and the server. The osc from openSUSE:Tools:Unstable is meanwhile waiting for a server side source service run after commit and updates your local copy to merge such changes back. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Wed, 4 May 2011 10:20:22 +0200 Stephan Kulow <coolo@novell.com> wrote:
With _service files you download the original tar ball too. Very often even more often than once.
for many people, download bandwidth is a less scarce resource than upload bandwidth
So what you're saying is that you want to update huge sources in the OBS without having them locally.
No, without having to first download them and then upload them into OBS when OBS can just download them from the original source. -- Stefan Seyfried "Dispatch war rocket Ajax to bring back his body!" -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch, 4. Mai 2011, 10:59:41 schrieb Stefan Seyfried:
On Wed, 4 May 2011 10:20:22 +0200 Stephan Kulow <coolo@novell.com> wrote:
With _service files you download the original tar ball too. Very often even more often than once.
That is only true when you build packages, not on initial checkout.
for many people, download bandwidth is a less scarce resource than upload bandwidth
right.
So what you're saying is that you want to update huge sources in the OBS without having them locally.
No, without having to first download them and then upload them into OBS when OBS can just download them from the original source.
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch, 4. Mai 2011, 11:20:40 schrieb Adrian Schröter:
Am Mittwoch, 4. Mai 2011, 10:59:41 schrieb Stefan Seyfried:
On Wed, 4 May 2011 10:20:22 +0200 Stephan Kulow <coolo@novell.com> wrote:
With _service files you download the original tar ball too. Very often even more often than once.
That is only true when you build packages, not on initial checkout.
for many people, download bandwidth is a less scarce resource than upload bandwidth
right.
The different ways how to deal with sources also in respect of bandwidth are documented here: http://doc.opensuse.org/products/draft/OBS/obs-best-practices_draft/cha.obs.... My personal recommendation as Factory policy would be: * make a global "download_files" with "mode=trylocal" the official openSUSE:Factory default policy: => with current osc versions the downloaded files gets committed and checked out as usual files. But they are validate on the server side when a URL as Source: is mentioned. => Make usage of Source: $URL/$file in spec files our default policy for Factory => people may able to skip "trylocal" in exceptional cases => _service: prefixed files will get generated on server side. but easily merged back as standard files later. * Still allow "download_url" source services in some cases, eg. packages with tar balls > 100MB when the files should not be checked out by default to allow fast fixing packages even with lower bandwidth. * Still allow "tar_scm" source services when an upstream project is not publishing tar balls and just offers svn/git repositories for their releases. I wanted to propose this in a more lengthy way, but I did not find the time yet :/ -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 5/3/2011 10:12 AM, Stephan Kulow wrote:
Hi,
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
Please comment or I will go forward.
Greetings, Stephan
Does everything actually have a valid url that goes to an archive? Some things just have web sites where the download link is a cgi, sometimes a poorly written cgi, and no direct simple link to the actual archive is offered, and trying to wget/curl the url doesn't work or doesn't work as desired. I actually have a couple of small packages of my own that I do not host anywhere else but in the build service. The build service essentially _is_ the most upstream root source for this code. Then there are git/hg/svn snapshots Then there are packages whose real original host/maintainer have long since gone away and all there are now are many places that maintain it, but none can be considered the new authoritative upstream maintainer. -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Dienstag, 3. Mai 2011 schrieb Brian K. White:
Does everything actually have a valid url that goes to an archive? Some things just have web sites where the download link is a cgi, sometimes a poorly written cgi, and no direct simple link to the actual archive is offered, and trying to wget/curl the url doesn't work or doesn't work as desired. Those don't use _service files atm either. So there is no change.
Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 5/4/2011 4:13 AM, Stephan Kulow wrote:
Am Dienstag, 3. Mai 2011 schrieb Brian K. White:
Does everything actually have a valid url that goes to an archive? Some things just have web sites where the download link is a cgi, sometimes a poorly written cgi, and no direct simple link to the actual archive is offered, and trying to wget/curl the url doesn't work or doesn't work as desired. Those don't use _service files atm either. So there is no change.
Greetings, Stephan
The point was that you suggested making the source url comment important instead of just a comment. I put whatever I can in the source url comment (in spec files) that will point the way to how to get the upstream source, and there is always _something_ I can put there, but it is not always something that can be automated, or that should be considered any more authoritative than the locally hosted copy, for instance, the debian or redhat version of a package that no longer has any official upstream source, or a known broken link that was the last known location and that's better than forgetting the info, or a wayback machine link, etc. Those and other possible things that might appear in the source url comment are not usable for automated download or verify, and sometimes there is no better thing to put there. So making that comment important is not sensible to me, at least not without making it voluntary/optional, where it's only invoked by putting a special string in there, or less-good, disabled by putting a special string in there. If that is not what you meant, or you meant to allow for this somehow, then never mind. -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch, 4. Mai 2011 schrieb Brian K. White:
On 5/4/2011 4:13 AM, Stephan Kulow wrote:
Am Dienstag, 3. Mai 2011 schrieb Brian K. White:
Does everything actually have a valid url that goes to an archive? Some things just have web sites where the download link is a cgi, sometimes a poorly written cgi, and no direct simple link to the actual archive is offered, and trying to wget/curl the url doesn't work or doesn't work as desired.
Those don't use _service files atm either. So there is no change.
Greetings, Stephan
The point was that you suggested making the source url comment important instead of just a comment.
I put whatever I can in the source url comment (in spec files) that will point the way to how to get the upstream source, and there is always _something_ I can put there, but it is not always something that can be automated, or that should be considered any more authoritative than the locally hosted copy, for instance, the debian or redhat version of a package that no longer has any official upstream source, or a known broken link that was the last known location and that's better than forgetting the info, or a wayback machine link, etc.
Those and other possible things that might appear in the source url comment are not usable for automated download or verify, and sometimes there is no better thing to put there. So making that comment important is not sensible to me, at least not without making it voluntary/optional, where it's only invoked by putting a special string in there, or less-good, disabled by putting a special string in there.
It has to be correct at the moment when you submit it. If it can't be correct for reasons given by you, then don't put them in the Source: url but in a comment. If it turns broken at some later point, it's of course out of your control, but that doesn't mean you should write bogus URIs in spec files just because you can. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Stephan Kulow wrote:
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
+1 for considering the current approach failed. +1 for not recompressing sources. However, I still think storing tarballs in some kind of source caching service rather than alongside the spec file would be beneficial. Not only for verification purpose but also to aid people with a slow connection. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 3.5.2011 16:12, Stephan Kulow wrote:
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
So I will only need to write Source: http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.38.tar.bz2 in the kernel spec file and it will automatically download the tarball it on the server, without me uploading it? Without the ugly _service:download_url:linux-2.6.38.tar.bz2 filename? That would be great! Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
-------- Original-Nachricht --------
Datum: Wed, 04 May 2011 14:27:33 +0200 Von: Michal Marek <mmarek@suse.cz> An: opensuse-factory@opensuse.org Betreff: Re: [opensuse-factory] RFC: forbid _service files
On 3.5.2011 16:12, Stephan Kulow wrote:
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
So I will only need to write Source: http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.38.tar.bz2 in the kernel spec file and it will automatically download the tarball it on the server, without me uploading it? Without the ugly _service:download_url:linux-2.6.38.tar.bz2 filename? That would be great!
+1 for that! (Ok different $SOURCENAME) ;-) Would ease working from home with that tiny upload bandwidth. Detlef
Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
-- NEU: FreePhone - kostenlos mobil telefonieren und surfen! Jetzt informieren: http://www.gmx.net/de/go/freephone -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch, 4. Mai 2011, 14:27:33 schrieb Michal Marek:
On 3.5.2011 16:12, Stephan Kulow wrote:
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
So I will only need to write Source: http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.38.tar.bz2 in the kernel spec file and it will automatically download the tarball it on the server, without me uploading it? Without the ugly _service:download_url:linux-2.6.38.tar.bz2 filename? That would be great!
No. It is enough to specify this url in spec file with "download_files", yes. But it will get downloaded, committed and server side redownloaded (to verify) when using the "mode=trylocal" approach. If the server decides that it does not match (maybe because you skipped to commit it together), it will store it beside with the prefix. In theory we could also add a mode which overrites your file, but I think this may cause more problems and confusions.
Michal
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 4.5.2011 15:38, Adrian Schröter wrote:
Am Mittwoch, 4. Mai 2011, 14:27:33 schrieb Michal Marek:
So I will only need to write Source: http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.38.tar.bz2 in the kernel spec file and it will automatically download the tarball it on the server, without me uploading it? Without the ugly _service:download_url:linux-2.6.38.tar.bz2 filename? That would be great!
No.
It is enough to specify this url in spec file with "download_files", yes.
But it will get downloaded, committed and server side redownloaded (to verify) when using the "mode=trylocal" approach.
"committed" means uploaded by osc to the build service? Could this be changed so that only the spec file and patches are uploaded, and the tarball is downloaded by the build service? Thanks Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Am Mittwoch, 4. Mai 2011, 15:58:20 schrieb Michal Marek:
On 4.5.2011 15:38, Adrian Schröter wrote:
Am Mittwoch, 4. Mai 2011, 14:27:33 schrieb Michal Marek:
So I will only need to write Source: http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.38.tar.bz2 in the kernel spec file and it will automatically download the tarball it on the server, without me uploading it? Without the ugly _service:download_url:linux-2.6.38.tar.bz2 filename? That would be great!
No.
It is enough to specify this url in spec file with "download_files", yes.
But it will get downloaded, committed and server side redownloaded (to verify) when using the "mode=trylocal" approach.
"committed" means uploaded by osc to the build service?
yes
Could this be changed so that only the spec file and patches are uploaded, and the tarball is downloaded by the build service?
As written in my mail, we follow the holy rule not to overwrite files from the user with files from a server side service run. This could be changed, but that will lead to all kind of confusions and clashes. The current approach makes absolute clear which files are submitted by the user and which are from the system. So, you don't need to review the tar ball for example, when you trust the upstream project and you verified that the url is correct. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 4.5.2011 16:51, Adrian Schröter wrote:
Am Mittwoch, 4. Mai 2011, 15:58:20 schrieb Michal Marek:
On 4.5.2011 15:38, Adrian Schröter wrote:
Am Mittwoch, 4. Mai 2011, 14:27:33 schrieb Michal Marek:
So I will only need to write Source: http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.38.tar.bz2 in the kernel spec file and it will automatically download the tarball it on the server, without me uploading it? Without the ugly _service:download_url:linux-2.6.38.tar.bz2 filename? That would be great!
No.
It is enough to specify this url in spec file with "download_files", yes.
But it will get downloaded, committed and server side redownloaded (to verify) when using the "mode=trylocal" approach.
"committed" means uploaded by osc to the build service?
yes
Could this be changed so that only the spec file and patches are uploaded, and the tarball is downloaded by the build service?
As written in my mail, we follow the holy rule not to overwrite files from the user with files from a server side service run.
This could be changed, but that will lead to all kind of confusions and clashes.
I understand this, but I my use case, I never want to upload the tarball to the buildservice, so there would be no conflict. The server would simply download the missing file. Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Le mercredi 04 mai 2011, à 16:59 +0200, Michal Marek a écrit :
I understand this, but I my use case, I never want to upload the tarball to the buildservice, so there would be no conflict. The server would simply download the missing file.
If we want to support this, then I think using something like the verify_file (which checks the checksum of a file) service would be needed: it's important that what the build service downloads matches what you expected it to download. Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On 4.5.2011 17:13, Vincent Untz wrote:
Le mercredi 04 mai 2011, à 16:59 +0200, Michal Marek a écrit :
I understand this, but I my use case, I never want to upload the tarball to the buildservice, so there would be no conflict. The server would simply download the missing file.
If we want to support this, then I think using something like the verify_file (which checks the checksum of a file) service would be needed: it's important that what the build service downloads matches what you expected it to download.
Yeah, why not. I could either upload the kernel.org provided signature or some hash. Michal -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Le mercredi 04 mai 2011, à 16:59 +0200, Michal Marek a écrit :
I understand this, but I my use case, I never want to upload the tarball to the buildservice, so there would be no conflict. The server would simply download the missing file.
If we want to support this, then I think using something like the verify_file (which checks the checksum of a file) service would be needed: it's important that what the build service downloads matches what you expected it to download. I talked with Michael about this about a month ago, this could be moved into
On Wednesday 04 May 2011 17:13:35 Vincent Untz wrote: the spec file too as an annotation. While I forget the exact 'annotation' syntax, it could look like this: #Source0-MD5: 1234567890 #Source0-SHA1: 0987654321 Source0: http://example.com/foo-0.1.tar.gz -- Mit freundlichen Grüßen, Sascha Peilicke http://saschpe.wordpress.com
Am Donnerstag, 5. Mai 2011, 10:30:20 schrieb Sascha Peilicke:
Le mercredi 04 mai 2011, à 16:59 +0200, Michal Marek a écrit :
I understand this, but I my use case, I never want to upload the tarball to the buildservice, so there would be no conflict. The server would simply download the missing file.
If we want to support this, then I think using something like the verify_file (which checks the checksum of a file) service would be needed: it's important that what the build service downloads matches what you expected it to download. I talked with Michael about this about a month ago, this could be moved into
On Wednesday 04 May 2011 17:13:35 Vincent Untz wrote: the spec file too as an annotation. While I forget the exact 'annotation' syntax, it could look like this:
#Source0-MD5: 1234567890 #Source0-SHA1: 0987654321 Source0: http://example.com/foo-0.1.tar.gz
Yes, we could support this via a service as well, but where is the sense if we track (at least md5) in our history anyway already ? It would be more interessting to maintain a defined set of gpg keys, where we have verified that they are from upstream projects and use them for validation (if the project is signing their tar balls at all). -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Tue, May 03, 2011 at 04:12:32PM +0200, Stephan Kulow wrote:
Hi,
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
+1 for not recompressing - in adtion please remove the ugly source is not bzipped - I'd say having the **same** source file is more important, than saving few bytes.
Please comment or I will go forward.
+1 from me BTW: I consider the separate source caching server idea from Ludwig as excellent. In that case we can reuse the similar scheme with _sources file, like Fedora has. Uploading can be done via web as a fallback method for those, who had a limited upload bandwitch. Regards Michal Vyskocil
On 5/5/2011 10:13 AM, Michal Vyskocil wrote:
On Tue, May 03, 2011 at 04:12:32PM +0200, Stephan Kulow wrote:
Hi,
After a series of broken packages because of _service files I would like to forbid them in openSUSE:Factory.
So far I've seen the use of _service files in factory as an experiment (I happily joined), but I see this experiment as failed.
What I would like to propose is: - ban recompress, download and tarscm as _services (everything else should have been banned before) - instead we verify the Source urls and make packages that have invalid URLs "broken", so you can't SR them to factory.
This means: the Source URL is now an important thing to change and no longer a bit rotting comment and we stop recompressing tars - this was a proposal from Adrian a while ago without much objection.
+1 for not recompressing - in adtion please remove the ugly source is not bzipped - I'd say having the **same** source file is more important, than saving few bytes.
I also highly disagree with the idea of re-packaging source archives in other formats. It's sort of tempting to think it's safe and doesn't count as modifying the data. But for example there are some sources that come from upstream as .zip from primarily Windows or Mac developers and have all manner of file/directory name oddness. I have seen some of these re-packaged into more normal looking archives, not only changing from .zip to .tar.gz, but changing the directory structure, removing capitalization, changing the root directory name to what the %setup macro expects, even removing dos line-endings from text files (instead of doing it via commands in the spec). This results in: * The rest of the spec defines how to unpack and build that specially crafted archive, NOT the original. The info about the original is lost. * There is no way to know that nothing _else_ was changed to suit the repackagers aesthetics. * It requires the same manual, work to be done over again the same way at the next update. * Some of the steps needed to create the package and build the source are not documented within the spec or patches. They were done manually by someone and then the spec just starts off expecting this to already have been done. This all made me decide at least for myself that there really is no valid place to draw the line as to what is ok to change and what should not be changed. The simplest, and the only trustworthy approach is just never change anything. Do all such work within the spec and/or via patches. Even an uncompressed tar is ok, since the encapsulating cpio in the src rpm is compressed anyways. -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
participants (15)
-
Adrian Schröter -
Brian K. White -
Cristian Morales Vega -
Detlef Steuer -
Greg KH -
Guido Berhoerster -
Jiri Slaby -
Ludwig Nussel -
Michal Marek -
Michal Vyskocil -
Richard Guenther -
Sascha Peilicke -
Stefan Seyfried -
Stephan Kulow -
Vincent Untz