[opensuse-buildservice] Re: [PATCH] [webui] more sophisticated check if a file is available on the mirror
The repserver has already some code to find out if a repo is published already or not. We could extend that and tell you already, if a package has been published or not. So we could avoid this additional traffic and IMHO also possible area of bugs. What do you think about that ? bye adrian Am Mittwoch, 14. April 2010 14:26:15 schrieben Sie:
From: Thomas Schmidt
--- src/webui/app/controllers/package_controller.rb | 35 ++++++++++----------- src/webui/app/views/project/repositories.html.erb | 2 +- 2 files changed, 18 insertions(+), 19 deletions(-)
diff --git a/src/webui/app/controllers/package_controller.rb b/src/webui/app/controllers/package_controller.rb index 105885f..54a3452 100644 --- a/src/webui/app/controllers/package_controller.rb +++ b/src/webui/app/controllers/package_controller.rb @@ -33,7 +33,6 @@ class PackageController < ApplicationController elsif @project.bugowner @bugowner_mail = Person.find_cached( @project.bugowner ).email.to_s end - fill_status_cache end
@@ -44,7 +43,7 @@ class PackageController < ApplicationController @dproject = params[:dproject] @filename = params[:filename] @fileinfo = Fileinfo.find_cached( :project => params[:dproject], :package => '_repository', :repository => params[:drepository], :arch => @arch, - :filename => params[:dname], :view => 'fileinfo_ext') + :filename => params[:dname], :view => 'fileinfo_ext') @durl = nil end
@@ -53,28 +52,15 @@ class PackageController < ApplicationController @repository = params[:repository] @filename = params[:filename] @fileinfo = Fileinfo.find_cached( :project => @project, :package => @package, :repository => @repository, :arch => @arch, - :filename => @filename, :view => 'fileinfo_ext') - puts @fileinfo + :filename => @filename, :view => 'fileinfo_ext') @durl = repo_url( @project, @repository ) + "/#{@fileinfo.arch}/#{@filename}" - puts @durl - if @durl - uri = URI.parse( @durl ) - response = nil - Net::HTTP.start(uri.host, uri.port) do |http| - response = http.head uri.path - end - @durl = nil if response.code.to_i >= 400 - else - logger.debug "no repository url" - end - if @user and not @durl - # use API for logged in users if the mirror is not available + if !file_available?( @durl ) and @user + # only use API for logged in users if the mirror is not available @durl = rpm_url( @project, @package, @repository, @arch, @filename ) end end
def binaries - @repository = params[:repository] @buildresult = Buildresult.find( :project => @project, :package => @package, :repository => @repository, :view => ['binarylist', 'status'] ) @@ -835,6 +821,19 @@ class PackageController < ApplicationController
private
+ def file_available? url, max_redirects=5 + uri = URI.parse( url ) + Net::HTTP.start(uri.host, uri.port) do |http| + logger.debug "Checking url: #{url}" + response = http.head uri.path + if response.code.to_i == 302 and response['location'] and max_redirects > 0 + return file_available? response['location'], (max_redirects - 1) + end + response.code.to_i == 200 ? true : false + end + end + + def get_files( project, package ) # files whose name ends in the following extensions should not be editable no_edit_ext = %w{ .bz2 .dll .exe .gem .gif .gz .jar .jpeg .jpg .lzma .ogg .pdf .pk3 .png .ps .rpm .svgz .tar .taz .tb2 .tbz .tbz2 .tgz .tlz .txz .xpm .xz .z .zip } diff --git a/src/webui/app/views/project/repositories.html.erb b/src/webui/app/views/project/repositories.html.erb index 615c395..00dbb55 100644 --- a/src/webui/app/views/project/repositories.html.erb +++ b/src/webui/app/views/project/repositories.html.erb @@ -1,7 +1,7 @@ <% @crumb_list = [ link_to( 'Projects', :action => :list_public), - link_to( @project.name, :action => :show, :project => @project.name ), "Repositories" + link_to( @project, :action => :show, :project => @project ), "Repositories" ] -%>
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 14.04.2010 14:36, Adrian Schröter wrote:
The repserver has already some code to find out if a repo is published already or not.
We could extend that and tell you already, if a package has been published or not.
So we could avoid this additional traffic and IMHO also possible area of bugs.
What do you think about that ?
I don't know. The implemented way is only a head request... Can we rely on the repservers published state? Maybe there get files removed on the public server after being published or sth. like that? We already have a published state in the buildresult, is this related? Greetings
bye adrian
Am Mittwoch, 14. April 2010 14:26:15 schrieben Sie:
From: Thomas Schmidt
--- src/webui/app/controllers/package_controller.rb | 35 ++++++++++----------- src/webui/app/views/project/repositories.html.erb | 2 +- 2 files changed, 18 insertions(+), 19 deletions(-)
diff --git a/src/webui/app/controllers/package_controller.rb b/src/webui/app/controllers/package_controller.rb index 105885f..54a3452 100644 --- a/src/webui/app/controllers/package_controller.rb +++ b/src/webui/app/controllers/package_controller.rb @@ -33,7 +33,6 @@ class PackageController < ApplicationController elsif @project.bugowner @bugowner_mail = Person.find_cached( @project.bugowner ).email.to_s end - fill_status_cache end
@@ -44,7 +43,7 @@ class PackageController < ApplicationController @dproject = params[:dproject] @filename = params[:filename] @fileinfo = Fileinfo.find_cached( :project => params[:dproject], :package => '_repository', :repository => params[:drepository], :arch => @arch, - :filename => params[:dname], :view => 'fileinfo_ext') + :filename => params[:dname], :view => 'fileinfo_ext') @durl = nil end
@@ -53,28 +52,15 @@ class PackageController < ApplicationController @repository = params[:repository] @filename = params[:filename] @fileinfo = Fileinfo.find_cached( :project => @project, :package => @package, :repository => @repository, :arch => @arch, - :filename => @filename, :view => 'fileinfo_ext') - puts @fileinfo + :filename => @filename, :view => 'fileinfo_ext') @durl = repo_url( @project, @repository ) + "/#{@fileinfo.arch}/#{@filename}" - puts @durl - if @durl - uri = URI.parse( @durl ) - response = nil - Net::HTTP.start(uri.host, uri.port) do |http| - response = http.head uri.path - end - @durl = nil if response.code.to_i >= 400 - else - logger.debug "no repository url" - end - if @user and not @durl - # use API for logged in users if the mirror is not available + if !file_available?( @durl ) and @user + # only use API for logged in users if the mirror is not available @durl = rpm_url( @project, @package, @repository, @arch, @filename ) end end
def binaries - @repository = params[:repository] @buildresult = Buildresult.find( :project => @project, :package => @package, :repository => @repository, :view => ['binarylist', 'status'] ) @@ -835,6 +821,19 @@ class PackageController < ApplicationController
private
+ def file_available? url, max_redirects=5 + uri = URI.parse( url ) + Net::HTTP.start(uri.host, uri.port) do |http| + logger.debug "Checking url: #{url}" + response = http.head uri.path + if response.code.to_i == 302 and response['location'] and max_redirects > 0 + return file_available? response['location'], (max_redirects - 1) + end + response.code.to_i == 200 ? true : false + end + end + + def get_files( project, package ) # files whose name ends in the following extensions should not be editable no_edit_ext = %w{ .bz2 .dll .exe .gem .gif .gz .jar .jpeg .jpg .lzma .ogg .pdf .pk3 .png .ps .rpm .svgz .tar .taz .tb2 .tbz .tbz2 .tgz .tlz .txz .xpm .xz .z .zip } diff --git a/src/webui/app/views/project/repositories.html.erb b/src/webui/app/views/project/repositories.html.erb index 615c395..00dbb55 100644 --- a/src/webui/app/views/project/repositories.html.erb +++ b/src/webui/app/views/project/repositories.html.erb @@ -1,7 +1,7 @@ <% @crumb_list = [ link_to( 'Projects', :action => :list_public), - link_to( @project.name, :action => :show, :project => @project.name ), "Repositories" + link_to( @project, :action => :show, :project => @project ), "Repositories" ] -%>
-- Thomas Schmidt (tschmidt [at] suse.de) SUSE Linux Products GmbH :: Research & Development :: Tools "Don't Panic", Douglas Adams (1952 - 11.05.2001) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wed, Apr 14, 2010 at 02:43:44PM +0200, Thomas Schmidt wrote:
On 14.04.2010 14:36, Adrian Schröter wrote:
The repserver has already some code to find out if a repo is published already or not.
We could extend that and tell you already, if a package has been published or not.
So we could avoid this additional traffic and IMHO also possible area of bugs.
What do you think about that ?
I don't know. The implemented way is only a head request... Can we rely on the repservers published state? Maybe there get files removed on the public server after being published or sth. like that? We already have a published state in the buildresult, is this related?
The "published" state of the repository doesn't help at all, as you don't know which packages are the unpublished ones if the state is not "published". Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Mittwoch, 14. April 2010 14:45:44 schrieb Michael Schroeder:
On Wed, Apr 14, 2010 at 02:43:44PM +0200, Thomas Schmidt wrote:
On 14.04.2010 14:36, Adrian Schröter wrote:
The repserver has already some code to find out if a repo is published already or not.
We could extend that and tell you already, if a package has been published or not.
So we could avoid this additional traffic and IMHO also possible area of bugs.
What do you think about that ?
I don't know. The implemented way is only a head request... Can we rely on the repservers published state? Maybe there get files removed on the public server after being published or sth. like that? We already have a published state in the buildresult, is this related?
The "published" state of the repository doesn't help at all, as you don't know which packages are the unpublished ones if the state is not "published".
Yes, the current information are not enough for this. But I think we could extend the current code, which lists all binary results of a package. The check would need to validate if the package was prepared for publish (in :repo) and if the publish happened already. Btw, there is still the problem, that it is sort of unknown in which sub directory the package will land (i586/ or noarch/ or suse/x86_64/ or amd64/ or all/ or ...) depending on the repository mode. But the current code seems also only to guess here. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Mittwoch, 14. April 2010 15:28:09 schrieb Adrian Schröter:
Am Mittwoch, 14. April 2010 14:45:44 schrieb Michael Schroeder:
On Wed, Apr 14, 2010 at 02:43:44PM +0200, Thomas Schmidt wrote:
On 14.04.2010 14:36, Adrian Schröter wrote:
The repserver has already some code to find out if a repo is published already or not.
We could extend that and tell you already, if a package has been published or not.
So we could avoid this additional traffic and IMHO also possible area of bugs.
What do you think about that ?
I don't know. The implemented way is only a head request... Can we rely on the repservers published state? Maybe there get files removed on the public server after being published or sth. like that? We already have a published state in the buildresult, is this related?
The "published" state of the repository doesn't help at all, as you don't know which packages are the unpublished ones if the state is not "published".
Yes, the current information are not enough for this. But I think we could extend the current code, which lists all binary results of a package. The check would need to validate if the package was prepared for publish (in :repo) and if the publish happened already.
Btw, there is still the problem, that it is sort of unknown in which sub directory the package will land (i586/ or noarch/ or suse/x86_64/ or amd64/ or all/ or ...) depending on the repository mode. But the current code seems also only to guess here.
As just discussed with Michael, we see some more problems. For example .noarch.rpm files or packages where the packager is playing with the release number would not necessary the same on download.o.o. Yes, in most cases it should not matter. But when you debug some error (for example a noarch.rpm which builds differently on the architectures due to some bug) this could drive you crazy, because you may get the wrong package. It might be possible to do this check via the /published route to guarantee that the file is published and really the file on download.o.o is from the build shown in the webui. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Mittwoch, 14. April 2010 16:11:02 schrieb Adrian Schröter:
As just discussed with Michael, we see some more problems. For example .noarch.rpm files or packages where the packager is playing with the release number would not necessary the same on download.o.o.
Yes, in most cases it should not matter. But when you debug some error (for example a noarch.rpm which builds differently on the architectures due to some bug) this could drive you crazy, because you may get the wrong package.
It might be possible to do this check via the /published route to guarantee that the file is published and really the file on download.o.o is from the build shown in the webui.
So you're not worried about traffic at all? A head request to the download url sounds way less traffic than one to three api->backend ping pongs. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (4)
-
Adrian Schröter -
Michael Schroeder -
Stephan Kulow -
Thomas Schmidt