[opensuse-buildservice] spec file wizard - how to proceed?
Hi, as you probably noticed, the buildservice web interface now has a wizard that helps with writing spec files. Yesterday, I improved it to generate better spec files for perl and python modules (not yet on build.o.o) and I hot already advice on how to add support for apache modules and ruby gems ;-). Of course I'd like the wizard to be able to package as many kinds of packages as possible, but I see some problems I'd like to solve somehow before adding more features: Problem #1 (more theoretical than practical): the main part of the wizard is implemented in the frontend (aka API), the webclient only translates xml to html forms and vice versa. The reason for this is that I'd like to add a wizard interface also to osc one day. The problem is that so far, the frontend has been just a simple proxy to the backend, with access control, without any fancy functionality. Now the wizard adds potentially buggy code that doesn't necessarily need to be there (it would be easily implemented as a standalone application talking to the frontend). The only reason for having the wizard on the frontend is to be able to use it from different clients easily. Problem #2 (practical): To implement good heuristics, the wizard at least needs to look into the source tarball (if not run some commands...). Here comes the problem: 'tar tf foo-1.1.tar.bz2' can take a *lot* of cpu time, so where to perform such expensive actions? a) on the frontend - can be abused for DoS easily b) let the client send the filelist along with the tarball - makes the client implementation too complicated, plus moves the DoS problem from the frontend to the webclient c) on the backend, dispatching the jobs to workers - would also allow for more dangerous analysis of the package, but would impose additional delays d) create a separate wizard service and do all the job there - would also solve problem #1, but would make the client implementation a bit more difficult (need to talk to two different servers) e) [insert great idea here]? Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wednesday 18 June 2008 16:24:35 Michal Marek wrote: ..
Problem #2 (practical): To implement good heuristics, the wizard at least needs to look into the source tarball (if not run some commands...). Here comes the problem: 'tar tf foo-1.1.tar.bz2' can take a *lot* of cpu time, so where to perform such expensive actions? a) on the frontend - can be abused for DoS easily b) let the client send the filelist along with the tarball - makes the client implementation too complicated, plus moves the DoS problem from the frontend to the webclient c) on the backend, dispatching the jobs to workers - would also allow for more dangerous analysis of the package, but would impose additional delays d) create a separate wizard service and do all the job there - would also solve problem #1, but would make the client implementation a bit more difficult (need to talk to two different servers) e) [insert great idea here]?
What is with having an own service for this but handle it via the API ? We could run this service inside of a chroot or maybe even on a remote system, but not necessary on the backend server. This service would work on the scanning jobs sequential to avoid the DoS. You would need to implement some kind of waiting procedure in the frontend for this to give the user feedback ... bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Adrian Schröter wrote:
On Wednesday 18 June 2008 16:24:35 Michal Marek wrote: ..
Problem #2 (practical): To implement good heuristics, the wizard at least needs to look into the source tarball (if not run some commands...). Here comes the problem: 'tar tf foo-1.1.tar.bz2' can take a *lot* of cpu time, so where to perform such expensive actions? ... What is with having an own service for this but handle it via the API ? We could run this service inside of a chroot or maybe even on a remote system, but not necessary on the backend server. This service would work on the scanning jobs sequential to avoid the DoS. You would need to implement some kind of waiting procedure in the frontend for this to give the user feedback
Sounds like a good idea - implement the wizard in the API, but have a separate service (behind the API) to extract tarballs etc. What do the others think? From an administrative POV - is it possible to get a (virtual) machine for this? (I know you guys are currently busy with the storage problems...) Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Michal Marek wrote:
Adrian Schröter wrote:
On Wednesday 18 June 2008 16:24:35 Michal Marek wrote: ..
Problem #2 (practical): To implement good heuristics, the wizard at least needs to look into the source tarball (if not run some commands...). Here comes the problem: 'tar tf foo-1.1.tar.bz2' can take a *lot* of cpu time, so where to perform such expensive actions? ... What is with having an own service for this but handle it via the API ? We could run this service inside of a chroot or maybe even on a remote system, but not necessary on the backend server. This service would work on the scanning jobs sequential to avoid the DoS. You would need to implement some kind of waiting procedure in the frontend for this to give the user feedback
Sounds like a good idea - implement the wizard in the API, but have a separate service (behind the API) to extract tarballs etc. What do the others think? From an administrative POV - is it possible to get a (virtual) machine for this? (I know you guys are currently busy with the storage problems...)
You could consult your implementation with Andrei Oprisan. He is working on an more intelligent wizard on his Google SoC project: http://www.apkg.org/ He would be happy, if the service API would be compatible with his future APKG frontend API. -- Best Regards / S pozdravem, Stanislav Brabec software developer --------------------------------------------------------------------- SUSE LINUX, s. r. o. e-mail: sbrabec@suse.cz Lihovarská 1060/12 tel: +420 284 028 966, +49 911 740538747 190 00 Praha 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz/ --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Michal Marek wrote:
Adrian Schröter wrote:
On Wednesday 18 June 2008 16:24:35 Michal Marek wrote: ..
Problem #2 (practical): To implement good heuristics, the wizard at least needs to look into the source tarball (if not run some commands...). Here comes the problem: 'tar tf foo-1.1.tar.bz2' can take a *lot* of cpu time, so where to perform such expensive actions? ... What is with having an own service for this but handle it via the API ? We could run this service inside of a chroot or maybe even on a remote system, but not necessary on the backend server. This service would work on the scanning jobs sequential to avoid the DoS. You would need to implement some kind of waiting procedure in the frontend for this to give the user feedback
Sounds like a good idea - implement the wizard in the API, but have a separate service (behind the API) to extract tarballs etc. What do the others think? From an administrative POV - is it possible to get a (virtual) machine for this? (I know you guys are currently busy with the storage problems...)
I'm working on it (again), see http://idea.opensuse.org/content/ideas/package-wizard-for-the-build-service Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Michal Marek wrote:
Michal Marek wrote:
Sounds like a good idea - implement the wizard in the API, but have a separate service (behind the API) to extract tarballs etc. What do the others think? From an administrative POV - is it possible to get a (virtual) machine for this? (I know you guys are currently busy with the storage problems...)
I'm working on it (again), see http://idea.opensuse.org/content/ideas/package-wizard-for-the-build-service
So, here is what I have so far: - a server that queues request to inspect tarballs, hidden behind the API: https://build.opensuse.org/package/show?package=obs-wizard&project=home%3Amichal-m - patch to the frontend wizard that uses the server to gather information about uploaded source tarballs. It generates the %build and %install sections based on these information (e.g. if the tarball has CMakeLists.txt, run cmake). The patch is not in svn, because it requires the wizard service to function properly. I'll make some screenshots later today.
participants (3)
-
Adrian Schröter -
Michal Marek -
Stanislav Brabec