[opensuse-buildservice] How does obs manage the packages and projects files in backend?
Hi, guys I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service. I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code? thanks bamvor -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side. The source service server process is responsible to run them on server side. You can find some bits of docu here: http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_... And an example here: https://build.opensuse.org/package/show/OBS:Server:Unstable/build This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server? bye adrian -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Andrian, When using source services (such as download_url or gbs), are files (like _service:recompress:tar_scm:obs-build-20150320.tar.gz brought in through source services) still stored under "/obs/sources"? or only "_service" file itself stored under "obs/sources"? Also does "/obs/trees" store the md5sum record for only "_service" file or "/obs/trees" also include the md5sum record for files that brought in through source services, such as _service:recompress:tar_scm:obs-build-20150320.tar.gz? Rick -----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Monday, March 30, 2015 4:32 AM To: opensuse-buildservice@opensuse.org Cc: Bamvor Jian Zhang; Dingtianhong Subject: Re: [opensuse-buildservice] How does obs manage the packages and projects files in backend? On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side. The source service server process is responsible to run them on server side. You can find some bits of docu here: http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_... And an example here: https://build.opensuse.org/package/show/OBS:Server:Unstable/build This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server? bye adrian -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Dienstag, 31. März 2015, 22:37:11 wrote Rick Liu:
Hi Andrian,
When using source services (such as download_url or gbs), are files (like _service:recompress:tar_scm:obs-build-20150320.tar.gz brought in through source services) still stored under "/obs/sources"? or only "_service" file itself stored under "obs/sources"?
source service generate files are committed in the same way as user commits files. To avoid conflicts these files have therefore the _service: prefix, which can not be modified directly by the user.
Also does "/obs/trees" store the md5sum record for only "_service" file or "/obs/trees" also include the md5sum record for files that brought in through source services, such as _service:recompress:tar_scm:obs-build-20150320.tar.gz?
/obs/trees contains all files. Most important ones are the ...-MD5SUMS files, which list the files which are part of a commit. This file contains the file names and their md5sums. And also the special /SERVICE tag refering to source service generated files. Can you give me some insight why this is important for you? Do you plan to work on some code here? We could maybe share efforts, since we have also the git(/svn/..) mirroring on our agenda ... thanks adrian
Rick
-----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Monday, March 30, 2015 4:32 AM To: opensuse-buildservice@opensuse.org Cc: Bamvor Jian Zhang; Dingtianhong Subject: Re: [opensuse-buildservice] How does obs manage the packages and projects files in backend?
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side.
The source service server process is responsible to run them on server side. You can find some bits of docu here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
And an example here:
https://build.opensuse.org/package/show/OBS:Server:Unstable/build
This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server?
bye adrian
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 2015/4/1 15:00, Adrian Schröter wrote:
On Dienstag, 31. März 2015, 22:37:11 wrote Rick Liu:
Hi Andrian,
When using source services (such as download_url or gbs), are files (like _service:recompress:tar_scm:obs-build-20150320.tar.gz brought in through source services) still stored under "/obs/sources"? or only "_service" file itself stored under "obs/sources"?
source service generate files are committed in the same way as user commits files. To avoid conflicts these files have therefore the _service: prefix, which can not be modified directly by the user.
Also does "/obs/trees" store the md5sum record for only "_service" file or "/obs/trees" also include the md5sum record for files that brought in through source services, such as _service:recompress:tar_scm:obs-build-20150320.tar.gz?
/obs/trees contains all files. Most important ones are the ...-MD5SUMS files, which list the files which are part of a commit. This file contains the file names and their md5sums. And also the special /SERVICE tag refering to source service generated files. how and where does obs store the project relative file? Sorry, I do not have the obs server in my hand.
Can you give me some insight why this is important for you? Do you plan to work on some code here? We could maybe share efforts, since we have also the git(/svn/..) mirroring on our agenda ... Is there details about this plan? I found an old GSoC project about this[1]. Is it the similar ideas?
regards bamvor [1] https://en.opensuse.org/openSUSE:Build_Service_Concept_GitSupport
thanks adrian
Rick
-----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Monday, March 30, 2015 4:32 AM To: opensuse-buildservice@opensuse.org Cc: Bamvor Jian Zhang; Dingtianhong Subject: Re: [opensuse-buildservice] How does obs manage the packages and projects files in backend?
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side.
The source service server process is responsible to run them on server side. You can find some bits of docu here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
And an example here:
https://build.opensuse.org/package/show/OBS:Server:Unstable/build
This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server?
bye adrian
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 1. April 2015, 22:59:59 wrote Zhangjian:
On 2015/4/1 15:00, Adrian Schröter wrote:
On Dienstag, 31. März 2015, 22:37:11 wrote Rick Liu:
Hi Andrian,
When using source services (such as download_url or gbs), are files (like _service:recompress:tar_scm:obs-build-20150320.tar.gz brought in through source services) still stored under "/obs/sources"? or only "_service" file itself stored under "obs/sources"?
source service generate files are committed in the same way as user commits files. To avoid conflicts these files have therefore the _service: prefix, which can not be modified directly by the user.
Also does "/obs/trees" store the md5sum record for only "_service" file or "/obs/trees" also include the md5sum record for files that brought in through source services, such as _service:recompress:tar_scm:obs-build-20150320.tar.gz?
/obs/trees contains all files. Most important ones are the ...-MD5SUMS files, which list the files which are part of a commit. This file contains the file names and their md5sums. And also the special /SERVICE tag refering to source service generated files. how and where does obs store the project relative file? Sorry, I do not have the obs server in my hand.
Which file do you mean? the project meta data? It is also stored in the source repository, but the current version is also placed inside the projects/ directory for fast access.
Can you give me some insight why this is important for you? Do you plan to work on some code here? We could maybe share efforts, since we have also the git(/svn/..) mirroring on our agenda ...
Is there details about this plan? I found an old GSoC project about this[1]. Is it the similar ideas?
it goes into that direction, yes. However, the concept was written from a group which never followed up on it.
[1] https://en.opensuse.org/openSUSE:Build_Service_Concept_GitSupport
thanks adrian
Rick
-----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Monday, March 30, 2015 4:32 AM To: opensuse-buildservice@opensuse.org Cc: Bamvor Jian Zhang; Dingtianhong Subject: Re: [opensuse-buildservice] How does obs manage the packages and projects files in backend?
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side.
The source service server process is responsible to run them on server side. You can find some bits of docu here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
And an example here:
https://build.opensuse.org/package/show/OBS:Server:Unstable/build
This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server?
bye adrian
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi, Adrian I am try to refresh my memory, please correct me :p 1. source service on server mean that this service file is obs server? 2. how does obs get the source if there is no source service in a project? Does the source came from user commit? 3. how does obs maintain the branch and link package in the same project when using the source service in server side? and how about package cross the prject? On 2015/4/2 0:57, Adrian Schröter wrote:
On Mittwoch, 1. April 2015, 22:59:59 wrote Zhangjian:
On 2015/4/1 15:00, Adrian Schröter wrote:
On Dienstag, 31. März 2015, 22:37:11 wrote Rick Liu:
Hi Andrian,
When using source services (such as download_url or gbs), are files (like _service:recompress:tar_scm:obs-build-20150320.tar.gz brought in through source services) still stored under "/obs/sources"? or only "_service" file itself stored under "obs/sources"?
source service generate files are committed in the same way as user commits files. To avoid conflicts these files have therefore the _service: prefix, which can not be modified directly by the user.
Also does "/obs/trees" store the md5sum record for only "_service" file or "/obs/trees" also include the md5sum record for files that brought in through source services, such as _service:recompress:tar_scm:obs-build-20150320.tar.gz?
/obs/trees contains all files. Most important ones are the ...-MD5SUMS files, which list the files which are part of a commit. This file contains the file names and their md5sums. And also the special /SERVICE tag refering to source service generated files. how and where does obs store the project relative file? Sorry, I do not have the obs server in my hand.
Which file do you mean? the project meta data? Yes. prjconf for example. It is also stored in the source repository, but the current version is also placed inside the projects/ directory for fast access.
Can you give me some insight why this is important for you? Do you plan to work on some code here? We could maybe share efforts, since we have also the git(/svn/..) mirroring on our agenda ...
Is there details about this plan? I found an old GSoC project about this[1]. Is it the similar ideas?
it goes into that direction, yes. However, the concept was written from a group which never followed up on it.
[1] https://en.opensuse.org/openSUSE:Build_Service_Concept_GitSupport
thanks adrian
Rick
-----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Monday, March 30, 2015 4:32 AM To: opensuse-buildservice@opensuse.org Cc: Bamvor Jian Zhang; Dingtianhong Subject: Re: [opensuse-buildservice] How does obs manage the packages and projects files in backend?
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side.
The source service server process is responsible to run them on server side. You can find some bits of docu here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
And an example here:
https://build.opensuse.org/package/show/OBS:Server:Unstable/build
This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server?
bye adrian
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Freitag, 3. April 2015, 17:22:40 wrote Zhangjian:
Hi, Adrian
I am try to refresh my memory, please correct me :p
1. source service on server mean that this service file is obs server?
not sure if I understand you, but any service in a _service file can be configured to run local (osc) or remote on the OBS server by bs_service: http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
2. how does obs get the source if there is no source service in a project? Does the source came from user commit?
yes, the sources are always hosted by bs_srcserver process. Sources commited by the user, but also generated ones by the service server.
3. how does obs maintain the branch and link package in the same project when using the source service in server side? and how about package cross the prject?
service generated files are not taken from a linked package.
On 2015/4/2 0:57, Adrian Schröter wrote:
On Mittwoch, 1. April 2015, 22:59:59 wrote Zhangjian:
On 2015/4/1 15:00, Adrian Schröter wrote:
On Dienstag, 31. März 2015, 22:37:11 wrote Rick Liu:
Hi Andrian,
When using source services (such as download_url or gbs), are files (like _service:recompress:tar_scm:obs-build-20150320.tar.gz brought in through source services) still stored under "/obs/sources"? or only "_service" file itself stored under "obs/sources"?
source service generate files are committed in the same way as user commits files. To avoid conflicts these files have therefore the _service: prefix, which can not be modified directly by the user.
Also does "/obs/trees" store the md5sum record for only "_service" file or "/obs/trees" also include the md5sum record for files that brought in through source services, such as _service:recompress:tar_scm:obs-build-20150320.tar.gz?
/obs/trees contains all files. Most important ones are the ...-MD5SUMS files, which list the files which are part of a commit. This file contains the file names and their md5sums. And also the special /SERVICE tag refering to source service generated files. how and where does obs store the project relative file? Sorry, I do not have the obs server in my hand.
Which file do you mean? the project meta data? Yes. prjconf for example. It is also stored in the source repository, but the current version is also placed inside the projects/ directory for fast access.
Can you give me some insight why this is important for you? Do you plan to work on some code here? We could maybe share efforts, since we have also the git(/svn/..) mirroring on our agenda ...
Is there details about this plan? I found an old GSoC project about this[1]. Is it the similar ideas?
it goes into that direction, yes. However, the concept was written from a group which never followed up on it.
[1] https://en.opensuse.org/openSUSE:Build_Service_Concept_GitSupport
thanks adrian
Rick
-----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Monday, March 30, 2015 4:32 AM To: opensuse-buildservice@opensuse.org Cc: Bamvor Jian Zhang; Dingtianhong Subject: Re: [opensuse-buildservice] How does obs manage the packages and projects files in backend?
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side.
The source service server process is responsible to run them on server side. You can find some bits of docu here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
And an example here:
https://build.opensuse.org/package/show/OBS:Server:Unstable/build
This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time.
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server?
bye adrian
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Samstag, 28. März 2015, 18:02:57 wrote Bamvor Jian Zhang:
Hi, guys
I am trying to build the internal build service as well as integrate the build service with our development flow. Currently, our packages are maintained in svn and git. I want to learn that how does suse manage the packages and projects in build service backend. E.g. What happened in obs server after osc commit? I do not mean that how does the build service fetch the code through service.
The only way to let OBS fetching sources are currently source services. These are small modules which can either run on client or server side.
The source service server process is responsible to run them on server side. You can find some bits of docu here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.source_...
And an example here:
https://build.opensuse.org/package/show/OBS:Server:Unstable/build
This way works and can even be triggered without credentials, however it stores for every commit a new tar ball. So not very optimized disk usage wise. We want to improve that in future by mirroring git/svn/... and create the tar ball during build time. Got it. And how does obs management the branch and link project? Is there a service for pull the package from parent package and apply
On 2015/3/30 19:32, Adrian Schröter wrote: the diff?
I heard from irc that suse use some sort of file based method to do this. If it is true, where could I find feature lists and/or source code?
Not sure what you mean with "file based" here, the storage on the source server?
Yeah. I guess you already answer my question. regards bamvor
bye adrian
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (4)
-
Adrian Schröter -
Bamvor Jian Zhang -
Rick Liu -
Zhangjian (Bamvor)