[opensuse-buildservice] deploying the OBS backend to multiple servers
Hi all, I was doing some thoughts about what is the best way for load balancing by deploying the backend to multiple servers. I have given this some thoughts and would be interested to hear the opinion of people with experience on deploying things to multiple servers or by those that actually implemented those things. My understanding is that in principle I could deploy each and every backend module (like bs_dispatcher or bs_publish) could be deployed to a different system as long as they have a shared /srv/obs directory. I am not saying that doing things in that an extreme way is reasonable but just wanted to check whether this is technically correct or whether someone sees a problem with that? Next I did a quick analysis of what backend module seems to access which parts of the /srv/obs namespace and found the following relations: bs_srcserver reads and writes to /build, /db/published, /db/source, /db/request, /diffcache, /projects, /remotecache, /sources, /trees, and /events/service bs_repserver reads and writes to /build, /workers, /repos, /events/$ARCH, /events/dispatch, and /events/signer bs_repserver reads from /db/published, /info, and /events/publish bs_scheduler reads and writes to /jobs, /info, and /events/$ARCH bs_scheduler reads from /repos bs_dispatcher reads and writes to /workers, /jobs, /events/dispatch, and /events/repository bs_warden reads and writes to /workers and /jobs bs_signer reads and writes to /jobs, /events/$ARCH, and /events/signer bs_publisher reads and writes to /build, /db/published, /repos, /repos_sync, and /events/publish bs_sshgit reads and writes to /projects This information was gathered by a lot of grepping through the source code and thus is not necessarily complete or completely correct. What I'd be interested here is now whether someone could tell me whether this is correct or what is missing/wrong in this analysis. What is completely missing from this analysis is the /upload directory, since I saw that multiple modules were using that one but it seemed they were using completely independent namespaces within there. Is this correct? The background of this analysis is that if I splitted different modules to different servers that I knew which directories should be shared among which hosts via NFS and which of them could be on the local disk. Or would you generally recommend to share the whole filesystem space among them since relying on the above information is generally to fragile? Are there people with a general recommendation on how to split the various modules to different servers. When I talked to Adrian once he suggested to run the WebUI+API on one machine, the srcserver on another one and the rest on a third one (at least if I remember correctly). For sure the workers will run on separate machines in any case. Is this what people think is a good balancing or does someone come up with different ideas? Could someone describe the infrastructure of build.opensuse.org? Which parts are running on separate servers, what parts of the relevant filesystems are network mounted on the machines from where and so on? Robert BTW: I created a graph of the information mentioned above with graphviz. If someone with better insight can provide the missing pieces and fixes and people think this is interesting for OBS documentation I am happy to contribute that. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wed, Aug 31, 2011 at 01:12:58PM +0200, Robert Schiele wrote:
This information was gathered by a lot of grepping through the source code and thus is not necessarily complete or completely correct. What I'd be interested here is now whether someone could tell me whether this is correct or what is missing/wrong in this analysis.
I think it's pretty correct.
What is completely missing from this analysis is the /upload directory, since I saw that multiple modules were using that one but it seemed they were using completely independent namespaces within there. Is this correct?
That's just a tmp dir.
The background of this analysis is that if I splitted different modules to different servers that I knew which directories should be shared among which hosts via NFS and which of them could be on the local disk. Or would you generally recommend to share the whole filesystem space among them since relying on the above information is generally to fragile?
Are there people with a general recommendation on how to split the various modules to different servers. When I talked to Adrian once he suggested to run the WebUI+API on one machine, the srcserver on another one and the rest on a third one (at least if I remember correctly). For sure the workers will run on separate machines in any case. Is this what people think is a good balancing or does someone come up with different ideas?
The design is: - one source server running bs_srcserver [ - one source service server, running bs_service ] - multiple repository servers, each running a bs_repserver bs_sched bs_dispatch bs_warden bs_signer The "multiple" part is not really implemented yet (but it would not be too hard to implement it). The publishing part is missing from the list, currently you would need to run a bs_publish on each repserver host as well. Maybe this can be relaxed in the future.
Could someone describe the infrastructure of build.opensuse.org? Which parts are running on separate servers, what parts of the relevant filesystems are network mounted on the machines from where and so on?
We use the above layout (with just one repo server). We do *not* use NFS. We had many troubles with NFS in the old autobuild system, thus the obs is designed to not use NFS. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi Michael, Thanks for the quick and detailed answer! On Wed, Aug 31, 2011 at 1:32 PM, Michael Schroeder <mls@suse.de> wrote:
On Wed, Aug 31, 2011 at 01:12:58PM +0200, Robert Schiele wrote:
This information was gathered by a lot of grepping through the source code and thus is not necessarily complete or completely correct. What I'd be interested here is now whether someone could tell me whether this is correct or what is missing/wrong in this analysis.
I think it's pretty correct.
Ah cool! Just by grepping for file names and variable names. Seems you have used the variable names in quite a consistent fashion!
The design is: - one source server running bs_srcserver
[ - one source service server, running bs_service ]
- multiple repository servers, each running a bs_repserver bs_sched bs_dispatch bs_warden bs_signer
The "multiple" part is not really implemented yet (but it would not be too hard to implement it).
The publishing part is missing from the list, currently you would need to run a bs_publish on each repserver host as well. Maybe this can be relaxed in the future.
We do *not* use NFS. We had many troubles with NFS in the old autobuild system, thus the obs is designed to not use NFS.
Ok, if you don't use NFS I have one further question: According to my analysis all three of bs_srcserver, bs_publisher, and bs_repserver seem to access /build and /db/published. Now if you don't use NFS those directories are not shared among those and the srcserver will see different stuff there than the other two. Is that ok or do I have to do something about that? Or is my analysis just incorrect in that specific case? Robert -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Wed, Aug 31, 2011 at 04:22:45PM +0200, Robert Schiele wrote:
Ok, if you don't use NFS I have one further question: According to my analysis all three of bs_srcserver, bs_publisher, and bs_repserver seem to access /build and /db/published.
No, seems I missed that. bs_srcserver doesn't access /build, bs_repserver and bs_publisher doesn't access /db/published (bs_repserver and bs_publisher used to own the /db/published database, but we moved it to the source server to speed up the searches.) M. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (2)
-
Michael Schroeder
-
Robert Schiele