Mailinglist Archive: opensuse-buildservice (200 mails)

< Previous Next >
[opensuse-buildservice] Elastic Build Service - provisioning workers flexibly to Amazon EC2
Hello,

As my Master's Thesis project, I have been developing an elastic build service
based on OBS. It is a private instance of OBS, with a simple service manager
included. This service manager monitors the local build hosts and decides
whether it should spawn more workers to an Infrastructure-as-a-Service provider
to help out with the workload.

I have been running a test environment, having one local build host with 3
workers and 0-15 build hosts in Amazon each running a single worker. The
service manager runs as a script once every minute using cron. In practice,
when builds are triggered, the service manager requests new machines in 1-2
minutes, depending how long it takes for the server to schedule the jobs. In
total, it takes about 3-5 minutes for the build hosts to boot up in Amazon,
open a VPN connection to our OBS network, start up workers and start building.
Machines are launched into Amazon if there are jobs to be done in the queue and
if there are no idle local workers. Idle Amazon workers are killed the next
time they are about to be rebilled (Amazon bills you every full hour from
instance launch time).

This service manager script has been committed to the OBS repo:
https://gitorious.org/opensuse/build-service/blobs/master/dist/elastic_build_service_manager.py

There is a catch however. The Amazon Machine Image (AMI) that I am using has
been configured manually, meaning that IP addresses and other data the build
host needs to connect to the server are specific to our setup, and thus cannot
be shared. I have added to the script a really short summary of what you need
to setup your own elastic build service, and plan to add more detailed
instructions. The best solution would be to create a generic cloud worker
appliance image with SUSE Studio that has all the needed packages (obs-worker,
openvpn) installed. Then everyone could use the same machine image and the
build hosts would be configured on boot time using Amazon's user_data (when
requesting a new instance, it is possible to define arbitrary shell commands to
run during boot).

This is not production-ready. There is no persistent, shared cache in the cloud
yet (could S3 be used or do we need to run a fileserver) and the build
environment is not sandboxed into a virtual environment (OBS support for LXC
needed). Please comment and share your ideas.


Best Regards,
Ville Seppänen
Tieto Corporation

--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse-buildservice+help@xxxxxxxxxxxx

< Previous Next >
This Thread
  • No further messages