[opensuse-buildservice] test of obs-server.x86_64-2.2.116-Build1.2.raw in a update from version 2.1
Hi, I want share my expirence during a update of a test install from image obs-server.x86_64-2.1.16-Build1.6.raw.bz2 to obs-server.x86_64-2.2.116-Build1.2.raw.bz2. It was a more or less standard setup following the howto on a x86_64 physical host, booted from USB stick and have a LVM setup for the storage: pingi8:/usr/lib/obs/server # ll /dev/OBS/ total 0 drwxr-xr-x 2 root root 240 Jan 30 10:13 . drwxr-xr-x 22 root root 4560 Jan 30 10:13 .. lrwxrwxrwx 1 root root 7 Jan 30 10:13 cache -> ../dm-1 lrwxrwxrwx 1 root root 7 Jan 30 10:13 server -> ../dm-2 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_1 -> ../dm-3 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_2 -> ../dm-4 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_3 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_4 -> ../dm-6 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_1 -> ../dm-7 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_2 -> ../dm-8 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_3 -> ../dm-9 lrwxrwxrwx 1 root root 8 Jan 30 10:13 worker_swap_4 -> ../dm-10 Under 2.1 a small local project was copied from OBS and setuped for local build, for 11.4 and 12.1 (32 and 64 bit) and OBS remote link (to opensuse.org) 2.1 was running fine, every package did build. Then I replaced the 2.1.16 USB Stick whith the 2.2.116 version. I did this after a full backup and a clean shutdown. First boot die crash the MySQL database completely during the MySQL update process. The database did not longer work and could not be repaired (BUG 743697 was submitted). Later I restored the database backup on an other 12.1 machine and did the update manually - without any error. So it looks like that if the update is running on boot time something goes wrong (I did repeat it 3 times). Then I copied the updated data base to the target machine again and did the database migration manually - also without any error. After reboot MySQL was not running, because it did start before the /obs directory was mounted - fixed with Required-Start: $network $remote_fs obsstoragesetup in the mysql startscript and insserv. obsstoragesetup did not mount /dev/OBS/cache, moved the mount in the startscript before the worker setups. Then apache did start before MYSQL was started, because it was in systemd defined as service. Deleted apache2.service file, now apache2 starts as expected after the required services are running. Still wrong rights on many files under /obs (because of changed UID/GID values, repaired with chown -R obsrun:obsrun /obs/* chown -R mysql:mysql /obs/MySQL No access to http://server:82 Error 403 - link not allowed, because root dir is /srv/obs/repos which is symlinked to /obs/repos. I tried option FollowSymLinks in the *82 vhost config in /etc/apache2/vhosts.d/obs.conf and did a restart - still not working. Then I changed the root dir to /obs/repos in /etc/apache2/vhosts.d/obs.conf - works, now I have access to the repos again and also zypper works with the test project. Main webinterface https://server also works, but builds did not. Also montitor shows, that the schedulers are not running, after rcobsscheduler start the processes seems to start and ps ax show them for 3-5 sec, then they die. Logfile looks like: http://pastie.org/3282456, last lines are: fetching remote project data for openSUSE.org:openSUSE:12.1 fetching remote project data for openSUSE.org:SUSE:SLE-11 sorting projects and repositories... setting up watcher for https://api.opensuse.org/public exiting (with complete info)... bye. Even reboot or several restarts of rcobsscheduler did not start it. For further debugging I started it with cd /usr/lib/obs/server strace ./bs_sched x86_64 and it run normally, also the web monitor changed from red to green, and builds are running now. I stopped the strace and did run rcobsserver start again - now also this works for x86_64 as it should !??? - i586 still not runing (did die after few secs). Repeated the same procedure for i586 without strace. cd /usr/lib/obs/server ./bs_sched i586 also this does work now and the i586 builds also succeeded. Now the 2.2.116 apliance seems to run just fine :-) Maybe this helps to find the remaining problems, if you want some logfiles request it. Best Karsten -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Dienstag, 31. Januar 2012, 01:59:52 schrieb Karsten Keil: ...
After reboot MySQL was not running, because it did start before the /obs directory was mounted - fixed with Required-Start: $network $remote_fs obsstoragesetup in the mysql startscript and insserv. obsstoragesetup did not mount /dev/OBS/cache, moved the mount in the startscript before the worker setups.
We found out yesterday that the "boot.lvm" script is not enabled by default anymore. Changed now in the appliance. The modification of the mysql init script is not a good way, because this would not work outside of the appliance and break package updates. We have the "Start-Before: mysql ssh" line instead in obsstoragesetup which should let it start before.
Then apache did start before MYSQL was started, because it was in systemd defined as service. Deleted apache2.service file, now apache2 starts as expected after the required services are running.
Actually I am only testing with sysvinit atm. Most likely the default will change for the appliance. However, apache2 and mysqld should be able to startup in parallel.
Still wrong rights on many files under /obs (because of changed UID/GID values, repaired with chown -R obsrun:obsrun /obs/*
Really ? I thought this is fixed meanwhile.
chown -R mysql:mysql /obs/MySQL
Yep, noticed also. ... moin adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Jan 31, 2012 at 09:34:29AM +0100, Adrian Schröter wrote:
Am Dienstag, 31. Januar 2012, 01:59:52 schrieb Karsten Keil: ...
After reboot MySQL was not running, because it did start before the /obs directory was mounted - fixed with Required-Start: $network $remote_fs obsstoragesetup in the mysql startscript and insserv. obsstoragesetup did not mount /dev/OBS/cache, moved the mount in the startscript before the worker setups.
We found out yesterday that the "boot.lvm" script is not enabled by default anymore. Changed now in the appliance.
Hmm, but I saw some extra mdadm and lvm calls in the obsstorage script, so it still works somehow, but maybe not early enough for mysql. What I still cannot explain is, that on the first boot /obs/MYSQL was here and mysql was started - because if not mysql-update would not start at all but it did - and destroyed the database. It also seems that the database migration does run on every reboot.
The modification of the mysql init script is not a good way, because this would not work outside of the appliance and break package updates.
Of course, this was my quick and dirty workaround.
We have the "Start-Before: mysql ssh" line instead in obsstoragesetup which should let it start before.
Then apache did start before MYSQL was started, because it was in systemd defined as service. Deleted apache2.service file, now apache2 starts as expected after the required services are running.
Actually I am only testing with sysvinit atm. Most likely the default will change for the appliance.
However, apache2 and mysqld should be able to startup in parallel.
Still wrong rights on many files under /obs (because of changed UID/GID values, repaired with chown -R obsrun:obsrun /obs/*
Really ? I thought this is fixed meanwhile.
Maybe it did not run completely because the mysql issue. Unfortunately I did not saved a ls -l /obs in this state.
chown -R mysql:mysql /obs/MySQL
Yep, noticed also.
Does this maybe explain the database update crash - wrong rights ? Any idea to the scheduler issue - why did a manual run be required and what does the difference be to a run from the obsscheduler script ? And KVM is missing on the appliance - intended or bug ? Karsten -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Dienstag, 31. Januar 2012, 12:14:55 schrieb Karsten Keil:
On Tue, Jan 31, 2012 at 09:34:29AM +0100, Adrian Schröter wrote:
Am Dienstag, 31. Januar 2012, 01:59:52 schrieb Karsten Keil: ...
After reboot MySQL was not running, because it did start before the /obs directory was mounted - fixed with Required-Start: $network $remote_fs obsstoragesetup in the mysql startscript and insserv. obsstoragesetup did not mount /dev/OBS/cache, moved the mount in the startscript before the worker setups.
We found out yesterday that the "boot.lvm" script is not enabled by default anymore. Changed now in the appliance.
Hmm, but I saw some extra mdadm and lvm calls in the obsstorage script, so it still works somehow, but maybe not early enough for mysql.
What I still cannot explain is, that on the first boot /obs/MYSQL was here and mysql was started - because if not mysql-update would not start at all but it did - and destroyed the database.
yes, me neither.
It also seems that the database migration does run on every reboot.
yes, there is no way to detect for the appliance if the database is new enough. Should not harm usually. ..
chown -R mysql:mysql /obs/MySQL
Yep, noticed also.
Does this maybe explain the database update crash - wrong rights ?
IMHO not. If mysqld has not enough permissions, it should also not be able to trash the file content. However, we need to fix it anyway.
Any idea to the scheduler issue - why did a manual run be required and what does the difference be to a run from the obsscheduler script ?
no. I need first to find out how to configure with kiwi to boot via sysvinit, (everything got rewritten so far). systemd boot is just hopelesse atm.
And KVM is missing on the appliance - intended or bug ?
kvm package is in the file list. Or what do you mean ?
Karsten
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Jan 31, 2012 at 03:18:44PM +0100, Adrian Schröter wrote:
Am Dienstag, 31. Januar 2012, 12:14:55 schrieb Karsten Keil:
On Tue, Jan 31, 2012 at 09:34:29AM +0100, Adrian Schröter wrote:
Am Dienstag, 31. Januar 2012, 01:59:52 schrieb Karsten Keil: ...
Any idea to the scheduler issue - why did a manual run be required and what does the difference be to a run from the obsscheduler script ?
no. I need first to find out how to configure with kiwi to boot via sysvinit, (everything got rewritten so far). systemd boot is just hopelesse atm.
Agree seems this is the better solution for now.
And KVM is missing on the appliance - intended or bug ?
kvm package is in the file list. Or what do you mean ?
It is not installed, also qemu is missing (in the raw image). On the 2.1 image it was here and my builds run via KVM, now they are running in chroot I think. Found something not working: Download of Logfiles via web. The life display during build is working, also the last part is shown if you klick on "succeed" but try to download gives an: Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, [no address given] and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request. Apache/2.2.21 (Linux/SUSE) Server at pingi8.linux-pingi.de Port 443 Karsten -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
JFYI, the current appliances are using sysvinit now and that solves most of the problems. I hope the database trashing will not happen again :( bye adrian Am Dienstag, 31. Januar 2012, 01:59:52 schrieb Karsten Keil:
Hi,
I want share my expirence during a update of a test install from image obs-server.x86_64-2.1.16-Build1.6.raw.bz2 to obs-server.x86_64-2.2.116-Build1.2.raw.bz2.
It was a more or less standard setup following the howto on a x86_64 physical host, booted from USB stick and have a LVM setup for the storage:
pingi8:/usr/lib/obs/server # ll /dev/OBS/ total 0 drwxr-xr-x 2 root root 240 Jan 30 10:13 . drwxr-xr-x 22 root root 4560 Jan 30 10:13 .. lrwxrwxrwx 1 root root 7 Jan 30 10:13 cache -> ../dm-1 lrwxrwxrwx 1 root root 7 Jan 30 10:13 server -> ../dm-2 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_1 -> ../dm-3 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_2 -> ../dm-4 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_3 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_4 -> ../dm-6 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_1 -> ../dm-7 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_2 -> ../dm-8 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_3 -> ../dm-9 lrwxrwxrwx 1 root root 8 Jan 30 10:13 worker_swap_4 -> ../dm-10
Under 2.1 a small local project was copied from OBS and setuped for local build, for 11.4 and 12.1 (32 and 64 bit) and OBS remote link (to opensuse.org) 2.1 was running fine, every package did build. Then I replaced the 2.1.16 USB Stick whith the 2.2.116 version. I did this after a full backup and a clean shutdown. First boot die crash the MySQL database completely during the MySQL update process. The database did not longer work and could not be repaired (BUG 743697 was submitted).
Later I restored the database backup on an other 12.1 machine and did the update manually - without any error. So it looks like that if the update is running on boot time something goes wrong (I did repeat it 3 times).
Then I copied the updated data base to the target machine again and did the database migration manually - also without any error.
After reboot MySQL was not running, because it did start before the /obs directory was mounted - fixed with Required-Start: $network $remote_fs obsstoragesetup in the mysql startscript and insserv. obsstoragesetup did not mount /dev/OBS/cache, moved the mount in the startscript before the worker setups.
Then apache did start before MYSQL was started, because it was in systemd defined as service. Deleted apache2.service file, now apache2 starts as expected after the required services are running.
Still wrong rights on many files under /obs (because of changed UID/GID values, repaired with chown -R obsrun:obsrun /obs/* chown -R mysql:mysql /obs/MySQL
No access to http://server:82 Error 403 - link not allowed, because root dir is /srv/obs/repos which is symlinked to /obs/repos. I tried option FollowSymLinks in the *82 vhost config in /etc/apache2/vhosts.d/obs.conf and did a restart - still not working. Then I changed the root dir to /obs/repos in /etc/apache2/vhosts.d/obs.conf - works, now I have access to the repos again and also zypper works with the test project. Main webinterface https://server also works, but builds did not. Also montitor shows, that the schedulers are not running, after rcobsscheduler start the processes seems to start and ps ax show them for 3-5 sec, then they die. Logfile looks like: http://pastie.org/3282456, last lines are:
fetching remote project data for openSUSE.org:openSUSE:12.1 fetching remote project data for openSUSE.org:SUSE:SLE-11 sorting projects and repositories... setting up watcher for https://api.opensuse.org/public exiting (with complete info)... bye.
Even reboot or several restarts of rcobsscheduler did not start it. For further debugging I started it with cd /usr/lib/obs/server strace ./bs_sched x86_64
and it run normally, also the web monitor changed from red to green, and builds are running now. I stopped the strace and did run rcobsserver start again - now also this works for x86_64 as it should !??? - i586 still not runing (did die after few secs). Repeated the same procedure for i586 without strace. cd /usr/lib/obs/server ./bs_sched i586
also this does work now and the i586 builds also succeeded.
Now the 2.2.116 apliance seems to run just fine :-)
Maybe this helps to find the remaining problems, if you want some logfiles request it.
Best Karsten -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Any plans to remove the links to openSUSE's board wiki from the API interfaces ? :) NM 2012/2/1 Adrian Schröter <adrian@suse.de>:
JFYI, the current appliances are using sysvinit now and that solves most of the problems.
I hope the database trashing will not happen again :(
bye adrian
Am Dienstag, 31. Januar 2012, 01:59:52 schrieb Karsten Keil:
Hi,
I want share my expirence during a update of a test install from image obs-server.x86_64-2.1.16-Build1.6.raw.bz2 to obs-server.x86_64-2.2.116-Build1.2.raw.bz2.
It was a more or less standard setup following the howto on a x86_64 physical host, booted from USB stick and have a LVM setup for the storage:
pingi8:/usr/lib/obs/server # ll /dev/OBS/ total 0 drwxr-xr-x 2 root root 240 Jan 30 10:13 . drwxr-xr-x 22 root root 4560 Jan 30 10:13 .. lrwxrwxrwx 1 root root 7 Jan 30 10:13 cache -> ../dm-1 lrwxrwxrwx 1 root root 7 Jan 30 10:13 server -> ../dm-2 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_1 -> ../dm-3 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_2 -> ../dm-4 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_3 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_root_4 -> ../dm-6 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_1 -> ../dm-7 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_2 -> ../dm-8 lrwxrwxrwx 1 root root 7 Jan 30 10:13 worker_swap_3 -> ../dm-9 lrwxrwxrwx 1 root root 8 Jan 30 10:13 worker_swap_4 -> ../dm-10
Under 2.1 a small local project was copied from OBS and setuped for local build, for 11.4 and 12.1 (32 and 64 bit) and OBS remote link (to opensuse.org) 2.1 was running fine, every package did build. Then I replaced the 2.1.16 USB Stick whith the 2.2.116 version. I did this after a full backup and a clean shutdown. First boot die crash the MySQL database completely during the MySQL update process. The database did not longer work and could not be repaired (BUG 743697 was submitted).
Later I restored the database backup on an other 12.1 machine and did the update manually - without any error. So it looks like that if the update is running on boot time something goes wrong (I did repeat it 3 times).
Then I copied the updated data base to the target machine again and did the database migration manually - also without any error.
After reboot MySQL was not running, because it did start before the /obs directory was mounted - fixed with Required-Start: $network $remote_fs obsstoragesetup in the mysql startscript and insserv. obsstoragesetup did not mount /dev/OBS/cache, moved the mount in the startscript before the worker setups.
Then apache did start before MYSQL was started, because it was in systemd defined as service. Deleted apache2.service file, now apache2 starts as expected after the required services are running.
Still wrong rights on many files under /obs (because of changed UID/GID values, repaired with chown -R obsrun:obsrun /obs/* chown -R mysql:mysql /obs/MySQL
No access to http://server:82 Error 403 - link not allowed, because root dir is /srv/obs/repos which is symlinked to /obs/repos. I tried option FollowSymLinks in the *82 vhost config in /etc/apache2/vhosts.d/obs.conf and did a restart - still not working. Then I changed the root dir to /obs/repos in /etc/apache2/vhosts.d/obs.conf - works, now I have access to the repos again and also zypper works with the test project. Main webinterface https://server also works, but builds did not. Also montitor shows, that the schedulers are not running, after rcobsscheduler start the processes seems to start and ps ax show them for 3-5 sec, then they die. Logfile looks like: http://pastie.org/3282456, last lines are:
fetching remote project data for openSUSE.org:openSUSE:12.1 fetching remote project data for openSUSE.org:SUSE:SLE-11 sorting projects and repositories... setting up watcher for https://api.opensuse.org/public exiting (with complete info)... bye.
Even reboot or several restarts of rcobsscheduler did not start it. For further debugging I started it with cd /usr/lib/obs/server strace ./bs_sched x86_64
and it run normally, also the web monitor changed from red to green, and builds are running now. I stopped the strace and did run rcobsserver start again - now also this works for x86_64 as it should !??? - i586 still not runing (did die after few secs). Repeated the same procedure for i586 without strace. cd /usr/lib/obs/server ./bs_sched i586
also this does work now and the i586 builds also succeeded.
Now the 2.2.116 apliance seems to run just fine :-)
Maybe this helps to find the remaining problems, if you want some logfiles request it.
Best Karsten -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- Nelson Marques /* http://www.marques.so nmo.marques@gmail.com */ -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Adrian Schröter -
Karsten Keil -
Nelson Marques