[opensuse-buildservice] Build fails because unit tests runs daemon
Hi list, I added a unit test to my project at home:jcalcote/dnx This project built on all flavors of SuSE linux, plus fedora, debian, and ubuntu 7 before I added this new unit test. Now it fails on all but the debian and ubuntu builds - probably because these platforms' hacked up control files don't run "make check". I can successfully build this package locally - it just won't build on the obs servers. DETAILS: The unit test I added was a shell script to load my client daemon, ensure that it initialized properly, then send it a kill signal and wait for it to cleanup its own lock file (dnxClient.pid). Now, before you jump all over me for trying to write files to system directories during a build, be aware that I've configured my daemon to use a test directory for such files. Here's the relevant portion of the test script: - - - - - - - - - - - - - - - - - - # Execute the dnx client as a daemon ./dnxClient -c $PWD/testrun/test.cfg -r $PWD/testrun \ -l $PWD/testrun/test.log -D $PWD/testrun/test.dbg # Give it time to create its lock file count=0 while ((count<15)) && [ ! -e $PWD/testrun/dnxClient.pid ]; do sleep 1; let count++ done # Ensure we HAVE a lock file now if [ ! -e $PWD/testrun/dnxClient.pid ]; then echo "dnxClient.pid NOT found - daemon didn't start!" exit 1 fi - - - - - - - - - - - - - - - - - - Here's a portion of the build log from the openSUSE:10.2 target: - - - - - - - - - - - - - - - - - - ... Making check in client make[1]: Entering directory `/usr/src/packages/BUILD/dnx-0.15.1/client' /usr/bin/make check-TESTS make[2]: Entering directory `/usr/src/packages/BUILD/dnx-0.15.1/client' dnxClient.pid NOT found - daemon didn't start! FAIL: dnxClientTest.sh ================================================ 1 of 1 tests failed Please report to dnx-devel@lists.sourceforge.net ================================================ make[2]: *** [check-TESTS] Error 1 ... - - - - - - - - - - - - - - - - - - You can see that the script prints this message when the dnxClient daemon doesn't write its lock file. FYI, the command line options are: -c - specify configuration file. -r - specify run path (where the pid file will be created) -l - specify the log file name. -D - specify the debug log file name. I'm guessing there's some rule against running programs on the build servers that use fork-fork-exec. Any help would be much appreciated. Thanks in advance, John --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 28 February 2008 20:48:53 wrote John Calcote:
Hi list,
I added a unit test to my project at home:jcalcote/dnx
This project built on all flavors of SuSE linux, plus fedora, debian, and ubuntu 7 before I added this new unit test. Now it fails on all but the debian and ubuntu builds - probably because these platforms' hacked up control files don't run "make check".
I can successfully build this package locally - it just won't build on the obs servers.
Maybe due to the fact that these have no internet access ? Another possibility would be that they do not like the XEN kernel for some reason. Please try to add an "strace" before the unit test to find out a bit more. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2008-02-28 21:03:30 +0100, Adrian Schröter wrote:
On Thursday 28 February 2008 20:48:53 wrote John Calcote:
Hi list,
I added a unit test to my project at home:jcalcote/dnx
This project built on all flavors of SuSE linux, plus fedora, debian, and ubuntu 7 before I added this new unit test. Now it fails on all but the debian and ubuntu builds - probably because these platforms' hacked up control files don't run "make check".
I can successfully build this package locally - it just won't build on the obs servers.
Maybe due to the fact that these have no internet access ? Another possibility would be that they do not like the XEN kernel for some reason.
Please try to add an "strace" before the unit test to find out a bit more.
or maybe it requires root access due to port < 1024? darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 28 February 2008 21:27:13 wrote Marcus Rueckert:
On 2008-02-28 21:03:30 +0100, Adrian Schröter wrote:
On Thursday 28 February 2008 20:48:53 wrote John Calcote:
Hi list,
I added a unit test to my project at home:jcalcote/dnx
This project built on all flavors of SuSE linux, plus fedora, debian, and ubuntu 7 before I added this new unit test. Now it fails on all but the debian and ubuntu builds - probably because these platforms' hacked up control files don't run "make check".
I can successfully build this package locally - it just won't build on the obs servers.
Maybe due to the fact that these have no internet access ? Another possibility would be that they do not like the XEN kernel for some reason.
Please try to add an "strace" before the unit test to find out a bit more.
or maybe it requires root access due to port < 1024?
in that case it should also fail with local build (using osc build or the build script directly). -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Getting strace to work was an interesting task - first I forgot to add
strace to buildrequires, then I forgot to use the -f flag to cause it
to follow forks... Anyway, it works not, and I'll never forget how to
do it again. :)
Here's a snippet of the log file starting from where strace logs the
creation of the lock file, and ending at the removal of that file
during initialization. I'm guessing that somewhere in the middle is a
failed call that causes the system to shutdown while on its way up.
- - - - - - - - - - - - - - - - -
open("/usr/src/packages/BUILD/dnx-0.15.1/client/testrun/dnxClient.pid",
O_RDWR|O_CREAT, 0644) = 6
flock(6, LOCK_EX|LOCK_NB) = 0
write(6, "11002\n", 6) = 6
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 7
bind(7, {sa_family=AF_INET, sin_port=htons(12480),
sin_addr=inet_addr("0.0.0.0")}, 16) = 0
socket(PF_NETLINK, SOCK_RAW, 0) = 8
bind(8, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(8, {sa_family=AF_NETLINK, pid=11002, groups=00000000}, [12]) = 0
time(NULL) = 1204230691
sendto(8, "\24\0\0\0\22\0\1\3#\32\307G\0\0\0\0\0\0\0\0", 20, 0,
{sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\350\0\0\0\20\0\2\0#\32\307G\372*\0\0\0\0\4\3\1\0\0\0I\0\0\0\0\0\0\0"...,
4096}], msg_controllen=0, msg_flags=0}, 0) = 232
recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\24\0\0\0\3\0\2\0#\32\307G\372*\0\0\0\0\0\0\1\0\0\0I\0\0\0\0\0\0\0"...,
4096}], msg_controllen=0, msg_flags=0}, 0) = 20
sendto(8, "\24\0\0\0\26\0\1\3$\32\307G\0\0\0\0\0\0\0\0", 20, 0,
{sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"<\0\0\0\24\0\2\0$\32\307G\372*\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"...,
4096}], msg_controllen=0, msg_flags=0}, 0) = 60
recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0,
groups=00000000},
msg_iov(1)=[{"\24\0\0\0\3\0\2\0$\32\307G\372*\0\0\0\0\0\0\1\0\0\0\10\0\1\0\177\0\0\1"...,
4096}], msg_controllen=0, msg_flags=0}, 0) = 20
close(8) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff4c6f8) = -1 ENOTTY
(Inappropriate ioctl for device)
time(NULL) = 1204230691
write(3, "[Thu Feb 28 20:31:31 2008] WLM: "..., 69) = 69
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff4c738) = -1 ENOTTY
(Inappropriate ioctl for device)
time(NULL) = 1204230691
write(3, "[Thu Feb 28 20:31:31 2008] Threa"..., 84) = 84
close(7) = 0
unlink("/usr/src/packages/BUILD/dnx-0.15.1/client/testrun/dnxClient.pid") = 0
close(6) = 0
- - - - - - - - - - - - - - - - -
After analyzing this trace I can see by mapping the write(3, ...)
calls to my source code that this code is failing:
// cache our (primary?) ip address in binary and string format
if (getifaddrs(&ifa) == 0)
{
u_int setflags = IFF_UP | IFF_RUNNING;
u_int clrflags = IFF_LOOPBACK;
struct ifaddrs * ifcur = ifa;
// locate the first proper AF_NET address in our interface list
while (ifcur && (ifcur->ifa_addr->sa_family != AF_INET
|| (ifcur->ifa_flags & setflags) != setflags
|| (ifcur->ifa_flags & clrflags) != 0))
ifcur = ifcur->ifa_next;
if (ifcur)
{
// cache binary and presentation (string) versions of the ip address
iwlm->myipaddr = (unsigned long)
((struct sockaddr_in *)ifcur->ifa_addr)->sin_addr.s_addr;
inet_ntop(ifcur->ifa_addr->sa_family,
&((struct sockaddr_in *)ifcur->ifa_addr)->sin_addr,
iwlm->myipaddrstr, sizeof iwlm->myipaddrstr);
}
freeifaddrs(ifa);
}
Does this tell you guys anything? I can't really see why it would
fail. I'm just trying to get my network interface list so I can send
my identity (ip address, in string format) to the server portion of my
package. If I can't get this information, I bail. I guess it'd be a
bit more robust if I just didn't care whether or not this code worked
- I don't use the address for anything except logging, but it should
work, shouldn't it?
Thanks again (in advance) for any insight you might have,
John
On Thu, Feb 28, 2008 at 1:27 PM, Marcus Rueckert
On 2008-02-28 21:03:30 +0100, Adrian Schröter wrote:
On Thursday 28 February 2008 20:48:53 wrote John Calcote:
Hi list,
I added a unit test to my project at home:jcalcote/dnx
This project built on all flavors of SuSE linux, plus fedora, debian, and ubuntu 7 before I added this new unit test. Now it fails on all but the debian and ubuntu builds - probably because these platforms' hacked up control files don't run "make check".
I can successfully build this package locally - it just won't build on the obs servers.
Maybe due to the fact that these have no internet access ? Another possibility would be that they do not like the XEN kernel for some reason.
Please try to add an "strace" before the unit test to find out a bit more.
or maybe it requires root access due to port < 1024?
darix
-- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2008-02-28 14:25:25 -0700, John Calcote wrote:
Getting strace to work was an interesting task - first I forgot to add strace to buildrequires, then I forgot to use the -f flag to cause it to follow forks... Anyway, it works not, and I'll never forget how to do it again. :)
Here's a snippet of the log file starting from where strace logs the creation of the lock file, and ending at the removal of that file during initialization. I'm guessing that somewhere in the middle is a failed call that causes the system to shutdown while on its way up.
- - - - - - - - - - - - - - - - - open("/usr/src/packages/BUILD/dnx-0.15.1/client/testrun/dnxClient.pid", O_RDWR|O_CREAT, 0644) = 6 flock(6, LOCK_EX|LOCK_NB) = 0 write(6, "11002\n", 6) = 6 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 7 bind(7, {sa_family=AF_INET, sin_port=htons(12480), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 socket(PF_NETLINK, SOCK_RAW, 0) = 8 bind(8, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(8, {sa_family=AF_NETLINK, pid=11002, groups=00000000}, [12]) = 0 time(NULL) = 1204230691 sendto(8, "\24\0\0\0\22\0\1\3#\32\307G\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\350\0\0\0\20\0\2\0#\32\307G\372*\0\0\0\0\4\3\1\0\0\0I\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 232 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0#\32\307G\372*\0\0\0\0\0\0\1\0\0\0I\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 sendto(8, "\24\0\0\0\26\0\1\3$\32\307G\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"<\0\0\0\24\0\2\0$\32\307G\372*\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 60 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0$\32\307G\372*\0\0\0\0\0\0\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(8) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff4c6f8) = -1 ENOTTY (Inappropriate ioctl for device) time(NULL) = 1204230691 write(3, "[Thu Feb 28 20:31:31 2008] WLM: "..., 69) = 69 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff4c738) = -1 ENOTTY (Inappropriate ioctl for device) time(NULL) = 1204230691 write(3, "[Thu Feb 28 20:31:31 2008] Threa"..., 84) = 84 close(7) = 0 unlink("/usr/src/packages/BUILD/dnx-0.15.1/client/testrun/dnxClient.pid") = 0 close(6) = 0 - - - - - - - - - - - - - - - - -
strace -s 1024 to get more infos from the log messages. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 28, 2008 at 2:37 PM, Marcus Rueckert
strace -s 1024
to get more infos from the log messages.
I'm guessing this is the portion you wanted - this is where the AF_NETLINK socket is opened for access to the interface list. socket(PF_NETLINK, SOCK_RAW, 0) = 8 bind(8, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0 getsockname(8, {sa_family=AF_NETLINK, pid=1435, groups=00000000}, [12]) = 0 time(NULL) = 1204237619 sendto(8, "\24\0\0\0\22\0\1\00335\307G\0\0\0\0\0\213\365\267", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\344\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\4\3\1\0\0\0I\0\0\0\0\0\0\0\7\0\3\0lo\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\n\0\1\0\0\0\0\0\0\0\0\0\n\0\2\0\0\0\0\0\0\0\0\0\10\0\4\0004@\0\0\f\0\6\0noqueue\0`\0\7\0en\333\ven\333\v\334)%\372\334)%\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\354\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\1\0\2\0\0\0C\20\0\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0\350\3\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\24\0\0\0\0\0\0\261\0\0\0\n\0\1\0\0PV\200\26L\0\0\n\0\2\0\377\377\377\377\377\377\0\0\10\0\4\0\334\5\0\0\17\0\6\0pfifo_fast\0\0`\0\7\0O^\21\0\264\271\24\0g1\16\27z\v\345\21\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\340\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\10\3\3\0\0\0\200\0\0\0\0\0\0\0\t\0\3\0sit0\0\0\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\1\0\0\0\0\0\10\0\2\0\0\0\0\0\10\0\4\0\310\5\0\0\t\0\6\0noop\0\0\0\0`\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 688 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\00035\307G\233\5\0\0\0\0\0\0\1\0\0\0I\0\0\0\0\0\0\0\7\0\3\0lo\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\n\0\1\0\0\0\0\0\0\0\0\0\n\0\2\0\0\0\0\0\0\0\0\0\10\0\4\0004@\0\0\f\0\6\0noqueue\0`\0\7\0en\333\ven\333\v\334)%\372\334)%\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\354\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\1\0\2\0\0\0C\20\0\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0\350\3\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\24\0\0\0\0\0\0\261\0\0\0\n\0\1\0\0PV\200\26L\0\0\n\0\2\0\377\377\377\377\377\377\0\0\10\0\4\0\334\5\0\0\17\0\6\0pfifo_fast\0\0`\0\7\0O^\21\0\264\271\24\0g1\16\27z\v\345\21\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\340\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\10\3\3\0\0\0\200\0\0\0\0\0\0\0\t\0\3\0sit0\0\0\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\1\0\0\0\0\0\10\0\2\0\0\0\0\0\10\0\4\0\310\5\0\0\t\0\6\0noop\0\0\0\0`\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 sendto(8, "\24\0\0\0\26\0\1\00345\307G\0\0\0\0\0\213\365\267", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"<\0\0\0\24\0\2\00045\307G\233\5\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1\10\0\2\0\177\0\0\1\24\0\3\0lo\0\0\0\0\0\0\0\0\0\0\0\0\0\0D\0\0\0\24\0\2\00045\307G\233\5\0\0\2\30\200\0\2\0\0\0\10\0\1\0\n3% \10\0\2\0\n3% \10\0\4\0\n3%\377\24\0\3\0eth0\0\0\0\0\0\0\0\0\0\0\0\0eue\0`\0\7\0en\333\ven\333\v\334)%\372\334)%\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\354\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\1\0\2\0\0\0C\20\0\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0\350\3\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\24\0\0\0\0\0\0\261\0\0\0\n\0\1\0\0PV\200\26L\0\0\n\0\2\0\377\377\377\377\377\377\0\0\10\0\4\0\334\5\0\0\17\0\6\0pfifo_fast\0\0`\0\7\0O^\21\0\264\271\24\0g1\16\27z\v\345\21\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\340\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\10\3\3\0\0\0\200\0\0\0\0\0\0\0\t\0\3\0sit0\0\0\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\1\0\0\0\0\0\10\0\2\0\0\0\0\0\10\0\4\0\310\5\0\0\t\0\6\0noop\0\0\0\0`\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\00045\307G\233\5\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\24\0\6\0\377\377\377\377\377\377\377\377\200\21\0\0\200\21\0\0@\0\0\0\24\0\2\00045\307G\233\5\0\0\n@\200\375\2\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0\2PV\377\376\200\26L\24\0\6\0\377\377\377\377\377\377\377\377\200\21\0\0\200\21\0\0eue\0`\0\7\0en\333\ven\333\v\334)%\372\334)%\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\354\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\1\0\2\0\0\0C\20\0\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0\350\3\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\24\0\0\0\0\0\0\261\0\0\0\n\0\1\0\0PV\200\26L\0\0\n\0\2\0\377\377\377\377\377\377\0\0\10\0\4\0\334\5\0\0\17\0\6\0pfifo_fast\0\0`\0\7\0O^\21\0\264\271\24\0g1\16\27z\v\345\21\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\340\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\10\3\3\0\0\0\200\0\0\0\0\0\0\0\t\0\3\0sit0\0\0\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\1\0\0\0\0\0\10\0\2\0\0\0\0\0\10\0\4\0\310\5\0\0\t\0\6\0noop\0\0\0\0`\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128 recvmsg(8, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\00045\307G\233\5\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\24\0\6\0\377\377\377\377\377\377\377\377\200\21\0\0\200\21\0\0@\0\0\0\24\0\2\00045\307G\233\5\0\0\n@\200\375\2\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0\2PV\377\376\200\26L\24\0\6\0\377\377\377\377\377\377\377\377\200\21\0\0\200\21\0\0eue\0`\0\7\0en\333\ven\333\v\334)%\372\334)%\372\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\354\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\1\0\2\0\0\0C\20\0\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0\350\3\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\24\0\0\0\0\0\0\261\0\0\0\n\0\1\0\0PV\200\26L\0\0\n\0\2\0\377\377\377\377\377\377\0\0\10\0\4\0\334\5\0\0\17\0\6\0pfifo_fast\0\0`\0\7\0O^\21\0\264\271\24\0g1\16\27z\v\345\21\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\340\0\0\0\20\0\2\00035\307G\233\5\0\0\0\0\10\3\3\0\0\0\200\0\0\0\0\0\0\0\t\0\3\0sit0\0\0\0\0\10\0\r\0\0\0\0\0\10\0\17\0\0\0\0\0 \0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\1\0\0\0\0\0\10\0\2\0\0\0\0\0\10\0\4\0\310\5\0\0\t\0\6\0noop\0\0\0\0`\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20 close(8) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfc7ab78) = -1 ENOTTY (Inappropriate ioctl for device) time(NULL) = 1204237619 write(3, "[Thu Feb 28 15:26:59 2008] Thread pool init failed: Unknown error 4294967295.\n", 78) = 78 close(7) = 0 In my code I'm looking for interfaces that have the IFF_UP and IFF_RUNNING flags set, but have the IFF_LOOPBACK flag cleared. --john --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 28 February 2008 23:32:35 wrote John Calcote:
On Thu, Feb 28, 2008 at 2:37 PM, Marcus Rueckert
wrote: strace -s 1024
to get more infos from the log messages.
I'm guessing this is the portion you wanted - this is where the AF_NETLINK socket is opened for access to the interface list.
IIRC, we do not even have a localhost network in the XEN setups. This is most likely the reason why it failes, because it does not find a single network. We wanted to add a local network during XEN build IIRC, which would solve this problem, I think. Can you make a bugreport to setup a 127.x network during XEN build against the build script and assign it to mls@novell.com ? thanks adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, Feb 29, 2008 at 08:49:04AM +0100, Adrian Schröter wrote:
IIRC, we do not even have a localhost network in the XEN setups.
No, localhost works. Otherwise a lot of other packages would be failing. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 28, 2008 at 02:25:25PM -0700, John Calcote wrote:
After analyzing this trace I can see by mapping the write(3, ...) calls to my source code that this code is failing:
// cache our (primary?) ip address in binary and string format if (getifaddrs(&ifa) == 0) { u_int setflags = IFF_UP | IFF_RUNNING; u_int clrflags = IFF_LOOPBACK; struct ifaddrs * ifcur = ifa;
// locate the first proper AF_NET address in our interface list while (ifcur && (ifcur->ifa_addr->sa_family != AF_INET || (ifcur->ifa_flags & setflags) != setflags || (ifcur->ifa_flags & clrflags) != 0)) ifcur = ifcur->ifa_next;
if (ifcur) { // cache binary and presentation (string) versions of the ip address iwlm->myipaddr = (unsigned long) ((struct sockaddr_in *)ifcur->ifa_addr)->sin_addr.s_addr; inet_ntop(ifcur->ifa_addr->sa_family, &((struct sockaddr_in *)ifcur->ifa_addr)->sin_addr, iwlm->myipaddrstr, sizeof iwlm->myipaddrstr); } freeifaddrs(ifa); }
Does this tell you guys anything? I can't really see why it would fail.
It's looking your a network interface that is not loopback. We don't have that on the build clients, we have just a single loopback interface. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Michael Schroeder wrote:
On Thu, Feb 28, 2008 at 02:25:25PM -0700, John Calcote wrote:
After analyzing this trace I can see by mapping the write(3, ...) calls to my source code that this code is failing:
// cache our (primary?) ip address in binary and string format if (getifaddrs(&ifa) == 0) { u_int setflags = IFF_UP | IFF_RUNNING; u_int clrflags = IFF_LOOPBACK; struct ifaddrs * ifcur = ifa;
// locate the first proper AF_NET address in our interface list while (ifcur && (ifcur->ifa_addr->sa_family != AF_INET || (ifcur->ifa_flags & setflags) != setflags || (ifcur->ifa_flags & clrflags) != 0)) ifcur = ifcur->ifa_next;
if (ifcur) { // cache binary and presentation (string) versions of the ip address iwlm->myipaddr = (unsigned long) ((struct sockaddr_in *)ifcur->ifa_addr)->sin_addr.s_addr; inet_ntop(ifcur->ifa_addr->sa_family, &((struct sockaddr_in *)ifcur->ifa_addr)->sin_addr, iwlm->myipaddrstr, sizeof iwlm->myipaddrstr); } freeifaddrs(ifa); }
Does this tell you guys anything? I can't really see why it would fail.
It's looking your a network interface that is not loopback. We don't have that on the build clients, we have just a single loopback interface.
Cheers, Michael.
Hi Michael, I realize this list is probably not the place to discuss this problem, but I don't get a chance to chat with you very often. :) I'll make this my last post on this subject. I'm thinking the right change here is to use the loopback address as a fallback. Normally, I'd want to identify myself to my server by a non-local address - it's silly to send 127.0.0.1 to a server to identify yourself - unless of course that really is your only address! I think I'll change this code so that it uses the first non-loopback address it finds, but if none exists, then it falls back to the loopback address. This way it can assume that if loopback is the only one it can find, then the server must also be local. Does this sound reasonable? Also, I've been wondering about this sort of code for years. I mean, if you're multi-homed, then which non-local address do you tell your server about? Clearly the right way is to calculate the route to your server, and ensure you're telling the server that your name is some address by which he really can reach you. In this case, it doesn't matter, as we're using this address more as a human-readable indication of which machine is processing the server's data, but in a more rigorous environment, where the address might be used by the server to return information to the client, you'd want to pick an address with a proper route. Is there some example code of doing this sort of thing that you're aware of? Regards, John --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, Feb 29, 2008 at 10:07:01AM -0700, John Calcote wrote:
I'm thinking the right change here is to use the loopback address as a fallback. Normally, I'd want to identify myself to my server by a non-local address - it's silly to send 127.0.0.1 to a server to identify yourself - unless of course that really is your only address! I think I'll change this code so that it uses the first non-loopback address it finds, but if none exists, then it falls back to the loopback address. This way it can assume that if loopback is the only one it can find, then the server must also be local. Does this sound reasonable?
Yes, sounds sane to me.
Also, I've been wondering about this sort of code for years. I mean, if you're multi-homed, then which non-local address do you tell your server about?
IMHO the routing layer is the natural enemy of multi homed ;-)
Clearly the right way is to calculate the route to your server, and ensure you're telling the server that your name is some address by which he really can reach you. In this case, it doesn't matter, as we're using this address more as a human-readable indication of which machine is processing the server's data, but in a more rigorous environment, where the address might be used by the server to return information to the client, you'd want to pick an address with a proper route.
Is there some example code of doing this sort of thing that you're aware of?
Uh, first of all, I'm no network expert at all. Having said that, I think a pretty standard way is to do a DNS lookup on the hostname and use what the DNS server indicates to be canonical. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (4)
-
Adrian Schröter
-
John Calcote
-
Marcus Rueckert
-
Michael Schroeder