[opensuse-packaging] systemd and enabling services at install
Apologies for the long email ... I have updated open-iscsi in openSUSE Factory and 13.1 to work with systemd. I have a couple of questions (below), but first some background on what I'm trying to do. As part of the process of updating open-iscsi to work with systemd, I tried to take advantage of the fact that systemd can start socket-based services on demand. This means that, if set up correctly, the iscsi daemon does not need to be running until and unless it is needed. To do this, I broke the open-iscsi System V init file into 3 services, based on work being done on open-iscsi upstream: - At the lowest level is the unit iscsid.socket - The iscsid.service unit is tied to the iscsid.socket unit, and - The iscsi.service unit The iscsid.socket unit just encapsulates the socket that the iSCSI services uses. The iscsid.service unit "owns" the iscsid iSCSI daemon. The iscsi.service unit, at the top level, encapsulates iSCSI sessions, which run on top of the iscsid.service service. If all three of these new units are enabled, then you get iSCSI as it was in pre-sysytemd days, i.e. an iscsid daemon running in the background, waiting to be needed. If you instead do not enable the iscsid.service, then you get the same functionality as if it was enabled, except that the daemon is not started unless needed. Question 1: What is the best way to ensure these units get enabled correctly, i.e. iscsid.socket and iscsi.service enabled and iscsid.service not enabled? Note that the previous (pre-systemd) open-iscsi package had a file /etc/init.d/boot-iscsi.early (or something like that), which was always installed as enabled. I eliminated the need for this seperate boot-support init file. I tried to ensure that this package got installed correctly by enabling the iscsid.socket and iscsi.service units automatically upon installation, but it looks like there is an openSUSE policy to not enable any service by default. There also seems to be exceptions, but both the reasons behind the policy and the exceptions to that policy do not seem documented. Question 2: What is the objection to enabling the iscsid.socket service, since it does not add any resource burden to the system unless it is needed and used? (Same question for iscsi.service.) Advice? Answers? -- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
El 23/10/13 15:10, Lee Duncan escribió:
- At the lowest level is the unit iscsid.socket - The iscsid.service unit is tied to the iscsid.socket unit, and - The iscsi.service unit
Ok, I assume before doing that you explored what others did in different distributions and you did not invented your own way...
Question 1: What is the best way to ensure these units get enabled correctly, i.e. iscsid.socket and iscsi.service enabled and iscsid.service not enabled? I tried to ensure that this package got installed correctly by enabling the iscsid.socket and iscsi.service units automatically upon installation, but it looks like there is an openSUSE policy to not enable any service by default. There also seems to be exceptions, but both the reasons behind the policy and the exceptions to that policy do not seem documented.
You have to open a request for the services to be added to the "preset" packages.. this is the way to enable services by default. -- "If debugging is the process of removing bugs, then programming must be the process of putting them in." - Edsger Dijkstra -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/23/2013 08:38 PM, Cristian Rodríguez wrote:
El 23/10/13 15:10, Lee Duncan escribió:
- At the lowest level is the unit iscsid.socket - The iscsid.service unit is tied to the iscsid.socket unit, and - The iscsi.service unit
Ok, I assume before doing that you explored what others did in different distributions and you did not invented your own way...
Question 1: What is the best way to ensure these units get enabled correctly, i.e. iscsid.socket and iscsi.service enabled and iscsid.service not enabled? I tried to ensure that this package got installed correctly by enabling the iscsid.socket and iscsi.service units automatically upon installation, but it looks like there is an openSUSE policy to not enable any service by default. There also seems to be exceptions, but both the reasons behind the policy and the exceptions to that policy do not seem documented.
You have to open a request for the services to be added to the "preset" packages.. this is the way to enable services by default.
He did that - and that's why I send him here. Since we do not want to have every package enable every service when it gets installed, we have a whitelist of services to start. It's not clear yet to me whether we should make an exception for iscsid like he requests. I need to reread Lee's nice summary another time to chime in, Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/23/2013 11:38 PM, Cristian Rodríguez wrote:
El 23/10/13 15:10, Lee Duncan escribió:
- At the lowest level is the unit iscsid.socket - The iscsid.service unit is tied to the iscsid.socket unit, and - The iscsi.service unit
Ok, I assume before doing that you explored what others did in different distributions and you did not invented your own way...
You assume correctly.
Question 1: What is the best way to ensure these units get enabled correctly, i.e. iscsid.socket and iscsi.service enabled and iscsid.service not enabled? I tried to ensure that this package got installed correctly by enabling the iscsid.socket and iscsi.service units automatically upon installation, but it looks like there is an openSUSE policy to not enable any service by default. There also seems to be exceptions, but both the reasons behind the policy and the exceptions to that policy do not seem documented.
You have to open a request for the services to be added to the "preset" packages.. this is the way to enable services by default.
-- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
El 23/10/13 15:10, Lee Duncan escribió:
Advice? Answers?
Ok, I just looked at the patch that implements this bits, some comments... - systemd unit files are NOT and should never be marked as %config - why is LOCK_DIR defined in the command line as /etc/iscsi ? that does not look quite right.. use /run/lock/iscsi and provide a tmpfiles.d snippet with d /run/lock/iscsi 0700 root root - - about starting the service only when needed.. if the presence of an iscsi device happens to trigger a kernel event you could detect such change with udev , create an appropiate rule and use TAG+="systemd", ENV{SYSTEMD_WANTS}+="yourservice.service" to automagically start the needed stuff on demand. I have never used iscsi so I have no idea if this the case. - iscsid.service is missing an "Also=iscsid.socket" in the [Install] section -- "If debugging is the process of removing bugs, then programming must be the process of putting them in." - Edsger Dijkstra -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
В Wed, 23 Oct 2013 16:39:26 -0300 Cristian Rodríguez <crrodriguez@opensuse.org> пишет:
- iscsid.service is missing an "Also=iscsid.socket" in the [Install] section
Hmm ... why? If service is unconditionally enabled anyway, what is the use of socket unit then? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/24/2013 12:39 AM, Cristian Rodríguez wrote:
El 23/10/13 15:10, Lee Duncan escribió:
Advice? Answers?
Ok, I just looked at the patch that implements this bits, some comments...
I appreciate your time.
- systemd unit files are NOT and should never be marked as %config
I have created Bug#847953 to track this issue, as well as one other issue, below.
- why is LOCK_DIR defined in the command line as /etc/iscsi ? that does not look quite right.. use /run/lock/iscsi and provide a tmpfiles.d snippet with d /run/lock/iscsi 0700 root root -
This is not a lock directory but rather the home of the open-iscsi run-time implementation. This directory has the open-iscsi configuration file, interface definitions and configurations, as well as holding the NODE and target persistence database. The term LOCK_DIR is perhaps unfortunate but is historic.
- about starting the service only when needed.. if the presence of an iscsi device happens to trigger a kernel event you could detect such change with udev , create an appropiate rule and use TAG+="systemd", ENV{SYSTEMD_WANTS}+="yourservice.service" to automagically start the needed stuff on demand. I have never used iscsi so I have no idea if this the case.
Kay has made it clear that it is the job of systemd and not udev to handle these device events, which is much of the motiavation for this open-iscsi makeover. There is no need for this extra udev clutter if either (1) the systemd unit files are configure correctly, or (2) the iscsi daemon is just configured to run all the time, as it is now. But the "systemd way" to handle this is via socket-based on-demand starting of the iscsi daemon.
- iscsid.service is missing an "Also=iscsid.socket" in the [Install] section
This is also added to bug#847953. Thanks again for your review. -- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
El 28/10/13 17:12, Lee Duncan escribió:
This is not a lock directory but rather the home of the open-iscsi run-time implementation. This directory has the open-iscsi configuration file, interface definitions and configurations, as well as holding the NODE and target persistence database. The term LOCK_DIR is perhaps unfortunate but is historic.
It is wrong anyway.. runtime status should not be there.. i looked at the source code.. #define LOCK_FILE LOCK_DIR"/lock" #define LOCK_WRITE_FILE LOCK_DIR"/lock.write" You should not create any runtime status in /etc.. specially not locks that should be placed in a directory that is guaranteed to be clean at startup.
Kay has made it clear that it is the job of systemd and not udev to handle these device events,
which is much of the motiavation for this
open-iscsi makeover. There is no need for this extra udev clutter if either (1) the systemd unit files are configure correctly, or (2) the iscsi daemon is just configured to run all the time, as it is now.
But the "systemd way" to handle this is via socket-based on-demand starting of the iscsi daemon.
You are misunderstading my suggestion.. I never told you that udev should do that or that it will do so with my suggestion.. it simply means that udev will tell systemd that the iscsi service ought to be included in the transaction because it is wanted. -- "Any real systematist (or scientist in general) has to be ready to heave all that he or she has believed in, consider it crap, and move on, in the face of new evidence. That is how we differ from clerics." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/29/2013 02:25 AM, Cristian Rodríguez wrote:
El 28/10/13 17:12, Lee Duncan escribió:
This is not a lock directory but rather the home of the open-iscsi run-time implementation. This directory has the open-iscsi configuration file, interface definitions and configurations, as well as holding the NODE and target persistence database. The term LOCK_DIR is perhaps unfortunate but is historic.
It is wrong anyway.. runtime status should not be there.. i looked at the source code..
#define LOCK_FILE LOCK_DIR"/lock" #define LOCK_WRITE_FILE LOCK_DIR"/lock.write"
I did not say there were not any lock files there. Open-iscsi mixes lock files, config files, and database files in a directory tree. As I said, it is historical, as are many of the systems that still use lock files in /etc.
You should not create any runtime status in /etc.. specially not locks that should be placed in a directory that is guaranteed to be clean at startup.
By "you" I will assume you mean the open-iscsi package maintainers. And they do not require the directory to be clean, at start up.
Kay has made it clear that it is the job of systemd and not udev to handle these device events,
which is much of the motiavation for this
open-iscsi makeover. There is no need for this extra udev clutter if either (1) the systemd unit files are configure correctly, or (2) the iscsi daemon is just configured to run all the time, as it is now.
But the "systemd way" to handle this is via socket-based on-demand starting of the iscsi daemon.
You are misunderstading my suggestion.. I never told you that udev should do that or that it will do so with my suggestion.. it simply means that udev will tell systemd that the iscsi service ought to be included in the transaction because it is wanted.
Perhaps you misunderstand my reply. I am asking why is it neccessary to have udev tell systemd anything? The daemon has to be running for the device to show up, so by the time udev gets called there is no need to tell iscsid anything.
-- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
El 28/10/13 18:34, Lee Duncan escribió:
On 10/29/2013 02:25 AM, Cristian Rodríguez wrote:
And they do not require the directory to be clean, at start up.
Still wrong... we do have that requirement.. that's why /var/lock existed and now /run/lock does. unless we want applications failing in subtle ways of course..
The daemon has to be running for the device to show up, so by the time udev gets called there is no need to tell iscsid anything.
That was the piece of the puzzle that wasn't clear to me, if there is no kernel event *before* starting the daemon then my suggestion does not apply. -- "Any real systematist (or scientist in general) has to be ready to heave all that he or she has believed in, consider it crap, and move on, in the face of new evidence. That is how we differ from clerics." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/29/2013 02:51 AM, Cristian Rodríguez wrote:
El 28/10/13 18:34, Lee Duncan escribió:
On 10/29/2013 02:25 AM, Cristian Rodríguez wrote:
And they do not require the directory to be clean, at start up.
Still wrong...
I find "wrong" a bit strong, as it is working as designed, and the system around it has changed. I would agree it should be updated, but it certainly is not currently breaking anything. But please feel free to file a bug on it, and assign it to me (lduncan at suse dot com).
we do have that requirement.. that's why /var/lock existed and now /run/lock does. unless we want applications failing in subtle ways of course..
I understand it is an issue, but one unrelated to systemd integration.
The daemon has to be running for the device to show up, so by the time udev gets called there is no need to tell iscsid anything.
That was the piece of the puzzle that wasn't clear to me, if there is no kernel event *before* starting the daemon then my suggestion does not apply.
-- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Cristian Rodríguez wrote:
El 23/10/13 15:10, Lee Duncan escribió:
Advice? Answers?
Ok, I just looked at the patch that implements this bits, some comments...
- systemd unit files are NOT and should never be marked as %config
- why is LOCK_DIR defined in the command line as /etc/iscsi ? that does not look quite right.. use /run/lock/iscsi and provide a tmpfiles.d snippet with d /run/lock/iscsi 0700 root root -
Don't. Simple rule, never use /run/lock (or /var/lock). If at all it's managed by lockdev. Never use it for anything else. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Lee, I need some more background: This is the client side of iscsi, you want to enable, correct? Client-side means, you have a server/SAN that provides iscsi storage and your local machine will access that remote server/SAN. So, do I need to edit anything to access the remote storage? Or is this done via command-line tools delivered with the package that need the service enabled? Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
I've been doing a lot of this recently so I'll add some commentary. To access the iscsi server/san you need to... - configure IP networking for the iscsi devices - enable/start iscsid.service - add software iscsi interfaces - add ethernet interface names to iscsi interfaces - discover iscsi targets - login to iscsi targets If you're using multipathing you need to configure it before any iscsi, and if those multiple targets are on the same IP Subnet, such as Dell Equallogic and FusionIO/NexGenStorage ioControl SAN's, you need to disable FW_KERNEL_SECURITY in the firewall and prevent arp flux and reverse path filtering before multipathing. -- Later, Darin On Fri, Oct 25, 2013 at 5:43 AM, Andreas Jaeger <aj@suse.com> wrote:
Lee, I need some more background:
This is the client side of iscsi, you want to enable, correct? Client-side means, you have a server/SAN that provides iscsi storage and your local machine will access that remote server/SAN.
So, do I need to edit anything to access the remote storage? Or is this done via command-line tools delivered with the package that need the service enabled?
Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/25/2013 02:43 PM, Andreas Jaeger wrote:
Lee, I need some more background:
This is the client side of iscsi, you want to enable, correct?
Yes, correct.
Client-side means, you have a server/SAN that provides iscsi storage and your local machine will access that remote server/SAN.
More likely you have an ethernet network and some hard or soft clients. I think of a SAN as separate from the other network traffic, and usually FC (though of course not always).
So, do I need to edit anything to access the remote storage? Or is this done via command-line tools delivered with the package that need the service enabled?
After installing open-iscsi, if it is configured correctly, i.e. either set up for socket-based activation (as I wish to do with systemd), or with the daemon already running, here are the steps you might use to get access to a remote device: root# iscsiadm -m discovery -t st -p 192.168.20.2 -l 192.168.20.2:3260,1 iqn.2001-04.net.gonzoleeman:test.disk.laptop.001 Logging in to [iface: default, target: iqn.2001-04.net.gonzoleeman:test.disk.laptop.001, portal: 192.168.20.2,3260] (multiple) Login to [iface: default, target: iqn.2001-04.net.gonzoleeman:test.disk.laptop.001, portal: 192.168.20.2,3260] successful. In other words, I have asked the iscsiadm program to discover targets at 192.168.20.2, and try to login to any targets it finds. If finds one target and succeeds at logging into it. After this step, a new device appears, /dev/sdb in my case. If I happened to have a partition from /dev/sdb listed in /etc/fstab as containing a filesystem and mark it appropriately then systemd will take care of mounting it for me. As part of this process it builds a target and node database, so that next time I don't need to rediscover targets unless I want to.
Andreas
-- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
I'm sorry, I forgot one aspect of the answer to your question Andreas, so I added it, below, at the bottom. On 10/29/2013 02:27 AM, Lee Duncan wrote:
On 10/25/2013 02:43 PM, Andreas Jaeger wrote:
Lee, I need some more background:
This is the client side of iscsi, you want to enable, correct?
Yes, correct.
Client-side means, you have a server/SAN that provides iscsi storage and your local machine will access that remote server/SAN.
More likely you have an ethernet network and some hard or soft clients. I think of a SAN as separate from the other network traffic, and usually FC (though of course not always).
So, do I need to edit anything to access the remote storage? Or is this done via command-line tools delivered with the package that need the service enabled?
After installing open-iscsi, if it is configured correctly, i.e. either set up for socket-based activation (as I wish to do with systemd), or with the daemon already running, here are the steps you might use to get access to a remote device:
root# iscsiadm -m discovery -t st -p 192.168.20.2 -l 192.168.20.2:3260,1 iqn.2001-04.net.gonzoleeman:test.disk.laptop.001 Logging in to [iface: default, target: iqn.2001-04.net.gonzoleeman:test.disk.laptop.001, portal: 192.168.20.2,3260] (multiple) Login to [iface: default, target: iqn.2001-04.net.gonzoleeman:test.disk.laptop.001, portal: 192.168.20.2,3260] successful.
In other words, I have asked the iscsiadm program to discover targets at 192.168.20.2, and try to login to any targets it finds.
If finds one target and succeeds at logging into it. After this step, a new device appears, /dev/sdb in my case. If I happened to have a partition from /dev/sdb listed in /etc/fstab as containing a filesystem and mark it appropriately then systemd will take care of mounting it for me.
As part of this process it builds a target and node database, so that next time I don't need to rediscover targets unless I want to.
Note: this example is only using the iscsid.socket and iscsid.serivce systemd units. I also created (taken from open-iscsi upstream) an "iscsi.service" unit. This higher-level unit is based on iscsid.service, and provices iscsi sessions as a service. So starting this unit logs into any iSCSI nodes you have marked appropriately in your iscsi node database, and stopping this service logs out of those targets. This optional higher-level service, if used, automates creating and destroying sessions on your favorite iSCSI targets, so that you don't have to manually log into and out of said targets unless you want to. -- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Lee, thanks for the lengthy explanation. I'll withdraw my objections, I think it makes sense to enable the services as you proposed and will approve your submission now, Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/29/2013 02:48 PM, Andreas Jaeger wrote:
Lee, thanks for the lengthy explanation. I'll withdraw my objections, I think it makes sense to enable the services as you proposed and will approve your submission now,
Andreas
Thank you Andreas. I believe the end user will be more pleased with the resulting system. -- Lee Duncan SUSE Labs -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Lee Duncan wrote:
I tried to ensure that this package got installed correctly by enabling the iscsid.socket and iscsi.service units automatically upon installation, but it looks like there is an openSUSE policy to not enable any service by default. There also seems to be exceptions, but both the reasons behind the policy and the exceptions to that policy do not seem documented.
Question 2: What is the objection to enabling the iscsid.socket service, since it does not add any resource burden to the system unless it is needed and used? (Same question for iscsi.service.)
As usual in so many places there is no written rule here I think. It's just that packages follow a common pattern for consistency. In case of services that is: - install it - configure it - start it - enable it if it should be available on next reboot as well Just because a service is installed or has been run once for testing doesn't mean it has to be kept running always. Also, if a service requires any kind of configuration there's no point of having it enabled by default. If a service allows users != root to access the socket, security should have a look at it as this could be an entry point for code execution resp privilege escalation just like an open port. It doesn't matter for any of the above whether the service is socket activated or actually running. The existing exeptions are either because those services are basically mandatory on any system (e.g. nscd), are expected to work in the default installation without configuration (printing, dns service discovery) or are legacy and noone dared to switch them off yet (postfix). So is there a good reason why iscsid should be treated differently? cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (6)
-
Andreas Jaeger -
Andrey Borzenkov -
Cristian Rodríguez -
Darin Perusich -
Lee Duncan -
Ludwig Nussel