[opensuse] System slowdowns
Hi - I am running an OpenSuSE Leap 42.2 system on an x64 system with a dual core processor. While using it I am experiencing periods of up to several minutes where I can hear the processor's cooling fan speed up and the system's responsiveness slows way down to where it is unusable. During these episodes I have had KSysguard running and top executing in a shell to see if I can figure out what is causing these episodes. The KSysguard graphical display will show that both processors are running at or near 100% yet the process table and top both show that the running processes are taking only a few percent (typically less than 4 or 5% for the worst process) and the total percentage of CPU time being used for all processes is nowhere near the 100% shown by the cpu usage graph. So how do I track down what application, process, or system service is chewing up my processors. I am experiencing these slowdowns approx 5 to 10 times per hour which is quite frustrating. I kinda suspect it is Firefox but I cannot say that definitively.. Suggestions? Marc.. -- "The Truth is out there" - Spooky -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On October 2, 2016 8:34:52 PM PDT, Marc Chamberlin <marc@marcchamberlin.com> wrote:
Hi - I am running an OpenSuSE Leap 42.2 system on an x64 system with a
dual core processor. While using it I am experiencing periods of up to several minutes where I can hear the processor's cooling fan speed up and the system's responsiveness slows way down to where it is unusable.
During these episodes I have had KSysguard running and top executing in
a shell to see if I can figure out what is causing these episodes. The KSysguard graphical display will show that both processors are running at or near 100% yet the process table and top both show that the running processes are taking only a few percent (typically less than 4 or 5% for the worst process) and the total percentage of CPU time being used for all processes is nowhere near the 100% shown by the cpu usage graph.
So how do I track down what application, process, or system service is chewing up my processors. I am experiencing these slowdowns approx 5 to
10 times per hour which is quite frustrating. I kinda suspect it is Firefox but I cannot say that definitively.. Suggestions?
Marc..
Was Ksystemguard set to show ALL processes. Or just your processes? -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/2/2016 8:41 PM, John Andersen wrote:
On October 2, 2016 8:34:52 PM PDT, Marc Chamberlin <marc@marcchamberlin.com> wrote:
Hi - I am running an OpenSuSE Leap 42.2 system on an x64 system with a
dual core processor. While using it I am experiencing periods of up to several minutes where I can hear the processor's cooling fan speed up and the system's responsiveness slows way down to where it is unusable.
During these episodes I have had KSysguard running and top executing in
a shell to see if I can figure out what is causing these episodes. The KSysguard graphical display will show that both processors are running at or near 100% yet the process table and top both show that the running processes are taking only a few percent (typically less than 4 or 5% for the worst process) and the total percentage of CPU time being used for all processes is nowhere near the 100% shown by the cpu usage graph.
So how do I track down what application, process, or system service is chewing up my processors. I am experiencing these slowdowns approx 5 to
10 times per hour which is quite frustrating. I kinda suspect it is Firefox but I cannot say that definitively.. Suggestions?
Marc.. Was Ksystemguard set to show ALL processes. Or just your processes? Sorry, ALL processes... Marc...
-- "The Truth is out there" - Spooky -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am Sun, 2 Oct 2016 21:05:01 -0700 schrieb Marc Chamberlin <marc@marcchamberlin.com>:
10 times per hour which is quite frustrating. I kinda suspect it is Firefox but I cannot say that definitively.. Suggestions?
Marc.. Was Ksystemguard set to show ALL processes. Or just your processes? Sorry, ALL processes... Marc...
Hi Marc, after last Kde update (but Leap 42.1) I have a similar effect of system slowdown. Unfortunately I can't say exactly what was updated. atop shows heavily activity on disk sdb (101%), which holds /home and other user data, as well as /tmp, /var and swap partitions, all ext4, when I force a file look up, like changing the mail folder in claws mail, as example. I've checked the partitions except /var and swap without errors. I'm unable to unmount /var and swap in init 1. But, I've found that, if I change from kde4/qt5/plasma desktop to xfce desktop the heavily disk activity doesn't raise up and shows typically around 0 to 5%. Are you able to change the desktop and check then the disk activity on your system? -- Mit freundlichen Grüßen Kind Regards Peter Ragosch -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/10/2016 05:34, Marc Chamberlin wrote:
Hi - I am running an OpenSuSE Leap 42.2 system on an x64 system with a dual core processor. While using it I am experiencing periods of up to several minutes where I can hear the processor's cooling fan speed up and the system's responsiveness slows way down to where it is unusable. During these episodes I have had KSysguard running and top executing in a shell to see if I can figure out what is causing these episodes. The KSysguard graphical display will show that both processors are running at or near 100% yet the process table and top both show that the running processes are taking only a few percent (typically less than 4 or 5% for the worst process) and the total percentage of CPU time being used for all processes is nowhere near the 100% shown by the cpu usage graph.
So how do I track down what application, process, or system service is chewing up my processors. I am experiencing these slowdowns approx 5 to 10 times per hour which is quite frustrating. I kinda suspect it is Firefox but I cannot say that definitively.. Suggestions?
Marc..
Try using atop, I use it to monitor build processes. It's not in the main distribution, I branched it from server:montoring in obs. You can find it via one click install or http://download.opensuse.org/repositories/home:/plater/Leap_42.2 . Regards Dave P -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-10-03 07:55, Dave Plater wrote:
Try using atop, I use it to monitor build processes. It's not in the main distribution, I branched it from server:montoring in obs. You can find it via one click install or http://download.opensuse.org/repositories/home:/plater/Leap_42.2 .
And run it in a terminal after doing "su -". Anyway, I don't see why "top" would not display the process using most of the cpu, though, even if used as user. Unless not sorting by default by cpu power. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On October 3, 2016 4:22:55 AM PDT, "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 2016-10-03 07:55, Dave Plater wrote:
Try using atop, I use it to monitor build processes. It's not in the main distribution, I branched it from server:montoring in obs. You can find it via one click install or http://download.opensuse.org/repositories/home:/plater/Leap_42.2 .
And run it in a terminal after doing "su -".
Anyway, I don't see why "top" would not display the process using most of the cpu, though, even if used as user. Unless not sorting by default by cpu power.
Still, why is George getting conflicting readings from Ksystemguard? Could it be that it is being blocked by whatever is using all the resources, such that it never actually sees the culprit in the list? Stuff happening in the kernel never shows up in top or Ksystemguard. I would ask if the disk drive light was on solid during the slow down. A failing drive sometimes acts like a super busy system, but doesn't show up in top. Maybe it's Time to run some S.M.A.R.T. commands against the disk, and if it has exhausted all of its spare tracks and blocks get it replaced. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-10-03 18:38, John Andersen wrote:
On October 3, 2016 4:22:55 AM PDT, "Carlos E. R." <> wrote:
Still, why is George getting conflicting readings from Ksystemguard?
Dunno. I prefer output from top or atop.
Could it be that it is being blocked by whatever is using all the resources, such that it never actually sees the culprit in the list?
I don't think so...
Stuff happening in the kernel never shows up in top or Ksystemguard.
That maybe.
I would ask if the disk drive light was on solid during the slow down.
A failing drive sometimes acts like a super busy system, but doesn't show up in top.
No, because in that case it is the processes waiting for operations on the disk to complete, and they take eons. CPU usage is about nil, maybe the disk firmware is busy. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 10/3/2016 7:16 PM, Carlos E. R. wrote:
On 2016-10-03 18:38, John Andersen wrote:
On October 3, 2016 4:22:55 AM PDT, "Carlos E. R." <> wrote:
Still, why is George getting conflicting readings from Ksystemguard? Dunno. I prefer output from top or atop.
Could it be that it is being blocked by whatever is using all the resources, such that it never actually sees the culprit in the list? I don't think so...
Stuff happening in the kernel never shows up in top or Ksystemguard. That maybe.
I would ask if the disk drive light was on solid during the slow down.
A failing drive sometimes acts like a super busy system, but doesn't show up in top. No, because in that case it is the processes waiting for operations on the disk to complete, and they take eons. CPU usage is about nil, maybe the disk firmware is busy.
Thanks John, Carlos for your thoughts... I ran the smartctl checks against my disk drives and yeah one of the drives is showing some failures although it's overall summary reports that the drive is healthy. I noted that my swap partition was on the drive that is showing errors, so decided to move the swap partition to a different drive to see if that would help. It didn't. I will go ahead and replace this drive just to be safe and report back if that indeed improves things... However, for my two cents worth, speaking as a computer scientist/engineer myself, I would agree with Carlos. Processes are suspend when they are waiting for I/O operations to complete. That is normal OS behavior. So if a disk drive is having a hard time reading or writing, then that would just result in the process being suspended for a longer period of time, while the drive (using it's own internal processor) figures out how to solve the problem it is having. And a suspended process is not using CPU cycles. So me thinks that disk drive errors should not result in high CPU usage, like I am seeing. Something is periodically and definitely using up a LOT of CPU time and top, atop, and ksystemguard are not reporting why. Although ksystemguard will graphically show that both of my CPU cores are running at or near 100% usage. And the total amount of CPU time being reported for each of the individual running processes does not add up to anything near 100% usage. Typically I see less than 15% usage total. Once in awhile I will see the total of the systemd and/or kdeinit5 processes climb up higher, during these episodes of 100% CPU usage, but again nowhere near 100%. I can believe it is something in the kernel itself, which these monitoring tools may not know how to report on. Or perhaps my system has been compromised by one heck of a smart virus or trojan that is able to hide it's activities, who knows. I do use it for a number of servers and it is exposed to the internet 24/7. I am not enough of an expert on the internals of the Linux kernel to know how to diagnose it or what tools I should/could use to figure out what it is doing. One other symptom that is interesting is that the display can become unresponsive to mouse/cursor movements/clicks as well. I don't know whether Linux handles mouse events by interrupt handlers or polls the mouse, but either way I would expect mouse events to be handled at a fairly high priority. The fact that the display becomes unresponsive to these mouse events seems to further point towards the idea that something going on within the kernel/OS itself. Marc.... -- "The Truth is out there" - Spooky -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-10-04 20:25, Marc Chamberlin wrote:
On 10/3/2016 7:16 PM, Carlos E. R. wrote:
Thanks John, Carlos for your thoughts... I ran the smartctl checks against my disk drives and yeah one of the drives is showing some failures although it's overall summary reports that the drive is healthy. I noted that my swap partition was on the drive that is showing errors, so decided to move the swap partition to a different drive to see if that would help. It didn't. I will go ahead and replace this drive just to be safe and report back if that indeed improves things...
Recently one of my disks developed an error. I figured, from smartctl, that the affected sectors were in the swap partition. So I just did swapoff on that one, wrote it entirely with zeroes, and run smartctl again. Clear. mkswap and activate it again. No issues. Watching it to see if the number of bad sectors increase, then consider replacing it.
However, for my two cents worth, speaking as a computer scientist/engineer myself, I would agree with Carlos. Processes are suspend when they are waiting for I/O operations to complete. That is normal OS behavior. So if a disk drive is having a hard time reading or writing, then that would just result in the process being suspended for a longer period of time, while the drive (using it's own internal processor) figures out how to solve the problem it is having. And a suspended process is not using CPU cycles. So me thinks that disk drive errors should not result in high CPU usage, like I am seeing.
Right.
Something is periodically and definitely using up a LOT of CPU time and top, atop, and ksystemguard are not reporting why. Although ksystemguard will graphically show that both of my CPU cores are running at or near 100% usage. And the total amount of CPU time being reported for each of the individual running processes does not add up to anything near 100% usage. Typically I see less than 15% usage total. Once in awhile I will see the total of the systemd and/or kdeinit5 processes climb up higher, during these episodes of 100% CPU usage, but again nowhere near 100%.
There is a toggle in top to show other threads, I think kernel threads. Wait a minute [...] Yes, I think it is "H".
I can believe it is something in the kernel itself, which these monitoring tools may not know how to report on. Or perhaps my system has been compromised by one heck of a smart virus or trojan that is able to hide it's activities, who knows. I do use it for a number of servers and it is exposed to the internet 24/7. I am not enough of an expert on the internals of the Linux kernel to know how to diagnose it or what tools I should/could use to figure out what it is doing.
One other symptom that is interesting is that the display can become unresponsive to mouse/cursor movements/clicks as well. I don't know whether Linux handles mouse events by interrupt handlers or polls the mouse, but either way I would expect mouse events to be handled at a fairly high priority. The fact that the display becomes unresponsive to these mouse events seems to further point towards the idea that something going on within the kernel/OS itself.
I had a curious issue several years ago. The system was very very busy. It turned out that it was process number 1, init. The culprit was my modem: unplug it and zero cpu. I rebooted my modem, problem solved. Apparently my modem was triggering many interrupts in the serial port, and again apparently they were handled by init. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 10/04/2016 12:22 PM, Carlos E. R. wrote:
There is a toggle in top to show other threads, I think kernel threads. Wait a minute [...] Yes, I think it is "H".
Man top says:
H :Threads-mode toggle When this toggle is On, individual threads will be displayed for all processes in all visible task windows. Otherwise, top displays a summation of all threads in each process.
It says nothing about kernel threads, and everything it lists has a pid, so not a kernel thread. I don't believe normal userland tools can monitor the kernel CPU usage. But I do agree that it sound like something in the kernel is probably the culprit here. The mouse problems seem worth investigating. Maybe dig in the junk drawer for a totally different kind of mouse (serial, bluetooth, usb-wired) and see if that changes anything. -- After all is said and done, more is said than done.
John Andersen wrote:
On 10/04/2016 12:22 PM, Carlos E. R. wrote:
There is a toggle in top to show other threads, I think kernel threads. Wait a minute [...] Yes, I think it is "H". It says nothing about kernel threads, and everything it lists has a pid, so not a kernel thread.
All threads on linux have pids as well, as the linux thread model is based on processes. BTW, kernel processes in "top" are shown in brackets.
I don't believe normal userland tools can monitor the kernel CPU usage.
Try "htop" (in package of the same name). It allows you to separately toggle display on/off of both user and kernel threads Press "H" to toggle user threads, and "K" to toggle kernel threads. It runs in a terminal window -- pick one that supports color (do any not, these days?). You can configure it how you like it and save the result (much like 'top'). Htop understand keyboard and mouse input and uses curses to display things. You'll want to stretch the window out horizontally. Also you'll want to make sure your terminal is in Unicode mode (unicode_start) if it isn't already, so the graphics characters in htop display correctly. Linda -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/4/2016 1:38 PM, Linda Walsh wrote:
John Andersen wrote:
There is a toggle in top to show other threads, I think kernel threads. Wait a minute [...] Yes, I think it is "H". It says nothing about kernel threads, and everything it lists has a
On 10/04/2016 12:22 PM, Carlos E. R. wrote: pid, so not a kernel thread.
All threads on linux have pids as well, as the linux thread model is based on processes. BTW, kernel processes in "top" are shown in brackets.
I don't believe normal userland tools can monitor the kernel CPU usage.
Try "htop" (in package of the same name). It allows you to separately toggle display on/off of both user and kernel threads
Press "H" to toggle user threads, and "K" to toggle kernel threads. It runs in a terminal window -- pick one that supports color (do any not, these days?). You can configure it how you like it and save the result (much like 'top'). Htop understand keyboard and mouse input and uses curses to display things. You'll want to stretch the window out horizontally. Also you'll want to make sure your terminal is in Unicode mode (unicode_start) if it isn't already, so the graphics characters in htop display correctly.
Linda
Hi Linda and thanks for the suggestion. I tried htop also, and no joy with it either. Though I am not sure what the "H" and "K" keys do when I use them. It appears that what they do is simply force an update, but I don't see any change in the output that differentiates between user and kernel threads. Marc.... -- "The Truth is out there" - Spooky -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-10-05 02:27, Marc Chamberlin wrote:
Hi Linda and thanks for the suggestion. I tried htop also, and no joy with it either. Though I am not sure what the "H" and "K" keys do when I use them. It appears that what they do is simply force an update, but I don't see any change in the output that differentiates between user and kernel threads.
Same here. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Marc Chamberlin wrote:
Hi Linda and thanks for the suggestion. I tried htop also, and no joy with it either. Though I am not sure what the "H" and "K" keys do when I use them. It appears that what they do is simply force an update, but I don't see any change in the output that differentiates between user and kernel threads.
Then something is broken on your machine. There is an option in Setup (F2) to change the color of threads, but it appears it is for all threads. But the system threads that disappear are not on the first screen (use the PageUp/PageDn keys to scroll pages, PageUp key to scroll up) If you want a solution to your problem, You'll be able to see what threads disappear or not, in this tools If I press "H", I see threads from transmission appearing/disappearing. You can see the effect (configuration-wise) by looking at "Setup and Display Options and see if Hide userland and/or hide kernel threads options have an 'x' next to them. If you have kernel-threads displayed, you would see them (I count 51 different types, 308 kernel threads total): bioset cpuhp crypto deferwq dio dm-thin dm_bufio_cache edac-poller ixgbe kauditd kblockd kcompactd0 kcompactd1 kcopyd kdevtmpfs kdmflush khugepaged khungtaskd kintegrityd kipmi0 kpsmoused ksmd ksoftirqd kswapd0 kswapd1 kthreadd kthrotld kvm-irqfd-clean kworker lru-add-drain migration netns oom_reaper rcu_bh rcu_preempt rcu_sched scsi_eh_0 scsi_eh_1 scsi_tmf_0 scsi_tmf_1 vmstat watchdog watchdogd writeback xfs-buf xfs-cil xfs-conv xfs-data xfs-eofblocks xfs-log xfs-reclaim xfs_mru_cache xfsaild xfsalloc ------ Those are the 1st word with parts after '/' chopped off -- still got a few dups w/scsi using '_', and a few others not using a separator char like kswap & kcompact -- but those are likely due to my machine having a NUMA memory layout associated with having 2 physical cpu-chips. Many of the threads come from needing a separate thread for each "Core", while others come from needing a separate thread for each block device you have a file system on. You can also see the kernel threads in "ps" using something like: ps -efl The kernel threads have square brackets '[]' around the process name (having brackets in the arguments section doesn't count). But some xfs/block dev threads: [xfsaild/dm-0] [xfsaild/dm-12] [xfsaild/dm-14] [xfsaild/dm-1] [xfsaild/dm-2] [xfsaild/dm-3] [xfsaild/dm-4] [xfsaild/dm-5] [xfsaild/dm-6] [xfsaild/sdc1] [xfsaild/sdc2] [xfsaild/sdc3] [xfsaild/sdc6] [xfsaild/sdc8] Some per-cpu threads: [ksoftirqd/0] [ksoftirqd/10] [ksoftirqd/11] [ksoftirqd/1] [ksoftirqd/2] [ksoftirqd/3] [ksoftirqd/4] [ksoftirqd/5] [ksoftirqd/6] [ksoftirqd/7] [ksoftirqd/8] [ksoftirqd/9] In "top" (and htop) the threads don't show the brackets -- but both tools show threads -- user and system. Another way of looking at these -- if you see a bunch of duplicate looking processes, ask if they correspond to a program you are running -- usually they won't. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/4/2016 1:22 PM, John Andersen wrote:
On 10/04/2016 12:22 PM, Carlos E. R. wrote:
There is a toggle in top to show other threads, I think kernel threads. Wait a minute [...] Yes, I think it is "H". Man top says: H :Threads-mode toggle When this toggle is On, individual threads will be displayed for all processes in all visible task windows. Otherwise, top displays a summation of all threads in each process.
It says nothing about kernel threads, and everything it lists has a pid, so not a kernel thread.
I don't believe normal userland tools can monitor the kernel CPU usage.
But I do agree that it sound like something in the kernel is probably the culprit here.
The mouse problems seem worth investigating. Maybe dig in the junk drawer for a totally different kind of mouse (serial, bluetooth, usb-wired) and see if that changes anything.
Thanks John for your suggestion. I tried a different mouse and no joy... Marc.... -- "The Truth is out there" - Spooky -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/04/2016 05:29 PM, Marc Chamberlin wrote:
The mouse problems seem worth investigating. Maybe dig in the junk drawer for a totally different kind of mouse (serial, bluetooth, usb-wired) and see if that changes anything.
Thanks John for your suggestion. I tried a different mouse and no joy... Marc....
Remind me again what kernel you are using, and what version of Vbox? I tend to purge my list mail fairly aggressivly. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/4/2016 6:04 PM, John Andersen wrote:
On 10/04/2016 05:29 PM, Marc Chamberlin wrote:
The mouse problems seem worth investigating. Maybe dig in the junk drawer for a totally different kind of mouse (serial, bluetooth, usb-wired) and see if that changes anything.
Thanks John for your suggestion. I tried a different mouse and no joy... Marc....
Remind me again what kernel you are using, and what version of Vbox? I tend to purge my list mail fairly aggressivly. Hi John - I am running Leap 42.1 (x86_64) I am not running VBox, (which I think what you are referring to is VirtualBox?) Interesting that you should ask though because I am playing around with Docker. However, I don't think my system slowdowns has anything to do with Docker as these slowdown issues preceded my installation of Docker. And these slowdowns occur regardless of whether Docker is running or not. (The only reason I am fooling around with Docker, FYI, is so I can get the latest version of the Apache James email server up and running, something that OpenSuSE does not directly support.)
Marc.... -- "The Truth is out there" - Spooky -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/10/2016 13:22, Carlos E. R. wrote:
On 2016-10-03 07:55, Dave Plater wrote:
Try using atop, I use it to monitor build processes. It's not in the main distribution, I branched it from server:montoring in obs. You can find it via one click install or http://download.opensuse.org/repositories/home:/plater/Leap_42.2 . And run it in a terminal after doing "su -".
Anyway, I don't see why "top" would not display the process using most of the cpu, though, even if used as user. Unless not sorting by default by cpu power.
I have several sata cables and one that actually works in my present box. Two identical sata drives, used for swap at one time, developed unfixable bad sectors. I had to use hdparm to recover them. The disks continually give ata reset messages in "sudo journalctl -f" tailed log and slow the system down noticably. I created a swap file for now and my system is fine until I trigger a memory swap to disk. Go and buy a new sata cable and "smartctl -t long /sd?"" if it completes without error you can rule out disk hardware 90% of the time. Dave P -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (6)
-
Carlos E. R. -
Dave Plater -
John Andersen -
Linda Walsh -
Marc Chamberlin -
Peter Ragosch