[opensuse] systemload / cpu's
Perhaps a silly question... I noticed on one of my machines, one of the cores was permanently 100% busy, while others were stile idle. (two dual-cores) This i noticed both for openvpn and rsync. For both my cpu is apperently the bottleneck, not my network. Aren't the processes supposed to be spread even on the different core's? hw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-03-20 16:47, Hans Witvliet wrote:
Aren't the processes supposed to be spread even on the different core's?
No. Not till developers learn how to do that, one of these years >:-) - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkuk9wwACgkQU92UU+smfQXirgCfUwlLBgS9s/xqn4JJVvjXzt+8 mjcAn1f6eaOvFDWOyKHCSsqf1wvKzuSj =DSM3 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. said the following on 03/20/2010 12:25 PM:
On 2010-03-20 16:47, Hans Witvliet wrote:
Aren't the processes supposed to be spread even on the different core's?
No.
Not till developers learn how to do that, one of these years >:-)
Aye, and with that comes a whole new raft of hazards and problems, including many to do with the various aspects of security.- -- Those of us in security are very much like heart doctors -- cardiologists. Our patients know that lack of exercise, too much dietary fat, and smoking are all bad for them. But they will continue to smoke, and eat fried foods, and practice being couch potatoes until they have their infarction. Then they want a magic pill to make them better all at once, without the effort. And by the way, they claim loudly that their condition really isn't their fault -- it was genetics, or the tobacco companies, or McDonalds that was to blame. And they blame us for not taking better care of them. Does this sound familiar? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sat, 2010-03-20 at 12:43 -0400, Anton Aylward wrote:
Aye, and with that comes a whole new raft of hazards and problems, including many to do with the various aspects of security.-
Security aspects of SMP? Apart from certain race condition exploits possibly being easier to achieve, I have no idea what that would be. Could you expand a little on what you mean? Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Anders Johansson said the following on 03/20/2010 02:54 PM:
On Sat, 2010-03-20 at 12:43 -0400, Anton Aylward wrote:
Aye, and with that comes a whole new raft of hazards and problems, including many to do with the various aspects of security.-
Security aspects of SMP?
Apart from certain race condition exploits possibly being easier to achieve, I have no idea what that would be. Could you expand a little on what you mean?
Right now its open-ended. Security isn't *just* about keeping hackers out. There are issues of integrity and availability to consider, never mind the Parkerian Hexad. Per has already mentioned that programmers are unfamiliar with the kind of multi-threading required for effective and efficient use of multi-core - arbitrary numbers not just dual core - programming. We can expect error and we cant be sure what those errors will be. While multi-threading is not new, few programmers have been forced to use it for real in their own code. Very few programmers have experience with *true* multiprocessing until now. With this new programming paradigm the average programmer is confronted with the need to understand much more and write code that is cleaner to a much higher degree than ever before. Very likely there will be pressure to make more applications multi-threaded, lest they'll be too slow to be competitive. It's also *much* harder to test and debug multi-threaded code. The languages we use, most notably C and C++, don't provide for much in the way of error avoidance or detection of interlock, contention and race conditions, either. The bugs are also going to be much more dependent on machine environment, other software and so on, making reproduction of issues incredibly difficult. Due to the multi-cores, we're probably at the threshold of an era with an entirely new set of program bugs. Classic statistical testing and even execution path analysis will be less effective that ever. We will need new design and programming paradigms, tools and debugging tools. I don't see them yet. -- "Each new law makes only a single guarantee. It will create new criminals." -- John Tandervold -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Anton Aylward wrote:
Due to the multi-cores, we're probably at the threshold of an era with an entirely new set of program bugs.
Multiple execution units are nothing new - they date back at least 20 years. I started writing multi-threaded code for IBM 3090s around 89/90. The multi execution unit boxes have been perfectly common in servers since then.
We will need new design and programming paradigms, tools and debugging tools. I don't see them yet.
Apples Grand Central Dispatch ? -- Per Jessen, Zürich (0.0°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Saturday 20 March 2010 02:00:26 pm Per Jessen wrote:
Anton Aylward wrote:
Due to the multi-cores, we're probably at the threshold of an era with an entirely new set of program bugs.
Multiple execution units are nothing new - they date back at least 20 years. I started writing multi-threaded code for IBM 3090s around 89/90. The multi execution unit boxes have been perfectly common in servers since then.
We will need new design and programming paradigms, tools and debugging tools. I don't see them yet.
Apples Grand Central Dispatch ? Actually goes back further than that. The old Xerox Sigma 9's and 8's ran multiple OS's, with while executing look ahead for commands. The memory and I/O processors were multi ported in the late 1960's. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
upscope said the following on 03/20/2010 05:49 PM:
Actually goes back further than that. The old Xerox Sigma 9's and 8's ran multiple OS's, with while executing look ahead for commands. The memory and I/O processors were multi ported in the late 1960's.
I worked with a GE machine that was designed for real time control (signalling, machinery, for example rail-road switching) that had FIVE ported memory/backplane. The granularity of memory access was 512 bytes. So one of the memory IO channels could be locked to the disk DMA controller and another to the tape controller, and since they were autonomous data units that took command blocks from memory they didn't need CPU intervention once those blocks were set up. This multi-channel memory is one reason why those old machines were so fast. Not only was the IO "autonomous" (not requiring CPU intervention) but it did not block CPU access to memory. And yes we've returned there. In various ways. But there's a problem Granularity of current access is limited by how big the chips are (until you use chips that migrate this down to the chip level). Sadly things are not always as simple as they seem. SMP kernel code for '*just*' two processors is pretty simple; doing it or N processors without spin locks isn't. I takes some redesign, as we've seen over the years in Linux. The same applies with threading. On the big IBM machines, the Xerox and the GE machines, while the principles were the same, the details were different. For example, I once wrote a striping disk controller for a GE machine where FOUR hardware controllers (each with a different spindle) each took over a memory channel simultaneously and **BLURP** that 4K segment hit the disk(s) damn fast and then unlocked the channels. It was tied to that very specific hardware. I couldn't be generalized. But there are many limitations with Linux. Yes, we can have custom drivers, but so much *has* to be general. And that isn't always easy. -- The great successful men of the world have used their imagination ... think ahead and create their mental picture in all it details, filling in here, adding a little there, altering this a bit and that a bit, but steadily building - steadily building. -- Robert Collier -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen said the following on 03/20/2010 05:00 PM:
Anton Aylward wrote:
Due to the multi-cores, we're probably at the threshold of an era with an entirely new set of program bugs.
Multiple execution units are nothing new - they date back at least 20 years. I started writing multi-threaded code for IBM 3090s around 89/90. The multi execution unit boxes have been perfectly common in servers since then.
Its not the people like thee and mee that I'm worried about. For the most part, its not even the people developing for Linux Its the numbers out there. For every one of us that have the experience you speak of there are ... Thousands of PC programmers who never really mastered dBase, pascal and other first generation languages who are now producing code for web sites. I recently met some web application programmers who didn't understand what I meant when I described a web site as a 'state machine'; other who didn't understand that they were dealing with a connectionless protocol. When I look to what tools there are, and then I look the to the tens of thousands of development tools that have nothing to do with threading ... I think the marketplace has a lot of catching up to do. When I look to the changes to the scheduling code that Linux has seen this last couple of years ... and I think about what was used in MUMPS and even VAX/VMS ... I wonder how much and how quickly we do learn from the past. And I wonder how much opportunity people like Per and myself have for 'mentoring'. In the 1970s I was working with SMP machines that not only were completely asynchronous but also did dynamic load balancing as processor failed or were replaced. In the 1980s I worked with Sequent machines that had libraries that allowed many of the program we currently think of as single threaded to be balanced across multiple processors - dynamically. In the 1990s I worked with database systems that used large shared memory segments and multiple dynamically created (short lived) processes (you can tell it wasn't Oracle!) to deal with various tasks like before and after imaging, queries and connections. In doing so they created many 'threads' using a more conventional model. I worked on SCADA systems using single-stack, mutli-threaded FORTH. Where is all this now? Oh, its there. I see it in some real-time work. But a lot of it seems to have been lost. I don't see the 'progression'. But how multi-threaded is your smart-phone? And then there are the people that think security is just about breaking in ... -- "What we have learned from others becomes our own by reflection". -- Ralph Waldo Emerson -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hello, On Sat, 20 Mar 2010, Anton Aylward wrote: [..]
We will need new design and programming paradigms, tools and debugging tools.
Functional programming languages (like Erlang) look very interesting in this light ... see http://en.wikipedia.org/wiki/Erlang_(programming_language) -dnh -- MISC You are the 4711th visitor of this page. -- man xawtv -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
See http://rationalsecurity.typepad.com/blog/multicore_crisis/ "All Your COTS Multi-Core CPU's With Non-Optimized Security Software Are Belong To Us..." -- "Tell me, and I'll forget. Show me, and I'll remember. Involve me, and I'll learn." -- Marla Jones -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hans Witvliet wrote:
Perhaps a silly question...
I noticed on one of my machines, one of the cores was permanently 100% busy, while others were stile idle. (two dual-cores)
This i noticed both for openvpn and rsync. For both my cpu is apperently the bottleneck, not my network.
Aren't the processes supposed to be spread even on the different core's?
Until you have enough load to use up more cores, there's no need to use them. /Per -- Per Jessen, Zürich (12.0°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen said the following on 03/20/2010 12:43 PM:
Until you have enough load to use up more cores, there's no need to use them.
Its not simply "load", its the nature of the load. Multiple cores make sense for the big servers where there are many simulates unblocked threads of execution. For a desktop machine where there is, for the most part, 'just you' running the foreground task, they offer little advantage. Yes, you may be running a download in the background, but that's going to be IO limited, not CPU limited. In fact most of the time your computer is waiting for you to it a key. Yes, there are situations where you might need more "cpu power", but stop and think if the situation is really CPU-limited or IO-limited. The example Hans gave got the right answer from Per. The code for those applications is not multi-threaded. It _could_ be, perhaps, but it isn't. Yes, 'rsync' could be written as a producer-consumer pair of processes, but it would become more complicated, more prone to design and logic errors and less 'general'. Do you want to optimise it for two cores, and them repeat Hans' complain when you are running a 4-core desktop? -- HTTP is like being married: you have to be able to handle whatever you're given, while being very careful what you send back. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Anton Aylward wrote:
The example Hans gave got the right answer from Per. The code for those applications is not multi-threaded. It _could_ be, perhaps, but it isn't. Yes, 'rsync' could be written as a producer-consumer pair of processes, but it would become more complicated, more prone to design and logic errors and less 'general'.
Given that many of todays and most of tomorrows CPUs are going to be multi-core, anyone writing a daemon (or something that has the same characteristics) should consider whether it would make sense to be multi-threaded.
Do you want to optimise it for two cores, and them repeat Hans' complain when you are running a 4-core desktop?
You should always write for N cores. You will never know how many execution units you're likely to have, so why optimize for a certain number. -- Per Jessen, Zürich (11.4°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen said the following on 03/20/2010 01:57 PM:
Anton Aylward wrote:
Do you want to optimise it for two cores, and them repeat Hans' complain when you are running a 4-core desktop?
You should always write for N cores. You will never know how many execution units you're likely to have, so why optimize for a certain number.
History tells me that yes, people *will* be that short sighted. -- For people who like peace and quiet: a phoneless cord. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sat, 2010-03-20 at 13:04 -0400, Anton Aylward wrote:
Multiple cores make sense for the big servers where there are many simulates unblocked threads of execution.
For a desktop machine where there is, for the most part, 'just you' running the foreground task, they offer little advantage.
Have you run bzip2 lately? I'm pretty glad that I have more than one core, so that I can still so something else on my machine while that is running. I also like to rip the movies I buy so I can put them on my media center machine - I could start that and then not use my machine at all for about 10 hours while that is running, or I could use my other cores. And this is not counting the innumerable applications and games that can use multiple cores internally. The usage scenario you describe sound more like win95 or DOS to me - in more modern times, there are huge benefits to having multiple CPUs/cores on a desktop Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-03-20 20:04, Anders Johansson wrote:
I also like to rip the movies I buy so I can put them on my media center machine - I could start that and then not use my machine at all for about 10 hours while that is running, or I could use my other cores.
Have you tried ffmpeg with libxvid? It actually runs slower in threaded mode (on the other hand, the mpeg codec runs faster). Converting with the xvid codec has to be run on one thread only, but you can convert several movies at the same time, one on each core. It depends a lot on the programmer, the benefits of having several cores are not automatic. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkulHwgACgkQU92UU+smfQUCPQCfYtnJ9qr7CLk24O0R7ylrxSBf kKQAn0so2mwYBw6pRW1yTQksZ47s+xxs =sNqw -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sat, 2010-03-20 at 20:16 +0100, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2010-03-20 20:04, Anders Johansson wrote:
I also like to rip the movies I buy so I can put them on my media center machine - I could start that and then not use my machine at all for about 10 hours while that is running, or I could use my other cores.
Have you tried ffmpeg with libxvid? It actually runs slower in threaded mode (on the other hand, the mpeg codec runs faster). Converting with the xvid codec has to be run on one thread only, but you can convert several movies at the same time, one on each core.
It depends a lot on the programmer, the benefits of having several cores are not automatic.
Most rippers run in a single thread. But it is not the only program running on my machine. And that part is automatic Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Anders Johansson said the following on 03/20/2010 03:04 PM:
The usage scenario you describe sound more like win95 or DOS to me - in more modern times, there are huge benefits to having multiple CPUs/cores on a desktop
Indeed there are. And sadly most people using desktops are running Windows and using it like W95. Unlike you (or me) they will wait for files to unzip, wait for things to compete. -- Nobody realizes that some people expend tremendous energy merely to be normal. --Albert Camus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sat, 2010-03-20 at 20:04 +0100, Anders Johansson wrote:
On Sat, 2010-03-20 at 13:04 -0400, Anton Aylward wrote:
Multiple cores make sense for the big servers where there are many simulates unblocked threads of execution.
For a desktop machine where there is, for the most part, 'just you' running the foreground task, they offer little advantage.
Have you run bzip2 lately? I'm pretty glad that I have more than one core, so that I can still so something else on my machine while that is running.
I also like to rip the movies I buy so I can put them on my media center machine - I could start that and then not use my machine at all for about 10 hours while that is running, or I could use my other cores.
And this is not counting the innumerable applications and games that can use multiple cores internally.
I know for sure, that apache2 (who can refork dozens of times) does spread the load... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Quoting Hans Witvliet
Perhaps a silly question...
I noticed on one of my machines, one of the cores was permanently 100% busy, while others were stile idle. (two dual-cores)
This i noticed both for openvpn and rsync. For both my cpu is apperently the bottleneck, not my network.
Aren't the processes supposed to be spread even on the different core's?
These programs, as are most programs (still), are single process/thread. They do not take advantage of multiple cores. Parallel programming is even more difficult than single process/thread programming to get right. If you have a tool that shows which CPU is busy, you may see them occasionally switch CPUs. It is a matter of tradeoffs, correctness should always be valued over speed and a programmer's time is always a limited resource. Just my $0.02USD, Jeffrey -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sat, 2010-03-20 at 16:47 +0100, Hans Witvliet wrote:
Perhaps a silly question...
I noticed on one of my machines, one of the cores was permanently 100% busy, while others were stile idle. (two dual-cores)
This i noticed both for openvpn and rsync. For both my cpu is apperently the bottleneck, not my network.
Aren't the processes supposed to be spread even on the different core's?
Yes, if you have multiple processes that all take up CPU time, they should get spread out among the cores. If that doesn't happen on your machine, it is strange Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (9)
-
Anders Johansson -
Anton Aylward -
Carlos E. R. -
Carlos E. R. -
David Haller -
Hans Witvliet -
Jeffrey L. Taylor -
Per Jessen -
upscope