[opensuse-factory] Phoronix has done some network tests windows vs a few distros (incl TW)
http://tinyurl.com/ydefbwq8 -- opensuse:tumbleweed:20180313 Qt: 5.10.0 KDE Frameworks: 5.43.0 - KDE Plasma: 5.12.3 - kwin 5.12.3 kmail2 5.7.2 - akonadiserver 5.7.2 - Kernel: 4.15.8-1-default - xf86-video-nouveau: 1.0.15 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 15-03-2018 a las 13:52, Ianseeks escribió:
This benchmarks are not particulary useful to developers or to take any remediation steps if needed. they appear to be provided purely for entertaiment. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2018-03-15 22:00, Cristian Rodríguez wrote:
El 15-03-2018 a las 13:52, Ianseeks escribió:
This benchmarks are not particulary useful to developers or to take any remediation steps if needed. they appear to be provided purely for entertaiment.
That is exactly the scope of Phoronix. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Op donderdag 15 maart 2018 22:52:04 CET schreef Jan Engelhardt:
On Thursday 2018-03-15 22:00, Cristian Rodríguez wrote:
El 15-03-2018 a las 13:52, Ianseeks escribió:
This benchmarks are not particulary useful to developers or to take any remediation steps if needed. they appear to be provided purely for entertaiment.
That is exactly the scope of Phoronix. And that is exactly why I don't visit Phoronix.
-- Gertjan Lettink, a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team Linux user #548252 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 15.03.2018 um 22:52 schrieb Jan Engelhardt:
On Thursday 2018-03-15 22:00, Cristian Rodríguez wrote:
El 15-03-2018 a las 13:52, Ianseeks escribió:
This benchmarks are not particulary useful to developers or to take any remediation steps if needed. they appear to be provided purely for entertaiment.
That is exactly the scope of Phoronix.
Care to explain how those network-benchmarks could be improved? The discussed benchmark you are slandering is only a preview, the full comparison is in the works. It was requested and payd for by a user, so it must be useful to him. As a Tumbleweed user I enjoy those benchmarks, so entertaining it is, yes. But at the same time they deliver also vaild data points, sometimes on a small scale (hardware wise), and sometimes with nice big comparisons when it comes to gaming / gpus. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday, March 16, 2018 9:33:52 AM CET tomtomme wrote:
That is exactly the scope of Phoronix.
Care to explain how those network-benchmarks could be improved? The discussed benchmark you are slandering is only a preview, the full comparison is in the works. It was requested and payd for by a user, so it must be useful to him.
As a Tumbleweed user I enjoy those benchmarks, so entertaining it is, yes. But at the same time they deliver also vaild data points, sometimes on a small scale (hardware wise), and sometimes with nice big comparisons when it comes to gaming / gpus.
I'm not a developer. But the one thing I heard mentioning in the past is that Michael is basically testing default configurations , which can be quite different between distros and are not always appropriate for he tested subject. I personally read phoronix a lot but I know how I take it. The Benchmarks sometimes seems to be made in a hurry and in a drive-by manner. Nobody disputes that he is a workoholic , with the frequency of updates to the site. But to be honest , what annoys me most is that he mostly links to his own posts instead of the source and his constant grandstanding , e.g. I was the first to report etc.. or as I have reported several times before .. I have not seen this specific benchmarks though. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Still, if somebody volunteered to investigate what exactly causes the performance difference and what could be done about it this would be positive for openSUSE. Of course, there are more important areas where contributors are needed but different people have different skills and interests. For some reader on this mailing list this may be the right thing to work on. Joachim -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 2018-03-16 11:39, Joachim Wagner wrote:
Still, if somebody volunteered to investigate what exactly causes the performance difference and what could be done about it this would be positive for openSUSE.
Of course, there are more important areas where contributors are needed
In other news, company CTS Labs, "a heretofore unknown Tel Aviv-based cybersecurity startup, has claimed it's found over a dozen security problems with AMD Ryzen and EPYC processors" (ZDNet), sends everyone on a wild goose chase for a day. And now we're supposed to hunt whatever flaws there may be in Larabel's setup? (Cf. post by M. Kubeček.) "Yeah [right]." (ZDnet) On top, the Phoronix setup is not completely specified for there to be an independent investigation as I see it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
And now we're supposed to [...]
Nobody is telling you what to do. There are pros and cons to any activity and to make an informed decision somebody needs to point out the other side.
(Cf. post by M. Kubeček.)
While this post seems to be written to dismiss the relevance of the report, it does provide valuable information both for the author and for anybody who wishes to dig deeper. Joachim -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday, 16 March 2018 9:33 tomtomme wrote:
Am 15.03.2018 um 22:52 schrieb Jan Engelhardt:
On Thursday 2018-03-15 22:00, Cristian Rodríguez wrote:
El 15-03-2018 a las 13:52, Ianseeks escribió:
This benchmarks are not particulary useful to developers or to take any remediation steps if needed. they appear to be provided purely for entertaiment.
That is exactly the scope of Phoronix.
Care to explain how those network-benchmarks could be improved? The discussed benchmark you are slandering is only a preview, the full comparison is in the works. It was requested and payd for by a user, so it must be useful to him.
As a Tumbleweed user I enjoy those benchmarks, so entertaining it is, yes. But at the same time they deliver also vaild data points, sometimes on a small scale (hardware wise), and sometimes with nice big comparisons when it comes to gaming / gpus.
Phoronix reviews and benchmark are famous... I remember Mr. Larabel's "review" of an openSUSE conference in 2012(?) focusing mostly on the fact that there were problems with food supply on the welcome party. :-) Benchmarking is hard. If you want to do it right, that is. It means thinking about how to do the setup to minimize random influence of other factors. Careful thinking about what you really want to measure and how to do it for the results to be relevant. Analyzing the results to make sure they make sense. And, of course, trying to explain the results, in particular those which stand out. This test? Exactly the opposite. He goes to long details to explain what disks do server and client have but not a word about NICs; "Gigabit Ethernet" is not nearly enough - and that hidden "Realtek PCIe GBE Family + Microsoft ISATAP" (???) isn't much better. One might think that for a "networking performance test" the NIC would be more important than the disk. No information about how were client and server connected. And, of course, not a word about netfilter configuration (which can affect the latency quite a log) and other networking settings; instead, we get detailed list of gcc flags used for build. It's proudly called "Network Benchmarks" or even "networking performance benchmarks" and yet, all the author did was running netperf TCP_RR and UDP_RR test (i.e. one very specific aspect of networking performance) with some parameters he didn't bother to tell us. From the results, it's apparent he didn't notice some tests show variance so high that the results should have been discarded and tests repeated. Not to mention that netperf has "-I" and "-i" parameters to make things even easier. Instead, he runs the same test with two different lengths. and presents the results separately. Why? The results should be the same within a margin of statistic error; if they are not, it should be a warning sign that something was wrong. Or should we perhaps believe the PC gets tired if you run the test for whole 6 minutes? One thing that really stands out is the UDP_RR test on Tumbleweed, in particular the 360s one. The variance itself should tell author the test went completely bonkers. Even if he didn't realize, he might have noticed even upper end of the indicated interval is still way below the results of the 60s test. Neither stopped him from publishing such completely unreliable results. Out of curiosity, I quickly ran netperf UDP_RR between my two machines, one running 42.3 with 4.15.8 Kernel:stable kernel (i.e. essentially Tumbleweed), the other Tumbleweed with 4.16-rc4 kernel from Kernel:HEAD. The hardware is definitely worse than Mr. Larabel's and NICs aren't anything special either (on-board Realtek 8168evl and common consumer grade Intel (82541GI)). Both machines are running a KDE desktop (I only wanted to get some idea about the results) so I used "-I 99 -i 20,5" to make sure the results are not completely random. The result I got was... wait for it... 10359.48. Even with 1400 bytes of request/response size, I still get 3257.53. As the highest result in the Phoronix article is 918.92, I feel Mr. Larabel owes us some information about what he was actually testing and how. (Honestly, over a millisecond per roundtrip on gigabit ethernet? That should give anyone a hint they are doing something wrong.) I just hope the next article isn't going to present TCP_STREAM results (measured on gigabit ethernet). That would be even more ridiculous. Michal Kubeček -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
[...] That should give anyone a hint they are doing something wrong.)
Thanks for pointing out issues with the benchmark and for trying to replicate some of the numbers. Given that openSUSE doesn't look good in the preliminary results, it would be beneficial to openSUSE if somebody contacted the author with constructive suggestions how to fix the most serious flaws in the benchmark, keeping in mind that he doesn't have time to go as deep as people here on the mailing list might wish. Joachim -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 16.03.2018 um 12:06 schrieb Joachim Wagner:
Given that openSUSE doesn't look good in the preliminary results, it would be beneficial to openSUSE if somebody contacted the author with constructive suggestions how to fix the most serious flaws in the benchmark,
Go ahead, do it. But assigning people with actual competence and knowledge to deal with a bullshit-fabricator like phoronix is a severe waste of scarce resources.
keeping in mind that he doesn't have time to go as deep as people here on the mailing list might wish.
Catrering to people who rely on phoronix for technical advice is a lost cause from the very start. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 17/03/18 06:50 AM, Stefan Seyfried wrote:
Catrering to people who rely on phoronix for technical advice is a lost cause from the very start.
You mean managers and other decision makers who want a simplistic answer to what us more technically minded people know is a multi-variant problem surrounded by a whole raft of situational and conditional if-but-maybe provisions to which there is no simple or reductionist solution. But there's no telling managers that. Never has been. Never will be. -- There is no such thing as a perfect leader either in the past or present, in China or elsewhere. If there is one, he is only pretending, like a pig inserting scallions into its nose in an effort to look like an elephant. - Liu Shao-ch'i -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 16.03.2018 um 09:33 schrieb tomtomme:
The discussed benchmark you are slandering is only a preview, the full comparison is in the works. It was requested and payd for by a user, so it must be useful to him.
Come on. I'm benchmarking stuff every day as part of my day job. Of course I ask my managers what kind of result they want to get. That's the same with the "paid for by" user. He wants to have "external facts" for a decision, e.g. to defend using his favorite distribution. There's many ways even a phoronix benchmark can be "useful" to someone. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Samstag, 17. März 2018, 11:53:59 CET schrieb Stefan Seyfried:
Am 16.03.2018 um 09:33 schrieb tomtomme:
The discussed benchmark you are slandering is only a preview, the full comparison is in the works. It was requested and payd for by a user, so it must be useful to him.
Come on. I'm benchmarking stuff every day as part of my day job. Of course I ask my managers what kind of result they want to get.
That's the same with the "paid for by" user. He wants to have "external facts" for a decision, e.g. to defend using his favorite distribution.
There's many ways even a phoronix benchmark can be "useful" to someone.
Hey, Stefan, should we do an article about the benchmarks you do - a professional one for print magazines? Yes, they still exist. I would ask my former employee - not only to set this Phoronix tests right again, but to have some reliable data source to compare. I'd be in. -- Markus Feilner Team Lead Documentation P.S.: I moved - new home address: Wöhrdstraße 10, 93059 Regensburg - - - _This incident will be documented._ - - - +49 173 5876 838 (also via Signal), privat: +49 170 302 7092 mfeilner@suse.[com|de] http://www.suse.com G+: https://plus.google.com/+MarkusFeilner Xing: http://www.xing.com/profile/Markus_Feilner LinkedIn: https://www.linkedin.com/in/markusfeilner #mfeilner: Jabber, Skype, Twitter openSUSE: http://www.opensuse.org - - - SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
Hi Markus, Am 19.03.2018 um 09:40 schrieb Markus Feilner:
Am Samstag, 17. März 2018, 11:53:59 CET schrieb Stefan Seyfried:>> Of course I ask my managers what kind of result they want to get.
You obviously did not read my message with enough attention :-)
Hey, Stefan, should we do an article about the benchmarks you do - a professional one for print magazines? Yes, they still exist. I would ask my former employee - not only to set this Phoronix tests right again, but to have some reliable data source to compare.
I don't think that an article about "producing benchmark results that prove exactly what you want to prove" is something I really want my name attached to :-P (And what a magazine for professionals would want to print) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Freitag, 23. März 2018, 19:00:50 CEST schrieb Stefan Seyfried:
Hi Markus,
Am 19.03.2018 um 09:40 schrieb Markus Feilner:
Am Samstag, 17. März 2018, 11:53:59 CET schrieb Stefan Seyfried:>> Of course I ask my managers what kind of result they want to get. You obviously did not read my message with enough attention
Hey, Stefan, should we do an article about the benchmarks you do - a professional one for print magazines? Yes, they still exist. I would ask my former employee - not only to set this Phoronix tests right again, but to have some reliable data source to compare.
I don't think that an article about "producing benchmark results that prove exactly what you want to prove" is something I really want my name attached to (And what a magazine for professionals would want to print) -- Stefan Seyfried
"For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
Interesting, something here must have gone terribly wrong in quoting. I can't remember writing that sentence and I would never do that. Maybe I just got that wrong. From what I understood, you know how to cheat in benchmarks. And I think telling people about how ugly people cheat (or accidentally do wrong things, maybe even subconciously) would be a great thing and a nice article - if that hasn't been written many a times. Yes, I still think an article about "How to get the benchmark results you want" might be interesting. Why? Because: Readers might like to read: a) "Beware! - here's how people fake benchmarks" b) "Beware! - How to recognize faked benchmarks" c) "What you should be aware of when creating benchmarks" Am I wrong? -- Markus Feilner Team Lead Documentation P.S.: I moved - new home address: Wöhrdstraße 10, 93059 Regensburg - - - _This incident will be documented._ - - - +49 173 5876 838 (also via Signal), privat: +49 170 302 7092 mfeilner@suse.[com|de] http://www.suse.com G+: https://plus.google.com/+MarkusFeilner Xing: http://www.xing.com/profile/Markus_Feilner LinkedIn: https://www.linkedin.com/in/markusfeilner #mfeilner: Jabber, Skype, Twitter openSUSE: http://www.opensuse.org - - - SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
participants (11)
-
Anton Aylward -
Cristian Rodríguez -
Ianseeks -
Jan Engelhardt -
Joachim Wagner -
Knurpht - Gertjan Lettink -
Markus Feilner -
Michal Kubecek -
Stefan Kunze -
Stefan Seyfried -
tomtomme