On 2016-02-25 08:05, Per Jessen wrote:

sdm wrote:

You pick one of the troubled areas, e.g. databases and you start analyzing what makes openSUSE slower. Typical things that might impinge on performance - filesystem setup, perhaps apparmor, sub-optimal core libraries perhaps.

If they are using the default install, on btrfs, it will be slow on databases. -- Cheers / Saludos, Carlos E. R. (from openSUSE Leap 42.1 x86_64 (test)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2016-02-25 09:57, Vojtěch Zeisek wrote:

Dne Čt 25. února 2016 09:50:26, Carlos E. R. napsal(a):

...

If they are using the default install, on btrfs, it will be slow on databases.

Default install is using special subvolume for default DB locations without extra Btrfs features slowing down the databases...

Ah, this is a good change, because I remember the recommendation to not place databases on btrfs. I don't see the word "database" in the Leap release notes, but I don't know where this change happened. Then we need to know if phoronix is locating their test databases there, or if they change the defaults. -- Cheers / Saludos, Carlos E. R. (from openSUSE Leap 42.1 x86_64 (test)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2016-02-25 10:55, Frans de Boer wrote:

On 02/25/2016 10:13 AM, Carlos E. R. wrote:

If one is ready the article well, you would understand that they did not changed one thing. It was testing "out-of-the-box". So they used the default XFS settings.

XFS is not the default for system databases. It is used only on /home. -- Cheers / Saludos, Carlos E. R. (from openSUSE Leap 42.1 x86_64 (test)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2016-02-25 08:05, Per Jessen wrote:

...

sdm wrote:

...

You pick one of the troubled areas, e.g. databases and you start analyzing what makes openSUSE slower. Typical things that might impinge on performance - filesystem setup, perhaps apparmor, sub-optimal core libraries perhaps.

If they are using the default install, on btrfs, it will be slow on databases.

From the article:

"... and its home directory is an XFS file-system from where the benchmarks were conducted." -- Per Jessen, Zürich (2.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2016-02-25 10:52, Per Jessen wrote:

...

Carlos E. R. wrote:

...

...

From the article:

"... and its home directory is an XFS file-system from where the benchmarks were conducted."

And they store the databases there?

Dunno, I can only tell you what the article says.

That's not the default for system databases.

What's a "system database" ? Leap uses mysql/mariadb by default, but the tests were done with postgres. -- Per Jessen, Zürich (2.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Vojtěch Zeisek wrote:

Dne Čt 25. února 2016 11:06:16, Per Jessen napsal(a):

...

Carlos E. R. wrote:

...

On 2016-02-25 10:52, Per Jessen wrote:

...

Carlos E. R. wrote: That's not the default for system databases.

What's a "system database" ? Leap uses mysql/mariadb by default, but the tests were done with postgres.

It it by default in /var/lib/pgsql on separate Btrfs subvolume.

I have no idea what kind of performance to expect with butterfs, but it's an interesting question. The article seemed keen to point out the /home was on XFS, and that tests were run from there. -- Per Jessen, Zürich (2.6°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Feb 25, 2016 at 2:28 PM, Vojtěch Zeisek wrote:

Copy-on-write is terribly slowing down any DB.

btrfs does not do any copy on write. It does "redirect on write" which is quite different. This does cause fragmentation; it is impossible to give blanket statement what effect it will have. It will likely impact sequential IO; it is much less likely impact random IO (unless your database is very small and would not be affected by seek latency otherwise).

Also snapshots don't make sense for DB (use DB's tools).

If you have suggestions how to make point in time online copies without impacting normal activity (terribly slowing down client access :) ) *and* revert to them in seconds for multiterabyte database (in order of several dozens TB) without using snapshots I am all ears. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Feb 25, 2016 at 2:56 PM, Vojtěch Zeisek wrote:

Dne Čt 25. února 2016 14:40:00, Andrei Borzenkov napsal(a):

...

On Thu, Feb 25, 2016 at 2:28 PM, Vojtěch Zeisek wrote:

...

Copy-on-write is terribly slowing down any DB.

btrfs does not do any copy on write. It does "redirect on write" which

https://btrfs.wiki.kernel.org/index.php/Main_Page „Btrfs is a new copy on write (CoW) filesystem for Linux...“ Did I misunderstand something? :-)

No; whoever created this page used confusing name. CoW snapshots existed long before btrfs was invented and has quite precise meaning in storage world - it is technology when old data is *copied* elsewhere before being overwritten by new data in place. btrfs does not overwrite old data ever - new data is written in free space, leaving old intact. This applies not only to user data, but to metadata as well - so you get completely new filesystem tree, and only when this tree is constructed, the top level pointer is "flipped".

...

is quite different. This does cause fragmentation; it is impossible to give blanket statement what effect it will have. It will likely impact sequential IO; it is much less likely impact random IO (unless your database is very small and would not be affected by seek latency otherwise).

I don't know the theoretical background. But many tests show Btrfs with CoW and subvolumes has bad performance impact to DB. See also https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Copy_on_Write_.28CoW.2...

...

...

Also snapshots don't make sense for DB (use DB's tools).

If you have suggestions how to make point in time online copies without impacting normal activity (terribly slowing down client access

Client access is what is counted, right? ;-)

Is it sarcasm? Database exists to serve clients and if clients are cut off for hours or days to perform backup, then yes, it counts. Or if clients must wait for days to revert to older date because someone made an error.

...

:) ) *and* revert to them in seconds for multiterabyte database (in order of several dozens TB) without using snapshots I am all ears.

I don't say it is a bug. It just isn't the best for all user cases...

You said "it does not make sense". It is very far from "isn't the best in all user cases". -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Feb 25, 2016 at 1:29 PM, Vojtěch Zeisek wrote:

OK, again. If some feature of Btrfs is making performance problems for DBs (do we agree it does?), then it does not make sense to use it for DB, doesn't it? Of course, there are dozens of user cases. Some are are suitable, some not.

Our primary use for openSUSE is as a data collection system that collects large amounts of data (megabytes a minute to various data files, often sustained for a couple of hours at a time) that is routinely copied off-system, and then the collected data files deleted. I have been wondering how btrfs use may impact this. I guess we will just have to set up a system and see what happens. Any predictions? -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Feb 25, 2016 at 2:07 PM, Anton Aylward wrote:

It may take a bit of experimentation. There's a lot out there on using XFS/realtime for database performance with oracle and postgress. You might use that as a bass.

We do have subsystems where we have used XFS. And that has been working fine. At the moment we are using ext4, and it is also working fine. The disks are SSD or fast SATA. They have been working fine. I don't have basic questions about this scenario. Have been doing this for years. And, of course, we will do testing before even considering using btrfs. What I do have are question on what btrfs brings to the situation. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Roger Oberholtzer wrote:

On Thu, Feb 25, 2016 at 2:07 PM, Anton Aylward wrote:

...

It may take a bit of experimentation. There's a lot out there on using XFS/realtime for database performance with oracle and postgress. You might use that as a bass.

We do have subsystems where we have used XFS. And that has been working fine. At the moment we are using ext4, and it is also working fine.

The disks are SSD or fast SATA. They have been working fine.

I don't have basic questions about this scenario. Have been doing this for years. And, of course, we will do testing before even considering using btrfs.

What I do have are question on what btrfs brings to the situation.

Given what you have described to us, absolutely nothing. -- Per Jessen, Zürich (2.6°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Feb 25, 2016 at 2:21 PM, Per Jessen wrote:

Roger Oberholtzer wrote:

...

Our primary use for openSUSE is as a data collection system that collects large amounts of data (megabytes a minute to various data files, often sustained for a couple of hours at a time) that is routinely copied off-system, and then the collected data files deleted.

Sounds like you ought to be using tape. Or a portable SSD drive.

Tape? Haven't heard that for years. Not in the cards:) Since the entire reason for the system existing is to collect this data, we will make the file systems whatever is best. We collect data to /home. Since that is pretty much all that ever happens on /home, there is little reason to let /home be an inappropriate file system and then put all the data elsewhere. We set up these systems via an OEM image created with KIWI. I see that KIWI allows us to select the file system for the created openSUSE OS image. I wonder of we can still select ext4 when building an install image for Leap. Or, more specifically, if such a selection is honored. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Le 25/02/2016 13:29, Vojtěch Zeisek a écrit :

OK, again. If some feature of Btrfs is making performance problems for DBs (do we agree it does?), then it does not make sense to use it for DB, doesn't it? Of course, there are dozens of user cases. Some are are suitable, some not.

performance is not the alpha and omega. Data security is. May be have stable backups with snapshots is worth having a small loss in performance. What where the performance one year ago... we lived with them :-). just my 2 cents :-) jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 02/25/2016 06:28 AM, Vojtěch Zeisek wrote:

Copy-on-write is terribly slowing down any DB. Also snapshots don't make sense for DB (use DB's tools). Maybe there are more problematic points. That is why Btrfs is the worst for any DB.

Indeed! back in the days when oracle didn't consider UNIX - Linux didn't exist then - worth bothering with, it made the case that the DBM should be managing disk access. Even when, grudgingly, oracle migrated to some of the Big iron UNIX implementation from SUN/servers, AIX/servers and HP/UX it still insisted on running on raw partitions and not a file system. I recall one Oracle support tech telling me that using files was was for toy databases lie dBase. The file system is just another layer of indirection. Having to play COW-games as well is an overhead. So why bother? I can think of two principal reasons. 1. Integrated backups. While the database-on-raw is efficient in one way, it makes backups awkward. It involves converting tables to files ("exporting") so that the normal backup tools can be used. Slow, inefficient and error prone. 2. File systems have got better. I often refer to the early days, the V7FS, the early Berkeley FFS. They were easy to understand, in some ways easier than MinixFS. But they were not by any means efficient even by the metrics of those days. Modern file systems are not only more efficient but have facilities that those antediluvian dinosaurs never had, such as expanded file space and size, large block allocation and preallocation, journalling and more. While BtrFS, like LVM and XFS, can span more than one volume, a more useful multi-spindle approach is available with RAID, again something that wasn't there when Oracle only ran on IBM mainfrmes. Ultimately though, the oracle experience is not the beast way of looking at things. Oracle, like the original DB2, was a huge single processor with multi-thread. That's not the way UNIX/Linux views things. The *NIX model is of small, possibly dynamic, processes that have a low creation overhead. This contrasts heavily with the mainframe and even the VAX/VMS model of the long lived, fixed number of process 'slots' and 'transient program area' area. Its why, at the time, UNIX was so revolutionary. UNIX-friendly databases were things like Progress which had a number of cooperating processes which could be brought up and discarded much the way that a large shell script might work in conjunction with there services like the line printer daemon. Somewhere along the line IBM realised that if it was to run DB2 efficiently on its UNIX-like AIX it had to move away from the monolithic model to something like Progress. Sort of. At the same time, the AIX file system - JFS - was very supportive, behaving line an integrated, journal version of LVM. It was easy to group tables to a logical volume that was striped and cached in a specific manner; hence tuning was very flexible. Add to that it being well integrated to IBM's multiprocessor support. The "downside", if you want to consider it that way, with MySQL/MariaDB or Postgress on line is that they are not architected for a specific filesystem :-) If you don't like that "downside" then go with one of the proprietary databases on BigIron. Its worth googling for the performance of databases on XFS, JFS, ext4 and so on. Not just the reports from phoronix, but others as well. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2016-02-25 11:06, Per Jessen wrote:

...

Carlos E. R. wrote:

...

...

That's not the default for system databases.

What's a "system database" ? Leap uses mysql/mariadb by default, but the tests were done with postgres.

Somewhere under /var, used by databases running as services, ie, started by systemd, and accessible by anybody in the computer, under control of the database engine. Postgres is installed that way.

You can also have "user" databases, stored completely on your own home. KDE uses some. But this also means that the "defaults" for those databases are defined by the tester, not by the system install.

Okay, it's not a distinction I am used to. To me, a database is a database is a database, no matter how it's started, where the data is placed or which user it runs with. I thought KDE simply used the standard mysql/mariadb install, I wasn't aware of any per-user instances. -- Per Jessen, Zürich (2.6°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 02/25/2016 10:55 AM, Carlos E. R. wrote:

On 2016-02-25 12:09, Per Jessen wrote:

...

Carlos E. R. wrote:

...

Okay, it's not a distinction I am used to. To me, a database is a database is a database, no matter how it's started, where the data is placed or which user it runs with. I thought KDE simply used the standard mysql/mariadb install, I wasn't aware of any per-user instances.

I think it can use the libraries, but not the engine. I'm unsure of this. Good question.

However, some KDE apps sometimes use the system database engine to place there a database, and for this it requires the user/admin to allow it, or initiate the database.

Exactly. KDE's Akonadi will use the system database, and in many cases simply turning on ANY akonadi services will load mysqld which consumes an inordinate amount of memory to serve (typically) a single user. And that can in turn feed baloo. But it is still not clear that these tests used any thing that might have touched the system database. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2016-02-25 21:41, John Andersen wrote:

On 02/25/2016 10:55 AM, Carlos E. R. wrote:

Exactly. KDE's Akonadi will use the system database, and in many cases simply turning on ANY akonadi services will load mysqld which consumes an inordinate amount of memory to serve (typically) a single user.

When KDE uses a system database it uses a lot of resources, yes. But my own database uses very few. I don't know why. Maybe KDE does a lot of accessing.

And that can in turn feed baloo.

But it is still not clear that these tests used any thing that might have touched the system database.

Right. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlbPiQAACgkQja8UbcUWM1y1ZAD/fXDNvkflq9HSI6e483m3ZmnP 5XzryDbfU/P5inJtwIQBAJEPRhuA+NO4EBubo3nM1Rno1On3P0N+DNQkCoZJGfgV =XlSd -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2/25/2016 11:22 PM, Per Jessen wrote:

John Andersen wrote:

...

Exactly. KDE's Akonadi will use the system database, and in many cases simply turning on ANY akonadi services will load mysqld which consumes an inordinate amount of memory to serve (typically) a single user.

It shouldn't be so bad - for Akonadi use, I can't imagine it using more than 300-400M? Most of that will be for indexes and probably unused. Even on a meagre office desktop, it should not cause any issues. Is akonadi default in Leap 42? I'll have to check.

Given that most users won't have a ton of data in there from akonadi you might be right, but then they could have got by with something a lot lighter than mysql, which itself can chew up a lot of runtime memory. KDE use LMDB for baloo, (which is very lightweight, and fast), and almost everything akonodi puts into mysql also gets fed to baloo. Seems like a mysql is a complexity that really isn't necessary. http://vhanda.in/blog/2015/10/baloo-5-15/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Knurpht - Gertjan Lettink wrote:

Op woensdag 24 februari 2016 20:42:35 CET schreef sdm:

...

This is at least the second article I've ready where Leap's benchmarks pretty dismal:

http://www.phoronix.com/scan.php?page=article&item=2016-feb-6way&num=1

...

Why is Leap running at the back of the pack, and is there anything we can do to speed the OS up through patching, so the benchmarks made against it allows for more interest amongst people comparing Linux distros? I'm not impressed at all by the latest benchmarks released by Phoronix.

sdm

My 2 cents: too many unknown variables.

Admitted, IMHO Phoronix never brings answers, rather raises questions. One example: why mention that XFS is used for /home and not mention what is being used for "/"?

The others were all using ext4, it was probably a significant difference worth mentioning. I read it to mean "openSUSE was tested on XFS, the others on ext4.", no more, no less.

What I want from a distro is not a good benchmark score, but stuff like having a webserver up and running in < 5 minutes without having to edit dozens of files. Like a good and stable desktop experience.

We all have different needs. To me, boot-time is mostly irrelevant. My servers spend way more time on POST, memory check and waiting for network interfaces (wicked easily adds 30 seconds) anyway. Most of them are only booted once a year :-) My desktop systems need to be 100% stable, perform reasonably well on lower to mid-range hardware, and not change very much. (people don't like change).

IMHO these benchmarks are useful to a very limited audience, not the audience that is going to make openSUSE grow.

The OP did write "... for more interest amongst people comparing Linux distros". I agree that benchmarks such as this are of little use, but not everyone will see it that way, so we shouldn't just dismiss them. -- Per Jessen, Zürich (2.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org