[opensuse-factory] journalctl very slow
Hi, I'am trying to watch the logs but it's a bit too slow. journalctl is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal? journalctl also consumes 100% CPU the whole time so the machine is useless currently. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 01/11/13 08:26, Ruediger Meier escribió:
Hi,
I'am trying to watch the logs but it's a bit too slow.
Well Known issue, no solution for now. and if I were you I won't be holding my breath for one to be provided anytime soon. (last discussed in early Oct @systemd-devel thread title Fwd: Journalctl performance) This happens almost exclusively when the journal is stored in rotating media. If this bothers you, just set Storage=volatile in in /etc/systemd/journal.conf and if you need to retain the logs for more than the current boot, install rsyslog. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 1 Nov 2013 16:18, Cristian Rodríguez <crrodriguez@...> wrote:
El 01/11/13 08:26, Ruediger Meier escribió:
Hi,
I'am trying to watch the logs but it's a bit too slow.
Well Known issue, no solution for now. and if I were you I won't be holding my breath for one to be provided anytime soon. (last discussed in early Oct @systemd-devel thread title Fwd: Journalctl performance)
This happens almost exclusively when the journal is stored in rotating media.
If this bothers you, just set Storage=volatile in in /etc/systemd/journal.conf and if you need to retain the logs for more than the current boot, install rsyslog.
Niggle: isn't that /etc/systemd/journald.conf ? Also, see: man 5 journald.conf But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year. Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog. - Yamaban.
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default. I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins. -Archie -- Archie L. Cobbs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 01 November 2013, Archie Cobbs wrote:
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins.
Where was already a big discussion before syslog was removed at a time when journald was still even more broken than now. As usualy "the decision has been made" anyway. Sysadmins are not openSUSE targets. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 01/11/13 13:03, Archie Cobbs escribió:
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins.
If you install a traditional syslog implementation there will be no difference.. also people keep repeating "opaqueness and cumbersomeness" but never care to elaborate exactly what they mean in a detailed form and compared to what. That said, embedded developers from Samsung and Intel do care about this slowness and are working on it (much better approach than whining and complaining I must add) -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 11/01/2013 01:03 PM, Cristian Rodríguez pecked at the keyboard and wrote:
El 01/11/13 13:03, Archie Cobbs escribió:
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins.
If you install a traditional syslog implementation there will be no difference.. also people keep repeating "opaqueness and cumbersomeness" but never care to elaborate exactly what they mean in a detailed form and compared to what.
That said, embedded developers from Samsung and Intel do care about this slowness and are working on it (much better approach than whining and complaining I must add)
It has become the means of complaining since the elitist devs won't bother to look at bug reports without just marking them as "Won't Fix" or "to be fixed in a future version (AKA no one knows when)". People have been complaining about logging for a long time and now we're told "too bad, we don't care about sysadmins". -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Fri, 01 Nov 2013 14:03:09 -0300 Cristian Rodríguez <crrodriguez@opensuse.org> пишет:
El 01/11/13 13:03, Archie Cobbs escribió:
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins.
If you install a traditional syslog implementation there will be no difference..
Then it should be default setting in openSUSE until problem with journald slowness is fixed, should not it?
also people keep repeating "opaqueness and cumbersomeness" but never care to elaborate exactly what they mean in a detailed form and compared to what.
That said, embedded developers from Samsung and Intel do care about this slowness and are working on it (much better approach than whining and complaining I must add)
The question was "why it became default when it had obvious problems"; do not pretend you do not see the difference. It is not about development of code, it is about selecting distribution default behavior. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 02/11/13 00:25, Andrey Borzenkov escribió:
The question was "why it became default when it had obvious problems"; do not pretend you do not see the difference. It is not about development of code, it is about selecting distribution default behavior.
Last time I did a clean installation, package systemd-logger was not installed by default (neither is listed to be installed in current opensuse-patterns) therefore no persistent journal, nothing gets written to /var/log/journal..(directory absent) and I had to remove rsyslog in order to only use the journal. (they conflict with each other) zypper in rsyslog Loading repository data... Reading installed packages... Resolving package dependencies... Problem: systemd-logger-208-6.1.x86_64 conflicts with namespace:otherproviders(syslog) provided by rsyslog-7.4.4-2.1.2.x86_64 Solution 1: deinstallation of systemd-logger-208-6.1.x86_64 Solution 2: do not install rsyslog-7.4.4-2.1.2.x86_64 (this is technically wrong but well..) If this has changed somehow.. I do not know.. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andrey Borzenkov <arvidjaar@gmail.com> wrote:
В Fri, 01 Nov 2013 14:03:09 -0300 Cristian Rodríguez <crrodriguez@opensuse.org> пишет:
El 01/11/13 13:03, Archie Cobbs escribió:
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins.
If you install a traditional syslog implementation there will be no difference..
Then it should be default setting in openSUSE until problem with journald slowness is fixed, should not it?
also people keep repeating "opaqueness and
cumbersomeness"
but never care to elaborate exactly what they mean in a detailed form
and compared to what.
That said, embedded developers from Samsung and Intel do care about this slowness and are working on it (much better approach than whining and
complaining I must add)
The question was "why it became default when it had obvious problems"; do not pretend you do not see the difference. It is not about development of code, it is about selecting distribution default behavior.
Since it is relatively small, is there a way to page it into ram before starting to parse it. Ie. Add a patch to do a linear read of the entire fire before starting the random I/O. For systems with some spare ram that should drastically accelerate parsing the file and for systems tight in ram it would not hurt much. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 2013-11-02 05:27, Greg Freemyer wrote:
Since it is relatively small, is there a way to page it into ram before starting to parse it. Ie. Add a patch to do a linear read of the entire fire before starting the random I/O. For systems with some spare ram that should drastically accelerate parsing the file and for systems tight in ram it would not hurt much.
How small is small? And if it's "big enough", it will be evicted from the cache sooner than you think. So no, that does not help. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 02/11/13 00:25, Andrey Borzenkov escribió:
The question was "why it became default when it had obvious problems"; do not pretend you do not see the difference. It is not about development of code, it is about selecting distribution default behavior.
Ok, I just picked openSUSE 13.1 RC2 KDE Live and installed it on a virtual machine - rsyslog is installed by default. - there is no systemd-logger installed, therefore no /var/log/journal, therefore no persistent journal. This is exactly like there was before, the default has not been changed, nothing needs to be documented in the release notes. etc. I do not know where people got this idea in the first place. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
* Cristian Rodríguez <crrodriguez@opensuse.org> [2013-11-02 05:32]:
El 02/11/13 00:25, Andrey Borzenkov escribió:
The question was "why it became default when it had obvious problems"; do not pretend you do not see the difference. It is not about development of code, it is about selecting distribution default behavior.
Ok, I just picked openSUSE 13.1 RC2 KDE Live and installed it on a virtual machine
- rsyslog is installed by default. - there is no systemd-logger installed, therefore no /var/log/journal, therefore no persistent journal.
This is exactly like there was before, the default has not been changed, nothing needs to be documented in the release notes. etc.
I do not know where people got this idea in the first place.
https://github.com/openSUSE/patterns/pull/58 -- Guido Berhoerster -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 02/11/13 06:12, Guido Berhoerster escribió:
Yeah, not merged yet.. it says "after 13.1" -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 01 November 2013, Cristian Rodríguez wrote:
El 01/11/13 08:26, Ruediger Meier escribió:
Hi,
I'am trying to watch the logs but it's a bit too slow.
Well Known issue, no solution for now. and if I were you I won't be holding my breath for one to be provided anytime soon. (last discussed in early Oct @systemd-devel thread title Fwd: Journalctl performance)
How could this journald logging daemon got released and (used in openSUSE per default) if it's not possible to view the logs at all?
This happens almost exclusively when the journal is stored in rotating media.
If this bothers you, just set Storage=volatile in in /etc/systemd/journal.conf and if you need to retain the logs for more than the current boot, install rsyslog.
I have rsyslog installed but I've been told to watch the journal to debug initrd problems. But this is actually not possible. BTW my production machines (still all 11.4) have average uptimes between 200 and 400 days. So I expect that the "current boot logs" would be as big as this one on my test machine. The journal disk-usage is currently 1.3G, which is far to much. The same as compressed text files would be probably about 20M (factor 60!). It really does not make sense that syslog was removed per default and replaced by such a monster which is using 60 times more space and we are not even able to view that log unless we want to wait a whole week while the machine is unusable on 100% load. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 01/11/13 13:02, Ruediger Meier escribió:
On Friday 01 November 2013, Cristian Rodríguez wrote:
How could this journald logging daemon got released and (used in openSUSE per default) if it's not possible to view the logs at all?
The logs can be viewed, it is just slow with non-SSD drives.. it has to be used by default because it is mandatory ! I have explained this I do not know how many times.
I have rsyslog installed but I've been told to watch the journal to debug initrd problems. But this is actually not possible.
ALL messages that the journal captures are forwarded to the syslog implementation if installed.
It really does not make sense that syslog was removed per default
according to the patterns file, rsyslog is still a suggested package, it is not required however. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 1 Nov 2013 17:43, Cristian Rodríguez <crrodriguez@...> wrote:
El 01/11/13 13:02, Ruediger Meier escribió:
On Friday 01 November 2013, Cristian Rodríguez wrote:
How could this journald logging daemon got released and (used in openSUSE per default) if it's not possible to view the logs at all?
The logs can be viewed, it is just slow with non-SSD drives.. it has to be used by default because it is mandatory ! I have explained this I do not know how many times.
Could we put a short info on that (journald slow with non-SSD drives) and the "should have" package rsyslog or similar for server use into the Release Notes, for the search term "logging" ?
I have rsyslog installed but I've been told to watch the journal to debug initrd problems. But this is actually not possible.
ALL messages that the journal captures are forwarded to the syslog implementation if installed.
Also well worth mentioning, see above. - Yamaban
On Friday 2013-11-01 17:57, Yamaban wrote:
On Fri, 1 Nov 2013 17:43, Cristian Rodríguez <crrodriguez@...> wrote:
El 01/11/13 13:02, Ruediger Meier escribió:
On Friday 01 November 2013, Cristian Rodríguez wrote:
How could this journald logging daemon got released and (used in openSUSE per default) if it's not possible to view the logs at all?
The logs can be viewed, it is just slow with non-SSD drives.. it has to be used by default because it is mandatory ! I have explained this I do not know how many times.
Could we put a short info on that (journald slow with non-SSD drives) and the "should have" package rsyslog or similar for server use into the Release Notes, for the search term "logging" ?
Even with rsyslog, `systemctl status` searches the journal which can take noticable time. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 1 Nov 2013 17:57:28 +0100 (CET) Yamaban <foerster@lisas.de> wrote:
Could we put a short info on that (journald slow with non-SSD drives) and the "should have" package rsyslog or similar for server use into the Release Notes, for the search term "logging" ?
Release Notes is better handled via bugzilla (for now). Product: openSUSE 13.1 Component: Release Notes That way documentation guys will get message. For cross reference, you can list bug report URL here, and thread reporting proposal for change in the bug report. -- Regards, Rajko. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Cristian Rodríguez wrote:
El 01/11/13 13:02, Ruediger Meier escribió:
On Friday 01 November 2013, Cristian Rodríguez wrote:
How could this journald logging daemon got released and (used in openSUSE per default) if it's not possible to view the logs at all?
The logs can be viewed, it is just slow with non-SSD drives.. it has to be used by default because it is mandatory ! I have explained this I do not know how many times.
I have rsyslog installed but I've been told to watch the journal to debug initrd problems. But this is actually not possible.
ALL messages that the journal captures are forwarded to the syslog implementation if installed.
It really does not make sense that syslog was removed per default
according to the patterns file, rsyslog is still a suggested package, it is not required however.
https://bugzilla.novell.com/show_bug.cgi?id=843657 -- Per Jessen, Zürich (16.4°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 01/11/13 13:02, Ruediger Meier escribió:
The journal disk-usage is currently 1.3G, which is far to much.
Far too much..in what kind of system .. do you have harddisks from the 90's ? in 2011, the average *consumer* hard drive was 590 GB.. increasing size at a rate of 39% per year. The same
as compressed text files would be probably about 20M (factor 60!).
The journal is already compressed (XZ is used.. ) it is of course much bigger than a text file because it has metadata attached. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 01 November 2013, Cristian Rodríguez wrote:
El 01/11/13 13:02, Ruediger Meier escribió:
The journal disk-usage is currently 1.3G, which is far to much.
Far too much..in what kind of system .. do you have harddisks from the 90's ? in 2011, the average *consumer* hard drive was 590 GB.. increasing size at a rate of 39% per year.
How ignorant and blind are you ... Journalctl showing the logs with 2K/s - that's 60's style In 2013 systemd-journal (the default in most distros :) needs a week to show such a 1.3G journal ... it can't even handle 90's file sizes but you complain that users don't always want to keep their / small. vm servers are hosting dozens or hundreds of journald "powered" systems. They all need much more space / network bandwidth / traffic / time / energy to backup. This is one of our typical productive openSUSE 11.4 installations: Filesystem Size Used Avail Use% Mounted on rootfs 3.0G 1.1G 1.8G 38% / devtmpfs 243M 88K 243M 1% /dev tmpfs 246M 4.0K 246M 1% /dev/shm /dev/vda1 3.0G 1.1G 1.8G 38% / Upgrading to journald per default means either only keeping logs for a few last days or double hd space. We have rpmlint complaining about a few KB documentation in non-doc packages (and much more similar warnings). Not even significant when journald is in use. But not allowed to complain about a journal wasting giga bytes? Even you always try to convince us that that tmpfs (maybe < 4-8GB on average systems) are enough /tmp for everybody. How does this match? /var/log has to be 201x style but /tmp must be 90' style? We have still many NFS client desktops with <80GB HD where / is ~20GB (90% full) and thre rest is local /tmp storage for the developers and number crunchers. Each wasted GB would be painful for our users and then for our NFS server.
The same
as compressed text files would be probably about 20M (factor 60!).
The journal is already compressed (XZ is used.. ) it is of course much bigger than a text file because it has metadata attached.
Haha. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 01/11/13 15:19, Ruediger Meier escribió:
On Friday 01 November 2013, Cristian Rodríguez wrote:
El 01/11/13 13:02, Ruediger Meier escribió:
The journal disk-usage is currently 1.3G, which is far to much.
Far too much..in what kind of system .. do you have harddisks from the 90's ? in 2011, the average *consumer* hard drive was 590 GB.. increasing size at a rate of 39% per year.
How ignorant and blind are you ...
So you disagree with me..:-)
Journalctl showing the logs with 2K/s - that's 60's style
In 2013 systemd-journal (the default in most distros :) needs a week to show such a 1.3G journal ... it can't even handle 90's file sizes but you complain that users don't always want to keep their / small.
if 1.3G is too big for your usecase, you can configure the maximum size that the journal can use on disk, or configure it to only retain the logs of the current boot at a particular limit (embedded developers do this all the time as they have limited ram, slower media) or set the Journal storage to none, which is not recommended and then you must have a syslog implementation installed .you will loose pretty much all systemd functionality that depends on logging.
We have rpmlint complaining about a few KB documentation in non-doc packages (and much more similar warnings). Not even significant when journald is in use. But not allowed to complain about a journal wasting giga bytes?
Big difference, the size of the journal does not affect the size of the installation media, that's what rpmlint cares about.
Even you always try to convince us that that tmpfs (maybe < 4-8GB on average systems) are enough /tmp for everybody.
Not for everybody, but for the average use, it is not the default and has nothing to do with journal either. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Nov 1, 2013 at 3:19 PM, Ruediger Meier <sweet_f_a@gmx.de> wrote:
On Friday 01 November 2013, Cristian Rodríguez wrote:
El 01/11/13 13:02, Ruediger Meier escribió:
The journal disk-usage is currently 1.3G, which is far to much.
Far too much..in what kind of system .. do you have harddisks from the 90's ? in 2011, the average *consumer* hard drive was 590 GB.. increasing size at a rate of 39% per year.
How ignorant and blind are you ...
Journalctl showing the logs with 2K/s - that's 60's style
In 2013 systemd-journal (the default in most distros :) needs a week to show such a 1.3G journal ... it can't even handle 90's file sizes but you complain that users don't always want to keep their / small.
This is because journal is using memory-mapped access. Memory-mapped access is terribly bad at predicting I/O patterns. This may be a kernel issue more than a journal issue, but the journal can help the kernel predict better. I suggest monitorion I/O with iostat to see what the issue is. There should be a lot of read-request-merges since watching a journal log should be sequential access. If read-request-merges are low and avg request sizes too, then it's probably fragmentation. I'll be happy to provide a few patches for you to try trying to address that, if you're willing to rebuild and test. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sun, Nov 3, 2013 at 3:42 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
I suggest monitorion I/O with iostat to see what the issue is. There should be a lot of read-request-merges since watching a journal log should be sequential access. If read-request-merges are low and avg request sizes too, then it's probably fragmentation.
I'll be happy to provide a few patches for you to try trying to address that, if you're willing to rebuild and test.
Or better yet. Send me the journal file somehow? That way I can test myself. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sunday 03 November 2013, Claudio Freire wrote:
On Sun, Nov 3, 2013 at 3:42 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
I suggest monitorion I/O with iostat to see what the issue is. There should be a lot of read-request-merges since watching a journal log should be sequential access. If read-request-merges are low and avg request sizes too, then it's probably fragmentation.
I'll be happy to provide a few patches for you to try trying to address that, if you're willing to rebuild and test.
Or better yet. Send me the journal file somehow?
That way I can test myself.
Thanks a lot for your efforts but unfortunately I had removed that useless journal stuff already. Actually I also don't really believe that you could accelerate it by factor 1000 to make it at least a bit more useful. Maybe it's this hopeless looking situation why upstream does not "want" to fix that, instead telling us that our needs or hardware is wrong. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Nov 5, 2013 at 8:15 PM, Ruediger Meier <sweet_f_a@gmx.de> wrote:
On Sunday 03 November 2013, Claudio Freire wrote:
On Sun, Nov 3, 2013 at 3:42 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
I suggest monitorion I/O with iostat to see what the issue is. There should be a lot of read-request-merges since watching a journal log should be sequential access. If read-request-merges are low and avg request sizes too, then it's probably fragmentation.
I'll be happy to provide a few patches for you to try trying to address that, if you're willing to rebuild and test.
Or better yet. Send me the journal file somehow?
That way I can test myself.
Thanks a lot for your efforts but unfortunately I had removed that useless journal stuff already. Actually I also don't really believe that you could accelerate it by factor 1000 to make it at least a bit more useful. Maybe it's this hopeless looking situation why upstream does not "want" to fix that, instead telling us that our needs or hardware is wrong.
Well, without knowing what the bottleneck is, it's hard to tell. My money was either I/O slowness due to fragmentation (easy to fix with a posix_fallocate call) or repeated decompression overhead (fixed not so easily but still easy-ish with a block cache). With a patch that addresses the bottleneck, I bet upstream would pay attention. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-11-01 at 12:18 -0300, Cristian Rodríguez wrote:
This happens almost exclusively when the journal is stored in rotating media.
If this bothers you, just set Storage=volatile in in /etc/systemd/journal.conf and if you need to retain the logs for more than the current boot, install rsyslog.
Eleanor4:~ # less /etc/systemd/journal.conf /etc/systemd/journal.conf: No such file or directory Eleanor4:~ # Eleanor4:~ # cat /etc/os-release NAME=openSUSE VERSION="13.1 (Bottle)" VERSION_ID="13.1" PRETTY_NAME="openSUSE 13.1 (Bottle) (x86_64)" ID=opensuse ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:opensuse:13.1" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://opensuse.org/" ID_LIKE="suse" Eleanor4:~ # Eleanor4:~ # rpm -q rsyslog rsyslog-7.4.4-2.1.2.x86_64 Eleanor4:~ # Eleanor4:~ # locate journal.conf Eleanor4:~ # locate -i journal.conf Eleanor4:~ # Default install... - -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlJz8aIACgkQtTMYHG2NR9VMtACfXdyIqIGFnHyngGe7shxXXx4E CLkAn03aUyx04lzFH+g626gY7Q/3GS6C =WyNL -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-11-01 at 19:23 +0100, Carlos E. R. wrote:
On Friday, 2013-11-01 at 12:18 -0300, Cristian Rodríguez wrote:
If this bothers you, just set Storage=volatile in in /etc/systemd/journal.conf and if you need to retain the logs for more than the current boot, install rsyslog.
Eleanor4:~ # less /etc/systemd/journal.conf /etc/systemd/journal.conf: No such file or directory
It is /etc/systemd/journald.conf - -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlJz8mwACgkQtTMYHG2NR9W8OgCfedOM3aQB06Ugl7kbO4PJRRhl HMkAoIfnlu8LQk7PDNWYBNe4qN8uwJR1 =xbUC -----END PGP SIGNATURE-----
El 01/11/13 15:26, Carlos E. R. escribió:
If this bothers you, just set Storage=volatile in in /etc/systemd/journal.conf and if you need to retain the logs for more than the current boot, install rsyslog.
Eleanor4:~ # less /etc/systemd/journal.conf /etc/systemd/journal.conf: No such file or directory
It is /etc/systemd/journald.conf
Yes, I missed a "d" in the mail, I guess people can figure that out themselves no ? ;-P -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 01.11.2013 12:26, schrieb Ruediger Meier:
Hi,
I'am trying to watch the logs but it's a bit too slow.
journalctl
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk? Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;) -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 1 Nov 2013 23:45, Stefan Seyfried <stefan.seyfried@...> wrote:
Am 01.11.2013 12:26, schrieb Ruediger Meier:
Hi,
I'am trying to watch the logs but it's a bit too slow.
journalctl
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Shall we hope you've used the [sarcasm] tag for your reply? TBH, I've been looking at the journald code wrt the on disk ops, and, my impression was: Somebody took code for a strict In-RAM data structure and used that for the On-Disk ops. IMHO a no-go for any 'real' core OS developer, but, well, look at who's in charge of that project upstream.... For me, the journald storage code is a thing to be replaced, at whole. Linus would never allow that level of code quality anywhere in the Kernel tree. No other syslog implementation is done that bad. And, no, I'm no C / C++ programmer, I'll keep my hand off that. - Yamaban. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 01/11/13 20:05, Yamaban escribió:
IMHO a no-go for any 'real' core OS developer, but, well, look at who's in charge of that project upstream....
And that's your argument, an ad-homimem attack.. all nicely bundled with a "not true scotsman fallacy" .,. is that the best you can do ?
Linus would never allow that level of code quality anywhere in the Kernel tree. Linus has no authority in userspace, he is just another guy with an opinion..
No other syslog implementation is done that bad.
The journal is not a syslog implementation. comparing pears with apples. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 11/01/13 19:17, Cristian Rodríguez pecked at the keyboard and wrote:
El 01/11/13 20:05, Yamaban escribió:
IMHO a no-go for any 'real' core OS developer, but, well, look at who's in charge of that project upstream....
And that's your argument, an ad-homimem attack.. all nicely bundled with a "not true scotsman fallacy" .,. is that the best you can do ?
Linus would never allow that level of code quality anywhere in the Kernel tree. Linus has no authority in userspace, he is just another guy with an opinion..
No other syslog implementation is done that bad.
The journal is not a syslog implementation. comparing pears with apples.
Touchy touchy are we. When a piece of software is crap it is crap. No need for a PhD here to see the results. -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Nov 1, 2013 at 8:17 PM, Cristian Rodríguez <crrodriguez@opensuse.org> wrote:
IMHO a no-go for any 'real' core OS developer, but, well, look at who's in charge of that project upstream....
And that's your argument, an ad-homimem attack.. all nicely bundled with a "not true scotsman fallacy" .,. is that the best you can do ?
Don't be negligently dismissive. He's on target. It's a bad data structure for disk access, it was a bad idea, and it's just a matter of the devs getting the necessary years of woes to be able to give up their egos and accept that. Nevertheless, my previous mail stands. It's possible to improve it without much effort, by telling the kernel what to expect. On Fri, Nov 1, 2013 at 3:53 PM, Cristian Rodríguez <crrodriguez@opensuse.org> wrote:
Journalctl showing the logs with 2K/s - that's 60's style
In 2013 systemd-journal (the default in most distros :) needs a week to show such a 1.3G journal ... it can't even handle 90's file sizes but you complain that users don't always want to keep their / small.
if 1.3G is too big for your usecase, you can configure the maximum size that the journal can use on disk, or configure it to only retain the logs of the current boot at a particular limit (embedded developers do this all the time as they have limited ram, slower media) or set the Journal storage to none, which is not recommended and then you must have a syslog implementation installed .you will loose pretty much all systemd functionality that depends on logging.
I think the amount of metadata should also be configurable, if that's the main reason for bloat. I suspect it's not the case though. Also, having the journal, a memory-mapped write-heavy structure, on an SSD, is a good way to burn it prematurely. On Sat, Nov 2, 2013 at 12:25 AM, Andrey Borzenkov <arvidjaar@gmail.com> wrote:
El 01/11/13 13:03, Archie Cobbs escribió:
On Fri, Nov 1, 2013 at 10:38 AM, Yamaban <foerster@lisas.de> wrote:
But, yes, clear upstream trouble, that is: ignored by those in charge, sounds of '... rotating media are a dying type ...' where made, when asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install a full 'traditional' syslog.
Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more opaque and cumbersome. Logs are like bread and water to sysadmins.
If you install a traditional syslog implementation there will be no difference..
Then it should be default setting in openSUSE until problem with journald slowness is fixed, should not it?
Yes. I think it is, in fact. Or at least seemed to last time I installed 13.1 RC1. Will have to double-check. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2013-11-01 23:45, Stefan Seyfried wrote:
Am 01.11.2013 12:26, schrieb Ruediger Meier:
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Bad programming, IMHO. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Sat, Nov 02, 2013 at 12:17:26AM +0100, Carlos E. R. wrote:
On 2013-11-01 23:45, Stefan Seyfried wrote:
Am 01.11.2013 12:26, schrieb Ruediger Meier:
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Bad programming, IMHO.
Really? Patches are always gladly accepted upstream for this, and numerous people have looked at it to try to firstly determine what exactly the problem is, and no one has been able to figure that out. Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.) So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring... greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 02 November 2013, Greg KH wrote:
On Sat, Nov 02, 2013 at 12:17:26AM +0100, Carlos E. R. wrote:
On 2013-11-01 23:45, Stefan Seyfried wrote:
Am 01.11.2013 12:26, schrieb Ruediger Meier:
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Bad programming, IMHO.
Really? Patches are always gladly accepted upstream for this, and numerous people have looked at it to try to firstly determine what exactly the problem is, and no one has been able to figure that out. Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
Yes probably the design is more bad than the actual implementation. Anyway, it was very easily predictable that we would get problems like this. All I complain about is that we (the users/admins) get such young, untested, unstable, broken stuff per default. There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work? cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Nov 02, 2013 at 07:01:03PM +0100, Ruediger Meier wrote:
On Saturday 02 November 2013, Greg KH wrote:
On Sat, Nov 02, 2013 at 12:17:26AM +0100, Carlos E. R. wrote:
On 2013-11-01 23:45, Stefan Seyfried wrote:
Am 01.11.2013 12:26, schrieb Ruediger Meier:
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Bad programming, IMHO.
Really? Patches are always gladly accepted upstream for this, and numerous people have looked at it to try to firstly determine what exactly the problem is, and no one has been able to figure that out. Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
Yes probably the design is more bad than the actual implementation.
Anyway, it was very easily predictable that we would get problems like this. All I complain about is that we (the users/admins) get such young, untested, unstable, broken stuff per default.
Yes, we all write crappy software and have no idea what we are doing and should listen to everyone who tells us to stop because they are the ones who know best. Give me a break.
There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work?
It works just fine for me here, on three other distros who have been shipping it for a long time (Fedora and Arch and Gentoo). If you really hate this type of thing, and don't want progress, there's always Slackware. As it is, openSUSE is already playing catch-up with those distros, how far behind do you want to see us be? greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Greg KH wrote:
On Sat, Nov 02, 2013 at 07:01:03PM +0100, Ruediger Meier wrote: [snip]
There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work?
It works just fine for me here, on three other distros who have been shipping it for a long time (Fedora and Arch and Gentoo). If you really hate this type of thing, and don't want progress, there's always Slackware. As it is, openSUSE is already playing catch-up with those distros, how far behind do you want to see us be?
Playing catch-up is fine with me, trailing edge is a much safer place to be. Anyway, what does our mission statement say? -- Per Jessen, Zürich (12.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Nov 02, 2013 at 08:52:33PM +0100, Per Jessen wrote:
Greg KH wrote:
On Sat, Nov 02, 2013 at 07:01:03PM +0100, Ruediger Meier wrote: [snip]
There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work?
It works just fine for me here, on three other distros who have been shipping it for a long time (Fedora and Arch and Gentoo). If you really hate this type of thing, and don't want progress, there's always Slackware. As it is, openSUSE is already playing catch-up with those distros, how far behind do you want to see us be?
Playing catch-up is fine with me, trailing edge is a much safer place to be. Anyway, what does our mission statement say?
"Have a lot of fun..." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Greg KH wrote:
On Sat, Nov 02, 2013 at 08:52:33PM +0100, Per Jessen wrote:
Greg KH wrote:
On Sat, Nov 02, 2013 at 07:01:03PM +0100, Ruediger Meier wrote: [snip]
There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work?
It works just fine for me here, on three other distros who have been shipping it for a long time (Fedora and Arch and Gentoo). If you really hate this type of thing, and don't want progress, there's always Slackware. As it is, openSUSE is already playing catch-up with those distros, how far behind do you want to see us be?
Playing catch-up is fine with me, trailing edge is a much safer place to be. Anyway, what does our mission statement say?
"Have a lot of fun..."
Haha, good point! That's not the whole answer though. One of these days we ought to address that question thoroughly. -- Per Jessen, Zürich (7.4°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 02 November 2013, Greg KH wrote:
On Sat, Nov 02, 2013 at 07:01:03PM +0100, Ruediger Meier wrote:
On Saturday 02 November 2013, Greg KH wrote:
On Sat, Nov 02, 2013 at 12:17:26AM +0100, Carlos E. R. wrote:
On 2013-11-01 23:45, Stefan Seyfried wrote:
Am 01.11.2013 12:26, schrieb Ruediger Meier:
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Bad programming, IMHO.
Really? Patches are always gladly accepted upstream for this, and numerous people have looked at it to try to firstly determine what exactly the problem is, and no one has been able to figure that out. Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
Yes probably the design is more bad than the actual implementation.
Anyway, it was very easily predictable that we would get problems like this. All I complain about is that we (the users/admins) get such young, untested, unstable, broken stuff per default.
Yes, we all write crappy software and have no idea what we are doing and should listen to everyone who tells us to stop because they are the ones who know best.
Give me a break.
There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work?
It works just fine for me here, on three other distros who have been shipping it for a long time (Fedora and Arch and Gentoo).
Does Gentoo and Arch really force you to switch to systemd yet?
If you really hate this type of thing, and don't want progress, there's always Slackware. As it is, openSUSE is already playing catch-up with those distros, how far behind do you want to see us be?
I don't say something against shipping "progress" but against changing well tested and stable defaults far to early. For example there was absolutely no need that upgrading from to 12.1 or 12.2 replaced sysvinit by systemd. There would have been enough pro-active testers. No need to bother everybody. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Nov 02, 2013 at 10:35:49PM +0100, Ruediger Meier wrote:
It works just fine for me here, on three other distros who have been shipping it for a long time (Fedora and Arch and Gentoo).
Does Gentoo and Arch really force you to switch to systemd yet?
Arch switched a long time ago, and Gentoo makes you do it if you want to run GNOME, which is fine with me.
If you really hate this type of thing, and don't want progress, there's always Slackware. As it is, openSUSE is already playing catch-up with those distros, how far behind do you want to see us be?
I don't say something against shipping "progress" but against changing well tested and stable defaults far to early.
Please define "early" in a way that is sufficient for everyone.
For example there was absolutely no need that upgrading from to 12.1 or 12.2 replaced sysvinit by systemd. There would have been enough pro-active testers. No need to bother everybody.
How much "bother" was it really? And how would delaying cause that "bother" to go away except to postpone it? Anyway, that's not the issue here at all, we aren't going to rehash the systemd issue anymore. Again, if you don't like it, there are still distros out there without systemd, feel free to use one of them if you like. I hear some people like Ubuntu these days... Best of luck, greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 02/11/13 15:01, Ruediger Meier escribió:
There is already a distribution (Fedora) which is testing all this alpha software. They have never cared about compatibility or the upgrade scenario - that's ok because you know that. Why openSUSE does not just wait until at least one distribution exists where all this funny freedesktop stuff works as it's supposed too work?
In that case we will not be able to influence any design decision, any of the features, etc. essentially we will be the last to attendant to arrive to the party and will have to plumb our way up in the most painful manner in existence. In reality, we cannot afford to be late, due to a number of factors, including resource constrains. If you expect things to work the way you just told, I am very sorry to inform you that things will get much, much worst for you in the near future and you might want to consider a dying OS family like the BSDs. Previously I would have recommended you Slackware, but recently Pat V stated that he is not ruling out switching to systemd if necessary, however he did rule out going to openRC. -- "Judging by their response, the meanest thing you can do to people on the Internet is to give them really good software for free". - Anil Dash -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 02.11.2013 17:03, schrieb Greg KH:
Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
Just try to use it on old (slow) rotating rust. 100% reproducible on all my 2.5" IDE HDDs here. If I were to guess, I'd vote it's just horribly fragmenting the files: shake -vvv shows thousands of fragments, probably due to lots of holes in the database files.
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
The developers of core system software should be forced to use 10 year old hardware to test their creation in daily use ;-) -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 02 November 2013, Stefan Seyfried wrote:
Am 02.11.2013 17:03, schrieb Greg KH:
Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
Just try to use it on old (slow) rotating rust. 100% reproducible on all my 2.5" IDE HDDs here.
BTW my problem does not looked like HD speed issue. journalctl was constantly on 100% CPU. So I assume that it couldn't handle more input data in the same time (unless it's even more broken than thought).
If I were to guess, I'd vote it's just horribly fragmenting the files: shake -vvv shows thousands of fragments, probably due to lots of holes in the database files.
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
The developers of core system software should be forced to use 10 year old hardware to test their creation in daily use ;-) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Nov 2, 2013 at 3:38 PM, Ruediger Meier <sweet_f_a@gmx.de> wrote:
On Saturday 02 November 2013, Stefan Seyfried wrote:
Am 02.11.2013 17:03, schrieb Greg KH:
Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
Just try to use it on old (slow) rotating rust. 100% reproducible on all my 2.5" IDE HDDs here.
BTW my problem does not looked like HD speed issue. journalctl was constantly on 100% CPU. So I assume that it couldn't handle more input data in the same time (unless it's even more broken than thought).
Oh, about that, taking a look at perf-top could also give a hint. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Nov 02, 2013 at 07:06:35PM +0100, Stefan Seyfried wrote:
Am 02.11.2013 17:03, schrieb Greg KH:
Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
Just try to use it on old (slow) rotating rust. 100% reproducible on all my 2.5" IDE HDDs here.
If I were to guess, I'd vote it's just horribly fragmenting the files: shake -vvv shows thousands of fragments, probably due to lots of holes in the database files.
What filesystem are you using? What is the % used of that partition? I did try this on rust, but not a really slow one, I'll dig up an old disk and try that next time I can.
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
The developers of core system software should be forced to use 10 year old hardware to test their creation in daily use ;-)
Then nothing would ever get written because we would all be taking days to build our kernels... And you wouldn't get new features for that new hardware (transactional memory, huge memory sizes, large numbers of cpus, etc.) You can't have it both ways, sorry. If you want, just run 10 year old software on that hardware, it should work just fine. greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 02.11.2013 20:00, schrieb Greg KH:
On Sat, Nov 02, 2013 at 07:06:35PM +0100, Stefan Seyfried wrote:
If I were to guess, I'd vote it's just horribly fragmenting the files: shake -vvv shows thousands of fragments, probably due to lots of holes in the database files.
What filesystem are you using? What is the % used of that partition?
ext3 and ext4, between 50% to 90% full.
I did try this on rust, but not a really slow one, I'll dig up an old disk and try that next time I can.
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
The developers of core system software should be forced to use 10 year old hardware to test their creation in daily use ;-)
Then nothing would ever get written because we would all be taking days to build our kernels...
compiling is allowed on current hardware ;-)
And you wouldn't get new features for that new hardware (transactional memory, huge memory sizes, large numbers of cpus, etc.) You can't have it both ways, sorry.
If you want, just run 10 year old software on that hardware, it should work just fine.
I'll probably just go for busybox-init/mdev instead of systemd/udev on those machines -- boots exactly as fast but is much less demanding on the hardware (and on admin-maintainance). And the feature set is good enough. My work machine has now gotten a SSD due to the "systemctl status foo.service" taking ages (and "git status" in kernel trees, too) For my wife's old machine, I recently bought an IDE-SSD to get it to boot in an acceptabe time frame with current 13.1. That's of course not only systemd's fault, the whole userspace stuff is getting much fatter all the time. Have fun, seife -- Stefan Seyfried "If your lighter runs out of fluid or flint and stops making fire, and you can't be bothered to figure out about lighter fluid or flint, that is not Zippo's fault." -- bkw -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Nov 02, 2013 at 10:36:38PM +0100, Stefan Seyfried wrote:
Am 02.11.2013 20:00, schrieb Greg KH:
On Sat, Nov 02, 2013 at 07:06:35PM +0100, Stefan Seyfried wrote:
If I were to guess, I'd vote it's just horribly fragmenting the files: shake -vvv shows thousands of fragments, probably due to lots of holes in the database files.
What filesystem are you using? What is the % used of that partition?
ext3 and ext4, between 50% to 90% full.
Ok, that helps, both of these, when things start to get full start thrashing in well-known ways to find free blocks. thanks, greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 2013-11-02 22:36, Stefan Seyfried wrote:
My work machine has now gotten a SSD due to the "systemctl status foo.service" taking ages (and "git status" in kernel trees, too)
For my wife's old machine, I recently bought an IDE-SSD to get it to boot in an acceptabe time frame with current 13.1. That's of course not only systemd's fault, the whole userspace stuff is getting much fatter all the time.
Yeah, let's go back to the DOS times, where there was just 60 or so programs in C:\DOS (something like /bin). Tab completion would complete in a blink — man, where have we developed toward. Well, with SYSV scripts going away thanks to systemd, you save the overhead of interpreting (repetitive) sh code. So now, your boot phase should be more IO-bound, and SSD is a good investment in that regard. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2013-11-02 at 09:03 -0700, Greg KH wrote:
On Sat, Nov 02, 2013 at 12:17:26AM +0100, Carlos E. R. wrote:
That's only 4K/s. Is this normal?
Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-) The file fragmentation of the journal files and the performance of the "database" is absolutely totally horrible. It gets slightly better once you dedicate a fast SSD to the journal ;)
Bad programming, IMHO.
Really? Patches are always gladly accepted upstream for this, and numerous people have looked at it to try to firstly determine what exactly the problem is, and no one has been able to figure that out. Including the fact that most of us who have looked at it, can't even duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem for a small number of systems that people are not ignoring...
Maybe you need expert database engine programmers in the team. I'm not. If programmers are used to excellent and modern hardware, you can not make code that works well with the kind of hardware normal people use. This happened to me, as a programmer... it is not idle talk. Remember that one of the sale points of Linux was that it brings new life to older hardware that no longer works well with the last Windows version. If you insist that openSUSE must run on SSD media, you are breaking contact with the masses of Linux users out there. - -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlJ1eBMACgkQtTMYHG2NR9UxrACfRAKyrXZ9YjXhQXXNJN2S5ROX O/cAni4CA69vw8v12mJAYbwO6MkwSpoA =4DOU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Friday 01 November 2013 13.26:14 Ruediger Meier wrote:
Hi,
I'am trying to watch the logs but it's a bit too slow.
journalctl
is running since 8 hours now and it has printed only 120MB text yet. That's only 4K/s. Is this normal? journalctl also consumes 100% CPU the whole time so the machine is useless currently.
cu, Rudi
/me just loving journald + systemd-logger journalctl is a perfect tool excepted it slowlyness (even on ssd) but you certainly don't want more than since the last boot so using -b fix for 98% my use case. syntax color, search, bold, export in text file etc ... Thanks to have invented that tool! -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (18)
-
Andrey Borzenkov -
Archie Cobbs -
Bruno Friedmann -
Carlos E. R. -
Carlos E. R. -
Claudio Freire -
Cristian Rodríguez -
Greg Freemyer -
Greg KH -
Greg KH -
Guido Berhoerster -
Jan Engelhardt -
Ken Schneider - openSUSE -
Per Jessen -
Rajko -
Ruediger Meier -
Stefan Seyfried -
Yamaban