Created attachment 788309 [details]
journal files

First of all apologies for the title, but I did't know how else to phrase it
since I don't know the exact reasons just yet.

We are observing some slowness in the Leap 15 upstream OpenStack jobs and it
all comes down to journalctl being terribly slow compared to Leap 42.3

Here are some details

== 42.3 ==
journal file: ecb95df5ea41489fa1825f7afd92e4d3/system.journal    2018-11-02
16:09    24M

logs:
2018-11-02 16:11:55.805918 | opensuse-423 | ++ sudo journalctl
--file=/var/log/journal/ecb95df5ea41489fa1825f7afd92e4d3/system.journal
--output=cat
2018-11-02 16:11:55.876491 | opensuse-423 |
2018-11-02 16:11:55.876582 | opensuse-423 | real    0m0.071s
2018-11-02 16:11:55.876611 | opensuse-423 | user    0m0.055s
2018-11-02 16:11:55.876637 | opensuse-423 | sys    0m0.015s
2018-11-02 16:11:55.876765 | opensuse-423 | ++ sudo journalctl
--file=/var/log/journal/ecb95df5ea41489fa1825f7afd92e4d3/system.journal
--output=json-pretty
2018-11-02 16:11:56.697738 | opensuse-423 |
2018-11-02 16:11:56.697909 | opensuse-423 | real    0m0.821s
2018-11-02 16:11:56.697984 | opensuse-423 | user    0m0.687s
2018-11-02 16:11:56.698046 | opensuse-423 | sys    0m0.133s

== 15.0 ==
journal file: 2209f71a310e4383849df3b36e462922/system.journal    2018-11-02
16:15    24M


2018-11-02 16:20:12.898136 | opensuse-150 | ++ sudo journalctl
--file=/var/log/journal/2209f71a310e4383849df3b36e462922/system.journal
--output=cat
2018-11-02 16:20:13.118726 | opensuse-150 |
2018-11-02 16:20:13.118847 | opensuse-150 | real    0m0.222s
2018-11-02 16:20:13.118907 | opensuse-150 | user    0m0.162s
2018-11-02 16:20:13.119013 | opensuse-150 | sys    0m0.059s
2018-11-02 16:20:13.119268 | opensuse-150 | ++ sudo journalctl
--file=/var/log/journal/2209f71a310e4383849df3b36e462922/system.journal
--output=json-pretty
2018-11-02 16:20:16.389780 | opensuse-150 |
2018-11-02 16:20:16.389932 | opensuse-150 | real    0m3.271s
2018-11-02 16:20:16.390028 | opensuse-150 | user    0m2.594s
2018-11-02 16:20:16.390117 | opensuse-150 | sys    0m0.672s
I do realize that the journal files are different but the size of them is the
same

====
As we can see, the same operation in Leap 15 needs +3 seconds. It does not
matter if we use --output=cat or --output=json-pretty.

The OS configuration is the same (rootfs being ext4, both are virtual machines
etc)

Unfortunately I am not able to quickly reproduce this in a Leap 15 vagrant
environment or by loading the same files in my TW system. In both cases
journactl is rather fast. However, the problem is 100% consistent in the
upstream jobs so something must be wrong there.

Let me know if you need be to provide more information. I will also keep
investigating myself.

I have also attached the 2 files.