[oS-en] I need to purge the journal for a bugzilla
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I need to purge the output from journalctl before posting it in a bugzilla. I want to remove mail and news entries. News entries add up to megabytes, and mail is private. I do: journalctl --boot=-2 | grep -v "texpire\[" | grep -v "fetchnews\[" > journal_nonews But this doesn't work on multiline reports like these: Feb 18 12:32:00 Telcontar fetchnews[15388]: sent ARTICLE 321110 command, in pipe: 2 Feb 18 12:32:00 Telcontar fetchnews[15388]: sent ARTICLE 321111 command, in pipe: 3 Feb 18 12:32:00 Telcontar fetchnews[15388]: store: try filters on header "Path: uni-berlin.de!fu-berlin.de!eternal-september.org!feeder3.eternal-september.org!news.et> From: Boris <Boris@invalid.invalid> Newsgroups: alt.comp.hardware Subject: Re: How Do SSDs Wear Out? Date: Tue, 18 Feb 2025 01:42:22 -0000 (UTC) Organization: A noiseless patient Spider Lines: 213 Message-ID: <XnsB289B41E2295ABorisinvalidinvalid@135.181.20.170> References: <XnsB285DDB4AC76FBorisinvalidinvalid@135.181.20.170> <vomtr3$3cnqe$1@dont-email.me> Injection-Date: Tue, 18 Feb 2025 02:42:23 +0100 (CET) Injection-Info: dont-email.me; posting-host="c64b607c5d3255373c7670fad39a075e"; logging-data="1481251"; mail-compla> User-Agent: Xnews/2006.08.24 Cancel-Lock: sha1:mPni5aIW+7kzN+kq1TNpc4DGcSg= Xref: uni-berlin.de alt.comp.hardware:321109 " Feb 18 12:32:00 Telcontar fetchnews[15388]: killfilter: trying filter for .* Feb 18 12:32:00 Telcontar fetchnews[15388]: pcre filter: /^From:.*APKr@domein\.nl.*/ did not match How can I remove an entire "facility" from journalctl? I did not find that in the manual. I see how to include a facility, but not how to remove one. Or, a grep recipe that cleans a multiline entry. I would use the syslog instead, which is already filtered, but it appears that the syslog doesn't contain all boot and kernel messages. I googled. Found <https://askubuntu.com/questions/1464751/telling-journalctl-to-show-all-but-a-certain-facility> where it seems the way is to do: journalctl --facility=kern,user,mail,....,local7 listing all facilities except mail and news. So I try: journalctl --boot=-2 --facility=auth,authpriv,cron,daemon,ftp,kern,local0,local1,local2,local3,local4,local5,local6,local7,lpr,mail,news,syslog,user,uucp > journal_purged and compare to the output from: journalctl --boot=-2 I get: -rw-r--r-- 1 cer users 23299642 Feb 26 12:34 journal_all -rw-r--r-- 1 cer users 18743605 Feb 26 13:02 journal_purged Both files should be the same, but they are not. Am I missing some facility? I use "meld" to find out the differences. Lines like this are missing: Feb 18 12:17:58 Telcontar sddm[2599]: Initializing... That's fac 1. kern? -rw-r--r-- 1 cer users 23299642 Feb 26 12:34 journal_all -rw-r--r-- 1 cer users 18743605 Feb 26 13:12 journal_purged Still lines missing, and it is still sddm: Feb 18 12:17:58 Telcontar sddm[2599]: Initializing... Read the manual again. --facility= Filter output by syslog facility. Takes a comma-separated list of numbers or facility names. The names are the usual syslog facilities as documented in syslog(3). --facility=help may be used to display a list of known facility names and exit. Let's try. Telcontar:~ # journalctl --facility=help Available facilities: kern user mail daemon auth syslog lpr news uucp cron authpriv ftp 12 13 14 15 local0 local1 local2 local3 local4 local5 local6 local7 Telcontar:~ # So I run journalctl --boot=-2 --facility=kern,user,mail,daemon,auth,syslog,lpr,news,uucp,cron,authpriv,ftp,12,13,14,15,local0,local1,local2,local3,local4,local5,local6,local7 > journal_purged Sigh. Still parts are missing: -rw-r--r-- 1 cer users 23299642 Feb 26 12:34 journal_all -rw-r--r-- 1 cer users 18743605 Feb 26 13:20 journal_purged It is still the sddm entries missing. I know it is fac=1 by comparison with my syslog: <1.4> 2025-02-24T01:03:30.963177+01:00 Telcontar sddm 2599 - - Signal received: SIGTERM *..... I print the <fac.prio> numbers. I will have to report with megabytes missing, from those unknown entries. Must be a bug with journalctl. Removing mail and news makes a difference in size: -rw-r--r-- 1 cer users 23299642 Feb 26 12:34 journal_all -rw-r--r-- 1 cer users 3777192 Feb 26 13:24 journal_purged But there are also those unknown entries, also megabytes lost. - -- Cheers Carlos E. R. (from 15.6 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCZ78L/Bwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVoLcAni8/2Z/Qf+YbtY1uUrrn Hg8clOdzAJ9hE86ZjY0vH2+QrTAPvc3R60c6Cg== =AguY -----END PGP SIGNATURE-----
On 2025-02-26 14:34, Andrei Borzenkov wrote:
Thanks. There is no facility. That's why the filter fails: Tue 2025-02-18 12:17:58.798499 CET [s=918bfecad21d42deb9d5de39b387f803;i=2020bf;b=509c9ed60df74c27a8db34cbd742e899;m=3d08fc1;t=62e68cb1394a8;x=eaecf4382c2d69a3] _BOOT_ID=509c9ed60df74c27a8db34cbd742e899 _MACHINE_ID=2ce1d54548517a7307c1c2bc38206d00 _HOSTNAME=Telcontar PRIORITY=7 _UID=0 _GID=0 _SELINUX_CONTEXT=unconfined _SYSTEMD_SLICE=system.slice _TRANSPORT=journal _CAP_EFFECTIVE=1ffffffffff _RUNTIME_SCOPE=system _SYSTEMD_CGROUP=/system.slice/display-manager.service _SYSTEMD_UNIT=display-manager.service _SYSTEMD_INVOCATION_ID=df5baf58bb93441b9b4e8cda6cdc78cf CODE_FILE=unknown CODE_LINE=0 CODE_FUNC=unknown SYSLOG_IDENTIFIER=sddm _PID=2599 _COMM=sddm _EXE=/usr/bin/sddm _CMDLINE=/usr/bin/sddm MESSAGE=Xauthority path: "/run/sddm/xauth_eBLdze" _SOURCE_REALTIME_TIMESTAMP=1739877478798499 Tue 2025-02-18 12:17:58.798515 CET [s=918bfecad21d42deb9d5de39b387f803;i=2020c0;b=509c9ed60df74c27a8db34cbd742e899;m=3d09062;t=62e68cb139549;x=abf76af6229d948a] ... -- Cheers / Saludos, Carlos E. R. (from 15.6 x86_64 at Telcontar)
26.02.2025 17:17, Carlos E. R. wrote:
Yes, journal does not support negative or "has (not)" style of matches.
Transport is journal which means SDDM is using native journal API and is responsible for providing any metadata. bor@bor-Latitude-E5450:~$ journalctl -b --system -o json | jq -r ._TRANSPORT | sort -u driver journal kernel stdout syslog bor@bor-Latitude-E5450:~$ And if we look where SYSLOG_FACILITY is missing bor@bor-Latitude-E5450:~$ journalctl -b --system -o json | jq -r 'select(has("SYSLOG_FACILITY")|not)| ._TRANSPORT' | sort -u journal bor@bor-Latitude-E5450:~$ and bor@bor-Latitude-E5450:~$ journalctl -b --system -o json | jq -r 'select(has("SYSLOG_FACILITY")|not)| ._CMDLINE' /usr/libexec/udisks2/udisksd /usr/libexec/gnome-remote-desktop-daemon --system /usr/libexec/udisks2/udisksd /usr/libexec/geoclue /usr/libexec/udisks2/udisksd /usr/libexec/udisks2/udisksd /usr/libexec/udisks2/udisksd bor@bor-Latitude-E5450:~$ In principle, you can use JSON output format together with any JSON processing tool to filter out anything you (do not) want and cobble together log lines in any form.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2025-02-26 a las 21:29 +0300, Andrei Borzenkov escribió:
A sad omission. When submitting to bugzilla I do not want to include my mail log, for privacy; nor do I want to include news because they are several megabytes of useless entries. And news entries are multiline, at least in my system. The amount of missing entries is substantial: cer@Telcontar:~/Bugzilla/nuevo_20250226> journalctl --boot=-2 --facility=kern,user,mail,daemon,auth,syslog,lpr,news,uucp,cron,authpriv,ftp,12,13,14,15,local0,local1,local2,local3,local4,local5,local6,local7 | tee journal_fac | wc -l 178781 cer@Telcontar:~/Bugzilla/nuevo_20250226> journalctl --boot=-2 | tee journal_all | wc -l 221479 cer@Telcontar:~/Bugzilla/nuevo_20250226> l total 41080 drwxr-xr-x 2 cer users 4096 Feb 26 19:36 ./ drwxrwxr-x 93 cer root 8192 Feb 26 19:32 ../ - -rw-r--r-- 1 cer users 23299642 Feb 26 19:36 journal_all - -rw-r--r-- 1 cer users 18743605 Feb 26 19:36 journal_fac cer@Telcontar:~/Bugzilla/nuevo_20250226>
Hum. Sigh. JSON is totally out of my experience. In <https://unix.stackexchange.com/questions/272662/how-do-i-clear-journalctl-entries-for-a-specific-unit-only> they suggest using a script to remove entries: +++····················· Use my Python 3 program copy_journal.py on the journal files in /var/log/journal from which you want to remove entries. For instance, to make a copy of system.journal without log entries for NetworkManager.service: $ journalctl --file=system.journal | wc 167 1934 18825 $ journalctl --file=system.journal | grep -v NetworkManager | wc 77 881 8421 $ python3 copy_journal.py --remove-unit=NetworkManager.service system.journal system-without-nm.journal $ journalctl --file=system-without-nm.journal | wc 77 881 8421 ·····················++- Said script is here: https://github.com/Mortal/cournal/blob/master/copy_journal.py But it goes by units, not facilities. And it seems to operate on the entire journal, which is gigabytes. I need only one boot session. Normally I would use grep, but news fac has multiline entries, so no go. Anyway, I have submitted the bugzilla, attaching the journal minus mail and news, and minus the megabytes of entries without facility. I told Bugzilla of the issue, I hope the surviving log entries are enough (kernel mostly are the interest). Otherwise, maybe they can suggest another method of purging the journal. Surely, I am not the only one needing to purge the journal for privacy when submitting a bugzilla. I have the syslog, but strangely, it has many more lines than the journal; and the boot part is different. cer@Telcontar:~/Bugzilla/nuevo_20250226> wc -l syslog journal_all journal_fac 332772 syslog 221479 journal_all 178781 journal_fac 733032 total cer@Telcontar:~/Bugzilla/nuevo_20250226> The syslog should be basically the same as journal_fac. Ie, syslog is complete with mail and news filtered, and also local1..7 are not there (tradition). And journal_fac was obtained with all facilities except mail and news. - -- Cheers, Carlos E. R. (from openSUSE 15.6 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCZ79wBhwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVcKEAn33UhkueDXFkOtwY0A0h 9RcH4Wq4AJ42DyLzm8DXK9YEIViFj8IhZWj0Nw== =15wX -----END PGP SIGNATURE-----
On Wed, 26 Feb 2025 21:29:06 +0300, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
OK. Thanks for demonstrating how useful that 'jq' tool is, Andrei. Having never used it before, I was encouraged to learn a bit. Here is some code to remove the mail, news, and authpriv (suggested) facility lines from journalctl output, using pieces copied from you and from others who needed to exclude some things. BTW, journalctl(1) does have one exclude type of option: "-T, --exclude-identifier=SYSLOG_IDENTIFIER Exclude messages for the specified syslog identifier" The code: # Filters the output of 'journalctl ... -o json' is_facility() { # usage: is_facility {=|!} facility_name # Arg 1: "=" for string equality match. # Arg 1: "!" to match if not equal. # facility_name may be a number. Outputs a match expression for 'jq'. # Values from: /usr/include/sys/syslog.h local N case "${2-}" in kern) N=0 ;; user) N=1 ;; mail) N=2 ;; daemon) N=3 ;; auth) N=4 ;; syslog) N=5 ;; lpr) N=6 ;; news) N=7 ;; uucp) N=8 ;; cron) N=9 ;; authpriv) N=10 ;; ftp) N=11 ;; 12) N=12 ;; 13) N=13 ;; 14) N=14 ;; 15) N=15 ;; local0) N=16 ;; local1) N=17 ;; local2) N=18 ;; local3) N=19 ;; local4) N=20 ;; local5) N=21 ;; local6) N=22 ;; local7) N=23 ;; *) local Num_re='^(0|[1-9][0-9]*)$' if [[ ${2-} =~ $Num_re ]] ;then N="$2" else printf '*** is_facility(): arg 2: %s\n' "${2-}" >&2 return 2 fi ;; esac case "${1-}" in '='|'!') : ;; *) printf '*** is_facility(): arg 1: %s\n' "${1-}" >&2 return 2 ;; esac printf '(.SYSLOG_FACILITY %s= "%d")\n' "$1" "$N" } SEL_EXPR="$(is_facility = authpriv) or $(is_facility = mail) or $(is_facility = news)" #SEL_EXCLUDE='' SEL_EXCLUDE=' | not' TIMESTAMP_SPEC='._SOURCE_REALTIME_TIMESTAMP | tonumber/1000000 | strftime("%Y-%m-%d %H:%M:%S")' OUT_ARRAY_SPEC="[(${TIMESTAMP_SPEC}), ._HOSTNAME, .SYSLOG_IDENTIFIER, ._PID, .MESSAGE]" journal_exclude() { # usage: journal_exclude [json_journal_file ...] # Output is piped through 'less', so it may be viewed, or directed # somewhere. # The awk gensub() call undoes the escaping done by jq's @tsv operator, # assuming that '\n', '\r', '\t', or '\' only appear in the MESSAGE field. jq -r "select( (${SEL_EXPR})${SEL_EXCLUDE}) | (${OUT_ARRAY_SPEC}) | @tsv" \ "$@" \ |awk -F '\t' \ '{print $1, $2, ($3 "[" $4 "]:"), gensub( /\\([nrt\\])/, "\\1", "g", $5)}' \ |less -S } Call it like this: $ sudo journalctl -o json -b |journal_exclude I haven't checked its handling of multi-line messages. -- Robert Webb
On 2025-03-02 08:32, Robert Webb via openSUSE Users wrote:
journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 --exclude-identifier=news --exclude-identifier=mail journalctl: unrecognized option '--exclude-identifier=news' I don't see that option in the manual. This is Leap 15.6 journalctl --version systemd 254 (254.23+suse.141.g9376e684d0) +PAM +AUDIT +SELINUX +APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
It doesn't produce any output :-? $> journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 -o json | ./journal_exclude $> I assumed it is a bash script, so I added the line "#!/bin/bash" at the start. Otherwise "file journal_exclude" says it is ascii text. jq is installed. awk is installed. -- Cheers / Saludos, Carlos E. R. (from 15.6 x86_64 at Telcontar)
On Mon, 3 Mar 2025 12:17:28 +0100, "Carlos E. R." <robin.listas@telefonica.net> wrote:
Hmm, so it's new. For Tumbleweed: $ journalctl --version systemd 257 (257.3+suse.3.ge03ffd74c4) +PAM +AUDIT +SELINUX +APPARMOR +IMA +IPE -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON -UTMP +SYSVINIT +LIBARCHIVE
It defined the function 'journal_exclude()'. In your script, you need to call it at the end with (make sure your script name is different): journal_exclude "$@" Now for the mea culpa. I only tested this on a very limited set of journalctl JSON output. With the full since-boot output, there are many errors due to messages not having the _SOURCE_REALTIME_TIMESTAMP field. Substituting __REALTIME_TIMESTAMP fixes that. In addition, if you want to compare with diffs, the time format should match, so the 'TIMESTAMP_SPEC=' line changes to: TIMESTAMP_SPEC='.__REALTIME_TIMESTAMP | tonumber/1000000 | strflocaltime("%b %d %H:%M:%S")' As for the gensub() call in the awk print line, which "undoes the escaping done by jq's @tsv operator", that is bogus! It does not. I don't know where my brain was then. I have since removed awk ("Bad dog!") and put the output handling into the jq code. That removes the need for the @tsv conversion to interface to awk, so there is not any escaping now. There was also an issue with the particular pid field not always being there, and the same with the identifier field. Some extra conditional logic deals with those. Now I have it as a script, attached[1], just as you made it. To change the specified exclusions, you need to edit the setting of the 'Sel_expr' variable in the script. It can't be made to be input as a command-line arg yet because there is a need to call the internal is_facility() function within the value. If I put that into the jq code also, then it will be possible. But I have to learn more. The current state, testing with a more than 6000 line journalctl JSON output, and diffing the script output against the plain journalctl output, is that there are differences due to tabs, which can be ignored with the '-b' option of diff, and also some unknown (to me) invisible differences. Otherwise, it looks like the script output matches journalctl when setting $Sel_expr to not exclude anything. To get test data: $ sudo journalctl -b -o json > journal-json.log ;sudo journalctl -b > journal-plain.log Edit journal-plain.log to remove extra lines. To test: $ ./journal-exclude.bash journal-json.log |diff -ub journal-plain.log - |less -S For regular use: $ sudo journalctl -o json <options> |./journal-exclude.bash [1] Attached: journal-exclude.bash -- Robert Webb
On 2025-03-04 13:33, Robert Webb via openSUSE Users wrote:
Ah. Now it produces a lot of: jq: error (at <stdin>:2607): null (null) cannot be parsed as a number jq: error (at <stdin>:2608): null (null) cannot be parsed as a number mixed with the output. Let's see your new version instead :-)
I did: cer@Telcontar:~/Bugzilla/Bug_1237776 - Machine went unresponsive/two> journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 -o json > journal-json.log cer@Telcontar:~/Bugzilla/Bug_1237776 - Machine went unresponsive/two> journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 > journal-plain.log cer@Telcontar:~/Bugzilla/Bug_1237776 - Machine went unresponsive/two> (I can run journalctl as my user because it belongs to group "systemd-journal". cer@Telcontar:~/Bugzilla/Bug_1237776 - Machine went unresponsive/two> ./journal-exclude-2.bash journal-json.log > journal-filtered.log jq: error (at journal-json.log:2674): string ("Feb 24 11:...) and array ([91,48,58,4...) cannot be added jq: error (at journal-json.log:2675): string ("Feb 24 11:...) and array ([91,48,58,4...) cannot be added cer@Telcontar:~/Bugzilla/Bug_1237776 - Machine went unresponsive/two> -rw-r--r-- 1 cer users 9120208 Mar 4 23:06 journal-filtered.log -rw-r--r-- 1 cer users 228684353 Mar 4 23:03 journal-json.log -rw-r--r-- 1 cer users 24668998 Mar 4 23:03 journal-plain.log Then I compare using "meld" because I'm more used to it than to diff. meld journal-plain.log journal-filtered.log There are some differences in whitespace, I think you mentioned it: original: Feb 24 11:20:54 Telcontar kernel: rcu: RCU event tracing is enabled. Feb 24 11:20:54 Telcontar kernel: rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=32. Feb 24 11:20:54 Telcontar kernel: Trampoline variant of Tasks RCU enabled. Feb 24 11:20:54 Telcontar kernel: Rude variant of Tasks RCU enabled. Feb 24 11:20:54 Telcontar kernel: Tracing variant of Tasks RCU enabled. filtered: Feb 24 11:20:54 Telcontar kernel: rcu: RCU event tracing is enabled. Feb 24 11:20:54 Telcontar kernel: rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=32. Feb 24 11:20:54 Telcontar kernel: Trampoline variant of Tasks RCU enabled. Feb 24 11:20:54 Telcontar kernel: Rude variant of Tasks RCU enabled. Feb 24 11:20:54 Telcontar kernel: Tracing variant of Tasks RCU enabled. There are some differences in timestamps - original: Feb 24 11:21:07 Telcontar systemd[1]: Mounting /boot... Feb 24 11:21:07 Telcontar systemd[1]: data-storage_c.mount: Directory /data/storage_c to mount over is not empty, mounting anyway. Feb 24 11:21:07 Telcontar systemd[1]: Mounting /data/storage_c... Feb 24 11:21:07 Telcontar systemd[1]: data-storage_d.mount: Directory /data/storage_d to mount over is not empty, mounting anyway. Feb 24 11:21:07 Telcontar systemd[1]: Mounting /data/storage_d... filtered: Feb 24 11:21:08 Telcontar systemd[1]: Mounting /boot... Feb 24 11:21:08 Telcontar systemd[1]: data-storage_c.mount: Directory /data/storage_c to mount over is not empty, mounting anyway. Feb 24 11:21:08 Telcontar systemd[1]: Mounting /data/storage_c... Feb 24 11:21:08 Telcontar systemd[1]: data-storage_d.mount: Directory /data/storage_d to mount over is not empty, mounting anyway. Feb 24 11:21:08 Telcontar systemd[1]: Mounting /data/storage_d... I see more occurrences of timestamps differences in the output. Example, much later: Mar 03 01:00:04 Telcontar systemd[1]: Started Timeline of Snapper Snapshots. vs Mar 03 01:00:10 Telcontar systemd[1]: Started Timeline of Snapper Snapshots. or Mar 03 01:00:20 Telcontar rtkit-daemon[5418]: The canary thread is apparently starving. Taking action. vs Mar 03 01:00:40 Telcontar rtkit-daemon[5418]: The canary thread is apparently starving. Taking action. or Mar 03 01:05:44 Telcontar smartd[1495]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 70 vs Mar 03 01:06:19 Telcontar smartd[1495]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 70 This is the filtering at work: Feb 24 11:35:37 Telcontar polkitd[1472]: Loading rules from directory /etc/polkit-1/rules.d Feb 24 11:35:37 Telcontar polkitd[1472]: Loading rules from directory /usr/share/polkit-1/rules.d :-) Funny that this is "authpriv". I have the same line in syslog: <10.5> 2025-02-24T01:58:19.559623+01:00 Telcontar polkitd 1453 - - Loading rules from directory /etc/polkit-1/rules.d So it is facility 10, authpriv. Working as intended :-) I might remove authpriv filtering from the script, and then open the file in an editor to manually remove really private entries. I can produce them with: journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 --facility=authpriv > journal_authpriv.log That way, the times when I log in via ssh can be seen, they are mentioned in the bugzilla report. A multiline entry with whitespace - original: Feb 24 11:35:38 Telcontar boot.local[1764]: /dev/sda: setting standby to 210 (17 minutes + 30 seconds) filtered: Feb 24 11:35:38 Telcontar boot.local[1764]: /dev/sda: setting standby to 210 (17 minutes + 30 seconds) Filtering mail and news, as intended: Feb 24 11:35:43 Telcontar clamd[2455]: Received 0 file descriptor(s) from systemd. Feb 24 11:35:43 Telcontar clamd[2455]: clamd daemon 1.4.2 (OS: Linux, ARCH: x86_64, CPU: x86_64) Feb 24 11:35:43 Telcontar clamd[2455]: Log file size limited to 1048576 bytes. Feb 24 11:35:43 Telcontar clamd[2455]: Reading databases from /var/lib/clamav Feb 24 11:35:43 Telcontar clamd[2455]: Not loading PUA signatures. Feb 24 11:35:43 Telcontar clamd[2455]: Bytecode: Security mode set to "TrustSigned". Feb 24 11:35:43 Telcontar texpire[2456]: config: debugmode is 1001 Ah, we arrive at the gist of my bug report. Original: Mar 03 00:49:53 Telcontar kernel: ---[ end trace 0000000000000000 ]--- Mar 03 00:49:53 Telcontar kernel: amdgpu 0000:27:00.0: amdgpu: GPU reset begin! Mar 03 00:48:58 Telcontar dovecot[2595]: imap(21237): Warning: Time jumped forwards 18.971933 seconds Mar 03 00:49:19 Telcontar rpc.mountd[2464]: v4.0 client detached: 0xf4c994fe67bc4b7f from "192.168.1.16:951" Mar 03 00:50:30 Telcontar kernel: watchdog: BUG: soft lockup - CPU#9 stuck for 21s! [kworker/u64:2:1587] filtered: Mar 03 00:49:53 Telcontar kernel: ---[ end trace 0000000000000000 ]--- Mar 03 00:49:53 Telcontar kernel: amdgpu 0000:27:00.0: amdgpu: GPU reset begin! Mar 03 00:49:53 Telcontar rpc.mountd[2464]: v4.0 client detached: 0xf4c994fe67bc4b7f from "192.168.1.16:951" Mar 03 00:50:30 Telcontar kernel: watchdog: BUG: soft lockup - CPU#9 stuck for 21s! [kworker/u64:2:1587] The filter is working as intended, but the dovecot filtered line is interesting for the bug. Time jumped forward 19 seconds, matches with "CPU#9 stuck for 21s!". It is a pity, because the intention is to remove my email data and correspondents from the log, but that line has no privacy concerns. Well, it can not be helped. The missing whitespace might produce some difficulties reading the log - original: Mar 03 01:11:36 Telcontar kernel: rcu: INFO: rcu_preempt self-detected stall on CPU Mar 03 01:11:36 Telcontar kernel: rcu: 3-....: (14996 ticks this GP) idle=7180/1/0x4000000000000000 softirq=27814637/27814637 fqs=1883 Mar 03 01:11:36 Telcontar kernel: rcu: hardirqs softirqs csw/system Mar 03 01:11:36 Telcontar kernel: rcu: number: 0 0 0 Mar 03 01:11:36 Telcontar kernel: rcu: cputime: 0 0 20712 ==> 20712(ms) Mar 03 01:11:36 Telcontar kernel: rcu: (t=15000 jiffies g=62413365 q=10816 ncpus=12) Mar 03 01:11:36 Telcontar kernel: rcu: rcu_preempt kthread starved for 5163 jiffies! g62413365 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=5 Mar 03 01:11:36 Telcontar kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. Mar 03 01:11:36 Telcontar kernel: rcu: RCU grace-period kthread stack dump: filtered: Mar 03 01:11:36 Telcontar kernel: rcu: INFO: rcu_preempt self-detected stall on CPU Mar 03 01:11:36 Telcontar kernel: rcu: 3-....: (14996 ticks this GP) idle=7180/1/0x4000000000000000 softirq=27814637/27814637 fqs=1883 Mar 03 01:11:36 Telcontar kernel: rcu: hardirqs softirqs csw/system Mar 03 01:11:36 Telcontar kernel: rcu: number: 0 0 0 Mar 03 01:11:36 Telcontar kernel: rcu: cputime: 0 0 20712 ==> 20712(ms) Mar 03 01:11:36 Telcontar kernel: rcu: (t=15000 jiffies g=62413365 q=10816 ncpus=12) Mar 03 01:11:36 Telcontar kernel: rcu: rcu_preempt kthread starved for 5163 jiffies! g62413365 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=5 Mar 03 01:11:36 Telcontar kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. Mar 03 01:11:36 Telcontar kernel: rcu: RCU grace-period kthread stack dump: Ah, this is important. Original: Mar 03 01:50:01 Telcontar CRON[18600]: (root) CMDEND ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 1 1) Mar 03 01:50:24 Telcontar systemd-coredump[18120]: Process 2805 (X) of user 0 dumped core. Stack trace of thread 2844: #0 0x00007f636baa941c __pthread_kill_implementation (libc.so.6 + 0xa941c) #1 0x00007f636ba57842 raise (libc.so.6 + 0x57842) #2 0x00007f636ba3f5cf abort (libc.so.6 + 0x3f5cf) ... omitting lines ELF object binary architecture: AMD x86-64 Mar 03 01:50:24 Telcontar kglobalaccel5[29272]: The X11 connection broke (error 1). Did the X11 server die? filtered - the contents of the coredump are missing: Mar 03 01:50:01 Telcontar CRON[18600]: (root) CMDEND ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 1 1) Mar 03 01:50:24 Telcontar systemd-coredump[18120]: Mar 03 01:50:24 Telcontar kglobalaccel5[29272]: The X11 connection broke (error 1). Did the X11 server die? Thankyou for the script, it helps a lot. Just that missing coredump at the end is important. Big multiline message, I guess. -- Cheers / Saludos, Carlos E. R. (from 15.6 x86_64 at Telcontar)
On Tue, 4 Mar 2025 23:59:57 +0100, "Carlos E. R." <robin.listas@telefonica.net> wrote:
Right. :-)
Let's see your new version instead :-)
This is something I haven't seen. I better check whether the posted script is identical to what I have been using. Would you be able to post those two JSON log lines? sed -n -e '2674,2675p' journal-json.log > add_errors.log Since the script was expecting two strings to add (concatenate), and the second item in the error is an array of numbers, I suspect this is what is happening[2]: "A field that contains non-printable or non-UTF8 is serialized as a number array instead. This is necessary to handle binary data in a safe way without losing data, since JSON cannot embed binary data natively. Each byte of the binary field will be mapped to its numeric value in the range 0…255." I don't know if that is going to be easy to convert.
There are some differences in whitespace, I think you mentioned it:
Yes, I saw the same, but have not investigated it yet.
Maybe these are related to the add errors above since they involved the timestamps.
This is the filtering at work: [...]
Thanks very much for the feedback.
If it is too big, it might not have been included in the exported JSON.[2]: "The JSON serializer can optionally skip huge (as in larger than a specific threshold) data fields from the JSON object. If that is enabled and a data field is too large, the field name is still included in the JSON object but assigned null." [2] https://systemd.io/JOURNAL_EXPORT_FORMATS/#journal-export-format -- Robert Webb
On 2025-03-05 01:55, Robert Webb via openSUSE Users wrote:
If I'm not mistaken, these: {"_STREAM_ID":"aa9171a41bbd4258959287b1814df831","_SELINUX_CONTEXT":"unconfined\n","__REALTIME_TIMESTAMP":"1740393369322372","_AUDIT_LOGINUID":"1000","_PID":"5417","_SYSTEMD_OWNER_UID":"1000","_RUNTIME_SCOPE":"system","_AUDIT_SESSION":"4","_SYSTEMD_UNIT":"user@1000.service","_BOOT_ID":"0a3828f43b5649c188f1bbee6ab7f468","_SYSTEMD_USER_UNIT":"wireplumber.service","__SEQNUM":"2325584","SYSLOG_FACILITY":"3","_COMM":"wireplumber","_TRANSPORT":"stdout","_CMDLINE":"/usr/bin/wireplumber","_SYSTEMD_USER_SLICE":"session.slice","_MACHINE_ID":"2ce1d54548517a7307c1c2bc38206d00","_HOSTNAME":"Telcontar","_GID":"100","_CAP_EFFECTIVE":"0","_SYSTEMD_INVOCATION_ID":"bbebce4a78ee4adebf57c3a6fee1b699","_SYSTEMD_CGROUP":"/user.slice/user-1000.slice/user@1000.service/session.slice/wireplumber.service","SYSLOG_IDENTIFIER":"wireplumber","__SEQNUM_ID":"f1e7af9f29494abe99932f67c9d50b73","MESSAGE":[91,48,58,49,53,58,49,53,46,53,57,52,55,54,49,55,56,49,93,32,91,53,52,49,55,93,32,27,91,49,59,51,51,109,32,87,65,82,78,32,27,91,49,59,51,55,109,73,80,65,77,97,110,97,103,101,114,32,27,91,49,59,51,52,109,105,112,97,95,109,97,110,97,103,101,114,46,99,112,112,58,49,53,52,32,27,91,48,109,78,111,32,73,80,65,32,102,111,117,110,100,32,105,110,32,39,47,117,115,114,47,108,105,98,54,52,47,108,105,98,99,97,109,101,114,97,39],"__CURSOR":"s=f1e7af9f29494abe99932f67c9d50b73;i=237c50;b=0a3828f43b5649c188f1bbee6ab7f468;m=3692de29;t=62ee0e88c4384;x=4665cc0efc549185","_EXE":"/usr/bin/wireplumber","_SYSTEMD_SLICE":"user-1000.slice","__MONOTONIC_TIMESTAMP":"915594793","PRIORITY":"6","_UID":"1000"} {"__SEQNUM":"2325585","_COMM":"wireplumber","_CAP_EFFECTIVE":"0","_CMDLINE":"/usr/bin/wireplumber","_SYSTEMD_USER_UNIT":"wireplumber.service","MESSAGE":[91,48,58,49,53,58,49,53,46,53,57,52,55,56,57,51,52,49,93,32,91,53,52,49,55,93,32,27,91,49,59,51,50,109,32,73,78,70,79,32,27,91,49,59,51,55,109,67,97,109,101,114,97,32,27,91,49,59,51,52,109,99,97,109,101,114,97,95,109,97,110,97,103,101,114,46,99,112,112,58,50,56,52,32,27,91,48,109,108,105,98,99,97,109,101,114,97,32,118,48,46,50,46,48],"SYSLOG_IDENTIFIER":"wireplumber","_SYSTEMD_USER_SLICE":"session.slice","_UID":"1000","_SYSTEMD_CGROUP":"/user.slice/user-1000.slice/user@1000.service/session.slice/wireplumber.service","_STREAM_ID":"aa9171a41bbd4258959287b1814df831","_SYSTEMD_UNIT":"user@1000.service","_BOOT_ID":"0a3828f43b5649c188f1bbee6ab7f468","_AUDIT_SESSION":"4","_SELINUX_CONTEXT":"unconfined\n","SYSLOG_FACILITY":"3","_PID":"5417","_SYSTEMD_SLICE":"user-1000.slice","__REALTIME_TIMESTAMP":"1740393369322372","_TRANSPORT":"stdout","_MACHINE_ID":"2ce1d54548517a7307c1c2bc38206d00","_RUNTIME_SCOPE":"system","_HOSTNAME":"Telcontar","_SYSTEMD_INVOCATION_ID":"bbebce4a78ee4adebf57c3a6fee1b699","_AUDIT_LOGINUID":"1000","_EXE":"/usr/bin/wireplumber","__SEQNUM_ID":"f1e7af9f29494abe99932f67c9d50b73","_SYSTEMD_OWNER_UID":"1000","__CURSOR":"s=f1e7af9f29494abe99932f67c9d50b73;i=237c51;b=0a3828f43b5649c188f1bbee6ab7f468;m=3692de29;t=62ee0e88c4384;x=787c99946485c559","__MONOTONIC_TIMESTAMP":"915594793","PRIORITY":"6","_GID":"100"}
Ah, I see. It would have taken me several long moons to get this far. -- Cheers / Saludos, Carlos E. R. (from 15.6 x86_64 at Telcontar)
On 2025-02-26 14:34, Andrei Borzenkov wrote:
Thanks. There is no facility. That's why the filter fails: Tue 2025-02-18 12:17:58.798499 CET [s=918bfecad21d42deb9d5de39b387f803;i=2020bf;b=509c9ed60df74c27a8db34cbd742e899;m=3d08fc1;t=62e68cb1394a8;x=eaecf4382c2d69a3] _BOOT_ID=509c9ed60df74c27a8db34cbd742e899 _MACHINE_ID=2ce1d54548517a7307c1c2bc38206d00 _HOSTNAME=Telcontar PRIORITY=7 _UID=0 _GID=0 _SELINUX_CONTEXT=unconfined _SYSTEMD_SLICE=system.slice _TRANSPORT=journal _CAP_EFFECTIVE=1ffffffffff _RUNTIME_SCOPE=system _SYSTEMD_CGROUP=/system.slice/display-manager.service _SYSTEMD_UNIT=display-manager.service _SYSTEMD_INVOCATION_ID=df5baf58bb93441b9b4e8cda6cdc78cf CODE_FILE=unknown CODE_LINE=0 CODE_FUNC=unknown SYSLOG_IDENTIFIER=sddm _PID=2599 _COMM=sddm _EXE=/usr/bin/sddm _CMDLINE=/usr/bin/sddm MESSAGE=Xauthority path: "/run/sddm/xauth_eBLdze" _SOURCE_REALTIME_TIMESTAMP=1739877478798499 Tue 2025-02-18 12:17:58.798515 CET [s=918bfecad21d42deb9d5de39b387f803;i=2020c0;b=509c9ed60df74c27a8db34cbd742e899;m=3d09062;t=62e68cb139549;x=abf76af6229d948a] ... -- Cheers / Saludos, Carlos E. R. (from 15.6 x86_64 at Telcontar)
26.02.2025 17:17, Carlos E. R. wrote:
Yes, journal does not support negative or "has (not)" style of matches.
Transport is journal which means SDDM is using native journal API and is responsible for providing any metadata. bor@bor-Latitude-E5450:~$ journalctl -b --system -o json | jq -r ._TRANSPORT | sort -u driver journal kernel stdout syslog bor@bor-Latitude-E5450:~$ And if we look where SYSLOG_FACILITY is missing bor@bor-Latitude-E5450:~$ journalctl -b --system -o json | jq -r 'select(has("SYSLOG_FACILITY")|not)| ._TRANSPORT' | sort -u journal bor@bor-Latitude-E5450:~$ and bor@bor-Latitude-E5450:~$ journalctl -b --system -o json | jq -r 'select(has("SYSLOG_FACILITY")|not)| ._CMDLINE' /usr/libexec/udisks2/udisksd /usr/libexec/gnome-remote-desktop-daemon --system /usr/libexec/udisks2/udisksd /usr/libexec/geoclue /usr/libexec/udisks2/udisksd /usr/libexec/udisks2/udisksd /usr/libexec/udisks2/udisksd bor@bor-Latitude-E5450:~$ In principle, you can use JSON output format together with any JSON processing tool to filter out anything you (do not) want and cobble together log lines in any form.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 El 2025-02-26 a las 21:29 +0300, Andrei Borzenkov escribió:
A sad omission. When submitting to bugzilla I do not want to include my mail log, for privacy; nor do I want to include news because they are several megabytes of useless entries. And news entries are multiline, at least in my system. The amount of missing entries is substantial: cer@Telcontar:~/Bugzilla/nuevo_20250226> journalctl --boot=-2 --facility=kern,user,mail,daemon,auth,syslog,lpr,news,uucp,cron,authpriv,ftp,12,13,14,15,local0,local1,local2,local3,local4,local5,local6,local7 | tee journal_fac | wc -l 178781 cer@Telcontar:~/Bugzilla/nuevo_20250226> journalctl --boot=-2 | tee journal_all | wc -l 221479 cer@Telcontar:~/Bugzilla/nuevo_20250226> l total 41080 drwxr-xr-x 2 cer users 4096 Feb 26 19:36 ./ drwxrwxr-x 93 cer root 8192 Feb 26 19:32 ../ - -rw-r--r-- 1 cer users 23299642 Feb 26 19:36 journal_all - -rw-r--r-- 1 cer users 18743605 Feb 26 19:36 journal_fac cer@Telcontar:~/Bugzilla/nuevo_20250226>
Hum. Sigh. JSON is totally out of my experience. In <https://unix.stackexchange.com/questions/272662/how-do-i-clear-journalctl-entries-for-a-specific-unit-only> they suggest using a script to remove entries: +++····················· Use my Python 3 program copy_journal.py on the journal files in /var/log/journal from which you want to remove entries. For instance, to make a copy of system.journal without log entries for NetworkManager.service: $ journalctl --file=system.journal | wc 167 1934 18825 $ journalctl --file=system.journal | grep -v NetworkManager | wc 77 881 8421 $ python3 copy_journal.py --remove-unit=NetworkManager.service system.journal system-without-nm.journal $ journalctl --file=system-without-nm.journal | wc 77 881 8421 ·····················++- Said script is here: https://github.com/Mortal/cournal/blob/master/copy_journal.py But it goes by units, not facilities. And it seems to operate on the entire journal, which is gigabytes. I need only one boot session. Normally I would use grep, but news fac has multiline entries, so no go. Anyway, I have submitted the bugzilla, attaching the journal minus mail and news, and minus the megabytes of entries without facility. I told Bugzilla of the issue, I hope the surviving log entries are enough (kernel mostly are the interest). Otherwise, maybe they can suggest another method of purging the journal. Surely, I am not the only one needing to purge the journal for privacy when submitting a bugzilla. I have the syslog, but strangely, it has many more lines than the journal; and the boot part is different. cer@Telcontar:~/Bugzilla/nuevo_20250226> wc -l syslog journal_all journal_fac 332772 syslog 221479 journal_all 178781 journal_fac 733032 total cer@Telcontar:~/Bugzilla/nuevo_20250226> The syslog should be basically the same as journal_fac. Ie, syslog is complete with mail and news filtered, and also local1..7 are not there (tradition). And journal_fac was obtained with all facilities except mail and news. - -- Cheers, Carlos E. R. (from openSUSE 15.6 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCZ79wBhwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVcKEAn33UhkueDXFkOtwY0A0h 9RcH4Wq4AJ42DyLzm8DXK9YEIViFj8IhZWj0Nw== =15wX -----END PGP SIGNATURE-----
On Wed, 26 Feb 2025 21:29:06 +0300, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
OK. Thanks for demonstrating how useful that 'jq' tool is, Andrei. Having never used it before, I was encouraged to learn a bit. Here is some code to remove the mail, news, and authpriv (suggested) facility lines from journalctl output, using pieces copied from you and from others who needed to exclude some things. BTW, journalctl(1) does have one exclude type of option: "-T, --exclude-identifier=SYSLOG_IDENTIFIER Exclude messages for the specified syslog identifier" The code: # Filters the output of 'journalctl ... -o json' is_facility() { # usage: is_facility {=|!} facility_name # Arg 1: "=" for string equality match. # Arg 1: "!" to match if not equal. # facility_name may be a number. Outputs a match expression for 'jq'. # Values from: /usr/include/sys/syslog.h local N case "${2-}" in kern) N=0 ;; user) N=1 ;; mail) N=2 ;; daemon) N=3 ;; auth) N=4 ;; syslog) N=5 ;; lpr) N=6 ;; news) N=7 ;; uucp) N=8 ;; cron) N=9 ;; authpriv) N=10 ;; ftp) N=11 ;; 12) N=12 ;; 13) N=13 ;; 14) N=14 ;; 15) N=15 ;; local0) N=16 ;; local1) N=17 ;; local2) N=18 ;; local3) N=19 ;; local4) N=20 ;; local5) N=21 ;; local6) N=22 ;; local7) N=23 ;; *) local Num_re='^(0|[1-9][0-9]*)$' if [[ ${2-} =~ $Num_re ]] ;then N="$2" else printf '*** is_facility(): arg 2: %s\n' "${2-}" >&2 return 2 fi ;; esac case "${1-}" in '='|'!') : ;; *) printf '*** is_facility(): arg 1: %s\n' "${1-}" >&2 return 2 ;; esac printf '(.SYSLOG_FACILITY %s= "%d")\n' "$1" "$N" } SEL_EXPR="$(is_facility = authpriv) or $(is_facility = mail) or $(is_facility = news)" #SEL_EXCLUDE='' SEL_EXCLUDE=' | not' TIMESTAMP_SPEC='._SOURCE_REALTIME_TIMESTAMP | tonumber/1000000 | strftime("%Y-%m-%d %H:%M:%S")' OUT_ARRAY_SPEC="[(${TIMESTAMP_SPEC}), ._HOSTNAME, .SYSLOG_IDENTIFIER, ._PID, .MESSAGE]" journal_exclude() { # usage: journal_exclude [json_journal_file ...] # Output is piped through 'less', so it may be viewed, or directed # somewhere. # The awk gensub() call undoes the escaping done by jq's @tsv operator, # assuming that '\n', '\r', '\t', or '\' only appear in the MESSAGE field. jq -r "select( (${SEL_EXPR})${SEL_EXCLUDE}) | (${OUT_ARRAY_SPEC}) | @tsv" \ "$@" \ |awk -F '\t' \ '{print $1, $2, ($3 "[" $4 "]:"), gensub( /\\([nrt\\])/, "\\1", "g", $5)}' \ |less -S } Call it like this: $ sudo journalctl -o json -b |journal_exclude I haven't checked its handling of multi-line messages. -- Robert Webb
On 2025-03-02 08:32, Robert Webb via openSUSE Users wrote:
journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 --exclude-identifier=news --exclude-identifier=mail journalctl: unrecognized option '--exclude-identifier=news' I don't see that option in the manual. This is Leap 15.6 journalctl --version systemd 254 (254.23+suse.141.g9376e684d0) +PAM +AUDIT +SELINUX +APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
It doesn't produce any output :-? $> journalctl --boot=0a3828f43b5649c188f1bbee6ab7f468 -o json | ./journal_exclude $> I assumed it is a bash script, so I added the line "#!/bin/bash" at the start. Otherwise "file journal_exclude" says it is ascii text. jq is installed. awk is installed. -- Cheers / Saludos, Carlos E. R. (from 15.6 x86_64 at Telcontar)
participants (3)
-
Andrei Borzenkov -
Carlos E. R. -
Robert Webb