-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2019-02-13 at 11:55 -0000, Dave Howorth wrote:
Carlos E. R. wrote:
Telcontar:~/Documents/lists/swapoff_test # time swapoff -a swapoff: /dev/sde1: swapoff failed: Cannot allocate memory
real 0m0.010s user 0m0.001s sys 0m0.003s
Interesting, so it seems that OOM killer doesn't get triggered in such a case. Would that be expected, or a bug?
I'd hope it would be expected. The swapoff fails, you get told and can shut down whichever processes you choose to free up enough memory. Or you can change your mind about shutting down swap. Versus, the OOM killer is invoked, randomly kills off a selection of your processes, and the swapoff succeeds. I know which I would prefer.
I then killed Firefox and Thunderbird, restarted them (the machine has to run my usual workload), then repeated the swapoff experiment. Telcontar:~/Documents/lists/swapoff_test # time swapoff -a real 3m30.374s user 0m0.001s sys 3m3.646s Telcontar:~/Documents/lists/swapoff_test # I was writing my notes on the terminal, when the system *collapsed* and would not respond. I managed to issue "swapon -a" on an ssh session already opened on another machine, which took perhaps half an hour to process. Then, with swap back on, the system responded, after also doing some OOM (on mysqld). Thunderbird was not running: cer@Telcontar:~> grep -i "oom" /var/log/messages <0.4> 2019-02-13 13:16:35 Telcontar kernel - - - [688971.796341] mysqld invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 <0.4> 2019-02-13 13:16:35 Telcontar kernel - - - [688971.796366] oom_kill_process+0x213/0x3d0 <0.6> 2019-02-13 13:16:35 Telcontar kernel - - - [688971.796474] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name <0.6> 2019-02-13 13:16:35 Telcontar kernel - - - [688971.860717] oom_reaper: reaped process 12455 (thunderbird-bin), now anon-rss:0kB, file-rss:0kB, shmem-rss:15884kB cer@Telcontar:~> Interestingly, both mysqld and Thunderbird have been killed differently (one by oom-killer, another by oom_reaper) Why mysqld I have no idea, but it could have been worse. See the "free -h" output at different stages: Telcontar:~/Documents/lists/swapoff_test # cat free_status.2.log Wed Feb 13 12:32:56 CET 2019 total used free shared buff/cache available Mem: 7.8G 5.0G 870M 503M 2.0G 2.1G Swap: 23G 2.2G 21G Wed Feb 13 12:46:01 CET 2019 total used free shared buff/cache available Mem: 7.8G 6.9G 151M 566M 797M 137M Swap: 0B 0B 0B Wed Feb 13 13:35:38 CET 2019 total used free shared buff/cache available Mem: 7.8G 5.6G 1.2G 547M 1.0G 1.4G Swap: 23G 269M 23G As you can see, when swap was turned off I only had 137M available RAM (151M free and 797M in cache). Barely sufficient to run the system, which collapsed soon and would not respond to the keyboard - not even the LED lights. *top sequence*: Status before running swapoff: top - 12:32:28 up 7 days, 22:38, 4 users, load average: 1,03, 0,69, 0,61 Tasks: 448 total, 1 running, 446 sleeping, 0 stopped, 1 zombie %Cpu(s): 6,4 us, 1,8 sy, 0,6 ni, 88,4 id, 2,9 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 895036 free, 5209780 used, 2056584 buff/cache KiB Swap: 25165820 total, 22824188 free, 2341632 used. 2160380 avail Mem Notice now that "free swap" goes to zero the next minute - probably when swapoff starts working (took 3:30 minutes): top - 12:33:28 up 7 days, 22:39, 4 users, load average: 1,23, 0,81, 0,66 Tasks: 447 total, 2 running, 444 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,6 us, 9,8 sy, 0,6 ni, 81,2 id, 2,6 wa, 0,0 hi, 0,2 si, 0,0 st KiB Mem : 8161400 total, 881484 free, 5215912 used, 2064004 buff/cache KiB Swap: 2035128 total, 0 free, 2035128 used. 2146120 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 15178 root 20 0 22636 1316 1148 0 R 33,44 0,016 0:20.09 swapoff 12957 cer 20 0 9926004 1,042g 409480 0 S 9,804 13,39 5:09.48 firefox 13141 cer 20 0 2424292 700828 194512 0 S 4,544 8,587 1:59.71 Web Content 4680 cer 39 19 2067524 20520 0 22300 S 2,663 0,251 311:38.78 tracker-extract 5910 root 20 0 60420 17824 2112 1900 S 2,280 0,218 266:24.87 iotop 13048 cer 20 0 2593004 685348 201584 0 S 2,247 8,397 1:15.42 Web Content 13090 cer 20 0 2579108 598056 209880 0 S 1,997 7,328 0:52.23 Web Content 12455 cer 20 0 3464248 1,023g 140108 0 S 1,481 13,14 1:48.88 thunderbird-bin 13121 cer 20 0 2684540 742012 196632 0 S 1,182 9,092 0:48.16 Web Content swapoff is working - see that firefox and thunderbird-bin are both running. top - 12:36:28 up 7 days, 22:42, 4 users, load average: 1,95, 1,19, 0,83 Tasks: 450 total, 2 running, 447 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,8 us, 23,5 sy, 0,6 ni, 63,4 id, 6,0 wa, 0,0 hi, 0,7 si, 0,0 st KiB Mem : 8161400 total, 123128 free, 6996448 used, 1041824 buff/cache KiB Swap: 78948 total, 0 free, 78948 used. 315948 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 15178 root 20 0 22636 1292 1124 0 R 87,63 0,016 2:58.03 swapoff 12957 cer 20 0 9926004 1,004g 368860 0 S 9,704 12,90 5:26.81 firefox 13141 cer 20 0 2426340 629064 166704 0 S 3,778 7,708 2:06.93 Web Content 4680 cer 39 19 2067524 42132 0 688 S 2,730 0,516 311:43.68 tracker-extract 5910 root 20 0 60420 19536 1996 72 S 2,513 0,239 266:29.23 iotop 13048 cer 20 0 2593004 658724 173924 0 S 2,397 8,071 1:19.72 Web Content 13121 cer 20 0 2686588 714216 169496 0 S 1,365 8,751 0:50.52 Web Content 13090 cer 20 0 2579108 574268 182952 0 S 1,282 7,036 0:54.64 Web Content 12455 cer 20 0 3464248 0,981g 84588 0 S 1,148 12,61 1:51.19 thunderbird-bin 3868 root 20 0 652028 199244 134556 28 S 0,782 2,441 72:53.03 X 4020 root 20 0 447228 8336 2620 68 S 0,682 0,102 7:32.27 udisksd 24809 root 20 0 41604 1052 0 8 S 0,649 0,013 29:00.71 top 5871 cer 20 0 40396 1024 0 40 S 0,632 0,013 67:09.08 top 52 root 20 0 0 0 0 0 S 0,583 0,000 34:02.25 kswapd0 the next minute, swapoff had finished: top - 12:37:28 up 7 days, 22:43, 4 users, load average: 1,00, 1,06, 0,81 Tasks: 446 total, 1 running, 444 sleeping, 0 stopped, 1 zombie %Cpu(s): 6,2 us, 4,2 sy, 0,6 ni, 85,9 id, 3,0 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 8161400 total, 227648 free, 7045396 used, 888356 buff/cache KiB Swap: 0 total, 0 free, 0 used. 278852 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 12957 cer 20 0 9926004 986,3m 323700 0 S 10,60 12,37 5:33.18 firefox 13141 cer 20 0 2424292 639536 131112 0 S 4,760 7,836 2:09.79 Web Content 13048 cer 20 0 2595052 678568 128068 0 S 3,113 8,314 1:21.59 Web Content 4680 cer 39 19 2067524 42744 0 76 S 2,780 0,524 311:45.35 tracker-extract 5910 root 20 0 60420 19512 1948 48 S 2,413 0,239 266:30.68 iotop 13090 cer 20 0 2579108 598900 146328 0 S 2,413 7,338 0:56.09 Web Content 13121 cer 20 0 2687612 737580 125348 0 S 1,897 9,037 0:51.66 Web Content 12455 cer 20 0 3464248 987,5m 59504 0 S 1,265 12,39 1:51.95 thunderbird-bin 3868 root 20 0 652028 198516 133824 24 S 0,649 2,432 72:53.42 X 5871 cer 20 0 40396 1024 0 40 S 0,616 0,013 67:09.45 top 24809 root 20 0 41604 1052 0 8 S 0,599 0,013 29:01.07 top 52 root 20 0 0 0 0 0 S 0,516 0,000 34:02.56 kswapd0 4234 cer 20 0 301208 19528 3520 0 S 0,250 0,239 27:34.45 gkrellm 4702 cer 20 0 732704 42912 0 52 S 0,233 0,526 23:40.80 tracker-store 4324 cer 20 0 524424 28988 5712 20 S 0,200 0,355 22:56.35 panel-6-weather 4330 cer 20 0 524364 27136 5776 24 S 0,200 0,332 23:03.13 panel-14-weathe 41 root 39 19 0 0 0 0 S 0,117 0,000 3:36.08 khugepaged 4242 cer 20 0 748680 61428 8712 80 S 0,100 0,753 8:45.65 xfce4-terminal 5907 cer 20 0 217984 13180 0 4 S 0,100 0,161 1:40.45 dirmngr 4322 cer 20 0 251216 18624 2468 0 S 0,067 0,228 6:31.89 panel-23-multil 4315 cer 20 0 710624 20768 1480 80 S 0,050 0,254 6:05.39 panel-19-pulsea 6744 cer 20 0 4057688 266724 48 932 S 0,050 3,268 4:51.09 java 24011 cer 20 0 40260 988 0 4 R 0,050 0,012 2:26.05 top 266 root 0 -20 0 0 0 0 S 0,033 0,000 0:50.70 kworker/2:1H top - 12:38:28 up 7 days, 22:44, 4 users, load average: 0,61, 0,94, 0,78 Tasks: 447 total, 1 running, 445 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,3 us, 1,4 sy, 0,6 ni, 90,5 id, 2,2 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 329408 free, 6922968 used, 909024 buff/cache KiB Swap: 0 total, 0 free, 0 used. 401424 avail Mem ... top - 12:46:29 up 7 days, 22:52, 4 users, load average: 0,66, 0,65, 0,67 Tasks: 447 total, 1 running, 445 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,8 us, 1,7 sy, 0,6 ni, 87,4 id, 4,5 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 8161400 total, 156276 free, 7192052 used, 813072 buff/cache KiB Swap: 0 total, 0 free, 0 used. 138704 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 12957 cer 20 0 9926004 990208 293952 0 S 9,407 12,13 6:23.92 firefox 13141 cer 20 0 2424292 554988 98292 0 S 4,096 6,800 2:30.53 Web Content 4680 cer 39 19 2067524 43068 264 16 S 2,714 0,528 311:59.87 tracker-extract 5910 root 20 0 60420 18296 684 0 S 2,481 0,224 266:43.44 iotop 13048 cer 20 0 2596076 715052 95864 0 S 2,231 8,761 1:33.88 Web Content 12455 cer 20 0 3472492 1,010g 37284 0 S 1,915 12,98 2:03.53 thunderbird-bin 13090 cer 20 0 2504132 562236 112660 0 S 1,265 6,889 1:03.27 Web Content 13121 cer 20 0 2688636 778320 98888 0 S 1,116 9,537 0:57.87 Web Content 3868 root 20 0 652028 196496 131804 24 S 0,932 2,408 72:56.89 X 52 root 20 0 0 0 0 0 S 0,716 0,000 34:05.19 kswapd0 4020 root 20 0 447228 6200 448 32 S 0,699 0,076 7:32.70 udisksd 5871 cer 20 0 40396 1024 0 40 S 0,616 0,013 67:12.75 top 24809 root 20 0 41604 1052 0 8 S 0,566 0,013 29:04.38 top 4242 cer 20 0 748680 61108 8376 64 S 0,416 0,749 8:46.30 xfce4-terminal 4702 cer 20 0 732704 42888 0 16 S 0,283 0,525 23:42.13 tracker-store 4234 cer 20 0 301208 19200 3136 0 S 0,266 0,235 27:35.84 gkrellm 4330 cer 20 0 524364 25604 4192 0 S 0,200 0,314 23:04.26 panel-14-weathe 4324 cer 20 0 524424 27068 3712 0 S 0,183 0,332 22:57.47 panel-6-weather 4762 cer 20 0 494612 32860 7268 72 S 0,167 0,403 0:15.99 lxterminal 41 root 39 19 0 0 0 0 S 0,067 0,000 3:36.35 khugepaged 15332 root 0 -20 0 0 0 0 S 0,067 0,000 0:00.04 kworker/u9:3 4250 cer 20 0 1074936 40172 3088 96 S 0,050 0,492 2:21.68 konversation 4315 cer 20 0 710624 19680 328 16 S 0,050 0,241 6:05.66 panel-19-pulsea 4322 cer 20 0 251216 17940 1784 0 S 0,050 0,220 6:32.20 panel-23-multil 6744 cer 20 0 4057688 266912 48 744 S 0,050 3,270 4:51.31 java Here top stopped collecting data for about half an hour (I have it dumping data every minute), the time while the system collapsed. As you can see kswapd0 is running full load trying to free memory dumping things to swap again - thunderbird doesn't appear, it was killed at 13:16:35 (by syslog entry, see top of email). Notice that only about 4 MB are available: top - 13:16:35 up 7 days, 23:22, 4 users, load average: 100,36, 97,43, 79,20 Tasks: 437 total, 5 running, 429 sleeping, 0 stopped, 3 zombie %Cpu(s): 0,2 us, 28,8 sy, 0,0 ni, 3,5 id, 66,9 wa, 0,0 hi, 0,6 si, 0,0 st KiB Mem : 8161400 total, 108228 free, 7405628 used, 647544 buff/cache KiB Swap: 0 total, 0 free, 0 used. 4528 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 52 root 20 0 0 0 0 0 R 100,0 0,000 43:25.98 kswapd0 2318 mysql 20 0 1887764 71392 0 20 S 144,8 0,875 5:10.93 mysqld 12957 cer 20 0 9926004 971900 271204 0 D 65,62 11,91 7:11.64 firefox 6744 cer 20 0 4057688 267328 48 328 S 41,47 3,276 5:21.47 java 2307 named 20 0 540744 203332 0 0 S 41,46 2,491 4:23.45 named 13141 cer 20 0 2424292 579728 75816 0 D 24,38 7,103 2:48.26 Web Content 29797 nscd 20 0 1025804 924 60 32 S 23,67 0,011 0:40.85 nscd 4315 cer 20 0 710624 19548 164 0 R 22,88 0,240 6:22.30 panel-19-pulsea 13121 cer 20 0 2690684 813840 76696 0 D 20,97 9,972 1:13.12 Web Content 4149 cer 20 0 240120 420 0 0 D 20,74 0,005 0:44.32 gpg-agent 13048 cer 20 0 2600172 752436 76500 0 D 20,35 9,219 1:48.68 Web Content 13090 cer 20 0 2504132 594280 92888 0 D 20,32 7,282 1:18.05 Web Content 4250 cer 20 0 1074936 37100 16 96 D 20,15 0,455 2:36.33 konversation 15753 cer 20 0 179320 572 0 0 D 20,01 0,007 0:14.55 nvidia-settings 1538 avahi 20 0 43808 440 0 0 D 19,22 0,005 0:22.54 avahi-daemon 2929 ntp 20 0 53120 756 0 0 D 15,51 0,009 0:38.71 ntpd 24809 root 20 0 41604 1068 8 0 R 15,06 0,013 29:15.33 top 4324 cer 20 0 524424 23752 396 0 D 14,99 0,291 23:08.37 panel-6-weather 4772 cer 20 0 468916 27192 0 60 D 14,95 0,333 2:08.14 contarcorreo 4234 cer 20 0 301208 16064 0 0 D 14,89 0,197 27:46.67 gkrellm 5871 cer 20 0 40396 1064 0 0 R 14,32 0,013 67:23.16 top 4330 cer 20 0 524364 21808 396 0 D 14,19 0,267 23:14.58 panel-14-weathe 3868 root 20 0 652028 194640 129924 0 S 14,03 2,385 73:07.09 X 2954 root 20 0 225140 8808 16 0 D 13,92 0,108 0:28.11 httpd-prefork 24011 cer 20 0 40260 2000 1004 0 R 13,92 0,025 2:36.37 top next iteration of top, I have swap again, but zero used - the good news is that now there are 433MB available memory, the system can breathe again: top - 13:17:36 up 7 days, 23:23, 4 users, load average: 37,39, 79,87, 74,30 Tasks: 433 total, 1 running, 431 sleeping, 0 stopped, 1 zombie %Cpu(s): 8,4 us, 2,0 sy, 0,7 ni, 83,0 id, 5,8 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem : 8161400 total, 172136 free, 6916048 used, 1073216 buff/cache KiB Swap: 25165820 total, 25165820 free, 0 used. 433128 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 52 root 20 0 0 0 0 0 S 37,29 0,000 54:35.07 kswapd0 12957 cer 20 0 9965688 1,216g 318196 0 S 0,606 15,63 7:22.52 firefox 1 root 20 0 240380 7920 4044 0 S 0,537 0,097 2:22.43 systemd 13141 cer 20 0 2457064 692188 129672 0 S 0,239 8,481 2:52.55 Web Content 1151 root 20 0 7132 1456 1152 0 S 0,167 0,018 0:15.29 mdadm 663 root 20 0 88996 9040 8000 0 S 0,141 0,111 1:47.81 systemd-journal 13048 cer 20 0 2687212 855316 106936 0 S 0,094 10,48 1:50.37 Web Content 4680 cer 39 19 2067524 44384 1580 16 S 0,090 0,544 312:03.11 tracker-extract 5910 root 20 0 60420 21772 4152 0 S 0,076 0,267 266:53.02 iotop 13090 cer 20 0 2506180 713180 132272 0 S 0,062 8,738 1:19.16 Web Content 13121 cer 20 0 2744956 920400 109940 0 S 0,060 11,28 1:14.19 Web Content 3868 root 20 0 636368 184712 119996 0 S 0,033 2,263 73:07.69 X 15778 root 20 0 0 0 0 0 S 0,031 0,000 0:00.56 kworker/0:2 5871 cer 20 0 40396 2616 1552 0 S 0,021 0,032 67:23.53 top 4196 cer 20 0 484456 24180 0 20 S 0,016 0,296 11:25.28 gvfs-udisks2-vo 4020 root 20 0 447228 8992 3240 32 S 0,014 0,110 7:40.01 udisksd 15819 root 20 0 0 0 0 0 S 0,012 0,000 0:00.21 kworker/0:0 839 root 20 0 67628 2572 0 0 S 0,010 0,032 0:01.32 systemd-udevd 4234 cer 20 0 301208 23828 7764 0 S 0,009 0,292 27:46.84 gkrellm 4702 cer 20 0 732704 47244 4320 16 S 0,009 0,579 23:42.45 tracker-store 4250 cer 20 0 1074936 56080 18996 96 S 0,007 0,687 2:36.46 konversation 4324 cer 20 0 524424 31668 8312 0 S 0,007 0,388 23:08.50 panel-6-weather 4330 cer 20 0 524364 28796 7384 0 S 0,007 0,353 23:14.71 panel-14-weathe 22 root 20 0 0 0 0 0 S 0,007 0,000 0:16.43 ksoftirqd/2 8 root 20 0 0 0 0 0 S 0,006 0,000 2:11.07 rcu_sched Three minutes later swap starts to store things, very slowly: top - 13:20:37 up 7 days, 23:26, 4 users, load average: 2,21, 43,88, 61,26 Tasks: 432 total, 2 running, 429 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,3 us, 1,2 sy, 0,8 ni, 90,7 id, 2,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 563748 free, 6518736 used, 1078916 buff/cache KiB Swap: 25165820 total, 25165564 free, 256 used. 830360 avail Mem top - 13:21:37 up 7 days, 23:27, 4 users, load average: 0,94, 35,94, 57,44 Tasks: 432 total, 1 running, 430 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,1 us, 1,2 sy, 0,6 ni, 91,2 id, 2,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 547248 free, 6533800 used, 1080352 buff/cache KiB Swap: 25165820 total, 25165564 free, 256 used. 814192 avail Mem top - 13:22:37 up 7 days, 23:28, 4 users, load average: 0,79, 29,55, 53,90 Tasks: 429 total, 2 running, 426 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,5 us, 1,5 sy, 0,7 ni, 92,0 id, 0,3 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 343844 free, 6739428 used, 1078128 buff/cache KiB Swap: 25165820 total, 25165308 free, 512 used. 609804 avail Mem top - 13:23:37 up 7 days, 23:29, 4 users, load average: 0,36, 24,19, 50,53 Tasks: 430 total, 1 running, 428 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,4 us, 1,1 sy, 0,7 ni, 92,0 id, 0,8 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 335460 free, 6746744 used, 1079196 buff/cache KiB Swap: 25165820 total, 25165308 free, 512 used. 602480 avail Mem ... top - 13:27:37 up 7 days, 23:33, 4 users, load average: 0,14, 10,90, 39,04 Tasks: 432 total, 2 running, 429 sleeping, 0 stopped, 1 zombie %Cpu(s): 5,4 us, 1,5 sy, 0,6 ni, 92,1 id, 0,3 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 170052 free, 6936852 used, 1054496 buff/cache KiB Swap: 25165820 total, 25041916 free, 123904 used. 412372 avail Mem ... top - 13:36:38 up 7 days, 23:42, 4 users, load average: 0,69, 2,08, 21,94 Tasks: 432 total, 1 running, 430 sleeping, 0 stopped, 1 zombie %Cpu(s): 7,0 us, 1,7 sy, 0,6 ni, 88,7 id, 2,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161400 total, 940524 free, 6162704 used, 1058172 buff/cache KiB Swap: 25165820 total, 24889596 free, 276224 used. 1190036 avail Mem I have top log and the messages log of the event. As you can see, it is not possible to run this 8GiB machine without Swap - I said it would not work and I was right. - -- Cheers, Carlos E. R. (from openSUSE 15.0 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXGQeNRwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVX+0AniuRB1NXHsDRAT6MvFoo Tm8aY3BZAKCYbVkrrpwr8jg1R9aVuE7DQmMXNg== =R39Z -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org