[opensuse] Slow amavis (11 seconds for small, text only, emails)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Recently I have configured postfix to send emails trhough it, instead of via thunderbird. It works fine. But amavis takes a longish time thinking about it: Jun 20 14:19:00 Elessar amavis[3728]: (03728-04) Passed CLEAN, [IPv6:::1] [IPv6:::1] <robin.listas@telefonica.net> -> <opensuse@opensuse.org>, Message-ID: <4C1E0723.2060602@telefonica.net>, mail_id: kM3AAtZAo7Jl, Hits: -0.057, size: 1857, queued_as: 5F09A2DFBB, 17172 ms That is 17 seconds of thinking! This is a fast machine, and anyway, cpu load was under 5% while I posted that email. I think amavis is doing some test that needs a remote check, and timing out. This is oS 11.2 (x64). Amavis has the default settings, I haven't touched it (except uncommenting a paragraph to enable clamvd). - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkweCZ0ACgkQU92UU+smfQU/5ACfdcjJ+nrZBQcqZkkgOncsvzZI emMAn1HAQTEWA6zkzurulk+feiKGSBum =OL80 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-20 14:29, Carlos E. R. wrote:
Hi,
Recently I have configured postfix to send emails trhough it, instead of via thunderbird. It works fine. But amavis takes a longish time thinking about it:
Jun 20 14:19:00 Elessar amavis[3728]: (03728-04) Passed CLEAN, [IPv6:::1] [IPv6:::1] <robin.listas@telefonica.net> -> <opensuse@opensuse.org>, Message-ID: <4C1E0723.2060602@telefonica.net>, mail_id: kM3AAtZAo7Jl, Hits: -0.057, size: 1857, queued_as: 5F09A2DFBB, 17172 ms
That is 17 seconds of thinking!
This is a fast machine, and anyway, cpu load was under 5% while I posted that email. I think amavis is doing some test that needs a remote check, and timing out.
This is oS 11.2 (x64). Amavis has the default settings, I haven't touched it (except uncommenting a paragraph to enable clamvd).
I increased the log verbosity of amavis, and found the point of delay: Jun 20 21:33:20 Elessar amavis[16122]: (16122-01) CALLING SA check Jun 20 21:33:29 Elessar amavis[16122]: (16122-01) spam_scan: score=-0.515 autolearn=ham tests=[AWL=-0.514,NO_RELAYS=-0.001] Ie, 9 seconds to run "spamassassin". Why? They are emails to this list, under 5 KB, with a fast machine. I need to increase the log verbosity inside the "CALLING SA check" section, to learn the cause. Any ideas? - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwecJAACgkQU92UU+smfQXuAgCfbP2GIbDyOYMJgjVCcaALCFbz BEgAnA8gabJsbB4SWh1A4WHHsPvGdM27 =DFC7 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Carlos E. R. <carlos.e.r@opensuse.org> [06-20-10 15:50]:
I increased the log verbosity of amavis, and found the point of delay:
Jun 20 21:33:20 Elessar amavis[16122]: (16122-01) CALLING SA check Jun 20 21:33:29 Elessar amavis[16122]: (16122-01) spam_scan: score=-0.515 autolearn=ham tests=[AWL=-0.514,NO_RELAYS=-0.001]
Ie, 9 seconds to run "spamassassin". Why? They are emails to this list, under 5 KB, with a fast machine.
I need to increase the log verbosity inside the "CALLING SA check" section, to learn the cause.
Any ideas?
spamd /usr/sbin/spamd -d -c -L -r /var/run/spamd.pid rcspamd -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-20 22:11, Patrick Shanahan wrote:
* Carlos E. R. <> [06-20-10 15:50]:
I need to increase the log verbosity inside the "CALLING SA check" section, to learn the cause.
Any ideas?
spamd
/usr/sbin/spamd -d -c -L -r /var/run/spamd.pid
rcspamd
I'm not using spamd in this machine; and anyway, amavis uses spamassassin internally, it ignores spamd. Anyway, I'll try with this email. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkweenEACgkQU92UU+smfQWvAACfQ+K1DR/Oh+Xdq0bqIcfwUSVB gGwAn0MPnWlDKw0WbHfBK/CXH7INzaF7 =wYyf -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-20 22:30, Carlos E. R. wrote:
On 2010-06-20 22:11, Patrick Shanahan wrote:
* Carlos E. R. <> [06-20-10 15:50]:
I need to increase the log verbosity inside the "CALLING SA check" section, to learn the cause.
Any ideas?
spamd
/usr/sbin/spamd -d -c -L -r /var/run/spamd.pid
rcspamd
I'm not using spamd in this machine; and anyway, amavis uses spamassassin internally, it ignores spamd. Anyway, I'll try with this email.
Nope. I started spamd, then restarted amavis: Jun 20 22:28:45 Elessar amavis[29816]: Creating db in /var/spool/amavis/db/; BerkeleyDB 0.39, libdb 4.5 Jun 20 22:28:45 Elessar amavis[29816]: SpamControl: initializing Mail::SpamAssassin Jun 20 22:28:45 Elessar amavis[29816]: SpamAssassin debug facilities: info Jun 20 22:28:55 Elessar amavis[29816]: SpamControl: init_pre_fork on SpamAssassin done Jun 20 22:28:55 Elessar amavis[29820]: TIMING [total 3 ms] - bdb-open: 3 (100%)100, rundown: 0 (0%)100 Jun 20 22:28:55 Elessar amavis[29820]: SpamControl: init_child on SpamAssassin done Jun 20 22:28:55 Elessar amavis[29821]: TIMING [total 3 ms] - bdb-open: 3 (100%)100, rundown: 0 (0%)100 Jun 20 22:28:55 Elessar amavis[29821]: SpamControl: init_child on SpamAssassin done Then I sent the previous email: Jun 20 22:30:41 Elessar postfix/smtpd[29878]: connect from localhost[::1] amavis starts checking that second: Jun 20 22:30:42 Elessar amavis[29820]: process_request: fileno sock=13, STDIN=0, STDOUT=1 and here there is a delay: Jun 20 22:30:42 Elessar amavis[29820]: (29820-01) ClamAV-clamd: Sending CONTSCAN /var/spool/amavis/tmp/amavis-20100620T223042-29820/parts\n to UNIX socket /var/lib/clamav/clamd-socket Jun 20 22:30:42 Elessar amavis[29820]: (29820-01) run_av (ClamAV-clamd): CLEAN Jun 20 22:30:42 Elessar amavis[29820]: (29820-01) run_av (ClamAV-clamd) result: clean Jun 20 22:30:56 Elessar amavis[29820]: (29820-01) spam_scan: score=-0.207 autolearn=ham tests=[AWL=-0.206,NO_RELAYS=-0.001] ... Jun 20 22:30:56 Elessar amavis[29820]: (29820-01) Passed CLEAN, [IPv6:::1] [IPv6:::1] <carlos.e.r@...> -> <opensuse@...>, Message-ID: <4C1E7A71.6040402@opensuse.org>, mail_id: uktzwy0AXK5c, Hits: -0.207, size: 1689, queued_as: BA5F03CD36, 14695 ms See? clamav does the job fast. After it returns, there is a 14 second delay (I reduced log verbosity to 3 from 4, the SA check doesn't show - I think. I'll change back to 4. Or 5). But anyway, running spamd does nothing. Verifying the cpu time - 0:00. Elessar:~ # ps afx | grep -i spam 30129 pts/2 S+ 0:00 | | \_ grep -i spam 29784 ? Ss 0:00 /usr/sbin/spamd -d -c -L -r /var/run/spamd.pid 29785 ? S 0:00 \_ spamd child 29786 ? S 0:00 \_ spamd child And the only log entries for spamd are: Jun 20 21:49:45 Elessar amavis[28581]: SpamControl: init_child on SpamAssassin done Jun 20 22:18:32 Elessar clamd[3356]: SelfCheck: Database status OK. Jun 20 22:28:35 Elessar spamd[29780]: logger: removing stderr method Jun 20 22:28:36 Elessar spamd[29784]: spamd: server started on port 783/tcp (running version 3.2.5) Jun 20 22:28:36 Elessar spamd[29784]: spamd: server pid: 29784 Jun 20 22:28:36 Elessar spamd[29784]: spamd: server successfully spawned child process, pid 29785 Jun 20 22:28:36 Elessar spamd[29784]: spamd: server successfully spawned child process, pid 29786 Jun 20 22:28:36 Elessar spamd[29784]: prefork: child states: IS Jun 20 22:28:36 Elessar spamd[29784]: prefork: child states: II Jun 20 22:28:43 Elessar amavis[29803]: logging initialized, log level 3, syslog: amavis.mail - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwefuEACgkQU92UU+smfQXRkQCeIRbxxgtdpFJykEr6ifzm0O/p HhQAnjGQiuacFPlM264rTsDTfwKMfk4r =IrSn -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Carlos E. R. <carlos.e.r@opensuse.org> [06-20-10 16:51]:
On 2010-06-20 22:11, Patrick Shanahan wrote: I'm not using spamd in this machine; and anyway, amavis uses spamassassin internally, it ignores spamd. Anyway, I'll try with this email.
Nope.
I started spamd, then restarted amavis:
Jun 20 22:28:45 Elessar amavis[29816]: Creating db in /var/spool/amavis/db/; BerkeleyDB 0.39, libdb 4.5 Jun 20 22:28:45 Elessar amavis[29816]: SpamControl: initializing Mail::SpamAssassin Jun 20 22:28:45 Elessar amavis[29816]: SpamAssassin debug facilities: info Jun 20 22:28:55 Elessar amavis[29816]: SpamControl: init_pre_fork on SpamAssassin done Jun 20 22:28:55 Elessar amavis[29820]: TIMING [total 3 ms] - bdb-open: 3 (100%)100, rundown: 0 (0%)100 Jun 20 22:28:55 Elessar amavis[29820]: SpamControl: init_child on SpamAssassin done Jun 20 22:28:55 Elessar amavis[29821]: TIMING [total 3 ms] - bdb-open: 3 (100%)100, rundown: 0 (0%)100 Jun 20 22:28:55 Elessar amavis[29821]: SpamControl: init_child on SpamAssassin done ...
Since amavisd-new-2.6.3 it is possible to use an external spamd server and let amavisd behave as a spamc client to it: @spam_scanners = ( ['SpamdClient', 'Amavis::SpamControl::SpamdClient' ], ); This was not intended as a mechanism to depend-on in a production environment, but more like a test vehicle for the external spam scanners feature which brought support for CRM114 and DSPAM. Adding a spamc/spamd protocol to that was just an exercise, and as a side product this also serves as a test environment for exercising spamd. from: http://old.nabble.com/spamd-and-multiple-amavisd-new-instances-installations... also http://www.mail-archive.com/amavis-user@lists.sourceforge.net/msg15192.html but http://www.ijs.si/software/amavisd/ says is part: -- performance .... * when configured to call Mail::SpamAssassin (this is optional), it orders SA to pre-load its config files and to precompile the patterns, so performance is at least as good as with spamc/spamd setup. All Perl modules are pre-loaded by a parent process at startup time, so forked children need not re-compile the code, and can hopefully share some memory for compiled code; so ?? -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-21 02:50, Patrick Shanahan wrote:
* Carlos E. R. <> [06-20-10 16:51]:
Nope.
...
Since amavisd-new-2.6.3 it is possible to use an external spamd server and let amavisd behave as a spamc client to it:
11.2 has amavisd-new-2.6.2-3.4, so no luck there. However, if the loading were the problem, I'm sure that the delay would not be 15 seconds: this is a fast machine, I would see the cpu busy or the disk busy. And the delay would be constant (or long the first time, short the second): but it goes from 9 to 20 seconds, and some under a second. All are mails to this list. Even a local mail gets delayed (21258 ms and 21244 ms right now). No, I'm sure that the SA part of amavis is waiting all that time for the result of something, without using cpu, without using HD, meaning it is something from the network. What I need is the debug log of the SA part. There is nothing logged. I need that to try to learn where it is stopping. The verbose log I'm getting is from amavis alone - look: Jun 21 17:32:38 Elessar amavis[30319]: (30319-04) calling SA parse, SA version 3.2.5, 3.002005, data as GLOB Jun 21 17:32:38 Elessar amavis[30319]: (30319-04) CALLING SA check Jun 21 17:32:59 Elessar amavis[30319]: (30319-04) prolong_timer spam_scan_sa_finish: timer set to 459 s Jun 21 17:32:59 Elessar amavis[30319]: (30319-04) spam_scan: score=3.328 autolearn=no tests=[AWL=0.430,NO_RELAYS=-0.001,TVD_SPACE_RATIO=2.899] Jun 21 17:32:59 Elessar amavis[30319]: (30319-04) prolong_timer spam_scan: remaining time = 459 s It calls SA, and SA returns after 21 seconds with a full report done. No verbose log of the inside process. I need that. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwfh9EACgkQU92UU+smfQXI5ACfWHOLPfPcvJ+ipAfsLnK9msj8 0AgAn0wZtOVxAlVkg1Sv6Wy/ZxqeI+Rb =Ayl9 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Carlos E. R. <robin.listas@telefonica.net> [06-21-10 11:41]:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2010-06-21 02:50, Patrick Shanahan wrote:
* Carlos E. R. <> [06-20-10 16:51]:
Nope.
...
Since amavisd-new-2.6.3 it is possible to use an external spamd server and let amavisd behave as a spamc client to it:
11.2 has amavisd-new-2.6.2-3.4, so no luck there.
http://download.opensuse.org/repositories/server:/mail/openSUSE_11.2/i586/am... http://download.opensuse.org/repositories/server:/mail/openSUSE_11.2/x86_64/... -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-21 17:53, Patrick Shanahan wrote:
* Carlos E. R. <> [06-21-10 11:41]:
On 2010-06-21 02:50, Patrick Shanahan wrote:
* Carlos E. R. <> [06-20-10 16:51]:
Nope.
...
Since amavisd-new-2.6.3 it is possible to use an external spamd server and let amavisd behave as a spamc client to it:
11.2 has amavisd-new-2.6.2-3.4, so no luck there.
http://download.opensuse.org/repositories/server:/mail/openSUSE_11.2/i586/am... http://download.opensuse.org/repositories/server:/mail/openSUSE_11.2/x86_64/...
No, I'm not going that route. I simply want to get the verbose log, not install anything, chancing that it solves or not the problem. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwfsoIACgkQU92UU+smfQWfMgCgi0t47pj/Cz8BZ291b01OT1dg bG4An0aa+u4JltAjVk4Lcb14YnTo28PP =Xi09 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-21 17:40, Carlos E. R. wrote:
What I need is the debug log of the SA part. There is nothing logged. I need that to try to learn where it is stopping.
The verbose log I'm getting is from amavis alone - look:
Jun 21 17:32:38 Elessar amavis[30319]: (30319-04) calling SA parse, SA version 3.2.5, 3.002005, data as GLOB Jun 21 17:32:38 Elessar amavis[30319]: (30319-04) CALLING SA check Jun 21 17:32:59 Elessar amavis[30319]: (30319-04) prolong_timer spam_scan_sa_finish: timer set to 459 s Jun 21 17:32:59 Elessar amavis[30319]: (30319-04) spam_scan: score=3.328 autolearn=no tests=[AWL=0.430,NO_RELAYS=-0.001,TVD_SPACE_RATIO=2.899] Jun 21 17:32:59 Elessar amavis[30319]: (30319-04) prolong_timer spam_scan: remaining time = 459 s
It calls SA, and SA returns after 21 seconds with a full report done. No verbose log of the inside process. I need that.
Anyone? - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwfsucACgkQU92UU+smfQWuqwCgkX/UDVTjvWCHHEsEadU7xUui AVsAn3Hwh4LY8gadsDFFmYrhkMC42zrr =qcZN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-21 20:43, Carlos E. R. wrote:
On 2010-06-21 17:40, Carlos E. R. wrote:
What I need is the debug log of the SA part. There is nothing logged. I need that to try to learn where it is stopping.
Anyone?
Add: $sa_debug = 1; to /etc/amavisd.conf. And found the problem: Jun 22 01:18:36 Elessar amavis[14512]: (14512-01) SA dbg: async: aborting after 8.251 s, deadline shrunk: DNSBL-A, dns:A:elessar.valinor.rhsbl.ahbl.org. And the problem was reported here a year ago (oS 10.3): http://old.nabble.com/dnsbl-checks-time-out-td22222481i20.html and here: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6075 The recommended solution is to set up a local domain server in my machine (although I doubt the explanation). I usually do (set up a cache dns), but I didn't in this partition. I'll try that, time permitting. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwf//sACgkQU92UU+smfQVuSgCfYt6tHPyA0CzPa2NlTQBYs0VC I8kAnRYIIPWmAO4eFwOCCW95lPUl00wW =jutf -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2010-06-20 14:29, Carlos E. R. wrote:
Hi,
Recently I have configured postfix to send emails trhough it, instead of via thunderbird. It works fine. But amavis takes a longish time thinking about it:
Jun 20 14:19:00 Elessar amavis[3728]: (03728-04) Passed CLEAN, [IPv6:::1] [IPv6:::1] <robin.listas@...> -> <opensuse@...>, Message-ID: <4C1E0723.2060602@telefonica.net>, mail_id: kM3AAtZAo7Jl, Hits: -0.057, size: 1857, queued_as: 5F09A2DFBB, 17172 ms
That is 17 seconds of thinking!
Just install bind. It just works. The culprit, they think, is the house router, which supposedly can not handle too many dns request simultaneously, and, supposedly, does not reply. I haven't tested this, but... that's what the bugzilla at SA concluded a year ago. In any case, a local bind daemon configured as cached dns (the default config as installed from the oS rpm) works. - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAkwhHe8ACgkQU92UU+smfQX2GgCbBp1Y4tvJp3hydnCG0zcdblqM iXAAnA4srFa/g+3anD+E9dJUkdkcsLw8 =BzOC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Carlos E. R. <carlos.e.r@opensuse.org> [06-22-10 16:34]:
Just install bind. It just works.
good :^) -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (3)
-
Carlos E. R. -
Carlos E. R. -
Patrick Shanahan