[Bug 956558] New: Base:System/nfs-utils: Bug GSS is disabled in config, but see rpc-svcgssd.service ERROR ( GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE) on boot
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 Bug ID: 956558 Summary: Base:System/nfs-utils: Bug GSS is disabled in config, but see rpc-svcgssd.service ERROR ( GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE) on boot Classification: openSUSE Product: openSUSE.org Version: unspecified Hardware: x86-64 OS: openSUSE 42.1 Status: NEW Severity: Normal Priority: P5 - None Component: 3rd party software Assignee: neilb@suse.com Reporter: 9b3e05a5@opayq.com QA Contact: opensuse-communityscreening@forge.provo.novell.com Found By: --- Blocker: --- I'm running Opensuse Leap 42.1 server. nfs-kernel-server is installed & running systemctl status nfs-server nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled) Drop-In: /usr/lib/systemd/system/nfs-server.service.d └─nfsserver.conf Active: active (exited) since Tue 2015-11-24 14:11:42 PST; 3h 49min ago Main PID: 2098 (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs-server.service rpm -q --whatprovides /usr/lib/systemd/system/nfs-server.service nfs-kernel-server-1.3.0-9.1.x86_64 The default status of RPC services is systemctl list-unit-files | grep -i rpc var-lib-nfs-rpc_pipefs.mount static auth-rpcgss-module.service static rpc-gssd.service static rpc-statd-notify.service static rpc-statd.service static rpc-svcgssd.service static rpcbind.service enabled rpcbind.socket enabled rpcbind.target static NFS is configured to disable GSS (no krb5) grep -i gss /etc/sysconfig/nfs NFS_SECURITY_GSS="no" GSSD_OPTIONS="" SVCGSSD_OPTIONS="" NFS_GSSD_AVOID_DNS="no" but on boot, there's a rpc-svcgssd FAILure journalctl -xb | egrep -i "gss|rpc" Nov 24 14:10:27 xensvrkernel: RPC: Registered named UNIX socket transport module. Nov 24 14:10:27 xensvrkernel: RPC: Registered udp transport module. Nov 24 14:10:27 xensvrkernel: RPC: Registered tcp transport module. Nov 24 14:10:27 xensvrkernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Nov 24 14:10:28 xensvrrpc.statd[293]: Version 1.3.0 starting Nov 24 14:10:28 xensvrrpc.statd[293]: Failed to open directory sm: No such file or directory Nov 24 14:10:28 xensvrrpc.statd[293]: Initializing NSM state Nov 24 14:10:28 xensvrrpc.statd[293]: Running as root. chown /var/lib/nfs to choose different user Nov 24 14:10:37 xensvrrpcbind[289]: rpcbind terminating on signal. Restart with "rpcbind -w" Nov 24 14:10:37 xensvrrpcbind[289]: cannot open file = /var/lib/rpcbind/rpcbind.xdr for writing Nov 24 14:10:37 xensvrrpcbind[289]: cannot save any registration Nov 24 14:10:37 xensvrrpcbind[289]: cannot open file = /var/lib/rpcbind/portmap.xdr for writing Nov 24 14:10:37 xensvrrpcbind[289]: cannot save any registration
Nov 24 14:10:39 xensvrsystemd[1]: Cannot add dependency job for unit gssproxy.service, ignoring: Unit gssproxy.service failed to load: No such file or directory.
Nov 24 14:10:39 xensvrsystemd[1]: Starting Kernel Module supporting RPCSEC_GSS... Nov 24 14:10:39 xensvrsystemd[1]: Starting RPCbind Server Activation Socket. Nov 24 14:10:39 xensvrsystemd[1]: Listening on RPCbind Server Activation Socket. Nov 24 14:10:39 xensvrsystemd[1]: Starting RPC Port Mapper. Nov 24 14:10:39 xensvrsystemd[1]: Reached target RPC Port Mapper. Nov 24 14:10:39 xensvrsystemd[1]: Started Kernel Module supporting RPCSEC_GSS. Nov 24 14:10:43 xensvrsystemd[1]: Mounting RPC Pipe File System... Nov 24 14:10:43 xensvrsystemd[1]: Starting RPC Bind... Nov 24 14:10:43 xensvrsystemd[1]: Mounted RPC Pipe File System. Nov 24 14:10:44 xensvrsystemd[1]: Started RPC Bind. Nov 24 14:11:33 xensvrsystemd[1]: Starting RPC security service for NFS client and server... Nov 24 14:11:33 xensvrsystemd[1]: Starting RPC security service for NFS server... Nov 24 14:11:33 xensvrsystemd[1]: Started RPC security service for NFS client and server. Nov 24 14:11:33 xensvrsystemd[1]: rpc-svcgssd.service: control process exited, code=exited status=1 Nov 24 14:11:33 xensvrsystemd[1]: Failed to start RPC security service for NFS server. Nov 24 14:11:33 xensvrsystemd[1]: Unit rpc-svcgssd.service entered failed state.
Nov 24 14:10:45 xensvrrpc.svcgssd[1326]: ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - Unsupported key table format version number Nov 24 14:10:45 xensvrrpc.svcgssd[1326]: unable to obtain root (machine) credentials Nov 24 14:10:45 xensvrrpc.svcgssd[1326]: do you have a keytab entry for nfs/
@ in /etc/krb5.keytab? Nov 24 14:11:41 xensvrrpc.mountd[2083]: Version 1.3.0 starting Nov 24 14:11:41 xensvrrpc.statd[2092]: Version 1.3.0 starting Nov 24 14:11:41 xensvrrpc.statd[2092]: Flags: TI-RPC Nov 24 14:11:50 xensvrsystemd[1]: Starting Quota RPC monitor... Nov 24 14:11:50 xensvrsystemd[1]: Started Quota RPC monitor.
If GSS is properly disabled (is config sufficient?), why is the service start being attempted in the first place? attempt/service non-start could be logged, but should not report ERROR. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c1 boo35 boo35 <9b3e05a5@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |neilb@suse.com Flags| |needinfo?(neilb@suse.com) --- Comment #1 from boo35 boo35 <9b3e05a5@opayq.com> --- Note 2 rpc-*gssd.service files systemctl list-unit-files | grep gss auth-rpcgss-module.service static rpc-gssd.service static rpc-svcgssd.service static both provided by CLIENT 'nfs-client' package, rpm -q --whatprovides /usr/lib/systemd/system/rpc-gssd.service /usr/lib/systemd/system/rpc-svcgssd.service nfs-client-1.3.0-9.1.x86_64 nfs-client-1.3.0-9.1.x86_64 but both (1) providing SERVER security svcs, and (2) looking very similar cat rpc-gssd.service [Unit] Description=RPC security service for NFS client and server DefaultDependencies=no Conflicts=umount.target Requires=var-lib-nfs-rpc_pipefs.mount After=var-lib-nfs-rpc_pipefs.mount ConditionPathExists=/etc/krb5.keytab PartOf=nfs-utils.service Wants=nfs-config.service After=nfs-config.service [Service] EnvironmentFile=-/run/sysconfig/nfs-utils Type=forking ExecStart=/usr/sbin/rpc.gssd $GSSDARGS cat rpc-svcgssd.service [Unit] Description=RPC security service for NFS server DefaultDependencies=no Requires=var-lib-nfs-rpc_pipefs.mount After=var-lib-nfs-rpc_pipefs.mount local-fs.target PartOf=nfs-server.service PartOf=nfs-utils.service After=gssproxy.service ConditionPathExists=|!/run/gssproxy.pid ConditionPathExists=|!/proc/net/rpc/use-gss-proxy ConditionPathExists=/etc/krb5.keytab Wants=nfs-config.service After=nfs-config.service [Service] EnvironmentFile=-/run/sysconfig/nfs-utils Type=forking ExecStart=/usr/sbin/rpc.svcgssd $SVCGSSDARGS Why are there 2 similar services here? Wondering if that's causal at boot. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 boo35 boo35 <9b3e05a5@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|3rd party software |Basesystem Version|unspecified |Leap 42.1 Product|openSUSE.org |openSUSE Distribution Target Milestone|--- |Leap 42.1 QA Contact|opensuse-communityscreening |qa-bugs@suse.de |@forge.provo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c2
Dieter Jurzitza
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c3
--- Comment #3 from Dieter Jurzitza
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c4
Neil Brown
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c5 boo35 boo35 <9b3e05a5@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(neilb@suse.com) --- Comment #5 from boo35 boo35 <9b3e05a5@opayq.com> --- (In reply to Neil Brown from comment #4)
1/ rpc-svcgssd.service shouldn't try do start unless you have the file /etc/krb5.keytab. This is due to the line "ConditionPathExists=/etc/krb5.keytab" in /usr/lib/systemd/system/rpc-svcgssd.service Do you have the file?
Yes
Do you want to have it?
Yes, as I have other occassional, non-RPC-related uses for kerberos.
Can you remove it?
In theory, certainly; in practice, however ...
What then happens?
... Breaks other kerberos usage We already have config options grep -i gss /etc/sysconfig/nfs | grep -v ^# NFS_SECURITY_GSS="no" SVCGSSD_OPTIONS="" GSSD_OPTIONS="" NFS_GSSD_AVOID_DNS="no" if NFS_SECURITY_GSS == no, that should be the sufficient source to disable. Why is ConditionPathExists=/etc/krb5.keytab uses in lieu of that config option to make that decision? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c6
Dieter Jurzitza
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c7
--- Comment #7 from Dieter Jurzitza
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c8
Neil Brown
Why is
ConditionPathExists=/etc/krb5.keytab
uses in lieu of that config option to make that decision?
Partly because that it what upstream nfs-utils does, and there is a lot of value in using the same systemd unit files as upstream (but then, I put that stuff upstream, that that isn't really an excuse). Partly because systemd doesn't make it at all easy to selectively enable/disable services using a env variables in a config file. Is this causing any actual problems apart from the unsightly error message? If not it might be simplest just to find a way to hide the error message. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c9
--- Comment #9 from Neil Brown
nfs-server and nfs-client contain references to gssproxy they should not contain,
I think they *should* contain those references, so that if/when the package is added the dependencies will automatically do the right thing. They are soft dependencies so the missing spec files is not fatal. Maybe systemd should be less verbose about them. I very much want to keep the same unit files as upstream has. I wonder if they is a way to create null unit files that automagically get over-ridden when the real spec files get installed.... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c10 boo35 boo35 <9b3e05a5@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(9b3e05a5@opayq.co | |m) | --- Comment #10 from boo35 boo35 <9b3e05a5@opayq.com> --- (In reply to Neil Brown from comment #8)
Is this causing any actual problems apart from the unsightly error message?
Well, maybe ... http://lists.opensuse.org/wicked-devel/2015-12/msg00000.html NFS-related dependencies are definitely at issue, between that thread and this bug, at least. Whether they're related or not, I don't yet know. I'm poking at it trying to figure out what's causal etc. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c11
--- Comment #11 from Dieter Jurzitza
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c12
Neil Brown
(In reply to Neil Brown from comment #8)
Is this causing any actual problems apart from the unsightly error message?
Well, maybe ...
http://lists.opensuse.org/wicked-devel/2015-12/msg00000.html
This looks like it might be the same as bug 947483 ... which you probably cannot see because it is against SLES12-SP1 rather than opensuse. But I believe SLE12 fixes get into Leap, so it should be fixed.... but it looks like it isn't. The fix removes references to $remote_fs from /etc/init.d/named and /etc/init.d/lwresd
NFS-related dependencies are definitely at issue, between that thread and this bug, at least.
Whether they're related or not, I don't yet know. I'm poking at it trying to figure out what's causal etc.
Yeah, it's complex. And systemd is new and different and there is a bit of a learning curve. systemd doesn't make it at all easy for service execution to be controlled by a separate config file. I've managed something, but I'm not sure I like it. 1/ edit /usr/lib/systemd/scripts/nfs-utils_env.sh and add if [ "$NFS_SECURITY_GSS" == "yes" ]; then SVCGSS_PROG=/usr/sbin/rpc.svcgssd else SVCGSS_PROG=/bin/true fi somewhere before "mkdir -p /run/sysconfig" 2/ add echo "SVCGSS_PROG=$SVCGSS_PROG" in the block of similar commands at the end of that file. 3/ create directory /usr/lib/systemd/system/rpc-svcgssd.service.d and create a file in that directory called "prog.conf" containing [Service] ExecStart= ExecStart=/bin/sh -c "${SVCGSS_PROG} ${SVCGSSDARGS}" 4/ run: systemctl daemon-reload 5/ run: systemctl restart nfs-config.service Now if you "systemctl restart rpc-svcgss" it should not run the daemon (unless NFS_SECURITY_GSS is "yes"). I think I'd rather modify rpc-svcgssd in some what... not sure yet. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c13 --- Comment #13 from boo35 boo35 <9b3e05a5@opayq.com> --- (In reply to Neil Brown from comment #12)
Well, maybe ...
http://lists.opensuse.org/wicked-devel/2015-12/msg00000.html
This looks like it might be the same as bug 947483 ... which you probably cannot see because it is against SLES12-SP1 rather than opensuse. But I believe SLE12 fixes get into Leap, so it should be fixed.... but it looks like it isn't.
Since opensuse/suse are "sharing", that situation should be addressed/improved (elsewhere)
The fix removes references to $remote_fs from /etc/init.d/named and /etc/init.d/lwresd
On the machine in question here, ls -al /etc/init.d/*{lwres,named}* ls: cannot access /etc/init.d/*lwres*: No such file or directory ls: cannot access /etc/init.d/*named*: No such file or directory
systemd doesn't make it at all easy for service execution to be controlled by a separate config file. I've managed something, but I'm not sure I like it. ... Now if you "systemctl restart rpc-svcgss" it should not run the daemon (unless NFS_SECURITY_GSS is "yes").
Well, that _is_ the end goal.
I think I'd rather modify rpc-svcgssd in some what... not sure yet.
but, this still nags at me as being incorrect /usr/lib/systemd/system/rpc-svcgssd.service ... ConditionPathExists=/etc/krb5.keytab ... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c14
--- Comment #14 from Neil Brown
but, this still nags at me as being incorrect
/usr/lib/systemd/system/rpc-svcgssd.service ... ConditionPathExists=/etc/krb5.keytab ...
I think that is definitely correct. If /etc/krb5.keytab is not present, thenre there is no point running the service. The problem is that it isn't sufficient. We want another test - if disabled in config file then don't run it either. systemd doesn't help with that. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558 http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c15 --- Comment #15 from boo35 boo35 <9b3e05a5@opayq.com> --- Perhaps too many places for configuration? Yep, asking about the wisdom of /etc/sysconfig/* ... Some comments/approaches for thought https://coreos.com/os/docs/latest/using-environment-variables-in-systemd-uni... http://0pointer.de/blog/projects/on-etc-sysinit.html TBH, I have absolutely _no_ idea of the strategy & thinking @ *Suse of maintaining/deprecating /etc/sysconfig/usage. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c16
--- Comment #16 from Neil Brown
Perhaps too many places for configuration? Yep, asking about the wisdom of /etc/sysconfig/* ...
Good question to ask. I've just posed it on a suse-internal list. Maybe something helpful will arise.
Some comments/approaches for thought
https://coreos.com/os/docs/latest/using-environment-variables-in-systemd- units.html
Evironment variables are clearly useful.
I've seen that. I don't agree.
TBH, I have absolutely _no_ idea of the strategy & thinking @ *Suse of maintaining/deprecating /etc/sysconfig/usage.
Me neither. Maybe I'll find out. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=956558
http://bugzilla.opensuse.org/show_bug.cgi?id=956558#c17
--- Comment #17 from Neil Brown
The decision for systemd has been made, now IMHO there is a need to comply to it.
alternately, we could fix systemd. I really do think that systemd is at fault here. This is an optional dependency and so should not generate a warning. Probably a "debug" or "notice" would be appropriate, which are easy to filter out. I cannot re-assign this bug to systemd-maintainers that this is a secondary issue. the rpc.svcgssd messages are the primary issue. So I suggest that you create a new bug against systemd. It might be worth noting that there is a real (though minor) bug in systemd relating to this. If you create an empty file e.g. /etc/systemd/system/gssproxy.service, the message with change from "failed to load" to "is masked". The code tries to make this a "debug" level message for "Wants" dependencies. But it fails because error codes have changed. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com