* Bernhard Voelker [04-22-14 11:11]:

On 04/22/2014 04:53 PM, Patrick Shanahan wrote:

...

Apr 22 10:39:14 Crash nfs[15602]: mount.nfs: mount options: "rw,bg,retry=5,_netdev" Apr 22 10:40:42 Crash systemd[1]: nfs.service operation timed out. Terminating. Apr 22 10:40:42 Crash systemd[1]: Failed to start LSB: NFS client services. Apr 22 10:40:42 Crash systemd[1]: Unit nfs.service entered failed state. Apr 22 10:43:28 Crash mount[16304]: mount to NFS server '192.168.1.3' failed: timed out, retrying Apr 22 10:43:28 Crash mount[16303]: mount to NFS server '192.168.1.3' failed: timed out, retrying Apr 22 10:43:28 Crash mount[16305]: mount to NFS server '192.168.1.3' failed: timed out, retrying Apr 22 10:47:43 Crash mount[17037]: mount to NFS server '192.168.1.3' failed: timed out, retrying Apr 22 10:47:43 Crash mount[17038]: mount to NFS server '192.168.1.3' failed: timed out, retrying Apr 22 10:47:43 Crash mount[17041]: mount to NFS server '192.168.1.3' failed: timed out, retrying

Did the NFS server on 192.168.1.3 also die?

yes but restarted and reports active/running

Are the services needed by NFS running?

$ /usr/sbin/rpcinfo -p 192.168.1.3

rpcinfo -p 192.168.1.3 rpcinfo: can't contact portmapper: RPC: Remote system error - Connection timed out

What shares does it export?

on 192.168.1.3, exportfs: / <world> /mnt/ExTernal <world> /mnt/photos <world> /var <world> /srv <world> /mnt/g7_unk3 <world> /home <world>

$ /usr/sbin/showmount -e 192.168.1.3

showmount -e 192.168.1.3 clnt_create: RPC: Port mapper failure - Timed out Ok, appears there is more info but it means ?? to me :^( tks, -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

* Bernhard Voelker [04-22-14 12:02]:

On 04/22/2014 05:38 PM, Patrick Shanahan wrote:

...

rpcinfo -p 192.168.1.3 rpcinfo: can't contact portmapper: RPC: Remote system error - Connection timed out

...

showmount -e 192.168.1.3 clnt_create: RPC: Port mapper failure - Timed out

Both timed out, i.e. there is a networking problem - as Anton already suspected.

Use a TCP-traceroute to the portmapper (111) to see where it's blocked:

$ traceroute --tcp -p 111 192.168.1.3

Well, since I stopped the server firewall, restart nfs, mounted the shares from remote and restarted the firewall, there is no failure. But I will retain this and if it happens again, try. What I do not understand is how stopping the firewall and then restarting it does not destroy the connection as it apparently thought it was originally supposed to do. And no changes have been made to the server in between nfs working and it not :^( ??? puzzled. tks, -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

* Bernhard Voelker [04-22-14 13:47]:

On 04/22/2014 06:31 PM, Patrick Shanahan wrote:

...

Well, since I stopped the server firewall, restart nfs, mounted the shares from remote and restarted the firewall, there is no failure. But I will retain this and if it happens again, try.

What I do not understand is how stopping the firewall and then restarting it does not destroy the connection as it apparently thought it was originally supposed to do. And no changes have been made to the server in between nfs working and it not :^( ??? puzzled.

I'm no NFS expert, but as far as I understand "mount -t nfs ..." first tries to resolve the NFS service on the server side by contacting the portmapper. Therefore my (wild) guess is that only NFS is configured in the firewall, but the portmapper is blocked. Once the connection is up, the portmapper is not needed anymore, and you could start the firewall ... and NFS would still be working. Just a guess.

BTW: you can consult your firewall logs - there should be an entry about an connecting attempt to port 111 being blocked.

BTW2: there may even be 2 (or more) firewalls be involved, of course ...

In "BTW2", disabling *one* and suceeding would appear to disprove? Firewall logs from the server show access from 192.168.1.10 on dpt=111 to 192.168.1.3 dropped until I stopped the firewall and accessed the shares. Upon restarting the firewall access is allowed ??? No other changes were made on the server, 192.168.1.3, before or after problem occurred. tks, -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 4/22/2014 1:41 PM, Anton Aylward wrote:

On 04/22/2014 12:31 PM, Patrick Shanahan wrote:

...

What I do not understand is how stopping the firewall and then restarting it does not destroy the connection as it apparently thought it was originally supposed to do.

That bothers me as a well There could be a security hole sitting there.

Firewalls usually do not disrupt pre-existing connections/sockets. Removing the firewall, simply allows the source and destination to continue unhindered. Reestablishing the firewall, and that same stream is allowed because it meets the rules. A simple firewall is distinct from a router. But even most iptables based router implementations would allow established connections to continue. -- _____________________________________ ---This space for rent--- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

* Anton Aylward [04-22-14 17:32]: [...]

Now I don't have access to to Patrick's logs, so maybe something else is going on, but one interpretation of what *has* been reported is that the firewall allows established connections to continue though they don't meet the rules. If that is the case then there is a security problem.

From the server, 192.168.1.3 egrep 192.168.1.10.*111 /var/log/firewall

http://wahoo.no-ip.org/~pat/firewall.log.txt What "rules" are of interest and how to show them?

I agree with you, John, about what firewalls OUGHT to do. I am concerned about the evidence as to what this firewall *IS* doing.

interesting :^) -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

* David Haller [04-24-14 02:51]:

Hello,

On Tue, 22 Apr 2014, Patrick Shanahan wrote:

...

...

From the server, 192.168.1.3 egrep 192.168.1.10.*111 /var/log/firewall

http://wahoo.no-ip.org/~pat/firewall.log.txt

I notice that the failed attempts were when the client (.10) used priviledged ports (under 1024) as source-ports. When it worked, unpriviledged ports were used.

...

What "rules" are of interest and how to show them?

Those on both machines from:

iptables -vL input_int | egrep 'nfs|rpc'

server: 192.168.1.3 iptables: No chain/target/match by that name. client: 192.168.1.10 iptables: No chain/target/match by that name.

iptables -vL output_int | egrep 'nfs|rpc'

server: iptables: No chain/target/match by that name. client: iptables: No chain/target/match by that name.

iptables -vL input_ext | egrep 'nfs|rpc'

server: 0 0 LOG udp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW udp dpt:sunrpc LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:sunrpc 0 0 LOG tcp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW tcp dpt:sunrpc LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:sunrpc 0 0 LOG udp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW udp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs client: 0 0 LOG udp -- any any anywhere anywhere ctstate NEW udp dpt:sunrpc LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:sunrpc 0 0 LOG tcp -- any any anywhere anywhere ctstate NEW tcp dpt:sunrpc LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:sunrpc 0 0 LOG udp -- any any anywhere anywhere ctstate NEW udp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs

iptables -vL output_ext | egrep 'nfs|rpc'

server: 0 0 LOG udp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW udp dpt:sunrpc LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:sunrpc 0 0 LOG tcp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW tcp dpt:sunrpc LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:sunrpc 0 0 LOG udp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW udp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs 0 0 LOG tcp -- any any anywhere anywhere limit: avg 3/min burst 5 ctstate NEW tcp dpt:nfs LOG level warning tcp-options ip-options prefix "SFW2-INext-ACC-RPC " 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:nfs client: iptables: No chain/target/match by that name. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/22/2014 06:38 PM, John Andersen wrote:

The security violation you postulate was letting a connection happen BEFORE you launched your firewall.

NO! The security violation I postulated involved the firewall being down FOR ANY REASON. Lets leave aside misconfiguration ... In Patrick's case we have an example of the generic Something isn't working... I wonder if the firewall is in the way.... Lets take it down for a while and experiment... Patrick remembered to put it back up PROMPTLY, but we don't know what the critical aperture is. I've read that newly installed Windows machines on the 'Net can be hacked in a couple of minutes between being connected and firewall software being installed/configured. My POV is that security rule is a security rule. If the firewall rules prohibit a connection then they should ensure that connection does not exist as opposed to merely 'cannot be initiated'. I understand the explanation you give and don't disagree with it. Yes you have explained what is going on. I think you are quite correct.

You can ssh into a remote server, dick around with the firewall settings and stop/restart it without worrying about killing your own ssh connection, and potentially leaving your remote machine in a broken and vulnerable state. That original connection will persist.

I don't see the problem. The rules permit a ssh connection. My 'proposed' idea about tearing down connections that violate the rules would allow this connection because the rules permit it. I'm NOT saying that starting the firewall should tear down all connections. I'm talking about connections that the firewall would otherwise prohibit. A few years ago, not long after AIX was released, I had an argument with a senior executive at IBM. My attitude towards security configuration lies on the side of "everything that is not explicitly allowed is prohibited". I held that all the services that inetd, for example, supported should be default off. He insisted that the way AIX was then shipped with most of them default on was correct, and that it was the sysadmins responsibility to determine what needed to be turned off for security purposes. I had seen too many novice admins who didn't know enough about security, never mind fringes of security that hackers exploit, which is why I had come to adopt the 'everything off' stance. If the users needed it they would demand it and it could be turned on. There are services such as tftp that are fringe/specialized yet offer entrances to hackers, and I can see no reason to ship them default on. In the long run I feel justified; most of the systems I deal with now - Linux, AIX and other non-Windows systems - are shipped 'default off' for many facilities, especially inetd and xinetd. We can't expect sysadmins to be aware of every security hole that might be generated. We need to have aggressive defaults. Yes, I realise that many of these will require, no will DEMAND more vigilance on the part of the people building the isntllation kits. We've recently had a discussion of Apparmor and Dovecot and I think that's a good posterboy. Installing Dovecot _should_ include installing the Apparmor rules that allow it to be run. Heck, installing it bring in the relevant PAM entries, doesn't it? -- "Realizing the importance of the case, my men are rounding up twice the number of usual suspects" -- Cpt Renault to Major Strasser, Cassablanaca -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/22/2014 07:16 PM, Patrick Shanahan wrote:

* Anton Aylward [04-22-14 19:11]: [...]

...

In Patrick's case we have an example of the generic

Something isn't working... I wonder if the firewall is in the way.... Lets take it down for a while and experiment...

Patrick remembered to put it back up PROMPTLY, but we don't know what the critical aperture is.

I've read that newly installed Windows machines on the 'Net can be hacked in a couple of minutes between being connected and firewall software being installed/configured.

More the point, I disconnected the modem/router while the firewall was down :^). Only security would be local/internal and I am in *complete* control there!

And I only did the mechanical fix because it is much faster than reconfiguring the router to deny ALL outside.

God security practice never the less :-) -- No one who's seen it in action can say the phrase "government help" without either laughing or crying. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/22/2014 07:36 PM, John Andersen wrote:

On 4/22/2014 4:09 PM, Anton Aylward wrote:

...

...

...

You can ssh into a remote server, dick around with the firewall settings and stop/restart it without worrying about killing your own ssh connection, and potentially leaving your remote machine in a broken and vulnerable state. That original connection will persist.

...

I don't see the problem. The rules permit a ssh connection. My 'proposed' idea about tearing down connections that violate the rules would allow this connection because the rules permit it.

Ah, I see. You're obviously one of those guys who never makes an error! ;-) Me? I fuckup all the time. I accidentally get my Nics mixed up, deny inbound ssh, make typos that leave everything blocked, or nothing at all blocked, etc.

I'm a belt and braces and rivets and measure twice before deciding not to cut today but think about it overnight type of guy. I once referred t myself as a 'paid to be paranoid' type of sysadmin and was taken to task for that, being told that paranoia is an illness and not a security stance. I try very very had to make sure my errors are (a) low impact and (b) reversible. Just don't mention that to SheWhoSallNotBeNamed, please.

...

I'm NOT saying that starting the firewall should tear down all connections. I'm talking about connections that the firewall would otherwise prohibit.

Yeah, I understand. Give it a few weeks and SystemD will probably take over that too, but in the meantime netfilter doesn't know about those open connections, and making it do so, may open as many security holes as it closes.

Patrick's assumption that a *pre-existing* connection should be stopped by a new firewall rule is simply not the case today, but it is a common misconception. So much so that it is FAQ Question 4B in the Shorewall Firewall guide. http://shorewall.net/3.0/FAQ.htm#faq4b

Indeed. As you point out, that is the way iptables has been set up to work. I'm not saying that is not how things work; I'm just saying that, rightly or wrongly, there is a potential security risk. Patrick's strategy of disconnecting the router was one very effective way of mitigating that risk. Could iptables be set up so that a firewall will tear down an existing connection that violates its rules? I have no doubt it could. The way things work not seem to be a well established convention, sort of like what side of the road to drive on. So long as everyone follows the same convention we have a predictable behaviour. The 'logical' argument that we should drive on the let in the northern hemisphere and on the right in the southern so as to counteract the natural spin of tornadoes is totally beside the point: like in RubyonRails, convention trumps everything. I think Patrick is eminently sensible. A simple and effective solution. He's obviously one of us 'security paranoid' types. -- intaxification (n): Euphoria at getting a tax refund, which lasts until you realize it was your money to start with. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

* John Andersen [04-22-14 19:38]: [...]

Patrick's assumption that a *pre-existing* connection should be stopped by a new firewall rule is simply not the case today, but it is a common misconception. So much so that it is FAQ Question 4B in the Shorewall Firewall guide. http://shorewall.net/3.0/FAQ.htm#faq4b

Patrick should test by restarting NFS, not *just* restarting the firewall.

During an access lull, I restarted the server and the shared directories were automagically handled as expected. Manupilating the firewall was not necessary. # systemctl status nfs nfsserver nfs.service - LSB: NFS client services Loaded: loaded (/etc/init.d/nfs) Drop-In: /run/systemd/generator/nfs.service.d └─50-insserv.conf-$remote_fs.conf Active: active (running) since Tue 2014-04-22 20:38:49 EDT; 1h 46min ago Process: 2225 ExecStart=/etc/init.d/nfs start (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs.service ├─2267 /usr/sbin/rpc.gssd -D -p /var/lib/nfs/rpc_pipefs └─2283 /usr/sbin/rpc.idmapd -p /var/lib/nfs/rpc_pipefs Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: can't open /var/lib/nfs/rpc_pipefs/gssd/clntXX/info: No such file or directory Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: failed to read service info Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: can't open /var/lib/nfs/rpc_pipefs/gssd/clntXX/info: No such file or directory Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: failed to read service info Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: can't open /var/lib/nfs/rpc_pipefs/gssd/clntXX/info: No such file or directory Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: failed to read service info Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: can't open /var/lib/nfs/rpc_pipefs/gssd/clntXX/info: No such file or directory Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: failed to read service info Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: can't open /var/lib/nfs/rpc_pipefs/gssd/clntXX/info: No such file or directory Apr 22 20:45:02 wahoo rpc.gssd[2267]: ERROR: failed to read service info nfsserver.service - LSB: Start the kernel based NFS daemon Loaded: loaded (/etc/init.d/nfsserver) Active: active (running) since Tue 2014-04-22 20:38:49 EDT; 1h 46min ago Process: 2266 ExecStart=/etc/init.d/nfsserver start (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfsserver.service ├─2315 /usr/sbin/rpc.mountd └─2319 /usr/sbin/rpc.statd --no-notify Apr 22 20:38:49 wahoo rpc.mountd[2315]: Version 1.2.8 starting Apr 22 20:38:49 wahoo rpc.statd[2319]: Version 1.2.8 starting Apr 22 20:38:49 wahoo rpc.statd[2319]: Flags: TI-RPC Apr 22 20:38:49 wahoo systemd[1]: Started LSB: Start the kernel based NFS daemon. I have no idea why the gssd errs as it is not configured in server or client..... go figure ????..... :^) -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2014-04-23 04:30, Patrick Shanahan wrote:

I have no idea why the gssd errs as it is not configured in server or client.....

I see those errors here, too. No idea what they are, things work. So far... -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

On 04/22/14 17:38, Patrick Shanahan wrote:

* Bernhard Voelker [04-22-14 11:11]:

...

Are the services needed by NFS running?

$ /usr/sbin/rpcinfo -p 192.168.1.3

rpcinfo -p 192.168.1.3 rpcinfo: can't contact portmapper: RPC: Remote system error - Connection timed out

In your first email you reported that rpcbind runs. If that happens again, try to restart rpcbind.socket, too. Maybe you need to stop rpcbind.socket, restart rpcbind.service, and start rpcbind.socket. I had this problem, too. My interim solution has been to stop and disable rpcbind.socket. In my setup, stuff works without systemd-socket-activated rpcbind, the daemon alone is sufficient. I don't know that the Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod, Roedermark, Germany Email: jschrod@acm.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org