OpenSSH 3.3p1 on Suse 7.1 - lockup?
I've read most of the recent discussion about Openssh 3.3p1 but haven't seen this particular issue so... I installed the 3.3p1 patch on several Suse 7.1 boxes, 7 in the UK that I can reach locally yesterday and they all seem fine and 5 more in another country that I can't get to without a plane ticket :-( Sequence of installation was to use YOU to apply the patch while logged on via SSH on all machines then to shutdown -r now them, wait a bit then log back on. So far so good on all boxes. However, within 30 minutes of the reboot on the 5 machines that I cannot reach locally, 2 of them have become inaccessible. They don't ping and nmap with the -P0 option doesn't get any response from them. That looks pretty dead to me. Neither of these two machines has done this before and up until now, they've up and running for 113 days without any issue. I can't categorically state that it is the Openssh patch that's done this since I can't find anyone around to go and look at them to find out if they're sitting with an Ooops message or what's wrong with them. But it's suspicious enough that I've backed out 3.3p1 on the machines I can still get to and gone back to 2.9.9p2-98 for now. And, yes, if I'd read the mailing list before I put the patches on then I probably wouldn't have bothered :-) With issues like this, maybe Suse should pull these particular patches off the web page/ftp site? Especially since it appears that the 2.9.9p2 rpm's aren't vulnerable to the exploit that the advisory is meant to fix. Trevor Hemsley, Security Specialist, Atos Origin Ltd, Whyteleafe, +44-(0)1883-628139 [This electronic transmission and any files attached to it are strictly confidential and intended solely for the addressee. If you are not the intended addressee, you must not disclose, copy or take any action in reliance of this transmission. If you have received this transmission in error, please notify us by return and delete the same. The views expressed in this electronic transmission do not necessarily reflect those of Atos Origin or any of its subsidiary companies. Although the sender endeavours to maintain a computer virus free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Thank You.]
I can't categorically state that it is the Openssh patch that's done this since I can't find anyone around to go and look at them to find out if they're sitting with an Ooops message or what's wrong with them. But it's suspicious enough that I've backed out 3.3p1 on the machines I can still get to and gone back to 2.9.9p2-98 for now.
I have removed all openssh packages from the ftp server to make sure that the packages don't break anything when installed. We will have better packages to offer on Monday.
And, yes, if I'd read the mailing list before I put the patches on then I probably wouldn't have bothered :-)
With issues like this, maybe Suse should pull these particular patches off the web page/ftp site? Especially since it appears that the 2.9.9p2 rpm's aren't vulnerable to the exploit that the advisory is meant to fix.
Trevor Hemsley, Security Specialist, Atos Origin Ltd, Whyteleafe, +44-(0)1883-628139
Thanks, Roman. -- - - | Roman Drahtmüller <draht@suse.de> // "You don't need eyes to see, | SuSE Linux AG - Security Phone: // you need vision!" | Nürnberg, Germany +49-911-740530 // Maxi Jazz, Faithless | - -
I installed the 3.3p1 patch on several Suse 7.1 boxes, 7 in the UK that I can reach locally yesterday and they all seem fine and 5 more in another country that I can't get to without a plane ticket :-( Sequence of installation was to use YOU to apply the patch while logged on via SSH on all machines then to shutdown -r now them, wait a bit then log back on. So far so good on all boxes. However, within 30 minutes of the reboot on the 5 machines that I cannot reach locally, 2 of them have become inaccessible. They don't ping and nmap with the -P0 option doesn't get any response from them. That looks pretty dead to me.
Neither of these two machines has done this before and up until now,
Never update packages and then reboot, if it's not needed. Normally rcsshd restart would have been enough (no kernelpatch was made). Shure, YOU did only update ssh-package - There were apache-updates as well! Next time first have a look, what has been updated (YOU shows it after downloading and setup). Then restart services by hand and only reboot, if kernel is patched. I handle things with caution, since I had several bad experiances. they've
up and running for 113 days without any issue.
I can't categorically state that it is the Openssh patch that's done this since I can't find anyone around to go and look at them to find out if they're sitting with an Ooops message or what's wrong with them. But it's suspicious enough that I've backed out 3.3p1 on the machines I can still get to and gone back to 2.9.9p2-98 for now.
Openssh.org did send the really useful announcement too late. This says, that the old ssh shipped with suse is not vulnerable in the default config. Another thing is, that the 3.3 is too buggy (no kompression, pam not 100% supported). Philippe
On Saturday 29 June 2002 12:34 am, Philippe Vogel wrote:
Never update packages and then reboot, if it's not needed.
Really? If you never test that the machine will really reboot after an update you leave this unplesent discovery to happen after the next power failure. That way, things you broke with the update but did not discover will surely be a great supprise. After applying upgrades to any packages which are normally started at system boot up, a responsible administrator will TEST that it actually will come up properly. -- _________________________________________________ No I Don't Yahoo! And I'm getting pretty sick of being asked if I do. _________________________________________________ John Andersen / Juneau Alaska
* Philippe Vogel wrote on Sat, Jun 29, 2002 at 10:34 +0200:
Never update packages and then reboot, if it's not needed.
I don't think so. Better risk the need for manual attention after a failed reboot on a scheduled, known point that after i.e. a power loss! Otherwise you risk that the servers will not boot up successfully after an power- or UPS failure. In that case, you'll not only have to work around the usual things, but additionally against the wrong configuration - well, and this maybe weeks after you changed something! I saw big troubles when booting up complete server centers in after such failures. It's important to know the boot order, it may happend that failures occure since hosts other servers depend on need just to much time to boot (DNS-Servers, for instance). In such a case you may stand in front of a system out of corder completly, just since a DNS server *was* not ready. This is very difficult to trace! Maybe it's a good practise to reboot after such changes. Maybe in the evening on a scheduled, communicated time. Noone should blame you if you reboot something in such cases, but if anything fails later, they will!
Then restart services by hand and only reboot, if kernel is patched. I handle things with caution, since I had several bad experiances.
This has disadvantages. We had a nice detail problem once upon a time. For some unknown reason, on one host the /etc/rc.config file was destroyed (some binary contents). Well, it was a big problem to restore it from backup, we had the last working on a history tape only. All the time the server was down! Another indident: rc.config has to START_WHATVER="". The first was "yes", and on first view nobody noted that at the end of file there was another entry with "no". Starting per Hand worked of course... Well, this took manual investigation at a time we had much other trouble... oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.
Hi, first, this is only partly security related. Second this is a philosophical question. We are not talking about critical systems, as such may not be rebooted, even in case of power loss. For systems that are meant to run all the time, the automatic start of services is just a backup system. If this is so important to someone, the only solution is to have an identical system to test (and even that is not a 100% structure). Usually such systems may not be rebootet just to test if they resume normal operation if they are rebooted after a power loss or such. So it is clear, that we have to live with whatever comes, we just have to be prepared to it. I wish to thank whoever reported this, as it gave me the opportunity to test this with a backup machine, and this backup system came up after a reboot after the upgrade without much problems. I assume the critical systems will too, but I cannot reboot them just to test if this will work (not to mention that some systems need about an hour from the shutdown command till they are ready to service again). I also wish to thank the SuSE team for the fast reaction, even if it was not necessary to provide an upgrade to 3.3x, it was the optimal thing to do. Not to mention that security is also a psychological structure, and being able to tell customers "the manufacturer recommended prcedure is allready in place" makes them feel good. mike
participants (6)
-
Hemsley, Trevor -
John Andersen -
Philippe Vogel -
Roman Drahtmueller -
Steffen Dettmer -
Thomas Michael Wanka