I'm going to guess that you didn't install your Xen on your HostOS using "the" recommended standard procedure... Which is to use the YaST Virtualization module. If you did that, then you shouldn't have variations. Also, you would be prompted to install a bridge device. That said, I don't know how old your HostOS installations are (except for any you say you just installed). I am not involved with or track what the Xen team does, but the Linux kernel has undergone several related changes over the past 8 years or so... There first was the "pre-ip" era when we used usermode utilities like ipconfig regularly. Then, the ip tools were introduced, I don't remember if they were introduced initially as User mode tools are not. Today we have version 2 of ip tools which are implemented as kernel mode modules which comprise not only of the expanded ip commands which more or less replace everything "pre-ip tools" but also bridging devices and more. In other words, practically everything networking that used to be implemented as its own utility can now likely be done with an "ip" command. That <might> explain the differences in your two installations but shouldn't be a cause of problems... The "old way" should still be supported, but as complex as the modifications to the Xen kernel likely are, could be signposts to possible issues. Bottom line is that I don't think you should have to get down into the weeds as you've done and might be counter-productive, higher up methods should work that accommodate whatever is happening at a lower level. Blabbering away... Tony On Mon, Dec 23, 2019 at 1:26 AM Jan Beulich wrote:

On 21.12.2019 07:31, Glen wrote:

...

So just for laughs, I ran an lsmod in both modes, and sorted and diffed them:

The "clean" guest (which appears to be stable), has these four kernel modules not present on the upgraded guest:

iptable_raw nf_conntrack_ftp nf_nat_ftp xt_CT

The "dup'ped" guest (which seems to be crashable on a large local rsync) has these modules not present on a clean install:

auth_rpcgss br_netfilter bridge

One of these two is, according to my experience, a fair candidate for your problems. I'm not a networking specialist at all, so I can't give any suggestions on how to convert the upgraded guest to a network config not requiring these modules. (Trying to get rid of br_netfilter alone may be easier, but again I'm not really knowledgeable in this area at all.)

Jan

...

grace intel_rapl ipt_MASQUERADE llc lockd nf_conntrack_netlink nf_nat_masquerade_ipv4 nfnetlink nfs_acl nfsd overlay sb_edac stp sunrpc veth xfrm_algo xfrm_user xt_addrtype xt_nat

Both guests share these additional sysctl.conf settings:

kernel.panic = 5 vm.panic_on_oom = 2 vm.swappiness = 0 net.ipv6.conf.all.autoconf = 0 net.ipv6.conf.default.autoconf = 0 net.ipv6.conf.eth0.autoconf = 0 net.ipv4.tcp_fin_timeout = 10 net.ipv4.tcp_tw_reuse = 0

The dup'ped guest has these additional sysctl.conf settings:

net.ipv4.tcp_tw_recycle = 0 net.core.netdev_max_backlog=300000 net.core.somaxconn = 2048 net.core.rmem_max=67108864 net.core.wmem_max=67108864 net.ipv4.ip_local_port_range=15000 65000 net.ipv4.tcp_sack=0 net.ipv4.tcp_rmem=4096 87380 67108864 net.ipv4.tcp_wmem=4096 65536 67108864

all of which have, more or less, worked well in the past (when everything was on 42.3) and may or may not be relevant here.

I'm sorry, I feel like I'm missing something obvious here, but I can't see it. I would be grateful for any guidance or insights into this. Yes, in addition to trying to upgrade my client in place to 15.1, I could just build a new guest by hand, but that would be even more time-consuming and seems like it should not be necessary. If I might quote from the kernel, "Dazed and confused, but trying to continue" is exactly how I'm feeling here. Why could this guest be hanging? Why does an NMI bring it back? What should I do next? Anything anyone would be willing to point me to or suggest would be gratefully appreciated.

Glen

-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Jan, Tony - Thank you both for your responses. I have more information now which might be helpful, I'll provide it after I answer your comments. On Mon, Dec 23, 2019 at 1:26 AM Jan Beulich wrote:

...

br_netfilter bridge One of these two is, according to my experience, a fair candidate for your problems.

Thank you. I'll focus in on these and see what I can do. On Mon, Dec 23, 2019 at 11:00 AM Tony Su wrote:

I'm going to guess that you didn't install your Xen on your HostOS using "the" recommended standard procedure... Which is to use the YaST Virtualization module. If you did that, then you shouldn't have variations. Also, you would be prompted to install a bridge device.

Sorry, I was not clear, my fault. My HostOS is OpenSuse 15.1 across all hosts. On two of the hosts, it is a fresh load from the downloaded ISO, with only the defaults plus the Xen patterns selected. Following the fresh load, I did a zypper update, and I did the recommend standard procedure, using Yast2 to install the virtualization support. That process did indeed prompt me to create a bridge, and I did. It seemed to me to be the same procedure. The other two hosts were fresh loaded (in the past) at 42.3, using the same procedure then, and have since been zypper-dup'ped to 15.0 and then 15.1 per the upgrade procedure. All four hosts seem "clean"... and the problem exists with guests on all four hosts. But to be clear - the hosts are not freezing up or losing network connectivity at all. The hosts are fine. It is only the guests that are having issues.

That said, I don't know how old your HostOS installations are (except for any you say you just installed)

The two fresh hosts were loaded about 6 weeks ago. The other two were dup'ped about 12 weeks ago. All have been zypper-updated to the latest stuff since then. All four hosts run only Xen, no other stuff at the same time. Just stock OpenSuse 15.1, and Xen Dom0.

Blabbering away...

Please continue! I read everything else you said with interest. It's often the case that one thing one person says can trigger something for someone else, and I'm hoping that happens here. I am very grateful for the background, history and detail. On Mon, Dec 23, 2019 at 11:20 AM Tony Su wrote:

AFAIC all your /etc/sysctl/ settings look benign,

Thanks.

But be aware that there is an effort to deprecate /etc/sysctl.

I wasn't aware of that *sigh* but thank you.

Your post suggests you think that your problem might be network or disk related...

Maybe. I honestly don't know. All I know is, when I have a guest running on any of my hosts, even if that guest is idle (because, for example, it's a copy of a production machine) and therefore not getting internet traffic or usage, I can make the host crash by rsyncing a lot of data over a crossover cable at maximum speed. I haven't tested letting the host "just sit there", I suppose that'd be a good bracketing test; for now, it seems like an rsync read triggers the issue, so I assume (perhaps incorrectly) it's network or disk. Everything else in your email read, understood, and appreciated, and

https://sites.google.com/site/4techsecrets/optimize-and-fix-your-network-con...

I will read that after I send this. So where I am now is this: I have a guest machine image running 42.3. There are four copies of this guest, running on four different hosts. The hosts are running 15.1. Two are fresh loads, two are zypper-dup'ped from 42.3 fresh loads. The guest machines have had problems on all four hosts. 1. The "production" guest is running 42.3 When its host was also on 42.3, it was rock solid. When I dup'ped the host to 15.1, the guest started going into the weeds every 5-7 days at random. This is the one I first reported in the 42.3 thread. Olaf suggested installing the SLE12-SP5 kernel on that guest. I did that roughly 72 hours ago. So far no issues, but it needs more time. I previously thought that I had to destroy/recreate this guest (as I mentioned in my thread); I now realize (see #2 below) that if it crashed again I should be able to recover it with an xl trigger nmi. 2. One of the backup guests, whose job it was to just rsync from production, is a stock 42.3 guest still running the 4.4 (42.3) kernel. It's host has also been dupped to 15.1. It has never had a problem, until today, when it went into the weeds. I was able to recover it using xl trigger nmi. 3. A third backup guest is also a stock 42.3 guest running the 4.4 kernel. Its host, however, was a fresh load of 15.1. It has locked up once as well. 4. My fourth guest is a copy of the original 42.3 which the guest itself has been zypper-dup'ped to 15.1 I have been copying data from this machine, which causes it to freeze up, which is also recoverable with an NMI. So.... * Problem exists on any 15.1 host, whether fresh loaded or dup'ped. * Problem exists on multiple copies of this particular guest, whether at 42.3 or dup'ped to 15.1 The SLE12-SP5 kernel *might* have resolved this, but more time is needed to be sure. Problem seems to be confined to (ccopies of) this particular guest , for this particular client. * Problem *seems* to not exist on any of my other guests from different origins/other clients running 42.3 or 15.1, dup'ped or fresh, although I suppose there could still be broken guests that just haven't crashed yet. But testing a fresh loaded 15.1 guest I could not get it to crash. * Problem *seems* to be related to utilization of network or disk, the more utilization, the more frequent the hang. No log output on guest at all to indicate why. Virtual hardware just.... stops. I'm literally just guessing at this point, but that is why I suspect something in this particular guest. It could be a legacy thing - this guest was last freshly loaded at 13.1, and has been zypper-dup'ped step by step ever since. That could be it, but I have other guests with similar histories that are not malfunctioning. This guest is also running the extra modules I mentioned, and I'm going to look at that. In addition, this guest runs things like Docker, Elastic Search, Kibana, and other programs that tend to eat CPU and IO even when they're idle (grumble grumble). I can't help but wonder if one of these might be contributing. But the thing is, when the machine hangs, it literally just... hangs. The host can't detect it, it still shows b/r states and normal usage, the only thing the host sees is "Guest network stalled". But the guest console is frozen, and literally you'd think you'd have to destroy/recreate (as I did). Only by chance did I discover that an NMI recovered it. After recovery, the host literally just starts running again, just as it was, right from where it left off, except for the clock. So it's as if the guest is literally stopping at a (virtual) hardware level... and hanging.... and then continuing on when I NMI it. So it seems to me that processes on the guest, even Docker, would show some sign of trouble beforehand. But there is none. Loads are normal, iotop is normal, I've literally say on the regular top with a 1.0 second refresh and had a guest hang on me right while I was looking at it - and there is literally no warning at all. I mean at this point I'm toying with an every-minute cronjob on the host like: * * * * * ping -c4 -w5 [myhostip] &>/dev/null || xl trigger [guestdomid] nmi Meaning that, as soon a the host can't ping the guest, assume it's in the weeds, and NMI it. I shouldn't have to live like that, but at least I'd sleep through the night. So I hope this clarifies. I'm kind of depressed that zypper-dup'ping the guest to 15.1 didn't solve this - I hope the SLES kernel does. But if not, it seems like a fresh load/recreate of the guest is all I can do, and I'll do it if I have to, but I'm hoping this rings a bell for someone who can point me to some additional data or solution. Thank you all for your patience and support! With great respect and appreciation, Glen -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

On Tue, Dec 24, 2019 at 6:10 AM Glen wrote:

Argh.

On Mon, Dec 23, 2019 at 2:47 PM Glen wrote:

...

Maybe. I honestly don't know. All I know is, when I have a guest running on any of my hosts, even if that guest is idle (because, for example, it's a copy of a production machine) and therefore not getting internet traffic or usage, I can make the host crash by rsyncing a lot of data over a crossover cable at maximum speed. I

Typo. Ugh.

The host does not ever crash.

s/host crash/guest crash/ => I can make the GUEST crash by rsyncing a lot of data.

If it's easily reproducible, then the simplest way to troubleshoot is to use another distro as guest, or at least another kernel. You said that this guest runs docker, right? Then it should be easy enough to try install a new guest with another distro (e.g. ubuntu, centos, whatever), install docker on it, and try to replicate your problem. If the culprit is bridge/netfilter module, you could probably test with installing another kernel. Try kernel:HEAD? https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.ref... In any case, those two methods can help verify whether the problem is in the kernel (module), or something in leap userland (including its docker version). -- Fajar -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

I'm really surprised, I've never heard that any Guest running any virtualization on any kind of hypervisor would have any kind of problem running docker and having a networking problem. There should be two possible approaches to your problem based on a common idea... All network bridge devices are the same regardless of how they were created and if they were created by a virtualization tool or not. Once created, they should be discoverable by any entity that needs to use a bridge device. For this reason, although it's possible to cascade bridge devices, you should avoid if possible. So your likely choices, 1. If you create a bridge device in the Guest, it has to be bound to a network interface which communicates correctly with the HostOS network interface which in the case of Xen installed on openSUSE/SUSE likely itself has a bridge device bound to the network interface. Yes, that starts getting complicated, so... 2. You can go "old school" and implement docker networking the old way without a bridge device. Here is one old piece of documentation that describes how things used to be done and should still work https://github.com/putztzu/docker/blob/master/docs/installation/linux/SUSE.m... Otherwise, if you want to configure the first scenario above , you'll have to post your bridge device configurations for both in your Guest and your HostOS describing the actual physical interfaces they're bound to and what type of bridge device each are (are all bridging, are any NAT? Depending on the type of bridge may also need further details). Personally though, I like to live a life as uncomplicated as possible, I even avoid virtual switches as much as possible unless there is a real need that can't be accomplished by using bridge devices intelligently. HTH, TSU On Sun, Jan 5, 2020 at 8:41 AM Glen wrote:

On Mon, Dec 23, 2019 at 1:26 AM Jan Beulich wrote:

...

On 21.12.2019 07:31, Glen wrote:

...

So just for laughs, I ran an lsmod in both modes, and sorted and diffed them: br_netfilter bridge One of these two is, according to my experience, a fair candidate for your problems. I'm not a networking specialist at all, so I can't give any suggestions on how to convert the upgraded guest to a network config not requiring these modules. (Trying to get rid of br_netfilter alone may be easier, but again I'm not really knowledgeable in this area at all.)

All -

During the past two weeks I've been trying to hunt these down, and I have discovered why these modules are loading.

The Xen guest is running Docker.... and Docker needs those modules to provide access to the outside world for docker container networking.

Those two modules don't exist in /etc/modprobe.d, and I can rmmod them without damaging the Xen guest at all - but when I do, the Docker containers lose their network connectivity. None of my other guests run Docker, which is why those modules don't exist in the other guests, and why, I'm now guessing, the other guests don't have crash issues.

*sigh*

So I may be in a chicken-and-egg scenario here, but I'll just put this out there - has anyone tried running Docker on a Xen guest, and/or could there be anything in those two modules Jan mentioned, or in Docker, that could be causing or related to these issues?

I'd be grateful for any thoughts anyone might have!

Glen -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Hi everyone - So my two threads really turn out to be one thread, and I'm replying to both here. I apologize for the mess. First, I really appreciate all the responses and pointers, I'm very grateful for all the help. THANK YOU to all those who responded on either thread! To recap, essentially what I have is: 1. I've always run a bunch of high-traffic 42.3 Xen PV guests on 42.3 hosts. 2. I upgraded the 42.3 hosts to 15.1, via online upgrade and then via fresh load. 3. When I did that, *one* of my 42.3 guests started hanging at random every 2-7 days. The hangs seemed to be related to high network and/or disk traffic. I discovered by accident that if I did an "xl trigger nmi" I could "unhang" the guest and make it resume duty, more or less, without a reboot, but I have no idea why the hang occurs or why it's recoverable in that way. Chasing this down has been painful. It initially looked like sshd was the culprit, but it wasn't. I thought the kernel mismatch might be the issue, but other 42.3 guests run on their 15.1 hosts without a problem, and upgrading the guest to 15.1 didn't solve it. Olaf has been pointing me to new kernels, and that helped somewhat - moving to the SLE kernel extended the guest uptime from a few days to a few weeks (buying me much needed sleep, thank you!) but I still don't have a solution. The problem seems to be in this particular guest... somewhere. The problem seems to travel with the guest: If I clone the guest, and bring it up elsewhere that clone also has the problem. So I've resorted to making a copy of the guest and staging it on a different host just so I can stress-test it. To stress-test it, I basically initiate lots of high-traffic requests against the troubled guest from an outside source. Initially, the guest was hanging during a single full outbound rsync. To prevent SSD wear I modified the command and used a hack to simulate the traffic. If I boot the guest, and, from a different connected machine, do stuff like: nohup ssh 192.168.1.11 tar cf - --one-file-system /a | cat > /dev/null & nohup ssh 192.168.1.11 cat /dev/zero | cat > /dev/null & (where 1.11 is the troubled guest, and /a is a 4TB filesystem full of data) I can make the troubled guest hang in somewhere between 45 minutes and 12 hours. Thanks to Olaf, Jan, Tony and Fajar, I've been able to try a number of things, but so far, I've had no luck: Uprading openssh to the latest version did not solve it. Upgrading the guest to 15.1 (unifying the kernels) did not solve it. Upgrading the 42.3 guest kernel to a different version helped... but did not solve it. Removing some possibly problematic kernel modules did not solve it. Removing Docker did not solve it. I had optimizations from the Xen best practices page in /etc/sysctl.conf for just this guest - removing those did not solve it. The only solution I've found seems to be starting the guest over fresh. If I do a fresh load of 15.1 as a guest and mount that same /a filesystem, and do those same tests... the freshly-loaded guest works fine.... it's rock solid. I had those same tests running against the freshly-loaded guest for over 24 hours and it did just fine. I can literally just swap out root filesystem images - booting the troubled guest's root filesystem results in the hangs - booting the fresh-load seems completely reliable. In short it seems to me now that there's something in this particular guest's root filesystem image... something I can't find... that is causing this. The image started as a 13.1 (thirteen point one) fresh load (years ago) and has been in-place upgraded ever since.... so I am concluding that something bad has been brought forward that I'm not aware of. It seems at this point that I just need to rebuild the guest as a fresh 15.1 load, reinstalling only what I currently need, and going from there, and so that's what I'm going to do. The usual absence of useful log data when the machine crashes is frustrating, and the time it takes (1-12 hours) to make a test machine crash makes the test process slow, so I'm feeling like I should just abandon this and replace the guest. If any of this triggers anything for anyone, please let me know. Otherwise, I'm continuing to stress test my freshly-loaded guest for a few more days (just to be sure) and then I'll start the reconnect and replacement process. It really would have been nice to find out what on that troubled guest was causing the issue, but it's probably some legacy thing brought forward that's causing instability, and since each test cycle takes so long, the "process of elimination" could take months or more. And of course as soon as I send this, something new will break, making this all invalid. :-) Anyway THANK YOU ALL for your support and help here, I am very grateful! Glen -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Hi Tony - Thanks for the reply! On Wed, Jan 8, 2020 at 9:24 AM Tony Su wrote:

Was thinking along the same lines as the last resort... If you can pin down the problem as a HostOS problem, Shouldn't your Xen Guests(with docker containers within) be easily migratable to a newly built HostOS? Don't know what else is running on your system, but typically it's a recommended best practice to keep the HostOS in a multi-apartment system as simple and uncomplex as possible.

Yes but I must have explained it backwards. At this point, I think it's a problem inside the guest, not the host. I have 9 hosts, and about 16 guests, and the problem follows this particular guest. All hosts are solid. All the other guests are solid. It's only this one guest (or copies of it) that fails, no matter which host I run it on. So I am going to fresh-load the guest, and just reconnect services, and yes, it won't be that complex, it just "a pain" and I was hoping to avoid it.

And a general observation, If this machine was originally installed as a 13.1 and survived every upgrade for each openSUSE version from then to now, IMO that's quite an accomplishment.

This made me smile. At this point, all of my hosts are freshly-loaded 15.1. But this guest - yes, it's last fresh load was back on 13.1, and we've just "zypper dup"'ped it at each new release, all the way up until now. But it's clearly time to bite the bullet and reload the guest fresh. THANK YOU again for your support and responses as I've tried to figure this out! Glen -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org