Hi list, I already tried several channels to address this issue but noone answered yet. Last week we upgraded our Mitaka cloud to Ocata (via Newton, of course), and also upgraded the cloud nodes from openSUSE Leap 42.1 to Leap 42.3. There were some issues as expected, but no showstoppers (luckily). So the cloud is up and working again, but our monitoring shows a high CPU load for cinder-volume service on the control node, caused by endless connections to the ceph cluster: ---cut here--- 2017-10-17 11:04:48.211 10175 DEBUG cinder.manager [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:183 2017-10-17 11:04:48.212 10175 DEBUG oslo_messaging._drivers.amqpdriver [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] CAST unique_id: 83824da546134dd5b57eb14fe07c2e41 FANOUT topic 'cinder-scheduler' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:551 2017-10-17 11:04:48.214 10175 DEBUG oslo_messaging._drivers.amqpdriver [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] CAST unique_id: efe284dad8f44f66b1140c9cf8c1ecd0 exchange 'openstack' topic 'cinder-scheduler' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:562 2017-10-17 11:04:48.215 10175 DEBUG oslo_service.periodic_task [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2017-10-17 11:04:48.216 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:48.608 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:48.633 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:49.632 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:50.673 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:52.066 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:53.492 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:58.745 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:04.134 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:04.876 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:09.776 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:10.499 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:15.622 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:15.757 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:19.078 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:19.939 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.086 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.454 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.620 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.692 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:22.744 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:24.134 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:27.478 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:28.400 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:28.520 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:48.216 10175 DEBUG oslo_service.periodic_task [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2017-10-17 11:05:48.217 10175 DEBUG cinder.manager [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:183 ---cut here--- I added some log statements to analyze the traffic, and obviously all existing volumes (currently 24 volumes) are examined all the time to "_get_usage_info". I tried to change the values for report_interval without any impact. Is there any way to reduce the cinder-volume traffic? Since we didn't really test this on Newton - we just did the necessary steps to be able to upgrade to Ocata - we can't really tell if this is a Newton or Ocata issue, but it was definitely NOT a Mitaka issue, we compared the network monitoring to the results before the upgrade, it's quite a big difference. I'd appreciate any insight! Regards, Eugen -- Eugen Block voice : +49-40-559 51 75 NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77 Postfach 61 03 15 D-22423 Hamburg e-mail : eblock@nde.ag Vorsitzende des Aufsichtsrates: Angelika Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 Vorstand: Jens-U. Mozdzen USt-IdNr. DE 814 013 983 -- To unsubscribe, e-mail: opensuse-cloud+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-cloud+owner@opensuse.org