Hi Andy, On 15.09.2017 14:14, Andy Wojnarek wrote:

I did not run the database migration, I was running through the OpenSuSE Implementation guide for Ocata – since there isn’t one for Pike yet:

https://docs.openstack.org/ocata/install-guide-obs/

Not sure, but I think the new guide is at https://docs.openstack.org/install-guide/ .

There isn’t anything there for a neutron db sync during the controller portion of the guide. How does one get involved with OpenSuSE OpenStack documentation? I’d like to contribute.

At the end of every openstack doc, there is a bug report symbol to report issues. I guess a good start for new documentation contributors is https://docs.openstack.org/contributor-guide/ Best, Tom

Thanks for the link, I’ll run the DB migration/sync.

Thanks, Andrew Wojnarek | Sr. Systems Engineer | ATS Group, LLC mobile 717.856.6901 | andy.wojnarek@TheATSGroup.com Galileo Performance Explorer Blog http://galileosuite.com/blog/ Offers Deep Insights for Server/Storage Systems

On 9/15/17, 5:20 AM, "Thomas Bechtold" wrote:

Hi Andy,

On 15.09.2017 04:25, Andy Wojnarek wrote: > I’m trying to submit a bug, but on https://bugzilla.opensuse.org/, but there does not seem to be a component for OpenStack under any of the products.

Indeed. I requested a new component and it should be available in the next 24 hours.

> I’m testing a fresh install of Pike on 42.3, and I’m getting neutron database errors: > > 2017-09-14 22:17:47.431 24052 ERROR oslo_messaging.rpc.server DBError: (pymysql.err.InternalError) (1054, u"Unknown column 'subnets.project_id' in 'field list'") [SQL: u'SELECT subnets.project_id AS subnets_project_id, subnets.id AS subnets_id, subnets.name AS subnets_name, subnets.network_id AS subnets_network_id, subnets.segment_id AS subnets_segment_id, subnets.subnetpool_id AS subnets_subnetpool_id, subnets.ip_version AS subnets_ip_version, subnets.cidr AS subnets_cidr, subnets.gateway_ip AS subnets_gateway_ip, subnets.enable_dhcp AS subnets_enable_dhcp, subnets.ipv6_ra_mode AS subnets_ipv6_ra_mode, subnets.ipv6_address_mode AS subnets_ipv6_address_mode, subnets.standard_attr_id AS subnets_standard_attr_id, standardattributes_1.id AS standardattributes_1_id, standardattributes_1.resource_type AS standardattributes_1_resource_type, standardattributes_1.description AS standardattributes_1_description, standardattributes_1.revision_number AS standardattributes_1_revision_number, standardattributes_1.created_at AS standardattributes_1_created_at, standardattributes_1.updated_at AS standardattributes_1_updated_at, subnetpools_1.project_id AS subnetpools_1_project_id, subnetpools_1.id AS subnetpools_1_id, subnetpools_1.name AS subnetpools_1_name, subnetpools_1.ip_version AS subnetpools_1_ip_version, subnetpools_1.default_prefixlen AS subnetpools_1_default_prefixlen, subnetpools_1.min_prefixlen AS subnetpools_1_min_prefixlen, subnetpools_1.max_prefixlen AS subnetpools_1_max_prefixlen, subnetpools_1.shared AS subnetpools_1_shared, subnetpools_1.is_default AS subnetpools_1_is_default, subnetpools_1.default_quota AS subnetpools_1_default_quota, subnetpools_1.hash AS subnetpools_1_hash, subnetpools_1.address_scope_id AS subnetpools_1_address_scope_id, subnetpools_1.standard_attr_id AS subnetpools_1_standard_attr_id, standardattributes_2.id AS standardattributes_2_id, standardattributes_2.resource_type AS standardattributes_2_resource_type, standardattributes_2.description AS standardattributes_2_description, standardattributes_2.revision_number AS standardattributes_2_revision_number, standardattributes_2.created_at AS standardattributes_2_created_at, standardattributes_2.updated_at AS standardattributes_2_updated_at \nFROM subnets LEFT OUTER JOIN subnetpools AS subnetpools_1 ON subnets.subnetpool_id = subnetpools_1.id LEFT OUTER JOIN standardattributes AS standardattributes_1 ON standardattributes_1.id = subnetpools_1.standard_attr_id LEFT OUTER JOIN standardattributes AS standardattributes_2 ON standardattributes_2.id = subnets.standard_attr_id'] > > I confirmed there is no column ‘project_id’ in the subnets table inside the neutron database.

Did you sync the neutron database[1]? We do the sync also during a neutron-server restart[2] via systemd but ignore failures here. Maybe you started neutron-server before you configured the database string in neutron.conf ?

[1] https://docs.openstack.org/mitaka/networking-guide/migration-database.html [2] https://build.opensuse.org/package/view_file/Cloud:OpenStack:Pike/openstack-...

Best,

Tom N�����r��y隊Z)z{.��%��~�{.n�+������Ǩ�?�r��i�m��0��ޙ�������?rZ.w�0�����Ǩ�

Update: I compared the Ocata cinder code to Newton and Mitaka, an that function causing all this traffic to the cluster is indeed new since Ocata. So at least it's explainable, but how can I configure it properly? Any ideas? Regards, Eugen Zitat von Eugen Block :

Hi list,

I already tried several channels to address this issue but noone answered yet.

Last week we upgraded our Mitaka cloud to Ocata (via Newton, of course), and also upgraded the cloud nodes from openSUSE Leap 42.1 to Leap 42.3. There were some issues as expected, but no showstoppers (luckily).

So the cloud is up and working again, but our monitoring shows a high CPU load for cinder-volume service on the control node, caused by endless connections to the ceph cluster:

---cut here--- 2017-10-17 11:04:48.211 10175 DEBUG cinder.manager [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:183 2017-10-17 11:04:48.212 10175 DEBUG oslo_messaging._drivers.amqpdriver [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] CAST unique_id: 83824da546134dd5b57eb14fe07c2e41 FANOUT topic 'cinder-scheduler' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:551 2017-10-17 11:04:48.214 10175 DEBUG oslo_messaging._drivers.amqpdriver [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] CAST unique_id: efe284dad8f44f66b1140c9cf8c1ecd0 exchange 'openstack' topic 'cinder-scheduler' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:562 2017-10-17 11:04:48.215 10175 DEBUG oslo_service.periodic_task [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2017-10-17 11:04:48.216 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:48.608 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:48.633 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:49.632 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:50.673 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:52.066 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:53.492 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:04:58.745 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:04.134 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:04.876 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:09.776 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:10.499 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:15.622 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:15.757 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:19.078 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:19.939 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.086 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.454 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.620 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:20.692 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:22.744 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:24.134 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:27.478 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:28.400 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:28.520 10175 DEBUG cinder.volume.drivers.rbd [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] connecting to ceph (timeout=-1). _connect_to_rados /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:300 2017-10-17 11:05:48.216 10175 DEBUG oslo_service.periodic_task [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215 2017-10-17 11:05:48.217 10175 DEBUG cinder.manager [req-61732ae6-4e69-42f2-b8ec-036bdb9fbaef - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:183 ---cut here---

I added some log statements to analyze the traffic, and obviously all existing volumes (currently 24 volumes) are examined all the time to "_get_usage_info". I tried to change the values for report_interval without any impact. Is there any way to reduce the cinder-volume traffic? Since we didn't really test this on Newton - we just did the necessary steps to be able to upgrade to Ocata - we can't really tell if this is a Newton or Ocata issue, but it was definitely NOT a Mitaka issue, we compared the network monitoring to the results before the upgrade, it's quite a big difference.

I'd appreciate any insight!

Regards, Eugen

-- Eugen Block voice : +49-40-559 51 75 NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77 Postfach 61 03 15 D-22423 Hamburg e-mail : eblock@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 Vorstand: Jens-U. Mozdzen USt-IdNr. DE 814 013 983

-- Eugen Block voice : +49-40-559 51 75 NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77 Postfach 61 03 15 D-22423 Hamburg e-mail : eblock@nde.ag Vorsitzende des Aufsichtsrates: Angelika Mozdzen Sitz und Registergericht: Hamburg, HRB 90934 Vorstand: Jens-U. Mozdzen USt-IdNr. DE 814 013 983 -- To unsubscribe, e-mail: opensuse-cloud+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-cloud+owner@opensuse.org