October 2018 - openSUSE Public Cloud - openSUSE Mailing Lists

[opensuse-cloud] Nova imports flat images from base file despite ceph backend
by Eugen Block 10 Oct '18

10 Oct '18

Hi list, this week I noticed something strange in our cloud (Ocata on Leap 42.3). We use Ceph as backend for nova, glance and cinder, everything really works like a charm. But from time to time we've noticed that some instances take much longer to launch than others. So I wanted to take a look what's happening, turned on debug logs and found that in some cases (I have no idea how to reproduce yet) there is an image downloaded to /var/lib/nova/instances/_base which then is used to import it back to Ceph, that is obviously the reason for the delay. The problem is that this new instance is not CoW, it's a flat rbd image. Here are some relevant logs (instance_id: 65567fc1-017f-45dc-b0ee-570c44146119, image_id: 0da1ba0f-c504-45ea-b138-16026aec022b) ---cut here--- [...] 2018-10-04 11:46:39.189 10293 DEBUG nova.compute.manager [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 65567fc1-017f-45dc-b0ee-570c44146119] Start spawning the instance on the hypervisor. _build_and_run_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:1929 [...] 2018-10-04 11:46:39.192 10293 INFO nova.virt.libvirt.driver [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 65567fc1-017f-45dc-b0ee-570c44146119] Creating image 2018-10-04 11:46:39.220 10293 DEBUG nova.virt.libvirt.storage.rbd_utils [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] rbd image 65567fc1-017f-45dc-b0ee-570c44146119_disk does not exist __init__ /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py:77 2018-10-04 11:46:39.241 10293 DEBUG nova.virt.libvirt.storage.rbd_utils [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] rbd image 65567fc1-017f-45dc-b0ee-570c44146119_disk does not exist __init__ /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py:77 2018-10-04 11:46:39.245 10293 DEBUG oslo_concurrency.lockutils [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Lock "3c60237fbf59101d3411c4f795d0a72b82752e0b" acquired by "nova.virt.libvirt.imagebackend.fetch_func_sync" :: waited 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 2018-10-04 11:46:39.246 10293 DEBUG oslo_concurrency.lockutils [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Lock "3c60237fbf59101d3411c4f795d0a72b82752e0b" released by "nova.virt.libvirt.imagebackend.fetch_func_sync" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 2018-10-04 11:46:39.266 10293 DEBUG nova.virt.libvirt.storage.rbd_utils [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] rbd image 65567fc1-017f-45dc-b0ee-570c44146119_disk does not exist __init__ /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py:77 2018-10-04 11:46:39.269 10293 DEBUG oslo_concurrency.processutils [req-85d728c3-5da1-4b37-add7-b956b4b2bb3d df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Running cmd (subprocess): rbd import --pool images /var/lib/nova/instances/_base/3c60237fbf59101d3411c4f795d0a72b82752e0b 65567fc1-017f-45dc-b0ee-570c44146119_disk --image-format=2 --id openstack --conf /etc/ceph/ceph.conf execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:355 [...] # no parent data control:~ # rbd info images/65567fc1-017f-45dc-b0ee-570c44146119_disk rbd image '65567fc1-017f-45dc-b0ee-570c44146119_disk': size 6144 MB in 1536 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.c8f88a6b8b4567 format: 2 features: layering, exclusive-lock, object-map flags: create_timestamp: Thu Oct 4 11:46:39 2018 ---cut here--- ################################################################### For comparison I moved the respective base file and tried to spawn a new instance from the same glance image: ---cut here--- [...] 2018-10-04 10:30:29.412 2336 DEBUG nova.compute.manager [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 91d0b930-97b0-4dd0-81b4-929599b7c997] Start spawning the instance on the hypervisor. _build_and_run_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:1929 [...] 2018-10-04 10:30:29.415 2336 INFO nova.virt.libvirt.driver [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] [instance: 91d0b930-97b0-4dd0-81b4-929599b7c997] Creating image 2018-10-04 10:30:29.435 2336 DEBUG nova.virt.libvirt.storage.rbd_utils [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] rbd image 91d0b930-97b0-4dd0-81b4-929599b7c997_disk does not exist __init__ /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py:77 2018-10-04 10:30:29.455 2336 DEBUG nova.virt.libvirt.storage.rbd_utils [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] rbd image 91d0b930-97b0-4dd0-81b4-929599b7c997_disk does not exist __init__ /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py:77 2018-10-04 10:30:29.492 2336 DEBUG oslo_concurrency.lockutils [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Lock "3c60237fbf59101d3411c4f795d0a72b82752e0b" acquired by "nova.virt.libvirt.imagebackend.fetch_func_sync" :: waited 0.035s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270 2018-10-04 10:30:29.527 2336 DEBUG nova.virt.libvirt.imagebackend [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Image locations are: [{u'url': u'rbd://655cb05a-435a-41ba-83d9-8549f7c36167/images/0da1ba0f-c504-45ea-b138-16026aec022b/snap', u'metadata': {}}, {'url': u'rbd://655cb05a-435a-41ba-83d9-8549f7c36167/images/0da1ba0f-c504-45ea-b138-16026aec022b/snap', 'metadata': {}}] clone /usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py:903 2018-10-04 10:30:29.681 2336 DEBUG nova.virt.libvirt.imagebackend [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] Selected location: {u'url': u'rbd://655cb05a-435a-41ba-83d9-8549f7c36167/images/0da1ba0f-c504-45ea-b138-16026aec022b/snap', u'metadata': {}} clone /usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py:912 2018-10-04 10:30:29.682 2336 DEBUG nova.virt.libvirt.storage.rbd_utils [req-942ba103-1932-4adf-b9b2-670e1a2fc126 df7b63e69da3b1ee2be3d79342e7992f3620beddbdac7768dcb738105e74301e 2e3c3f3822124a3fa9fd905164f519ae - - -] cloning images/0da1ba0f-c504-45ea-b138-16026aec022b@snap to None/91d0b930-97b0-4dd0-81b4-929599b7c997_disk clone /usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py:236 [...] # instance has parent data control:~ # rbd info images/91d0b930-97b0-4dd0-81b4-929599b7c997_disk rbd image '91d0b930-97b0-4dd0-81b4-929599b7c997_disk': size 8192 MB in 1024 objects order 23 (8192 kB objects) block_name_prefix: rbd_data.c8abcb6b8b4567 format: 2 features: layering, exclusive-lock, object-map flags: create_timestamp: Thu Oct 4 10:30:29 2018 parent: images/0da1ba0f-c504-45ea-b138-16026aec022b@snap overlap: 6144 MB ---cut here--- Why is there a local copy of the image in the first place? Obviously nova doesn't need that and creates clones successfully without them. The directory /var/lib/nova/instances/_base only contains 6 images. I would be thankful for any explanation for this behaviour. Regards, Eugen

3 5

[opensuse-cloud] Devstack support for Leap 15.0
by Antonio Ojea 09 Oct '18

09 Oct '18

Hi all, Just good news for the openSUSE cloud community, Devstack supports openSUSE Leap 15.0 since this patch [1] got merged today. Cheers, A.Ojea [1] https://review.openstack.org/#/c/576798/ -- To unsubscribe, e-mail: opensuse-cloud+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-cloud+owner(a)opensuse.org

1 0

[opensuse-cloud] Networking provided via meta-data does not work
by Ben Howard 02 Oct '18

02 Oct '18

Hello from DigtialOcean! I am the engineer at DigitalOcean that is responsible for making distributions work on our cloud. I have been attempting to work with community members to try and get OpenSUSE on DigitalOcean, but those efforts have failed. The root of the problem is that DigitalOcean (like a few others clouds) does not have DHCP. To get around this we have implemented both ConfigDrive and our DataSource that handles how to configure a network in Cloud-init. On other distributions, this method works just fine. However the OpenSUSE Cloud-init distribution module does NOT handle configuring networking when the network configuration comes from either the ConfigDrive or DigitalOcean Cloud-init datasources. To whit: CONSOLE: 2017-11-01 14:47:22,468 - __init__.py[WARNING]: apply_network_config is not currently implemented for distribution '<class 'cloudinit.distros.opensuse.Distro'>'. Attempting to use apply_network /var/log/cloud-init.log Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cloudinit/distros/__init__.py", line 346, in _bring_up_interface (_out, err) = util.subp(cmd) File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1849, in subp cmd=args) ProcessExecutionError: Unexpected error while running command. Command: ['ifup', 'eth0'] Exit code: 162 Reason: - Stdout: eth0 setup-in-progress Stderr: - (That was taken from a OpenSUSE LEAP 42.3 image booted on DigitalOcean). Anyhow, this is blocking bringing OpenSUSE to DigitalOcean. If someone would like to work with me to produce upstream images that work -- and can configure networking, I would love the help. I can give free instances (just ping me off list with your DigitalOcean account). Thanks, Ben -- To unsubscribe, e-mail: opensuse-cloud+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-cloud+owner(a)opensuse.org

3 7