Comment # 4 on bug 1171257 from Jochen Breuer

(In reply to Alberto Planas Dominguez from comment #3)
> Some errors that I see on the master:
> 
> [DEBUG   ] LazyLoaded local_cache.prep_jid
> [DEBUG   ] Sending event: tag = 20200506112531080684; data = {'minions':
> ['10.0.2.15'], '_stamp': '2020-05-06T11:25:31.094373'}
> [DEBUG   ] Sending event: tag = salt/job/20200506112531080684/new; data =
> {'jid': '20200506112531080684', 'tgt_type': 'glob', 'tgt': '*', 'user':
> 'root', 'fun': 'test.ping', 'arg': [], 'minions': ['10.0.2.15'], 'missing':
> [], '_stamp': '2020-05-06T11:25:31.097904'}
> [DEBUG   ] Adding minions for job 20200506112531080684: ['10.0.2.15']
> [INFO    ] User root Published command test.ping with jid
> 20200506112531080684
> [DEBUG   ] Published command details {'fun': 'test.ping', 'arg': [], 'tgt':
> '*', 'jid': '20200506112531080684', 'ret': '', 'tgt_type': 'glob', 'user':
> 'root'}
> [DEBUG   ] Signing data packet
> [DEBUG   ] salt.crypt.get_rsa_key: Loading private key
> [DEBUG   ] salt.crypt.sign_message: Signing message.
> [DEBUG   ] Sending payload to publish daemon. jid=20200506112531080684
> size=436
> [DEBUG   ] Connecting to pub server:
> ipc:///var/run/salt/master/publish_pull.ipc
> [DEBUG   ] Sent payload to publish daemon.
> [DEBUG   ] Publish daemon received payload. size=436
> Process salt.transport.zeromq.<class 'method'>._publish_daemon:
> Traceback (most recent call last):
>   File "/usr/lib64/python3.8/multiprocessing/process.py", line 315, in
> _bootstrap
>     self.run()
>   File "/usr/lib64/python3.8/multiprocessing/process.py", line 108, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/lib/python3.8/site-packages/salt/transport/zeromq.py", line
> 920, in _publish_daemon
>     unpacked_package = salt.payload.unpackage(package)
>   File "/usr/lib/python3.8/site-packages/salt/payload.py", line 47, in
> unpackage
>     return salt.utils.msgpack.loads(package_, use_list=True)
>   File "/usr/lib/python3.8/site-packages/salt/utils/msgpack.py", line 128,
> in unpackb
>     return msgpack.unpackb(packed, **_sanitize_msgpack_kwargs(kwargs))
>   File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0:
> invalid start byte
> [INFO    ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
> 
> 
> A bit later I can also see this:
> 
> [DEBUG   ] Adding minions for job 20200506112536159600: ['10.0.2.15']
> [INFO    ] User root Published command saltutil.find_job with jid
> 20200506112536159600
> [DEBUG   ] Published command details {'fun': 'saltutil.find_job', 'arg':
> ['20200506112531080684'], 'tgt': ['10.0.2.15'], 'jid':
> '20200506112536159600', 'ret': '', 'tgt_type': 'list', 'user': 'root'}
> [DEBUG   ] Signing data packet
> [DEBUG   ] salt.crypt.get_rsa_key: Loading private key
> [DEBUG   ] salt.crypt.sign_message: Signing message.
> [DEBUG   ] Sending payload to publish daemon. jid=20200506112536159600
> size=489
> [DEBUG   ] Sent payload to publish daemon.
> [INFO    ] Process <bound method ZeroMQPubServerChannel._publish_daemon of
> <salt.transport.zeromq.ZeroMQPubServerChannel object at 0x7f791113f550>>
> (4011) died with exit status 1, restarting...
> [DEBUG   ] Started 'salt.transport.zeromq.<class 'method'>._publish_daemon'
> with pid 4786
> [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at
> 0x7f790fce4ac0> exception was never retrieved: Traceback (most recent call
> last):
>   File "/usr/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309,
> in wrapper
>     yielded = next(result)
>   File "/usr/lib/python3.8/site-packages/salt/utils/process.py", line 531,
> in run
>     self.check_children()
>   File "/usr/lib/python3.8/site-packages/salt/utils/process.py", line 556,
> in check_children
>     for pid, mapping in six.iteritems(self._process_map):
> RuntimeError: dictionary keys changed during iteration
> [INFO    ] Starting the Salt Publisher on tcp://0.0.0.0:4505
> [INFO    ] Starting the Salt Puller on
> ipc:///var/run/salt/master/publish_pull.ipc
> [DEBUG   ] Publish daemon getting data from puller
> ipc:///var/run/salt/master/publish_pull.ipc
> [DEBUG   ] Publish daemon received payload. size=489
> Process salt.transport.zeromq.<class 'method'>._publish_daemon:
> Traceback (most recent call last):
>   File "/usr/lib64/python3.8/multiprocessing/process.py", line 315, in
> _bootstrap
>     self.run()
>   File "/usr/lib64/python3.8/multiprocessing/process.py", line 108, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/lib/python3.8/site-packages/salt/transport/zeromq.py", line
> 920, in _publish_daemon
>     unpacked_package = salt.payload.unpackage(package)
>   File "/usr/lib/python3.8/site-packages/salt/payload.py", line 47, in
> unpackage
>     return salt.utils.msgpack.loads(package_, use_list=True)
>   File "/usr/lib/python3.8/site-packages/salt/utils/msgpack.py", line 128,
> in unpackb
>     return msgpack.unpackb(packed, **_sanitize_msgpack_kwargs(kwargs))
>   File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0:
> invalid start byte
> [DEBUG   ] Performing fileserver updates for items with an update interval
> of 60
> [DEBUG   ] Updating roots fileserver cache

Weird! I just did a fresh install and I don't get a UnicodeDecodeError. But I
can confirm that it's not working and that there seems to be the return
missing. I've tested the same package (3000.2) for Leap 15.1 (Python 3.6) and
this is working. So I guess we are running into a Python 3.8 problem. I'll do
some cherry-picks that upstream did for Python 3.8 compatiblity and then I'll
re-run my tests.