My theory was correct, but it isn't gnutls_deinit but gnutls_global_deinit. Xvnc calls it when cleaning after a failed connection and then any following connections fail in gnutls calls... It was fixed upstream in: https://github.com/TigerVNC/tigervnc/commit/8aa4bc53206c2430bbf0c8f4b642f59a379ee649