RTMPE support can be provided by libgcrypt and gmp there is no need for GnuTLS there. OpenSSL cannot be enabled because of licensing incompatibilities. Can you please provide an rtmpe stream that fails with ffmpeg so I can take a look?