I have just rechecked. I am not having a problem. I tested with both GDM and with SDDM. Logout from plasma-wayland and into plasma X -- no problem. However, gpg-agent is not running here. My understanding is that gpg-agent is not automatically started. Rather, it is started when first needed. So maybe you are doing something to start it.