I'm not convinced this is related to the gfx driver. Did you start sway with --unsupported-gpu Maybe sway tries some fallback, when not specified.