Looks like I found exactly my problem with mouse: https://gitlab.freedesktop.org/xorg/driver/xf86-input-libinput/-/issues/38 and https://github.com/ckb-next/ckb-next/issues/747. Completely forgot that I'm using ckb-next, sorry about that. But original issue might be different.