Well, it's stranger than I thought. I removed gtk-font-name from .gtkrc-2.0 and .config/gtk-3.0/settings.ini and libreoffice becomes responsive again.