The easiest trick is to run "make localmodconfig" once. This will create a stripped kernel config that consists with only the currently loaded modules. It'll help reducing the build time dramatically.