Invisible X?

Howdy! I like to have as many as possible ports closed. Is it possible to configure X in a way that it's only "visible" for localhost/127.0.0.1 ? --Ragnar