Why more than 1 hole in FW for IPSec

Hi, if my VPN GW is behind the firewall, why is it - generally, in theory - neccessary to have the firewall open IP 50, 51 and UDP 500 (ISAKMP). Why - in theory - is hasn't everything been designed to use ONE single connection through one port and protocol? Why not e. g. use SSL/TLS with PKI and that's it? Higher and lower level traffic can be packed and tunneled through even a higher network layer, can't it? Greetings, René