This might be because I have KASLR enabled, maybe passing nokaslr on the kernel command line will fix it...