Port scans, what do to?!

Hi Gurus ;P I see it in a pretty relaxing way when i get log entries from a ip 0.0.0.0 or a scan from someone. I never tried to scan back or something, whats the point?! ( Portsentry is actually banning about one ip a every 2nd day) Do u usually just irgnore them? Write a mail to your ISP ? How do i know if a boy was pressing a few buttons or if someone seriusly tried to gain access? Where is the line between script kiddy and attacker? I would like to hear your opinion about this! Spiekey