I'm afraid i have to reinstall the machine, but before i do it want to know what and how happened.

That is the normal way. Reinstall your box after doing a forensic analysis on what is happen until now. Fix the problem and harden the box.

If someone of you experienced with this and could give good advices about what to do and how i can analyse who logged it would be appreciated.

As above - doing a forensic analysis. The're a lot of documents on it. Try e.g. google: forensic analysis honey pot.

Put your hard disc into another box and mount the filesystem(s) r/o too examine what logfiles tell you about some actions of the attacker - but normally they should be cleaned. (apache/mailserver logs too)

Do a string analysis on the whole disc and search for log-strings with a date entry related to date of attack in order to identify the way attacker became root.

TIA, istvan

Ciao

Michael