Found an excellent guide (in Swedish) to remove PulseAudio. Basically do this:
Check that your system uses Pulseaudio (it says clearly by Card and Chip) in the top left corner.
sudo rm /etc/X11/Xsession.d/70pulseaudio
sudo apt-get remove pulseaudio
sudo apt-get install esound
And check that it is not using PulseAudio any longer.
I will report after thorough testing if it works fine.