When you reach the forth floor, you'll be greeted by some nasty goblins, but you're used to that by now. Hit the towers in this order:
2 1 3 4
(imagining that those numbers are the four corners of the floor you're on.)
Once you've cleaned out all the towers, go up the staircase that is offset compared to the others in the south east section of the floor.