No, it's not just you, it's considered a scary movie and I read on Cracked that the kid actors themselves were scared during the psychedelic boat ride.

The oompa loompas look frightening as well.

Impressive that you remember your nightmare that many years later.