Ballast pump won't turn off!

So the automatic bilge pump in my boat won't turn off. Well thats not exactly true it well turn off when there's water above it!?! Its working backwards? If there's no water it runs constantly, when water makes the float rise it turns off? I'm not sure what happened as i haven't touched it or any of its wires. Theres a nob on the side of it you can turn and it'll shut off but as soon as you let go of it it turns right back on? Any ideas?

Sounds like it's wired backwards buddy, may have happened on your gas tank/ballast removal.