So I've got it fixed, here is what I did:
Sprayed everything that moves down with WD-40.
Replaced the brake drums.
Flushed the old fluid out (I Really think this was the problem, because after I got all the air and old fluid out, the actuator felt much firmer than when I started).
Everything is good to go now!