Charged shots shooting wrong way with controller

I find that whenever I play Azazel or have any other charged shot item when I release the charge it sometimes just shoots in some random directions other than where I was aiming. I use a controller and use the joystick to shoot. I press in the direction to charge and release to fire. I suspect that the elasticity in the joystick is causing it to snap back to center and a little further, causing it to shoot in the opposite direction on release. Is this a problem anyone else has had? Is it my controller? Is there any way to fix this or should I just start using the buttons instead?