Polarization-controlled orbital angular momentum switching in nonlinear wave mixing