First, don't keep dangerous things near your baby. When he puts something in his mouth, tell him "No!" and take it away from him. It is more important to take it away from him than to scold him.
Also, you and your partner should talk with each other to coordinate on how to train your baby.