First, it will take lots of practice to get it solid, but knowing what to pay attention to when you practice makes it a little better. To keep the yoyo from hitting your finger, you will need to pay attention to hitting the string above the yoyo a bit higher up the string.
Once you’re hitting high enough, to keep the yoyo form sliding to the side, you need to make sure your finger is perfectly perpendicular to the string. If your finger tilts up or down, the yoyo will flip either left or right.
The parallax from your viewpoint may make it look like your finger is right, but if the yoyo misses, it isn’t. Play with the angle of your finger until it is right, then work on getting that feel down so you can do it without really even looking.