I'd be afraid to try and drill out post holes as well and definitely would need a press and proper vise. Doing it by hand would be asking for serious issues and potential injury. Even cordless drills have some wicked torque when they want to, getting the bit perfectly centered with no wobble from the drill shaft (or bit) would be tricky.
Hitting the hole even a little off center may cause the bit to slip or try to bend it as it tries to follow the hole (if not fully centered) causing the bit to grab/break. Same thing with doing it by hand, the bit gets cocked at all it's gonna jam and grab and flip shit all over the place if the bit doesn't snap off first. That doesn't even cover the potential damage to the posts.
Unfortunately I've had to learn the hard way trying to widen a hole out and either not having a proper press on hand or being lazy and thinking 'aw I got this, it'll take me 2min'. Nope. lol. You'd think I would have learned the first time, apparently I have a short memory. More times than not it bites me in the ass.