Intermittent pulling when low speed braking over bumps

Pulling one way or the other can be caused by tires.
Swap the front wheels left to right and right to left and see if it still pulls
more, or pulls less.
Mine drifts slightly right. Whoever did the alignment didn't take in effect that
the road is not level, there is usually .5 degree slant to allow the water from rain
to drain.
The crappy tires I got last time made it worse. I swopped the front tires and it drove
straight.

You should check all the bushings for control arms. I would bet that they are worn.
The ones on my van where shot. I had to turn completely one way in a circle once so
that it would drive and brake straight....

 

This sounds to me like a combination of things.

There is a classic Prius normal behavior that many people notice (and are often startled by, the first time), when the car is being gently braked and then encounters some road imperfection. The normal feeling is that the car let up on the brakes for an instant, or slightly leapt ahead.

What happens is that the gentle braking is being handled by regen, using only the front wheels, and the road imperfection makes the car think "whoops! sketchy traction, I'll switch from regen to friction brakes, they can grip with four wheels, not just two". That's not the same as applying ABS, but it's a needed step so the car is prepared to do that if it later decides there's still sketchy traction and just switching to four-wheel friction brakes hasn't helped enough.

The car lets off the regen braking right away and it takes an instant for the friction brakes to reach the same braking rate, so that's why you normally feel a sort of mini-lunge ahead right while it's in that transition.

It does not normally feel like a pull to either side, just a mini-lunge straight ahead. But I am thinking if that normal behavior got combined with some issue where your friction brakes are pulling to the side, that could feel as you've described.