Upgrades to Increase Crosswind Resistance

Is it that bad? what's the speed differential?

I would think any suspension improvements would help.

 

I have the same car you have. Depending on the $$$ available, I'd recommend, in this order:
- the underbody center brace by Rude person's (cures about 70% of the crosswind tug)
- the front underbody brace by Rude person's (tightens up the front of the car - for example, I don't feel the ICE starting since this brace got bolted on)
...the above mods are easiest and least costly, and have a huge impact on crosswind instability - but fellow PChat member Rude person's only makes them in batches when the planets align just right - so you might want to ping him via private message...
- much better cornering responsiveness: throw on a TRD rear anti-sway bar. Easy mod to bolt on but you'll spend some $$$.
- effortless and railroad-accurate steering at any speed : try the Cusco front Strut Tower Brace. More $$$, and the most complex piece to put on, because of all the disassembly/reassembly of the plastic stuff under the hood.

 

I think it has to do with energy efficient tires that we have, if you have stickier tires you should feel less crosswind jolt.

I have all the above mentioned mods to our Prius except the Rude person's mid/rear cross brace. I can still feel the car jerk whenever I get hit with a strong crosswind.

 

An upgraded center brace sure helped the Gen 2, I have not been in a Gen 3.
Cusco Center Power Brace for Toyota Prius ZVW30 [CUS-951 492 C] EDO Performance

TUB153F Tanabe 2 Point Sustec Front Under Brace for 10-up Toyota Prius at Andy's Auto Sport

 

Front sway bar upgrade! That the only parts that will push back when one side is lift up or dips down! Or lowering springs and even adjustable coil overs.