Prius Fob stopped unlocking doors but car starts

Basically I have an 06, where i have two key fobs that were working up until a week ago. I noticed that my main fob that i use everyday with the car stopped locking/unlocking the doors when i pressed the key on the remote. However, it had no problems starting the car or if i press the button on my door it'll lock the door perfectly fine. Not only that but to unlock the car i can unlock it by putting my hand on the handle.

I just got a hold of batteries, and replaced them. Nothing, it still won't unlock by pressing the "unlock/lock" button, however, i got a hold of my other spare key and it works fine! I don't know what else to do, since it works fine, and my main key doesn't. Any ideas?