Oh wow, that's a bit hard to say, since it's kinda been different through each playthrough. There's rarely ever been one single Pokemon that's really stood out in the rivals' teams for me except for the starters, but that's to be expected. I'm gonna make an attempt to listn them though. I'll be excluding the B/W(2) rivals though, since i haven't played either of those games yet.

Blue

Here's a rather easy one, because with all the times I've played the games, one Pokemon has actually stood out in his team. While I'd like to say his Pidgeot, just because it's super awesome, his Alakazam has always been there to seriously bug me. I always had a tough time against it in R/B, since I didn't know what weakness Psychic types had (not sure if they were actually weak to Ghost Pokemon back then or if they had no weaknesses), so I never had any Pokemon to take it out quickly. And then it was just super strong. Most of the time I lost at least one Pokemon to it, and even though I haven't played the games in a long time, I'm pretty sure that's not something that would've changed.

Silver

Crobat. It was a Pokemon I really admired in the first few playthroughs, just because I thought it seemed so cool. At the time I had no clue how to get it, and quite frankly didn't want to raise a Zubat just to get it, because out of the three Pokemon of that line, I only really liked Crobat. It wasn't actually that much of a challenge, as I usually had Pokemon to counter it with, however it did at times outmatch me with its speed. But the main reason I really liked it was because of its design and how cool it was compared to the rest of its line.

May/Brendan
That's harder to say. I never really thought the third generation games lived up to its predecessors when it came to the rivals, as neither May, Brendan or Wally really stood out that much, probably because they replaced each other, but also because I didn't really saw any rivalry between them. It was more between friends. The same thing can really be said about the rivals from B/W, except Cheren stood out more as a rival than any of the others mentioned in that group. But when it comes to my favourite of May/Brendan's Pokemon, I'd have to say Wailmer. It isn't my favourite because I like the Pokemon, but because it was the hardest to beat of all their Pokemon. My Pokemon always got confused by its Water Pulse, and I usually ended up losing a Pokemon in a battle against it. They didn't get the Pokemon till late on though, so there were only like 2 or 3 battles with it. But yeah, it was still one of the harder ones.

Wally

With Wally it's definitely Gardevoir. Out of all of Wally's Pokemon, Gardevoir was definitely the most well trained and hardest, even though some of the others had more potential. I'd say this was the only rival where I could really feel a bond with one of their Pokemon with, and it definitely helped him in battle. I played through a ROM hack of Emerald once, where the difficulty was raised, and out of all the trainers I got to face before I gave up, Wally was the most challenging, solely because of his Gardevoir. If I didn't plan my strategy right and if I missed with any of my moves, it could wipe out my team faster than a Pikachu using Quick Attack >_< But I managed to squeeze through my battles, all my money spent on potions which were then spent on the battle, and most of my Pokemon fainted. Yeah, he's definitely the one that has been the biggest challenge, however that was only because of the improved difficulty. Otherwise he's usually extremely easy.

Barry

Definitely Staraptor. It was always the biggest challenge when battling against Barry because of its type combination covering a lot of my Pokemon, and with the great moves it had as well as how strong it was, it was a pretty big threat in any battle in which it was involved. Usually I sent out my own Staraptor to pit against his, though usually mine was more well trained, so mine often won. But yeah, biggest reason of all is that it's a BIIIIIIIIIIIIIIIIIIIIIIIIIIRD.