In my opinion, the races that struggle the most against Orcs are those with weak defenses and a focus on magic. Orcs have a high attack stat, dealing tremendous damage, and with the right build and a bit of luck, they attack first, and their main attackers (Cyclops, Wyverns, and level 5 units) reach the enemy immediately. Mages might only manage to cast three spells before losing.
I consider Demons to be the most dangerous opponents for Orcs, as they will strike first, and then the Orcs will be put to the test (and while Orcs have decent defense, it's definitely less than the attack power of Demons). Imps can completely overwhelm the Orcs with their spells, as Orcs don't have much knowledge. Humans are also dangerous for Orcs, as they can withstand attacks well, and their hero actively supports the troops with Light magic or weakens the Orcs with Dark magic. I can't say much about Dwarves, as I haven't played against them much.
However, in general, I think it all depends on the player. For example, you can easily defeat Mages with Orcs, or you can lose badly. And it also depends on luck, what artifacts you find, what your starting position is, and so on. I wouldn't say there's a truly noticeable difference...
P.S. In general, I consider Tower and Mages to be losing matchups, and Demons, Humans, and perhaps Necromancers with their Dark magic to be difficult matchups.
Bow down and hear our cry,
Our earthly rulers falter,
Our people drift and die,
The walls of gold entomb us,
The swords of scorn divide,
Take not thy thunder from us,
Take away our pride.
G.K. Chesterton: English Hymnal