You're right, it is very much the same if you look at it that way. Like I said, it both isn't nice.
You might even say that women obsessed with an actor are worse, because they don't usually just want to have sex with him, but they fantasize about being in a relationship with him and being loved by him, (well, most women do) which is certainly worse than just wanting to fuck somebody else to most people. Not to me though, I'm strange like that.
I just think that is has a lot more of an humiliating air to the partner when a man gets sexual satisfaction from a porn girl than a girl fantasizing about marrying Brad Pitt. That isn't something she does, and therefore it is much more innocent to me. After all, thoughts are never quite real unless you speak them out loud, which is when they start getting shapes, etc.
Plus, most women obsessed with Hugh Grant, Ben Affleck or Justin Timberlake are not older than 16 and mostly immature, while a man finding joy in porn is ageless.
But like I said, it both isn't quite right in a sexually active relationship.