The difference between a player making a mistake and a ref making a mistake is in some ways an unfair comparison.
A player can make up for his mistake in future matches (perhaps). The ref has no such opportunity and therefore his mistake can be more costly, (loose points, league position, promotion, relegation and possibly money)
As for fans making mistakes they have no bearing on the outcome of the game/season etc.
Of course the player has to put up with the often unhelpful comments of the clubs own supporters but the ref come the final whistle picks up his fee and sods off home with out a care in the world for the consequencies of his mistakes. That is if he believes it is possible for him to be wrong on occasion.
I would have to disagree with the last bit of your comment here. Referees and Assistants make mistakes, they are, like you and I, only human, recognise their mistakes and, in a good number of cases do have a care in the world for the consequences of the mistake.
Also, when you're talking about Blue Square Level Officials their time really is not their own. As a level 4 Official (Blue Square North Assistant) you are at the beck and call of the FA at any time. I know a guy personally who was called at work in Manchester City Centre at 2.45 on a Tuesday and told that he was on the line at Barrow that night.
A guy on the Premiership Line whom I officiated with at Crewe Academy a couple of years ago hadn't been home for just short of two weeks, only saw his wife when they exchanged his case of dirty clothes for the case of ones she'd washed at some Motorway Services and hadn't even spoken to his Children for a week
If, as happens at Conference (BSP) level the Officials performance is assessed then the consequences can be major for that official. I know of cases this year where Assistants have been taken off BSP matches to which they had originally been appointed due to a poor performance either on the line or in the middle of a match at lower level.
So, what I think I'm trying to say is that it isn't just a case of turning up, doing the match, picking up the money and being home for Dr Who, there's a lot more commitment than that.