Apart from Colchester, which was a sudden event and over very quickly, I remember being terrified for the whole game at Frickley in an FA Cup tie in the mid 80s.
I think it was around the time of the miners strike and the ground was full of very angry yorkshiremen who clearly weren't scared of the police and were itching for a fight. I think Phil Gardner scored for us right at the end of the game after we had been totally outplayed for the first 90 minutes, but without conceding.
That only made matters worse, on the way out I just buried my head in my coat and jumped on a bus that happened to be passing, without caring where it went. Fortunately it went to Barnsley and I was able to get a train home!!