This is one of those discussions that I guess people keep coming back to. Here's my thoughts:
Series 1 - tough to say, since most of the teams made "series 1 mistakes", that is to say mistakes that occurred through natural unfamiliarity with the format. Of course, James' fatal sidestep in series 3 comes under this category but by that point in the show was just sheer stupidity!
Series 2 - yes, Jamie was treated unfairly, no question. Especially compared to the hints that Neil got (does anyone else think that some of that team were dyslexic, and that's why they got the hints?).
Series 3 - Leo's riddle was a stinker, and Simon's amulet didn't have a "tracker facility" - even a little shadow on the floor would have helped. Scott's jump was indeed a good one, and it always threw me that they suddenly died. Cliff, there's arguments for and against there. Ross was unlucky too, that cliff path was tough to judge.
Series 4 - Alistair was unlucky getting the CoB like that, they were really on the ball for the whole of their quest (they didn't get a single riddle wrong). The backwards riddles that Vicky (or was it Nicky?) faced were confusing, that was very unfair, and Jeremy was a little hard done by too. Giles - absolutely was treated unfairly. Those long-winded spyglass sequences were the killer. They should have done what they did in series 1 and edit things a bit.
Series 5 - Sarah was hard done by, they were a very capable team, and they did eventually give Elita that gem. Alex's "slowness" on the causeway was a debatable issue, which I guess is why they put a visible clock there next season. Christopher needed three objects from one clue room to successfully complete level 2, so he was very unfairly treated indeed.
Series 6 - Matt's causeway death has always rankled with me, because it doesn't actually look like he put a foot wrong on it. And yes, Sofia didn't even get a chance to fail on the Great Causeway. Btw - Ben on that same causeway, the advisors used logic to work out the sequence.
Series 7 - Ben and the scattered spell was very unfair, considering the ease of the clues by this point in the series, the need to use that darkness powder or whatever it was, was not at all obvious. Usually, even in the first series, it's obvious what needs to be done, the question is do they have the means?
Series 8 - Daniel's CoB experience was indeed unfair. Also, Michael is another one that can be argued both ways.