Are Video Game Reviews Broken?

[BreakABone]

I don't know if we've had a thread like this before, but this recent article by Jim Sterling really bought the attention to them once more.

http://www.gamefront.com/gears-of-wa...-divine-right/

The general idea is that the video game review scale, is a really small scale. While it tends to exist from 1-10, for the most part, it is usually between 7-10 (70-100) and he points out the flack he got for giving Gears 3 an 8, even though that is a really respectable score.

Thoughts on this? Any reviews/reviewers you tend to stick with? Reason? Any sites you avoid?

[magus113]

I generally avoid IGN. If I take any opinions from reviews at all it's usually from GameTrailers, and Joystiq, but sometimes I just like to try it out for myself, because it's all really in how I take it..not how someone else does.

[gekko]

I am going to completely ignore the question you asked and say this:

What is it with Twitter that causes people to forget all about professionalism?

[Vampyr]

I just read Giant Bomb reviews. It's on a 5 star basis which makes a lot of sense. They rate games all over that scale.

[Typhoid]

I've always figured the reason every game (from a major developer) gets a 6-7 (out of 10) if it's a bad game is because if reviewers shit all over a game - even if it's bad - that company will be less likely to send that review company more free games to review. It's a catch 22. If you honestly call a game shitty, and you're a credible reviewer, that company probably won't eagerly hand over their next game to you. Obviously some probably would. But it's a fine line on both sides.


I've never ever ever ever paid attention to game ratings, though. They do nothing to tell you about the actual game considering it's entirely subjective to the handful of people reviewing it - none of which are you. And in the game playing and enjoying experience, what matters to you is the only thing that's important.

I've had countless fun hours playing shittily reviewed games - because games tend to not just be rated out of 5 or 10. They're rated out of 5 or 10 on a grading curve that encompasses all games that are like it.

An average game (of ____ genre) would get an (let's say) 8. Unless a game that was similar was released recently. Then you can't just rate the new game out of 10. You're really rating it out of whatever the most similar game got. "Well it'd get an 8, but _____ got an 8.5, and this is nowhere near that, so we'll HAVE to give it a 6.5 or 7."
Which in the grand scheme of things means absolutely nothing.

Now, clearly there are comparable things for certain game types. Racing games and FPS games are pretty easily reviewable on top of each other. You can definitely compare Bad Company2 to CoD. But to compare an RPG for the Wii to a (different) RPG for the PC/PS3/360 doesn't make much sense, but that's sort of what they have to do.

I view it like being in different leagues. The New York Yankess, good team. Great team. Let's give them a 9 out of 10. Not perfect, but damn good. Now let's take a minor league baseball team. Let's go with the Hickville Poopshooters. Now the Hickville Poopshooters could possibly be the best minor league team. So they'd get an 8 or 9 out of 10. However, in the grand scheme of baseball they would hardly even show up on the scale considering the likes of the Yankees. You can't give the Poopshooters an 8 if you gave the Yankees a 9, despite the fact they're in different leagues and will never play against eachother. So now you give the Poopshooters a 3 out of 10, because while they're good, they're not Yankee good. But this would probably upset the people of Hickville who are sitting there thinking "Hey, we're not even in the same LEAGUE as the Yankees, so why should we be compared to them? We're good in our own right." To which reviewers would say "Baseball is baseball".

Everything I said was probably in the OP's link, but that just shows I read nothing.


Edit: I don't mean to make it seem like I think games don't get shitty reviews. they do all the time. But reviewers seem to be afraid of giving a truthful review to a bad game if that game is from a large developer - unless the game is offensively bad. I just typically find that the reviews by the general populace after the game is released is by far better, and more reliable than a handful of companies who get sent the games for free saying how good or bad it is.

[Ginkasa]

Reviews don't mean much to me. If I want a game I'll generally get it regardless. I read reviews more to find out more about what's in a game than how good the game is.

The only time I'm swayed either for or against buying a game is if the general consensus (press and public) says its good/bad. I don't hinge it on what or a few particular websites.

[Angrist]

Hm reviewing Wii games is difficult. Graphics for example: would a 10 be for the best possible on the Wii, or the best possible on whatever console? THen Wii games would never score higher than a 7.
So I know a lot of gaming sites rate stuff based on what's realistic on that platform. And I think that's fair.

Also speaking of graphics... what exactly are you rating? The technical aspects? That way Wii Sports should probably get a 2 for graphics. But when you look at how good something is on the eye... Wii Sports can suddenly get an 8 for graphics. It just looks fun.

Anyway... do I care about review ratings... Sometimes. There are some games which I just know I'm going to like (Zelda, Starcraft/Warcraft, Mario platformers...). But when I want to try something new, I'll read reviews and I'll also pay attention to the ratings. Only when I think the bad rating was giving for the wrong reasons will I try a game.

[KillerGremlin]

Yup.

I don't know what it is...I'm not a statistician. It's broken math and a broken game industry where reviewers are tied too closely (AKA in bed) with developers. Before I even proceed I just want to point out that this mother fucker only spent 15 hours playing the game before dishing out an 8/10. What the fucking hell, right? Until you have beat the game, decided on its impact to the industry, determined its replay value, etc., it is PETTY to even try to give a game a score. This is an industry problem. Games are not movies. Hell, music is not movies. Movies are the only medium that really benefit from rapid digestion, and doesn't necessarily warrant repeat viewings to rate. Furthermore, a movie's true brilliance is often written about in hindsight many years later. Many movies that received rave reviews would go on to become serious genre-benders or would leave a big imprint in Hollywood. Some movies that received poor reviews would go on to become genre-benders and leave a big imprint in Hollywood.

The point is movies can be rated using rapid reaction and short digestion periods. Games cannot.

The whole concept of rushing out a game review is only beneficial to the developer. If you can't see that you are a moron and deserve to part with your money. There's a reason IGN cranks out a score the day of the game's release: SO PEOPLE WILL BUY IT. Don't even try to pretend that the reviewers and the game developers aren't jacking each other off.

If you want an accurate, normal-ish distribution of game reviews, go read Amazon.com. These are user reviewers that typically represent a game accurately. The downside to this is that you often need to wait several months after the game is released to get a really accurate sense of what to expect.

The problem, I think, is that we use percentages from 0 to 100% or scores from 1 to 10. If you look at the US school system, we give scores like:
A 90-100
B 80-89
C 70-79
D 65-69
F 0-64

Look where the largest range of points is located: 0-64 is failing. Whereas 80-100 are A and B scores, or really really good.

I'm not really sure of the methodology behind the US grading system, I assume it tries to create some sort of normal distribution where most people get Cs, and some people get Bs and Ds, and very few people get As and Fs. And in order to meet this distribution the points are distributed as such where a 70-79 is average.

So here is this system that everyone in this country is raised on where a 70% or 80% is just average or okay, so anything that is good or really good should be a 90% or 100%. So we run into the obvious dilemma:

There's a lot of good games, they should all get 80%, 90% or 100%

or

We need to change the industry-reviewer standards so that a 60% or 70% is the new good, and 80%, 90% and 100% are reserved for those top-notch titles like OoT or Half-Life 2

or

Do away with this ridiculous 1-10 or 0-100% system and roll with a new system. I think doing a 1-5 scale would be good.

1 = Really shitty game. Not just shitty gameplay, but glitches and bugs.

2 = Really underwhelming. Game has serious flaws.

3 = Average. Does everything the genre is supposed to, but doesn't go above and beyond. May not have a ton of replay value, isn't changing the industry.

4 = This is a game is an impressive example of the genre. Has polished gameplay, might replay again. You would be doing okay buying this game.

5 = Top of the class. This is THAT ONE GAME that you will be talking about 10 years from now. This is a genre-changer, so innovative you have to replay it just to see stuff you missed.

With this system, the reviewer in BaB's article would probably give the game a 3 or a 4. And then the reviewer would have to write up a review that supports that decision. And people would have to actually read the review. But this isn't beneficial to the industry or the game developers. Also, in an ideal world, the review for the game wouldn't come out until a few months later to see whether or not the replay value, multiplayer, and online gameplay held up. It's a lot more work to write and read a review that uses a scale of 1-5 vs. a review that just shits out a percentage value that everyone in America is comfy with.

You could also use letter scores
A = Genre changer (OoT, Half-Life, GTA3, etc.)
B = Genre over-achiever (Vice City, Halo 2, etc.)
C = Genre average
D = Genre Sub-par
F = Abortion

It's the same, ordinal, 1-5 scale. The only problem is people immediately associate a percentage score with a letter grade.

I haven't read a single IGN/Gamespot/Big Media review since the Gamecube Generation. IGN can go fuck itself. The reviews are absolute bullshit and are completely contrived. I prefer Amazon, Yahtzee, or editorials by Joe-averages like me. In fact, I figured Youtube would do away with IGN by now. You can go check out video reviews by normal gamers on video sites, you can read opinions about games on social news sites like Reddit, and you have massive shopping sites with reviews like Amazon.com and *cringe* even Walmart or Best Buy.

And when someone eventually writes an empirical, educational book on video gaming (and they will because it has happened with Cinema and Music), I promise you they will point out how sites like IGN contributed to the orgy of sequels and rehash titles that plague the industry. It's just a huge circle jerk, and the two people who win are the game developers, and the nerdy game reviewers who get to play the title before everyone else.

Isn't that the incentive to review video games, to be first? Everyone else gets shafted, unless you're wise to the giant flaws in IGN's rating system.

[Professor S]

Broken? Not any more broken than anyone's opinion can be.

All reviews can be reduced to the opinion of one person, regardless than what "scientific" method is used. It's always best to read the article and evaluate the reviewers taste in games and then judge whether or not you felt the praise or criticism to be based on fair comparisons/judgements.

If I need a quick and dirty review, I use metacritic. If I want to see a number I want to know how the majority of reviewers rated it, not just one.

[KillerGremlin]

Originally Posted by Professor S

Broken? Not any more broken than anyone's opinion can be.

All reviews can be reduced to the opinion of one person, regardless than what "scientific" method is used. It's always best to read the article and evaluate the reviewers taste in games and then judge whether or not you felt the praise or criticism to be based on fair comparisons/judgements.

If I need a quick and dirty review, I use metacritic. If I want to see a number I want to know how the majority of reviewers rated it, not just one.

Here's the thing: it would seem that one person's opinion is swayed by perception. It sounds like from the article that BaB linked to that there is a lot of hazing within the network of early adopters from other reviewers, and more disturbingly, from developers themselves.

It seems like developers have created a privilege system where they give reviewers games in advance, with one small caveat: the review needs to be positive.

It's gray area when reviewers write a review out of fear, or when the looming shadow of the developer is standing over them. The solution would be simple: IGN could simply turn down the early review copies of the game. But IGN doesn't need to do that. All they need to do is create a review scale where they don't get penalized for rating a game accordingly.

The point of finding a scientific or streamlined review system isn't to challenge the fact that a game review comes down to the opinion of one person. We all know reviews are 100% subjective, I think Typhoid hit that point well enough. The point of revamping the way games are scored might help reduce confusion and simplify things. And it would remain ambiguous too.

I dunno, that's just my feeling on the issue.

I enjoy meta-websites. Rottentomates is awesome, but you need to know how to use it. If a movie gets a 75%, it simply means that a majority of the critics liked the movie. From there you need to go and read individual critic reviews, and see how they scored the movie. I have a handful of movie critics who I fall back on, so I get a feel for how the general population feels about the movie, and then I read specific reviews.

[Vampyr]

Originally Posted by Angrist

Hm reviewing Wii games is difficult. Graphics for example: would a 10 be for the best possible on the Wii, or the best possible on whatever console? THen Wii games would never score higher than a 7.
So I know a lot of gaming sites rate stuff based on what's realistic on that platform. And I think that's fair.

Also speaking of graphics... what exactly are you rating? The technical aspects? That way Wii Sports should probably get a 2 for graphics. But when you look at how good something is on the eye... Wii Sports can suddenly get an 8 for graphics. It just looks fun.

Anyway... do I care about review ratings... Sometimes. There are some games which I just know I'm going to like (Zelda, Starcraft/Warcraft, Mario platformers...). But when I want to try something new, I'll read reviews and I'll also pay attention to the ratings. Only when I think the bad rating was giving for the wrong reasons will I try a game.

It's my opinion that Wii games should almost always get low scores on graphics, unless they are very artistically unique.

They shouldn't get a free pass because Nintendo flaked out on HD.

[gekko]

I find it a bit odd that the same posts trying to argue that there is bribery between the media and publishers are the ones which are against corporate news outlets (IGN, etc.). You actually have the logic backwards. Places like IGN have very strict policies on these types of practices. Certainly there isn't quite the level of journalistic integrity you would find working at 60 Minutes, but it's a very serious business.

[KillerGremlin]

Originally Posted by gekko

I find it a bit odd that the same posts trying to argue that there is bribery between the media and publishers are the ones which are against corporate news outlets (IGN, etc.). You actually have the logic backwards. Places like IGN have very strict policies on these types of practices. Certainly there isn't quite the level of journalistic integrity you would find working at 60 Minutes, but it's a very serious business.

Does IGN not do early reviews at the behest of the publisher?

I'd trust major gaming sites more if not for stories like the famous Kane & Lynche debacle back at Gamespot, so color me skeptical. Also, I would say 60 Minutes and any major news outlet is about as credible as a Tijuana asshole. You want to give it several look-overs and make sure it doesn't wink back at you before you dive in. Any mainstream news source comes with a series of baggage inherent to being a mainstream news source.

[Angrist]

Originally Posted by Vampyr

It's my opinion that Wii games should almost always get low scores on graphics, unless they are very artistically unique.

They shouldn't get a free pass because Nintendo flaked out on HD.

But still, look at Metroid Prime 3, Zelda Twilight Princess, Monster Hunter 3 and Super Mario Galaxy. Technically they'd get a 6 or so: low res, too few polygons. But when you take a step back and only pay attention to how it looks to you.... they suddenly get an 8 or better.

[gekko]

Certainly the publisher provides the journalists with the copies of the game before it's released, but it's not with any agreement that the game will receive a high score. Reviews are advertisement, and almost certainly ensure when the game releases there's an article on the front page of every site featuring it. Sure a good review is always what they want, but this is video games, not politics. It's not ALL corrupt.

Yes, there is certainly a relationship there. IGN would be nothing if it couldn't get the news from publishers, but IGN is also the largest gaming site on the web, and publishers want their game on the front page. Larger sites like IGN have strict policies to prevent any possible issues. IGN has policies against journalists receiving gifts from publishers, and the journalists are also deliberately not told about advertisement deals. For example, the journalist reviewing Gears 3 isn't allowed to know that the advertising department was planning to give Gears 3 the full-page ad this week because it might influence the score. If the game for a 5/10, that ad would still have gone up.

Now there are certainly examples of corruption in the past, but that kind of thing can easily kill a journalist's career and severely damage the reputation of a company. For a fan site or blog, that means nothing. For a site like IGN, those guys (some trained journalists, some not) are making a living doing that job, so that's a huge risk. Also it's a big risk to News Corp. if they get caught. There's famous stories of this kind of thing happening to companies as big as CBS (with the famous Wigand story), so I'm not saying it's out of the question. However, it's certainly not the norm.