Kotaku: Reviewing the Reviewers: Or How To Break A Broken System

Kotaku writes: "Just what you've been waiting for: A site that lets your critique the critics, review the reviewers, shit all over the shit all overers.

Website Criticosm allows you to give reviewers a review score of their own. They then use a reviewer's average score to weight the importance of their score for a game when calculating the average score a game received."

[Foxgod]

Who is gonna review the reviewers of the reviewers?
If a reviewer gives an opinion about a reviewer, it better be a reviewer with backbone.

This stuff seems to be based on public opinion, sadly public opinion is even dumber then Anonymous blogging.

[rockleex]

I do.

[Mo0eY]

You are right, Foxgod. Kotaku is smothered in bias for the Xbox 360. Why cater to the public with a review of other reviewers when you hold your own personal vendetta?

No thanks.

[Tony P]

So basically this is the official site to whinge when a game scores more or less than you think it should.

Awesome. /s

[sikbeta]

Tooltaku Reviewing the reviewers, sorry but this guys has anything better do?

This is insanely Stupid, who'll take THE TIME to do some cr@p like that

[Gen0ne]

Is this fact? Or your take? I've always felt they were pretty good at staying non-biased. Or am I missing something? Examples please, to help support your claim.

[xabmol]

Someone hasn't been watching the Bonus Round...

[Foxgod]

Before you know it, people will start claiming that PSM is favoring the 360.
Hey, maybe people favor the 360 cause it good, ever thought of that?

Instead of blatantly acting like its a bad thing to favor the 360.

[xabmol]

Michael McWhertor, Senior Editor of Kotaku.com, is a blatant FANBOY.

So obvious, they all call 'em out on it!

[BeaArthur]

Meta scores are usually a good gauge of how good or bad a game is. If you do your do diligence in researching a game you should be fine.

[Oner]

Exactly. It is just 1 component to utilize ~ not to be used as the decisive factor because it always comes down to YOUR OWN experience with the title.

And for the record it is probably better to use N4G's Meta Rating's as they don't leave out particular sites or "weight" some less/more than others.

[Unicron]

I'll stick with valuing my own opinion over some random "journalist" thank ya.

[Rockox]

Right there with you.

[Gen0ne]

An interesting idea though. Just not enough for me to partake in.

One second thought, reading some of the reviews of the reviewers could prove hilarious. I dunno, this could be comedy gold.

[Unicron]

I've said this before, but I really like Kotaku's system. I may not always agree with their opinions, but I really feel listing the pros/cons and how much of the game was actually played is a great way to do things. It makes people... gasp... read!

Bubbles for ya both.

[Rockox]

I don't often concern myself with reading reviews, so Unicron when you mentioned Kotaku's reviewing system, I had no idea what you meant. So I looked at a recent review and whattaya know...no numerical or alphabetical rating. It was glorious. If I wanted to know what the reviewer thought about the game, I actually had to read the review front to back. I wish more sites would adopt a similar system.

[huga_muga]

Only gamers use their own review "opinion".

people should need to stop supporting review scores.

[BeaArthur]

Plenty of people who aren't fanboys use reviews. If a game has no demo reading the reviews is the only way to try and determine how good it is without throwing $60 at something that might suck. I'm all for forumlating your own opinion but I'm also not interested in wasting my money.

[cereal_killa]

There's nothing wrong with people looking at reviews but what bothers me is people using some peoples opinions as the word of the Gaming Gods I see fanboys on here all the time saying "wait till Game Trailers or IGN or Game Informers reviews to come out before I play this game". I cant stand Kotaku but they do the best reviews NO score, No stars, No letters nothing but there opinion on if they like it or not and that should be the way all reviews are. Fanboys would have nothing to gloat about and would probably cry because they can't make fun of the other consoles games. Hell I wish all together they stop with scores on reviews and keep sales of games and consoles private to only developers, publishers, manufacturers and share holders.

[Unicron]

Huga is right. Reviews are good to inform, but review SCORES are utterly useless and 100% arbitrary.

[Montrealien]

so, this is how it works.

you read many reviews, you start getting familiar with certain people that write reviews, some share your tastes, some don`t. However, in the end, the only opinion that truly matters is your own. But that does not mean that you can`t get the opinion from other people you trust to see if you may like a game or not? Of course not. However the most important thing is that you don`t judge a game simply based on what others think.

[BeaArthur]

Unicron...technically the numerical score/letter grade is just a representation of the written opinion. I like the numerical score because it provides me with a quantitative evaluation of the game. I agree, the written portion is far more informative than any score could ever be; but if they just have the written review with no score then it makes it hard to determine whether they think the game is average or above average. A written review with no score could leave you wondering whether they thought it deserved a 7.5 (relatively average) or an 8.5 (a pretty solid/good game).

I don't know if people are actually understanding what I am trying to say though. I am not saying that professional (and yes I will use the term loosely) reviewers are the only evaluation of a game you should have. I'm not saying that at all. What I am saying is that if there is no demo and I am trying to decide whether to pick this game up at it's release then the only information I really have to go on (other than random gameplay clips and preliminary evaluations of an unfinished product) are the reviews. With that being said I have had plenty of situations where the reviews said it was great and I hated it and plenty of occasions where the reviewers thought it was terrible and I loved it. The point I am making is that I think review are a good preliminary evaluation tool. They are by no means the word of law.

[Unicron]

I agree bea, they ARE a good evaluation tool. The problem is people are turning them into law, and that scares me, as many people are missing out on some GREAT titles. But to your other points -

My problem with the score is that they are quantifying opinion. Someone has yet to tell me the actual difference between a title scoring a 9.3 and a 9.4 for example. I have people trying to tell me that a game is 0.1 better than another? Yet they are in different genres, with different gameplay focuses? There is no mathematical way to really arrive at these scores, they are essentially just made up. IF they just had the written part, why would you need the score? A well written review SHOULD include the very point of them saying "I feel this game is average/above average/great/amazing/timeless /for fans only," no?

You also have to look at how twisted and deranged scoring scales have become. A 7.5 is now a flop? A game getting a 9.6 should have gotten a 10? The review scale means not what it used to. Consistently high reviews on high profile franchises also makes me wonder... is the game THAT good, or is it the pull of the marketing team/promises to the publisher? But I digress...

[BeaArthur]

Unicron...What scares me is all the hate that seems to come from these reviews. To me the real issue is the reader's getting so worked up over an opinion. You will never be able to satisfy everyone with whatever review/score it gets and therefore you will always have those people that cry foul. So often I see people claiming they are now going to boycott a site that they have been a loyal reader of because they don't agree with one review.

Very True, there is no way to decipher a 9.3 from a 9.4. In that respect you are absolutely correct but giving a numeric score on a well written review is kind of ironic. If these reviews were all written well then we wouldn't even be having this conversation about the state of game reviews. True, a good review shouldn't need to add a numerical score but since the credibility of reviews is being evaluated we almost have to have a numeric score so that the review makes sense. At the same time though if we didn't have numeric scores I think that would actually be worse because then we would have the same reviews actually saying whether the game is good or not. The number is subjective. Although it is commonly agreed upon that a 7 is an average at best game having someone actually say this game is average would create an even bigger firestorm than just giving it the 7. But I see your point though. If the review just said average the reader could make up their own mind as to what that meant instead of seeing a number and saying "that's not average that is bad".

I think the term flop gets used too infrequently. To me a flop is a game that had a ton of hype and failed miserably. Some would say MAG is a flop but I would say it is a good to average shooter based on who you ask. By my definition I don't know if I've ever seen a true flop. I would agree that big name franchises probably get preferential treatment but at the same time I would say that they also pump out better games more consistently. I'm not ignorant to the fact that some publishers probably grease the hands but that is why we have meta scores. If I see a bunch of 8 and then a 9.5 I know that the game is more of an 8. Like wise if I see a bunch of 8.5 and then a 6 I know the game is probably more of an 8.5.

[Rockox]

The sad part is that even after a review without a number/letter/star system is released, I bet people would STILL go to the reviewer and say "So, if you HAD to give the game a rating, what would you give it? From your review, it sounds like it'd be, oh I don't know, an 8.7. Amirite? Amirite?" Some people just can't stand for things to not be rated as such.

[Socomer 1979]

Its a chance to hurt reviewers just as much as reviewers hurt developers.

The next time these crumb bums go up for a promotion this will be thier metacritic. I suggest we use it & curb these guys with our power.

Happy days are a comin!

[BeaArthur]

So you would give a reviewer a bad review to what end? If you give them a bad review to "curb these guys with our power" than you are only making the problem worse. I mean what do we all really want? We want an unbiased evaluation of a game within the guidelines of a well formulated criteria. If you start trying to "curb" the reviewers then they will start reviewing games in order to please the people who are reviewing them which is exactly what we don't want them to do.

[Socomer 1979]

Firstly, these reviewers suck.
its painfully obvious many dont even know how to play a game, finish a game, have preferences, are unprofessional, continuously creating flamebait and then place the blame on fanboys, sell a game short, overhype, late to a review a game, are cynical, are jaded, are lame, do not relate to hardcore gamers, misinforming the public, lack of details, forget, are greedy, combative & did i say suck?

we need more optimistic reviewers who are in it for the challenges and the fun. I have more respect for user reviews than i do the so called professional garbage ive been reading for too many years now.
If you are a blogger who is scared of this tool, you know what? Get better. Reviewers now have too much of a big ego where they believe they are gods of gaming and could give a crap about stealing developers development money for marketing.

[BeaArthur]

You say the reviewers should "get better"; but what makes them good or bad? Lets be honest, you have no idea what the reviewer is looking for when they review a game. They might think graphics are important while you think story is important. Telling someone to get better on something that is subjective doesn't make sense. You have all these varied issues with reviewers and none of them are actually quantifiable, and then you tell them to get better.

I mean the conspiracy theories are so laughable. You have so many opinions on something you are probably completely ignorant on. You are also probably the guy who thinks you know how to fix his favorite football team.

You don't have to like or read reviews but don't sit there and act like they have no idea what they are doing, or that they are biased, or insinuate that the get paid off. Is the process perfect? No, and it never will be but lets be honest, you have no idea how to improve it.

[Sackdude]

Your wrong, there are a lot a games thet got low scores and these games were my fav,

Reviews are opinion, which can be different from your opinion, that makes reviews not even 50% reliable, you can rent a game if there is no demo for it.

also huga is right, this gen all a see is "my game got 0.1 better score then your game" and all that garbage,

also i don't need journalist who don't know nothing about gaming, to review a game, ill review it my self, because i know a good game when i see it.

[BeaArthur]

samsamurai...I agree, there are opinions that seep into reviews (that's pretty much unavoidable) and I'm going to sit here and say what your favorite game means to you. But if the majority of reviewers give your favorite game a low score and you think it's great, which one of you would be perceived as correct? Now again I'm not trying to tell you how to feel about your game and their are no right and wrongs when it comes to reviews, but if 30 evaluators are telling me a game isn't very good and you are telling me you love the game, I am probably going to go with the 30 people that tell me the game sucks. There have been plenty of games where I disagree with reviewers opnions but in my experience when I go out and try a game that got universally panned, I find they were typically right.

So by all means play whatever game you want but I'm going to do everything I can to not waste my money and if that means reading reviews, so be it.

[freelancerx8]

this shouldve been here along time ago, i like how the dtoid author sais that we shouldnt use a broken system to review a broken system, yet dtoid will hapily keep using numbers to review their games. screw that you cant say that something isnt fair and then turn around and keep doing the exact same thing. i hope this really catches on and maybe it will force review writers to actually think twice about the quality of their writing and how they review games.

[Socomer 1979]

what you are talking about is mind control.
if this guy who youve known on n4g for years is saying a game is good & you know what kind of gamer he is but you decide to go with 30 professional reviewers who say the game sucks is a form of mind control.

If you actually opened your eyes a bit you will realize that your friend really enjoys the game and is reccomending it to you. the professionals are not marketing the game because the publishers gave them no incentive to. no ads. no plans for future ads for other games. a company argument over rates can affect this game.
in which case, you lose. your trust is in the wrong place.

[BeaArthur]

Actually, that has nothing to do with mind control. Besides no one was talking about coveting the opinion of someone you know over 30 people you don't. If I knew someone really well and they said a game was great and he and I had the same taste in video games, than yeah, sure, I would value his opinion more but no one was talking about that.

I was talking about a situation where there is no demo and the game has not yet be released and I'm trying to decide whether to pick it up or not. I mean your whole theory about my "friend" recommending a game is based on the idea that: 1.) my friend actually knows how to evaluate a game, 2.) that this friend and I have the same taste in games. Suppose he and I don't like the same games and he is saying that this game is good and 30 evaluaters are saying it sucks. Should I go with his opinion?

[kingme71]

Doomed to fail. For every 1 person that reviews a reviewer based on that person's effectiveness to carry out valid, unbiased and professional reviews you will have 9 others that rate them based on whether they agree with the score or not. If the review of the reviewer had any bearing on that person's job security, you can bet their reviews would no longer be about the quality of the game and all about keeping their job.

[Oztheboss]

Oh this is fantastic

[HeroComplex]

Kotaku + controversial articles/reviews = lots of website hits

Criticosm + angry fans who've read controversial articles (i.e. Kotaku) = lots of website hits

OK, who else thinks Criticosm is owned by Kotaku? Seems like Kotaku has written an article to promote their new website to get people to go there after getting upset over an article they've posted. Pure genius!!!

[xtremeimport]

$100 for the arc....why do I think this is far from being correct.