Template talk:Ratings

Perfect vs Excellent
From Talk:Stereoscopic 3D Support and Compatibility/Sandbox, it occurred to me that we should bring one of the things proposed there over here! We should change "Perfect" to "Excellent". Excellent provides a way to allow minor errors in, such as shader compilation stuttering that is minor for every single game, without it interfering with the rating. And it should finally ease the "this is not perfect!" arguments, since the bar would be lower! It's also a drop in change for the 5th star, and very simple for us to do. What do you think guys? - MaJoR (talk) 10:13, 29 November 2015 (CET)


 * One thing to note is that the emulator would need to be updated too. I wanted to ask here first though. I'm pretty sure it will go well there, but it's something to consider when thinking about this. - MaJoR (talk) 11:29, 29 November 2015 (CET)


 * One other area to remember in this renaming is the 5 star VersionCompatibilityVersion templates. Kolano (talk) 23:20, 2 December 2015 (CET)

I very much dislike revising things so that the top rating isn't perfect. If some title is not perfect, then there should still be things to be worked on in the emulator related to it, and I don't what that to be lost as we start applying 5 star to imperfectly emulated titles. In theory we should have already resolved "shader compilation stuttering" resulting in imperfect ratings by migrating those problems to "Emulation Info" rather than under "Problems". Kolano (talk) 17:44, 29 November 2015 (CET)


 * We should be careful with "Perfect" rating, like, should we consider a game that need specific settings as "Perfect" or if a game needs settings that'll make it run very slow but accurate, should it get a Perfect rating too? That's the main issue I'm seeing with our current system, we lack a well defined set of aspects to quickly define what specific rating a game should get. Our current definitions are too vague and leads to different understandings, like this particular case... - Jhonn (talk) 18:23, 29 November 2015 (CET)


 * That is kind of what this is meant to deal with. We've had games that were perfect, and when a super minor bug is discovered, even though it was there all along, it is demoted to 4 stars. By lowering the standard just a little bit, a lot of problems will be reduced. "Perfect" is an impossible standard! A little give will smooth out a lot of issues in the rating system, imo. - MaJoR (talk) 08:43, 30 November 2015 (CET)


 * "Perfect" in technical sense, yes, it's impossible to emulate a game like perfect. When it comes to rate the compatibility with a game, it's true that when there's a minor (yet legit) bug discovered then it will receive 4 stars in an instant, it's also far easier to comprehend and manage. Just wait until that game has zero active problems in the "Problems" section. Be grateful that there is Emu Info section exists for non-genuine problems! It's matter of what in sense does the term "Perfect" refer to. And for some reason the term "Excellent" doesn't get along well with "Playable" and other terms in the compatibility rating list. Lucario (talk) 11:02, 30 November 2015 (CET)


 * Why would Excellent not get along with Playable? :/ It seems fine to me. - MaJoR (talk) 14:27, 30 November 2015 (CET)


 * The definition behind five rating sets feels distinctive from each others. I think it's perfect (not technical of course!). The definition of "Excellent" will blur the bar between "Playable" and "Perfect" and will not end well as if there's debate between whether the problem is quite "minor" or not. We shouldn't be splitting hairs there. With "Perfect", anytime there is a problem then let the other four rating sets do the job. They're based on where the point it crashed during the emulation. If no crash, gameplay progress fine, but still has problem, "Playable" it is. Lucario (talk) 07:34, 2 December 2015 (CET)


 * Hmm... How about this - we still have minor issues demoting from 5 stars to 4, so perfect would more or less just be renamed. But even like that, I still think excellent is a good thing to move to. Excellent would more or less be the same as Perfect, but with a small change - the global bugs, undiscovered bugs, and all of those things we ignore and allow us to call it perfect just because the problems area is empty? It's tolerable now, because it's NOT PERFECT! Perfect is perfect, and as long as we have shader compilation stuttering, *nothing can be perfect*. Moving to this would give us the wiggle room to actually say something is excellent and allow us to actually mean it. It wouldn't be about changing our rating system at all, it would only be about changing the word we use to describe the 5th star to better reflect how we are using it. - MaJoR (talk) 17:30, 2 December 2015 (CET)


 * I wonder if the shader compilation stuttering is pc performance dependent? Hypothetically, there is one badass PC with latest gen specs, will we still see stutter there? If not, then no way it will hurt the compatibility rating of the game. We shouldn't mix pc performance with compatibility rating system. Anyway, there are side effects to consider about when changing from "perfect" and its description to "excellent". People don't have the exact same mindsets as what we're having here (I'm okay with "Excellent" as long as it's exactly the same as "perfect" in the compatibility rating system, just becoming less too technical behind its meaning). Some may rate the game as 5 stars despite there's some active issues in the problem section that may seem extreme minor to them but didn't know that the 5 stars was only when the problem section is empty. Lucario (talk) 09:49, 3 December 2015 (CET)


 * A computer with the absolute latest and best parts overclocked to the maximum will still see shader compilation stuttering. It's a fundamental difference between how the GC/Wii and PCs work, so it isn't related to performance at all. Another example of this is issues with forced perspective to get around CRT timing issues, which creates subtle bugs in many GameCube titles, such as one pixel of masking being wrong, one pixel high jittering (which is a bigger problem worthy of the problems area), and other minor things without easy answers. All of these minor problems I'm fine saying a game is "Excellent", but "Perfect"? It feels wrong to me to ignore those things... - MaJoR (talk) 10:10, 3 December 2015 (CET)


 * What about that PC from the year of 3,000? Just kidding! I read somewhere that shader caching will reset every time there a new GPU driver installed. It's still PC dependent in some sense, no? I figured out why using "Excellent" in compatibility rating system feels funny to me. Dolphin is never perfect, that's certainly a given. When it comes to "Compatibility rating"? It just asks if a game works fine on Dolphin, so if there's a crash, glitch, or such thing that disrupts player's gameplay progress, rate it. "Excellent" is probably far more appropriate term when it comes to "Accuracy rating" or so what we'd call it. I actually don't consider the accuracy flaw a "problem worthy" if it doesn't disrupt gameplay progress at all (or catching their attention in x1 native IR). Unless I'm not understanding you correctly, does the one pixel high jittering still happening in x1 native IR? Any subtle bugs only visible with IR higher than x1 are not the real emulation problem. People can still provide patch for it in the Enhancements section. Tales of Symphonia is one good example. Lucario (talk) 01:07, 5 December 2015 (CET)


 * "When it comes to "Compatibility rating"? It just asks if a game works fine on Dolphin, so if there's a crash, glitch, or such thing that disrupts player's gameplay progress, rate it." By that definition, once a game is completable, it is "perfect". That's ridiculous, and not what anyone expects by the definition of perfect. The plain common-sense interpretation of perfect compatibility is no problems at all ever under any circumstances period. That you can play this game and never encounter a bug whatsoever. I absolutely do not think that the name "compatibility rating" vs. "accuracy rating" influences the definition of the word "Perfect". Not to mention we are using it as an accuracy rating and a compatibility rating in one measure. Anyway, I'm not going to talk about the details of these bugs or try to justify these bugs to you, but if you actually thought Dolphin was emulating perfectly, you need to move past that. It does not, and never has, and any dev would tell you that. There are fundamental problems with how Dolphin emulates the GC/Wii on PCs that cannot just be worked around. There is absolutely no game that qualifies as perfect.
 * Anyway, this conversation is starting to go into circles, so I'm going to push it toward a conclusion and ask the other admins to weigh in. My point is simple: Dolphin can never be perfect! And yet we are ignoring global problems and unknown bugs to give games the immutable "perfect" rating. By changing from Perfect to Excellent, and do no other change to our ratings template, we are making the wording used in the compatibility rating better match how we are using it! And that's it! We'll actually be able to tell someone that a game is excellent without lying to them. That is what bothers me about "perfect" and why I want to make the change. - MaJoR (talk) 09:48, 5 December 2015 (CET)


 * I don't think you get where I'm at... I brought up "Accuracy" vs "Compatibility" because the way you said with "Dolphin is not perfect" sounds like you're thinking about "Accuracy rating" rather than "Compatibility rating" on this subject. "Perfect" in accuracy would mean 1:1 identical to the real console which is impossible, of course. It'd be lying to people about this. I'm just saying that the "Perfect" is applicable term for the "Compatibility rating" even if the emulator is not perfect. In other words, it's "Perfect" when a player/tester played the game on x1 native IR at countless time in every possible way and found no apparent bugs so far since. It's pretty much what "Compatibility rating" was asking for. "Excellent" is another applicable term but it's got problem with how to handle bugs that may be minor to some but not quite to others.


 * I'm starting to feel stupid especially after long discussion like this. Please disregard my opinions if you think I'm still wrong. Lucario (talk) 01:34, 6 December 2015 (CET)

After the whole discussion I'm in favour of moving to Excellent as well as in essence it would just adjust to what we were doing in the past years, what we always did was tagging games that have no known problems as "Perfect" even when that certainly is not the case! (e.g. a less-known game that we don't know it have issues because nobody reported -- it can occur even with popular games, Super Smash Bros. Melee for example kept tagged as "Perfect" for a long time even with its known problems!). So, I suggest we move on with Excellent and then rename the relevant parts (compatibility table and chart, ping delroth or Parlane to update the wording in the main site as well, etc) - Jhonn (talk) 17:33, 5 December 2015 (CET)

(Outdented Jhonn and My responses, it gets silly that deep, and they are general responses anyway)

First off "Excellent" does not equal "Perfect" even if we try to define it as if it did. It seems to set a lower bar for entry and is likely to lead to arguments over games with real, but minor, issues. I understand the counterpoint that almost nothing can be emulated in an exacting enough way to be perfect. I don't feel it's a very strong point though, as the only example of anything currently reported on the wiki seems to be shader compilation stuttering, which is now classified as an "Emulation Information" topic and as such won't effect ratings. Unless there is some other widespread issue that can't be similarly addressed I don't think arguments against "Perfect" hold much water.

That being said, I would be supportive of getting text added to the wiki for the few other things Major listed (but I'm unaware of specific examples of them)... ...these sorts of things likely should be listed on the wiki, and if they cause titles to no longer meet our "Perfect" rating (i.e. title can be played without issues /w default configuration) so be it. Even if they are global problems effecting a large number of titles.
 * issues with forced perspective to get around CRT timing issues, which creates subtle bugs in many GameCube titles, such as one pixel of masking being wrong
 * one pixel high jittering (which is a bigger problem worthy of the problems area)
 * other minor things without easy answers

Major seems hung up on classifying something as "Perfect" that is not, and also seems to be of the opinion that many games currently classified as Perfect are not. It's unclear how it was handled prior to the wiki, but generally I'm confused how imperfect games would have received perfect ratings. The three cases I can think of: ...but I'd hope those aren't common occurrences. It also doesn't seem like it could be that widespread a problem, as there are only 295 titles, out of multiple thousands, that receive a perfect rating. If some of those have issues we should investigate them, but I'm not clear why we should assume that to be the case (and if we can assume such, we should drop them all to "Playable" now). Kolano (talk) 22:29, 5 December 2015 (CET)
 * Minor bugs missed by tester
 * New bugs introduced over time
 * Games maliciously rated as perfect by unscrupulous wiki editors


 * Had some further discussion with Major via IRC last night. She was supposed to update here but that apparently didn't happen. A rundown (with some minor edits to shorten things) appears below.

(12:04:53 AM) MaJoR1: perfect doesn't mean "no issues after subtracting issues we do not declare are issues" it means "no issues period" (12:05:55 AM) MaJoR1: who are you to declare that an issue is not relevant enough to say something is or isn't perfect? (12:06:39 AM) MaJoR1: by moving to excellent, to me, it means owning up to that. Yes we're making a value judgement, and we always have, so instead of pretending it's perfection, let's call it how we are using it, and accept that we are making judgements ... (12:08:36 AM) MaJoR1: to me, you appear to be hiding behind the same beauracratic structures of lies that we have always used to justify what someone decided to call the ratings back in 2008, while avoiding the basic reality of it all (12:08:40 AM) MaJoR1: it's frustrating… (12:09:38 AM) Kolano: My concern is that to a random wiki editor "Excellent" is likely to mean just that, "good" but not "perfect". So by not using the term "Perfect" we are likely to increase the likelihood of bogus 5 star ratings. We already have to roll things back frequently as is, but we at least have a easily explained rule for doing so. (12:10:41 AM) MaJoR1: hmm, that is a legitimate concern… (12:12:46 AM) MaJoR1: we'd need to refine what we call the rating in the explanation text (12:13:40 AM) MaJoR1: it could encourage more incorrect 5 stars, but, honestly that is already an issue, if we set it up correctly, it shouldn't make it worse (12:14:13 AM) MaJoR1: then again it might, and then again it might reduce it, we have no way to know that :/ (12:14:20 AM) Kolano: I think that's a solution I'm more comfortable with. Also, again, I'm perfectly happy not rate things as perfect if there are actual issues with them. But we need to know what those issues are. Per my response above to shader compilation, there may be possible resolutions in time, There are likely to always be subtle problems too, but at some point I do think it's possible to draw a line. (12:15:52 AM) MaJoR1: I'm still not happy with the word perfect… but if you can make the text clear that it is not "Not issues at all" and that subjective estimates are involved "relevant issues", I think I could be ok with that (12:16:34 AM) MaJoR1: I still think moving to excellent would be better, but eh, compromise means you don't get exactly what you want! :P (12:18:51 AM) MaJoR1: if you are ok with that, I can write up a formal proposal on the wiki and put it up, so we can bikeshed on the definition of perfect ;) (12:19:41 AM) Kolano: That's perfectly fine, was going to ask you to do so, so I can get back to Fallout 4. Also, don't think compromising let's you get out of explaining some of the problems you were referencing. ;) ... (12:23:25 AM) Kolano: k, I think I've seen the jittering thing, I would like more specifics on the off by 1 pixel masking issue. (12:23:43 AM) MaJoR1: you haven't seen it? (12:23:56 AM) MaJoR1: with crop off, lots and lots of games have the mask off by one pixel (12:24:07 AM) MaJoR1: it's an old old old dolphin problem (12:24:24 AM) MaJoR1: yet another reason why I wish crop was default :/ (12:26:34 AM) Kolano: I had never used crop till the other day, but no I don't think I've noticed it. At the same time I'm not clear what "it" is so it's hard to say. (12:28:02 AM) MaJoR1: when a fullscreen mask is in effect, often a one pixel tall line on the bottom of the screen will be outside of that mask (12:28:19 AM) MaJoR1: this could have been addressed by the off-by-one fixes that were done recently, though… (12:28:30 AM) MaJoR1: phire has been working on these oddities for a while (12:30:53 AM) Kolano: I'm sure it's prevalent, but do you know a specifically effected title I can look at. (12:31:39 AM) MaJoR1: I've been using crop almost exclusively since that feature was added, I'd need to turn crop off and check through games (12:34:29 AM) MaJoR1: sorry, you know how it is when in a rapid discussion like that… (12:34:46 AM) Kolano: k. I'd still like to have some better clarity, but this seems to be another thing I wouldn't have a problem having listed on the wiki, even if it impacts some of the ratings. It also seems like it may be fixable, so again I'm not clear that lowering current ratings should be a huge concern. (12:35:28 AM) MaJoR1: I was using it as an example of impact of why perfect was not applicable (12:35:41 AM) MaJoR1: but in the solution we talked about it isn't as big of a deal


 * So it seems Major may be OK with revising the description associated with the "Perfect" rating to something like Perfect: No relevant issues instead of changing the rating title. I'm still hoping we can get clearer descriptions of the global problems she's been referring to, as well.


 * That description next to Perfect rating... it's felt wrong for me too. It's what made "perfect" sounds too literal (really... it might've been the root of this whole argument.) and doesn't do well with minor global and non-genuine (emu info) problems. "No relevant issues" is really good one. I'm in for this change! "relevant" seems interchangeable so "Game-specific" or "Unique" comes into mind also. They attribute toward to game more and disregarding global/non-genuine problems. Lucario (talk) 22:42, 6 December 2015 (CET)


 * Just a quick response that "Global Problems" or other non-unique problems should influence ratings in the same way that normal problems would (though I think we may violate that on some VC titles today). Kolano (talk) 02:20, 7 December 2015 (CET)


 * We would have to update the compatibility rating in all affected games back and forward each time a global problem is found then fixed. I don't think that's easy task. I must make an exception for significant global problems like ones in VC titles though. Lucario (talk) 08:38, 7 December 2015 (CET)


 * Yes, we would. I'd need to look through again but I think the current places we have violators are likely due to the global problem only effecting the PAL versions of things. That's a separate thing we need to discuss, I'll add a topic below. In any case, where we have global problems today there aren't enough titles with ratings for it to be much of an issue. I guess it could be problematic at some future point /w more global problems defined. Kolano (talk) 09:01, 7 December 2015 (CET)


 * I've came up with a better description just today. It's "No reported issues". It sounds more accurate and honest. Far more appropriate for the games that haven't been played or tested much yet. Lucario (talk) 07:30, 11 December 2015 (CET)


 * There are global issues, so that's worse! Issues have been reported, we're just choosing that they are not relevant. No relevant issues is the most honest option so far. - MaJoR (talk) 13:36, 11 December 2015 (CET)

So, conclusions? I read the new "no relevant issues" proposal and meh, looks like we're exchanging "six" for "half a dozen", essentially not changing anything. Considering Dolphin is an emulator of a complex console, unless some day it becomes an accuracy focused emulator (like higan), calling the top-most rating as Perfect doesn't look completely right to me. In other words, my vote goes now to Excellent as well - Jhonn (talk) 20:27, 10 December 2015 (CET)


 * At least "No relevant issues" is better than saying "No issues at all!". The latter one sounds misleading and childish so I'm in for this change, whether it's simply a placeholder for the upcoming better description. Have any idea how to write description for "Excellent" also? I'm ok for either but may blur the bar between "perfect" and "playable" and that's something we're afraid of. Lucario (talk) 00:28, 11 December 2015 (CET)


 * I think Excellent: No relevant issues" is fine. Unless someone has a better idea. - MaJoR (talk) 13:36, 11 December 2015 (CET)

Regions and Ratings
We currently have a few NES titles rated as 5 star, even though they have an open Global Problem. I think we've allowed that as the issue only effects PAL titles, but I'm not really sure. In any case, how should compatibility differences across regions effect ratings? I'd lean toward lowest compatibility winning out, but that doesn't seem to be the tact we've taken thus far. Kolano (talk) 08:59, 7 December 2015 (CET)


 * The non-English material may be receiving too much credit for this English-centric website if it's under the spotlight over the English edition material. Wikipedia has similar rule for the cover art from different regions here: wikipedia:wikipedia:WikiProject_Video_games/Article_guidelines. But I have this kind of mindset that there should be zero problem in the problem section to qualify 5 stars so I'm very unsure which one is right. Lucario (talk) 10:22, 7 December 2015 (CET)


 * I believe the standard, for example if you can play it in most regions and not in one, is to rate it according to the majority. If two regions don't work, and one does, rate it as not working. If two regions work, and one doesn't, rate it as working. But um, honestly, this is not something we run into often. By the time we created a ratings template (2011, 3.0 era), most of these issues were resolved. I looked at brawl's history, since Brawl didn't work in NTSC and NTSC-J until, but all regions were working before 2.0, and well before we even had a ratings template! And we only even know that because in 2011 I went back and tested it to find when it was fixed as part of a clean up! Most of that knowledge was lost. :/ But anyway, my point is we don't have the best track record on this since it's so rare for us to encounter it; this wiki really kicked in after most of the glaring region differences were addressed, and by then it was safe for us to assume region parity. - MaJoR (talk) 13:39, 10 December 2015 (CET)