Erroneous league assignment

[GreatGonzales]

(07-26-2012 07:41 AM)Kamikaze28 Wrote:  Slightly longer answer: Outwitters doesn't use ELO. Explaining the inner workings of the current system would take quite an amount of time which is better spent on fixing bugs and improving the system. Knowing how it works doesn't make any difference other than satiating your curiosity. Even if an official explanation with formulas and everything existed, very few people would be able to or interested in understanding it. The current system has been designed so that players worry less about the outcomes of their games and more about playing, enjoying and giving their best at all times.

I guess my desire to understand the system is because it did not make sense to me that I was still in gifted after winning so consistently for so long. I wanted to know how you are calculating my skill level, so I can better manage my expectations for when I will be promoted. I am using past tense because I just got promoted to masters! Hooray.

Also I was under the impression that Outwitters uses some form of ELO based on something ArtNJ said back when he was a betatester. Not sure why you appear to have such animosity for the system...it is very reliable and elegant in its simplicity. But not the only system that works I am sure.

[Syvan]

Given the similarities between this ladder and Starcraft 2's, I find this explanation of SC2's ladder to be satisfying even if it isn't exactly true.

I suspect it is based on the MMR system which uses a skill rating and a confidence score for that skill. When you are winning constantly, your skill increases, but the system loses confidence in your skill making promotions slightly more challenging.

[Kamikaze28]

When you say

(07-26-2012 10:48 AM)GreatGonzales Wrote:  I wanted to know how you are calculating my skill level, so I can better manage my expectations for when I will be promoted.

I would like to emphasize my signature. I'm not a dev - just a beta tester.

(07-26-2012 10:48 AM)GreatGonzales Wrote:  Not sure why you appear to have such animosity for the system...it is very reliable and elegant in its simplicity. But not the only system that works I am sure.

I don't harbor animosities against ELO. It is a basic system with known flaws and problems. It is rather easy to implement, I'll give it that - but from a game design perspective, there are significant drawbacks, which were pointed out in the second thread I linked you.

[ArtNJ]

Facts: Great Big War Game uses ELO, works great. Hero Academy League uses ELO, works great. Outwitters system -- still doesnt work great. ELO is *proven* to be easy and workable in a game like this, so I am still not sure (after having discussed this issue with you already) what supposed flaws you are referring to Kamikaze. The only thing we discussed and agreed on was that ELO does need some modest start up time to start meaningfully sorting folks -- but its not much, as Great Big War Game is currently showing.

Blizzard's unique system may work for Blizzard, but it appears highly doubtful that a 2 man shop can just copy it and not get burned, as OML has been burned (in my opinion).

[Kamikaze28]

(07-26-2012 09:30 PM)ArtNJ Wrote:  Facts: Great Big War Game uses ELO, works great. Hero Academy League uses ELO, works great. Outwitters system -- still doesnt work great. ELO is *proven* to be easy and workable in a game like this, so I am still not sure (after having discussed this issue with you already) what supposed flaws you are referring to Kamikaze. The only thing we discussed and agreed on was that ELO does need some modest start up time to start meaningfully sorting folks -- but its not much, as Great Big War Game is currently showing.

Blizzard's unique system may work for Blizzard, but it appears highly doubtful that a 2 man shop can just copy it and not get burned, as OML has been burned (in my opinion).

I will not deny that ELO works in the way it was supposed to: it gauges skill relativistically and can be used for more or less fair match ups. It was designed to do that - for competitive chess players.
The flaws I've been referring to are not directed at the workings of the system, but rather at the implications regarding user experience. Seeing your rating influences your behavior just as behavior influences your rating: you start to worry and may even reduce your activity in favor of protecting your rating. Seeing the impending loss of rating in a game you are about to lose to a lower rated player may encourage stalling or other mean behavior.

One other thing (that just came to my mind) which is kind of a concern with the functionality of ELO: it was never designed with the possibility of having more than one game at a time. Imagine this scenario: you are highly rated (say 1800) and currently winning against an equally highly rated player - what's to stop you from purposefully losing some games in the meantime to increase the point loss for your opponent? You are competing for ranks after all and dropping him lower is beneficial to you. Knowing the formulas even enables you to calculate exactly how many games you need to lose to break even, so that you stay at your 1800 rating, but your opponent loses points as if he lost to a 1700-rated player.
Similarly, you can hold out on winning games to cushion an eventual loss against an equally rated player. In other words: you can trick the system.

Of course, you can always say that no-one would worry this much (I'd disagree), because we are playing Outwitters for fun. But why then do we need a rating system designed for chess championships?

Regarding the ramp-up time: I'd argue that any rating system needs that - and the way I remember our agreement was that this is not a difference between the current system and ELO, but a commonality.

[GreatGonzales]

I think your concerns would be sufficiently mitigated by merely keeping the precise ELO score of every player confidential, much in the same way Outwitters does not communicate everyone's rank. Very simple. The problem you describe with playing multiple games at once, eh, I don't think that's a big deal. If ELO is secret, then there is no way to abuse the system. It will all even out in the end, and people will be placed as they should. That's the whole point of the leagues, right? That people are matched with players of similar skill level? I cannot think of a better way to do this this than ELO.

[Kamikaze28]

(07-26-2012 11:44 PM)GreatGonzales Wrote:  I think your concerns would be sufficiently mitigated by merely keeping the precise ELO score of every player confidential, much in the same way Outwitters does not communicate everyone's rank. Very simple.

Since the rating system of Outwitters is a black box, arguing over the inner workings of said black box is kind of pointless, since everybody involved (except the devs) is guessing.

(07-26-2012 11:44 PM)GreatGonzales Wrote:  The problem you describe with playing multiple games at once, eh, I don't think that's a big deal. If ELO is secret, then there is no way to abuse the system. It will all even out in the end, and people will be placed as they should. That's the whole point of the leagues, right? That people are matched with players of similar skill level? I cannot think of a better way to do this this than ELO.

ELO is not the best system there is, if you want to dig deeper into this matter, I recommend the Wikipedia article which gives details on how it is not perfect (warning: there are statistics and mathematics involved).

ELO might be the system which is most easily understood by interested people and the most widely spread rating system for exactly that reason but it is 52 years old - and improvements have been made since then.
As much as I enjoy simple and elegant solutions which are easily understood and explained - good rating systems resist simplicity, they become more complex over time.
At some point in this complexity-increase there comes a point where explaining it to people without the necessary knowledge in statistics or mathematics becomes quite a challenge thus foregoing any explanation and just declaring the system a black box is the rational way to go.

Until this point, the most common complaint on these forums about the current system is, that it's not going fast enough for some players - which I can sympathize with. But given enough games, I have not heard anyone being matched properly. The main discussion revolves around the question of "how many games is enough games?".

[ArtNJ]

Kamikaze,

With respect to the many games at once issue, its actually not a big deal. Correspondence chess has been dealing with that for decades. Basically, if a player delays a loss to gain points, then when they do take the loss they lose a lot more points. Conversely, if they take the loss immediately, then the subsequent wins yield more points. Yes, you can game the system to get over a rating threshhold for an achievement or to try and temporarily and briefly grab a certain spot on the leaderboard, but in the long run its very counterproductive.

As far as the encouraging people to play point, its hard to debate because there is no data so we are speculating about people's feelings. Certainly Blizzard put a lot of thought into it, but clearly there are people that are very turned off by the Blizzard system. Moreover, because its much more complicated than an ELO system, OML did not get it right out of the box, and it may still not be right.

I wont claim I can prove anything with this data, but I offer you the following information for what its worth: in the Hero Academy League (800 registered players, at least half of whom are or were truly active) about 1/3 of the most active players by games completed are outside the top 40 rated players, and some of the most active players are actually poorly rated. ELO, and the ability to tangibly track changes in their ability, motivated them (a lot!) to try and improve. My belief is that if you look at the most active Outwitters players, the percentage of people that are very active, but weak, wont be much different from the Hero Academy League.

Well, I guess that data does prove one thing -- that the Blizzard hypothesis is at least debateable, agree?

Anyway, OML made its decision, its set in stone. All they can do now is try and tweak the formulas so that the best players get top competition before they get bored and quit. Surely we can agree that no one should go 30-0 and be in Gifted.

[Kamikaze28]

(07-27-2012 12:09 AM)ArtNJ Wrote:  With respect to the many games at once issue, its actually not a big deal. Correspondence chess has been dealing with that for decades. Basically, if a player delays a loss to gain points, then when they do take the loss they lose a lot more points. Conversely, if they take the loss immediately, then the subsequent wins yield more points. Yes, you can game the system to get over a rating threshhold for an achievement or to try and temporarily and briefly grab a certain spot on the leaderboard, but in the long run its very counterproductive.

True enough, but the short term repercussions from these kinds of cheesing may discourage long-term motivation thus rendering the point mute.

(07-27-2012 12:09 AM)ArtNJ Wrote:  As far as the encouraging people to play point, its hard to debate because there is no data so we are speculating about people's feelings. Certainly Blizzard put a lot of thought into it, but clearly there are people that are very turned off by the Blizzard system. Moreover, because its much more complicated than an ELO system, OML did not get it right out of the box, and it may still not be right.

Blizzard did not document their system in any way I am aware of - any available material has been reverse-engineered by the community. I don't know what kind of research Alex did before implementing this system. I don't know if there are books or papers on these kinds of systems, because that's not my area of study. What I do know is that if you think you can do something better than others, you are bound to take some risks.
Whether or not the system is 'right' is a very subjective matter and a definitive assessment requires amounts of data we don't have access to. We are limited to our own experience to form our opinion.

(07-27-2012 12:09 AM)ArtNJ Wrote:  I wont claim I can prove anything with this data, but I offer you the following information for what its worth: [...]
Well, I guess that data does prove one thing -- that the Blizzard hypothesis is at least debateable, agree?

Agreed.

(07-27-2012 12:09 AM)ArtNJ Wrote:  Anyway, OML made its decision, its set in stone. All they can do now is try and tweak the formulas so that the best players get top competition before they get bored and quit. Surely we can agree that no one should go 30-0 and be in Gifted.

I understand your point, but once again I'd like to explain that statements like "no one should go 30-0 and be in Gifted." are highly relative, especially in the beginning of a rating system. How good were these 30 opponents? The system doesn't know this from the start. It's bound to be cautious. 30 wins against unranked or freshly ranked players hardly means anything. 30 wins against Masters or Super Titans with high confidence ratings is a very valid reason not to be in Gifted League.

[GreatGonzales]

(07-27-2012 12:46 AM)Kamikaze28 Wrote:  I understand your point, but once again I'd like to explain that statements like "no one should go 30-0 and be in Gifted." are highly relative, especially in the beginning of a rating system. How good were these 30 opponents? The system doesn't know this from the start. It's bound to be cautious. 30 wins against unranked or freshly ranked players hardly means anything. 30 wins against Masters or Super Titans with high confidence ratings is a very valid reason not to be in Gifted League.

I mean I think going 30-0 and not being promoted is just bad, no question about it. FYI I would say the average person I was being matched up with was high rated Gifted people. But if after 30 consecutive wins I am still being matched up with people in Gifted league, then that is the fault of the match-making system. So perhaps that's where the problem lies. FYI, I am in Masters now...which occurred probably after about 35 consecutive wins. I'm not sure on the exact number.

To be fair, I think part of my problem is that I don't get excited about my "rank" within a league, because I understand that the number of points someone has is largely dependent on their activity. So that I was increasing in my league rank did not satiate my ego; I wanted league promotion, and it happened too slowly imo.