PDA

View Full Version : Ressource Flood and Screw



Pages : 1 [2]

Werlix
11-12-2013, 06:45 PM
Werflix, you should just give up. Every time you prove someone's feelings wrong with data they will always have another theory on when or what is not working right. Soon we will be talking about lunar cycles.

I'm tempted to give up but I feel like I have a little fight left in me :)

I've gotten testing a new opening hand in the client down to about 9 seconds per test which means when I have a spare 30 mins I can run through a couple hundred hands to test any kind of hypothesis. I'm keen to just bust down any tinfoil hat theory thrown at me... my body is ready

Xtopher
11-12-2013, 07:33 PM
Check out this thread: http://community.wizards.com/forum/magic-online-general/threads/1025716

And that's the second or third one of these in the history of the forums. The others were just as long.

Seriously, though, as long as you're having fun, keep at it.

Gwaer
11-12-2013, 07:34 PM
The real issue here is that werlix hasn't proved anything. I trust that most people realize that his certainty is evidence that he shouldn't be listened to.

He keeps saying all past data points to everything being normal. But that's just not true. Two all resource hands in 500 hands is not normal or expected. We'd need to draw a lot more hands to see if that fades into the background or not. It's just a case of someone drawing conclusions on limited information. It's just as bad as the people saying something is broken.

Xenavire
11-12-2013, 08:16 PM
The real issue here is that werlix hasn't proved anything. I trust that most people realize that his certainty is evidence that he shouldn't be listened to.

He keeps saying all past data points to everything being normal. But that's just not true. Two all resource hands in 500 hands is not normal or expected. We'd need to draw a lot more hands to see if that fades into the background or not. It's just a case of someone drawing conclusions on limited information. It's just as bad as the people saying something is broken.

With a 24 resource deck I have had 3-4 no resource opening hands, with mulligans being almost as bad. I accept that I am probably just unlucky, but I have played less than 200 games at this point (to my knowledge). Now, logically, I shouldn't have many repeats of this for thousands of games, right?

I know probability hates me though. I don't feel that there is anything inherently wrong with the randomiser in my own experience, but the randomiser does seem to hate me :)


On your hands with no resources, did you have clumps of non-resource cards that would have been next to each other in your decklist? If not, what would cause the server load to randomize 60% of your deck, but not 100% of it?

Same question for those saying that the cards are "clumped", but not immediately next to each other (i.e. if I have two in my opening hand, then I'm more likely to draw a third in the first few turns, etc,) Server load may theoretically cause the deck not to be randomized, but how would it cause a "weak shuffle" where cards get broken up but not very far from each other? The deck is either randomized or it's not. Unless it's programmed really strangely (like simulating an actual shuffle and server load causing it to crash in the middle of the first riffle), then how does it wind up "slightly randomized"?

Live games - I would have been stupid to keep those hands, so I mulliganed. The mulligans were all 1 resource, or 2 resource (wrong colour, multi shard deck). I had to go down to 4, and one time versus the AI the hands were so unplayable I ended up with 1 card.

I don't know if there is any clumping. But with a decently constructed deck, I was getting a pretty bad run. Last 2 days, while the server load was high, compared to a very good run before and during the weekend. If you can make some sense of it, be my guest, but for now it is either a bad case of random chance, or a case of immense server load.

Xtopher
11-12-2013, 08:29 PM
The real issue here is that werlix hasn't proved anything. I trust that most people realize that his certainty is evidence that he shouldn't be listened to.

He keeps saying all past data points to everything being normal. But that's just not true. Two all resource hands in 500 hands is not normal or expected. We'd need to draw a lot more hands to see if that fades into the background or not. It's just a case of someone drawing conclusions on limited information. It's just as bad as the people saying something is broken.

Is this you being serious or more of your social experiment ? It's impossible for me to tell, 'cause it seems the same as what you were posting earlier.

No one on this forum will ever be able to prove the shuffler is sufficiently random. There will always be another case that wasn't considered that someone will want tested. Much easier to demonstrate it's not random, since you only need one statistically relevant example, that can then be repeated as often as necessary.

Werlix
11-12-2013, 08:38 PM
The real issue here is that werlix hasn't proved anything. I trust that most people realize that his certainty is evidence that he shouldn't be listened to.

He keeps saying all past data points to everything being normal. But that's just not true. Two all resource hands in 500 hands is not normal or expected. We'd need to draw a lot more hands to see if that fades into the background or not. It's just a case of someone drawing conclusions on limited information. It's just as bad as the people saying something is broken.

It is hard to take you seriously now that you've admitted you were making stuff up to try and prove a point or something but I'll bite.

Basically the null hypothesis here is that the shuffler is random, from all the data we've gathered nothing has pointed with any level of significance to proving it wrong. Your example of two 7x resource hands out of 500 isn't nearly enough proof. Of all the results that do have enough data points they all trend towards the expected values.

I'm currently still collating data that can add to previous results to add more certainty...

Werlix
11-12-2013, 08:43 PM
With a 24 resource deck I have had 3-4 no resource opening hands, with mulligans being almost as bad. I accept that I am probably just unlucky, but I have played less than 200 games at this point (to my knowledge). Now, logically, I shouldn't have many repeats of this for thousands of games, right?

I know probability hates me though. I don't feel that there is anything inherently wrong with the randomiser in my own experience, but the randomiser does seem to hate me :)

With 24 resources you're likely to hit a 0 resource hand every 46 or so hands. So if you've had ~160 games you could expect about 3 to 4 no resource hands. Seems you're bang on :)

Try this site for calculating hypergeometric calculations: http://stattrek.com/online-calculator/hypergeometric.aspx (eg pop size 60, successes 24, sample size 7, successes in sample 0)

noragar
11-12-2013, 08:52 PM
No one on this forum will ever be able to prove the shuffler is sufficiently random. There will always be another case that wasn't considered that someone will want tested. Much easier to demonstrate it's not random, since you only need one statistically relevant example, that can then be repeated as often as necessary.

Well actually, finding one statistically relevant example wouldn't be enough to prove it. If we're testing at a 95% significance level, then we'd expect that one out of twenty trials (5%) would come up significant as a false positive. So if a bunch of people keep doing tests, eventually a few are going to turn out significantly different from the expectation, but even that wouldn't necessarily prove anything.

It would take a higher percentage than that of tests coming out bad or a couple results that are so extreme so as to be beyond the range of possibility.

Xtopher
11-12-2013, 09:02 PM
Well actually, finding one statistically relevant example wouldn't be enough to prove it. If we're testing at a 95% significance level, then we'd expect that one out of twenty trials (5%) would come up significant as a false positive. So if a bunch of people keep doing tests, eventually a few are going to turn out significantly different from the expectation, but even that wouldn't necessarily prove anything.

It would take a higher percentage than that of tests coming out bad or a couple results that are so extreme so as to be beyond the range of possibility.
That's what I was referring to when I said the test could be repeated as often as necessary. Originally the sentence said "as often as necessary to convince everyone there's a problem.", but then for some reason I cut the end off of it.

The point is, the experiment has to be repeatable and give similar results each time it's attempted.

Gwaer
11-12-2013, 09:20 PM
I'm very serious when I say we don't have enough data. And people drawing conclusions that everything is fine from the data we do have are incredibly brazen to the point of ridiculousness. Anyone that will say everything is fine after two data points in a category... There are no words for how crazy that is.


And once again the alpha code that is randomizing the decks don't really matter, we don't know if it will change, and if it changes mid test that would give invalid results. In that light what's the point of doing any more conclusion drawing, or data gathering until a more polished state?

Niedar
11-12-2013, 09:31 PM
Here is 1,000 hands in the test draw window with 24 resources. Maybe I will go up to 5k tomorrow but lets be honest it will never be enough for some people.

https://www.dropbox.com/s/jiab9sxa60smg49/lands.csv

And the results:

0 Lands: 0.029000

1 Lands: 0.122000

2 Lands: 0.275000

3 Lands: 0.305000

4 Lands: 0.196000

5 Lands: 0.065000

6 Lands: 0.007000

7 Lands: 0.001000

Gwaer
11-12-2013, 09:41 PM
Niedar, you've never said or done a single useful thing in this thread or any other, you're right. I don't believe that you would spend any amount of time actually recording data or being useful. Maybe if you did it on a stream and I could watch you doing it. But I'd still probably be pretty certain it was trickery rather than you being constructive. But I'm sure someone will take your contributions, and draw conclusions from them.

Niedar
11-12-2013, 09:46 PM
You are right, I would not manually do it, I automated it and as predicted another excuse.

Gwaer
11-12-2013, 09:48 PM
How did you automate it, pray-tell? I'd be very interested in that.

Niedar
11-12-2013, 09:51 PM
I wrote a very simple python script using sikuli (http://www.sikuli.org/) the likes of which I am not going to be posting considering it can be modified for other purposes.

Gwaer
11-12-2013, 09:54 PM
I apologize, perhaps you've been useful after all.

Showsni
11-13-2013, 12:58 AM
I'm surprised it worked; from my tests, if you perform 250 or so random draws the program then always crashes. How did you get 1000 without a crash? Or did it drash 4 times but you just picked up and restarted?

Niedar
11-13-2013, 01:39 AM
Yeah, I originally left it on overnight a couple days ago and came back to it crashed so I just set it up to do sets of 200 and then restart the client.

Vorpal
11-13-2013, 11:06 AM
I don't personally have any reason to believe the shuffler is broken.

I simply take issue with much of the methodology attempting to prove it couldn't possibly be broken. Either the sample sizes are much too small, or the test rests on a series of unfounded assumptions and handwaving.

Trying to prove the in game deck shuffler isn't broken by demonstrating the 'draw a test hand' shuffler is not broken is a fools errand for anyone who knows anything about how computers handle random number generation. If there is a problem with the seed values they choose in a multiplayer game, there is no reason at all to suppose that would show up in either a single player game against the AI or in the test draws.

We simply don't know enough about how CZE has structured their process.

Werlix
11-13-2013, 12:44 PM
I don't personally have any reason to believe the shuffler is broken.

I simply take issue with much of the methodology attempting to prove it couldn't possibly be broken. Either the sample sizes are much too small, or the test rests on a series of unfounded assumptions and handwaving.

Trying to prove the in game deck shuffler isn't broken by demonstrating the 'draw a test hand' shuffler is not broken is a fools errand for anyone who knows anything about how computers handle random number generation. If there is a problem with the seed values they choose in a multiplayer game, there is no reason at all to suppose that would show up in either a single player game against the AI or in the test draws.

We simply don't know enough about how CZE has structured their process.

Of course we can't prove that "it couldn't possibly be broken" we'd have to spend eternity doing that.

The null hypothesis here is that the deck shuffler is fine. It's presented as random, we can assume that it's random unless someone has an objection. If someone has an objection we can test their hypothesis and see if it is true.

So far every hypothesis (that the shuffler isn't random) that we've gathered sufficient data on has proven to be false.

Of course anyone that still feels there's a problem can blame all sorts of temporary / server / atmospheric / cosmic issues that cause them to get 3 copies of a card but we can't really do much with that. We can however disprove simple hypotheses like "too many/not enough resources show up in the opening hand" just as Niedar has (provided the shuffler isn't different between the deck tester and in-game)

Werlix
11-13-2013, 12:45 PM
Here is 1,000 hands in the test draw window with 24 resources. Maybe I will go up to 5k tomorrow but lets be honest it will never be enough for some people.

https://www.dropbox.com/s/jiab9sxa60smg49/lands.csv

And the results:

0 Lands: 0.029000

1 Lands: 0.122000

2 Lands: 0.275000

3 Lands: 0.305000

4 Lands: 0.196000

5 Lands: 0.065000

6 Lands: 0.007000

7 Lands: 0.001000

Good work :) Those numbers appear to agree very closely with what would be expected.

Werlix
11-13-2013, 12:47 PM
It'd be great if a CZE engineer could take 5 seconds to confirm/deny in here whether the deck tester shuffler and in-game shufflers use the same randomisation process :)

Shaqattaq
11-13-2013, 12:49 PM
I can confirm that our engineers love this thread and have tested the randomization process.

Xenavire
11-13-2013, 12:55 PM
I can confirm that our engineers love this thread and have tested the randomization process.

Thanks Shaq :)

I did notice you didn't mention if it was broken or not, mr tricky hahahaha :D

Shaqattaq
11-13-2013, 01:00 PM
I think the data in this thread answers that question =)

Werlix
11-13-2013, 01:37 PM
I think the data in this thread answers that question =)

Thanks Shaq, made my day :)

DeusPhasmatis
11-13-2013, 03:26 PM
I can confirm that our engineers love this thread and have tested the randomization process.

<3

mudkip
11-13-2013, 07:54 PM
Here is 1,000 hands in the test draw window with 24 resources. Maybe I will go up to 5k tomorrow but lets be honest it will never be enough for some people.

https://www.dropbox.com/s/jiab9sxa60smg49/lands.csv

And the results:

0 Lands: 0.029000

1 Lands: 0.122000

2 Lands: 0.275000

3 Lands: 0.305000

4 Lands: 0.196000

5 Lands: 0.065000

6 Lands: 0.007000

7 Lands: 0.001000

Niedar, is there any way you can change the script to record every card? That would be really good to explain some of the other concerns in this thread.

Niedar
11-14-2013, 02:58 AM
It is possible but would require a little more work to do but I am not sure that there is a reason to since if there is a problem it would show up in the resource test as well. I can certainly do it, I just don't really think it adds anything. That is why I eliminated doing mulligans as well although I think testing mulligans actually is more useful.

The 5k dataset will have to wait until I feel like updating my script, with the patch today the way you get to the hand testing screen has changed.

Edit: I don't even see it as an option anymore, has it been removed?
Edit Again: I found it, it is only accessible through the bar plot icon on the right where your Champion is supposed to be. To CZE, this is a very unintuitive location for that icon.

Hatts
11-14-2013, 08:30 AM
I think the data in this thread answers that question =)

/cheer

Statistics wins the day again! If only this could be the end of it. Good job Werlix, Niedar and others who've been fighting the good fight against the ignorant :)

Storm_Fireblade
11-14-2013, 08:35 AM
Personally I'm not that worried about the ressource number rather than the general "grouping" of cards, which happens way too often and sort of makes it feel like there is something wrong with the shuffler/ressource distribution. I've had hands like this one here 4-5 times now and I played a few dozen games, maybe slightly above 100.

http://www.simply-gaming.com/opening_hand.jpg

I feel like having seen this myself a few times already is way more, than it statistically should happen. Maybe I'm just that unlucky at the moment, but I don't know.

Werlix
11-14-2013, 01:35 PM
...general "grouping"...happens way too often...makes it feel like...something wrong...4-5 times now...played a few dozen games...I feel like...

Hang on... are we still talking about this?

Pretty sure a CZE engineer posted 1 page ago confirming that they've tested the randomisation process and our data showing correct distribution of cards aligns with their testing.

And here I was resting upon my mountain of vindication...

As a side note I hope you kept that hand, it looks sweet :)

Storm_Fireblade
11-14-2013, 02:16 PM
Hang on... are we still talking about this?

Pretty sure a CZE engineer posted 1 page ago confirming that they've tested the randomisation process and our data showing correct distribution of cards aligns with their testing.

And here I was resting upon my mountain of vindication...

As a side note I hope you kept that hand, it looks sweet :)

I kept it and won actually, yes. But only because I was lucky enough not to see any heat wave from my ruby/wild playing opponent :) That aside, Its very impolite to quote someone and delete part of the text to rephrase the whole context. I only wanted to show a picture of what I actually meant 30 pages ago. It never felt weird to get screwed or flooded, since thats part of the game. What felt odd, was HOW that happenend. And wouldn't you agree, that seeing opening hands like this 4-5 times by now, even though I only played maybe a 100 games, is way too often?

Only getting 1 ressource is one thing. But I'm seeing 3-4 copies of one card way too often. What has been tested here is how the ressource distribution is handled. And that might be fine, yeah. But the data in this thread isn't saying anything about the possibility of some form for "grouping problem".

Werlix
11-14-2013, 02:46 PM
I kept it and won actually, yes. But only because I was lucky enough not to see any heat wave from my ruby/wild playing opponent :) That aside, Its very impolite to quote someone and delete part of the text to rephrase the whole context. I only wanted to show a picture of what I actually meant 30 pages ago. It never felt weird to get screwed or flooded, since thats part of the game. What felt odd, was HOW that happenend. And wouldn't you agree, that seeing opening hands like this 4-5 times by now, even though I only played maybe a 100 games, is way too often?

Only getting 1 ressource is one thing. But I'm seeing 3-4 copies of one card way too often. What has been tested here is how the ressource distribution is handled. And that might be fine, yeah. But the data in this thread isn't saying anything about the possibility of some form for "grouping problem".

Sorry for the partial quote, just getting a bit tired of presenting research and data then having people talk about how they just "feel" something is wrong.

The problem here is your phrase "hands like this". This suggests you are identifying patterns after you've seen them which is a bad way to go about it. Now that you suspect something is wrong you should define an exact pattern, determine the expected probability of the pattern, then test the observed occurrence of that pattern over many samples. Eg probability of getting a 4-of any of the cards in my deck in my opening hand.

And also our previous tests are relevant to "grouping" concerns. The tests on resource counts would have been affected if cards were more likely to "group" together. Also I did a simple "grouping" test earlier in the thread, quoted below:


So in my tests I just test one thing: are the first two cards in my hand the same card? (I also mix it up and sometimes look at the last two instead, just in case) This is a simple way to test one variable over and over and get some reliable results on how random the randomiser is.

Given true randomness the probability of the first two cards being the same should be 3/59, am I right? This is 0.0508474576271186.

Currently my results are:

Hands drawn......: 520
Duplicates.......: 27

Expected result..: 0.0508474576271186
Actual result....: 0.0519230769230769

Regardless of all this talk, a CZE engineer has confirmed that they've tested the randomiser in Hex and has said that our data aligns with their research. So if there's nothing wrong with the randomiser then there's nothing wrong with "grouping".

What may be possible here is that real life shuffling methods may make "grouping" less common (due to weaving techniques) than should actually happen with true randomness. This would be something interesting to research...

Storm_Fireblade
11-14-2013, 02:59 PM
I actually overlooked your "grouping test" so thanks a lot for that. I still feel, no, it still is statisitically odd, that I've seen similar hands (with 3-4 copies of the same card) a handful of times during my games up to now. I am fully aware though, that those dozens of games I played aren't enough to be representative. People can sometimes get lucky, unlucky, whatever you will call it.

Since we have a few numbers here, that state everything should be fine, I'll simply leave it at that and write down for every game I play how many ressources and duplicate cards I had in my hand(s). That way I'll either come up with enough data to voice some concerns or I'll have some data to reassure myself, that even though I probably will remember all those times I got screwed/flooded much better, it still isn't happening too often.

Niedar
11-14-2013, 03:01 PM
You better be playing with the same deck then or all your data will be contaminated.

Creepshow
11-14-2013, 06:34 PM
personally, I find the ratio to be comparable to live TCGs. Ive played TCGs for almost two decades now and Ive almost allways stuck with a (1:3 resource to non-resource) structure. I have not had too many issues where HEX is concerned, and im still using the same ratio

DeusPhasmatis
11-14-2013, 07:06 PM
personally, I find the ratio to be comparable to live TCGs. Ive played TCGs for almost two decades now and Ive almost allways stuck with a (1:3 resource to non-resource) structure. I have not had too many issues where HEX is concerned, and im still using the same ratio

1:3 is 25%, or 15 resources in a 60 card deck. Perhaps you meant 1:2 (20 resources), or 2:3 (24 resources)?

mudkip
11-14-2013, 07:31 PM
It is possible but would require a little more work to do but I am not sure that there is a reason to since if there is a problem it would show up in the resource test as well. I can certainly do it, I just don't really think it adds anything. That is why I eliminated doing mulligans as well although I think testing mulligans actually is more useful.

The 5k dataset will have to wait until I feel like updating my script, with the patch today the way you get to the hand testing screen has changed.

Edit: I don't even see it as an option anymore, has it been removed?
Edit Again: I found it, it is only accessible through the bar plot icon on the right where your Champion is supposed to be. To CZE, this is a very unintuitive location for that icon.

See the previous few posts ;)

If we get a big batch of data from someone for all of their draws, people can process the data to form their own conclusions.

I could easily process it to debunk the grouping thing people keep talking about. In fact it would be fun :)

Gwaer
11-14-2013, 07:44 PM
You're always jumping to conclusions on limited data werlix, Will isn't an engineer, he's a product manager and community coordinator guy. Who said that the CZE engineers love this thread, furthermore there easily could be a problem between the randomizer and the user, cards do not just magically appear after a randomizer runs. I don't see the harm in people testing and collating data.

Werlix
11-14-2013, 08:08 PM
You're always jumping to conclusions on limited data werlix, Will isn't an engineer, he's a product manager and community coordinator guy. Who said that the CZE engineers love this thread

Ok I didn't know what job title Shaq has but he posted on behalf on the engineers.


furthermore there easily could be a problem between the randomizer and the user, cards do not just magically appear after a randomizer runs.

No not magically - they are drawn from the top of the randomised deck. If there's a problem with grouping then there's a problem with the randomiser. If there's no problem with the randomiser then there's no problem with grouping.


I don't see the harm in people testing and collating data.

Neither! I've been the most vocal about both collecting data and encouraging others to do so. We still need to find this elusive data that prooves there's a problem with the draws :)

Xtopher
11-15-2013, 01:36 AM
Those numbers are actually quite far off where they should be, we need a lot more tests to be sure though, check back in after you hit 1,000.
This is the post where all the issues people are having stem from. The numbers were not "far off" statistically speaking, as has been demonstrated repeatedly.

Have we collected enough data to say with 99% certainty that the shuffler is in perfect working order? NO. OTOH, statistical analysis of the data collected has so far shown there is NOT a problem.

Maybe when you posted that, it was your social experiment stage.

Gwaer
11-15-2013, 03:05 AM
Nah, the high end numbers were far off. We shouldn't have seen 2 hands of all resources in such a small window, and you really should just never ever draw conclusions from data before you have at least 5 hits of every possible outcome. Thats where I got annoyed at people declaring any conclusion for or against. There just wasn't enough to go on to make assertions like that.


My social experiment is less about riling people up and more about creating an opening to correct me, so that at least I would know that person knew what they were talking about.

Eierdotter
11-15-2013, 03:21 AM
what a long thread... with even Shag trolling in some posts.

so far what i got out of this thread is:

the statistical distribution overall works as intended, for example with 24 resource, most games you get 2-5 resource, and very rare you get 0,1,6,7 resource.
But the RNG-generator seems to need some work, since a lot of times you get sort of the same hands or several copies of the same card.

So one group says "A works, it is fine" the other group says " B is broken, needs to be fixed"
good job fighting over this for 30 pages.

Gwaer
11-15-2013, 03:32 AM
So one group says "A works, it is fine" the other group says " B is broken, needs to be fixed"
good job fighting over this for 30 pages.

Your reading comprehension is very bad. I don't think anyone is saying it is broken. Several people have popped in saying it feels off and offering anecdotal evidence. But I don't think anyone is taking that as proof there is a problem.

Group A, has been saying collect more data.
Group B has been saying everything is fine.

I'm firmly in group A. Honestly I hope to be able to plop down 10,000 hands and say everything is falling into acceptable levels and all is well. There can be no downside from doing the work. Either everything is fine and we can all point to it and say so, or there's a problem that can be corrected before release. *shrug* I just like the numbers and it gives me something to do when I'm not busy elsewhere.

Eierdotter
11-15-2013, 04:08 AM
seems i was not super clear to understand, (lazy me trying to write as less as possible^^)

with A i meant the statistical resource distribution in the starting hand - also referred as resource flood/screw -
after seeing the huge samplesizes that devoted people posted (500 samples seem small in the big picture but is a lot for a single person to gather). I get the "we need more data"-point, but to me it looks like it will just blend more into the expected statistic, beeing 1 or 2 % off seems fine with the current samples to me. And as you -Gwaer- mentioned, there is no "enough data".

with B i mean the card distribution, seeing stuff like
111222122133422210000100010003040011002220 resource numbers each hand. (i typed in random numbers here)
seems clumped and indicates, that for several hands the same RNG number seed was chosen from the system.
this point can not be confirmed by anyone who has no insight view to the system. but some people refer to this as beeing broken.

To be fair point B is not the topic.

So the first guy says it is broken (means point B)
Another guy says it works fine (means point A)
Gwaer stops by and says "we need more data" btw 2.5k posts #YOLO! (works for point A and B)

so thats how i see this discussion going

Gwaer
11-15-2013, 04:20 AM
If something that should be happening 0.000X percent of the time were happening 1.000X percent of the time. I would not think that's okay. =P but I do get your point. For people not interested in this stuff it probably looks silly, it's the best kind of arguing in my opinion. At least there is a verifiable truth in here somewhere. If we look hard enough.

Storm_Fireblade
11-15-2013, 04:25 AM
Just to be clear here...what would benefit the data the most?

If I was to make a deck with 24 ressources and use the test draw function in the deck builder, while writing down the number of ressources in the opening hand and, lets say, the first two mulligans. Furthermore I would write down any time I got duplicate cards or odd ressource distributions (f.x. yesterday I had 5 blood ressources in the opening hand. Which is fine number wise, but if I play 9 blood and 15 wild it feels weird again).

Would that help? I don't mind spending some time to collect some data honestly. Just want to make sure its the right format :)

Eierdotter
11-15-2013, 04:34 AM
If something that should be happening 0.000X percent of the time were happening 1.000X percent of the time. I would not think that's okay. =P but I do get your point. For people not interested in this stuff it probably looks silly, it's the best kind of arguing in my opinion. At least there is a verifiable truth in here somewhere. If we look hard enough.

exactly this "it should happen 0.00X % but happens 1.00X %" that is a "not enough data" for me

having the extreme case happen with a small sample makes it a huge difference, but with enough sample it usually evens out (but when is the "enough" point reached)
playing poker and getting 2 royal flush in 10 games seems broken, not getting one the next months evens it out.

@Storm_Fireblade:
i think we are not at a level to create a viable sample for 2 shard colors with different amounts of resources. the current samples are 24 resource and 36 other cards.
and if you collect data it needs to be all the data, not something like "i got 23 hands with only 1 resource over 200 hands"

Storm_Fireblade
11-15-2013, 06:08 AM
I made about 50 testdraws now and wrote down the ressources and any multiple cards (ressources at 4+, basic cards at 2+):

http://www.simply-gaming.com/testdraws.jpg

Link to the image (http://www.simply-gaming.com/testdraws.jpg)

Now, let me start by saying two things:

1) I know, that 50 draws aren't enough for any statistically proof.
2) I don't mind to go through a lot more draws, if it might help.

But since those 50 draws pretty much showed a similar pattern to what I "felt" before, lets say for a second, that 50 draws are enough to make a first conclusion. Now, could someone talented, when it comes to statistics, answer me the following question please: "Out of ~50 testdraws, did I see way too many hands with 3 copies or does the math proof, that this kind of frequency actually is totally fine?"

I'm asking because the number of times I saw 3 copies here is pretty similar to what I felt while just playing games. And I felt, that it happened way too often. If on the other hand this frequency actually is close to what it should be, then I could tell myself, that there simply is something wrong with my perception here :)

LLCoolDave
11-15-2013, 07:19 AM
Odds of your starting hand having multiple copies of a card occurrences happening. Assuming your deck is 24 resources 36 other cards with those consisting of 9 cards with 4 copies each. (call them orc, dwarf, elf, human etc)
2+ of a non-resource 44.44%
3+ of a non-resource 3.43%

Having multiple copies of a non-resource card in your starting hand is not particularly rare. The reason is that while having 3+orcs is only .39% (hypergeometric distribution) having 3+orcs or 3+humans is .77% and having 3+ orcs or 3+ dwarfs or 3+ elfs or 3+humans etc becomes reasonably likely.

These are numbers from earlier in the topic. I don't have access to a good statistics software right now to do more analysis of your data, but at a first glance, it looks pretty reasonable still. It seems your counts are slightly over, but probably still within a reasonable confidence level given the very small sample size. If anything, I would love you to use separate columns for 7 card, 6 card and 5 card hand results because the way you present the data is CLEARLY framed to make it look like there are more anomalous (for whatever that means) hands than there should be by marking any occurrence of a noteworthy hand within the 3 mulligans per hand in the first column.

A rough back of the envelope sketch shows that getting 3 or more triplets in a 50 card sample should happen about 20% of the time, so it's really not that unusual. Somebody who has more time and more experience in this subject could probably provide some proper math on this, especially concerning the variance involved in a multivariate hypergeometric experiment.

If anything, this is the first data set shown in this entire topic that makes me feel that some further investigation might be warranted.

mudkip
11-15-2013, 07:56 AM
I just made up an excel sheet to simulate a draw (24 resource, 9 lots of 4*Cards) and from 10,000 samples I have these results:


Total >=2 >=3 =4
%age 100.00% 56.82% 7.47% 0.44%
Count 10000 5682 747 44


I've F9'd a few times and the 2+ draw is consistently 55-57%

I'd love some live data to compare with.

LLCoolDave
11-15-2013, 08:03 AM
Are you sure that your simulation accounts for cards being removed form the deck for each subsequent draw? I.e. that after you have already drawn a Dwarfen Statistical Example there are only 3 of them left in the deck to draw in subsequent draws for the hand. Because your numbers seem a bit high to me, especially compared to the numbers from earlier in the topic, so I'd just like to make sure there isn't a simple mistake in your simulation. If your numbers are correct and the previous calculation was not then that 50 hand sample would line up very well.

mudkip
11-15-2013, 08:13 AM
Are you sure that your simulation accounts for cards being removed form the deck for each subsequent draw? I.e. that after you have already drawn a Dwarfen Statistical Example there are only 3 of them left in the deck to draw in subsequent draws for the hand. Because your numbers seem a bit high to me, especially compared to the numbers from earlier in the topic, so I'd just like to make sure there isn't a simple mistake in your simulation. If your numbers are correct and the previous calculation was not then that 50 hand sample would line up very well.

Yeah, it took a bit of fiddling. I made a function that makes a random integer (1-60), but it re-rolls if that number is taken previously. Then each number goes to a Vlookup.

To test if it's accurate I counted the resources:


Resource 0 1 2 3 4 5 6 7
%age 2.70% 12.96% 25.43% 29.86% 19.83% 7.68% 1.47% 0.07%
10000 270 1296 2543 2986 1983 768 147 7


Is that close to being the agreed percentages? I have to go to work and can't find out. I'll attach my sheet if it's accurate so others can play.

BossHoss
11-15-2013, 08:59 AM
Here is a quick Excel shuffler I whipped up right now for those interested:
- Select Cell A1
- Enter the Fx as:
=RAND()
- Click and Drag bottom right cell corner extending down to 60
You should now have 60 different random numbers

- Select Cell B1
- Enter the number 1
- Select B2
- Enter the number 2
- Select both cells then Click and Drag bottom right cell corner extending down to 60
You should now have 60 sequential numbers labeling the random numbers to their left

- Select Cell C1
- Enter the Fx as:
=LARGE($A$1:$A$60,B1)
- Click and Drag bottom right cell corner extending down to 60
You should now have all the random numbers from column A rearranged into descending order

- Select Cell D1
- Enter the Fx as:
=MATCH(C1,$A$1:$A$60,0)
- Click and Drag bottom right cell corner extending down to 60
You should now have a complete deck of cards with a random shuffle in column D

D1-D7 should represent your hand and f9 will re-shuffle

Shaqattaq
11-15-2013, 10:09 AM
I appreciate the collection of data in this thread. In general, discussion on this matter suffers due to the participants being a self-selective group. Those experiencing statistical anomalies-- which is inevitable considering the number of hands being dealt-- are going to be driven to discuss their experiences whereas those who are not generally do not enter the discussion.

Storm_Fireblade
11-15-2013, 10:16 AM
I appreciate the collection of data in this thread. In general, discussion on this matter suffers due to the participants being a self-selective group. Those experiencing statistical anomalies-- which is inevitable considering the number of hands being dealt-- are going to be driven to discuss their experiences whereas those who are not generally do not enter the discussion.

As long as we treat the data objectively and try to include all informations rather than those supporting a certain feeling or experience, it shouldn't matter, that this is sort of a self-selevtive group discussing here :) I'll try to come up with a few hundred, maybe even 1.000 testdraws myself during the weekend, keeping my current formating. Maybe that will be enough to either confirm, that more research would be useful or that everything is fine and it must be my own perception, which is wrong here :)

mudkip
11-15-2013, 10:28 AM
... whereas those who are not generally do not enter the discussion.

I'm trying to be involved and I've consistently argued that everything's fine.

Shaqattaq
11-15-2013, 12:23 PM
I'm trying to be involved and I've consistently argued that everything's fine.

Generally, players just keep playing the game and go about their routine. Sure, there are people who engage the community in discussion about how everything's ok. Overall, though, it's important to have that data so make sure we're all working off the same information and drawing conclusions from a solid set of numbers.

mudkip
11-15-2013, 03:21 PM
Okay, I have my Excel simulator corrected! For my random generator, I was comparing the drawn number to the name of the previous drawn card (I.e. my generator was saying "3" =/= "Resource" so it wasn't re-rolling). LLCoolDave was right :)

The resources are properly lined up with the expected percentages now, so I've worked out the chance of drawing multiples. The chance of drawing all 4 in the opening hand is incredibly small: Under 0.1%, ie less than 1 in 1000.

I'd seriously love it if someone could provide some live data!


Calibration:
Resource count 0 1 2 3 4 5 6 7
Expected 2.16% 12.10% 26.94% 30.87% 19.64% 6.93% 1.25% 0.09%
%age 100.00% 2.09% 11.81% 27.51% 30.83% 19.35% 6.84% 1.49% 0.09%
Count 8000 167 945 2201 2466 1548 547 119 7


Count of multiple cards (Except Resources) - 60 Cards, 24 Resource, 9 of 4*Cards

Total >=2 >=3 =4
%age 48.64% 3.48% 0.08%
Count 8000 3891 278 6

Storm_Fireblade
11-15-2013, 04:55 PM
I've been increasing my testdraws to 150 and changed the formatting a little bit. You can find the file containing everything HERE (http://www.simply-gaming.com/testdraws.ods).

Out of 150 testdraws (always mulligan down to 6 and 5) I did came across 3 copies of a single card that often:

At 7 Cards: 6
At 6 Cards: 7
At 5 Cards: 3

To me that feels like it is way too often, but as I mentioned before, that might only be my perception thats wrong here. Not sure, what an accurate calculation would be compared to this frequency.

Out of the same 150 testdraws I came across 4 Ressources of one shard, while having 12 of both, that often:

At 7 Cards: 7
At 6 Cards: 2
At 5 Cards: 2

Maybe someone can say something about, whether or not I did encounter 4 ressources of the same shard or 3 copies of the same card too often or not.

I'll add more games tomorrow, since 150 isn't that much yet.

mudkip
11-15-2013, 05:27 PM
From my experiment before, 6/150 or 4% is very close to the expected amount.

Storm_Fireblade
11-15-2013, 05:42 PM
From my experiment before, 6/150 or 4% is very close to the expected amount.

I'll add some more draws tomorrow to see, if anything major changes. Aside from that, if your calculations are correct and ~3,5% of all opening draws are expected to have 3 copies of a single card, then that already confirms, that my perception here is way off. I feel that 6 out of 150 opening hands with 3 copies of the same card is way too much and maybe thats the problem really.

Regardless of that, I'll add more draws during the weekend anyway, to get a better set of data. It can't hurt :)

MuffLord4
11-16-2013, 04:23 AM
What's the TLDR of this thread?

Cards are "stuck" together imo.

Malakili
11-16-2013, 06:05 AM
What's the TLDR of this thread?



The Hex playerbase, like most of the human species, is really bad at statistics.

ursa23
11-16-2013, 06:11 AM
The Hex playerbase, like most of the human species, is really bad at statistics.

+1. Everyone here should read "The Drunkard's Walk."

mudkip
11-16-2013, 10:07 AM
What's the TLDR of this thread?

"Cards are "stuck" together imo. "

Yep. That there's clustering of draws.


Edit: Attached is the excel drawing simulator I was using. Drag down Row 18 to get more samples (I trimmed it down for file space)

You could use it for mulligans too by clearing Column N, then Column M, etc.

1288

McKizz
11-16-2013, 05:11 PM
I'm still running into HUGE clumping issues. I get a 3-of nearly every time I draw.

Deadhorn
11-16-2013, 09:44 PM
As a student of statistics and also someone who is constantly frustrated by the RNG in this game I have tried to reconcile what I know with what I'm experiencing. Here is what my conclusion is:

These problems seem worse than they are because matches can end incredibly quickly when your opponent is playing a fast deck (MOST decks). There is therefore less time over the course of a match for reversion to the mean. This is the same reason football is a very difficult sport to bet on compared to, say, basketball. There are very few iterations of key events in a football game, whereas basketball has countless repetitions within a single game, and more games to smooth out curves.

This is a serious design issue the developers need to consider. If they are confident their RNG is working correctly - and it is likely it is - they need to consider how to design around the fact that a bad draw can easily decide a match with the current game speed.

Storm_Fireblade
11-17-2013, 02:46 AM
I did increase my data from 150 to 250 draws (going to add more today). Just wanted to share the results so far. You can still find the file --> HERE (www.simply-gaming.com/testdraws.ods) <--.

Out of 250 testdraws (always mulligan down to 6 and 5) I did came across 3 copies of a single card that often:

At 7 Cards: 9 => 3,6%
At 6 Cards: 9 => 3,6%
At 5 Cards: 6 => 2,4%


IF mudkips calculations --> HERE <-- are correct (I'm not saying they aren't. I'm just not that into statistics, that I'm able to say right now, they are correct), then my live data (testdraw function inside the deck builder) confirms so far, that at least in the opening hand everything is fine. I'll try get to 1.000 draws as soon as possible. If the data stays that way, I'm fine putting the topic to rest and happy, that it only might be my perception that is off here.


Out of the same 250 testdraws I came across 4+ Ressources of one shard, while having 12 of both, that often:

At 7 Cards: 13 => 5,2%
At 6 Cards: 7 => 2,8%
At 5 Cards: 2 => 0,8%

Ressources in general (ignoring the shards)

Out of the same 250 testdraws I came across 5 Ressources (didnt see 6 or 7 yet) that often:

At 7 Cards: 17 => 6,8%
At 6 Cards: 9 => 3,6%
At 5 Cards: 1 => 0,4%

Out of the same 250 testdraws I came across 0 Ressources that often:

At 7 Cards: 2 => 0,8%
At 6 Cards: 14 => 5,6%
At 5 Cards: 19 => 7,6%

While the opening hands with 6,8% (5 ressources) and 0,8% (0 ressources) look absolutely ok to me, I get the feeling that experiencing 0 ressources in a 24 shards deck 5,6% when mulligan to 6 and 7,6% when mulligan to 5 is way too much. But again, thats just a feeling I gladly put to rest, if someone could share any calculations :)

Maybe mudkip could add the numbers for 6 and 5 cards to his calculations? :)

FlyingMeatchip
11-17-2013, 03:51 AM
How bout both players start with 2 resources and draw the remaining 5 cards at the start of the game.

Willd
11-17-2013, 03:53 AM
I did increase my data from 150 to 250 draws (going to add more today). Just wanted to share the results so far. You can still find the file --> HERE (www.simply-gaming.com/testdraws.ods) <--.

Out of 250 testdraws (always mulligan down to 6 and 5) I did came across 3 copies of a single card that often:

At 7 Cards: 9 => 3,6%
At 6 Cards: 9 => 3,6%
At 5 Cards: 6 => 2,4%

IF mudkips calculations --> HERE <-- are correct (I'm not saying they aren't. I'm just not that into statistics, that I'm able to say right now, they are correct), then my live data (testdraw function inside the deck builder) confirms so far, that at least in the opening hand everything is fine. I'll try get to 1.000 draws as soon as possible. If the data stays that way, I'm fine putting the topic to rest and happy, that it only might be my perception that is off here.


For 7 cards this is more or less exactly what you'd expect (the numbers from earlier are correct). It's a bit high for 6 and 5 card hands.


Out of the same 250 testdraws I came across 4+ Ressources of one shard, while having 12 of both, that often:

At 7 Cards: 13 => 5,2%
At 6 Cards: 7 => 2,8%
At 5 Cards: 2 => 0,8%

This is a pretty close to exactly correct in all three cases (expected are 5%, 2.4%, 0.8%),


Ressources in general (ignoring the shards)

Out of the same 250 testdraws I came across 5 Ressources (didnt see 6 or 7 yet) that often:

At 7 Cards: 17 => 6,8%
At 6 Cards: 9 => 3,6%
At 5 Cards: 1 => 0,4%

Expected percentages of exactly 5 resources are 6.9%, 3.1%, 0.8%.


Out of the same 250 testdraws I came across 0 Ressources that often:

At 7 Cards: 2 => 0,8%
At 6 Cards: 14 => 5,6%
At 5 Cards: 19 => 7,6%

While the opening hands with 6,8% (5 ressources) and 0,8% (0 ressources) look absolutely ok to me, I get the feeling that experiencing 0 ressources in a 24 shards deck 5,6% when mulligan to 6 and 7,6% when mulligan to 5 is way too much. But again, thats just a feeling I gladly put to rest, if someone could share any calculations :)

Maybe mudkip could add the numbers for 6 and 5 cards to his calculations? :)

Expected percentages of 0 resources are 2.2%, 3.9%, 6.9%.

I used http://deckulator.appspot.com/ to get all the numbers here. It looks to me like pretty much everything is very close to expected. The only exception is the 0 resource hands, but given the sample size and the low likelihood of "successes" it's probably more surprising that everything else is so close than that there is one that is an outlier.

Storm_Fireblade
11-17-2013, 04:12 AM
Thank you very much Willd. I really appreciate it. My dataset is already been increased to 350. I'll go up to 500 during the next hours and share the results again. Hopefully we'll see it getting even closer to the expected results. I'm more than happy to find out, that everything is fine here :)

Storm_Fireblade
11-17-2013, 07:17 AM
I did increase my data from 250 to 500 draws Here are the results. You can still find the file --> HERE (www.simply-gaming.com/testdraws.ods) <--.

Out of 500 testdraws (always mulligan down to 6 and 5) I did came across 3 copies of a single card that often:

At 7 Cards: 16 => 3,2%
At 6 Cards: 17 => 3,4%
At 5 Cards: 7 => 1,4%

One single time I did see 4 copies of a single card (at 5 cards). But thats after all just something very rare, not entirely impossible.

Out of the same 500 testdraws I came across 4+ Ressources of one shard, while having 12 of both, that often:

At 7 Cards: 16 => 3,2%
At 6 Cards: 15 => 3,0%
At 5 Cards: 4 => 0,8%

Ressources in general (ignoring the shards)

Out of the same 500 testdraws I came across 5 Ressources that often:

At 7 Cards: 42 => 8,4%
At 6 Cards: 17 => 3,4%
At 5 Cards: 3 => 0,6%

I saw 6 ressources 3 times (at 7 cards). Same as 4 copies, very rare but not entirely impossible. So I'm fine with seeing that once in a while.

Out of the same 500 testdraws I came across 0 Ressources that often:

At 7 Cards: 6 => 1,2%
At 6 Cards: 22 => 4,4%
At 5 Cards: 32 => 6,4%


I can see some numbers that are not completely as expected, but thats no surprise, given the dataset only contains 500 draws. What I do conclude out of these 500 draws (at least for myself) is though, that the difference between the data and the expected numbers isn't off by that much and probably a result of the limited testdata. So unless someones uses my 500 draws to come up with any conclusions, where we need to check something out further after all, I'll stop here being fine with the fact, that the randomizer seems fine overall.

Vorpal
11-17-2013, 01:23 PM
How bout both players start with 2 resources and draw the remaining 5 cards at the start of the game.

The problem with this I could give my deck, say, 2 resources, and then a billion 1 cost and 2 cost cards and just probably run right over the other guy, if roughly half his draws are resources and none of mine are.

dachewster99
11-20-2013, 03:26 AM
I will say that the shuffle isnt good in general ATM. I will draw the EXACT same troops after like 4 mulligans, or the same thing with resources.

FlyingMeatchip
11-20-2013, 03:31 AM
The problem with this I could give my deck, say, 2 resources, and then a billion 1 cost and 2 cost cards and just probably run right over the other guy, if roughly half his draws are resources and none of mine are.

That is a good point. It would allow an unfair advantage if a deck is all low cost.

FlyingMeatchip
11-20-2013, 03:32 AM
There can still be a minimum resource requirement in the decks....say 20.

Storm_Fireblade
11-20-2013, 03:56 AM
I will say that the shuffle isnt good in general ATM. I will draw the EXACT same troops after like 4 mulligans, or the same thing with resources.

1: How often did you really encounter that?
2: If you mulligan to 6, every one of these six draws still has a chance of 4/60 to draw a certain troop, if you play a staple of them. So obviously you might mulligan into the "same" troops once in a while. Especially since you probably even did draw into a different copy, than before, which could as well have been a different card alltogether.

joseph5185
11-22-2013, 05:49 AM
The problem with this I could give my deck, say, 2 resources, and then a billion 1 cost and 2 cost cards and just probably run right over the other guy, if roughly half his draws are resources and none of mine are.

lol...a billion? really? .. =P

noragar
11-22-2013, 10:17 AM
lol...a billion? really? .. =P

Not in Set 1 since there aren't that many 1 and 2 drops yet and you can still only put 4 of each card in your deck. But once they get up to 250 million weenies released, go ahead and put them all in since they're all pretty much interchangeable anyway. Then LOL when you get matched up against a mill deck.

Tinuvas
11-23-2013, 12:02 PM
...Especially since you probably even did draw into a different copy, than before, which could as well have been a different card alltogether.

This could just be my perception of things, but when I mulligan, sometimes a couple of cards actually stay in my hand. After that the rest of my hand is drawn, which may or may not include copies of said cards. I have a theory on WHY that happens, namely that some cards are actually the previous cards that would be in your hand anyway. For example, I have 4 bottled vitae, copy A,B,C, and D. First draw has copies A and B in it. Mulligan. Now I get copies A and C. WHEN I'm mulliganning, copy A will actually 'stay' in my hand and not draw like copy C will. Completely off topic and probably irrelevant, but I think it's a neat feature of the digital format if it's true. That and we'll be more able to see this happen when we have extended art/alternate art/foil features on the cards.

If this WAS true, and confirmable, you could bounce calculations off of that tidbit all day long too. I hope I didn't open another can of worms.

mudkip
11-23-2013, 12:22 PM
I will say that the shuffle isnt good in general ATM. I will draw the EXACT same troops after like 4 mulligans, or the same thing with resources.

You need some data to make a claim like that. Anecdotes aren't evidence.

Storm_Fireblade
11-23-2013, 12:47 PM
You need some data to make a claim like that. Anecdotes aren't evidence.

Excactly. I would even have bet money, that the randomizer isn't workling as intended and that I do encounter way too many duplicated cards in one hand. After doing some research I'm pretty sure, that this is just a personal feeling, that has nothing to do with the reality.

mmt260
11-23-2013, 01:41 PM
lol cant believe ive found a thread already where people are bitching about the randomiser not working. Same thing happens in magic and poker(ppl bitching), and all I can say to you is its a lot more accurate than real life.

Storm_Fireblade
11-23-2013, 01:53 PM
To be fair mmt260, people here weren't only "bitching" but did actually put a lot of efforts into actually testing the randomizer to make sure it isn't bugged. What if it were? Would you prefer to simply live with it without anyone voicing their concerns? Everything seems fine, but it as lot easier to accept that, if you did some tests, rather than just saying "how could this company make any mistakes"? :)

VoidInsanity
11-23-2013, 03:56 PM
In the recent patch I have noticed things to be a lot less "broken" in terms of card generation for the most part, the card shards of fate has allowed me to test how well a deck is shuffled (10 of each + 8 shards of fate + 2 random cards). Since every card I own is mana and it shows every mana card in my deck in order from top-bottom, I can check to see how well its shuffled. No obvious flaws with it in the few tests I ran.

I hope to get someone to do the same test with me a few times using Curse of Oblivion instead for more accurate results.

Storm_Fireblade
11-23-2013, 04:06 PM
You did play 8 Shards of Fate? Thats broken already! :)

VoidInsanity
11-23-2013, 05:51 PM
You did play 8 Shards of Fate? Thats broken already! :)

Its a Resource card with a significant downside to using it, limiting it to 4 in its current form where its 0/1 without a charge makes zero sense.

Storm_Fireblade
11-23-2013, 05:56 PM
Any form for treshold fixing should be limited. Shards of Fate is a 4 copy per deck card. Anything else dosn't make sense in my opinion! :)

VoidInsanity
11-23-2013, 10:32 PM
As of todays patch things have gone back to as they were it seems, getting a lot of clumps of cards again. This not happen as much in the patch before todays one.

DeusPhasmatis
11-23-2013, 10:59 PM
As of todays patch things have gone back to as they were it seems, getting a lot of clumps of cards again. This not happen as much in the patch before todays one.

Onyxia Deep Breathes more!

jgsugden
11-23-2013, 11:44 PM
You may all be arguing over nothing as the randomization may be a placeholder.

Gwaer
11-24-2013, 12:36 AM
It's funny how people come in here thinking there's an argument going on.

mudkip
11-24-2013, 10:05 AM
You may all be arguing over nothing as the randomization may be a placeholder.
Very unlikely. http://forums.cryptozoic.com/showthread.php?t=29508&p=317530&viewfull=1#post317530


It's funny how people come in here thinking there's an argument going on.
Yes it is. But the thread has to stick around, otherwise we will have many more threads on the subject.

As soon as combat logs come in, this whole thing will be debunked, as we will be able to program a way to quickly tell people they're imagining things.

McKizz
12-05-2013, 09:27 PM
I don't buy it. My personal experience is that I get screwed more times than not. I shouldn't have to increase from 24 to 26 resources just to appease the RNG gods.

Is the "randomizer" using static objects to "randomize" ??

For instance, I've heard of randomizers that use something static (eg: a player name), converts it to a number, inputs that number into an RNG formula, and outputs results. Funktion says he has pretty normal, nothing-out-of-the-ordinary resource screw, whereas I have back to back games (and 3 games out of 5) where I draw 4 resources in 16 turns. I'm playing 24, including 4 Shards of Fate, 11 of one, and 9 of the other.

By the time I died, 20 of my 41, 40, and 41 cards were resources. Statistically improbable? Yep. Happen a lot more frequently to me? Yep. Just bad luck? Maybe. But I don't buy it. I can have a ton of TCG bad luck. But there becomes a point when it just isn't fun being bent over by RNG. #frustrated

Edit: I will note that it has been worse since the patch, since it was noted above.

Niedar
12-05-2013, 11:53 PM
That is cool, maybe you should actually get some data that backs up your speculation because all data collected so far implies no problem.

McKizz
12-06-2013, 01:35 AM
Statistically improbable? Yep. Happen a lot more frequently to me? Yep. Just bad luck? Maybe. But I don't buy it. I can have a ton of TCG bad luck. But there becomes a point when it just isn't fun being bent over by RNG.

Looks like you missed that part. Also, what's the need for being so haughty?


It's funny how people come in here thinking there's an argument going on.
Maybe because of posts like the above?

Jugan
12-06-2013, 01:53 AM
To be fair mmt260, people here weren't only "bitching" but did actually put a lot of efforts into actually testing the randomizer to make sure it isn't bugged. What if it were? Would you prefer to simply live with it without anyone voicing their concerns? Everything seems fine, but it as lot easier to accept that, if you did some tests, rather than just saying "how could this company make any mistakes"? :)



The shuffler is working properly.


People who are complaining about getting flooded or screwed:
1. Have limited experience with how TCG's work
or
2. Want something to complain about
or
3. Are making an excuse to themselves in a feeble attempt save their ego from being bruised
or
4. Have poor deck building skills / Using a bad deck
or
5. A combination of the above

If you understand what "variance" and "random" mean, then there's no reason to be complaining about a few bad games when an employee confirmed the shuffler is working.

Emergence
12-06-2013, 02:13 AM
I don't buy it. My personal experience is that I get screwed more times than not. I shouldn't have to increase from 24 to 26 resources just to appease the RNG gods.

Is the "randomizer" using static objects to "randomize" ??

For instance, I've heard of randomizers that use something static (eg: a player name), converts it to a number, inputs that number into an RNG formula, and outputs results. Funktion says he has pretty normal, nothing-out-of-the-ordinary resource screw, whereas I have back to back games (and 3 games out of 5) where I draw 4 resources in 16 turns. I'm playing 24, including 4 Shards of Fate, 11 of one, and 9 of the other.

By the time I died, 20 of my 41, 40, and 41 cards were resources. Statistically improbable? Yep. Happen a lot more frequently to me? Yep. Just bad luck? Maybe. But I don't buy it. I can have a ton of TCG bad luck. But there becomes a point when it just isn't fun being bent over by RNG. #frustrated

Edit: I will note that it has been worse since the patch, since it was noted above.

Assuming they're using a decent RNG, the seed they use doesn't really matter as long as it's not constantly getting reseeded with the same seed (which it's not since people aren't drawing cards in the EXACT same order multiple times).

Additionally yeah that's bad luck, but isn't enough to say there's anything wrong with the RNG. Think of how many different possible combinations of cards you could draw in 20 turns, and consider that individually each one is equally likely. 5-10 games is such an incredibly small sample size that you don't have any statistical power to draw any conclusions about the distribution of draws, so while yeah stuff may seem unlikely there's no statistical basis to complain unfortunately. Human beings in general are notorious about seeing patterns in data where there really isn't any pattern at all. A couple well known examples would be the hot-hand fallacy (http://en.wikipedia.org/wiki/Hot-hand_fallacy) and gambler's fallacy (http://en.wikipedia.org/wiki/Gambler's_fallacy) (which is what people are believing when they use small sample sizes to argue something is wrong). Some people intuitively understand statistics, but many (arguably most) don't. I really don't very well myself :( but I've taken a lot of math so I know about it even if I don't have the intuition for it.

McKizz
12-06-2013, 03:04 AM
I understand what you're saying, Emergence. And I appreciate where you're coming from. But just saying that the "sample size isn't big enough" also doesn't prove there's NOT a problem. Think about how this is perceived by paying customers in the future:

"Yeah dude, sorry that you have bad luck so frequently. Murphy must love you. But your sample size is too low. So unless you play 1000 games and track every game, you'll just have to trust us. I'm sure you just don't understand statistics."

When this goes live post-beta, if users perceive there is an RNG problem they won't invest money into the game. Despite their sample size only being maybe 10 games. I have faith in CZE and I wouldn't have backed it if I didn't believe in them. So no, I'm not threatening to quit. But from a business standpoint, if there are many people that feel there is an RNG problem and that it's not isolated, they won't spend their money on the game... or they may quit entirely.

I just played another game where I drew a 27 cards, including starting hand, and got 4 resources. By the end of the game, I had drawn 33 cards and had 5 resources. All 5 were Blood. My distribution is 12 Blood, 9 Diamond, 4 Shards of Fate. I wish I knew when the next resource would have come, but I died. That makes the third time tonight and the 4th time in two days. And I just increased the amount of resources in my deck by one, up to 25, to combat this problem.

Banquetto
12-06-2013, 03:06 AM
When this goes live post-beta, if users perceive there is an RNG problem they won't invest money into the game.

Oh, I can guarantee you that when the game goes live, plenty of users will perceive that there is an RNG problem, whether there is or not.

Coherent_Light
12-06-2013, 08:58 AM
Is the "randomizer" using static objects to "randomize" ??

For instance, I've heard of randomizers that use something static (eg: a player name), converts it to a number, inputs that number into an RNG formula, and outputs results. Funktion says he has pretty normal, nothing-out-of-the-ordinary resource screw, whereas I have back to back games (and 3 games out of 5) where I draw 4 resources in 16 turns. I'm playing 24, including 4 Shards of Fate, 11 of one, and 9 of the other.

By the time I died, 20 of my 41, 40, and 41 cards were resources. Statistically improbable? Yep. Happen a lot more frequently to me? Yep. Just bad luck? Maybe. But I don't buy it.

I agree with you that there are pitfalls when constructing an RNG. However, testing one, using brute force algorithms (that are pretty bulletproof) is so trivial that I would be surprised if the dev team didn't do it every time a single line of code in the shuffler function or RNG was altered.

Shuffle the cards, then grab a slice of X cards from a random position in the deck. Record (or ave or whatever) the number of resources. Repeat Y times, whether that is 1 million, run the program for 2 hours, overnight, however long. Check to see if the expected distribution of resources in the slices matches the composition of the deck.

I would expect that the code for the RNG and shuffler function would be some of the first things to be finished (untouched from months ago), and I would be surprised if someone didn't run some sort of check similar to what I described as its something you can whip up in literally minutes and run while you work on other things.

I've seen some weird things too, but I think it is unlikely that there is some sort of issue with either the RNG implementation or the shuffling function.

Xtopher
12-06-2013, 09:39 AM
When this goes live post-beta, if users perceive there is an RNG problem they won't invest money into the game. Despite their sample size only being maybe 10 games. I have faith in CZE and I wouldn't have backed it if I didn't believe in them. So no, I'm not threatening to quit. But from a business standpoint, if there are many people that feel there is an RNG problem and that it's not isolated, they won't spend their money on the game... or they may quit entirely.
I don't think it will make any difference business-wise. People have been complaining about the MTGO shuffler for more than a decade and they do millions in sales every year.

There will always be shuffler complaints, especially from players that primarily play paper. CZE has stated the shuffler is fine. If they lie once, they're dead, so I don't think they'd fabricate a statement about something so trivial and easy to address.

Edit: Just wanted to add that as far as I know, Wizards has never made a statement that the shuffler is fine, yet people spend all kinds of money there. We did get Elf to test it for us once and he did 10 million shuffles and the results were almost exactly what they should be. Things were looking kind of off around the 10,000 mark, though. So, it really takes a lot of testing.

Jugan
12-06-2013, 10:52 AM
I understand what you're saying, Emergence. And I appreciate where you're coming from. But just saying that the "sample size isn't big enough" also doesn't prove there's NOT a problem. Think about how this is perceived by paying customers in the future:

"Yeah dude, sorry that you have bad luck so frequently. Murphy must love you. But your sample size is too low. So unless you play 1000 games and track every game, you'll just have to trust us. I'm sure you just don't understand statistics."

When this goes live post-beta, if users perceive there is an RNG problem they won't invest money into the game. Despite their sample size only being maybe 10 games. I have faith in CZE and I wouldn't have backed it if I didn't believe in them. So no, I'm not threatening to quit. But from a business standpoint, if there are many people that feel there is an RNG problem and that it's not isolated, they won't spend their money on the game... or they may quit entirely.

I just played another game where I drew a 27 cards, including starting hand, and got 4 resources. By the end of the game, I had drawn 33 cards and had 5 resources. All 5 were Blood. My distribution is 12 Blood, 9 Diamond, 4 Shards of Fate. I wish I knew when the next resource would have come, but I died. That makes the third time tonight and the 4th time in two days. And I just increased the amount of resources in my deck by one, up to 25, to combat this problem.

When someone claims there is a God, and a person disagrees, the burden of proof is on the claimer.

When someone claims war crimes have been committed, and a person disagrees, the burden of proof is on the claimer.

When someone claims there's a problem with the shuffler, and a CZE employee publicly states there is no problem, the burden of proof is on the claimer. Your sample size is indeed too small; it's laughable. You just had a few games where you ran dry. Show us a scientific sample that proves the shuffler is broken, and then we'll talk. Nobody cares that you lost 3 games.

DeusPhasmatis
12-06-2013, 11:08 AM
I understand what you're saying, Emergence. And I appreciate where you're coming from. But just saying that the "sample size isn't big enough" also doesn't prove there's NOT a problem.

The prior probability that a perfectly functional RNG system generates a large number of complaints about randomness is very, very high (basically 100%). You need extraordinarily strong evidence to overcome this. That some people have a problem is not sufficient, because it's expected whether or not the RNG actually works.

Shaqattaq
12-06-2013, 12:42 PM
Considering our engineering team has the most experienced TCG engineer on the planet and others who have worked extensively on digital trading card games, we should all trust them when they say the shuffler is working properly.

DackFayden
12-06-2013, 12:59 PM
Considering our engineering team has the most experienced TCG engineer on the planet and others who have worked extensively on digital trading card games, we should all trust them when they say the shuffler is working properly.

Why not just make a sticky addressing the shuffler issue?

While I understand CZE may see it as a non-issue the number of people who keep complaining could be reduced if there was a stickied thread, where no one else could post, that just said, "The is no issue with the shuffler. It is random"

Gwaer
12-06-2013, 01:02 PM
I honestly think this thread serves a better purpose than a blanket statement like that. There are tons of informative posts in here. Just keep directing people to this thread, where actual work has been done on both sides.

mudkip
12-06-2013, 02:43 PM
Considering our engineering team has the most experienced TCG engineer on the planet and others who have worked extensively on digital trading card games, we should all trust them when they say the shuffler is working properly.

Shaqattaq, you can scream until you're blue in the face about this, but people won't believe you. People will keep asking questions until there's an effective combat log system, then we can say to people "show us your logs" so we can prove that their anecdotal evidence is crap.

It's good that this thread is open because it keeps all of silly shuffler paranoia in the one place.

Lawlschool
12-06-2013, 03:06 PM
Dear everyone, learn about the "Availability Heuristic." If it "feels" like the shuffler is off, it's only because you're focusing on the times you remember where the shuffler felt off, not all the times it worked fine.

DeusPhasmatis
12-06-2013, 03:16 PM
we can prove that their anecdotal evidence is crap

You can't disprove evidence. The only way to actually solve the problem is to teach people proper statistics, but most of the people who'll complain won't be willing to learn.


Dear everyone, learn about the "Availability Heuristic." If it "feels" like the shuffler is off, it's only because you're focusing on the times you remember where the shuffler felt off, not all the times it worked fine.

No, there will be people who correctly identify extreme outliers. The problem is that they can't see all the other games played by other people, so they don't see how infrequent their experience is in the grand scheme of things.

Vohl
12-06-2013, 03:28 PM
Considering our engineering team has the most experienced TCG engineer on the planet and others who have worked extensively on digital trading card games, we should all trust them when they say the shuffler is working properly.

As somebody else highlighted a bit further down unless the players can see some sort of log of what has been played and drawn they will only focus on the bad times when they got screwed or flooded. Are there any plans to implement a sidebar which tracks the cards that are drawn/played or something of a similar nature?

Shaqattaq
12-06-2013, 03:45 PM
Why not just make a sticky addressing the shuffler issue?

While I understand CZE may see it as a non-issue the number of people who keep complaining could be reduced if there was a stickied thread, where no one else could post, that just said, "The is no issue with the shuffler. It is random"

We have a Known Issues section in the patch notes. The shuffler is not in the Known Issues section. I don't think the amount of complaining would be reduced due to the brain's need to connect pieces through narrative or detect patterns.

Certainly, we would be open to anybody who can provide evidence that the shuffler is a problem. So, if people are able to collaborate and find information contrary to what those people in this thread have already discovered, please post it. But our engineers have been through this before on multiple projects. They say it's sound, and all data we have on it also says the process is sound.

stiii
12-06-2013, 04:02 PM
We have a Known Issues section in the patch notes. The shuffler is not in the Known Issues section. I don't think the amount of complaining would be reduced due to the brain's need to connect pieces through narrative or detect patterns.

Certainly, we would be open to anybody who can provide evidence that the shuffler is a problem. So, if people are able to collaborate and find information contrary to what those people in this thread have already discovered, please post it. But our engineers have been through this before on multiple projects. They say it's sound, and all data we have on it also says the process is sound.

This post is so awesome. I wonder if people would like any other things that work confirmed as working?

Yoss
12-06-2013, 04:06 PM
I can see it now.

Alpha Patch Notes

Fixed:
1.
2.
3.

Known Issues:
1.
2.
3.

Things That Work:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
.
.
.

DackFayden
12-06-2013, 04:23 PM
We have a Known Issues section in the patch notes. The shuffler is not in the Known Issues section. I don't think the amount of complaining would be reduced due to the brain's need to connect pieces through narrative or detect patterns.

Certainly, we would be open to anybody who can provide evidence that the shuffler is a problem. So, if people are able to collaborate and find information contrary to what those people in this thread have already discovered, please post it. But our engineers have been through this before on multiple projects. They say it's sound, and all data we have on it also says the process is sound.

Those magic words " all data we have on it also says the process is sound"

That data is irrefutable proof that can finally lay the issue to rest. Although sharing it might create some intellectual property issues... but if you can share then PLEASE SHARE. Math rarely lies

Also the phrasing of the rest of the post just made me wanna say "There are known knowns and known unknowns, but there are also unknown unknowns. We're doing the best we can"

McKizz
12-06-2013, 05:02 PM
That data is irrefutable proof that can finally lay the issue to rest. Although sharing it might create some intellectual property issues... but if you can share then PLEASE SHARE.

I agree.


No, there will be people who correctly identify extreme outliers. The problem is that they can't see all the other games played by other people, so they don't see how infrequent their experience is in the grand scheme of things.

Well put. Another reason shared data and logs would/will be beneficial.


Nobody cares that you lost 3 games.

I still have yet to understand why people feel the need to bring such negativity into a conversation. You can say exactly the same things with more tact. But... thanks for your feedback anyway.

Storm_Fireblade
12-10-2013, 08:54 AM
Ok, here is definitively something wrong now. I made 500 testdraws a while ago and didn't see a single opening hand with 0 ressources, while playing 24. Out of my last 5 games 3 or 4 of my opening hands had 0 ressources, while having 25 in the deck. Thats not just bad luck, especially since I had 0 ressources like 2-3 times yesterday as well. Whatever you guys did to the shuffler, please change it back.

I simply can not imagine this is a coincidence. Soo many 0 ressource opening hands? Like 2-3 times in a row even? I was sure the shuffler was fine, when I made all my testdraws. But this?

Kaga
12-10-2013, 09:34 AM
The main concept that seems to be lost on everyone that I am seeing (hit home by Storm's post about 2-3 times in a row) is one that was brought up a couple of pages ago by Emergence. That of the Gambler's Fallacy (http://en.wikipedia.org/wiki/Gambler's_fallacy) for those who would like a TL;DR of the concept, it is the incorrect assumption that a given event is more or less likely as a result of past events. Each game is independant of each other. You have the exact same likelyhood of drawing 0 resources from one game to the next. Making the fact that it happens multiple times in a row completely irrelevant. It is in large part due to the small sample size that you are feeling this way. The more draws you test, the closer you will approach to a more appropriate statistical distribution. I know this has been essentially said over and over but bears repeating.

It doesn't matter how many times you flip tails on the coin, the probability of flipping tails on the next throw is exactly 50%.
The odds of a series of 5 throws being TTTTT, or HHHHH, or HHTHT, or HHTTT, etc are exactly the same.

I have played.. a lot of games in the last few weeks I've been in, and so far I don't see an abnormal amount of resource flood or screw in my games. It is a well documented psychological phenomenon to focus on the bad over the good. If I had more time or perhaps better coding experience I would love to set up a test of 500-1000 draws, which would put this issue to rest. At least for the time being.

DackFayden
12-10-2013, 11:09 AM
The main concept that seems to be lost on everyone that I am seeing (hit home by Storm's post about 2-3 times in a row) is one that was brought up a couple of pages ago by Emergence. That of the Gambler's Fallacy (http://en.wikipedia.org/wiki/Gambler's_fallacy) for those who would like a TL;DR of the concept, it is the incorrect assumption that a given event is more or less likely as a result of past events. Each game is independant of each other. You have the exact same likelyhood of drawing 0 resources from one game to the next. Making the fact that it happens multiple times in a row completely irrelevant. It is in large part due to the small sample size that you are feeling this way. The more draws you test, the closer you will approach to a more appropriate statistical distribution. I know this has been essentially said over and over but bears repeating.

It doesn't matter how many times you flip tails on the coin, the probability of flipping tails on the next throw is exactly 50%.
The odds of a series of 5 throws being TTTTT, or HHHHH, or HHTHT, or HHTTT, etc are exactly the same.

I have played.. a lot of games in the last few weeks I've been in, and so far I don't see an abnormal amount of resource flood or screw in my games. It is a well documented psychological phenomenon to focus on the bad over the good. If I had more time or perhaps better coding experience I would love to set up a test of 500-1000 draws, which would put this issue to rest. At least for the time being.

1) Gamblers fallacy is right. Chance of good hand or whatev is the same per hand drawn. You're never due for a bad/good hand.
2) Don't use coins for gamblers fallacy in a series way. The 50/50 prob is throwing of your point. TTTT = HHHH only because chance of H = T = 50%. In general looking at the chance of series given that all the events haven't occurred yet isn't gamblers fallacy. It's once the events have occured that gamblers fallacy is an issue.
3) I agree from my experience shuffler is random, but forums are where issues like this should be discussed and debated Eventually someone who cares will run test hands, or CZE will release their test. Then we can all learn about randomness

Coherent_Light
12-10-2013, 11:24 AM
The main concept that seems to be lost on everyone that I am seeing (hit home by Storm's post about 2-3 times in a row) is one that was brought up a couple of pages ago by Emergence. That of the Gambler's Fallacy (http://en.wikipedia.org/wiki/Gambler's_fallacy) for those who would like a TL;DR of the concept, it is the incorrect assumption that a given event is more or less likely as a result of past events. Each game is independant of each other. You have the exact same likelyhood of drawing 0 resources from one game to the next. Making the fact that it happens multiple times in a row completely irrelevant.

Something occurring multiple times in a row is not irrelevant. The chance of seeing no shards in a 60 card deck, 25 shards, 7 card starting hand is .0174 (<2%) if that that occurred three times in a row, it *could* be relevant as the odds of that happening are 1 in 190000 or so.

That said, one thing to consider is how many people are playing and how many hands have been delt. The very first 3 hands dealt in the game are very unlikely to be shardless, but deal enough hands and you are going to see everything under the sun.

As I posted earlier I think it is unlikely that there is a problem with the RNG or shuffling algorithm for one simple reason: while there can be some pitfalls in programming a RNG, testing one using brute force methods is dead simple. I'm sure the devs have already verified that the RNG/shuffler are working fine.

Kaga
12-10-2013, 11:27 AM
Thanks for the feedback Dack, I like to use the coins when I talk about statistical issues because I feel that the 50/50 chance is much easier to comprehend in a raw form like that, and it's easy to illustrate to someone that while HHHHH seems statistically unlikely, that exact sequence has the same probability as any other given sequence of 5 throws. You are probably right that it may have not been a good example for my post directly about gambler's fallacy.

In any case. Each draw is a statistically separate event (including mulligans) and it looks like some of our intrepid compatriates assembled a few hundred test draws back on page 3 (http://forums.cryptozoic.com/showthread.php?t=29508&page=3) silly me, reading page 1, and then 30 onwards. Thier results can be found on this spreadsheet (https://docs.google.com/spreadsheet/ccc?key=0AgrNS7FanmrudFhmYVhaT0UzUk5rampSMF9VR0diN mc#gid=0) for those interested, the numbers from each of them, even with such a relatively small sample size as 100 are fairly close to what one would expect, and the average of all of thier data is even closer. But again, in practice the larger the sample pool the more they will approach expected distribution, but have no garuntee of ever being spot on, as is the nature of statistics and randomness. This graph of the results (http://forums.cryptozoic.com/showthread.php?t=29508&page=13&p=314588&viewfull=1#post314588) is also very relavent and buried in the middle of the thread, a very digestable and easy to read chart of the results.

The point is, everyone should see the data that our fine companions have collected, and hopefully having the numbers to back it up should put your mind somewhat at ease.


Something occurring multiple times in a row is not irrelevant. The chance of seeing no shards in a 60 card deck, 25 shards, 7 card starting hand is .0174 (<2%) if that that occurred three times in a row, it *could* be relevant as the odds of that happening are 1 in 190000 or so.

My point with the original post was that something happening 2 or 3 times in a row is irrelevant statistically. Each draw is independant of the other, you have the same odds of drawing the cards the second time as you did the first. You would be right if it was occuring an abnormal amount of the time as you say, but based on the data other's collected that does not appear to be the case.

Coherent_Light
12-10-2013, 11:47 AM
My point with the original post was that something happening 2 or 3 times in a row is irrellevant statistically. Each draw is independant of the other, you have the same odds of drawing the cards the second time as you did the first. You would be right if it was occuring an abnormal amount of the time as you say, but based on the data other's collected that does not appear to be the case.
Something unlikely occurring multiple times in a small sample size can be significant. The other posted stated Out of my last 5 games 3 or 4 of my opening hands had 0 ressources, while having 25 in the deck. -- if that is correct, it is an unlikely occurrence. You've made your point with the gambler's fallacy - in this day and age who thinks that? I tend to interpret Storm's post more towards what I stated above then the fact he said 'in a row'.

Again though, these things are bound to happen if you play long enough.

edit: to some degree we are talking past each other I think. While what I said is correct, one important thing to consider is Storm's perspective of something occurring during his 'last 5 games' when he could have played 100 and thus his sample size isn't 5 clearly. The more you play the greater the likelyhood of 'streaks'.

Coherent_Light
12-10-2013, 02:27 PM
I simply can not imagine this is a coincidence. Soo many 0 ressource opening hands? Like 2-3 times in a row even? I was sure the shuffler was fine, when I made all my testdraws. But this?
So, looking at this from a streak perspective is kind of interesting. If Storm has played 100 games, the probability of a streak of 3 (or more) no shard skunk hands is ~ 1 in 1971. If 200 games its 1 in 976.

Again, I'm in no way saying anything about the RNG, only that for him, it was certainly bad luck and understandable he would complain.

I'm using the online calculator below and using my previous figure of a .0174 chance of drawing no resources. Testing it on some simpler examples seemed to work fine, but I would welcome someone checking the math.
http://www.pulcinientertainment.com/info/Streak-Calculator-enter.html

Storm_Fireblade
12-10-2013, 04:12 PM
I agree, that playing much does increase the chance of a streak. By now I've probably played hundreds of matches. And I was one of those considering bugs in the randomizer and because of that I went through 500 testdraws, writing down my observations. As a result I was confident, that everything is fine and only my perception was off. During the last days (not sure if 2, 3 or 4) I did encounter 0 ressources in my opening hand frequently though. And after today with like 3 times in 5 matches and even 2-3 in a row, thats did feel so odd. Especially since I had several 0 ressource hands with 25 ressources in the deck. It might still just be a coincidence, but if that really should be bad luck...then I don't know. Guess I'm just the unluckiest person playing Hex right now :)

EDIT: HERE (forums.cryptozoic.com/showthread.php?t=29508&p=318527&viewfull=1#post318527) is my post about the 500 testdraws, including a link to the data.

DackFayden
12-10-2013, 05:18 PM
I agree, that playing much does increase the chance of a streak. By now I've probably played hundreds of matches. And I was one of those considering bugs in the randomizer and because of that I went through 500 testdraws, writing down my observations. As a result I was confident, that everything is fine and only my perception was off. During the last days (not sure if 2, 3 or 4) I did encounter 0 ressources in my opening hand frequently though. And after today with like 3 times in 5 matches and even 2-3 in a row, thats did feel so odd. Especially since I had several 0 ressource hands with 25 ressources in the deck. It might still just be a coincidence, but if that really should be bad luck...then I don't know. Guess I'm just the unluckiest person playing Hex right now :)

EDIT: HERE (forums.cryptozoic.com/showthread.php?t=29508&p=318527&viewfull=1#post318527) is my post about the 500 testdraws, including a link to the data.

Yay some actual testing on the randomizer.

I feel like the real complaint here is people don't like losing a game just cause they had resource problems in the opener. So its not a randomizer issue more of a deckbuilding and game mechanic(mulligan) issue, heavier on deck-building portion though.

ossuary
12-10-2013, 05:23 PM
I don't see how you can legitimately claim that 0 resources 3 times in a row with 25/60 resources is a DECKBUILDING problem, considering the accepted "right" number of lands is usually 23-24.

I don't want to get into the whole RNG thing, but you can't fault the deckbuilding for that one, at least. :)

Avaian
12-10-2013, 05:49 PM
I don't see how you can legitimately claim that 0 resources 3 times in a row with 25/60 resources is a DECKBUILDING problem, considering the accepted "right" number of lands is usually 23-24.

I don't want to get into the whole RNG thing, but you can't fault the deckbuilding for that one, at least. :)

This reminds me of a game I was playing the other day with my 'Bunnies!' Deck. (24/60 resources)

Started with 2 resources in hand, starting second and didn't get my third resource until turn 5. :(

I also was able to play 2 Moon'ariu Sensei, so that was two additional card draws. I ended up winning that game thanks to 'Sensei of the Wounded Petal' and a couple other factors.

DackFayden
12-10-2013, 06:33 PM
I don't see how you can legitimately claim that 0 resources 3 times in a row with 25/60 resources is a DECKBUILDING problem, considering the accepted "right" number of lands is usually 23-24.

I don't want to get into the whole RNG thing, but you can't fault the deckbuilding for that one, at least. :)

I guess there is a non-zero chance that a player receives all bad hands when they play a tcg. Normally that would suck, but look at it this way. You are that 1-1000000, so feel special you're a legend

ossuary
12-10-2013, 08:33 PM
People who know me can verify that my luck with RNG and dice is legendarily bad. I'm famous for missing important skill checks by 1, or rolling a 1 at the worst possible moment. In World of Warcraft, I would normally roll 100s constantly for trash items like gems, then roll 3 or 4 for the big boss drops. One time I was 2nd to last, rolled a 97, and figured I finally won one... and the last guy rolled 99.

In any sufficiently sized sample of statistical data, someone has shit luck. :)

Storm_Fireblade
12-11-2013, 05:27 AM
4-5 matches today and again 1-2 times 0 ressources in my opening hand. I'm probably gonna keep track with everything again, but this amount of 0 ressources is ridiculous by now :)

Kaga
12-11-2013, 08:46 AM
Storm the amount of 0 resource hands you are getting does seem a bit ridiculous o.o
I swear, if I haden't been pretty deep in statistics and gone through this entire thread yesterday I might be a randomizer problem convert by now. I don't know that anyone has done mass draw tests since the last few patches. Not that they touched the randomizer, but it might be pertinent to do some tests again. If I have some time this weekend I'll run through a few hundred using the same 24 resource/36 non-resource test they did before. I haven't noticed many problems myself, all my decks are 23-25 resource. Idunno. Maybe you did something to make Hex angry man.

Coherent_Light
12-11-2013, 08:54 AM
4-5 matches today and again 1-2 times 0 ressources in my opening hand. I'm probably gonna keep track with everything again, but this amount of 0 ressources is ridiculous by now :)
From here on out, you could keep meticulous records of how many games played, how many 0 shard opening hands you are dealt and see the probability of that happening. The key though is meticulous. In my next 5 games played, there is a pretty big difference between the odds of 1 or 2 no shard 7 card hands occurring. On the order of (1 in 12) for one vs (1 in 342) for > one... This is assuming 60 cards, 25 lands.

This online calculator is handy.
http://stattrek.com/online-calculator/binomial.aspx

Storm_Fireblade
12-11-2013, 08:56 AM
I started to write down every opening hand now. Its not gonna grow as fast as the last time, where I used the "test hand"-feature of the deckbuilder, but I will at least be able to track, whether or not that just was an amazingly unlucky streak :) If its going to continue this way, I'll probably test the shuffler again. But we'll see :)

ossuary
12-11-2013, 10:46 AM
Frankly, if we want to be super thorough, we would need to test several thousand hands with the deckbuilder's test hand as well as the actual start-of-match shuffle... there's a possibility one could be functioning different than the other, but we'd have to test both equally to show whether or not there's a statistical difference.

Doing 500 hands with the test drawer is easy... doing it 500 times against the AI and then immediately conceding would take a bit longer just because of the loading time. But we'd have to do both to prove that they're both working the same. :)

Rapierian
12-12-2013, 10:42 AM
This seemed to happen to me quite a bit last night. For the life of me I could not draw a resource, and I even started to load my decks with extra resources. No better.

Qing
12-16-2013, 05:53 AM
You can be unlucky in a game involving luck.

escapeRoute
12-16-2013, 07:02 AM
yeahm yesterday i was playing against this dude that started 2 of our 3 tournament matches with 1 (!!!!!!!!) resource and then was surprised he ended up screwed and not drawing it in the next 3/4 turns...

i am more and more convinced that those who are bothered by this problem are those who dont know how to deal with it... im sorry if that sounds rude, but thats what it is

Svenn
12-16-2013, 07:39 AM
I once got mana screwed in a (physical) MTG limited tournament 3 games in a row with 40% resources in my deck! MTG Randomizer is broken!

Unfortunately, any video game with RNG is bound to have the constant threads of "RNG is broken!" People get unlucky and start looking for all kinds of things to blame.

Gwaer
12-16-2013, 08:31 AM
I once got mana screwed in a (physical) MTG limited tournament 3 games in a row with 40% resources in my deck! MTG Randomizer is broken!

Unfortunately, any video game with RNG is bound to have the constant threads of "RNG is broken!" People get unlucky and start looking for all kinds of things to blame.


It is my sincerest belief that these threads in the hex forums are an excellent outlet for these issues. People who are upset come and vent their frustration, then move on feeling better and finding the game more enjoyable since they've gotten it off their chest. The people who are very very upset and can't let it go can help by providing actual data to add to the work that has already been done potentially uncovering bugs in the process as well as learning a bit about confirmation bias and various other mechanisms our brains use to trick us. To be perfectly honest, the only problem with these threads I can see in these hex forums are the people who are so absolutely certain there is not a problem that they come in here and abuse people with equally incorrect assumptions or bad math, statistics or psychology. Luckily though people on both sides are above average to what you find on other forums and all of the problems are relatively minor.

ossuary
12-16-2013, 10:01 AM
Luckily though people on both sides are above average to what you find on other forums and all of the problems are relatively minor.

Speak for yourself, Hitler. ;)

stiii
12-16-2013, 11:09 AM
Speak for yourself, Hitler. ;)

Nice try but even that won't end this thread.

Shaqattaq
12-16-2013, 12:01 PM
It is my sincerest belief that these threads in the hex forums are an excellent outlet for these issues. People who are upset come and vent their frustration, then move on feeling better and finding the game more enjoyable since they've gotten it off their chest. The people who are very very upset and can't let it go can help by providing actual data to add to the work that has already been done potentially uncovering bugs in the process as well as learning a bit about confirmation bias and various other mechanisms our brains use to trick us. To be perfectly honest, the only problem with these threads I can see in these hex forums are the people who are so absolutely certain there is not a problem that they come in here and abuse people with equally incorrect assumptions or bad math, statistics or psychology. Luckily though people on both sides are above average to what you find on other forums and all of the problems are relatively minor.

I agree that the way people treat each other is the biggest challenge in these threads, but that's not unique to this subject in particular. I think that the biggest issue with the randomization discussion specifically is that players are perhaps too bold with their assertions that there is a problem with the randomizer when there is not, so less active community members jump into the thread and believe there is a problem, so they go tell others.

Xtopher
12-16-2013, 12:14 PM
To be perfectly honest, the only problem with these threads I can see in these hex forums are the people who are so absolutely certain there is not a problem that they come in here and abuse people with equally incorrect assumptions or bad math, statistics or psychology.
The biggest problem I see is a combination of poor reading comprehension, poor understanding of statistics, and the thread getting "reset" periodically because yet another person posts their anecodote without paying attention to the hundreds of posts that have come before.

But, yeah, on the plus side I guess it is a good dumping ground for people to work out their frustrations.

ossuary
12-16-2013, 12:32 PM
I agree that the way people treat each other is the biggest challenge in these threads, but that's not unique to this subject in particular. I think that the biggest issue with the randomization discussion specifically is that players are perhaps too bold with their assertions that there is a problem with the randomizer when there is not, so less active community members jump into the thread and believe there is a problem, so they go tell others.

I'd say that's a pretty accurate analysis. The whole psychological imperative to remember negative events better and for longer than positive or expected events (apparently it's an evolutionary survival trait) certainly doesn't help... it guarantees that people will keep "discovering" this "problem" over and over again, every time someone happens to get that 1 in 1,000 draw where they mull to 4 and still have no resources. ;)

mudkip
12-16-2013, 03:46 PM
I think that the biggest issue with the randomization discussion specifically is that players are perhaps too bold with their assertions that there is a problem with the randomizer when there is not, so less active community members jump into the thread and believe there is a problem, so they go tell others.

The only way to stop that is for someone to make an awesome thread debunking it and sticky that. That way, people will be directed to a thread full of facts instead of paranoia and emotion.

ossuary
12-16-2013, 04:12 PM
The only way to stop that is for someone to make an awesome thread debunking it and sticky that. That way, people will be directed to a thread full of facts instead of paranoia and emotion.

Please god no... we already have 304 quintillion stickied threads. ;)

DackFayden
12-16-2013, 05:32 PM
Please god no... we already have 304 quintillion stickied threads. ;)

Maybe CZE just needs to be transparent on the issue and show the randomizer. Then all the computer/stats oriented people can go all out.

Ooooorrrr.... CZE could just ignore the issue and do nothing. Status Quo never hurt/helped anyone. Maybe keeping par is the easiest way out

Shaqattaq
12-16-2013, 06:17 PM
Maybe CZE just needs to be transparent on the issue and show the randomizer. Then all the computer/stats oriented people can go all out.

Ooooorrrr.... CZE could just ignore the issue and do nothing. Status Quo never hurt/helped anyone. Maybe keeping par is the easiest way out

We consider the randomizer part of our game code and cannot release that publicly. As others have shown with their own data collection and methods, in addition to our own internal testing, the randomizer works correctly.

mudkip
12-16-2013, 06:31 PM
Please god no... we already have 304 quintillion stickied threads. ;)

This is something that people frequently discuss. Even if there is already a thread, it's one of those issues that people feel compelled to make a new thread about when they don't get the perfect hand for the last 3 games.

It would be smarter for the community if people were directed to some propaganda on the issue to diffuse their fears.


Maybe CZE just needs to be transparent on the issue and show the randomizer. Then all the computer/stats oriented people can go all out.
I disagree. If by chance their is some way to exploit it, someone would find a way.

As an end user, the randomizer is and should remain a black box. If you can prove that the output isn't what you expect (hint: you can't) then there would be cause for alarm.

Chark
12-16-2013, 06:56 PM
@Shaqattaq: maybe have an engineer talk about the randomizer in an upcoming article on the site? This way we can link to the article when these threads pop up?

Gwaer
12-16-2013, 07:32 PM
That kind of stuff is really very unlikely to do any good. They will *know* something is wrong with it, and the devs just haven't found it yet, or are covering it up, or some other nonsense. It's much more effective for people to come to the realization gradually by doing a little work, and reading up on it.

escapeRoute
12-17-2013, 01:41 AM
That kind of stuff is really very unlikely to do any good. They will *know* something is wrong with it, and the devs just haven't found it yet, or are covering it up, or some other nonsense. It's much more effective for people to come to the realization gradually by doing a little work, and reading up on it.

exactly, it doesnt matter how much cze talk bout it, it will just be another proof for the conspiracy theory of some people... its just better to let it go and maybe, occasionally, say "its all right people, nothing to see here, go home"

Storm_Fireblade
12-17-2013, 04:08 AM
It was absolutely incredible, that I got like a dozen opening hands with 0 ressources out 2-3 three dozens matches and 25 ressources in the deck. I'm pretty sure statistically its ALMOST impossible. But, I've been keeping track for a while and in several dozens of games after this very strange streak, I only saw 0 ressources 1-2 more times. Now I keep getting flooded with 11 ressources out of 18 cards and other amazing things. But I'm absolutely sure, this is just bad luck. Obviously there could have been something wrong with the randomizer before and Cryptozoic made some changes without telling us, but right now, I'm confident that everything is fine.

Malakili
12-17-2013, 04:58 AM
There was an interesting keynote by Sid Meier a few years back about designing Civilization, and interestingly they got feedback that people were upset when losing certain fights that they felt they should win based on the %, and especially upset when they lost several in a row. He said in the end they had to make it so the game intentionally fudged the numbers on purpose so you couldn't go on extreme bad luck streaks.

I think you can get away with that in a primarily PvE game like Civ, but in a PvP game it makes me nervous to rig things like that.

ossuary
12-17-2013, 05:15 AM
Statistics are weird. People just don't understand that. For example, if you asked someone to work out the likelihood of flipping a coin and having it come up heads 6-8 times in a row, they would (naturally) conclude that percent chance is EXTREMELY low (8 is roughly .39%). What they fail to realize is that if you flip a coin a thousand times (or 150,000 times, for a match-making online game that flips coins to see who wins), it is statistically LIKELY that somewhere in that thousand flips, you will have a run of 8 or even 10.

Students in statistics classes are often asked by their professors to do this very thing. The professors can always tell who really flipped the coins and who just wrote down fake results, because the fake results invariably have far too few run-on series.

The point of all this is that perception colors results. People see patterns everywhere, especially ones that they feel harm them. It's nature... but that doesn't mean we can't try to diffuse it.

mudkip
12-17-2013, 06:55 AM
I think you can get away with that in a primarily PvE game like Civ, but in a PvP game it makes me nervous to rig things like that.

Yep, every roll of the dice should be independent. People would game the system otherwise.

KeeblerElf
02-03-2014, 11:04 PM
I know I'm late to this party, but I was just directed here from a thread I started on the same topic: I drew 3 sets of 4 identical cards, one set after the other after the other. I calculated the probability to be less than 3.5 in a 1,000,000 (it was actually much less, but this is the best upper bound I have based on my memory of how many cards I had left), and here I am! (I have also drawn a hand of 7 resources at least twice, with 24 / 60 cards being resources, a probability of less than 9 in 10,000 each time.)


This is how the game works: deck is shuffled randomly, cards are drawn from the top one at a time.

If we can prove that the shuffler works randomly then we can prove there is no problem with any kind of anomalies (eg 3-card "clumps").

Are you claiming 3-card clumps turn up too often? If so you're also claiming that the randomiser is not random. There's no way the randomiser can be truly random AND somehow give you more 3-card clumps than it should.

Just to be clear - any computerized shuffle is not truly random, as there is no known way to generate a sequence of random numbers using a computer. (That's not to say that we can't digitize the result of a random physical process, though; see random.org for one such service.)

Also, a truly random sequence can (of course) give more clumps than predicted; since it's random, we can only predict probabilistically. Maybe I'm not understanding what you're saying here, though, as there are a couple hundred posts of context I haven't read through!

My main concern with the shuffle is that it might "compensate" rather than "flood" its distributions. For example, the game may monitor how often you've drawn a resource, and realize that you haven't drawn any in six hands; it then draws five in a row for you to compensate for this, so that if you were to look at all the draws up to that point the number of drawn resources wouldn't seem unusual. This is compensating (rather than flooding), and it is not the way any decent shuffler should work (because then draws are, of course, not independent). Flooding is essentially a term used to describe the independent nature of each draw: there is a decided distribution (determined by the distribution of cards in the deck), and each draw independently is considered only with that, and not with previous draws.

What's worse is that a compensation method may pass simple tests of randomness; after all, it's possible to statistically mimic the results of a random distribution with a non-random process.

Someone has mentioned that Cryptozoic has made official posts asserting that the shuffler is of high quality, but I can't find those posts. Could anyone point me to them? Thanks!

BlackRoger
02-04-2014, 04:09 AM
I would like to start off by saying that if I sound a little condescending, I'm sorry, it's just this has come up so many many times and we are all tired from these arguments.
So with this in mind, the answers to your questions:


I know I'm late to this party, but I was just directed here from a thread I started on the same topic

I've seen your other thread, and I'm quite sure you were directed here so you could read a bit about what other people have researched and concluded about the randomizer.
Maybe you should read some of it, especially the last few pages, before copy pasting here your post.



: I drew 3 sets of 4 identical cards, one set after the other after the other. I calculated the probability to be less than 3.5 in a 1,000,000 (it was actually much less, but this is the best upper bound I have based on my memory of how many cards I had left), and here I am! (I have also drawn a hand of 7 resources at least twice, with 24 / 60 cards being resources, a probability of less than 9 in 10,000 each time.)


While I'm a bit skeptical about the ordering you mentioned, the question is how many times did this happen to you?
If it happens more than 20 times out of a million then something is off.
Hell, if you ever see it happening again something may be wrong, but just once is still statistically fine.


Just to be clear - any computerized shuffle is not truly random, as there is no known way to generate a sequence of random numbers using a computer. (That's not to say that we can't digitize the result of a random physical process, though; see random.org for one such service.)

Since its impossible for a human being to distinguish between true randomization from computer generated one, this point is mute.


My main concern with the shuffle is that it might "compensate" rather than "flood" its distributions. For example...

So after a very small sample of games you have reached a conspiracy theory that the shuffler is doing mean things behind our backs.
If you go back and read threads like this and many others, they are filled with so many such theories, all of them with different weird ideas.
My point is, if you look at a sample small enough, you can get to any number of conspiracy theories, so before going full power on analyzing the consequences of said "compensations", maybe you should try to actually validate this theory by using a larger sample size?
Especially considering it makes absolutely no sense for CZE to do these weird shuffling experiments behind our backs, they have much more important content to deliver atm then to mess around with complex mathematical equations for no good reason.

The people in this thread have checked thousands of samples and reached the conclusion it is random, if you do not intend to properly check your theories then they are just that, random conjectures.



Someone has mentioned that Cryptozoic has made official posts asserting that the shuffler is of high quality, but I can't find those posts. Could anyone point me to them? Thanks!

Some of CZE's messages (there were others):
http://forums.cryptozoic.com/showthread.php?t=29508&page=28&p=317530&viewfull=1#post317530
http://forums.cryptozoic.com/showthread.php?t=29508&page=40&p=328596&viewfull=1#post328596

DackFayden
02-04-2014, 11:55 AM
Disappointed with the condescending tone BlackRoger, the guy was trying to add to the conversation. If you cannot appraoch it calmly then simply don't waste your time responding. We don't need that negativity.

On your issue/conclusion KeeblerElf:


Just to be clear - any computerized shuffle is not truly random, as there is no known way to generate a sequence of random numbers using a computer. (That's not to say that we can't digitize the result of a random physical process, though; see random.org for one such service.)

So far I agree with you. If CZE released their shuffler algorithm there would be no more inconclusive discussion, but I understand for Intellectual Property reasons that may not be possible. So let the discussion commence.


Also, a truly random sequence can (of course) give more clumps than predicted; since it's random, we can only predict probabilistically.

Yes clumping events have a non-zero chance so they are possible, but look up the law of large numbers when you get a chance. Quick summary of that law: with a large enough sample size we'll observe our expected value.

MEANING: If we make a stat breakdown for outcomes of top X cards(which assumes actual randomness) of a deck then if we perform an experiment where we draw the top X cards. If we do it long enough the distibutions we observe IRL will be **very** if not exactly like we predicted. That's why people on these forums are always saying don't go saying shuffler is bonkers when you jsut get 1 weird game.

SIDE NOTE: If someone with more knowledge of Stochastic processes could chime in here. Maybe tell us how many runs we would need to do to have a valid conclusion that'd be great.


"compensate" rather than "flood" its distributions. For example, the game may monitor how often you've drawn a resource, and realize that you haven't drawn any in six hands; it then draws five in a row for you to compensate for this, so that if you were to look at all the draws up to that point the number of drawn resources wouldn't seem unusual.

Interesting... But just so everyone else understands(reading comprehension check here)and correct me if I'm wrong Keebler, you're using that resource compesation thing as an example of a compensation algorithm. So you're just saying A) CZE may just be implementing a some sort of compensation algorithm instead of a truly/psuedo random one. You're NOT saying that B) You believe they're using a resource compensation system

If A is your assertion then you have my interest, even though I'm with the majority that thinks the shuffler is fine. What type of compensation algorithm do you think they're using. Personally I haven't noticed anything weird, but I also havent done calculations with a large sample size.

For that last part BlackRoger has the correct links. He also has a point that you might want to read other threads. Dont waste too much time reading them though as most are just both sides of the issue QQing and not actually doing anything productive. I think ossuary has a thread where he collected actual data, I may be wrong.

Good luck on your Journey

KeeblerElf
02-04-2014, 02:27 PM
I would like to start off by saying that if I sound a little condescending, I'm sorry, it's just this has come up so many many times and we are all tired from these arguments.
So with this in mind, the answers to your questions:

No offense taken. I don't think anyone has claimed to be the first to wonder whether there is something wrong with the shuffling. I appreciate your taking the time to respond to each point, and I likewise hope I don't come off as condescending.


I've seen your other thread, and I'm quite sure you were directed here so you could read a bit about what other people have researched and concluded about the randomizer.
Maybe you should read some of it, especially the last few pages, before copy pasting here your post.

Indeed; this is how I was able to quote another post in mine. But it's reasonable to have missed something over the 40+ pages of this thread, isn't it?


While I'm a bit skeptical about the ordering you mentioned, the question is how many times did this happen to you?
If it happens more than 20 times out of a million then something is off.
Hell, if you ever see it happening again something may be wrong, but just once is still statistically fine.

The issue is that I've just begun to play Hex within the past week or so, so I don't have that many games under my belt. I've experienced several highly unlikely events (only some are outlined above) in a relatively small number of shuffles, so naturally I'm going to wonder how effective is the shuffling in the game.


Since its impossible for a human being to distinguish between true randomization from computer generated one, this point is mute.

This is flat-out incorrect in many cases, as any computer-generated sequence is deterministic (ie, with enough analysis, it can indeed be predicted). Besides being wrong, there is a great variance in the quality of a computer-generated sequence of pseudorandom numbers, so it's really not a moot point.


So after a very small sample of games you have reached a conspiracy theory that the shuffler is doing mean things behind our backs.
If you go back and read threads like this and many others, they are filled with so many such theories, all of them with different weird ideas.
My point is, if you look at a sample small enough, you can get to any number of conspiracy theories, so before going full power on analyzing the consequences of said "compensations", maybe you should try to actually validate this theory by using a larger sample size?
Especially considering it makes absolutely no sense for CZE to do these weird shuffling experiments behind our backs, they have much more important content to deliver atm then to mess around with complex mathematical equations for no good reason.

The people in this thread have checked thousands of samples and reached the conclusion it is random, if you do not intend to properly check your theories then they are just that, random conjectures.

I stated my thought as a concern; I'm not sure how you're getting a conspiracy theory out of it, unless I have some secret fan club that believes I'm correct (and that's just silly). I understand that you've seen many posts about this before, but I am not one of those authors, and I don't hold whatever strange beliefs you seem to think they hold. However, I've read of games (released by major companies) using this compensation method for RNG before, so I was naturally concerned that Hex does it, too (in other words, this isn't something I made up as an explanation for what I perceive as the game's picking on me). This is far from an unreasonable concern, regardless of how others might treat it, as Hex relies fundamentally on well-shuffled decks. There is absolutely nothing wrong with calling to attention strange results, as it's far easier for a community to come to a conclusion than it is for an individual (ie, more data and more thinking). There's no reason why a user should have to have spent hours doing research before even mentioning an idea, and this attitude makes for a much less open environment. If users are tired of this being called to attention, perhaps it's best if any tests using large sample sizes would be stickied for easy access, as it's silly to be upset that all these people aren't wading through hundreds of posts just hoping to find some results.


Some of CZE's messages (there were others):
http://forums.cryptozoic.com/showthread.php?t=29508&page=28&p=317530&viewfull=1#post317530
http://forums.cryptozoic.com/showthread.php?t=29508&page=40&p=328596&viewfull=1#post328596

Thanks for providing these links. My main concern is that CZE does indeed have engineers checking these things, instead of mathematicians. This is a little like asking a nurse, rather than a doctor, to oversee a surgical procedure. The nurses may be on staff, and they do have some medical knowledge, but it's a better idea to call in an expert for something so critical.

KeeblerElf
02-04-2014, 02:44 PM
Disappointed with the condescending tone BlackRoger, the guy was trying to add to the conversation. If you cannot appraoch it calmly then simply don't waste your time responding. We don't need that negativity.

On your issue/conclusion KeeblerElf:



So far I agree with you. If CZE released their shuffler algorithm there would be no more inconclusive discussion, but I understand for Intellectual Property reasons that may not be possible. So let the discussion commence.

Thanks for your reply. Let the discussion continue!


Yes clumping events have a non-zero chance so they are possible, but look up the law of large numbers when you get a chance. Quick summary of that law: with a large enough sample size we'll observe our expected value.

MEANING: If we make a stat breakdown for outcomes of top X cards(which assumes actual randomness) of a deck then if we perform an experiment where we draw the top X cards. If we do it long enough the distibutions we observe IRL will be **very** if not exactly like we predicted. That's why people on these forums are always saying don't go saying shuffler is bonkers when you jsut get 1 weird game.

SIDE NOTE: If someone with more knowledge of Stochastic processes could chime in here. Maybe tell us how many runs we would need to do to have a valid conclusion that'd be great.

I actually do mathematics for a living, so I have a little knowledge of this. One issue is that the law of large numbers assumes you have identically distributed random variables, and we only have pseudorandom number generation (again, unless CZE uses some random physical process and digitizes those results - I doubt it, but I have no way of knowing). So it would be incorrect logically to (1) do an analysis of a large number of hands, then (2) observe that the hands seem to converge to the expected distribution, and finally (3) conclude from this that the algorithm is random.

There are many non-random processes that can produce a distribution that one would expect from a random process. Many of these processes may also yield a higher frequency of events that should be very improbable (a compensation algorithm might easily have both of these properties: it compensates for what hasn't been drawn, so that after many draws it has given what appears to be a reasonable distribution; but to get there, the compensating draws had to be very improbable). One of my concerns is that something like this is going on.


Interesting... But just so everyone else understands(reading comprehension check here)and correct me if I'm wrong Keebler, you're using that resource compesation thing as an example of a compensation algorithm. So you're just saying A) CZE may just be implementing a some sort of compensation algorithm instead of a truly/psuedo random one. You're NOT saying that B) You believe they're using a resource compensation system

If A is your assertion then you have my interest, even though I'm with the majority that thinks the shuffler is fine. What type of compensation algorithm do you think they're using. Personally I haven't noticed anything weird, but I also havent done calculations with a large sample size.

That's right - the resource compensation scheme I mentioned was just an example of how a compensation algorithm could work. I'm not asserting that this is what Hex's shuffler does (in fact, the only thing I'm asserting about Hex's shuffler is that it's worth investigating), but I'm saying it's one possible explanation for some of the weird behavior I'm seeing in my drawing patterns. It is not the only explanation, and may not even be a particularly good explanation. But if it happens to be correct, then we're in trouble and Hex is in trouble.


For that last part BlackRoger has the correct links. He also has a point that you might want to read other threads. Dont waste too much time reading them though as most are just both sides of the issue QQing and not actually doing anything productive. I think ossuary has a thread where he collected actual data, I may be wrong.

Good luck on your Journey

Thanks for the advice; I'm glad you recognize that it can be difficult to strike the right balance between reading through countless complaints and spending time looking for actual results.

Emergence
02-04-2014, 02:55 PM
I don't have enough time right now to write up a full response, but I do want to clarify one thing: modern random number generators are EXTREMELY good at producing sequences of numbers which are indistinguishable from uniformly at random unless you have an absurdly large number of draws (and I mean really absurdly. It would probably take decades of cataloging every shuffle in Hex to do so, if not longer) or have information about the seed.

I'll write more later. Promise :P

Umaro
02-04-2014, 03:52 PM
If you're still collecting data for this, here's 100 test draws with mulligans. I was bored and interested. 24 resources in a 60 card deck.



# of Resources
Initial Hand
Mull to 6
Mull to 5
Mull to 4


0
5
5
3
12


1
12
14
24
35


2
36
35
37
34


3
34
33
26
16


4
9
10
8
3


5
3
3
2
N/A


6
1
0
N/A
N/A


7
0
N/A
N/A
N/A

KeeblerElf
02-04-2014, 08:05 PM
I don't have enough time right now to write up a full response, but I do want to clarify one thing: modern random number generators are EXTREMELY good at producing sequences of numbers which are indistinguishable from uniformly at random unless you have an absurdly large number of draws (and I mean really absurdly. It would probably take decades of cataloging every shuffle in Hex to do so, if not longer) or have information about the seed.

I'll write more later. Promise :P

That's fantastic (and I actually mean that without sarcasm). But the question isn't really whether humans are capable of generating pseudorandom number sequences with certain properties of random number sequences; the question is whether Hex's shuffler is "good enough" for the game's requirements. Even if there exists a perfect computerized shuffler that would do just what we need for the game, it's useless (for this discussion) if that's not the one that's implemented in Hex. Surely you would agree?

I thought I'd write this quick response, so that you wouldn't have to waste time writing about the current state of technology, as that's not really the point of questioning Hex's shuffler. On the other hand, if you're going to argue that all modern shufflers are "good enough," so that it doesn't really matter which particular shuffler Hex is using, then that's worth thinking about (and still leaves open the possibility of its not being implemented correctly, etc.). In any event, all we can do is speculate unless we have solid data. Of course, feel free to expatiate on the topic as much as you feel is necessary.

Emergence
02-04-2014, 11:08 PM
That's fantastic (and I actually mean that without sarcasm). But the question isn't really whether humans are capable of generating pseudorandom number sequences with certain properties of random number sequences; the question is whether Hex's shuffler is "good enough" for the game's requirements. Even if there exists a perfect computerized shuffler that would do just what we need for the game, it's useless (for this discussion) if that's not the one that's implemented in Hex. Surely you would agree?

I thought I'd write this quick response, so that you wouldn't have to waste time writing about the current state of technology, as that's not really the point of questioning Hex's shuffler. On the other hand, if you're going to argue that all modern shufflers are "good enough," so that it doesn't really matter which particular shuffler Hex is using, then that's worth thinking about (and still leaves open the possibility of its not being implemented correctly, etc.). In any event, all we can do is speculate unless we have solid data. Of course, feel free to expatiate on the topic as much as you feel is necessary.

What I'm saying is that unless they are completely incompetent computer scientists (which I sincerely doubt they are), then they did not come up with their own random number generator and instead are using one of the many well tested, documented, and widely used ones. Once you have a working random number generator, randomly permuting the order of a list of items is trivial.

Under this assumption, collecting a large number of hands and analyzing the results works fine. The results should be distributed identically at random. Additionally, any "clumping" in the shuffler would be directly observed by doing such tests since within hands you're looking at a sequence of draws, not individual ones. As such, any inconsistencies in the distribution the shuffler yields would be detectable (though granted you'll need a massive sample size doing this with just draws of 7 cards as compared to looking at the order of the entire deck) assuming that the game isn't doing any of the compensating you're talking about.

I would be VERY surprised if it were compensating considering Cryptozoic is very serious about supporting an e-sports scene for the game, and messing with the probabilities of draws would be a huge issue for high level competitive play. Also considering how frequently the complaints about the shuffler revolve around disadvantageous things happening, I doubt that any compensation is going on.

EDIT:
Another piece of evidence which goes against the idea of any compensating going on is that when using Shards of Fate or Curse of Oblivion, you see a random ordering of your or your opponents cards. It used to be that this order was maintained as long as nothing caused the deck to be shuffled. I'm pretty sure those issues have been resolved (cards which allow you to see the order of a deck now cause the deck to subsequently be shuffled), but what this means is that the order is something pre-determined. The game is not randomly drawing cards off the top of the deck. As a result, any "smoothing" of draws could be calculated before hand and any necessary adjustments would be made such that players would essentially never be resource screwed/flooded. But since there's strong evidence that the game knows the order of your deck at all times and you still frequently see flooding or screwing of resources, the idea of compensating your draws sounds extremely unlikely to me.
/EDIT

Keep in mind that every sequence of 4 non-resource cards you could draw in a row is extremely "rare". A small number of samples containing a couple different "rare" events is not something from which we can draw any statistically significant conclusions from. If you can predict with greater than expected accuracy (assuming truly random shuffles) the frequency with which a given rare event occurs then yeah, you're on to something. But until someone can do that, it's all speculation and all the data/theories I've seen when put under scrutiny have resulted in the conclusion that nothing wrong has been detected in the shuffler.

To be clear, I'm not on some crusade to "prove" the shuffler is working. I fully recognize that any statistics we can realistically do cannot "prove" that the shuffler works in a certain way. They can only disprove, at some level of certainty, a given hypothesis of how it works. I'm completely open to people posting statistically rigorous evidence that something is wrong, evaluating it, and adjusting my opinion accordingly. The problem is that people AREN'T doing that. I have yet to see a single post of someone claiming something is wrong who actually collected a reasonable initial sample of data and then perform a reasonable statistical test on it to back up their claim.

KeeblerElf
02-05-2014, 03:39 PM
What I'm saying is that unless they are completely incompetent computer scientists (which I sincerely doubt they are), then they did not come up with their own random number generator and instead are using one of the many well tested, documented, and widely used ones. Once you have a working random number generator, randomly permuting the order of a list of items is trivial.

Well, there are two cases: either (1) they came up with their own generator, in which case there is (more) reason to worry, or (2) they did not come up with their own generator, in which case I'd like to see the code to set my mind at ease (read: why hide the code? as you point out, it's completely trivial if you're using an established generator, such as the Mersenne twister). Of course, it's their code, and they can do with it whatever they please, for any reason; they don't need to justify not releasing their own intellectual property. This is just the question that makes me a bit uneasy. But I'm probably just being silly.


Under this assumption, collecting a large number of hands and analyzing the results works fine. The results should be distributed identically at random. Additionally, any "clumping" in the shuffler would be directly observed by doing such tests since within hands you're looking at a sequence of draws, not individual ones. As such, any inconsistencies in the distribution the shuffler yields would be detectable (though granted you'll need a massive sample size doing this with just draws of 7 cards as compared to looking at the order of the entire deck) assuming that the game isn't doing any of the compensating you're talking about.

I would be VERY surprised if it were compensating considering Cryptozoic is very serious about supporting an e-sports scene for the game, and messing with the probabilities of draws would be a huge issue for high level competitive play. Also considering how frequently the complaints about the shuffler revolve around disadvantageous things happening, I doubt that any compensation is going on.

EDIT:
Another piece of evidence which goes against the idea of any compensating going on is that when using Shards of Fate or Curse of Oblivion, you see a random ordering of your or your opponents cards. It used to be that this order was maintained as long as nothing caused the deck to be shuffled. I'm pretty sure those issues have been resolved (cards which allow you to see the order of a deck now cause the deck to subsequently be shuffled), but what this means is that the order is something pre-determined. The game is not randomly drawing cards off the top of the deck. As a result, any "smoothing" of draws could be calculated before hand and any necessary adjustments would be made such that players would essentially never be resource screwed/flooded. But since there's strong evidence that the game knows the order of your deck at all times and you still frequently see flooding or screwing of resources, the idea of compensating your draws sounds extremely unlikely to me.
/EDIT

While I'd like to agree with this last paragraph, I'm simply not seeing the logic of it (provided I'm understanding what you're saying). Seeing a purported sequence of the next m cards does not mean that the order of more than m cards has been determined, and it doesn't even mean that the order of those m cards was determined before the request to see them.


Keep in mind that every sequence of 4 non-resource cards you could draw in a row is extremely "rare". A small number of samples containing a couple different "rare" events is not something from which we can draw any statistically significant conclusions from. If you can predict with greater than expected accuracy (assuming truly random shuffles) the frequency with which a given rare event occurs then yeah, you're on to something. But until someone can do that, it's all speculation and all the data/theories I've seen when put under scrutiny have resulted in the conclusion that nothing wrong has been detected in the shuffler.

To be clear, I'm not on some crusade to "prove" the shuffler is working. I fully recognize that any statistics we can realistically do cannot "prove" that the shuffler works in a certain way. They can only disprove, at some level of certainty, a given hypothesis of how it works. I'm completely open to people posting statistically rigorous evidence that something is wrong, evaluating it, and adjusting my opinion accordingly. The problem is that people AREN'T doing that. I have yet to see a single post of someone claiming something is wrong who actually collected a reasonable initial sample of data and then perform a reasonable statistical test on it to back up their claim.

I agree wholeheartedly. Likewise, I'm not attempting to prove or assert that the shuffler doesn't work properly; rather, I'm trying to make sure the community is thinking critically and mathematically about the concern (not assertion) that it might not be doing its job. I'm also trying to ensure that any data collected is not interpreted with a specific outcome in mind. Part of doing this is to point out that non-random processes can give results similar to random processes in large-scale distribution, but with very different behavior at a smaller scale (ie, the scale that affects individual games). This is undesirable, and I'm only concerned about it because of the strange number of very unlikely events I've seen in the few games I've played (if I hadn't seen them, I'd likely write off most concerns about the shuffler and quietly look at the data that will eventually be released by especially helpful members of the community).

ossuary
02-05-2014, 03:53 PM
Thanks for providing these links. My main concern is that CZE does indeed have engineers checking these things, instead of mathematicians. This is a little like asking a nurse, rather than a doctor, to oversee a surgical procedure. The nurses may be on staff, and they do have some medical knowledge, but it's a better idea to call in an expert for something so critical.

Hah! You haven't known as many doctors and nurses as I have. :)

Frankly, more often than not you'd be in better hands with the nurses. As long as they're not the ones that take their union status and hours-long-during-work-hours hang-out chats REALLY seriously, anyway. ;)

Emergence
02-05-2014, 07:57 PM
Well, there are two cases: either (1) they came up with their own generator, in which case there is (more) reason to worry, or (2) they did not come up with their own generator, in which case I'd like to see the code to set my mind at ease (read: why hide the code? as you point out, it's completely trivial if you're using an established generator, such as the Mersenne twister). Of course, it's their code, and they can do with it whatever they please, for any reason; they don't need to justify not releasing their own intellectual property. This is just the question that makes me a bit uneasy. But I'm probably just being silly.

I agree that knowing the algorithm they use would be the ideal solution to the question, but I honestly don't expect them to publish in much detail what the shuffler is doing behind the scenes. As such, I'm more concerned with what we can test; draws.



While I'd like to agree with this last paragraph, I'm simply not seeing the logic of it (provided I'm understanding what you're saying). Seeing a purported sequence of the next m cards does not mean that the order of more than m cards has been determined, and it doesn't even mean that the order of those m cards was determined before the request to see them.


With Curse of Oblivion you used to be able to see their entire deck completely in order, and that order was verified to be preserved unless something else caused the deck to be shuffled. As such, until it was fixed, it was banned from community tournaments and was considered to be in poor taste to use for casual games. For Shards of Fate, you would just see the order in which you were going to draw shards (though not how many cards were between them), which was still an issue though not as big of one. That has also been fixed as far as I'm aware. So unless they completely re-wrote the entire underlying architecture of how the deck is stored for draws since fixing the issues of being able to gain unfair information (which seems like a weird fix when simply causing decks to be re-shuffled after a use of those cards would be all that's necessary), the order of your deck is explicitly known by the game at all times.



I agree wholeheartedly. Likewise, I'm not attempting to prove or assert that the shuffler doesn't work properly; rather, I'm trying to make sure the community is thinking critically and mathematically about the concern (not assertion) that it might not be doing its job. I'm also trying to ensure that any data collected is not interpreted with a specific outcome in mind. Part of doing this is to point out that non-random processes can give results similar to random processes in large-scale distribution, but with very different behavior at a smaller scale (ie, the scale that affects individual games). This is undesirable, and I'm only concerned about it because of the strange number of very unlikely events I've seen in the few games I've played (if I hadn't seen them, I'd likely write off most concerns about the shuffler and quietly look at the data that will eventually be released by especially helpful members of the community).

Admittedly the tests people are doing aren't the most rigorous in the world, and it's entirely possible that some subtle inconsistencies are getting washed out in the way that data is being collected. However, all statistical tests so far have turned up nothing out of the ordinary. If you have a more targeted test in mind, and have the time and energy to collect the data (and perhaps recruit people to help), absolutely please do so. I would myself, but I lack said time and energy at the moment. Grad school is hard :( I haven't actually even played Hex for a few weeks.

Edit: one somewhat realistic way Cryptozoic could help us to answer this issue without talking explicitly about their algorithm is to make there be some way to batch dump hundreds or thousands of shuffles of a given deck to file. Then it would actually be pretty easy to write up a little script to analyze any number of things about the behavior of the shuffler. It probably wouldn't even take that long to implement such a dump function (maybe an hour or two at most unless there's some huge issue I'm not thinking of that would complicate things).

filipcic
02-06-2014, 10:27 AM
When you talk about alghoritm, since I'm a programer, I would do the following:
function shuffle()
{
For each card in my deck
{
Card.SortOrder = GetRandomNumber()
}
Return all cards sorted by SortOrder.
}
GetRandomNumber() is a function provided by the programming language I'm using.
For example: http://msdn.microsoft.com/en-us/library/system.random(v=vs.110).aspx

And that's about it. Of course I could be wrong, but good programers should be lazy and they certainly shouldn't invent hot water.

DackFayden
02-06-2014, 01:20 PM
When you talk about alghoritm, since I'm a programer, I would do the following:
function shuffle()
{
For each card in my deck
{
Card.SortOrder = GetRandomNumber()
}
Return all cards sorted by SortOrder.
}
GetRandomNumber() is a function provided by the programming language I'm using.
For example: http://msdn.microsoft.com/en-us/library/system.random(v=vs.110).aspx

And that's about it. Of course I could be wrong, but good programers should be lazy and they certainly shouldn't invent hot water.

Yeah even doing it that way is good enough. I'm like 90% sure whatever method they use is good enough.

I actually think the way you put it is what they actually do:

1) Iterate through elements of a deck
2) Assign random weight/sortOrder to each card
3) Return elem in sortOrder

However do be aware there may be issues with how languages handle the random concept. For example if the Random seed (http://en.wikipedia.org/wiki/Random_seed) is given as part of the random sequence, thats an issue.

But again pretty sure CZEs got this