Verified Commit e3d2ac62 authored by Markus Shepherd's avatar Markus Shepherd 🙊
Browse files

optimisation

parent 1b7467a3
Pipeline #341324 passed with stage
in 17 seconds
......@@ -38,7 +38,7 @@ Don't worry too much about the details though – *adding dummy votes* is really
OK, so that's the concept, but crucially that's not all the details. You still need to choose *how many* dummy votes you want to add and *what value* they should take. Since People on the Internet™ who disagree with your ranking will try to manipulate it in whatever way they can, sites are usually very cagey about said details. [IMDb used to be more transparent](https://en.wikipedia.org/wiki/IMDb#Rankings), [as was BGG](https://www.boardgamegeek.com/thread/103639/new-game-ranking-system), but now we have to dig a little deeper.
Let's start from the easier of the two, the value of the dummy votes. It is commonly chosen to represent some *prior mean*, i.e., some decent estimate of the rating a new game in the database would have. A frequent choice would be to use the average rating across *all* games. It's a fair assumption – without further information about a game, we don't know if it's any better or worse than the average game. However, Scott Alden actually gave away the answer in that interview from the beginning: BGG chose the dummy value to be \\(5.5\\). Their rationale is that ratings range from \\(1\\) through \\(10\\), so \\(5.5\\) is the midpoint. Of course, people tend to rather play and rate much more the games they like, and so the average rating is around \\(7\\). Opting for the lower value here is part of the design of the ranking: it means a new game would enter the ranking rather at the end of the pack. On the other hand, using the mean as the dummy value means a new game is placed more or less in the middle. It is worth mentioning that IMBd does use the mean (or at least used to), but they only ever publish the top 250 movies, and don't care about the crowd behind.
Let's start from the easier of the two, the value of the dummy votes. It is commonly chosen to represent some *prior mean*, i.e., some decent estimate of the rating a new game in the database would have. A frequent choice would be to use the average rating across *all* games. It's a fair assumption – without further information about a game, we don't know if it's any better or worse than the average game. However, Scott Alden actually gave away the answer in that interview from the beginning: BGG chose the dummy value to be **\\(5.5\\)**. Their rationale is that ratings range from \\(1\\) through \\(10\\), so \\(5.5\\) is the midpoint. Of course, people tend to rather play and rate much more the games they like, and so the average rating is around \\(7\\). Opting for the lower value here is part of the design of the ranking: it means a new game would enter the ranking rather at the end of the pack. On the other hand, using the mean as the dummy value means a new game is placed more or less in the middle. It is worth mentioning that IMBd does use the mean (or at least used to), but they only ever publish the top 250 movies, and don't care about the crowd behind.
The other value, the *number* of dummy votes, requires more work. Because some of the details and data are unknown, we cannot actually pin down the exact number that BGG is using. Instead, we'll try three different approaches, and compare their results.
......@@ -58,9 +58,11 @@ So, there's about \\(1830\\) dummy ratings, end of story. Right? Unfortunately,
And this plot is even cropped, the results vary from \\(-1.4\\) million to \\(+660\\) thousand.
<!-- TODO: add mean and median -->
What's going on, why are the results so inconsistent? The problem is the ranking's *secret sauce*. Both IMDb and BGG stress is that they only consider *regular* voters for their rankings. That's the most mysterious part of the system as it's the easiest to manipulate, so we'll just have to take their word for it. For this investigation it means that the average rating BGG publishes includes all the ratings, but the geek score might *not*.
Still, clearly something is happening around the \\(1500\\) ratings mark, so we are at least getting closer to an answer. If exact calculations won't work, maybe we can approximate the correct value instead?
Still, clearly something is happening around the **\\(1500\\) ratings** mark, so we are at least getting closer to an answer. If exact calculations won't work, maybe we can approximate the correct value instead?
# Trial & error
......@@ -70,11 +72,19 @@ What we can do now is fairly simply and quickly compute the rankings for differe
{{< img src="num_dummies_corr" alt="TODO" >}}
The best correlation of around \\(0.996\\) is achieved with \\(1489\\) dummy ratings. However, it is worth noticing that the changes in the correlation are very, *very* small throughout the range we examined here (\\(1000\\) to \\(2500\\)), so let's dig still a little deeper.
The best correlation of around \\(0.996\\) is achieved with **\\(1489\\) dummy ratings**. However, it is worth noticing that the changes in the correlation are very, *very* small throughout the range we examined here (\\(1000\\) to \\(2500\\)), so let's dig still a little deeper.
# Optimisation
# Optimization
What we have here at hand is actually a classic optimisation task: a real valued function in one unknown (or two if we allow a variable dummy value as well) which we'd like to maximise. This is a well-studied field, and with just a few lines of code we're up and running.[^notebook]
```python
TODO
```
Unsuprisingly, we get almost the same result as above: the best possible correlation is \\(0.996\\) with around **\\(1486\\) dummy ratings**.
But since we made it this far, let's take it one step further. So far, we tried to optimise the correlation in order to recreate BGG's ranking. However, we can also try to recreate the actual *geek scores*. That is, we can look for the number of dummy ratings that will yield the closest to the actual geek score with our calculations. What exactly we mean by "closest" is up to us to define. A common metric is the *mean squared error*.[^root] It's not worth getting into the maths here either, but the general idea is that we want to punish outliers in our estimates more (qudratically so) the further away they lie from the actual datapoint. Long story short, this yields a minimum for around **\\(1630\\) dummy ratings**.
# External resources
......@@ -82,3 +92,5 @@ TODO
* https://youtu.be/Y1t_0LhpDmU
[^jotl]: {{% game 291457 %}}Jaws of the Lion{{% /game %}} is something of an exception here and will undoubtably shoot into the BGG top 10 very soon. In fact, it might be the only game with the potential to unseat {{% game 174430 %}}Gloomhaven{{% /game %}} as the number 1.
[^notebook]: TODO
[^root]: It's probably even more common to use the *root* mean squared error, but for boring mathematical reasons, it doesn't make a difference when it comes to optimisation. In fact, we could even drop the word *mean* from our metric and still obtain the same optimal point, but then we'd have to implement it ourselves, so there's no point.
......@@ -32,10 +32,10 @@ z
<defs>
<path d="M 0 0
L 0 3.5
" id="mfefc5e0bfc" style="stroke:#000000;stroke-width:0.8;"/>
" id="m77f4fe6811" style="stroke:#000000;stroke-width:0.8;"/>
</defs>
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="69.218182" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="69.218182" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_1">
......@@ -87,7 +87,7 @@ z
<g id="xtick_2">
<g id="line2d_2">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="109.8" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="109.8" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_2">
......@@ -129,7 +129,7 @@ z
<g id="xtick_3">
<g id="line2d_3">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="150.381818" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="150.381818" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_3">
......@@ -164,7 +164,7 @@ z
<g id="xtick_4">
<g id="line2d_4">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="190.963636" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="190.963636" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_4">
......@@ -212,7 +212,7 @@ z
<g id="xtick_5">
<g id="line2d_5">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="231.545455" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="231.545455" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_5">
......@@ -269,7 +269,7 @@ z
<g id="xtick_6">
<g id="line2d_6">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="272.127273" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="272.127273" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_6">
......@@ -285,7 +285,7 @@ z
<g id="xtick_7">
<g id="line2d_7">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="312.709091" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="312.709091" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_7">
......@@ -301,7 +301,7 @@ z
<g id="xtick_8">
<g id="line2d_8">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="353.290909" xlink:href="#mfefc5e0bfc" y="252"/>
<use style="stroke:#000000;stroke-width:0.8;" x="353.290909" xlink:href="#m77f4fe6811" y="252"/>
</g>
</g>
<g id="text_8">
......@@ -321,10 +321,10 @@ z
<defs>
<path d="M 0 0
L -3.5 0
" id="me21733a940" style="stroke:#000000;stroke-width:0.8;"/>
" id="mcf9366d2ae" style="stroke:#000000;stroke-width:0.8;"/>
</defs>
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#me21733a940" y="228.166299"/>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#mcf9366d2ae" y="228.166299"/>
</g>
</g>
<g id="text_9">
......@@ -404,7 +404,7 @@ z
<g id="ytick_2">
<g id="line2d_10">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#me21733a940" y="194.934153"/>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#mcf9366d2ae" y="194.934153"/>
</g>
</g>
<g id="text_10">
......@@ -422,7 +422,7 @@ z
<g id="ytick_3">
<g id="line2d_11">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#me21733a940" y="161.702007"/>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#mcf9366d2ae" y="161.702007"/>
</g>
</g>
<g id="text_11">
......@@ -451,7 +451,7 @@ z
<g id="ytick_4">
<g id="line2d_12">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#me21733a940" y="128.469861"/>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#mcf9366d2ae" y="128.469861"/>
</g>
</g>
<g id="text_12">
......@@ -469,7 +469,7 @@ z
<g id="ytick_5">
<g id="line2d_13">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#me21733a940" y="95.237715"/>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#mcf9366d2ae" y="95.237715"/>
</g>
</g>
<g id="text_13">
......@@ -487,7 +487,7 @@ z
<g id="ytick_6">
<g id="line2d_14">
<g>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#me21733a940" y="62.005569"/>
<use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#mcf9366d2ae" y="62.005569"/>
</g>
</g>
<g id="text_14">
......@@ -504,7 +504,7 @@ z
</g>
</g>
<g id="line2d_15">
<path clip-path="url(#p7011128740)" d="M 69.218182 240.679539
<path clip-path="url(#p837225bdf6)" d="M 69.218182 240.679539
L 75.914182 206.691243
L 80.378182 186.004291
L 80.986909 183.507391
......@@ -786,7 +786,7 @@ L 388.8 34.56
</g>
</g>
<defs>
<clipPath id="p7011128740">
<clipPath id="p837225bdf6">
<rect height="217.44" width="334.8" x="54" y="34.56"/>
</clipPath>
</defs>
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment