When I was a young man taking A-Level Maths, the subject that I liked the least was statistics. It seemed to be to be arbitrary and messy, especially when compared to the coherent and definite world of pure mathematics.
Recently, I have been studying statistics at a deeper level, and I have come to recognise the value of statistics across many domains, and it’s inherent and crucial role in mathematics.
During A-Level Mathematics, statistics formulas are given without a complete explanation of how they were discovered or what they mean. We were told to calculate the standard deviation, or the variance or R squared. We were given the formulas and told what meaning we should infer from the results. But there was no depth. It all just seemed completely arbitrary to me.
Conversely, when studying pure mathematics at A-Level, it felt like it was anything but arbitrary. All the theories seemed to fit together in a coherent way. Everything followed the same rules and I could see what was going on in the underlying maths.
I actually understand why A-Level maths is taught this way. The proofs of the theories of statistics are probably too difficult and involved for the A-Level student. It makes sense to give the students the tools they need, without telling them how to make those tools. But it does leave the student feeling as though the tools are a little bit random.
Go A Little Deeper
Now I am reading text called “Grinstead and Snell’s Introduction To Probability”. This text describes, demonstrates and proves each formula and theorem as it goes. The authors also explain the historical context behind the development of each theory, which goes a step further in explaining the necessity of the theorems and their use.
At the end of each chapter, there are plenty of problem exercises, which go over the material presented in the chapter, and also stretch the theory to its limits. So you need to really push your understanding in order to answer the harder questions.
Overall it’s a very satisfying way to learn. I feel like I’m actually learning the basis of probability theory, and it’s anything but arbitrary.
Importance Of Statistics
I’ve also come to recognise the importance of statistics in understanding the world around us. I used to think that statistics was the boring bit that you did at the end of each experiment or study, just to prove that your results actually mean something.
But now I realise that proving that your results actually mean something is of course the most important aspect of any study. And if you can understand and bring statistical theory into the heart of your work then you can produce much better research.
Statistical Basis of Emergent Phenomena
I’ve also been thinking recently on emergence, or how large-scale patterns emerge from the interactions of many small parts. So for example how birds flock. It turns out emergence happens at every scale of the material world, from thermodynamics to galaxy formation. I have started to think that the basis of emergence is statistical i.e that the emergent patterns at the large scale are the statistical outcome of many interactions on the low scale.
Look at an area where statistics is commonly used, say gambling. We can see that the profits that a casino makes emerge from the interactions of many small bets. You could perform a statistical analysis on those bets, and casinos do, in order to accurately predict the casino’s profits.
When we look at things in the world that appear to be deterministic and very predictable, such as the ideal gas law. It turns out that this apparently deterministic law is actually statistical in nature, but due to the very high number of molecules combining to create the emergent ideal gas law, the probability of the gas behaving outside of the range of what we expect is vanishingly small. This isn’t just conjecture on my part, for example using statistical mechanics you can actually determines the ideal gas law from a statistical treatment of individual gas molecules.
So the 17 year old me was very wrong. Statistics is not arbitrary, and it is not boring. It is in fact an exciting area of mathematics borne out of pragmatism and necessity. It is the language of the material world, the language of thermodynamics and it may just be the language of emergence.