4min 50sec clip - Correlation vs Causation Explained: Why Patterns Can Mislead Us

Graphic - Patterns of use of audio media throughout a typical week by adults 2025

 

An earlier graphic posted showed the pattern of use of selected TV and video and gaming activities throughout the day by adults showed weekday peaks around 9am, noon and 9pm and on the weekend the peaks were the same am & pm but there wasn't a noon peak.

In the graphic above on audio the biggest radio peak weekdays is around 9am whereas the biggest streamed music peak weekdays is around 6pm.  On weekends both radio and streamed music seem to peak around noon.

Source of graphic above - Ofcom's Media Nations UK 2025 document published 30 July 2025

Patterns of technology domain combinations - overview

 

Source: https://www.weforum.org/publications/technology-convergence-report-2025/

Unveiling Hidden Patterns in Big Data: The Power of New Statistical Techniques

Introduction - The explosion of data in recent years has transformed the way we live, work, and make decisions. Big Data, characterized by vast volumes, high velocity, and varied varieties of information, holds immense potential for businesses, researchers, and policymakers. However, the sheer magnitude of Big Data can be overwhelming, making it challenging to extract meaningful insights. Thankfully, the advent of new statistical techniques has revolutionized our ability to effectively discover patterns in this sea of information.

Photo by Markus Spiske on Unsplash

Traditional Methods vs. New Statistical Techniques - Traditional statistical methods, while reliable, often fall short when dealing with Big Data. They were originally designed for smaller datasets and struggle to handle the complexity and scale of modern data streams. Enter new statistical techniques that have emerged to address these limitations.

Machine Learning Algorithms: Machine learning algorithms, such as deep learning and random forests, have become invaluable tools for uncovering patterns in Big Data. These algorithms can identify complex relationships and hidden patterns that traditional methods might miss. For example, in the field of healthcare, machine learning models can analyze vast patient data to predict disease outbreaks, improve treatment outcomes, and identify potential areas for cost reduction.

Data Preprocessing: New techniques in data preprocessing help cleanse and transform raw data into a usable format. This includes handling missing values, scaling features, and encoding categorical variables. These steps are crucial for improving the performance of machine learning models and extracting meaningful patterns from the data.

Dimensionality Reduction: Dimensionality reduction techniques like Principal Component Analysis (PCA) and t-distributed Stochastic Neighbor Embedding (t-SNE) enable us to reduce the complexity of high-dimensional datasets. By visualizing and analyzing data in lower dimensions, researchers can more effectively identify patterns and clusters.

Advanced Sampling Methods: Traditional random sampling techniques may not be efficient for Big Data. New methods like stratified sampling and bootstrapping allow researchers to obtain representative subsets of data for analysis, reducing computational load without sacrificing accuracy.

Time Series Analysis: In domains like finance, climate science, and IoT, new statistical techniques for time series analysis have emerged. These methods can detect trends, seasonality, and anomalies in time-stamped data, helping organizations make data-driven decisions.

Bayesian Inference: Bayesian methods provide a powerful framework for estimating parameters and making predictions in Big Data scenarios. They are particularly useful when dealing with uncertainty and can be applied to various fields, from marketing to astrophysics.

Challenges and Considerations - While these new statistical techniques offer tremendous promise, they come with their own set of challenges. Some of these challenges include:

Computational Resources: Analyzing Big Data often requires substantial computational resources, including high-performance computing clusters and GPUs.

Data Privacy and Ethics: As data collection and analysis become more sophisticated, ethical considerations around data privacy and security become increasingly important.

Interpretability: Some machine learning models, such as deep neural networks, can be difficult to interpret. Researchers must strike a balance between accuracy and interpretability when applying these techniques.

Conclusion - The ability to effectively discover patterns in Big Data is essential for making informed decisions in various fields. New statistical techniques and machine learning algorithms have proven to be indispensable tools for this purpose, enabling us to unlock valuable insights from the vast sea of information. As these techniques continue to evolve, they will play an even more significant role in shaping our data-driven future. However, it is crucial to approach Big Data analysis with care, considering computational resources, ethics, and interpretability to ensure that the insights gained are both meaningful and responsible.

Source: I asked Chat GPT3.5 to "write a short article about how new statistical techniques cam more effectively discover patterns in big data" and then made some minor formatting changes and added a picture

1 min clip - Anywhere can happen

found via a laughingsquid post

critical thinking no.5: the gambler's fallacy

script by mike mcrae and james hutson, recording and music by audrey studios, animated and directed by James Hutson, produced by Bridge 8, found via brainpickings 

in summary

- if you watch a coin flip 9 times - it comes up heads 2 times and tails 7 times

- what is going to come up next?

- as tails has been having a good run maybe that will continue - or because of that maybe it is time for heads?

- our brains are very good at recognising patterns - sometime when they are not there

- in fact there is a 50% chance of heads or tails every time the coin is flipped - it doesn't matter what came before and luck doesn't come into it at all

- but it is hard to shake the idea that there is a pattern in there somewhere - maybe we just need to look hard enough - this is known as the gambler's fallacy (our assumption that probability changes as a result of past events)

- we are wired to see things as if they are related 

- so for example - people take pills and feel better - but a lot of logic and probability is needed to determine whether the pills were truly responsible

- just because one thing follows another it does not mean they are linked - there could be other factors or it could just be coincidence

- to know for sure you have to test the circumstances again and again  - to look for those other factors that could disprove the link-  this reinforces confidence that your pattern is true

- our brains may see patterns - and this is often useful - but it takes science to prove that those patterns are real