Unlocking the Power of Average Models in Data Science
Written on
Chapter 1: Understanding Average Models
Did you enjoy your math and science lessons? Perhaps the equation Y = mx + c sounds familiar? This formula serves as the cornerstone of various models we encounter in both science and mathematics. Its essence is straightforward: events in our universe rarely occur without a cause. Therefore, we can define the Y variable as what we wish to observe, while the X variables represent the factors that influence or lead to Y.
Every time we seek to elucidate an observation by formulating hypotheses or presenting evidence, we are essentially constructing a model. This model doesn’t necessarily have to be mathematical or scientific. For instance, a statement like "Wealth can lead to psychopathy" is still a model (Y = psychopathy, X = wealth level).
That example, however, is rather extreme and challenging to substantiate. Today, we focus on models that are more conventional.
“Such a basic model!”
What exactly constitutes an average model? From a scientific standpoint, an average model involves aggregating the outcomes from various modeling techniques related to a specific topic and utilizing this collective average as the final answer. This technique is more precisely termed ensemble modeling.
Imagine participating in a machine learning competition tasked with predicting stock market trends. Instead of diving into the mathematics and market variables to create something innovative, you simply consult each competing team for their predictions. By averaging their forecasts, you submit this as your own prediction. Congratulations, you have just employed an ensemble model.
Interestingly, this approach frequently yields superior results compared to those who attempted to craft original algorithms. Even if you don't claim the top spot, your method usually lands comfortably above average in rankings. For instance, when comparing index funds, less than 20% of investors consistently outperform them over a decade.
You might consider this a coincidence, but extensive lab studies have led researchers to recognize that this seemingly "lazy" modeling approach often delivers better performance over time. Ensemble models typically exhibit lower variance and bias right from the start and can be easily refined without complex methods (for example, by integrating predictions with low correlation and assigning different weights based on confidence levels).
It seems somewhat unjust to suggest that the simplest algorithm, which merely aggregates the answers of its competitors and presents the average as its own, can outperform algorithms that painstakingly gather data and conduct thorough analyses. This contradicts the assumption that greater effort should yield better results.
But why do ensemble models consistently outperform specialized modeling strategies, especially in forecasting, despite their simplicity? Before delving into that, let’s briefly revisit the fundamentals of statistical modeling.
Section 1.1: The Basis of Statistical Modeling
Individuals build models for various reasons, with common goals including making predictions or assessing the influence of particular variables on outcomes. Before making any predictions, a model must establish a relationship using a set of X variables to determine the Y outcome. For example, predicting future sales based on historical sales data (Y = future sales, X = past sales, time).
Predictions derived from statistical modeling are inherently flawed due to the limitations of statistical techniques. One frequent issue is the inability of statistics to reliably identify causal relationships. Even when a causal link is established, accurately measuring the effect of X on Y—whether linear or nonlinear—can be challenging.
Moreover, statistical models require substantial amounts of high-quality historical data to be dependable. They also presuppose that the future will be driven by the same variables that explained the past. Misguided assumptions or poor-quality data can undermine the model’s accuracy. Additionally, real-world complexities mean that outcomes are often influenced by a multitude of variables.
Considering that many real-world relationships are dynamically interconnected, it becomes computationally impractical to incorporate every variable into every model. Furthermore, statistical models base long-term predictions on the assumption that their short-term predictions will hold true.
As a result, the confidence intervals of predictions tend to widen significantly over time. While forecasting for tomorrow might be straightforward, predicting events a decade ahead becomes nearly impossible.
Given the complexities of modeling, there is no singular correct approach. As the saying goes, "All models are wrong, but some are more useful than others."
Section 1.2: The Logic Behind Ensemble Models
Picture yourself as a journalist tasked with uncovering dirt on a local politician. To effectively report, you must track your subject and document any noteworthy actions. Following him everywhere isn’t practical, as he would likely notice and object.
To gather data effectively, you need a strategic plan. One method could involve installing cameras, but subtler approaches—like researching his schedule and frequent haunts—might yield better results. Timing is also crucial; for instance, a Saturday evening may not be ideal for catching him in action, but Monday at 10 AM could be fruitful.
After gathering this information, you wait to see if your predictions hold true. This illustrates the process of constructing and selecting variables for a predictive model.
Now, imagine you’re not alone in this endeavor. Your company has hired ten other journalists, each employing distinct methodologies and criteria for determining when and where noteworthy events might unfold. This creates a competitive environment among you to see who accurately predicts the politician’s actions.
In this scenario, the ten journalists symbolize ten different predictive models aiming for the same goal. This highlights that there is no single correct way to make predictions. Each journalist possesses their unique mental model for achieving the desired outcome.
Now, envision a new contender— a private investigator. Instead of starting from square one, he understands who already has the necessary information about the politician. Legally, he compels you and the other journalists to share your data and insights. He then utilizes this collective intelligence to formulate his own predictions about the politician's activities. This encapsulates the essence of ensemble modeling.
While this hypothetical scenario has its flaws (like assuming all journalists base their models on identical data), it illustrates that the private investigator is likely to have a better chance of success, even though you and your colleagues invested more time and effort into your models.
Chapter 2: Potential Pitfalls of Ensemble Models
Although the private investigator’s approach is clever, it’s not infallible. In reality, nothing about the future is certain. There will always be instances where one model outperforms others, and there’s no assurance that a given methodology will consistently yield superior results.
Ensemble models may falter under certain conditions. For example, if the predictions from various models are highly correlated, the investigator faces a dilemma. If all ten journalists provide similar answers, relying on multiple sources adds little to no predictive value. He would essentially be gathering the same information.
Additionally, if all models are of low quality, averaging poor predictions won’t yield valuable insights. The outcome of averaging faulty data remains faulty.
Another scenario where ensemble modeling struggles is when it includes models offering radically different predictions. If the private investigator consults too many journalists with conflicting opinions, deriving a coherent judgment becomes challenging, even if all the journalists are competent.
Ensemble Modeling in Everyday Life
You don't need to be a data scientist or AI specialist to apply ensemble modeling principles in your daily life. In reality, we engage in this practice frequently. Nature has long been the ultimate optimizer, and when research demonstrates that ensemble methods enhance rewards relative to risks, it’s clear that biology has been employing this strategy long before we formalized it.
Personal Growth
"You are the average of the five people you spend the most time with." This popular saying has circulated widely, and while there’s no exact science to back it, it holds practical wisdom. The idea is that we tend to become the average byproduct of our surroundings over time. Thus, it’s essential to actively seek out a positive environment or associate with high-quality individuals if we aspire to better ourselves.
If you wish to be a writer, surround yourself with writers. If you aspire to be an artist, engage with artists. If your goal is to become a master in any field, associate with those who excel in it. Conversely, if you prefer mediocrity, you might find yourself in the company of those who embody that.
Given the established reliability of ensemble models in controlled studies, if we assume our behavior adjusts to optimize learning from our relationships, it follows that we can leverage this effect to meet our personal growth objectives.
Pop Culture
Popular culture encompasses the dominant practices, beliefs, and objects recognized by a society at a given time. Its characteristics include accessibility, relatability, and minimal resistance to acceptance.
This premise explains why pop culture often produces content appealing to the lowest common denominator—think cute animals, wealth, and attractive individuals—while steering clear of polarizing or complex subjects like politics or advanced mathematics.
Additionally, pop culture tends to recycle ideas, rarely offering new perspectives or risks. It often opts for familiar messages that attract attention without pushing boundaries.
In this sense, pop culture can be viewed as an "ensemble" of cultural influences, optimized to maximize rewards relative to the risks involved in crafting its narratives.
Politics
Everyone has the right to an opinion, yet when contrasting views compete, they fall into a spectrum (e.g., left-right). Terms like far-left, left, centrist, right, and far-right illustrate this spectrum.
Individuals with opinions far from the center quickly find that expressing those views can lead to complications. Conversely, centrist opinions typically invite less scrutiny.
It’s no surprise that those holding extreme ideological views tend to be less vocal, as expressing unpopular opinions can pose existential threats. In contrast, voicing widely accepted beliefs tends to yield benefits.
The hallmark of ensemble models is their tendency to appear less erroneous. They often lack the conviction to take risks and instead seek a middle ground by integrating various viewpoints.
Mathematically, this approach has proven effective in research contexts and is frequently accepted as a means of making predictions or informed decisions. In controlled environments, many can compensate for inaccuracies. However, real-world situations often demand precise outcomes.
To appear less wrong, one may have to sacrifice the pursuit of being right. While ensemble forecasts yield generalized predictions, they rarely align perfectly with specific future scenarios. If you aim to excel in predicting outcomes, broad statements are your best strategy.
Interestingly, a common method to appear more accurate in predictions is to minimize the number of incorrect statements or make generalized predictions. As Gordon points out, it’s about the art of not appearing foolish.
Even in the absence of mathematical validation for the advantages of ensemble forecasting, many instinctively adopt this practice in their lives. Generally, these practices are harmless and often optimal (e.g., following senior advice, submitting average academic references).
However, Gordon cautions that when individuals prioritize superficial rewards (like appearances) by relying on average predictions, they may stifle the capacity to take significant risks and make bold hypotheses—actions often necessary for uncovering deeper truths.
In conclusion, simplistic debates and basic hypotheses do not advance knowledge. We must recognize the stark contrast between lab conditions and the real world, where our expectations differ significantly.
Closing Thoughts
While we are naturally inclined to play it safe and adopt basic strategies—since these often yield the best rewards relative to risks—our efforts to optimize survival can sometimes be misguided.
It’s human to fear being wrong, looking foolish, or losing status, as these fears can pose existential threats. Challenge authority and you may face dire consequences. Propose a revolutionary idea and risk being ostracized.
Yet, history teaches us that taking risks in our predictions and statements—while it may not be fatal—can lead to discoveries closer to the truth, ultimately providing more options for survival.
For instance, asserting that diabetes correlates with Alzheimer's is a safe statement. However, claiming diabetes causes Alzheimer's is riskier (and incorrect), but disproving that claim is crucial for scientific progress, as it allows us to explore new potential causes.
I believe we all possess the capacity for more. Let us dare to voice our thoughts, make predictions, and embrace the possibility of being wrong. Each misstep in our predictions is a chance to refine our models and hypotheses.
In the lab, basic approaches may yield rewards, but in life, it is those who dare to take risks who ultimately thrive.