I think speculation and guesswork is perfectly fine. It’s part of a path towards an answer. However, that speculation and guesswork needs to have its uncertainty clearly indicated.
I'll give an example using football.
As “analytics” have emerged, everyone has their own model to give a guideline on decisions. This is done using things like “win probability” of all the possible choices and outcomes. You can do out the math, using a model, to say something like “going for it gives you a 35% chance to win, and kicking the field goal gives you a 33% chance”.
And that sounds great. But, all the numbers that go into that math are incredibly noisy, with very small sample sizes. A great kicker has a better chance of making a field goal than a bad kicker, and they can account for that, to a point. But they can’t really account for that, plus the specific weather conditions, plus the kicker is a little sore today, …
And the chances of a stop, and of scoring if you’re successful, etc, are even worse, because it’s specific to how your offense matches up to that defense, plus the context of the game, the context in the game/moment, etc.
It’s perfectly fine, and reasonable, to use a model as the best indicator you have and make a decision aided by that model. But the correct way to present statistical models is provide some guidance on how uncertain it is, in addition to the raw number. If you phrase that “35% +/- 10% if you go for it, 33% +/- 10% if you kick”, you realize that there’s a significant range where a better model might tell you to make the opposite decision, and it’s a lot closer to a toss up.
But despite the inherent uncertainty due to the limited sample sizes used to create the models, you see “analytics experts” all over the place calling coaches morons for decisions that are pretty ambiguous because their specific model gives one decision a small edge and it didn’t work out. If they had explicitly evaluated and acknowledged the uncertainty of their model given the factors it can’t account for, they would have a much clearer picture of what the decision actually was.
Make guesses. Speculate. But make it clear (to others, and yourself) what you’re doing so the guesses aren’t given more weight than they deserve.
I think speculation and guesswork is perfectly fine. It’s part of a path towards an answer. However, that speculation and guesswork needs to have its uncertainty clearly indicated.
I'll give an example using football.
As “analytics” have emerged, everyone has their own model to give a guideline on decisions. This is done using things like “win probability” of all the possible choices and outcomes. You can do out the math, using a model, to say something like “going for it gives you a 35% chance to win, and kicking the field goal gives you a 33% chance”.
And that sounds great. But, all the numbers that go into that math are incredibly noisy, with very small sample sizes. A great kicker has a better chance of making a field goal than a bad kicker, and they can account for that, to a point. But they can’t really account for that, plus the specific weather conditions, plus the kicker is a little sore today, …
And the chances of a stop, and of scoring if you’re successful, etc, are even worse, because it’s specific to how your offense matches up to that defense, plus the context of the game, the context in the game/moment, etc.
It’s perfectly fine, and reasonable, to use a model as the best indicator you have and make a decision aided by that model. But the correct way to present statistical models is provide some guidance on how uncertain it is, in addition to the raw number. If you phrase that “35% +/- 10% if you go for it, 33% +/- 10% if you kick”, you realize that there’s a significant range where a better model might tell you to make the opposite decision, and it’s a lot closer to a toss up.
But despite the inherent uncertainty due to the limited sample sizes used to create the models, you see “analytics experts” all over the place calling coaches morons for decisions that are pretty ambiguous because their specific model gives one decision a small edge and it didn’t work out. If they had explicitly evaluated and acknowledged the uncertainty of their model given the factors it can’t account for, they would have a much clearer picture of what the decision actually was.
Make guesses. Speculate. But make it clear (to others, and yourself) what you’re doing so the guesses aren’t given more weight than they deserve.