Once in a while I stumble upon something truly epic.

Image Courtesy of xkcd.com

This week, I’ve been lucky to see more than one example of people dropping the knowledge in a most delightful way.

So in honor of my 100 thousandth view (and before I go off to do the dishes), I’m dropping some links.

First off, thanks to Andres the Podcast now has a page.

Ty drops a great piece on the 72 win Chicago Bulls and how Rodman made them awesome.

DJ argues that the data points to the league getting better rather than the Lakers getting worse.

And finally Alex at SportsSkeptic drops an epic statistical beatdown in Illustrating Rosenbaum’s Folly. I’m actually jealous.

### Like this:

Like Loading...

*Related*

nerdnumbers

01/06/2011

Congrats on the 100K Arturo! Would you have believed me if I had told you a year ago you’d start a blog that would be linked by Dave Berri, Ted Leonsis, ESPN and Yahoo Sports?

arturogalletti

01/06/2011

Dre,

Not really. I started this as a personal growth exercise. I wanted to write and do analysis on a regular basis and I was hoping that the reps would make me better. The attention has just been a very nice bonus (and perhaps an indication that I’ve achieved my initial goals?).

You and Prof. Berri deserve a lot of the credit for this. You for providing the tools and DJ for letting me borrow his forum repeatedly (and giving me pub). I’m thanking you both publicly again 🙂

Guy

01/06/2011

Arturo: I’m surprised you would link to Sports Skeptics’ post on the Rosenbaum-Lewin paper. he provides a very bad misreading — almost willfully so — of that paper. Here is what I said at his site. Hopefully Alex will take the time to read the paper carefully, and post a correction at some point.

Alex: First, your core point that adding a team adjustment makes every metric equally good at predicting team wins is totally understood by Rosenbaum/Lewin (R/L). Indeed, that is exactly the point they were making! How do I know this? Because they say it very clearly: “Because the team adjustment forces each of the player metrics to add up to team efficiency, how well player metrics aggregated by team explain current team wins tells us nothing more than how well team efficiency explains team wins. Explaining current team wins cannot be used to evaluate player metrics; all player evaluation metrics with team adjustments will explain current team wins equally well.” (Perhaps you didn’t make it to page 7, and that explains your confusion?) Their point is that if a metric sums to team point differential, as WP does, it will explain differential nearly perfectly and explain 95% of wins. So R-L add a team adjustment to put every metric on a level playing field initially.

But they do not then compare the metrics in that initial year (there would be no point to doing that). The clearest indication you didn’t understand the paper is when you write about your simulation: “despite no players being traded or hurt or what have you.” The whole point of (R/L) is to see how well metrics do at predicting FUTURE wins, after team composition changes. That is why they look 2 and 3 years into the future, to maximize the amount of change in team rosters. While the team adjustments make all metrics appear equally good in year N, it doesn’t ensure parity once team composition starts to change in year N+1, N+2, and N+3. At that point, you start to find out which metrics have the strongest “signal” of individual players’ true productivity. If a metric can explain only year N well, but not later years when team composition changes, then it isn’t dividing up credit for team wins accurately. And of course, what they find is that Wins Produced does a terrible job of predicting future wins. This is the whole point of the paper, and you don’t even address it or seem to understand it.

It’s of course true that when R-L add a team adjustment to PER or Efficiency, they may make those metrics better than they currently are. But the team adjustment is very crude: it just allocates everything the underlying metric failed to explain to players based on their MP, as WP sort of does with opponent shooting efficiency. (And of course it has predictive value in your model added to a junk stat — it’s a partial measure of last year’s efficiency, and you don’t change personnel at all. Why wouldn’t it be predicitive?) If WP is doing a better job of estimating the true value of boxscore stats than PER or EFF, then WP should still do a better job of predicting future wins even when all metrics have the benefit of a team adjustment. But it doesn’t. Which tells us that WP does a very poor job of identifying players’ actual contributions to winning.

arturogalletti

01/06/2011

Guy,

There is a very sound and extremely well understood methodology behind building statistical models. Alex clearly understands that methodology, Rosembaum-Lewin clearly do not. The model described by R/L is the equivalent of the me taking a teams wins (or point margin so far) and assigning to each player on the team based on the percentage of the teams minutes played. Sure it’ll line up to the current season perfectly but the predictive value will be crap season to season. It’s as Alex says a bad use of statistics.

This is a piece, I meant to write. Alex beat me to it though. Damn him !!!! 🙂

Guy

01/06/2011

Arturo: You should not be jealous of someone who has written a post demonstrating that he totally misunderstood the paper. I think there are valid criticisms one can make of this paper. Or one could propose alternative ways to test the predictive power of these metrics. But Alex didn’t do any of that. He provides a thru the looking glass, up-is-down analysis that shows unambiguously that the whole paper was over his head. His whole post was a proof of Rosenbaum’s basic claim — that every metric will predict current wins — but Alex thinks he wrote a rebuttal! It’s just unbelievable. And I can assure you that Dan Rosenbaum’s knowledge of statistics and models is far greater and deeper than Alex’s, or yours, or mine (and David Berri’s, for that matter).

And your analogy could not be more ironic, since what the R-L paper shows is that the extremely crude model you describe — simply apportioning wins based on players’ minutes played — actually did a better job of predicting future team wins than Wins Produced! Which probably in part explains why the WOW network forecasts are proving so weak this year.

arturogalletti

01/06/2011

Guy,

You really don’t get it do you?

Guy

01/07/2011

Arturo: We have really passed through the looking glass now. Or, since you’re obviously a comics guy (me too), perhaps I should say we’re in Bizarro world. Somedude has it right: Alex confirmed Rosenbaum’s analysis, then calls it a rebuttal. But there isn’t anything he said that R-L would disagree with, or didn’t already say in that paper. So how can that possibly be a rebuttal?

I suggest you re-read the paper, with an open mind, and try to understand what they’re doing. It is a challenging and unconventional methodology, and there’s certainly room for fair criticism of it. But if you think Alex hit the mark, then you simply didn’t understand the paper. (And while you’re reading it, be sure to check out Table 3, which shows the extremely high correlation between WP and player rebounds. Since we know for sure that wins and rebounds are not in fact highly correlated, that should give you some hint of a problem.)

arturogalletti

01/07/2011

Guy,

I understand it perfectly. Alex is very much correct.

Not highly correlated? For team stats it’s opponent assists, opp. pts, opp. FGM then Defensive rebounds.

Guy

01/08/2011

Arturo: After posting a link to Wikipedia’s explanation of multicolinearlity, you give us these raw correlations? Really? Defensive rebounds are highly correlated with opponent missed shots (which are very important). So of course they are also correlated with wins. If you use dreb%, and put it in a regression with shooting efficiency, turnover, and fouls, you will find that dreb% plays a relatively small role in winning. Or you can just read EvanZ’s excellent posts on the Four Factors over at thecity2 dot com and see the answer. Rebounding over all determines about 15% of wins in the NBA. Rebounding matters, but it matters about as much as fouls, less than turnovers, and far, far less than shooting efficiency. And by the way, team Wins Produced tells the exact same story. It is only when you look at player WP that we find the illusion that rebounds are tremendously important.

EvanZ

01/09/2011

I can’t believe that Arturo continues to cite WP “predicts” 95% of team wins.

*ANY* model (including mine) that accounts for point differential will also account for wins! It’s not prediction – it’s ACCOUNTING. Why? Because point differential is 95% correlated with wins. I would never claim that my model “predicts” wins.

Prediction would be taking last year’s WP48 for Cleveland and projecting that they would have won 8 games at this point of the season. My model doesn’t do that, and I sure as hell know that WP doesn’t do that. Not even close.

Yet, I assume Arturo will tell us that WP “predicts” that Cleveland has won 6.8 games this season, based on the data from THIS season. Once again, that is not prediction. It’s ACCOUNTING.

some dude

01/09/2011

excellent point, EvanZ. Accounting was the word on the tip of my tongue last night in my “not all there” state.

WP does accounting, not predicting.

arturogalletti

01/09/2011

Evan,

There’s a saying in spanish that says you’re mixing gymnasium and magnesium.

The goal of a model is to predict what it’s modeling. All the models (+/-, WS and WP) correlate to wins but predictive ability has to do with year to year consistency at the player level. If we compare year to year numbers for those models WP does the best and is thus most predictive.It’s not perfect and there are opportunities. Which is what we continue to work on.

You’re entitled to your opinion and your model. I’m entitled to disagree. I’m also entitled to point out unsound models and math. And as Alex very rightly points out that study is very much that.

some dude

01/10/2011

No, it really doesn’t. You’re not predicting when you’re inputing data into the model it’s supposed to predict.

As I said, you’re comparing a poll on voters to predict a vote vs. a poll on voters with known absentee results to predict the vote.

If you want to make it make more sense, compare it without the team adjustments.

Another approach would be to take a projected WP48 for all players based on data from 2008-09 and before to predict the 2010 season wins. Then multiply the projection by actual 2010 minutes and see what it says. Now do that for every season going back. If that has a 95% correlation, I’ll be shocked.

Just look at some data from this year. It doesn’t make sense. Take a look at the lakers.

Fisher-Kobe-Artest-Odom-Gasol lineup is +9 per 100 possessions.

Fisher-Kobe-Barnes-Odom-Gasol lineup is -20 per 100 possessions.

According to WP Barnes is a star and Artest is just average. If this were true, why are the PD splits so different when using the same units? According to this season’s WP numbers, the latter lineup should be far superior to the former, but we see the opposite play out. Yeah, because there is a team adjustment, once you measure the team as a whole it will look pretty. But WP is not predicting how it’s happening accurately at all. if it were, Ron’s WP numbers should at the least be near Barnes (or really ahead of his).

And Barnes is twice the rebounder than Artest is. yet, the Artest lineup above is the better rebounding unit (52%) and the Barnes one is below average (48%).

Then you made a post trying to capture individual defense and adjusted your WP finding and it completely altered the WP rankings! And despite your own work demonstrating this, you agree with an article by Alex which not only confirms what he’s trying to refute, but his conclusion undermines your own work. I just don’t get your position.

Anyway, Wins Produced for 5 man units would make far more sense in an attempt to predict outcomes. That may be where this metric has potential.

arturogalletti

01/10/2011

SD,

We just see things differently is all. I don’t let it bother me.

bduran

01/10/2011

SD,

I went to 82games.com to look at the minutes played for the lineup you listed. One lineup played 594 minutes, the other 83. Probably not a great example.

some dude

01/09/2011

Wins Produced adds in a team adjustment for defense, so of course it will do better than a PER without a team adjustment. And as Alex pointed out, putting in a team adjustment in PER makes it just as predictable as WP!

Not to mention, taking a metric, like WP, with a team adjustment and then using that team adjustment to “predict” games in which the team adjustment is already accounted for is ridiculous.

You’re saying PER is worse at predicting wins than Wins Produced while ignoring the fact that wins produced is including KNOWN DATA into its prediction while PER has none of it. That would be like comparing a Gallup poll on an election based on pure sampling against a Rasmussen poll which includes known votes from absentee ballots already counted. It’s an unfair comparison. WP shouldn’t have the team adjustment in predicting because when you do, it’s no longer predicting everything.

Furthermore, I didn’t see a mention of DReb. I saw total rebounds and that is very poor according to your own list. And DReb suffers from OVB, which Guy describes.

Remember, correlation doesn’t always prove something.

bduran

01/10/2011

SD,

“if you apply the team adjustment actually used by WP to other metrics, like PER or NBA Efficiency, they still never reach the descriptive power of WP”

some dude

01/10/2011

because you’re adding the same team adjustment given to WP, which is nonsensical.

Alex’s post proves that a team adjustment done for each individual stat separately gets it to 95%. The predictive power is the same if you replicate the team adjustment process, not copying the WP one.

As for the 83 minute claim, arturo or another blogger on this network once argued it’s statistically significant enough. Even if we needed to get to 100 minutes to be there, the splits are so far apart that random noise cannot explain the issue.

Just go to last year with the Lakers. Both Pau and Odom were bigger producers during the regular season according to WP, but the Gasol-Bynum starting 5 was more productive than Fisher, Kobe, and Artest playing with Gasol and Odom or Odom and Bynum. And it wasn’t close, with the least used lineup at 398 minutes. According to WP, the Lakers should have played Odom more than Bynum, but the actual results contradict that.

arturogalletti

01/11/2011

sd,

It’s called sarcasm. If you work out the error for any model and add it back in you’ll see improvement. You’ll also get people looking at you funny.

The adjustments in WP for the teams are based on the teams actual stats not on a fudge factor to get the error corrected. And Win Shares does something similar but different (more like my combined WP number). +/- does adjustment at the player level (but there I’m not convinced the isolation is done right which is why the year to year correlation isn’t as good). All those models have their strengths and weakness but to Alex’s point (and mine) they come from a sound statistical base.

bduran

01/11/2011

SD,

Arturo said it exactly. There’s a difference between what WP does and what the paper did for the other metrics. I also seem to remember that Berri claims the team adjustment is pretty small ( my book is currently at a friends). Those other metrics required large adjustments to correlate with pont differential. I would be more interested in the same study with WPs position and team adjustment removed.

Also, I don’t think Arturo every claimed 83 minutes was enough for +-. I think it was claimed that 83 minutes was enough to get a decent correlation with end of the season results for WP, which is much more consistent. I would imagine that +- requires much more time than most box score metrics to correlate well with the whole season.

Lastly, you can’t dispprove a statistical measure by counter example. So the fact that you and Guy keep trying to point out specific examples may be interesting, but doesn’t really answer any questions.

some dude

01/11/2011

Arturo. That’s the entire point. The adjustment is using the actual stats. If you’re using the actual stats, then you’re not predicting.

You can’t throw in a team adjustment on current stats and then claim it’s predicting. And the team adjustment is called small by berri, but that’s his opinion and I disagree with that claim. Big and small are subjective claims.

Why won’t anyone run predictions based on WP projections (without current team adjustments) and multiply by current minutes. It’s called the scientific method. I suggest using it. WP is testable, so test it properly.

arturogalletti

01/12/2011

SD,

I’m thinking you don’t understand the adjustments. The two adjusments in the metrics are meant to account for the effect of pace and defense for a particular team. So if you play in Phoenix your stats will get dropped to account for the fast played pace. If you play in Boston the same thing happens for D. The adjustment is meant to provide a standard basis for comparison and it’s not a large number.

The real issue is the opponent adjustment. The assumption is that players are playing league average opponents adjusted for team D. This is were the real opportunities lie. This is what for example to me explains the Lebron- Cleveland situation in that we undervalued Lebron’s value in shutting down his opponent.

bduran

01/12/2011

SD,

As for +-, according to basketball value and adjusted +-, Lamar and Gasol are the best players on the Lakers. Trying to find one example of a player in just a couple of lineups using a highly variable metric to contradict WP is probably not a good idea.

BTW, I don’t think anyone actually believe’s that WP is going to “predict” the results of the next game any better than point differential so I’m not sure why people keep bringing it up. This is why Berri uses efficiency differential in the Truehoop Geek smackdown, because team WP is the same thing essentially.

I do agree that if WP is adding value to our analysis then we should be able to use it to predict the next season better than just using the previous seasons point differential.

arturogalletti

01/12/2011

bduran,

Except WP is point differential mapped to the players. That was the point of my point margin equation. The experience of mapping the whole league has shown (to me at least) that :

1.Refinement is needed on the opponent D taking it down to the player level (See James,Lebron)

2. Injury analysis and recovery curves are required. (See Brand,Elton)

3. Additional work on rookie analysis is needed (but we knew that)

4. Scouting and Insider access is a key advantage for handicaping (see Roy,Brandon)

And i’ll keep working at developing these tools.

some dude

01/06/2011

Arturo, I read his post after you linked it last night. It was late. I read some parts over again.

When I finished, I said to myself “I don’t understand. He is writing a post trying to claim Rosembaum is wrong, then spends the entire post saying Rosembaum is right, then concludes he’s wrong. huh? Maybe I need to sleep and re-read this.”

Apparently I was right. Guy saw it too. Alex’s seems to have miss understood what he was in fact doing. His entire post was validating Rosembaum’s paper, not refuting it. I don’t get how he thought otherwise.

Jon

01/06/2011

And expanding on the predicting future wins theme, all of this year’s WoW based predictions are still severely lagging behind the accuracy of others. Right now John Hollinger and the Sports Guy are the only 2 who’s predictions i have being slightly better than the opening over/under Las Vegas lines. Kelly Dwyer of Yahoo and the ESPN aggregate predictions are also tracking very closely to the Las Vegas accuracy. Then come models based on Win Shares and Adjusted Plus Minus from BR, followed by Bradford Doolittle’s NBAPET Model. Only then, with error and correlations much much worse than Hollinger/Sports Guy, come the predictions from Arturo, Ty, etc. To be specific, Hollinger’s predictions have an 85% r to actual right now, Simmons is at 84%. Arturo and Ty are at 72%. The total residual from Arturo/Ty’s predictions is 20% higher than that of the Hollinger/Simmons predictions (a massive amount imo). Though they are still ahead of SCHOENE and its epic prediction that NJ would finish within 2 games of Boston. But yet we can look forward to hundreds more articles on these sites about how delusional the likes of Simmons and even Hollinger are.

arturogalletti

01/06/2011

Jon,

I’ve never said Simmons is delusional . He’s actually well documented as a brilliant handicapper for the NBA. I’ve been a fan of the man since the Boston Sports Guy days.

My issue with Hollinger isn’t with the predictions (he uses differential and +/- for that) it’s with him trumpeting PER as a predictive statistic (it isn’t).

As for the predictions, this was meant (and has been) a learning and growth exercise. I didn’t expect to have my dyi model work from the start. The point was to get it done and look at the sources of error. I’ll keep at it and take my lumps accordingly.

ilikeflowers

01/06/2011

relevant comment thread

tl;dr

Given Model D = (Model A + Model B + Model C)

It does not follow that Model D Y U NO GOOD? = Model A Y U NO GOOD?

some dude

01/06/2011

I didn’t use any statisitical models and I’m doing better than WOW too. 😀

Neal Frazier

01/07/2011

Its been a while since I have seen a smack-down update – have the results flipped from the first few weeks when the WOW models were all in the lead? Where can I find the current standings in the smack-down?

MattB

01/07/2011

Guy,

I feel like I read your stuff more than any other blog (just through comments here and elsewhere).

Do you write regularly somewhere? If not, can you start? I really think you guys are all fascinating and offer a lot of insight.

Guy

01/08/2011

Thanks, Matt. Ironically enough, Dre has made the same suggestion (though I have the sneaking suspicion that is so I will go away). Maybe some day. Mainly, I hang out at Tom Tango’s The Book blog, which I highly recommend even if you’re not a big baseball fan. Lots of good discussion of all kinds of sports statistics issues there.