Should have plead the 5th

I read this post a while back and was amused to see the same study show up over at Brothers Judd. I pointed out that counter analysis to Judd and he did not react well. Being a person of low moral character, I couldn’t resist taunting the prisoner, but I thought I would cross post here before it got memholed.

It winds up with Judd writing

It’s next to the picture of the alternate America where Nixon and Bush are Democrats and JFK and Clinton Republicans.

Your problem here is that ideology forces you to deny facts. It would be more fruitful to expand the anlysis. For instance, how can you implicate Democrats in the Nixon and Bush years and even in the current slowdown? Look at the Congress. Likewise, how can you claim credit for the Clinton boom? Again, check Congress.

I couldn’t resist replying

I see. You mean that claiming that Nixon governed like a liberal Democrat is something only a hard core ideologue would espouse and indicates a strong denial of facts? Well, clearly, it would be wrong to trust the political analysis of such a person.

I would also note that expanding the analysis is, IMHO, de facto debunking the original claim which rests entirely on the notion that the President is the most important factor in economic performance. I am happy to see you come around to my point of view.

This, Mr. Burnet, is why I have so honed my memory.

Sadly, I really should be working on my new parsing engine. Sigh.

Comments — Formatting by Textile

I keep meaning to write on this over at the Secret Blog, but life keeps getting in the way.

So far as it goes, OJ is right. If the question is, does the economy do better with Republican or Democratic presidents, then you can’t just start switching them around. Nor have I ever heard this supposed rule of “robustness” before and, from a statistical point of view, it’s silly. The only statistical argument that is even close is that a given point is an outlier and you should just remove it from your data, but even that needs to be justified.

The real answer to this silly study is that it is atheoretical data mining, that the effect is small and, most importantly, this is a misuse of regression because the data violates a few basic assumptions of regression. Statistics can’t prove causation. It doesn’t even imply causation. It is at least as likely that the state of the economy causes one party to win (actually, it’s a lot more likely), then that the causation works the other way round. That’s why you can’t just comb through the data till you find a significant relationship. You have to go in with a convincing theory and then show, with statistics, that it is not inconsistent with the data. That’s all this shows and since you can’t actually argue convincingly that the president causes the economy, “not inconsistent” doesn’t mean much.

Now, I ran the data for GDP on SPSS, a statistics package I use, and found a significant relationship between the President’s party and gdp growth. The relationship I found is about what they found — the GDP does about 1.5% better per year when there is a Democratic president, p=0.25 (all numbers are approximate because I’m writing this from memory)). What does that “p” mean? It means that, if annual gdp growth is normally distributed, independent, doesn’t have skew or kurtosis (don’t ask), doesn’t suffer from heteroschedasticity (seriously, don’t ask), then there is a 2.5% chance that these results could just be random assuming that there is no relationship between the president’s party and gdp growth. Since significance is generally set at anything under 5%, the results are significant.

However, we know that the data isn’t independent. That is, one year’s gdp growth is a good predictor of the next year’s gdp growth, as opposed to pulling numbers out of a hat, where one number ins’t a good predictor of gdp growth. So this data violates one of the assumptions of regression, which throws off the estimate of p. Also, when we set the threshold for significance at 5% (alpha=.05), we’re assuming that a study is being done to test a good, logical hypothesis. But, by definition, alpha=.05 means that if we just looked at a bunch of random numbers, we would find significance 5 times for every hundred regressions we did. So you can always find significance in data if you look hard enough (if you torture the data). So, if I were peer reviewing this article, I would argue that, with a violation of the assumption of independence and no strong logical ex ante hypothesis being tested, alpha=.05 is too loose a test and alpha should be set lower. The strictest way to deal with this is to divide .05 by the number of regressions you run, but that’s probably too strict. I would probably settle for alpha=.01, although I haven’t tested the data for skew, kurtosis, etc., and might lower alpha even further if I did.

Even accepting the results, I got an R-squared of .07. That means that the President’s party only accounts for 7% of the variability in gdp growth and, of course, 93% of the variability comes from other influences. That’s a small effect. Of course, none of this means that gdp will grow faster under Obama than under McCain. There are certainly Republican presidents that have higher growth rates than Democratic presidents and, whatever that other 93% is, it can obviously over come the effect (if any) of party affiliation.

But that brings us to the real point, which is that you can’t use ordinary regression for this question. The question presented, does the president’s party matter to the economy, is a multi-level question. We’re asking questions about party but we’re looking at presidents. In statistical terms, we’re ignoring that presidents are “nested” within parties. Also, as noted above, we’re ignoring the fact that the independent variable is a time series of measurements that are not independent of each other. Both of those facts makes ordinary regression useless.

I’ve already talked about the problem of dependence. The problem with nesting is that it confuses the number of degrees of freedom in your sample. Let’s think about testing a new teaching method. We test 100 kids in math. Then we train two of their four teachers in our new teaching method so half the kids get the benefit of our method and half the kids are a control group. We retest at the end of the year and find that the kids taught with the new method do significantly better than the control. Eureka!, right? Well, maybe not.

For reasons that I think are intuitively obvious, it is more likely that a given difference in scores is significant if you have a large sample rather than a small sample. If you ask 10 friends who they’re going to vote for, and 5 say Obama, 4 say McCain and one says she doesn’t know yet, that 5-4-1 split doesn’t tell you as much as if Gallup asks 1000 people and find that 50% are going to vote for Obama, 40% are going to vote for McCain and 10% are undecided. If you have a thousand people, a difference of a few hundredths of a point can be significant. With 10 people, you need a big difference to be significant.

With our new math method, we don’t really know whether the relevant number in our sample is the 100 students or the 4 teachers. The scores of the students in a classroom are not independent of each other; because they have the same teacher, the same facilities, the same class time, the same environment, etc., the scores of one kid in the classroom are a better predictor of the scores of other kids in the classroom than of kids in other classrooms. In other words, the scores aren’t independent. In fact, there are two levels of effects here, the individual level and the group (classroom) level. But if you give SPSS scores from 100 kids, it assumes that your sample is 100 and bases it’s tests on that number. If what really matters is the teacher, then the test SPSS uses to calculate the p value will be too loose. A difference between the subject kids and the control group kids that wouldn’t be significant if n=4 will be significant because SPSS assumes that n=100.

Getting back to presidents and gdp growth, we have data that we know is not independent from 60 years, 11 presidents and 2 parties. In ordinary regression, n=60 but we’re trying to make an inference about the 2 parties. Almost certainly, the test used was too loose and easily satisfied. In any event, it’s completely unreliable.

I do have a software program that is designed to deal with nested data and with time series data. It calculates the within group and between group variance separately. If I get around to running this analysis in that program, I’ll post the results over at the secret blog along with the results from SPSS.

Oh, and another thing. Leaving out relevant variables also skews the calculation towards leniency, so what ever it is (other than randomness) that accounts for the other 93% of variance has to be included to get a good calculation.

So far as it goes, OJ is right. If the question is, does the economy do better with Republican or Democratic presidents, then you can’t just start switching them around.

I am with AOG on this. For the terms “Republican” and “Democrat” to mean anything in economic terms, they each have to be proxies for distinct approaches to managing a national economy. Proposing wage and price controls does not qualify as “Republican”, no matter who does it. Similarly, where most Democrats stand on economic matters now (to the extent congresscritters of any party are separable in even the most trivial fashion from companion animals in this regard) would be unrecognizable to a Democrat of, say, 1970.

Calling Nixon a Republican on economic matters, simply because of his party affiliation, is a world class category mistake. Provided, that is, if we are interested in economic matters rather than party labels. AOG did an outstanding job hoisting OJ on his own petard. I am astonished AOGs posts haven’t been deleted, or otherwise vandalized.

While I am indebted to you, David, for a fascinating (absolutely no tongue in cheek, BTW) statistics discussion, you failed to include a couple seemingly significant factors.

Presumably, it makes some difference if the President and the houses of Congress are the same party, or not.

Also, time seems worthy of consideration. As insufficiently enamored of Senator Obama as I may be, I wouldn’t dream of pinning upon him the totality of economic circumstances come January 20, 2009, or a substantial portion of them a year later.

Regardless of statistical niceties, the failure to take those into account speaks volumes about the author, none good.

Or about OJ, for that matter.

Bret Monday, 20 October 2008 at 23:13 |

hey skipper wrote: “*I wouldn’t dream of pinning upon him [Obama] the totality of economic circumstances come January 20, 2009, or a substantial portion of them a year later.*”

Certainly not a totality, but I might, depending on what happens, pin a substantial portion on him. If he raises taxes in the face of a recession and/or increases regulation with rhetoric promising even more government intrusion and further reduced economic freedom and the economy does poorly, then the data would fit my models of the world and I’ll be quick to blame him. Will I have statistical justification? Nope, there won’t be enough data, but I won’t much care.

Annoying Old Guy Tuesday, 21 October 2008 at 08:19 |

Mr. Cohen;

The “switching a data point” rule is a heuristic for determining if your data set is to small, a point you make well. My primary claim was that the study was bunk, which you also agree with. I am, honestly, still not sure what OJ’s point was, other than it’s delusional, somehow, to examine history or data from a counter-factual point of view. I remain convinced that the most interesting part of the whole exchange is that I was unable to find any examples of OJ doing that, a technique that is as natural to me as calculating.

Post a comment