Whose account of the response to Michael Mann's original "hockey stick" analysis is correct?

Question

New York has filed a lawsuit against several Big Oil firms on the grounds that they fomented contrarian opinion to undermine some findings in climate science.

In one part of the suit they claim the following (paragraph 86, my emphasis):

In the early 2000s, Exxon again attacked a respected scientist, Dr. Michael Mann. Dr. Mann had published a paper in peer-reviewed literature of what has come to be known as the “hockey stick” graph, which shows modern temperature sharply diverting from the temperatures of the last 1,000 years, and which was relied on by the IPCC in its 2001 report for its strengthened finding that humans were causing global warming, a report in which Exxon scientists participated. In response to the IPCC’s causal finding, Exxon sponsored its own bogus scientific research by paying $120,000 over the course of two years (2003–2004) to the Fraser Institute, a Canadian organization that specializes in climate denialism. Senior Fraser Institute Fellow Dr. Ross McKitrick and a co-author then published a supposed refutation of Dr. Mann’s “hockey stick” graph. Dr. McKitrick was an economist, not a scientist, and his co-author was a mining company executive. In 2003, the McIntyre and McKitrick paper was rushed into print, without peer review and, in a departure from the standard scientific practice, without offering Dr. Mann and his co-authors an opportunity to respond prior to publication. The McIntyre and McKitrick paper was subsequently debunked, but the smear of Dr. Mann’s work remains available on the web today and continues to be cited by climate deniers. Exxon’s promotion by deception thus lives on.

McKitrick responded by denying many of the claims. in particular he characterised the scientific response the the Mann paper like this:

The claim that our paper was “subsequently debunked” is an empty and worthless attempt to deceive the court as to the disposition of a debate that went on for several more years. It is notable that the assertion depends entirely on a citation to a blog post on Mann's own website and omits reference to any independent source. A more accurate summary of the ensuing events would be something like the following.

After publishing their 2003 E&E article and reviewing Mann's unpublished responses to it, McIntyre and McKitrick submitted an extended critique of the errors and misrepresentations in MBH98 to Nature magazine, which had published the first of the hockey stick papers. Nature solicited a response from Mann et al., and after examining it they ordered Mann et al. to publish a detailed correction and restatement of their methodology, which appeared in June 2006. M&M also extended their critique of Mann's statistical methodology and submitted it to GRL, which had published the 2nd hockey stick paper, and after peer review GRL published their study. Mann et al. never submitted a response. A panel led by Professor Wegman later conducted an independent review of the mathematical and statistical issues and upheld the M&M critique. A panel of the National Academy of Sciences also conducted an examination of the whole issue of paleoclimate reconstructions and upheld all the technical criticisms M&M made of Mann's work, going so far as to publish their own replication (North et al., 2006, pp. 90-91) of the spurious hockey stick effect M&M identified.

The M&M work remains “on the web” in large part because it is in print in peer reviewed journals. It is widely cited, including by the IPCC and the NAS, and in scientific studies such as: McShane, B.B. and A.J. Wyner, (2011). “A Statistical Analysis of Multiple Temperature Proxies: Are reconstructions of surface temperatures over the last 1000 years reliable?” Annals of Applied Statistics Volume 5, Number 1 (2011), 5-44.

It wouldn't be appropriate to test the whole set of claims on both sides here. Let's stick with the differing accounts of the Hockey Stick paper's history as it has been widely discussed and debated. Which one is more accurate (I'm acknowledging that neither may be the unvarnished truth)?

Answer 1

Summary: I'd rate the first quote you gave 95% accurate, while the 2nd one only 5% or so, mainly because the bulk of McIntyre and McKitrick (MM henceforth) claims/attacks from their paper(s) turned out to be unfounded, while the small part of their criticism that was factual (and resulted in corrigendum from Mann et al.) turned out inconsequential with respect to the main conclusion from Mann et al.

---

The first quote refers mostly to McIntyre and McKitrick 2003 which was indeed published quickly in a pretty obscure venue and without giving Mann's team a chance to responnd prior to publication. It is also true that a 2007 paper by different authors (than either set of disputants) refuted MM03 (and MM05 [see below]) in the sense that Mann's results could be duplicated. Quoting Wikipedia, which has has well-referenced article on this

McIntyre drafted an article before they first met on 19 September 2003, and they then worked together intensively on an extensive re-write. McKitrick suggested submitting the paper to Nature, but after drafting a short version to fit the word limit they submitted the full paper to Energy & Environment on 2 October. After review, resubmission on 14 October and further corrections, the paper was published on the web on 27 October 2003, only three and a half weeks after its first submission. Boehmer-Christiansen later said that she had published the paper quickly "for policy impact reasons, eg publication well before COP9", the United Nations Framework Convention on Climate Change negotiations in December 2003.

At the same time that the McIntyre and McKitrick (MM03) paper "Corrections to the Mann et al. (1998) Proxy Data Base and Northern Hemisphere Average Temperature Series" was published on the web, McIntyre set up climate2003 as a web site for the paper.

McIntyre and McKitrick said that they had not been able to replicate the Mann, Bradley and Hughes results due to problems with the data: although the sparse data for the earlier periods was difficult to analyse, their criticism was comprehensively refuted by Wahl & Ammann 2007. [...]

In 2007 the IPCC AR4 noted the MM03 claim that MBH98 could not be replicated, and reported that "Wahl and Ammann (2007) showed that this was a consequence of differences in the way McIntyre and McKitrick (2003) had implemented the method of Mann et al. (1998) and that the original reconstruction could be closely duplicated using the original proxy data."

The second quote you gave refers to a subsequent effort of McIntyre and McKitrick (2005)

In 2004 Stephen McIntyre blogged on his website climate2003.com about his efforts with Ross McKitrick to get an extended analysis of the hockey stick into the journal Nature. At this stage Nature contacted Mann, Bradley, and Hughes, about minor errors in the online supplement to MBH98. In a corrigendum published on 1 July 2004 they acknowledged that McIntyre and McKitrick had pointed out errors in proxy data that had been included as supplementary information, and supplied a full corrected listing of the data. They included a documented archive of all the data used in MBH98, and expanded details of their methods. They stated that "None of these errors affect our previously published results." [...]

On 14 October the McIntyre and McKitrick comment was submitted to Geophysical Research Letters, a publication of the American Geophysical Union, and the AGU consented to it being shown at their 13–17 December conference.

The proceedings of this conference were published in February 2005.

So the first quote is accurate in that MM03 was debunked by Wahl and Ammann 2007 (and the IPCC agreed with the latter). And 2nd quote is somewhat accurate in that the 2nd effort of McIntyre and McKitrick did result in a 2004 corrigendum by Mann et al. However it substantially misrepresents the impact MM05, including in the North report. Quoting again Wikipedia:

The contested principal component analysis methodology had a small tendency to bias results so was not recommended, but it had little influence on the final reconstructions, and other methods produced similar results.

And indeed while the North report mentions MM03 and MM05 on p.91, there's no mention of the minor issues found there in the summary of the report, which does have hokey sticks from multiple authors in its summary figure S1, including one from Mann & Jones 2003.

---

And in more detail, MM03 was a broadside salvo, here's just its abstract:

The data set of proxies of past climate used in Mann, Bradley and Hughes (1998, “MBH98” hereafter) for the estimation of temperatures from 1400 to 1980 contains collation errors, unjustifiable truncation or extrapolation of source data, obsolete data, geographical location errors, incorrect calculation of principal components and other quality control defects. We detail these errors and defects. We then apply MBH98 methodology to the construction of a Northern Hemisphere average temperature index for the 1400-1980 period, using corrected and updated source data. The major finding is that the values in the early 15th century exceed any values in the 20th century. The particular “hockey stick” shape derived in the MBH98 proxy construction – a temperature index that decreases slightly between the early 15th century and early 20th century and then increases dramatically up to 1980 — is primarily an artefact of poor data handling, obsolete data and incorrect calculation of principal components.

And Wahl & Ammann refutation:

Our results do not support the large upward “correction” to MBH reconstructed 15th century temperatures described in MM03 (p. 766) and MM05b (p. 71), and leave unaltered the single-bladed hockey stick conclusion of strongly anomalous late 20th century temperatures.

[...]

More generally, our results highlight the necessity of reporting skill tests for each reconstruction model. Taking this consideration into account, there is strong reason to conclude that the 15th century reconstructions reported in MM03, which show hemispheric temperatures much higher than those of the mid 20th century, do not have climatological meaning. This double-bladed hockey stick result, while computable using the MBH algorithm, does not pass standard validation tests (RE scores <0 for both calibration and verification). These validation results indicate that the annual climatological mean of the 1902–1980 calibration period would be a better predictor over 1854–1980 (the period of available instrumental values that can be used for comparison) than the reconstruction models of this scenario (cf. Cook et al., 1994). The same result holds for the somewhat lower-amplitude double-bladed hockey stick reconstruction reported in MM05b over 1400–1449, which also fails to pass calibration and verification tests for this period. Thus, the primary climatological argument offered by MM for rejecting the uniqueness of high late-20th century temperatures is found to be without merit, based on examination of the empirical quality of the reconstruction models these authors report.

Overall, the primary outcome from our results is that the work reported in MM03, MM05a, and MM05b does not provide substantive reason to invalidate the general conclusion of anomalous warmth in the later 20th century derived from the MBH reconstruction method and proxy data framework.

(MM05b was another broadside paper published in the same [obscure] venue as MM03, namely in Energy & Environment, which has an impact factor of 0.3. MM05a is their paper in Geophys. Res. Lett. (IF of 4 or so); its criticism is restricted to the principal component analysis method used by Mann et al.)

So it's fair to say that most of the criticism of MM turned out unfounded, while the criticism that was factual turned out rather inconsequential. Surely the opening claim from your 2nd quote that "The claim that our paper was “subsequently debunked” is an empty and worthless" doesn't hold water. So based on that I'd say the first description (MM papers [mostly] debunked) is 95% accurate, while the 2nd one is maybe 5% accurate, in that some of their criticism was substantive and acknowledged, but ultimately inconsequential.

---

And I've checked all of Mann's publications in Nature; there was no other corrigendum to the 1998 paper (besides the 2004 one I already mentioned). The only 2006 publication on Mann in Nature (M&M incorrectly mention 2006 as the year of the corrigendum) is a brief reply to a news report "Academy affirms hockey-stick graph" by Nature staff on the North report... and the reply is about a different (1999) paper and does not involve M&M's criticism(s), but they reply to a remark by North.

---

And since Avery's answer uncritically quotes from the blog of McIntyre a snippet which gives the impression that North fully agrees with Wegman's report (I deliberately ignored the latter insofar), here's the full extent of North's opinion on that:

Dr. Wegman’s criticisms of the statistical methodology in the papers by Mann et al were consistent with our findings. Our committee did not consider any social network analyses and we did not have access to Dr. Wegman’s report during our deliberations so we did not have an opportunity to discuss his conclusions. Personally, I was not impressed by the social network analysis in the Wegman report, nor did I agree with most of the report’s conclusions on this subject. As I stated in my testimony, one might erroneously conclude, based on a social network analysis analogous to the one performed on Dr. Mann, that a very active and charismatic scientist is somehow guilty of conspiring or being inside a closed community or ‘mutual admiration society’. I would expect that a social network analysis of Enrico Fermi or any of the other scientists involved with the development of modern physics would yield a similar pattern of connections, yet there is no reason to believe that theoretical physics has suffered from being a tight-knit community. Moreover, as far as I can tell the only data that went into Dr. Wegman’s analysis was a list of individuals that Dr. Mann has co-authored papers with. It is difficult to see how this data has any bearing on the peer-review process, the need to include statisticians on every team that engages in climate research (which in my view is a particularly unrealistic and unnecessary recommendation), or any of the other findings and recommendations in Dr. Wegman’s report. I was also somewhat taken aback by the tone of the Wegman Report, which seems overly accusatory towards Dr. Mann and his colleagues, rather than being a neutral, impartial assessment of the techniques used in his research. In my opinion, while the techniques used in the original Mann et al papers may have been slightly flawed, the work was the first of its kind and deserves considerable credit for moving the field of paleoclimate research forward. It is also important to note that the main conclusions of the Mann et al studies have been supported by subsequent research. Finally, while our committee would agree with Dr. Wegman that access to research data could and should be improved, as discussed on page 23 of the prepublication version of our report, we also acknowledge the complicated nature of such mandates, especially in areas such as computer code where intellectual property rights need to be considered.

This is from p. 592 in the full transcript. And from p. 92 for a bit more context:

MS. SCHAKOWSKY. I am trying very hard to understand the point of this hearing and this conflict because if we are through many studies come to the conclusion that there is such a thing is global warming, which is hard to deny on a day like today and yesterday, et cetera, although I am not the scientist, and that it at least in some part is caused by human activity, then why we are doing this really does escape me. I can understand why in academia you may have an interest in discrediting Mann and back and forth, but I am very concerned that this is being used in a way to discredit the whole notion that our country and the rest of the industrialized and developing ought to do anything about global warming, and that is why I asked you that question, Dr. Wegman, if this does not make you somewhat uncomfortable. Can you see in any way how this is being used and does it bother you?

DR. WEGMAN. Well, I can understand that it is your job to sort out the political ramifications of what I have said. In some sense it is not fair for you to say well, gee, you have reported on some fact and that is going to be used in a bad way. The other side of the coin is that, you have tried to get me to say that manmade carbon dioxide emissions are associated with the global warming.

MS. SCHAKOWSKY. Which you can’t, right, because you are not a climate scientist.

DR. WEGMAN. I cannot say that, but what I can say is that from 1850 to the present time, the global temperature rise is about 1.2 degrees Centigrade according to the Mann chart. One point two degrees Centigrade translates to about two degrees Fahrenheit. I challenge anybody to go out and tell the difference between 72 and 74 degrees Fahrenheit. What I do say and what I have said repeatedly is that you need to focus on the basic science. You need to understand what the transfer of heat from the ocean to the atmosphere, how that dynamic works, how the climate is going to change based on the physical mechanisms, a fundamental understanding of the physical mechanisms, not on some statistical estimation of those signals.

[SCHAKOWSKY time ran out after this.]

In other words, Wegman says he's not a climate scientist but he then tells the climate scientists what their "basic science" should be about (which should not include [statistically] estimating temperature, it seems, that probably should be left to statisticians??). Mmkay... now you can probably tell why North was "taken aback" Wegman's "tone".

Answer 2

I don't have time to respond to the entire question, but here's what he was talking about with this part:

A panel led by Professor Wegman later conducted an independent review of the mathematical and statistical issues and upheld the M&M critique. A panel of the National Academy of Sciences also conducted an examination of the whole issue of paleoclimate reconstructions and upheld all the technical criticisms M&M made of Mann's work, going so far as to publish their own replication.

This is backed up by the testimony of Gerald North, who chaired the NAS report, to the Wegman Report commission:

CHAIRMAN BARTON: Dr. North, do you dispute the conclusions or the methodology of Dr. Wegman’s report?

DR NORTH: No, we don’t. We don’t disagree with their criticism. In fact, pretty much the same thing is said in our report. But again, just because the claims are made, doesn’t mean they are false.

CHAIRMAN BARTON. I understand that you can have the right conclusion and that it not be–

DR. NORTH. It happens all the time in science.

CHAIRMAN BARTON. Yes, and not be substantiated by what you purport to be the facts but have we established–we know that Dr. Wegman has said that Dr. Mann’s methodology is incorrect. Do you agree with that? I mean, it doesn’t mean Dr. Mann’s conclusions are wrong, but we can stipulate now that we have–and if you want to ask your statistician expert from North Carolina that Dr. Mann’s methodology cannot be documented and cannot be verified by independent review.

DR. NORTH. Do you mind if he speaks?

CHAIRMAN BARTON. Yes, if he would like to come to the microphone.

MR. BLOOMFIELD. Thank you. Yes, Peter Bloomfield. Our committee reviewed the methodology used by Dr. Mann and his coworkers and we felt that some of the choices they made were inappropriate. We had much the same misgivings about his work that was documented at much greater length by Dr. Wegman.

I have nothing to add in response to Fizz's comment. I did not intend to make this answer about the politics of climate change. I simply desired to point that the Wegman report and the National Academy of Sciences agree that MM05 provided accurate technical criticisms of Mann's model, so McKitrick's summary of the Wegman and NAS panels is entirely accurate.