James Hansen took his degree in astrophysics, not normally a climate-related field. Nonetheless, few would argue that he is not an expert on climate change.
On the other hand, Freeman Dyson is possibly the second smartest person on the planet, a theoretical physicist who worked in the field of climate science for 15 years. And yet, because he does not support the consensus, climate activists dismiss him as unqualified.
How do we estimate the expertise of someone in a field where we ourselves are not expert?
This is a current events question, given the recent publication of ‘Consensus on Consensus: A Synthesis of Consensus Estimates on Human-Caused Global Warming,’ written by (among others) John Cook, Naomi Oreskes, Stefan Lewandowsky and William Anderegg, all authors of papers much criticized here.
The point of their paper is simple: The work of some of the co-authors of the paper were cricitized by Richard Tol. The thrust of his criticism is that many studies of climate consensus eliminate large amounts of data considered unqualified by the researchers. Tol writes, “Cook et al (2013) estimate the fraction of published papers that argue, explicitly or implicitly, that most of the recent global warming is human-made. They find a consensus rate of 96%–98%. Other studies (6) find different numbers, ranging from 47% in Bray and von Storch (2007) to 100% in Oreskes (2004)—if papers or experts that do not take a position are excluded, as in Cook et al. If included, Cook et al find a consensus rate of 33%–63%. Other studies range from 40% in Bray and von Storch (2007) to 96% in (Carlton et al 2015). Cook et al use the whole sample. Other studies find substantial variation between subsamples. Doran and Zimmerman (2009), for instance, find 82% for the whole sample, while the consensus in subsamples ranges from 47% to 97%. Verheggen et al (2014) find 66% for the whole sample, with subsample consensus ranging from 7% to 79%.”
This most recent paper by those for whom the Tol belled is an attempt to justify their decisions. Their reasoning is simple. If you eliminate the non-experts from the total being surveyed, the experts will agree with you.
In the Supplementary Information to their paper they write, “We define domain experts as scientists who have published peer-reviewed research in that domain, in this case, climate science.” (Despite this, they eliminate many peer-reviewed respondents in Verheggen et al, for example.)
As I mentioned the other day, a simple publication count is a remarkably weak way of estimating expertise. I wrote, “The weaknesses of publication records are:
1. Very capable younger scientists have not had time to establish a record of publications. Dismissing their opinions leads to loss of useful information.
2. As ‘alarmists’ like to point out whenever an older scientist expresses a skeptical viewpoint, at some point in the natural cycle of a person’s career, ongoing education becomes less important. One can make the case that someone reaching the end of their career actually knows less than a freshly minted scientist.
3. The tools and techniques used in tertiary education are different than they were when many older scientists were educated. In addition, new knowledge is incorporated into texts available to younger scientists. This again may advantage the young at the expense of the old.
4. Some scientists are co-authors of numerous papers for reasons other than their ability to contribute to the main body of the scientific arguments advanced in the paper. Their publication count may be more impressive than their actual command of the field.
5. Some very good scientists work outside the academic world and publication may not be a priority for them. Using publications as a proxy for expertise again may devalue their opinions.”
When I made those points to another of the paper’s co-authors (Bart Verheggen), he agreed but basically said it was the only way he could think of.
While Lewandowsky, Cook and Oreskes are not climate scientists, it seems that none of the team involved with the paper thought to look at how others evaluate expertise. It didn’t occur to them that there is a body of work that could have informed their paper. As many of the co-authors were in fact authors of papers cited in the most recent work, it really seems as if they missed the boat.
This surprises me a little, given the frequency with which they throw around the term ‘Dunning Kruger Effect,’ which describes the tendency of individuals to overestimate their own knowledge or abilities. It’s part of the field of expertise evaluation, yet that name is the only thing that seems to have stuck.
Expert recommendations are an oft-used technique to identify those with expertise. People refer those they think are experts and if enough of them do it,they are awarded the title.
Expertise is a highly relevant topic in the field of law, where my expert goes against your expert in the courtroom, and establishing who’s better is pretty important. It’s also important in discovery, especially with the new game of ‘e-discovery’, the evaluation of mountains of documents using software to sort it. Again, this is a well-researched topic ignored by Cook et al.
It’s relevant to military decision making, high technology research, and in academia.
In academia, a publication count is considered the crudest method of evaluating expertise, mostly for the reasons I cited above. More common are techniques such as citation measuring (how many times your work has been referenced by others) or impact measurement (the perceived quality of the journals where you are published, often combined with publication counts and citation counts).
I mentioned in my previous post that few of the co-authors have expertise in climate science. Fewer have experience in surveys. None appear to have relevant expertise in evaluating expertise.
They did not utilize the methods most commonly used and most trusted in academia. Worse, they do not appear to have consulted the large body of literature on the subject. There are no references to the appropriate literature in their paper.
They…just did a pub count and called it a day.
There’s no doubt they desperately need to defend their 97% claim of consensus. Going after Exxon and threatening to put skeptics in jail requires that high a level of confidence.
But you would think that if they were going to defend it, they would do a better job.
Most of the original papers referenced in their latest effort are remarkably weak. It seems they didn’t learn from experience.
Climate Scepticism 
??
And to think that I thought this a serious blog.
Bravo, in any case.
LikeLiked by 1 person
Never mind that even if their ‘analysis’ was correct, it is still meaningless (the scientific method doesn’t care about ‘consensus’), they also fail to account for expectancy bias: “We all got together to confirm we were right, and indeed, found that we were.”
LikeLiked by 1 person
I looked up Freeman Dyson’s publication on climate or “climate change” or “global warming”. Assuming I didn’t make some kind of mistake with the search, it returned 4 results. All 4 were in the New York Review of Books (not a peer-reviewed journal on climate, I think) and 3 of them seemed to refer to some exchange about something called “The Question of Global Warming”.
So, in what way did Freeman Dyson [work] in the field of climate science for 15 years?
LikeLike
Lest anyone be fooled, which I think unlikely, this wrangling over consensus is really a political activity and not a scientific one. Just another area of low quality papers diverting attention and energy from the real issues, which admittedly are much harder and not really accessible to most of the authors. Richard Tol is probably an exception whose contributions are important.
The real tragedy here is as the Economist laments with regard to science in general. The real loss from the fact that perhaps half of published results are wrong is the opportunity cost. How much good could all these people be doing if the system rewarded better behavior?
And that is perhaps the point of this genre of literature. It’s basically just another mechanism of denial that science has real problems that need to be addressed. I have a list now of 15 references from publications such as Science, The Lancet, The New England Journal of Medicine, the Economist, etc. taking aim at this scandal that impacts the reliability and reputation of science generally. Cook, Lew, and Oreskes are incapable of addressing it and are at great pains to deny that the problems exist.
LikeLiked by 2 people
I suspect that there would be a wide variation of opinions on what defines the realm of ‘Climate Science’, but I believe that most would agree that the Science of Meteorology would be a major player. (Of course Meteorology draws heavily upon other fields such as Physics, Mathematics and Statistics as there are extensive overlapping and complimentary aspects involved.) Just focusing upon Meteorology: What defines expertise in this field? Is it the number of publications? Or is it the number of major discoveries or contributions? Is it an individual’s fund of knowledge, or is it critical thinking ability? What is the acid test that separates a ‘true expert’ from the pack?
It is my contention that in the field of Meteorology there is insufficient public knowledge to answer the latter question. Why is this so? Well, where do you think many of the ‘best and brightest’ meteorologists are working? In Academia? For the Government? At a research institute? At your local TV station? Think so? Where else do meteorologists work? Well, follow the money. Where, in general, are the highest paying positions located for meteorologists? Answer: In the private sector, in forecasting, especially in Finance (hedge funds, commodity trading, casualty insurance, etc.) where you ‘live and die’ by your abilities. However, the latter circle of expertise usually doesn’t publish in journals as its work is proprietary and guarded. And note: the high paying arena places emphasis on ‘forecasting’ rather than ‘modeling’, and if one’s expertise doesn’t include advanced statistics, one may not survive long in this circle.
So are the usual suspects in Climate Science really experts? Are they into ‘climate modeling’ or ‘climate forecasting’? It seems that ‘climate forecasting’ is just being introduced to Climate Science (see below), but what do you think the unpublished ‘real experts’ have been doing for some time?
Click to access Forecasting-3.pdf
LikeLike
Dave L, I can testify from first hand experience about the gap between the “public science” such as it is and actual proprietary forecasting in aeronautical CFD. It is a huge gap with the “public science” being pretty badly biased and subjective and wholly inadequate for decisions of fiduciary responsibility. One must bear in mind that the Cook paper co-authors have uniformly little experience that would count here. Models of planet formation where one asks only for “qualitative understanding” is not really relevant here.
LikeLike
ATTP, From Wikipedia: “Around 1979, Dyson worked with the Institute for Energy Analysis on climate studies. This group, under the direction of Alvin Weinberg, pioneered multidisciplinary climate studies, including a strong biology group. Also during the 1970s, he worked on climate studies conducted by the JASON defense advisory group.[28] Dyson retired from the Institute for Advanced Study in 1994.”
ATTP, the fact that you are worrying about the number of Dyson’s publications on climate change is very, very odd.
LikeLiked by 1 person
Thomas, I’ve noticed that ATTP is very literal minded on this issue. He seems to have internalized the artificial University system of judging everything by the literature. As DaveL points out, that’s just a small slice of expertise.
LikeLike
DaveL’s point about where the real meteorology experts are working can equally be applied to experts in survey technique. The social scientists among the authors – Doran and Lewandowsky – have amply demonstrated that they wouldn’t last five minutes in a market research company.
LikeLiked by 2 people
Tom,
You claimed he had worked in climate science for 15 years. I looked up his publication record and – apart from some in the New York Review of Books – couldn’t find anything. I then asked you in what way he had worked in climate science for 15 years. You find that odd? Each to their own, I guess.
Bear the following in mind. Just because someone with no papers might have expertise in climate science, does not mean that everyone with no papers has expertise in climate science.
LikeLike
There are various flavors of expertise. And with all of them, there is the problem that it isn’t what you don’t know that causes trouble, but rather what you ‘know’ that isn’t so. There is a lot of both in ‘climate science’.
One example from AR4 and AR5. IPCC knows GCMs don’t do clouds well, yet IPCC ‘knows’ clouds have a substantial positive feedback. Dessler tried to observe this comparing allsky to clearsky, and said feedback was positive based on an r^2 of 0.02! Statistical joke. Essay Cloudy Clouds.
LikeLike
Gee, ATTP, what are you trying to say? Wo bu dong (I don’t understand). No comprendo. Are you saying he didn’t work in climate science for 15 years? Are you saying he didn’t publish? Are you saying your internet search skills need an upgrade?
I don’t think anyone here is suggesting that ‘everyone with no papers has expertise in climate science.’ Why do you introduce that?
What I thought I said quite clearly was that Freeman Dyson worked in climate science for 15 years.
LikeLiked by 1 person
Tom,
I assume your avatar picture is your grandfather. That’s nice of you.
Yes, I got that bit. I was simply trying to understand where you got that from, given that if he did work for 15 years in climate science, he appears to have published no papers. I realise that publication numbers are not necessarily a good way to determine someone’s expertise, but 0 papers in 15 years is pretty dismal.
Because you’re critiquing the choice of paper number as a metric for expertise. It’s certainly true that it’s not a perfect way to determine expertise and there may well be people with few – or no – papers who do have relevant expertise. However, if you are trying to extract a sample of people who may be regarded as having expertise, then paper number is one way to do so. The argument that some people with few – if any – papers might also have expertise does not make this a poor way to do it. The argument that publications are not necessarily a good way to judge people, again, is not an argument against doing this. If you wish to find a better way to determine expertise, feel free to do so. However, simply finding reasons to criticise what was done, appears to just be the standard way in which to dismiss a study the result of which you dispute, without actually doing any work to show that some alternative method would produce a different result (probably because an alternative method would not produce a different result).
LikeLike
Gee, ATTP–I thought I outlined several better ways to determine expertise above. Did you read the post? It’s what is called ‘accepted practice.’ Sometimes even ‘best practice.’
And gee again–the second smartest person on the planet worked in climate science for 15 years and didn’t publish much.
That could mean one of two things: One, that I am making everything up about Freeman Dyson just to tick you off. Alternatively, it might be evidence for my thesis here–that counting publications is a rotten way to determine expertise.
Which do you think it is?
LikeLiked by 2 people
Tom, The problem here for ATTP is that he is a coauthor on a low quality paper that is the latest in a long line of politically motivated papers of low scientific quality. As I said above, these people could actually make a real contribution if they weren’t distracted by this sideshow. Then ATTP comes here to defend this low quality paper. Why is it of any scientific importance at all to try to measure “consensus” using dubious measures of expertise? It is a political question. Some people who claim to be scientists don;t have the discipline to think honestly about the distinction, much less act on it.
LikeLiked by 2 people
What is really interesting however is the origins of scientism in the political left. The usual defense mechanisms become dogma. Dramatic problems with reproducibility are first ignored, then denied, and then minimized. In any case, for the scientism faithful, its more important to study consensus than reproducibility because the former is consistent with scientism, the worship of SCIENCE. The latter is a heretical activity that is best ignored. Ken Rice of course perfectly epitomizes this in that he is a leftist and also has a consistent track record of denial that reproducibility problems are important.
LikeLiked by 2 people
The whole Cooked-up climate charade reminds me of some of the words of the great Spike Milligan:
“My uncle was a great man, he told me so himself.
‘I am a great man,’ he said, and you cannot argue with facts like that.”
LikeLiked by 1 person
I looked up Ken Rice’s publication on climate or “climate change” or “global warming”. Assuming I didn’t make some kind of mistake with the search, it returned 0 (zero) results. Not even in the New York Review of Books.
LikeLiked by 1 person
Times Higher Education.
July 22, 2010
https://www.timeshighereducation.com/features/a-bona-fide-dispute/412601.article
LikeLike
Tom,
Yes, I did read the post. I’ve also pointed this out to you before. Impact measurements are not regarded as a good measure. There are numerous examples of people arguing that we should not judge a paper, or a researcher, on the basis of the impact factor of the journal in which their papers are published. Citation metrics are probably better. However, I do not think we had this, so we couldn’t use this. Suggesting an alternative that we could not actually use doesn’t seem like a particularly constructive suggestion.
clipe,
Your point?
LikeLike
So, yet once again your ‘team’ (tribe, club, gang, group) must resort to defending poorly constructed research, all but concealing real numbers in favor of a preferred metric that you don’t know how to measure.
Research design is supposed to come up with answers to this stuff before you get started. Did you ever meet with your co-authors prior to commencement of this? Not necessarily face to face, but we now have tools like Skype and Google hangouts.
It’s classic. It’s comic. You even exemplify the difficulties you face here in this thread. I say that publication counts are flawed as a standalone product. I say that Freeman Dyson should be considered expert because he worked 15 years in climate science. You say you are not even sure he worked in climate science… because… he…didn’t… publish. And… you… don’t… spend… five… minutes… on… the… internet…to… find… out.
LikeLiked by 3 people
ATTP seems to have difficulty holding more than one idea in his mind at a time. The relative value of citation metrics and impact factors may fascinate him personally, but he needs to bear in mind: that so-called “objective” quantitative methods of assessing expertise are inherently useless; that the fact that one might be marginally less useless than another doesn’t alter the fact; that the different studies used different criteria of assessment ; that the Cook metastudy seems to have used no criteria at all, but just arranged them subjectively on a piece of graph paper to make a nice pattern; that the idea of authors ganging together to do an objective metastudy confirming their own studies is risible; and that the presence of a number of co-authors with no expertise on a paper designed to emphasise the importance of expertise makes you all a laughing stock.
LikeLiked by 2 people
Responding to these points is simply not worth the effort. Anyone associated with this site who suggest others are a laughing stock, is not someone with whom it is worth exchanging views.
Here’s a key point. There’s is little doubt that the level of consensus amongst relevant experts is high. Maybe not 97%, but somewhere in the 90%. This site is associated with people who appear to dispute the consensus position. You’re of course free to do so. However, instead of having the courage to admit that your views are at odds with that of most relevant experts, you attack the studies that illustrate this level of consensus amongst experts. I don’t know why this is, but my guess is that you’re uncomfortable admitting that you hold views that are at odds with most relevant experts, and so would rather try to convince yourself – and others – that such a consensus does not exist.
LikeLike
ATTP, for someone who feels that responding to these points is not worth the effort, you certainly have been doing a lot of responding.
You do identify a (not the) key point. Most critics of your paper and the papers it tries to defend are happy to acknowledge a consensus of climate scientists. Indeed, two of the papers used by your study do a good job of identifying that consensus at around 66%.
What I personally object to is your claiming a 97% consensus by hiding the 66%,by magically claiming that 65 papers out of 12,000 odd is transformed into a 97% consensus, that changing keywords on a literature search until you find a convenient one yields no skeptical paper when 117 qualifying papers can be found on Skeptical Science. Other opponents have offered other reasons to object to your finding.
As I said earlier, a 66% consensus is adequate for scientific purposes. The only reason to strain and stretch for a mythical 97% is to provide cover for a set of policy proposals that are mostly daft.
It’s just amazing that you did such a poor job of it. There was a U.S. baseball manager named Casey Stengel who managed the New York Mets a long time ago. After one afternoon of inept play he was heard to lament “Can’t anybody here play this game?”
I think of that phrase quite often when I read your comments.
LikeLiked by 1 person
What exactly is an expert and does it affect the truth of what they say or think? Is someone who writes long and learned papers on the existence of god an expert? Well yes, on the literature and thoughts about god but without proof, the person is not an expert on the reality or not of a deity. A scientist who condemns homeopathy might be an expert on how science should be done and that homeopathy doesn’t comply with those rules, but is not an expert on homeopathy… it doesn’t mean that they’re wrong in claiming homeopathy is no better than a placebo.
Climate change and its supporters need consensus surveys because they can’t defend the real science. Like the religious cleric, they can become experts in what people believe is true but climate science is very short on proof of what is demonstrably true.
LikeLiked by 1 person
ATTP
”Anyone associated with this site who suggest others are a laughing stock, is not someone with whom it is worth exchanging views.”
When I worked in market research, if I had produced a graph like yours, I would have been a laughing stock among colleagues. I’d also have been in serious trouble with clients, who don’t like paying good money for garbage.
The fact that an astrophysicist can sign his name to garbage that would shame a junior marketing executive tells us something extremely important about our society that goes way beyond the ethics, intelligence or competence of the people involved. It’s about the role of status, prestige, and expertise in our society – a classic subject for study by sociologists. But the sociologists are in it up to their necks, so there’s no way that anyone with “expertise” is going to examine the problem. So what else can one do but try to reason woith people who, after all, must have some sense of what a graph should look like?
LikeLiked by 2 people
It appears to me that Climatology, once considered to be a physical science, is rapidly transforming into an exclusive political club.
As Groucho Marx once allegedly said: “I don’t want to belong to any club that would accept me as one of its members.”
LikeLiked by 2 people
Geoff,
Who are you trying to reason with?
LikeLike
There is no such thing as an ‘expert in climate science’ because ‘climate science’ is in fact climate sciences. Likewise, there’s no such thing as an expert in climate change because to fully understand climate change you have to have expertise in any number of climate sciences. One can be an expert oceanographer, atmospheric physicist, atmospheric chemist, meteorologist, geologist, palaeo-climatologist, solar physicist, climate modeler, etc. One can’t possibly be expert in all these fields but, in order to piece together the evidence for and against significant and/or dangerous anthropogenic global warming, expert knowledge gleaned from all these fields is required. Hence no doubt the often very long list of authors on climate science papers. But they are only experts in their chosen field. So if a scientist is co-author on a long list of papers which fall into the general category of ‘climate science’ that does not, per se, make him/her an expert in climate science.
LikeLiked by 3 people
Tom Fuller: ‘What I personally object to is … magically claiming that 65 papers out of 12,000 odd is transformed into a 97% consensus’
If you exclude all the engineers, economists, paediatricians and what have you, the 65 papers offering a humans-causing-more-than-half consensus falls to less than 50. (I made it 46. Others might differ by a few papers in either direction.)
Amusingly, one of the 65 papers (co-written by a lawyer/WWF activist and someone who might have an economics degree or might just be the wife of someone who owns a construction company and was briefly the minister in charge of tourism in Romania) was published in a notorious fake journal, _Metalurgia International_. See ‘Climate Changes And The Actions Of The European Union For Environmental Protection’.
(About half a dozen other papers in the 11,944 were published in the same fake journal. I haven’t checked them out – or looked for other fake journals, which are bound to be there.)
===
Obligatory (alas) disclaimer: Yes, the scientific consensus is probably in the high 90s. But if something can’t be measured, don’t cheat and pretend that it can be for what are essentially political reasons. Cook 2013 was crap in design, execution and follow-up. It’s astonishing that people still defend it.
LikeLiked by 1 person
I now make it 64 papers. (But perhaps I have buggered up the spreadsheet. Again.)
LikeLike
JAIME JESSOP at 18 Apr 16 at 8:56 pm
You are quite right about the varied nature of climate science / climatology. What also needs to be pointed out is what this broad field is not. It is not ethics, political philosophy, economics, public policy-making, public policy implementation or the conducting of opinion polls. Yet it would seem that the experts on climate know all about these things over and above the experts in their respective fields.
Climatology is a strange sort of science that instead of making verifiable content-rich empirical statements that relies on banal opinion statements divined from abstracts of papers from a huge variety of disciplines. Real science, in all its forms, has spent centuries trying to understand the real world in opposition to established dogmatic beliefs. Now Cook at al. are trying to maintain established beliefs in the name of science. They are modern day antidisestablishmentarianists.
LikeLiked by 1 person
I’ve often wondered how many people are involved in working out climate sensitivity, since everything else pivots on it.
LikeLike
TINYCO2 19 Apr 16 at 10:06 am
That is not quite correct. If climate sensitivity is low (less than 1) then the case for global emissions mitigation policies is undermined. There will be no future climate cataclysm, so we can leave it to some proper academic climatologists to decipher the real changes from the natural variation. But any human-caused changes will be adapted to in the same way as natural variation – by adaptation.
But if climate sensitivity is large and has highly adverse consequences, it does not mean that climate mitigation is the best policy. COP21 Paris clearly showed that all actual policies would make very little difference to global emissions, but where enacted such policies are very damaging.
In other words, demonstrating beyond reasonable doubt that climate sensitivity is low would undermine the case for climate mitigation policies. But demonstrating beyond reasonable doubt that climate sensitivity is high is a necessary, but far from sufficient, condition for showing that climate mitigation policies are beneficial. What is more, in answering the policy justification question, genuine climate scientists are mere technicians. The moral, economic and policy enactment areas are all quite different areas of expertise. The fact that the climate alarmists claim primacy in these areas based on pseudo opinion polls about belief in climate, whilst having failed to demonstrate competency (let alone expertise) in climatology, shows they are liability.
LikeLike
Mbc, ben pile has a whole blog devoted to the fact that, if climate gullibilism is real, it still does not mean that we follow the Greenpeace plan to global poverty
LikeLike
You will notice that ATTP left without responding to any substantive point.
LikeLike
MAN IN A BARREL says: 19 Apr 16 at 10:41 pm
I have also looked at the issue of policy. My main points are:-
1. The sum total of all proposed global policy initiatives, if fully enacted, would not make a significant difference to global GHG emissions. They only cover the period to 2030, and will not stop global emissions
from rising.
2. The UNFCCC before COP21 Paris knew this was the case. To claim that by 2100 policy would result in much lower emissions at present they used fantasy policy projections of other bodies for the period post 2030.
https://manicbeancounter.com/2015/11/29/unfccc-massively-overstates-impact-of-indcs-on-2100-emissions
LikeLike