Hiding the rise

Alternative title: The complete idiot’s guide to cherrypicking.

Willis Eschenbach (of Darwin Zero fame) has a post on Watt’s Up With That concerning the homogenisation process in Anchorage and Matanuska (both in Alaska). Matanuska was chosen for being close to Anchorage. But why start in Anchorage? No explanation is given. Something about this smells like cherrypicking – picking out a station that happened to have an odd-looking trend and expanding a conspiracy theory around it.

Well, two can play at this game. I wrote a little program to find stations that had a downward trend in homogenisation in the GHCN v2 data. Not anything even approaching being clever, I just picked a few of the stations that had a reasonable amount of data from the last 40 years and had less homogenisation in 2009 than in 1970. But let’s say I didn’t give that explanation, and just talked about the odd trend in Asheville? That would hardly be methodologically valid.

Here’s how the temperatures in Asheville, North Carolina, were homogenised:

I can’t explain the homogenisation in Asheville. Would it be reasonable to suppose that an AGW denialist hacked into the GHCN website and modified the data? Well, no. Whereas Eschenbach was excited by a .7°C increase spread out over 20 years, here we had a .375°C decrease in a single year. Note how this causes the homogenised line (red) to drop away from the raw data (blue).

What about Pohang, in South Korea?

These major drops cut off what had looked like a warming trend. Certainly Pohang was not homogenised to deliberately create an artificial warming trend! In two bursts within just 8 years (and at the start of the data set), temperatures are adjusted upwards by a startling .65°C!

And then there’s Cairns Airport. What the heck is going on here?

Is homogenisation being used to hide global warming all over the world?

Why is data being homogenised like this? Well, it’s unfortunate, but most of the various temperature stations were not set up to track climate. As such, they periodically got moved, changed, located in places that were more convenient to measure for the daily news rather than for tracking world climate, and so forth. This means that before trying to use these for studying climate it’s probably worthwhile controlling for these factors.

But, seriously, there are a whole bunch of factors, and between them they could cause adjustments – both up and down. And if you look at enough sites, you’ll no doubt find examples of both. So if someone is showing you one site’s homogenisation and asking you to draw conclusions of fraud on the basis of it – why that site? Why not any of the thousands of others? Are all the sites homogenised in the same direction? Or is it simply more likely that you are listening to someone who is cherry-picking, finding any anomaly and then wrapping it in a conspiracy and their own absolute certainty that global warming is a lie and that the scientists who give evidence in favour of it are liars and frauds?

Review: Team Foundation Server

Until a few months or so ago, the policy where I worked was to use Microsoft’s Visual Source Safe (VSS) for source control. Now, granted, VSS does have something of a reputation for corrupting your repository, which is generally considered to be an unforgivable sin for a source control system. That, however, never happened to me, and VSS wasn’t particularly unusable. However, my current employer has started using Microsoft’s newer source control system, Team Foundation Server (TFS), which, I can honestly say is the first source control system to ever make me wish desperately for the sweet release of a corrupted repository.

Many issues are related to the plugin, which crashes my development environment several times each day, and takes agonisingly long to do anything successfully (I have time to read the newspaper while waiting for my code to finish checking in. When checking in doesn’t cause it to crash). But, still…

Getting code out of the repository can be a problem. Sometimes, I only get the solution file and the directory structure (but no code files). Other times, I get the code, except for a handful of (as near as I can tell) random files which have been left behind. Granted, on rare occasions, I am able to retrieve the files I need.

Some more complex operations can get a little hairier. For example, I once spent the best part of a day fighting against TFS with the unreasonable goal of moving a folder. A colleague has been trying to set up daily builds, and TFS seems to have a peculiar habit of using old versions of some files for no apparent reason. Other files are unaccountably missing for the use of these builds.

There are random “features” a-plenty too. One time I deleted a project (from my local computer) that had a few changes, after which checking that project out from TFS it would come, but not have any source control bindings. I don’t think I ever bothered trying to deal with that; luckily, I didn’t need that project any more.

TFS takes the VSS’ working paths to stupid extremes with workspaces, creating a system that behaves according to a hierarchy of configuration-heavy and hard to determine rules. Certain actions, like retrieving a project you haven’t accessed before, can alter your workspace configurations without troubling to let you know about it. And why have workspaces in the first place? I’m yet to see any useful feature they add that wasn’t done better by, say, Subversion, a usable source control system, without an unmanageable mess of configuration and a nightmarish cacophony of bugs (in some situations, for example, which I have yet to isolate, it doesn’t put a project into the workspace it is logically supposed to go to, but place them… somewhere else, typically Visual Studio’s projects folder).

Over the last few months, every single developer on our team has spent at least a week fighting against TFS. That said, TFS has its advantages – for example, it’s quite possibly useful as a treatment for hypotension. Unfortunately, I can’t think of any others at the moment.

Is something rotten in Alaska?

Via an open thread on Deltoid, I discovered a link to this article by E.M. Smith (reposted on Watt’s Up With That), looking at an odd map he’d managed to generate using a temperate map generator on the NASA GISS site. The map generator’s pretty fun to play with.

A map of the temperate anomalies can be generated by entering a base period and a time period. As I understand it, it then takes the difference in mean temperature between the baseline period and the time period and draws that on a map. Pretty simple, right?

Now, if you use the same base period as time period, you’d expect that the map anomalies would all just be zero, right? Well, almost. The default settings exclude ocean data, and E.M. Smith does not change that. Without the ocean data and a 250km smoothing radius, you actually get the following map:
Temperature anomalies, time period 1951-1980, base period 1951-1980

What’s going on? Well, the short answer is that in the GHCN data, 9999 is used as a flag value to designate missing data (see the help file at the bottom of a map page, “Missing data are marked as 9999.”). As there’s no ocean data, 9999 appears there. Now, probably those should be greyed out. In maps that have a different base period and time period, grey is used to designate regions that don’t have any data.

However, the simple fact that this was almost certainly just displaying a flag value didn’t stop the conspiracy! Oh no! Presumably, those 9999 values are leaking into the real graphs and causing all the red values in a map like this one:
Temperature anomalies, 2009, base period 1951-1980

Nice idea. So I ran with it. Don’t know how long this GISS map stays up on their site, but I just did 2009 vs 2008 baseline. The “red” runs up to 10.3 C on the key.


So unless we’ve got a 10 C + heat wave compared to last year, well, I think it’s a bug

So I think this points to the ‘bug’ getting into the ‘non-NULL’ maps. Unless, of course, folks want to explain how it is 10 C or so “hotter” in Alaska, Greenland, and even Iran this year: what ought to be record setting hot compared to 1998…

I’ll leave it for others to dig up the actual Dec 1998 vs 2009 thermometer readings and check the details. I’ve got other things taking my time right now. So this is just a “DIg Here” from me at this point.

It’s not the color red that’s the big issue, it is the 9999 C attached to that color… Something is just computing nutty values and running with them.

BTW, the “missing region” flag color is supposed to be grey…

Now, this is something of a leap: how unlikely is it that unusual values in the ocean would magically happen to manifest themselves as warming in Alaska or Greenland – let alone Iran – rather than in, oh, say, the oceans. Never mind the idea that a modest change in temperatures between two years is especially unlikely. But, even though this claim is extremely unlikely, let’s do a little investigating.

So, the question: was the temperature in Alaska during December 2009 really 4-12.3°C warmer than 1998, or are those 9999s leaking through? This is what NASA’s GISS temperature map shows:

Happily, this is an easy question to answer if you actually look at the data. I downloaded the unadjusted mean GHCN data for the various sites in Alaska (the headers are 42570398000-425704820011). I picked out all the sites which had data for 2009 (I’ve also uploaded the raw data for 1998, 2008, 2009 for these sites so you can look at them if you like). Note that the temperature values are in tenths of a degree.



Dec 1998

Dec 2009


























St Paul




Cold Bay




King Salmon








Annette Island




I’ve helpfully marked highlighted the differences for those sites in Alaska in the region which are particularly red in the map. There appears to be some sort of correlation. The average temperature difference between Dec 1998 and Dec 2009 at those sites is 4.9°C warmer. The darkest shade of red represents an anomaly of between 4 and 12.3°C, so, Alaska is properly represented. The average, Alaska-wide, was 2.8°C warmer.

It’s not just me. A commenter on Watt’s Up With That, carrot eater, points out:

First station I tried: Goose, Newfoundland.


is 8.6 C warmer in Dec 09 than Dec 08.

Let’s look for other stations in red splotches in Dec 09, compared to Dec 08

Egesdesminde, Greenland 5.1 C
Fort Chimo, Canada. 10 C

Looks like I found your 10 C difference between Dec 08 and Dec 09. Third station I tried. Hence, the range of the colorbar.

Let’s see what else we find.
Danmarkshavn, Greenland. 2.7 C
Godthab Nuuk: 5 C
Inukjuak Quebec: 6.6 C
Coral Harbor: 8.6 C

So I’ve found a bunch of stations that are between 5 and 10 C warmer in Dec 09 compared to Dec 08.

This is a fun game, after all. Let’s say I want to find the biggest difference between Dec 09 and Dec 98. There are lots of red splotches on the map, and the colorbar has poor resolution. So I’ll download the gridded data and have a look.

Scrolling past all the 9999s for missing data, and I find that I should be looking at some islands north of Russia. I try some station called Gmo Im.E.T, and I get:

Dec 09 is 12.3 C warmer than Dec 98. First try.

So, yeah, this “bug” turned out to just be a weather fluctuation. Colour me surprised.