1. Attachments are working again! Check out this thread for more details and to report any other bugs.

Berkeley Earth Temperature Averaging Process

Discussion in 'Environmental Discussion' started by bwilson4web, Aug 1, 2012.

  1. bwilson4web

    bwilson4web BMW i3 and Model 3

    Joined:
    Nov 25, 2005
    27,067
    15,372
    0
    Location:
    Huntsville AL
    Vehicle:
    2017 Prius Prime
    Model:
    Prime Plus
    This is an interesting paper Berkeley Earth Temperature Averaging Process (Robert Rohde, Richard Muller, Robert Jacobsen, Saul Perlmutter, Author Rosenfeld, Jonathan Wurtele, Don Groom, Judith Curry, and Charlotte Wickham, July 1, 2012) that details how they approached data analysis of the surface temperature measurements. Key elements are automated, noise reduction and dealing with discontinuities in observed data. Although applied to earth surface temperature measurements from 1800 to current, the techniques have a wider application.

    A key element is recursive application of a "Gaussian regression or Kriging." (pp. 5) Significant additions include automated outlier identification and instead of discarding, giving them a low weighting. This prevents a few bad actors from skewing the primary signal yet preserves some of their contribution (pp. 7).

    They also automatically identify discontinuities in the data. From the paper, "These could be caused by undocumented station moves, changes in instrumentation, or just the construction of a building nearby." (pp. 8) They split the record at that point, "scalping", making a second station which can then be independently added to the primary signal reconstruction.

    Dealing with microclimate effects is handled by a function that uses latitude, elevation, and "geographic anomaly" (pp. 8) The latitude and elevation are normal, smooth functions. The geographic part deals with "Gulf Stream and Jet Stream."

    The upshot is this:
    The outliers and their weighting are handled automatically.

    Now I'm peripherially interested in man-made, global warming. At age 62, I'm unlikely to personally see the more severe effects although some of the more irritating aspects are self evident. But this technical approach to dealing with imperfect data of a complex system has application to similar complex systems. For example, my daily work with a reasonably sized, computer network, shares similar characteristics.

    Old school, I'll have to print the paper, pull down the source code, and do a through desk check. But from what I see, there is no systemic error that would induce results different from what they published in their summary:
    [​IMG]

    Bob Wilson
     
  2. austingreen

    austingreen Senior Member

    Joined:
    Nov 3, 2009
    13,525
    4,057
    0
    Location:
    Austin, TX, USA
    Vehicle:
    2018 Tesla Model 3
    Model:
    N/A
    Bob,
    Watts's objection seems to be how they handle UHI effect adjustments and bad thermometers. Normally this discussion is done by dueling papers. If the B.E.S.T. reconstruction is done well, and Watts is done well, the watt's reconstruction should fall within the error bars of the B.E.S.T. reconstruction. The service B.E.S.T. is doing is making a great deal of data easy to access and easier to review, as well as making reconstructions from that data easier. Most of the previous old reconstructions came from the CRU, and the methodology of reconstruction was not clear. GISS and NCDC use there own data for modern times for use old CRU temperature reconstructions.

    The paper that seems to have been rejected and asked to be resubmitted to JGR, seems to have a problem with clarity and scope. In this internet age, it seems both Muller and Watts want to argue through press release of unpublished papers. The paper has to do with methodology for these thermometers, but seems to claim at least according to some objections conclusions beyond what the data says. I didn't see any real objection to the methodology or temperature reconstruction, only what the numbers mean.

    IMHO the reconstruction and data sets are getting better, but the press releases about them are quite political. Muller said he wanted to take the politics out of it, but maybe he just wanted to be at the center of it instead of Phil Jones.
     
  3. bwilson4web

    bwilson4web BMW i3 and Model 3

    Joined:
    Nov 25, 2005
    27,067
    15,372
    0
    Location:
    Huntsville AL
    Vehicle:
    2017 Prius Prime
    Model:
    Prime Plus
    The term I use is an 'electro-political' issue where an engineering decision is short circuited by considerations having nothing to do with the facts and data. I ran into it every company I've worked for and over time, learned patience as politics can delay a correct decision but politics and personalities can't stop the natural world.

    If Muller and Watts want to have a cat fight, I don't care. Certainly the honorable competition has never shirked from using lay forums and lame assertions to deny reality. That Muller's team fights back using the same tools and techniques, fine by me. As much as we like reading dueling papers we also live in a world with petty politics and the ethics that comes with it.

    I'm more interested in the methodology because it may apply to my work. I'm not above reading and applying productive techniques to make my work easier and/or more accurate. This paper has a lot of good techniques so I don't mind if it is just the 'draft' before the final. Short of finding some obvious (here is where the 'hocky stick' variables are inserted), I'm keeping an open mind.

    Bob Wilson
     
  4. tochatihu

    tochatihu Senior Member

    Joined:
    Apr 10, 2004
    8,972
    3,501
    0
    Location:
    Kunming Yunnan China
    Vehicle:
    2001 Prius
    By the kindness of strangers I have received the results papers from both BEST and WUWT, but I hoping for the methods information as well.

    Kringing seems now widely regarded as the right way to do the thing. But back in the day, the objections of some geostatisticians to this particular 'black magic' was amusing. To say the least. Every bit as contentious as the worst to be seen in applied thermometry now.

    Also we should not lose sight of the difference of scale in this 'duel' 30 vs 260 years, US 6% of global land area.
     
  5. austingreen

    austingreen Senior Member

    Joined:
    Nov 3, 2009
    13,525
    4,057
    0
    Location:
    Austin, TX, USA
    Vehicle:
    2018 Tesla Model 3
    Model:
    N/A
    Trying to get a signal from sparse and bad data is a problem in many fields. Since changes happen so slowly it actually seems less contentious in the temperature record.

    NOAA Paleoclimatology Global Warming - The Data

    Both B.E.S.T and Watts try to improve on NCDC and deal with bad data. Both of them want to throw out CRU, because of the possibility of Jones manipulating the data. I don't think either disagrees strongly with the NCDC result. Does anyone know if either use the satellite record like GISS?

    Two widely recognized research programs have used the available instrumental data to reconstruct global surface air temperature trends from the late 1800's through today. Both use the same land-based thermometer measurement records from the GHCN, but the records contain some differences. These differences are due to different approaches to spatial averaging, the use and treatment of sea surface temperature data (from ship observations), and the handling of the influence of changes in land-cover (i.e., increases in urbanization). However, both show the same basic trends over the last 100 years. The units shown are departures from the 1960 - 1990 period. For larger viewing version of these graphed temperature records, please click here or on the graph. To view animations of mapped CRU-UK global temperatures, click here.

    Global surface temperatures are constantly monitored to update and assess temperature trends in real time.
    Satellite-derived Temperature Records


    Satellite measurements have been used to reconstruct global atmospheric temperatures since 1979. The expectation is that the lower atmospheric temperature has warmed as the concentration of carbon dioxide has increased. For some time, these measurements appeared to indicate that warming was less rapid than ground-based thermometers and paleoclimate data suggested. Recent research, using an updated correction for a long-term change in the timing of satellite measurements, shows closer agreement between satellite and ground-based data. These results are described in the Synthesis and Assessment report produced by the Climate Change Science Program in 2006 (Temperature Trends in the Lower Atmosphere: Steps for Understanding and Reconciling the Differences).



    B.E.S.T. tries to use all the thermometers in the world:) But 30 years of data on a small percentage does seem to miss the point.