When Is 88 Greater Than 246?

Apologies to my fans for not posting over the last few days.  An accumulation of personal and professional responsibilities diverted my attention. One reader wondered where I had gone. I am here, and I am back with a new CAO riddle.

Question: When is 88 greater than 246?

Answer: When you are discussing Best Available Science for wetlands with a nationally recognized expert.

During the past week, a new controversy arose at our Planning Commission meeting (see previous post) regarding the scientific justification for our buffers. Our County has based its wetland buffer proposal on a paper by Mayer (Mayer et al, 2007). Dr. Paul Adamus, who is an unpaid non-tenured courtesy faculty member at Oregon State, is the "nationally recognized" wetland expert serving as our County consultant. He has been unwavering in his support for the Mayer paper as the basis for our buffer sizes.

One of our local citizens decided to discuss the Mayer paper with Mayer himself. Interestingly, Mayer recommended several other papers that he thought provided equal or greater insight into the workings of buffers.  Mayer also acknowledged that his study had not demonstrated a strong relationship between pollutant removal/retention and buffer width. As Mayer shows in his paper, buffer width explains only about 9% of pollutant removal/retention. Pollutant removal depends 91% on something other than width.

An additional reference recommended by Mayer was a statistical meta-analysis by Zhang, et al., 2009. The Zhang paper is very interesting for several reasons. First, as mentioned, it is a meta-analysis, and we have been told by Dr. Adamus on innumerable occasions that meta-analysis is the best of the best. It's state of the art. It's the kind of analysis performed by epidemiologists. So, the Zhang paper would appear to be in the same category of excellence as the Mayer paper (and remember, Mayer himself is recommending Zhang). Second, the Mayer paper looks at only one pollutant, nitrogen/nitrate, whereas the Zhang paper looks at four: sediment, nitrogen, phosphorous, and pesticides. Third, Dr. Adamus admits that the Zhang paper already is part of the BAS adopted by the County.

One of the major conclusions of the Zhang paper is that buffers, for all practical purposes, have a maximum effective width. At some point, buffer width reaches diminishing returns. Had this principle been incorporated into our County's proposed buffer methodology, we would have ended up with smaller (probably much smaller) proposed buffers. Dr. Adamus acknowledges that he discarded the conclusions of the Zhang paper in favor of the Mayer paper. The question becomes why did Dr. Adamus choose one meta-analysis (Mayer) over another (Zhang)?

The Planning Commission asked Dr. Adamus that question: why Mayer and not Zhang? The answer from Adamus: Mayer used a bigger sample size. Adamus stated this as justification more than once during the Planning Commission meeting.

There are several difficulties with that answer, not the least of which is that it may not be true. The Zhang paper looked at 81 studies for sediment removal, 61 for nitrogen, 52 for phosphorous, and 52 for pesticides. The Mayer study looked at 0 studies for sediment removal, 88 studies for nitrogen, 0 for phosphorous, and 0 for pesticides. If you look at the data across all pollutants, Mayer's sample size was 88 and Zhang's was 246 (81+61+52+52). Mayer's sample size was greater only for nitrogen specifically (88 versus 61). On that basis, Adamus decided to use the Mayer nitrogen results as a proxy for every other pollutant, even though Zhang had actual data for other pollutants.

It's hard to completely fathom Dr. Adamus' answer, and he appears to be suggesting that data quantity trumps data quality too. As a result, not only his conclusion but his entire reasoning about statistical validity may be specious. Data quality and usability are critically important, and there are several acceptance criteria typically employed to evaluate data on those dimensions. They are called the PARCC criteria: precision, accuracy, representativeness, completeness, and comparability. Occasionally, these criteria are supplemented by a sixth criterion, sensitivity.

Sample size, if it has an effect, exerts its influence on these quality criteria, but sample size is not necessarily a worthwhile discriminator in its own right. Did Zhang's smaller nitrogen sample size affect the precision, for example? Actually, for the Zhang paper, the variance of the nitrogen data appears to be smaller than Mayer's, indicating that the Zhang data may be more precise even though it has fewer nitrogen data points than Mayer. Overall, based on a more thorough evaluation of acceptance criteria, Zhang may be the better study to use, even for nitrogen.

Mayer might think so.  That's probably why he recommended it.