GeoForecaster's Michael Kozuch and Lowell Whiteside didn't charge much for their earthquake forecasts. It might be argued that $9.95 a month doesn't harm anyone. For their money, customers got long lists of little earthquakes most of the time. Kozuch and Whiteside weren't harming their scientific reputations—they didn't have much of any. They weren't very smart Web marketers, hiding all of their methods and not allowing a free look at the product.
The Harm of Poor Predictions
But there are two kinds of harm in their approach. First, some day they might have issued a forecast for a large, damaging quake (although they never did), and the authorities would have had to judge their claim somehow as the clock ticked and the press clamored at the door. The longer they did business, such a forecast would have been inevitable, but in three years of selling their forecasts (except in California, where they were not licensed to practice) they never tried to warn officials. Perhaps they would rather have avoided such a forecast—and what would that say about their confidence in their methods?
Second, the public was led to believe that GeoForecaster was using the rigorous tools of science. As we've seen in the case of evolution, science has a precious reputation at stake that is easily degraded by a public that believes what pleases it. Keep those points in mind as I spell out the faults I see with the structure of their statistics. (You can confirm what follows from the archives of their Web site.)
The things I'm pointing out might seem subtle, but people with PhD's can point them out without a moment's thought. As Kozuch and Whiteside both have doctorates, these flaws can only be deliberate choices to use substandard methods.
- They counted random events as successful forecasts. Consider that in the regions they covered (California, Japan and Taiwan), earthquakes of all sizes occur constantly. Forecasting small quakes there is like forecasting waves at the beach. It is rare that a forecast will not include something they could call a complete success, and we know this because they proclaimed an overall success rate of 83 percent!
- Their criterion for a successful location was flawed. Instead of measuring the distance from an earthquake directly, as the radius of a circle, they used a longitude-latitude box. This allowed them to call a "hit" an event as much as 1.4 times farther than the radius. (See the illustration.)
- They assessed latitude and longitude separately. That way they could declare an "A" quality success for matching an event's latitude, even if the longitude was completely wrong. This procedure was not justified by scientific practice—or even common sense.
- There was a big loophole in the magnitude criterion: an event larger than the forecast could be counted as a success. Considering that all of southern California should be carpeted with forecasts for small quakes (GeoForecaster's go down to M2), magnitude-5 events like the quake of 22 February 2003 would always score as successes, even if they weren't predicted. Thus Kozuch and Whiteside claimed over 90 percent success for earthquakes larger than magnitude 6.5, yet they didn't say they actually issued forecasts for events of that size.
- They committed the basic statistical distortion of turning a semiquantitative measure of success (their 16-point ABCD metric) into a quantitative "AccuCast" index (0 to 100%), introducing false precision. It's the same error as reading the length of a car trip from your dashboard gauge, then reporting it in centimeters.
Still, if people thought it useful to get a weekly newsletter that claimed to announce every magnitude-2 event in advance, within 5 days and 80 kilometers, then they could pay their money and enjoy the illusion as long as GeoForecaster stayed in business. Not me. And the state of California agreed, forbidding the firm from serving its residents—as the firm disclosed in a discreet disclaimer.
PS: Lowell Whiteside wrote a stirring call to action a few years ago, in Seismological Research Letters: "A thousand years from now people will remember that scientists of our era solved the earthquake prediction problem," he said. "We can choose to be a part of that process or a hindrance to it, but in the end a solution will be found." Sometimes despite our best efforts, we choose wrong.