Evaluating weather forecast accuracy: an interview with Eric Floehr

Eric Floehr is the owner of ForecastWatch, a company that evaluates the accuracy of weather forecasts. In this interview Eric explains what his business does, how he got started, and some of the technology he uses.

JC: Let’s talk about your business and how you got started.

EF: I’m a programmer by trade. I got a computer science degree from Ohio State University and took a number of programming jobs, eventually ending up in management.

I’ve also always been interested in weather. A couple years ago my Mom showed me my baby book. At five years old it said “He’s interested in space, dinosaurs, and the weather.” I’m not as interested in dinosaurs now, but still interested in space and the weather.

When I was working as a programmer, and especially when I was a manager, I liked to do little programming projects to learn things. So when I ran across Python I thought about what I could write. I’d wondered whether there was any difference in the accuracy of various weather services — AccuWeather, Weather.gov, etc. Did they use different models, or did they all get their data from the National Weather Service and just package it up differently? So I wrote a little Python web scraper to pull forecasts from various places and compare it with observations. I kept doing that and realized there really were differences between the forecasters.

I didn’t start out for this to be a business. It just started out to satisfy personal curiosity. It just kept growing every year. In my last position before going out on my own I was CTO for a company that made a backup appliance. We got to the point where the product was mature and doing well. ForecastWatch was taking more and more of my time because I was getting more business from it, and so I decided to make the switch. That was March 2010. Revenue doubled over the next year and and it looks like this year it will double again. Things are going well and I really enjoy it.

JC: So you hadn’t been doing this that long when we met last year at SciPy in Austin.

EF: No, I’d only been doing this full time for a few months. But I’d been doing this part-time since 2004.

I didn’t have full-time revenue when I was doing this part-time. But it’s amazing. Once you have the time to focus on something, the opportunities that you hadn’t had time to notice before suddenly open up. Just the act of making something your focus almost makes your goal come to fruition. For years you think “too risky, too risky” and then once you make that jump, things fall in place.

JC: So what exactly is the product you sell?

EF: There are two main components. There’s an online component that is subscription-based. It provides monthly aggregated statistics on forecasts versus actual observations. It has absolute errors, min and max errors, Brier score, all kinds of statistics. It evaluates forecasts for precipitation, high and low temperature, opacity, wind speed and direction, etc. Meteriologist use those statistics to evaluate their forecasts to see how they’re doing relative to their peers.

The second component is research reports. Sometimes meteorologists will commission a report to show how well they’re doing. These reports are based on standard, widely-accepted metrics and time-frames, so they can’t just cherry-pick criteria they happened to do well on. But if they see there are statistics in ForecastWatch where they are doing really well, they might want to tell their customers. I’ve also created reports for media companies, large internet service providers, energy trading companies and other companies who were evaluating weather forecast providers or want some other data analysis related to weather forecasts.

Something else, and I don’t know whether this will become a major component, but another area some people are interested in is historical forecasts. I have agreements with some of the weather forecasting companies to sell their forecasts that are no longer forecasts. Some people find this information valuable. For example, a marketer with a major sports league wanted to know how weather forecasts affected attendance. Another example was an investment manager who was looking to invest in a business whose performance he believed had some correlation with weather forecasts. For example, a ski lodge might want to know how far out people base their decisions on forecasts.

I have this data back to 2004. It’s funny, but most weather forecasting companies historically have not kept their forecasts. Their bread-and-butter is the forecast in the future. Once that future becomes the past, they saw no value in that data until recently.

Incidentally, because I’m monitoring weather forecasters’ web sites, I sometimes let them know about errors they were unaware of.

JC: What volume of data are you dealing with?

EF: I have about 200,000,000 forecast data points back to 2004. I’m adding about 130,000 data points a day. My database is something on the order of 70 GB. That’s observation data, hourly forecasts, metadata, etc. Right now I’m looking at data from about 850 locations in the US and about 50 in Canada. I’m looking to expand that both domestically and internationally.

JC: So what kind of technology are you using?

EF: I’m running a LAMP stack: Linux, Apache, MySQL, Python. Originally I was on Red Hat Linux but I’ve switched to Ubuntu server. I’m using Django for the web site. Everything is in Python: the scrapers are in Python, the web site is in Python, all the administrative back-end is in Python.

There are two web sites right now: ForecastWatch.com, which is the subscription, professional site, and a free consumer site ForecastAdvisor.com. The consumer site will give you a local forecast and a measure of the accuracy for various forecasters for your weather.

JC: And who are your customers?

EF: All the major weather forecast companies. Also some financial companies, logistics and transportation companies, etc. I’m just starting to expand more into serving companies that depend on meterological forecasts whereas in the past I’ve focused directly on meterologists.

JC: Let’s talk a little more about the entrepreneurial aspect of your business.

EF: Well, for one thing, I don’t think I’d ever have done this if I’d thought about doing it to make money. There’s not an enormous market for this service, but in a way that’s good. I came from a completely technical background. There’s not a marketing or sales gene in my body and I’ve had to learn a lot. ForecastWatch has given me a great opportunity to learn about those non-technical areas of a business that were so foreign to me before.

I got into this entirely for my own use. And I thought that maybe there was already something that did what I wanted, and in the process of trying to find what’s out there I discovered an unmet need. Even though all the major forecasters said that accuracy was the number one thing they were interested in, they weren’t effectively measuring their accuracy. I thought that if I’m interested in this, maybe other people are too.

At first pricing was a mystery to me. Maybe I needed a new laptop, so I’d charge someone the price of a laptop for some analysis. I had to learn the value of my time and my product.

***

Some talks by Eric:

PyOhio 2009 talk about ForecastWatch
PyOhio 2010 panel on Python and entrepreneurship
SciPy 2010 talk

***

More interviews
More on entrepreneurship
More on Python

Tagged with: , ,
Posted in Business, Python
8 comments on “Evaluating weather forecast accuracy: an interview with Eric Floehr
  1. Nick Kotter says:

    Great story.

  2. Laura McLay says:

    Great interview–thanks for sharing.

  3. You might also want to include http://pycon.blip.tv/file/4879765/ in Eric’s talks. He just gave that talk on genetic algorithms two months ago at PyCon 2011, which was quite good.

  4. Tim H. says:

    Excellent interview! Thanks John and Eric.

  5. Speedmaster says:

    Fascinating topic, great interview!

    But does anyone else have trouble accessing http://forecastadvisor.com/ ?

  6. Shane says:

    Apparently, weather forecasting improvements from the US space program helped increase world agriculture output by 10% since 1960.

    I think this service is a great idea. I hope that he could expand the service to Ireland and Europe soon.

    I heard a few years ago that the ESA and some other organizations were launching forecasting satellites and so on to improve forecasting accuracy. I can’t believe that hundreds of millions are spent on weather forecasting and relatively little attention is paid to how accurate they may be.

    Sometimes, you might as well rely on reading the animal’s signs. ;)

  7. gwern says:

    To quote ciphergoth: http://lesswrong.com/r/discussion/lw/5bx/evaluating_weather_forecast_accuracy_an_interview/3zb0

    > Wow. How many things are there out there like this, where you just vaguely assume that someone’s already doing it – that weather companies are all data-crazy and keep every last byte and analyse it seven ways to Sunday and sell it in every way they can – and it turns out no-one was doing any such thing until this guy thought he’d write a Python script?

    Indeed.

  8. Have you ever noticed regularities in the residuals? For example:

    • errors not normally distributed / not symmetrically distributed about the actual observed value
    • in X season, the 14-day forecasts don’t adjust smoothly into the 5-day forecasts
    • the forecasts display an incorrect “momentum”
    • the relationship to historical norms is out of whack
    • relationships between different statistics is out of whack (maybe the prediction of daily low is less robust than the prediction of the time-path of temperature)
    • inappropriate response to instabilities in jetstream / hi lo pressure zones / something else that makes the forecast more unpredictable
    • any other battery of statistical tests on the (time×space×condition) residuals yielding nontrivial patterns

    In other words, do you think it’s possible that meteorologists are leaving some accuracy on the table? So that with only mathematics and no additional meteorological knowledge someone could improve the forecast?

    Lastly: do you think you could generate, with your data set, a “confidence interval” or Bayesian prior for what the weather will be (rather than a single-number prediction) ?

    …I don’t know if Mr Floehr still reads this page, but those are my questions. Thanks for a great interview on a fascinating topic!

5 Pings/Trackbacks for "Evaluating weather forecast accuracy: an interview with Eric Floehr"
  1. [...] Here is an interview with Eric Floehr at John D. Cook‘s very interesting blog. Floehr does exactly what I had wanted (assesses the accuracy of published weather forecasts), although for a cost. How curious am I? [...]

  2. [...] Evaluating weather forecast accuracy: an interview with Eric Floehr (johndcook.com) [...]

  3. [...] Floehr monitors weather forecasts for a living. Here’s what he had to say about when he started researching the accuracy of [...]

  4. [...] From The Signal and the Noise. The book gets some of its data from Eric Floehr of ForecastWatch. Read my interview with Eric here. [...]

  5. [...] Evaluating weather forecast accuracy: an interview with Eric Floehr Accuracy versus perceived accuracy [...]