Model Validation Technical Report

Object

Compare wind observations from weather stations and wind forecasted by weather models for various locations around the world in order to rank weather models in terms of wind accuracy. This study focuses on the following weather models: ECMWF, SPIRE, UKMO and GFS.

Dataset

  • The study covers a 4-month period (Feb 12th to June 13th, 2020)
  • 441 land-based stations and 166 buoys available (for ECMWF). Land based stations are mostly close to coastline and buoys are in open ocean.
  • For land based stations, the dataset is made of around 80,000 data points and for buoys around 15,000. By averaging those large dataset, we get a robust statistical value that removes any bad stations, bad forecast day, etc) in order to compare overall model performance.
  • The wind forecasts cover a 7 day period (day 1 to day 7)
  • ECMWF, SPIRE and GFS have a full day 7-day forecast, however UKMO however only has a 5-day forecast.

Methodology

For each model run (0Z and 12Z) we calculated the Mean Absolute Error (MAE) between the model and the observation. The MAE is calculated for each day of the forecast (day1 to day 7).

The resulting MAE is then averaged for every station and every model run, to get a robust statistical value that truly represents the overall accuracy of the model. Example: TWS MAE for ECMWF on Day1 = 2.6kn for land based stations. This means that for 82000 data points the average error is 2.6kn between the ECMWF model and all the observations on land.

The weather model comparison is done for land-based stations and another model comparison for buoys.

Analysis TWS MAE

Land Based Stations Analysis

TWS MAE
Day1
Day2
Day3
Day4
Day5
Day6
Day7
Average
ECMWF
2.6
2.7
2.8
3.0
3.2
3.4
3.7
3.1
SPRIRE
2.9
3.0
3.1
3.2
3.3
3.6
3.8
3.3
UKMO
3.1
3.2
3.4
3.5
3.7
3.4
GFS
3.2
3.3
3.4
3.6
3.8
4.0
4.1
3.6

TABLE: TWS MAE for Land-Based stations

Table Analysis:

  • ECMWF is ranked number 1. It is consistently the best on each day of the forecast (day+1 to day+7).
  • SPIRE is ranked number 2.
  • UKMO average is only done over the first 5 days of the forecast so it is not fair to compare to other model average which include a 7 day average. However over the first 5 days, UKMO is slightly better than GFS.

BUOY Analysis

TWS MAE
Day1
Day2
Day3
Day4
Day5
Day6
Day7
Average
SPIRE
2.5
2.6
2.8
3.2
3.5
3.8
4.3
3.2
ECMWF
2.5
2.7
3.0
3.2
3.6
3.9
4.3
3.3
UKMO
2.7
3.0
3.3
3.7
3.8
3.3
GFS
2.8
3.1
3.3
3.7
4.0
4.3
4.6
3.7

TABLE: TWS MAE for Buoy

Table Analysis:

  • SPIRE is ranked number 1.
  • SPIRE and ECMWF are close whereas GFS is significantly less accurate.
  • During the first 5 days UKMO is ranked number 3.

Analysis TWD MAE

Land Based Stations Analysis

TWD MAE
Day1
Day2
Day3
Day4
Day5
Day6
Day7
Average
ECMWF
20
21
23
26
30
34
40
28
SPIRE
22
24
25
27
30
33
36
28
UKMO
22
24
27
30
32
27
GFS
24
26
29
32
35
40
44
33

TABLE: TWD MAE for Land-Based stations

Table Analysis:

  • ECMWF and SPIRE are the best on average.
  • ECMWF is the best from day 1-3, and SPIRE is the best from Day 6-7

BUOY Analysis

TWD MAE
Day1
Day2
Day3
Day4
Day5
Day6
Day7
Average
SPIRE
17
19
21
23
26
30
33
24
UKMO
17
21
25
28
29
34
39
24
ECMWF
19
21
23
26
30
27
GFS
20
22
25
28
32
36
40
29

TABLE: TWD MAE for Buoy

Table Analysis:

  • SPIRE is ranked number 1. It is consistently the best on each day of the forecast.
  • ECMWF and UKMO are close. GFS least accurate.

Conclusion

By comparing forecasts to hundreds of wind observations around the globe, ECMWF and SPIRE both perform better than GFS and UKMO at forecasting the wind.

ECMWF and SPIRE are close in terms of ranking, and results suggest that ECMWF is slightly better close to shore (land-based station), and SPIRE is better on the open ocean (buoy).