The Persistence of Errors: How Evaluating Models over Data Partitions Relates to a Global Evaluation
Daniel Klotz, Martin Gauch, Grey Nearing, Sepp Hochreiter, and Frederik Kratzert
Skillful today, inept tomorrow. Today’s hydrological models have pronounced and complex error dynamics (e.g., small, highly correlated errors for low flows and large, random errors for high flows). Modelers generally accept that simple, variance based evaluation criteria — like the Nash-Sutcliffe Efficiency (NSE) — are not fully able to capture these intricacies. The (implied) consequences of this are however seldom discussed.
This contribution examines how evaluating the model over two data partitions (above and below a chosen threshold) relates to a global model evaluation of both partitions combined (i.e., the usual way of computing the NSE). For our experiments we manipulate dummy simulations with gradient descent to approximate specific NSE values for each partition individually. Specifically, we set the NSE for runoff values that fall below the threshold, and vary the NSE of the simulations above the threshold as well as the threshold itself. This enables us to study how the global NSE relates to the partition NSEs and the threshold. Intuitively, one would wish that the global NSE somehow reflects the performance on the partitions in a comprehensible manner. We do however show that this relation is not trivial.
Our results also show that subdividing the data and evaluating over the resulting partitions yields different information regarding model deficiencies than an overall evaluation. The downside is that we have less data to estimate the NSE. In the future we can use this for model selection and diagnostic purposes.
EGU23-15221, 2023-02-22.