Model performance
UniversalDiffEq.jl provides a number of functions to test the performance of NODE and UDE models on in-sample data. These tests validate the model fitting procedure. Functions to estimate the model's performance on out-of-sample data are discussed below and in the section on cross-validation.
Evaluating model fits to training data
There are two primary functions to evaluate model fits: plot_state_estimates and plot_predictions. The training procedure used by UniversalDiffEq.jl simultaneously smooths the training data and trains the parameters of the UDE model on the smoothed dataset. The function plot_state_estimates compares the smoothed time series produced by the training procedure to the observations in the dataset. The smoothed time series (grey line) needs to capture the main trends in training data (blue dots) for the model to accurately recover the dynamics in the dataset (Fig. 1).
Figure 1: Model fit to simulated training data
UniversalDiffEq.plot_state_estimates — Methodplot_state_estimates(UDE::UDE)Plots the values of the state variables estimated by the UDE model.
We can make this analysis a bit more rigorous by looking for correlations in the observation errors using the observation_error_correlations function. This creates a lag plot for each pair of variables in the model and calculates the correlation coefficent. Large correlations in the observation errors suggest the UDE model may be missing predictable variation in the dataset.
UniversalDiffEq.observation_error_correlations — Methodobservation_error_correlations(UDE)The first differnce plot of the observation errors $psilon_t$. This allows the user to check for autocorrelation in the observation errors.
The plot_predictions functions compares the predictions of UDE model one step into the future to the estimated sequence of state variables. This function quantifies the predictive accuracy of the model for in-sample data.
UniversalDiffEq.plot_predictions — Methodplot_predictions(UDE::UDE)Plots the correspondence between the observed state transitions and the predictions from the UDE model.
Model forecasting with testing data
There are also functions to compare the model predictions to out-of-sample data. The simplest is plot_forecast, which compares the observations in the test dataset to a deterministic simulation from the dataset, which starts at the first observation and runs to the end of the test data.
UniversalDiffEq.plot_forecast — Methodplot_forecast(UDE::UDE, test_data::DataFrame)Plots the model's forecast over the range of the test data along with the value of the test data.
It is also possible to test the performance of the models one time step into the future using the plot_predictions function. When a test dataset is supplied to the plot_predictions function, it will run a series of forecasts starting at each point in the dataset, predicting one time step into the future. The function returns a plot comparing the predicted and observed changes.
UniversalDiffEq.plot_predictions — Methodplot_predictions(UDE::UDE, test_data::DataFrame)Plots the correspondence between the observed state transitions in test data and the predictions from the UDE model.