Cross validation
Cross validation approximates the performance of a trained model on new data. These methods are useful for validating the performance of UDE models and for choosing between alternative models. UniversalDiffEq.jl supports two Cross validation methods, leave future out cross validation and k-fold cross validation. These two methods are based on the same assumption, we can approximate the models perforance on new data by testing it on data left out of the existing data set. Leave future out and k-fold cross validation differ in how these dat sets are constructed. Leave future out cross validation sequentially leaves data off the end of the time sereis data set with the goal of approximating the models preformance forecasting new observaitons. k-fold cross validation approximates the models ability to explain the historical dynamics of the time series by leaving block of consequtive observations out of the middel data set. Each method constructes several training and testing data sets to reduce the effect of random variation on the estiamtes of model perforamnce.
K-fold cross validation
k-fold cross validation breaks the data set up into k equally sized blocks of sequential observations. The algorithm trains the model on all but one of the blocks which is used as the testing data set and repeates this procedure leaving each block out one at a time.
The models performance is evalauted predicting one time step ahead in the testing data set. The initial points for the preditions are estimated using a particle filter algorithm, described in detail below. The forecasts are calcualted starting from the estiamte of the state variable estiamted by the particle filter and compared to the observed data point one step into the future.
UniversalDiffEq.cross_validation_kfold — Methodcrossvalidationkfold(model::UDE; kwagrs...)
This funciton approximates model performance on out of sample data by leaving blocks of consequtive observaitons out of the training data. The model is trained on the remiaining observations and the and the one step ahead prediction accuracy is calcualted on the testing data set. This procedure is repeated k times. ...
Arguments
k = 10 - the number of testing data sets BFGS = false - weather or not to train the models with the BFGS algorithm stepsize = 0.05 - the step size for the first round of gradient descent optimization maxiter = 500 - the number of iterations for the first round of gradinet descent stepsize2 = 0.05 - the step size for the second round of gradient descent maxiter2 = 500 - the number of iterations for the second round of gradient descent N = 1000 - the number of particle to use in the particle filter algorithm that estiamtes the states in the out of sample data nugget = 10^-10 - a small number to added variance terms in the particle filter algorith to improve numerical stability ...
Leave future out cross validation
Leave future out cross validation creates training data sets by leaving observation off of the end of the data set. The model performacne is calculated by forecasting over the length of the data set and and the forecasting skill is quatified by the squared error between the model predictions and the testing data. The process is repreated on new testing data sets constructed by reoving observaitons from the end of the data set.
UniversalDiffEq.leave_future_out_cv — Methodleave_future_out_cv(model::UDE; kwargs ...)Runs K fold leave future out cross validation and returns the mean squared forecasting error and a plot to visualize the model fits.
...
Arguments
model - the UDE model to test forecastlength = 5 - The number of data points to forecast forecastnumber = 10 - The number of trainin gnad testing data sets to create spacing = 2 - The number of data points to skip between the end of each new triaining data set BFGS=false - use BFGS algorithm to train the model stepsize = 0.05 - step size for first run of the ADAM algorithm maxiter = 500 - maximum iterations for first trial of ADAM stepsize2 = 0.01 - step size for second iteration of ADAM, only used of BFGS is false maxiter2 = 500 - step size for second iteration of ADAM, only used of BFGS is false ...
Particle filter algorithm
Particel filter algorithms are a method for estimating the vaue of unobserved state variables given a time series and state space model. The Particle filter algorithms used in the cross validation procedures use the trained UDE model for the deterministic components of the process and observation models. The process and observaiton errors are estiamted by calcualting the total variance of the process and observation residuals $\sigma_{total}$. The total variacne is then partitioned between process and observaiton error to match the ratio between the process and observaiton weights used to train the model. The full details of this algorithm are discussed in the supporting information to Buckner et al. 2024.