# Collocation Methods

## Introduction

Many methods for solving inverse problems involving ODE's need costly ODE solvers at each iteration to compute the loss. E.g. for L2 loss:

$\mathrm{Loss}_{\mathrm{L2}} = \left\|u - \hat{u}_{\mathrm{solver}}(u_0, t,p) \right\|_2$

Where $u$ is the observed data, $\hat{u}_{\mathrm{solver}}(u_0, t,p)$ is the ODE solver prediction of $u$, given initial conditions $u_0$, timespan $t$ and parameters $p$.

In order to reduce the computational cost, a collocation method may be used ^{[1]}, ^{[2]}. Collocation methods circumvents the need for an ODE solver by estimating the derivatives of the data. The loss is computed between these estimated derivatives and the system dynamics. E.g.

$\mathrm{Loss}_{\mathrm{L2}} = \left\|\hat{u}' - f(u, t,p) \right\|_2$

Where $\hat{u}'$ is the estimated derivatives of the observed data.

JuliaSimModelOptimizer implements three different collocation methods to estimate the derivative of the data: Kernel Collocation, Spline Collocation and Noise Robust Collocation.

## Kernel Collocation

`JuliaSimModelOptimizer.KernelCollocation`

— Type`KernelCollocation(; maxiters = nothing, maxtime = nothing, optimizer = IpoptOptimizer(; max_iter = maxiters), kernel = EpanechnikovKernel(), bandwidth = nothing, cutoff = (0.0, 0.0), kwargs...)`

A kernel based collocation method for solving the inverse problem. The derivatives of the data are estimated with a local linear regression using a kernel. The loss is then generated from the error between the estimated derivatives and the time derivatives predicted by solving the ODE system. Atleast one of `maxiters`

or `maxtime`

is required. If both of them are given, optimization stops after either of condition is first met.

**Keyword Arguments**

`maxiters`

: Maximum numbers of iterations when using the method with a time stepping optimizer.`maxtime`

: Maximum time for the optimization process.`optimizer`

: An Optimization.jl algorithm for the global optimization. Defaults to`Ipopt.Optimizer`

which can be used from`OptimizationMOI`

.`kernel`

: The kernel used to estimate the derivatives of the data. For more information, see: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631937/.`bandwidth`

: The bandwidth used for computing kernel weights for local kernel regression.`cutoff`

: The fraction of data in the beginning and end which is removed for loss computation.`kwargs`

: Keyword arguments passed onto`Optimization.solve`

.

## Spline Collocation

`JuliaSimModelOptimizer.SplineCollocation`

— Type`SplineCollocation(; maxiters = nothing, maxtime = nothing, optimizer = IpoptOptimizer(; max_iter = maxiters), interp = Datainterpolations.CubicSpline, interp_args = (), cutoff = (0.0, 0.0), kwargs...)`

A spline based collocation method for solving the inverse problem. The derivatives of the data are estimated by intepolating the data with a spline. The loss is then generated from the error between the estimated derivatives and the time derivatives predicted by solving the ODE system. Atleast one of `maxiters`

or `maxtime`

is required. If both of them are given, optimization stops after either of condition is first met.

**Keyword Arguments**

`maxiters`

: Maximum numbers of iterations when using the method with a time stepping optimizer.`maxtime`

: Maximum time for the optimization process.`optimizer`

: An Optimization.jl algorithm for the global optimization. Defaults to`Ipopt.Optimizer`

which can be used from`OptimizationMOI`

.`interp`

: The interpolation function used to estimate the derivatives of the data. For more information, see: DataInterpolations.jl.`interp_args`

: Extra arguments apart from states and time to be provided for methods like`BSplineInterpolation`

or`BSplineApprox`

of DataInterpolations.jl.`cutoff`

: The fraction of data in the beginning and end which is removed for loss computation.`kwargs`

: Keyword arguments passed onto`Optimization.solve`

.

## Noise Robust Collocation

`JuliaSimModelOptimizer.NoiseRobustCollocation`

— Type`NoiseRobustCollocation(; maxiters = nothing, maxtime = nothing, diff_iters, dx, α, optimizer = IpoptOptimizer(; max_iter = maxiters), tvdiff_kwargs = (), cutoff = (0.0, 0.0), kwargs...)`

A noise robust collocation method for solving the inverse problem. The derivatives of the data are estimated using *total variational regularized numerical differentiation* (`tvdiff`

). The loss is then generated from the error between the estimated derivatives and the time derivatives predicted by the ODE solver. Atleast one of `maxiters`

or `maxtime`

is required. If both of them are given, optimization stops after either of condition is first met.

For more information about the `tvdiff`

implementation, see: NoiseRobustDifferentiation.jl.

**Keyword Arguments**

`maxiters`

: Required. Maximum numbers of iterations when using the method with a time stepping optimizer.`optimizer`

: An Optimization.jl algorithm for the global optimization. Defaults to`Ipopt.Optimizer`

which can be used from`OptimizationMOI`

.`diff_iters`

: Required. Number of iterations to run the main loop of the differentiation algorithm.`α`

: Required. Regularization parameter. Higher values increase regularization strength and improve conditioning.`tvdiff_kwargs`

: The keyword arguments passed on to`NoiseRobustDifferentiation.tvdiff`

function for estimating derivatives.`cutoff`

: The fraction of data in the beginning and end which is removed for loss computation.`kwargs`

: Keyword arguments passed onto`Optimization.solve`

.

## CollocationData for plotting

`JuliaSimModelOptimizer.CollocationData`

— TypeA container for storing the original and the collocated data - both states and its derivatives, which can be used for visualization and validation purposes.

- 1Roesch, Elisabeth, Christopher Rackauckas, and Michael PH Stumpf. "Collocation based training of neural ordinary differential equations." Statistical Applications in Genetics and Molecular Biology 20, no. 2 (2021): 37-49.
- 2Liang, Hua, and Hulin Wu. "Parameter estimation for differential equation models using a framework of measurement error in regression models." Journal of the American Statistical Association 103, no. 484 (2008): 1570-1583. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631937/