Skip to content

Liquid Networks

Liquid Networks, specifically Liquid Time-Constant (LTC) networks, are a class of continuous-time Recurrent Neural Networks (RNNs) where the hidden state evolves according to a system of differential equations with state-dependent ("liquid") time constants.

They are a specialized subclass of Neural ODEs designed for stability, causality, and efficient modeling of stiff dynamical systems.

Definition and Formulation

While a standard Neural ODE defines dynamics as \(\dot{z} = f_\theta(z, t)\), an LTC network structures the differential equation based on the biophysics of leaky integrate-and-fire neurons and synaptic transmission.

The state evolution is governed by:

\[ \frac{dx(t)}{dt} = -\left[ \frac{1}{\tau} + S(t) \right] \odot x(t) + S(t) \odot A \]

where:

  • \(x(t)\) is the hidden state vector.
  • \(\tau\) is a fixed base time-constant.
  • \(S(t) = f_\theta(x(t), I(t))\) is a neural network (e.g., sigmoid-based) representing total synaptic input, dependent on state \(x(t)\) and external input \(I(t)\).
  • \(A\) is a bias parameter (resting potential or driving force).
  • \(\odot\) denotes element-wise multiplication.

The "Liquid" Time Constant

The term \(-[1/\tau + S(t)]\) acts as a time-varying relaxation rate. This means the effective time constant of the system, \(\tau_{\text{eff}}(t) = (1/\tau + S(t))^{-1}\), changes depending on the input and the current state. This adaptivity allows the network to handle data varying at multiple timescales within a single sequence.

Relationship with Neural ODEs

LTCs are effectively a structured Neural ODE.

  • Generic Neural ODE: \(\dot{x} = \text{MLP}(x, t)\)

    • Flexible, universal approximator.
    • Can be hard to train (stiffness).
    • No guarantee of stability.

  • Liquid Network: \(\dot{x} = -\frac{1}{\tau(x)} x + \dots\)

    • Constrained architecture.
    • Inductive bias for stable, decay-driven dynamics.
    • More interpretable in terms of signal processing.

In the context of SOFAx, Liquid Networks can serve as robust surrogates for subsystems that exhibit relaxation dynamics, or as stable components in a larger Hybrid Approach.

Closed-form Continuous-time (CfC) Models

Solving the LTC differential equation requires a numerical ODE solver (e.g., Runge-Kutta), which can be slow during training and inference.

Closed-form Continuous-time (CfC) networks provide a closed-form approximation to the solution of the LTC integral. By assuming piecewise interactions or using specific approximations, CfCs predict the state \(x(t)\) directly without a loop-based solver:

\[ x(t) \approx (x_0 - A) \odot \exp\left( - \left[ \frac{1}{\tau} + f_\theta(x_0, I) \right] t \right) \odot \sigma(g_\theta(x_0, I, t)) + A \]

(Note: The actual CfC architecture uses a more complex gating mechanism combining exponential decay and sigmoidal nonlinearities to approximate the integral solution).

Benefits:

  • Speed: Orders of magnitude faster than solver-based Neural ODEs.
  • Accuracy: Retains the stability and expressivity of LTCs.

See also