# Bayesian Temporal Matrix Factorization

**Published**: October 8, 2019

**Revised**: October 8, 2020

**Author**: Xinyu Chen [[**GitHub homepage**](https://github.com/xinychen)]

**Download**: This Jupyter notebook is at our GitHub repository. If you want to evaluate the code, please download the notebook from the [**transdim**](https://github.com/xinychen/transdim/blob/master/imputer/BTMF.ipynb) repository.

This notebook shows how to implement the Bayesian Temporal Matrix Factorization (BTMF), a fully Bayesian matrix factorization model, on some real-world data sets. To overcome the missing data problem in multivariate time series, BTMF takes into account both low-rank matrix structure and time series autoregression. For an in-depth discussion of BTMF, please see [1].

[1] Xinyu Chen, Lijun Sun (2019). Bayesian temporal factorization for multidimensional time series prediction. arXiv:1910.06366. [PDF]

## Bayesian Temporal Matrix Factorization Model

### 1 Model Specification

Following the general Bayesian probabilistic matrix factorization models (e.g., BPMF proposed by [Salakhutdinov & Mnih, 2008](https://www.cs.toronto.edu/~amnih/papers/bpmf.pdf)), we assume that each observed entry in $Y$ follows a Gaussian distribution with precision $\tau$:
\begin{equation}
y_{i,t}\sim\mathcal{N}\left(\boldsymbol{w}_i^\top\boldsymbol{x}_t,\tau^{-1}\right),\quad \left(i,t\right)\in\Omega.
\label{btmf_equation3}
\end{equation}

On the spatial dimension, we use a simple Gaussian factor matrix without imposing any dependencies explicitly:
\begin{equation}
\boldsymbol{w}_i\sim\mathcal{N}\left(\boldsymbol{\mu}_{w},\Lambda_w^{-1}\right),
\end{equation}
and we place a conjugate Gaussian-Wishart prior on the mean vector and the precision matrix:
\begin{equation}
\boldsymbol{\mu}_w | \Lambda_w \sim\mathcal{N}\left(\boldsymbol{\mu}_0,(\beta_0\Lambda_w)^{-1}\right),\Lambda_w\sim\mathcal{W}\left(W_0,\nu_0\right),
\end{equation}
where $\boldsymbol{\mu}_0\in \mathbb{R}^{R}$ is a mean vector, $\mathcal{W}\left(W_0,\nu_0\right)$ is a Wishart distribution with a $R\times R$ scale matrix $W_0$ and $\nu_0$ degrees of freedom.

In modeling the temporal factor matrix $X$, we re-write the VAR process as:
\begin{equation}
\begin{aligned}
\boldsymbol{x}_{t}&\sim\begin{cases}
\mathcal{N}\left(\boldsymbol{0},I_R\right),&\text{if $t\in\left\{1,2,...,h_d\right\}$}, \\
\mathcal{N}\left(A^\top \boldsymbol{v}_{t},\Sigma\right),&\text{otherwise},\\
\end{cases}\\
\end{aligned}
\label{btmf_equation5}
\end{equation}

Since the mean vector is defined by VAR, we need to place the conjugate matrix normal inverse Wishart (MNIW) prior on the coefficient matrix $A$ and the covariance matrix $\Sigma$ as follows,
\begin{equation}
\begin{aligned}
A\sim\mathcal{MN}_{(Rd)\times R}\left(M_0,\Psi_0,\Sigma\right),\quad
\Sigma \sim\mathcal{IW}\left(S_0,\nu_0\right), \\
\end{aligned}
\end{equation}
where the probability density function for the $Rd$-by-$R$ random matrix $A$ has the form:
\begin{equation}
\begin{aligned}
&p\left(A\mid M_0,\Psi_0,\Sigma\right) \\
=&\left(2\pi\right)^{-R^2d/2}\left|\Psi_0\right|^{-R/2}\left|\Sigma\right|^{-Rd/2} \\
&\times \exp\left(-\frac{1}{2}\text{tr}\left[\Sigma^{-1}\left(A-M_0\right)^{\top}\Psi_{0}^{-1}\left(A-M_0\right)\right]\right), \\
\end{aligned}
\label{mnpdf}
\end{equation}
where $\Psi_0\in\mathbb{R}^{(Rd)\times (Rd)}$ and $\Sigma\in\mathbb{R}^{R\times R}$ are played as covariance matrices.

For the only remaining parameter $\tau$, we place a Gamma prior  $\tau\sim\text{Gamma}\left(\alpha,\beta\right)$ where $\alpha$ and $\beta$ are the shape and rate parameters, respectively.

The above specifies the full generative process of BTMF, and we could also see the Bayesian graphical model shown in Figure 4. Several parameters are introduced to define the prior distributions for hyperparameters, including $\boldsymbol{\mu}_{0}$, $W_0$, $\nu_0$, $\beta_0$, $\alpha$, $\beta$, $M_0$, $\Psi_0$, and $S_0$. These parameters need to provided in advance when training the model. However, it should be noted that the specification of these parameters has little impact on the final results, as the training data will play a much more important role in defining the posteriors of the hyperparameters.

> **Figure 4**: An overview graphical model of BTMF (time lag set: $\left\{1,2,...,d\right\}$). The shaded nodes ($y_{i,t}$) are the observed data in $\Omega$.

### 2 Model Inference

Given the complex structure of BTMF, it is intractable to write down the posterior distribution. Here we rely on the MCMC technique for Bayesian learning. In detail, we introduce a Gibbs sampling algorithm by deriving the full conditional distributions for all parameters and hyperparameters. Thanks to the use of conjugate priors in Figure 4, we can actually write down all the conditional distributions analytically. Below we summarize the Gibbs sampling procedure.

#### 1) Sampling Factor Matrix $W$ and Its Hyperparameters

> For programming convenience, we use $W\in\mathbb{R}^{N\times R}$ to replace $W\in\mathbb{R}^{R\times N}$.

```python
import numpy as np
from numpy.linalg import inv as inv
from numpy.random import normal as normrnd
from scipy.linalg import khatri_rao as kr_prod
from scipy.stats import wishart
from scipy.stats import invwishart
from numpy.linalg import solve as solve
from numpy.linalg import cholesky as cholesky_lower
from scipy.linalg import cholesky as cholesky_upper
from scipy.linalg import solve_triangular as solve_ut
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def mvnrnd_pre(mu, Lambda):
    src = normrnd(size = (mu.shape[0],))
    return solve_ut(cholesky_upper(Lambda, overwrite_a = True, check_finite = False),
                    src, lower = False, check_finite = False, overwrite_b = True) + mu

def cov_mat(mat, mat_bar):
    mat = mat - mat_bar
    return mat.T @ mat

def ten2mat(tensor, mode):
    return np.reshape(np.moveaxis(tensor, mode, 0), (tensor.shape[mode], -1), order = 'F')
```

```python
def sample_factor_w(tau_sparse_mat, tau_ind, W, X, tau, beta0 = 1, vargin = 0):
    """Sampling N-by-R factor matrix W and its hyperparameters (mu_w, Lambda_w)."""

    dim1, rank = W.shape
    W_bar = np.mean(W, axis = 0)
    temp = dim1 / (dim1 + beta0)
    var_mu_hyper = temp * W_bar
    var_W_hyper = inv(np.eye(rank) + cov_mat(W, W_bar) + temp * beta0 * np.outer(W_bar, W_bar))
    var_Lambda_hyper = wishart.rvs(df = dim1 + rank, scale = var_W_hyper)
    var_mu_hyper = mvnrnd_pre(var_mu_hyper, (dim1 + beta0) * var_Lambda_hyper)

    if dim1 * rank ** 2 > 1e+8:
        vargin = 1

    if vargin == 0:
        var1 = X.T
        var2 = kr_prod(var1, var1)
        var3 = (var2 @ tau_ind.T).reshape([rank, rank, dim1]) + var_Lambda_hyper[:, :, None]
        var4 = var1 @ tau_sparse_mat.T + (var_Lambda_hyper @ var_mu_hyper)[:, None]
        for i in range(dim1):
            W[i, :] = mvnrnd_pre(solve(var3[:, :, i], var4[:, i]), var3[:, :, i])
    elif vargin == 1:
        for i in range(dim1):
            pos0 = np.where(sparse_mat[i, :] != 0)
            Xt = X[pos0[0], :]
            var_mu = tau[i] * Xt.T @ sparse_mat[i, pos0[0]] + var_Lambda_hyper @ var_mu_hyper
            var_Lambda = tau[i] * Xt.T @ Xt + var_Lambda_hyper
            W[i, :] = mvnrnd_pre(solve(var_Lambda, var_mu), var_Lambda)

    return W
```

#### 2) Sampling VAR Coefficients $A$ and Its Hyperparameters

**Foundations of VAR**

Vector autoregression (VAR) is a multivariate extension of autoregression (AR). Formally, VAR for $R$-dimensional vectors $\boldsymbol{x}_{t}$ can be written as follows,
\begin{equation}
\begin{aligned}
\boldsymbol{x}_{t}&=A_{1} \boldsymbol{x}_{t-h_1}+\cdots+A_{d} \boldsymbol{x}_{t-h_d}+\boldsymbol{\epsilon}_{t}, \\
&= A^\top \boldsymbol{v}_{t}+\boldsymbol{\epsilon}_{t},~t=h_d+1, \ldots, T, \\
\end{aligned}
\end{equation}
where
\begin{equation}
A=\left[A_{1}, \ldots, A_{d}\right]^{\top} \in \mathbb{R}^{(R d) \times R},\quad \boldsymbol{v}_{t}=\left[\begin{array}{c}{\boldsymbol{x}_{t-h_1}} \\ {\vdots} \\ {\boldsymbol{x}_{t-h_d}}\end{array}\right] \in \mathbb{R}^{(R d) \times 1}.
\end{equation}

In the following, if we define
\begin{equation}
Z=\left[\begin{array}{c}{\boldsymbol{x}_{h_d+1}^{\top}} \\ {\vdots} \\ {\boldsymbol{x}_{T}^{\top}}\end{array}\right] \in \mathbb{R}^{(T-h_d) \times R},\quad Q=\left[\begin{array}{c}{\boldsymbol{v}_{h_d+1}^{\top}} \\ {\vdots} \\ {\boldsymbol{v}_{T}^{\top}}\end{array}\right] \in \mathbb{R}^{(T-h_d) \times(R d)},
\end{equation}
then, we could write the above mentioned VAR as
\begin{equation}
\underbrace{Z}_{(T-h_d)\times R}\approx \underbrace{Q}_{(T-h_d)\times (Rd)}\times \underbrace{A}_{(Rd)\times R}.
\end{equation}

> To include temporal factors $\boldsymbol{x}_{t},t=1,...,h_d$, we also define $$Z_0=\left[\begin{array}{c}{\boldsymbol{x}_{1}^{\top}} \\ {\vdots} \\ {\boldsymbol{x}_{h_d}^{\top}}\end{array}\right] \in \mathbb{R}^{h_d \times R}.$$

**Build a Bayesian VAR on temporal factors $\boldsymbol{x}_{t}$**
\begin{equation}
\begin{aligned}
\boldsymbol{x}_{t}&\sim\begin{cases}\mathcal{N}\left(A^\top \boldsymbol{v}_{t},\Sigma\right),~\text{if $t\in\left\{h_d+1,...,T\right\}$},\\{\mathcal{N}\left(\boldsymbol{0},I_R\right),~\text{otherwise}}.\end{cases}\\
A&\sim\mathcal{MN}_{(Rd)\times R}\left(M_0,\Psi_0,\Sigma\right), \\
\Sigma &\sim\mathcal{IW}\left(S_0,\nu_0\right), \\
\end{aligned}
\end{equation}
where
\begin{equation}
\begin{aligned}
&\mathcal{M N}_{(R d) \times R}\left(A | M_{0}, \Psi_{0}, \Sigma\right)\\
\propto|&\Sigma|^{-R d / 2} \exp \left(-\frac{1}{2} \operatorname{tr}\left[\Sigma^{-1}\left(A-M_{0}\right)^{\top} \Psi_{0}^{-1}\left(A-M_{0}\right)\right]\right), \\
\end{aligned}
\end{equation}
and
\begin{equation}
\mathcal{I} \mathcal{W}\left(\Sigma | S_{0}, \nu_{0}\right) \propto|\Sigma|^{-\left(\nu_{0}+R+1\right) / 2} \exp \left(-\frac{1}{2} \operatorname{tr}\left(\Sigma^{-1}S_{0}\right)\right).
\end{equation}

**Likelihood from temporal factors $\boldsymbol{x}_{t}$**
\begin{equation}
\begin{aligned}
&\mathcal{L}\left(X\mid A,\Sigma\right) \\
\propto &\prod_{t=1}^{h_d}p\left(\boldsymbol{x}_{t}\mid \Sigma\right)\times \prod_{t=h_d+1}^{T}p\left(\boldsymbol{x}_{t}\mid A,\Sigma\right) \\
\propto &\left|\Sigma\right|^{-T/2}\exp\left\{-\frac{1}{2}\sum_{t=h_d+1}^{T}\left(\boldsymbol{x}_{t}-A^\top \boldsymbol{v}_{t}\right)^\top\Sigma^{-1}\left(\boldsymbol{x}_{t}-A^\top \boldsymbol{v}_{t}\right)\right\} \\
\propto &\left|\Sigma\right|^{-T/2}\exp\left\{-\frac{1}{2}\text{tr}\left[\Sigma^{-1}\left(Z_0^\top Z_0+\left(Z-QA\right)^\top \left(Z-QA\right)\right)\right]\right\}
\end{aligned}
\end{equation}

**Posterior distribution**

Consider
\begin{equation}
\begin{aligned}
&\left(A-M_{0}\right)^{\top} \Psi_{0}^{-1}\left(A-M_{0}\right)+S_0+Z_0^\top Z_0+\left(Z-QA\right)^\top \left(Z-QA\right) \\
=&A^\top\left(\Psi_0^{-1}+Q^\top Q\right)A-A^\top\left(\Psi_0^{-1}M_0+Q^\top Z\right) \\
&-\left(\Psi_0^{-1}M_0+Q^\top Z\right)^\top A \\
&+\left(\Psi_0^{-1}M_0+Q^\top Z\right)^\top\left(\Psi_0^{-1}+Q^\top Q\right)\left(\Psi_0^{-1}M_0+Q^\top Z\right) \\
&-\left(\Psi_0^{-1}M_0+Q^\top Z\right)^\top\left(\Psi_0^{-1}+Q^\top Q\right)\left(\Psi_0^{-1}M_0+Q^\top Z\right) \\
&+M_0^\top\Psi_0^{-1}M_0+S_0+Z_0^\top Z_0+Z^\top Z \\
=&\left(A-M^{*}\right)^\top\left(\Psi^{*}\right)^{-1}\left(A-M^{*}\right)+S^{*}, \\
\end{aligned}
\end{equation}
which is in the form of $\mathcal{MN}\left(\cdot\right)$ and $\mathcal{IW}\left(\cdot\right)$.

The $Rd$-by-$R$ matrix $A$ has a matrix normal distribution, and $R$-by-$R$ covariance matrix $\Sigma$ has an inverse Wishart distribution, that is,
\begin{equation}
A \sim \mathcal{M N}_{(R d) \times R}\left(M^{*}, \Psi^{*}, \Sigma\right), \quad \Sigma \sim \mathcal{I} \mathcal{W}\left(S^{*}, \nu^{*}\right),
\end{equation}
with
\begin{equation}
\begin{cases}
{\Psi^{*}=\left(\Psi_{0}^{-1}+Q^{\top} Q\right)^{-1}}, \\ {M^{*}=\Psi^{*}\left(\Psi_{0}^{-1} M_{0}+Q^{\top} Z\right)}, \\ {S^{*}=S_{0}+Z^\top Z+M_0^\top\Psi_0^{-1}M_0-\left(M^{*}\right)^\top\left(\Psi^{*}\right)^{-1}M^{*}}, \\
{\nu^{*}=\nu_{0}+T-h_d}.
\end{cases}
\end{equation}

```python
def mnrnd(M, U, V):
    """
    Generate matrix normal distributed random matrix.
    M is a m-by-n matrix, U is a m-by-m matrix, and V is a n-by-n matrix.
    """
    dim1, dim2 = M.shape
    X0 = np.random.randn(dim1, dim2)
    P = cholesky_lower(U)
    Q = cholesky_lower(V)

    return M + P @ X0 @ Q.T

def sample_var_coefficient(X, time_lags):
    dim, rank = X.shape
    d = time_lags.shape[0]
    tmax = np.max(time_lags)

    Z_mat = X[tmax : dim, :]
    Q_mat = np.zeros((dim - tmax, rank * d))
    for k in range(d):
        Q_mat[:, k * rank : (k + 1) * rank] = X[tmax - time_lags[k] : dim - time_lags[k], :]
    var_Psi0 = np.eye(rank * d) + Q_mat.T @ Q_mat
    var_Psi = inv(var_Psi0)
    var_M = var_Psi @ Q_mat.T @ Z_mat
    var_S = np.eye(rank) + Z_mat.T @ Z_mat - var_M.T @ var_Psi0 @ var_M
    Sigma = invwishart.rvs(df = rank + dim - tmax, scale = var_S)

    return mnrnd(var_M, var_Psi, Sigma), Sigma
```

#### 3) Sampling Factor Matrix $X$

**Posterior distribution**
\begin{equation}
\begin{aligned}
y_{it}&\sim\mathcal{N}\left(\boldsymbol{w}_{i}^\top\boldsymbol{x}_{t},\tau^{-1}\right),~\left(i,t\right)\in\Omega, \\
\boldsymbol{x}_{t}&\sim\begin{cases}\mathcal{N}\left(\sum_{k=1}^{d}A_{k} \boldsymbol{x}_{t-h_k},\Sigma\right),~\text{if $t\in\left\{h_d+1,...,T\right\}$},\\{\mathcal{N}\left(\boldsymbol{0},I\right),~\text{otherwise}}.\end{cases}\\
\end{aligned}
\end{equation}

If $t\in\left\{1,...,h_d\right\}$, parameters of the posterior distribution $\mathcal{N}\left(\boldsymbol{x}_{t}\mid \boldsymbol{\mu}_{t}^{*},\Sigma_{t}^{*}\right)$ are
\begin{equation}
\begin{aligned}
\Sigma_{t}^{*}&=\left(\sum_{k=1, h_{d}<t+h_{k} \leq T}^{d} {A}_{k}^{\top} \Sigma^{-1} A_{k}+\tau\sum_{i:(i,t)\in\Omega}\boldsymbol{w}_{i}\boldsymbol{w}_{i}^\top+I\right)^{-1}, \\
\boldsymbol{\mu}_{t}^{*}&=\Sigma_{t}^{*}\left(\sum_{k=1, h_{d}<t+h_{k} \leq T}^{d} A_{k}^{\top} \Sigma^{-1} \boldsymbol{\psi}_{t+h_{k}}+\tau\sum_{i:(i,t)\in\Omega}\boldsymbol{w}_{i}y_{it}\right). \\
\end{aligned}
\end{equation}

If $t\in\left\{h_d+1,...,T\right\}$, then parameters of the posterior distribution $\mathcal{N}\left(\boldsymbol{x}_{t}\mid \boldsymbol{\mu}_{t}^{*},\Sigma_{t}^{*}\right)$ are
\begin{equation}
\begin{aligned}
\Sigma_{t}^{*}&=\left(\sum_{k=1, h_{d}<t+h_{k} \leq T}^{d} {A}_{k}^{\top} \Sigma^{-1} A_{k}+\tau\sum_{i:(i,t)\in\Omega}\boldsymbol{w}_{i}\boldsymbol{w}_{i}^\top+\Sigma^{-1}\right)^{-1}, \\
\boldsymbol{\mu}_{t}^{*}&=\Sigma_{t}^{*}\left(\sum_{k=1, h_{d}<t+h_{k} \leq T}^{d} A_{k}^{\top} \Sigma^{-1} \boldsymbol{\psi}_{t+h_{k}}+\tau\sum_{i:(i,t)\in\Omega}\boldsymbol{w}_{i}y_{it}+\Sigma^{-1}\sum_{k=1}^{d}A_{k}\boldsymbol{x}_{t-h_k}\right), \\
\end{aligned}
\end{equation}
where
$$\boldsymbol{\psi}_{t+h_k}=\boldsymbol{x}_{t+h_k}-\sum_{l=1,l\neq k}^{d}A_{l}\boldsymbol{x}_{t+h_k-h_l}.$$

```python
def sample_factor_x(tau_sparse_mat, tau_ind, time_lags, W, X, A, Lambda_x):
    """Sampling T-by-R factor matrix X."""

    dim2, rank = X.shape
    tmax = np.max(time_lags)
    tmin = np.min(time_lags)
    d = time_lags.shape[0]
    A0 = np.dstack([A] * d)
    for k in range(d):
        A0[k * rank : (k + 1) * rank, :, k] = 0
    mat0 = Lambda_x @ A.T
    mat1 = np.einsum('kij, jt -> kit', A.reshape([d, rank, rank]), Lambda_x)
    mat2 = np.einsum('kit, kjt -> ij', mat1, A.reshape([d, rank, rank]))

    var1 = W.T
    var2 = kr_prod(var1, var1)
    var3 = (var2 @ tau_ind).reshape([rank, rank, dim2]) + Lambda_x[:, :, None]
    var4 = var1 @ tau_sparse_mat
    for t in range(dim2):
        Mt = np.zeros((rank, rank))
        Nt = np.zeros(rank)
        Qt = mat0 @ X[t - time_lags, :].reshape(rank * d)
        index = list(range(0, d))
        if t >= dim2 - tmax and t < dim2 - tmin:
            index = list(np.where(t + time_lags < dim2))[0]
        elif t < tmax:
            Qt = np.zeros(rank)
            index = list(np.where(t + time_lags >= tmax))[0]
        if t < dim2 - tmin:
            Mt = mat2.copy()
            temp = np.zeros((rank * d, len(index)))
            n = 0
            for k in index:
                temp[:, n] = X[t + time_lags[k] - time_lags, :].reshape(rank * d)
                n += 1
            temp0 = X[t + time_lags[index], :].T - np.einsum('ijk, ik -> jk', A0[:, :, index], temp)
            Nt = np.einsum('kij, jk -> i', mat1[index, :, :], temp0)

        var3[:, :, t] = var3[:, :, t] + Mt
        if t < tmax:
            var3[:, :, t] = var3[:, :, t] - Lambda_x + np.eye(rank)
        X[t, :] = mvnrnd_pre(solve(var3[:, :, t], var4[:, t] + Nt + Qt), var3[:, :, t])

    return X
```

#### 4) Sampling Precision $\tau$

```python
def sample_precision_tau(sparse_mat, mat_hat, ind):
    var_alpha = 1e-6 + 0.5 * np.sum(ind, axis = 1)
    var_beta = 1e-6 + 0.5 * np.sum(((sparse_mat - mat_hat) ** 2) * ind, axis = 1)
    return np.random.gamma(var_alpha, 1 / var_beta)
```

```python
def compute_mape(var, var_hat):
    return np.sum(np.abs(var - var_hat) / var) / var.shape[0]

def compute_rmse(var, var_hat):
    return  np.sqrt(np.sum((var - var_hat) ** 2) / var.shape[0])
```

#### 5) BTMF Implementation

```python
def BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter, multi_steps = 1):
    """Bayesian Temporal Matrix Factorization, BTMF."""

    dim1, dim2 = sparse_mat.shape
    d = time_lags.shape[0]
    W = init["W"]
    X = init["X"]
    pos_test = np.where((dense_mat != 0) & (sparse_mat == 0))
    dense_test = dense_mat[pos_test]
    del dense_mat
    ind = sparse_mat != 0
    pos_obs = np.where(ind)
    tau = np.ones(dim1)
    W_plus = np.zeros((dim1, rank))
    X_new_plus = np.zeros((dim2 + multi_steps, rank))
    A_plus = np.zeros((rank * d, rank))
    temp_hat = np.zeros(len(pos_test[0]))
    show_iter = 200
    mat_hat_plus = np.zeros((dim1, dim2))
    for it in range(burn_iter + gibbs_iter):
        tau_ind = tau[:, None] * ind
        tau_sparse_mat = tau[:, None] * sparse_mat
        W = sample_factor_w(tau_sparse_mat, tau_ind, W, X, tau, beta0 = 1, vargin = 0)
        A, Sigma = sample_var_coefficient(X, time_lags)
        X = sample_factor_x(tau_sparse_mat, tau_ind, time_lags, W, X, A, inv(Sigma))
        mat_hat = W @ X.T
        tau = sample_precision_tau(sparse_mat, mat_hat, ind)
        temp_hat += mat_hat[pos_test]
        if (it + 1) % show_iter == 0 and it < burn_iter:
            temp_hat = temp_hat / show_iter
            print('Iter: {}'.format(it + 1))
            print('MAPE: {:.6}'.format(compute_mape(dense_test, temp_hat)))
            print('RMSE: {:.6}'.format(compute_rmse(dense_test, temp_hat)))
            temp_hat = np.zeros(len(pos_test[0]))
            print()
        X_new = np.zeros((dim2 + multi_steps, rank))
        if it + 1 > burn_iter:
            W_plus += W
            A_plus += A
            X_new[: dim2, :] = X.copy()
            if multi_steps == 1:
                X_new[dim2, :] = A.T @ X_new[dim2 - time_lags, :].reshape(rank * d)
            elif multi_steps > 1:
                for t0 in range(multi_steps):
                    X_new[dim2 + t0, :] = A.T @ X_new[dim2 + t0 - time_lags, :].reshape(rank * d)
            X_new_plus += X_new
            mat_hat_plus += mat_hat
    mat_hat = mat_hat_plus / gibbs_iter
    W = W_plus / gibbs_iter
    X_new = X_new_plus / gibbs_iter
    A = A_plus / gibbs_iter
    print('Imputation MAPE: {:.6}'.format(compute_mape(dense_test, mat_hat[pos_test])))
    print('Imputation RMSE: {:.6}'.format(compute_rmse(dense_test, mat_hat[pos_test])))
    print()

    return mat_hat, W, X_new, A
```

# PeMS-4W

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-4w.csv', header = None)
dense_mat = data.values
random_mat = ten2mat(np.random.rand(data.values.shape[0], 288, 4 * 7), 0)
del data

missing_rate = 0.3

### Random missing (RM) scenario:
sparse_mat = np.multiply(dense_mat, np.round(random_mat + 0.5 - missing_rate))
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-4w.csv', header = None)
dense_mat = data.values
random_mat = ten2mat(np.random.rand(data.values.shape[0], 288, 4 * 7), 0)
del data

missing_rate = 0.7

### Random missing (RM) scenario:
sparse_mat = np.multiply(dense_mat, np.round(random_mat + 0.5 - missing_rate))
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-4w.csv', header = None)
dense_mat = data.values
dense_tensor = mat2ten(dense_mat, np.array([data.values.shape[0], 288, 4 * 7]), 0)
random_matrix = np.random.rand(data.values.shape[0], 4 * 7)

missing_rate = 0.3

### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[2]):
        binary_tensor[i1, :, i2] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
sparse_mat = ten2mat(np.multiply(dense_tensor, binary_tensor), 0)
del data, dense_tensor, binary_tensor
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-4w.csv', header = None)
dense_mat = data.values
dense_tensor = mat2ten(dense_mat, np.array([data.values.shape[0], 288, 4 * 7]), 0)
random_matrix = np.random.rand(data.values.shape[0], 4 * 7)

missing_rate = 0.7

### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[2]):
        binary_tensor[i1, :, i2] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
sparse_mat = ten2mat(np.multiply(dense_tensor, binary_tensor), 0)
del data, dense_tensor, binary_tensor
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

# PeMS-8W

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-8w.csv', header = None)
dense_mat = data.values
random_mat = ten2mat(np.random.rand(data.values.shape[0], 288, 8 * 7), 0)
del data

missing_rate = 0.3

### Random missing (RM) scenario:
sparse_mat = np.multiply(dense_mat, np.round(random_mat + 0.5 - missing_rate))
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-8w.csv', header = None)
dense_mat = data.values
random_mat = ten2mat(np.random.rand(data.values.shape[0], 288, 8 * 7), 0)
del data

missing_rate = 0.7

### Random missing (RM) scenario:
sparse_mat = np.multiply(dense_mat, np.round(random_mat + 0.5 - missing_rate))
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-8w.csv', header = None)
dense_mat = data.values
dense_tensor = mat2ten(dense_mat, np.array([data.values.shape[0], 288, 8 * 7]), 0)
random_matrix = np.random.rand(data.values.shape[0], 8 * 7)

missing_rate = 0.3

### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[2]):
        binary_tensor[i1, :, i2] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
sparse_mat = ten2mat(np.multiply(dense_tensor, binary_tensor), 0)
del data, dense_tensor, binary_tensor
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

```python
import numpy as np
import pandas as pd
np.random.seed(1000)

data = pd.read_csv('../datasets/California-data-set/pems-8w.csv', header = None)
dense_mat = data.values
dense_tensor = mat2ten(dense_mat, np.array([data.values.shape[0], 288, 8 * 7]), 0)
random_matrix = np.random.rand(data.values.shape[0], 8 * 7)

missing_rate = 0.7

### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[2]):
        binary_tensor[i1, :, i2] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
sparse_mat = ten2mat(np.multiply(dense_tensor, binary_tensor), 0)
del data, dense_tensor, binary_tensor
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 288])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %.2f minutes'%((end - start)/60.0))
```

# London 1-M data

```python
import numpy as np
np.random.seed(1000)

missing_rate = 0.3

dense_mat = np.load('../datasets/London-data-set/hourly_speed_mat.npy')
binary_mat = dense_mat.copy()
binary_mat[binary_mat != 0] = 1
pos = np.where(np.sum(binary_mat, axis = 1) > 0.7 * binary_mat.shape[1])
dense_mat = dense_mat[pos[0], :]

## Random missing (RM)
random_mat = np.random.rand(dense_mat.shape[0], dense_mat.shape[1])
binary_mat = np.round(random_mat + 0.5 - missing_rate)
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 20
time_lags = np.array([1, 2, 24])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

```python
import numpy as np
np.random.seed(1000)

missing_rate = 0.7

dense_mat = np.load('../datasets/London-data-set/hourly_speed_mat.npy')
binary_mat = dense_mat.copy()
binary_mat[binary_mat != 0] = 1
pos = np.where(np.sum(binary_mat, axis = 1) > 0.7 * binary_mat.shape[1])
dense_mat = dense_mat[pos[0], :]

## Random missing (RM)
random_mat = np.random.rand(dense_mat.shape[0], dense_mat.shape[1])
binary_mat = np.round(random_mat + 0.5 - missing_rate)
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 20
time_lags = np.array([1, 2, 24])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

```python
import numpy as np
np.random.seed(1000)

missing_rate = 0.3

dense_mat = np.load('../datasets/London-data-set/hourly_speed_mat.npy')
binary_mat = dense_mat.copy()
binary_mat[binary_mat != 0] = 1
pos = np.where(np.sum(binary_mat, axis = 1) > 0.7 * binary_mat.shape[1])
dense_mat = dense_mat[pos[0], :]

## Non-random missing (NM)
binary_mat = np.zeros(dense_mat.shape)
random_mat = np.random.rand(dense_mat.shape[0], 30)
for i1 in range(dense_mat.shape[0]):
    for i2 in range(30):
        binary_mat[i1, i2 * 24 : (i2 + 1) * 24] = np.round(random_mat[i1, i2] + 0.5 - missing_rate)
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 20
time_lags = np.array([1, 2, 24])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

```python
import numpy as np
np.random.seed(1000)

missing_rate = 0.7

dense_mat = np.load('../datasets/London-data-set/hourly_speed_mat.npy')
binary_mat = dense_mat.copy()
binary_mat[binary_mat != 0] = 1
pos = np.where(np.sum(binary_mat, axis = 1) > 0.7 * binary_mat.shape[1])
dense_mat = dense_mat[pos[0], :]

## Non-random missing (NM)
binary_mat = np.zeros(dense_mat.shape)
random_mat = np.random.rand(dense_mat.shape[0], 30)
for i1 in range(dense_mat.shape[0]):
    for i2 in range(30):
        binary_mat[i1, i2 * 24 : (i2 + 1) * 24] = np.round(random_mat[i1, i2] + 0.5 - missing_rate)
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 20
time_lags = np.array([1, 2, 24])
init = {"W": 0.1 * np.random.randn(dim1, rank), "X": 0.1 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

# Guangzhou 2-M data

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')['random_tensor']
dense_mat = tensor.reshape([tensor.shape[0], tensor.shape[1] * tensor.shape[2]])
missing_rate = 0.3

## Random missing (RM)
binary_mat = (np.round(random_tensor + 0.5 - missing_rate)
              .reshape([random_tensor.shape[0], random_tensor.shape[1] * random_tensor.shape[2]]))
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 80
time_lags = np.array([1, 2, 144])
init = {"W": 0.01 * np.random.randn(dim1, rank), "X": 0.01 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')['random_tensor']
dense_mat = tensor.reshape([tensor.shape[0], tensor.shape[1] * tensor.shape[2]])
missing_rate = 0.7

## Random missing (RM)
binary_mat = (np.round(random_tensor + 0.5 - missing_rate)
              .reshape([random_tensor.shape[0], random_tensor.shape[1] * random_tensor.shape[2]]))
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 80
time_lags = np.array([1, 2, 144])
init = {"W": 0.01 * np.random.randn(dim1, rank), "X": 0.01 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')['random_matrix']
dense_mat = tensor.reshape([tensor.shape[0], tensor.shape[1] * tensor.shape[2]])
missing_rate = 0.3

## Non-random missing (NM)
binary_tensor = np.zeros(tensor.shape)
for i1 in range(tensor.shape[0]):
    for i2 in range(tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
binary_mat = binary_tensor.reshape([binary_tensor.shape[0], binary_tensor.shape[1] * binary_tensor.shape[2]])
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 144])
init = {"W": 0.01 * np.random.randn(dim1, rank), "X": 0.01 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')['random_matrix']
dense_mat = tensor.reshape([tensor.shape[0], tensor.shape[1] * tensor.shape[2]])
missing_rate = 0.7

## Non-random missing (NM)
binary_tensor = np.zeros(tensor.shape)
for i1 in range(tensor.shape[0]):
    for i2 in range(tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
binary_mat = binary_tensor.reshape([binary_tensor.shape[0], binary_tensor.shape[1] * binary_tensor.shape[2]])
sparse_mat = np.multiply(dense_mat, binary_mat)
```

```python
import time
start = time.time()
dim1, dim2 = sparse_mat.shape
rank = 10
time_lags = np.array([1, 2, 144])
init = {"W": 0.01 * np.random.randn(dim1, rank), "X": 0.01 * np.random.randn(dim2, rank)}
burn_iter = 1000
gibbs_iter = 200
mat_hat, W, X_new, A = BTMF(dense_mat, sparse_mat, init, rank, time_lags, burn_iter, gibbs_iter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

### License

This work is released under the MIT license.


# "Spin Glass Models 3: Ising Model - Theory"
> "In this blog post we will introduce another model of spin glasses: the Ising model. We relax some of the simplifications of previous models to create something that more accurately captures the structure of real spin-glasses. We will look at this model mathematically to gain insight into its properties."
- toc: true
- author: Lewis Cole (2020)
- branch: master
- badges: false
- comments: false
- categories: [Spin-Glass, Magnet, Ising, Edwards-Anderson]
- hide: false
- search_exclude: false
- image: https://github.com/lewiscoleblog/blog/raw/master/images/spin-glass/Ising3d.png

___
This is the third blog post in a series - you can find the previous blog post [here](https://lewiscoleblog.com/spin-glass-models-2)
___

In this blog post we are going to look at another spin-glass model. In the previous post we looked at the Sherrington-Kirkpatrick model which allowed us to study the spin-glass analytically. However there is a certain lack of "realism" in this model - the infinite interaction range means that the Sherrington-Kirkpatrick (in a sense) does not occupy any space, there is no concept of dimension nor geometry. While providing interesting mathematical results, some of which appear to be universal to spin-glass systems, we would like to look at a different model that captures more of the real world system and perhaps might uncover new behaviours.

## Introducing the Ising Model
One model that we can look at is the Edwards-Anderson Ising model. We have actually looked at these models achronologically: Edwards-Anderson proposed the Ising model before Sherrington-Kirkpatrick proposed theirs. In fact Sherrington-Kirkpatrick developed their model partly due to the difficulty in deal with the Ising model. I have presented the models in this order in this series of blog posts because it feels more natural to look at models in increasing complexity order rather than presenting a historical account.

The main difference between Ising and Sherrington-Kirkpatrick is the extent over which the interactions can occur, instead of an infinite range the Ising model only allows "nearest-neighbour" interactions. This relaxation means that there is a concept of dimension and geometry to an Ising spin-glass. For example we could think of spins oriented on a line segment (a finite 1d Ising model), a square lattice (a 2d Ising model) or something more exotic like spins oriented on the nodes of a 32 dimensional hexagonal lattice. Through careful use of limits we could also consider infinite-dimensional Ising models too. The Hamiltonian follows the form one would expect:
$$ H = - \sum_{\lt x,y \gt} J_{xy} \sigma_x \sigma_y  - h \sum_x \sigma_x $$
Where we use the notation $\lt x,y \gt$ to denote the sum occuring over neighbour pairs.

As before the selection of $J_{xy}$ is somewhat arbitrary, however unlike the Sherrington-Kirkpatrick we do not have to scale this with lattice size to retain meaningful "average energy" or "average magnetism" measurements when we increase the size of the system. If we take $J_{xy} = J$ for some fixed $J$ we get the Ising model of Ferromagnetism, if we allow $J_{xy}$ to vary according to some distribution we get the Edwards-Anderson Ising Model. In this blog we will use the term "Ising model" to refer to either situation, the mathematics of moving from the fixed $J$ case usually involves integrating against the density function, this can lead to more complicated formulae so in this blog we will mainly focus on the ferromagnetic case unless otherwise stated.

As before spins can only take values $\pm 1$ - in some literature this is referred to as "Ising spins" (e.g. you can read references to "Sherrington-Kirkpatrick with Ising spins" - this means an infinite interaction range with binary spins).

A pictorial representation of a 3d square lattice Ising spin-glass can be seen below:

For the rest of this article we will limit ourselves to talking about "square" lattices since this is where the majority of research is focussed. Using different lattice types won't change the story too much.

In studying these models it is also worth noting what happens on the "boundary" of the lattice: for finite size systems this can become an issue. In this blog post we will skirt over the issue, where necessary we will assume periodic boundary conditions (i.e. there is a toroidal type geometry to the system) so each "edge" of the lattice is connected to an opposing one, so in a sense there are no boundary conditions to specify.

## 1D Ising Model
We will now consider a 1 dimensional Ising model where spins are located on a line segment. This simplication means that finding the ground state (minimal energy spin configuration) is trivial: we pick a site in the middle of the line segment (the origin) we will pick a spin ($\pm 1$) at random. We then propagate out from the origin in the left and right direction, we pick the spin for the next site according to the interaction strength for example: if $\sigma_0 = +1$ and $J_{0,1} > 0$ then $\sigma_1 = +1$ and so on. We can see that (as with all spin glass systems) there is a symmetry: if we flip all the spins direction we end up with an equivalent energy - we call these "pairs" of configurations. It doesn't take much convincing to realise that the pair of configurations we constructed is a unique minima for the system. To see this imagine we have only one pair of spins that opposes the direction indicated by the interaction strength. For a large enough system we can eventually find another interaction strength that is greater in magnitude (by definition having a spin pair as indicated by the sign of the interaction strength). By simply flipping the relative orientation of the pairs we will end up with a configuration of lower energy - which is a contradiction.

We will now look to solve the 1d Ising model analytically. For simplicity we will assume no external magnetic field, this just makes the formulae look "prettier". The crux of understanding any spin glass system is finding the Gibbs distribution, as before:
$$P(\sigma) = \frac{exp(-\beta H_{\sigma})}{Z_{\beta}}$$
Where:
$$H = - \sum_{\lt x,y \gt} J_{xy} \sigma_x \sigma_y$$
And $\beta$ is the inverse temperature. We can write an expression for the partition function as:
$$ Z_{\beta} = \sum_{\sigma} exp(-\beta H_{\sigma})  = \sum_{\sigma} exp(-\beta \sum_{\lt x,y \gt} J_{xy} \sigma_x \sigma_y)  $$
Suppose we look at a line segment with $N$ spins, we can then write this as:
$$ Z_{\beta} =\sum_{\sigma_1, ..., \sigma_N} exp(-\beta (J_{1,2}\sigma_1 \sigma_2 + J_{2,3}\sigma_2 \sigma_3 + ... + J_{N-1,N}\sigma_{N-1}\sigma_N))$$
We notice that we can factorise this summation as:
$$ Z_{\beta} =\sum_{\sigma_1, ..., \sigma_{N-1}} exp(-\beta (J_{1,2}\sigma_1 \sigma_2 + ... + J_{N-2,N-1}\sigma_{N-2}\sigma_{N-1}))\sum_{\sigma_N}exp(-\beta J_{N-1,N} \sigma_{N-1} \sigma_N)  $$
The internal sum can then be evaluated:
$$ \sum_{\sigma_N}exp(-\beta J_{N-1,N} \sigma_{N-1} \sigma_N) = exp(\beta J_{N-1,N} \sigma_{N-1} ) + exp(-\beta J_{N-1,N} \sigma_{N-1} ) = 2cosh(\beta J_{N-1,N} \sigma_{N-1}) = 2cosh(\beta J_{N-1,N}) $$
The last equality making use of the fact $\sigma_{N-1} = \pm 1$ and that $cosh(x) = cosh(-x)$. By repeating this process we can express the partition function as:
$$ Z_{\beta} = 2 \prod_{i=1}^{N-1} 2 cosh(\beta J_{i,i+1})  $$
We can evaluate this exactly. If we are sampling the interaction strengths according to some distribution we can find expected values (or other statistical properties) by integrating against the density function as usual, since this adds to notational complexity we will assume that the interaction strengths are fixed for now. We can then calculate thermal properties using the equations:

\begin{align}
F &= - \frac{1}{\beta} ln Z_{\beta} = - T ln 2 - T \sum_{i=1}^{N-1} ln (2 cosh(\beta J_{i,i+1})) \\
U &= - \frac{\partial}{\partial \beta} ln Z_{\beta} = - \sum_{i=1}^{N-1} J_{i,i+1} tanh(\beta J_{i,i+1}) \\
C &= \frac{\partial U}{\partial T} = \sum_{i=1}^{N-1} (\beta J_{i,i+1})^2 sech^2(\beta J_{i,i+1}) \\
S &= \frac{U - F}{T} = ln2 + \sum_{i=1}^{N-1} \left( -\beta J_{i,i+1} tanh(\beta J_{i,i+1}) + ln(2cosh(\beta J_{i,i+1}) \right)
\end{align}

Where:

F - is the Helmholtz-Free Energy

U - is the thermodynamic energy (ensemble average)

C - is the heat capacity

S - is the entropy

We can find the expected value of a given instantiation of interactions through an integral such as:
$$ \mathbb{E}(U) = - \sum_{i=1}^{N-1} \int J_{i,i+1} tanh(\beta J_{i,i+1}) Q(J_{i,i+1}) dJ_{i,i+1} $$
With $Q(J)$ being the density function of the distribution and the integral occuring over its support. Similar formulae exist for the other thermodynamic variables and you can calculate other statistics (e.g. variance) in the usual way.

We can plot the values of these variables for a given set of interaction weights:

```python
#hide
import warnings
warnings.filterwarnings('ignore')
```

```python
# This code creates a 4 figure plot showing how
# Thermodynamic variables change with temperature
# For a 1d Ising Model with Gaussian interactions

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Fix seed
np.random.seed(123)

# Fix number of points
N = 100

# Instantiate interaction strength
J = np.random.normal(0,1,size=N-1)

# Set temperature ranges
T_min = 1e-4
T_max = 5
T_steps = 1000
T = np.arange(T_steps)/T_steps *(T_max - T_min) + T_min
beta = 1 / T

# Set up holders for variables
F = np.zeros(T_steps)
U = np.zeros(T_steps)
C = np.zeros(T_steps)
S = np.zeros(T_steps)

# Loop over T_steps and calculate at each step
for i in range(T_steps):
    F[i] = - T[i] * np.log(2) - T[i] * (np.log(2*np.cosh(beta[i]*J))).sum()
    U[i] =  - (J * np.tanh(J * beta[i])).sum()
    C[i] = ((beta[i] * J)**2 * (np.cosh(beta[i] * J))**-2).sum()
    S[i] = (U[i] - F[i]) / T[i]

# Divide by number of points to give a scale invariant measure
F = F / N
U = U / N
C = C / N
S = S / N

# Create plots
fig, axs = plt.subplots(2, 2, figsize=(10,10), gridspec_kw={'hspace': 0.25, 'wspace': 0.25})
axs[0, 0].plot(T, F)
axs[0, 0].set_title("Free Energy")
axs[0, 0].set(xlabel="T", ylabel="F / N")

axs[0, 1].plot(T, U)
axs[0, 1].set_title("Average Energy")
axs[0, 1].set(xlabel="T", ylabel="U / N")

axs[1, 0].plot(T, C)
axs[1, 0].set_title("Heat Capacity")
axs[1, 0].set(xlabel="T", ylabel="C / N")

axs[1, 1].plot(T, S)
axs[1, 1].set_title("Entropy")
axs[1, 1].set(xlabel="T", ylabel="S / N")

plt.show()
```

From these plots we can see there is no phase transition taking place - this is true for all 1d Ising models.

We can also calculate the correlation function for the system. By following a similar line of logic as before we find:
$$ \langle \sigma_n \sigma_{n+r} \rangle = \prod_{i=0}^{r} tanh(\beta J_{n+i, n+i+1})$$

We can plot this as a function of $r$ starting at the first position:

```python
# This code creates a plot displaying
# The correlation function
# For a 1d Ising Model with Gaussian interactions

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Fix seed
np.random.seed(123)

# Fix number of points
N = 100

# Instantiate interaction strength
J = np.random.normal(0,1,size=N-1)

# Create holders for variables
r = np.arange(N)
corr_array = np.zeros(N)

# Fix beta
beta = 1

# Calculate correlations
tanh_array = np.tanh(beta * J)

for i in range(N):
    corr_array[i] = tanh_array[:i].prod()

plt.plot(r[0:20], corr_array[0:20])
plt.title("Correlation Function")
plt.ylabel(r"$\langle \sigma_{1} \sigma_{r} \rangle$")
plt.xlabel("r")
plt.xticks(np.arange(11)*2)
plt.ylim((-1,1))
plt.xlim((0,20))
plt.show()
```

As we can see there is a specific "structure" to the correlation function given an instantiation - this is not particularly surprising. If we picked an alternate starting state (e.g. the 2nd site) this graph can look totally different, the decay in absolute value will be similar however. We also notice that the correlation decays to zero fairly rapidly suggesting there isn't a long range structure to the model.

## 2d Ising Model
We now turn our attention to the 2d Ising model. The mathematics here will, understandably, get more complicated. We will require a bit more sophistication to solve this system. The solution was originally published by Lars Onsager in 1944, with alternative proofs and derivations published later. The derivation itself is fairly involved and would take a blog post (at least) by itself to cover - I may come back to this at a later date. For now I will simply present the result for the free energy (F) in absence of a magnetic field ($h=0$):
$$ - \frac{\beta}{N} F = ln2 + \frac{1}{8\pi^2} \int^{2\pi}_{0} \int^{2\pi}_{0} ln \left[ cosh(2 \beta J_H) cosh(2 \beta J_V) - sinh(2 \beta J_H)cos(\theta_1) - sinh(2 \beta J_V)cos(\theta_2) \right]d\theta_1 d\theta_2 $$
Where instead of $J_{xy}$ being sampled from a gaussian distribution we assume fixed interaction strengths $J_H$ and $J_V$ in the horizontal and vertical directions.

For simplicity we will take: $J_H = J_V = J$ and we can derive the thermodynamic properties (per site - e.g. $f=\frac{F}{N}$) of this system as:
\begin{align}
f &= \frac{-ln2}{2\beta} - \frac{1}{2\pi\beta} I_0(\beta, J)  \\
u &= -J coth(2\beta J) \left[ 1 + \frac{2}{\pi} \left( 2tanh^2(2\beta J) -1 \right) I_1(\beta, J) \right]      \\
c &= 2J\beta^2 \left[U csch(2\beta J)sech(2\beta J) + \frac{8J}{\pi} sech^2(2\beta J) I_1(\beta, J) - \frac{2 \beta J}{\pi}(cosh(4\beta J)-3)^2 sech^6(2\beta J) I_2(\beta, J) \right]     \\
s &= \frac{U - F}{T}
\end{align}

For convenience I created 3 new functions $I_0(\beta,J), I_1(\beta, J)$ and $I_2(\beta, J)$ to make the equations a little shorter. These functions are defined as:
\begin{align}
I_0(\beta, J) &= \int^\pi_0 ln \left[ cosh^2(2 \beta J) + sinh^2(2 \beta J) \sqrt{1 + csch^4(2 \beta J) - 2 csch^2(2 \beta J) cos(2\theta)} \right] d\theta \\
I_1(\beta, J) &= \int^{\frac{\pi}{2}}_0 \left[1 - 4csch^2(2\beta J)( 1 + csch^2(2\beta J))^{-2} sin^2(\theta) \right]^{-\frac{1}{2}} d\theta \\
I_2(\beta, J) &= \int^{\frac{\pi}{2}}_0 sin^2(\theta)\left[1 - 4csch^2(2\beta J)( 1 + csch^2(2\beta J))^{-2} sin^2(\theta) \right]^{-\frac{3}{2}} d\theta
\end{align}

As before we can produce plots of these:

```python
# This code creates a 4 figure plot showing how
# Thermodynamic variables change with temperature
# For a 2d Ferromagnetic Ising Model with fixed interaction

import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import quad
%matplotlib inline

# Fix seed
np.random.seed(123)

# Instantiate interaction strength
J = 1

# Set temperature ranges
T_min = 1e-4
T_max = 5
T_steps = 1000
T = np.arange(T_steps)/T_steps *(T_max - T_min) + T_min
beta = 1 / T

# Set up holders for variables
f = np.zeros(T_steps)
u = np.zeros(T_steps)
c = np.zeros(T_steps)
s = np.zeros(T_steps)

# Set up integrands for I0, I1, I2
def integrand0(x, b, j):
    return np.log(np.cosh(2*b*j)**2 + np.sinh(2*b*j)**2 * np.sqrt(1 + np.sinh(2*b*j)**(-4) - 2*np.sinh(2*b*j)**(-2) * np.cos(2*x)))

def integrand1(x, b, j):
    return (1 - 4*np.sinh(2*b*j)**(-2)*(1 + np.sinh(2*b*J)**(-2))**(-2)*np.sin(x)**2)**(-0.5)

def integrand2(x, b, j):
    return np.sin(x)**2 * integrand1(x, b, j)**3

# Loop over T_steps and calculate at each step
for i in range(T_steps):
    bt = beta[i]
    I0 = quad(integrand0, 0, np.pi, args=(bt, J))
    I1 = quad(integrand1, 0, np.pi/2, args=(bt, J))
    I2 = quad(integrand2, 0, np.pi/2, args=(bt, J))
    f[i] = - np.log(2) / (2 * bt) - I0[0] / (2 * np.pi * bt)
    u[i] = -J*np.tanh(2*bt*J)**(-1) * ( 1 + (2/np.pi)*(2*np.tanh(2*bt*J)**2 -1)*I1[0])
    c[i] = 2*bt**2*J*( u[i]*np.sinh(2*bt*J)**(-1)*np.cosh(2*bt*J)**(-1) + (8*J / np.pi)*np.cosh(2*bt*J)**(-2)*I1[0] - (2*bt*J/np.pi)*((np.cosh(4*bt*J)-3)**2)*np.cosh(2*bt*J)**(-6)*I2[0] )
    s[i] = (u[i] - f[i])*bt

# Create plots
fig, axs = plt.subplots(2, 2, figsize=(10,10), gridspec_kw={'hspace': 0.25, 'wspace': 0.25})
axs[0, 0].plot(T, f)
axs[0, 0].set_title("Free Energy")
axs[0, 0].set(xlabel="T", ylabel="f")

axs[0, 1].plot(T, u)
axs[0, 1].set_title("Average Energy")
axs[0, 1].set(xlabel="T", ylabel="u")

axs[1, 0].plot(T, c)
axs[1, 0].set_title("Heat Capacity")
axs[1, 0].set(xlabel="T", ylabel="c")

axs[1, 1].plot(T, s)
axs[1, 1].set_title("Entropy")
axs[1, 1].set(xlabel="T", ylabel="s")

plt.show()
```

We find there is a critical temperature $T_c$ where the specific heat equation diverges. We can compute the value of this as satisfying:
$$ sinh\left(\frac{2J_H}{T_c}\right)sinh\left(\frac{2J_V}{T_c}\right) = 1 $$
In the case where $J_H = J_V = J$ we have:
$$T_c = \frac{2 J}{ln\left(1+\sqrt{2}\right)} $$
This is an example of a second order phase transition since the discontinuity only arises under a second derivative.

For temperatres under this critical temperature we have that the spontaneous magnetization can be calculated as:
$$ m = \left[ 1 -  csch^2\left(\frac{2J_H}{T}\right)csch^2\left(\frac{2J_V}{T}\right)   \right]^{\frac{1}{8}} $$

We can plot this (for $J_H = J_V = J = 1$) as:

```python
# This code plots the spotaneous magnetization of
# a 2d Ising model on a square-lattice

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Set up temperature array
T_min = 1e-4
T_max = 5
T_steps = 1000
T = np.arange(T_steps)/T_steps *(T_max - T_min) + T_min

# Fix interaction strength constant
J = 1

m = np.power(np.maximum(1 - np.sinh(2*J/T)**-4,0), 1/8)

plt.plot(T,m)
plt.xlabel("T")
plt.ylabel("Magnetization")
plt.title("2d Ising Model Spontaneous Magnetization")
plt.xlim((0,5))
plt.show()

print("Critical Temp (Tc):", (2*J) / np.log(1 + np.sqrt(2)))
```

The correlation function is harder to compute, here we just present the correlation function between elements on the diagonal of the lattice:

$$ \langle \sigma_{0,0} \sigma_{N,N} \rangle = det \left[ A_N \right]$$
Where $A_N = (a_{n,m})_{n,m=1}^N$ is an $NxN$ matrix with entries:
$$ a_{n,m} = \frac{1}{2 \pi} \int_0^{2\pi} e^{i(n-m)\theta} \sqrt{\frac{sinh^2(2\beta J) - e^{-i\theta}}{sinh^2(2\beta J) - e^{i\theta}} } d\theta $$

We can plot this as so:

```python
# This code plots the diagnoal correlation function
# For the 2d square-lattice Ising model
# Lattice size = NxN
# At the critical temperature

import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import quad
%matplotlib inline

# Set up integrand function
imag = complex(0,1)
def integrand(x, n, m, beta, J):
    return np.exp(imag*(n-m)*x)*np.sqrt((np.sinh(2*beta*J)**2-np.exp(-imag*x))/(np.sinh(2*beta*J)**2-np.exp(imag*x)))

# Set up constants
J = 1
beta = np.log(1 + np.sqrt(2)) / (2*J)

# Set up arrays
N_max = 20
N_array = np.arange(N_max+1)
correl_array = np.zeros(N_max+1)

for x in range(N_max + 1):
    N = N_array[x]
    A_N = np.zeros((N, N))

    for n in range(N):
        for m in range(N):
            I = quad(integrand, 0, 2*np.pi, args=(n, m, beta, J))
            A_N[n,m] = I[0]/(2*np.pi)

    correl_array[x] = np.linalg.det(A_N)

# Plot correlations
plt.plot(N_array, correl_array)
plt.xlabel("N")
plt.ylabel(r"$\langle \sigma_{0,0} \sigma_{N,N} \rangle$")
plt.title("Correlation Function")
plt.xticks(np.arange(11)*2)
plt.ylim((0,1))
plt.xlim((0,20))
plt.show()
```

We can see that even though we have short range interactions (nearest neighbour) this gives rise to longer-range structure within the model. In fact we find for temperatures below the critical temperature there is an infinite correlation length, whereas for temperatures above the critical temperature the correlation length is finite.

## Infinte Dimension Ising Model
Next we consider the situation where the Ising model exists in infinite dimensions. In this case each site has infinitely many neighbours, as such a mean-field approximation is valid and we have a model that is somewhat similar to the Sherrington-Kirkpatrick fully connected geometry. (Please excuse the lack of rigour here; one has to be careful in how limits are defined and it is a non-trivial exercise. For this blog post I don't want to go down into that sort of detail.)

If each site has infinitely many neighbours we only need to concern ourselves with the ratio of positive and negative spin neighbours. By mean field if we take the probability of a positive spin as $p$ then via the Gibbs distribution we have:
$$ \frac{p}{1-p} = exp(2\beta H)$$
The average magnetization can then be calculated:
$$ \mathbb{E}\left[M\right] = (1)p + (-1)(1-p) = 2p - 1 = \frac{1 - exp(2\beta H)}{1 + exp(2\beta H)} = tanh(2 \beta H)$$

By investigating this function we can gain insight into spontaenous magnetization. Other similar arguments can be invoked for the other themodynamic properties. It is possible to show, as with the Sherrington-Kirkpatrick, that a phase transition occurs.

## n-d Ising Model ($n\geq3$)
Finally we look at the case where the dimension of the model is finite but strictly greater than 2. In this situation things get much trickier, in fact there are not many defined mathematical results in these systems and this is the subject of current research. As such I will just briefly outline some of the approaches that have been suggested to study these systems and their properties (presented in chronological order from when they were proposed):
* **Replica-Symmetry Breaking** - From our previous post on Sherrington-Kirkpatrick models we briefly looked at this sort of method. They were an obvious first choice in trying to deal with short range interactions. It has been shown that the "standard" techniques are insufficient but there have been extensions proposed that have shown some promise. Like in the infinite range model it suggests states have an ultrametric structure and uncountably many pure states.
* **Droplet Scaling** - The first such argument being presented by Rudolf Peierls. The main idea is to consider the arisal of "loops" or "islands" of spins (clusters of atoms all with the same spin being enclosed by some boundary). We then aim to count the number of such loops, often in high or low temperature ranges via approximation. This leads to only 2 pure states.
* **Chaotic Pairs** - Has properties somewhat similar to a combination of the preceeding 2 methods, like replica symmetry breaking there are infinitely many thermodynamic states however the relationship between them is much simpler and has simple scaling behaviour - much like droplet scaling.
* **TNT** - This interpretation does not itself specify the number of pure states, however it has been argued that it most naturally exists with 2 pure states - much like droplet scaling. However it has scaling properties more similar to that of replica symmetry breaking.

For completeness: a pure state of a system is a a state that cannot be expressed as a convex combination of other states.

As far as I am aware there is currently no conesensus on which (if any) of the options presented is the correct interpretation.

One of the main questions that we would like to answer is whether there is a phase transition (first order). This is unresolved currently. Another question we might ask is whether there exists multiple ground state pairs (i.e. we can find 2 different configurations that have minimal energy that are not merely negative images of each other) - again this is unresolved. In infinite dimensions we can see this would be true, in 1d we know this cannot be true - for other dimensions it is unclear (although it is believed it is probably not true for d=2).

In addition to this there are many other unanswered questions that are the subject of current research into Ising type models.

## Conclusion
In this blog post we have investigated some of the properties of square-lattice Ising models in various dimensions. In particular we have seen that there is no phase transition in 1d, a second order phase tranisition in 2d, a phase transition in infinite dimension and currently other dimensions are unresolved. We can see that the short-range interactions cause a lot of headaches when trying to analyse these systems. In the next blog post in this series we will begin to look at ways of simulating Ising models (and other spin glass models generally).

___
This is the third blog post in a series - you can find the next blog post [here](https://lewiscoleblog.com/spin-glass-models-4)