# Data Analysis and Machine Learning  Lectures: Optimization and  Gradient Methods

**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: **Sep 20, 2018**

Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license

## Optimization, the central part of any Machine Learning algortithm

Almost every problem in machine learning and data science starts with
a dataset $X$, a model $g(\beta)$, which is a function of the
parameters $\beta$ and a cost function $C(X, g(\beta))$ that allows
us to judge how well the model $g(\beta)$ explains the observations
$X$. The model is fit by finding the values of $\beta$ that minimize
the cost function. Ideally we would be able to solve for $\beta$
analytically, however this is not possible in general and we must use
some approximative/numerical method to compute the minimum.

## Steepest descent

The method of steepest descent The basic idea of gradient descent is
that a function $F(\mathbf{x})$,
$\mathbf{x} \equiv (x_1,\cdots,x_n)$, decreases fastest if one goes from $\bf {x}$ in the
direction of the negative gradient $-\nabla F(\mathbf{x})$.

It can be shown that if

$$
\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0
$$

for $\gamma_k$ small enough, then $F(\mathbf{x}_{k+1}) \leq
F(\mathbf{x}_k)$. This means that for a sufficiently small $\gamma_k$
we are always moving towards smaller function values, i.e a minimum.

## More on Steepest descent

The previous observation is the basis of the method of steepest
descent, which is also referred to as just gradient descent (GD). One
starts with an initial guess $\mathbf{x}_0$ for a minimum of $F$ and
computes new approximations according to

$$
\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0.
$$

The parameter $\gamma_k$ is often referred to as the step length or
the learning rate within the context of Machine Learning.

## The ideal

Ideally the sequence $\{ \mathbf{x}_k \}_{k=0}$ converges to a global
minimum of the function $F$. In general we do not know if we are in a
global or local minimum. In the special case when $F$ is a convex
function, all local minima are also global minima, so in this case
gradient descent can converge to the global solution. The advantage of
this scheme is that it is conceptually simple and straightforward to
implement. However the method in this form has some severe
limitations:

In machine learing we are often faced with non-convex high dimensional
cost functions with many local minima. Since GD is deterministic we
will get stuck in a local minimum, if the method converges, unless we
have a very good intial guess. This also implies that the scheme is
sensitive to the chosen initial condition.

Note that the gradient is a function of $\mathbf{x} =
(x_1,\cdots,x_n)$ which makes it expensive to compute numerically.

## The sensitiveness of the gradient descent

GD is sensitive to the choice of learning rate $\gamma_k$. This is due
to the fact that we are only guaranteed that $F(\mathbf{x}_{k+1}) \leq
F(\mathbf{x}_k)$ for sufficiently small $\gamma_k$. The problem is to
determine an optimal learning rate. If the learning rate is chosen too
small the method will take a long time to converge and if it is too
large we can experience erratic behavior.

Many of these shortcomings can be alleviated by introducing
randomness. One such method is that of Stochastic Gradient Descent
(SGD), see below.

## Gradient Descent Example

We revisit now our simple linear regression example with a linear polynomial.

```python
%matplotlib inline

# Importing various packages
from random import random, seed
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import sys

x = 2*np.random.rand(100,1)
y = 4+3*x+np.random.randn(100,1)

xb = np.c_[np.ones((100,1)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
print(beta_linreg)
beta = np.random.randn(2,1)

eta = 0.1
Niterations = 1000
m = 100

for iter in range(Niterations):
    gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)
    beta -= eta*gradients

print(beta)
xnew = np.array([[0],[2]])
xbnew = np.c_[np.ones((2,1)), xnew]
ypredict = xbnew.dot(beta)
ypredict2 = xbnew.dot(beta_linreg)
plt.plot(xnew, ypredict, "r-")
plt.plot(xnew, ypredict2, "b-")
plt.plot(x, y ,'ro')
plt.axis([0,2.0,0, 15.0])
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title(r'Gradient descent example')
plt.show()
```

## And a corresponding example using **scikit-learn**

```python
# Importing various packages
from random import random, seed
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import SGDRegressor

x = 2*np.random.rand(100,1)
y = 4+3*x+np.random.randn(100,1)

xb = np.c_[np.ones((100,1)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
print(beta_linreg)
sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)
sgdreg.fit(x,y.ravel())
print(sgdreg.intercept_, sgdreg.coef_)
```

    [[4.20374363]
     [2.83470272]]
    [4.21610176] [2.82633808]

    /usr/local/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:117: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.
      DeprecationWarning)

## Convex functions

Ideally we want our cost/loss function to be convex(concave).

First we give the definition of a convex set: A set $C$ in
$\mathbb{R}^n$ is said to be convex if, for all $x$ and $y$ in $C$ and
all $t \in (0,1)$ , the point $(1 − t)x + ty$ also belongs to
C. Geometrically this means that every point on the line segment
connecting $x$ and $y$ is in $C$ as discussed below.

The convex subsets of $\mathbb{R}$ are the intervals of
$\mathbb{R}$. Examples of convex sets of $\mathbb{R}^2$ are the
regular polygons (triangles, rectangles, pentagons, etc...).

## Convex function

**Convex function**: Let $X \subset \mathbb{R}^n$ be a convex set. Assume that the function $f: X \rightarrow \mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \in X$ and for all $t \in [0,1]$. If $\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.

## Conditions on convex functions

In the following we state first and second-order conditions which
ensures convexity of a function $f$. We write $D_f$ to denote the
domain of $f$, i.e the subset of $R^n$ where $f$ is defined. For more
details and proofs we refer to: [S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press](http://stanford.edu/boyd/cvxbook/, 2004).

**First order condition.**

Suppose $f$ is differentiable (i.e $\nabla f(x)$ is well defined for
all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$
is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds
for all $x,y \in D_f$. This condition means that for a convex function
the first order Taylor expansion (right hand side above) at any point
a global under estimator of the function. To convince yourself you can
make a drawing of f(x) = x^2+1 and draw the tangent line to $f(x)$ and
note that it is always below the graph.

**Second order condition.**

Assume that $f$ is twice
differentiable, i.e the Hessian matrix exists at each point in
$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its
Hessian is positive semi-definite for all $x\in D_f$. For a
single-variable function this reduces to $f''(x) \geq
0$. Geometrically this means that $f$ has nonnegative curvature
everywhere.

This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.

## More on convex functions

The next result is of great importance to us and the reason why we are
going on about convex functions. In machine learning we frequently
have to minimize a loss/cost function in order to find the best
parameters for the model we are considering.

Ideally we want the
global minimum (for high-dimensional models it is hard to know
if we have local or global minimum). However, if the cost/loss function
is convex the following result provides invaluable information:

**Any minimum is global for convex functions.**

Consider the problem of finding $x \in \mathbb{R}^n$ such that $f(x)$
is minimal, where $f$ is convex and differentiable. Then, any point
$x^*$ that satisfies $\nabla f(x^*) = 0$ is a global minimum.

This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.

## Some simple problems

1. Show that $f(x)=x^2$ is convex for $x \in \mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \in D_f$ and any $\lambda \in [0,1] $ $\lambda f(x) + (1-\lambda)f(y) - f(\lambda x + (1-\lambda) y ) \geq 0. $

2. Using the second order condition show that the following functions are convex on the specified domain.

 * $f(x) = e^x$ is convex for $x \in \mathbb{R}$.

 * $g(x) = -\ln(x)$ is convex for $x \in (0,\infty)$.

3. Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \in \mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex.

4. A norm is any function that satisfy the following properties

 * $f(\alpha x) = |\alpha| f(x)$ for all $\alpha \in \mathbb{R}$.

 * $f(x+y) \leq f(x) + f(y)$

 * $f(x) \leq 0$ for all $x \in \mathbb{R}^n$ with equality if and only if $x = 0$

Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).

## Revisiting our first homework

We will use linear regression as a case study for the gradient descent
methods. Linear regression is a great test case for the gradient
descent methods discussed in the lectures since it has several
desirable properties such as:

1. An analytical solution (recall homework set 1).

2. The gradient can be computed analytically.

3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates

We revisit the example from homework set 1 where we had

$$
y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
$$

with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally $\xi_i$ represents stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$.
The linear regression model is given by

$$
h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
$$

such that

$$
\hat{y}_i = \beta_0 + \beta_1 x_i.
$$

## Gradient descent example

Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$

t is convenient to write $\mathbf{\hat{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by

$$
\begin{equation}
X \equiv \begin{bmatrix}
1 & x_1  \\
\vdots & \vdots  \\
1 & x_{100} &  \\
\end{bmatrix}.
\label{_auto1} \tag{1}
\end{equation}
$$

The loss function is given by

$$
C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2
$$

and we want to find $\beta$ such that $C(\beta)$ is minimized.

## The derivative of the cost/loss function

Computing $\partial C(\beta) / \partial \beta_0$ and $\partial C(\beta) / \partial \beta_1$ we can show  that the gradient can be written as

$$
\nabla_\beta C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = 2X^T(X\beta - \mathbf{y}),
$$

where $X$ is the design matrix defined above.

## The Hessian matrix
The Hessian matrix of $C(\beta)$ is given by

$$
\hat{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1}  \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &  \\
\end{bmatrix} = 2X^T X.
$$

This result implies that $C(\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.

## Simple program

We can now write a program that minimizes $C(\beta)$ using the gradient descent method with a constant learning rate $\gamma$ according to

$$
\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots
$$

We can use the expression we computed for the gradient and let use a
$\beta_0$ be chosen randomly and let $\gamma = 0.001$. Stop iterating
when $||\nabla_\beta C(\beta_k) || < \epsilon = 10^{-8}$.

And finally we can compare our solution for $\beta$ with the analytic result given by
$\beta= (X^TX)^{-1} X^T \mathbf{y}$.

```python
import numpy as np

"""
The following setup is just a suggestion, feel free to write it the way you like.
"""

#Setup problem described in the exercise
N  = 100 #Nr of datapoints
M  = 2 #Nr of features
x  = np.random.rand(N) #Uniformly generated x-values in [0,1]
y  = 5*x**2 + 0.1*np.random.randn(N)
X  = np.c_[np.ones(N),x] #Construct design matrix

#Compute beta according to normal equations to compare with GD solution
Xt_X_inv = np.linalg.inv(np.dot(X.T,X))
Xt_y     = np.dot(X.transpose(),y)
beta_NE = np.dot(Xt_X_inv,Xt_y)
print(beta_NE)
```

    [-0.72135172  4.78496149]

## Gradient descent and Ridge

We have also discussed Ridge regression where the loss function contains a regularized given by the $L_2$ norm of $\beta$,

$$
C_{\text{ridge}}(\beta) = ||X\beta -\mathbf{y}||^2 + \lambda ||\beta||^2, \ \lambda \geq 0.
$$

In order to minimize $C_{\text{ridge}}(\beta)$ using GD we only have adjust the gradient as follows

$$
\nabla_\beta C_{\text{ridge}}(\beta)  = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (X^T(X\beta - \mathbf{y})+\lambda \beta).
$$

We can now extend our program to minimize $C_{\text{ridge}}(\beta)$ using gradient descent and compare with the analytical solution given by

$$
\beta_{\text{ridge}} = \left(X^T X + \lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y},
$$

for $\lambda = {0,1,10,50,100}$ ($\lambda = 0$ corresponds to ordinary least squares).
We can then compute $||\beta_{\text{ridge}}||$ for each $\lambda$.

```python
import numpy as np

"""
The following setup is just a suggestion, feel free to write it the way you like.
"""

#Setup problem described in the exercise
N  = 100 #Nr of datapoints
M  = 2   #Nr of features
x  = np.random.rand(N)
y  = 5*x**2 + 0.1*np.random.randn(N)

#Compute analytic beta for Ridge regression
X    = np.c_[np.ones(N),x]
XT_X = np.dot(X.T,X)

l  = 0.1 #Ridge parameter lambda
Id = np.eye(XT_X.shape[0])

Z = np.linalg.inv(XT_X+l*Id)
beta_ridge = np.dot(Z,np.dot(X.T,y))

print(beta_ridge)
print(np.linalg.norm(beta_ridge)) #||beta||
```

    [-0.76259836  4.89939464]
    4.958389265060428

## Stochastic Gradient Descent

Stochastic gradient descent (SGD) and variants thereof address some of
the shortcomings of the Gradient descent method discussed above.

The underlying idea of SGD comes from the observation that the cost
function, which we want to minimize, can almost always be written as a
sum over $n$ datapoints $\{\mathbf{x}_i\}_{i=1}^n$,

$$
C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$

## Computation of gradients

This in turn means that the gradient can be
computed as a sum over $i$-gradients

$$
\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}).
$$

Stochasticity/randomness is introduced by only taking the
gradient on a subset of the data called minibatches.  If there are $n$
datapoints and the size of each minibatch is $M$, there will be $n/M$
minibatches. We denote these minibatches by $B_k$ where
$k=1,\cdots,n/M$.

## SGD example
As an example, suppose we have $10$ datapoints $( \mathbf{x}_1,
\cdots, \mathbf{x}_{10} )$ and we choose to have $M=5$ minibathces,
then each minibatch contains two datapoints. In particular we have
$B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 =
(\mathbf{x}_9,\mathbf{x}_{10})$. Note that if you choose $M=1$ you
have only a single batch with all datapoints and on the other extreme,
you may choose $M=n$ resulting in a minibatch for each datapoint, i.e
$B_k = \mathbf{x}_k$.

The idea is now to approximate the gradient by replacing the sum over
all datapoints with a sum over the datapoints in one the minibatches
picked at random in each gradient descent step

$$
\nabla_\beta
C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta
c_i(\mathbf{x}_i, \mathbf{\beta}).
$$

## The gradient step

Thus a gradient descent step now looks like

$$
\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i,
\mathbf{\beta})
$$

where $k$ is picked at random with equal
probability from $[1,n/M]$. An iteration over the number of
minibathces (n/M) is commonly referred to as an epoch. Thus it is
typical to choose a number of epochs and for each epoch iterate over
the number of minibatches, as exemplified in the code below.

## Simple example code

```python
import numpy as np

n = 100 #100 datapoints
M = 5   #size of each minibatch
m = int(n/M) #number of minibatches
n_epochs = 10 #number of epochs

j = 0
for epoch in range(1,n_epochs+1):
    for i in range(m):
        k = np.random.randint(m) #Pick the k-th minibatch at random
        #Compute the gradient using the data in minibatch Bk
        #Compute new suggestion for
        j += 1
```

Taking the gradient only on a subset of the data has two important
benefits. First, it introduces randomness which decreases the chance
that our opmization scheme gets stuck in a local minima. Second, if
the size of the minibatches are small relative to the number of
datapoints ($M < n$), the computation of the gradient is much
cheaper since we sum over the datapoints in the k-th minibatch and not
all $n$ datapoints.

## When do we stop?

A natural question is when do we stop the search for a new minimum?
One possibility is to compute the full gradient after a given number
of epochs and check if the norm of the gradient is smaller than some
threshold and stop if true. However, the condition that the gradient
is zero is valid also for local minima, so this would only tell us
that we are close to a local/global minimum. However, we could also
evaluate the cost function at this point, store the result and
continue the search. If the test kicks in at a later stage we can
compare the values of the cost function and keep the $\beta$ that
gave the lowest value.

## Slightly different approach

Another approach is to let the step length $\gamma_j$ depend on the
number of epochs in such a way that it becomes very small after a
reasonable time such that we do not move at all.

As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.

In this way we can fix the number of epochs, compute $\beta$ and
evaluate the cost function at the end. Repeating the computation will
give a different result since the scheme is random by design. Then we
pick the final $\beta$ that gives the lowest value of the cost
function.

```python
import numpy as np

def step_length(t,t0,t1):
    return t0/(t+t1)

n = 100 #100 datapoints
M = 5   #size of each minibatch
m = int(n/M) #number of minibatches
n_epochs = 500 #number of epochs
t0 = 1.0
t1 = 10

gamma_j = t0/t1
j = 0
for epoch in range(1,n_epochs+1):
    for i in range(m):
        k = np.random.randint(m) #Pick the k-th minibatch at random
        #Compute the gradient using the data in minibatch Bk
        #Compute new suggestion for beta
        t = epoch*m+i
        gamma_j = step_length(t,t0,t1)
        j += 1

print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
```

## Conjugate gradient (CG) method
The success of the CG method  for finding solutions of non-linear problems is based
on the theory of conjugate gradients for linear systems of equations. It belongs
to the class of iterative methods for solving problems from linear algebra of the type

$$
\hat{A}\hat{x} = \hat{b}.
$$

In the iterative process we end up with a problem like

$$
\hat{r}= \hat{b}-\hat{A}\hat{x},
$$

where $\hat{r}$ is the so-called residual or error in the iterative process.

When we have found the exact solution, $\hat{r}=0$.

## Conjugate gradient method

The residual is zero when we reach the minimum of the quadratic equation

$$
P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b},
$$

with the constraint that the matrix $\hat{A}$ is positive definite and symmetric.
If we search for a minimum of the quantum mechanical  variance, then the matrix
$\hat{A}$, which is called the Hessian, is given by the second-derivative of the function we want to minimize.  This quantity is always positive definite.  In our case this corresponds normally to the second derivative of the energy.

## Conjugate gradient method, Newton's method first
We seek the minimum of the energy or the variance as function of various variational parameters.
In our case we have thus a function $f$ whose minimum we are seeking.
In Newton's method we set $\nabla f = 0$ and we can thus compute the next iteration point

$$
\hat{x}-\hat{x}_i=\hat{A}^{-1}\nabla f(\hat{x}_i).
$$

Subtracting this equation from that of $\hat{x}_{i+1}$ we have

$$
\hat{x}_{i+1}-\hat{x}_i=\hat{A}^{-1}(\nabla f(\hat{x}_{i+1})-\nabla f(\hat{x}_i)).
$$

## Conjugate gradient method
In the CG method we define so-called conjugate directions and two vectors
$\hat{s}$ and $\hat{t}$
are said to be
conjugate if

$$
\hat{s}^T\hat{A}\hat{t}= 0.
$$

The philosophy of the CG method is to perform searches in various conjugate directions
of our vectors $\hat{x}_i$ obeying the above criterion, namely

$$
\hat{x}_i^T\hat{A}\hat{x}_j= 0.
$$

Two vectors are conjugate if they are orthogonal with respect to
this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is conjugate to $\hat{t}$, then $\hat{t}$ is conjugate to $\hat{s}$.

## Conjugate gradient method
An example is given by the eigenvectors of the matrix

$$
\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j,
$$

which is zero unless $i=j$.

## Conjugate gradient method
Assume now that we have a symmetric positive-definite matrix $\hat{A}$ of size
$n\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector

$$
\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}.
$$

We assume that $\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions.
Then the $\hat{p}_{i}$  form a basis of $R^n$ and we can expand the solution
$  \hat{A}\hat{x} = \hat{b}$ in this basis, namely

$$
\hat{x}  = \sum^{n}_{i=1} \alpha_i \hat{p}_i.
$$

## Conjugate gradient method
The coefficients are given by

$$
\mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}.
$$

Multiplying with $\hat{p}_k^T$  from the left gives

$$
\hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b},
$$

and we can define the coefficients $\alpha_k$ as

$$
\alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k}
$$

## Conjugate gradient method and iterations

If we choose the conjugate vectors $\hat{p}_k$ carefully,
then we may not need all of them to obtain a good approximation to the solution
$\hat{x}$.
We want to regard the conjugate gradient method as an iterative method.
This will us to solve systems where $n$ is so large that the direct
method would take too much time.

We denote the initial guess for $\hat{x}$ as $\hat{x}_0$.
We can assume without loss of generality that

$$
\hat{x}_0=0,
$$

or consider the system

$$
\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0,
$$

instead.

## Conjugate gradient method
One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form

$$
f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n.
$$

This suggests taking the first basis vector $\hat{p}_1$
to be the gradient of $f$ at $\hat{x}=\hat{x}_0$,
which equals

$$
\hat{A}\hat{x}_0-\hat{b},
$$

and
$\hat{x}_0=0$ it is equal $-\hat{b}$.
The other vectors in the basis will be conjugate to the gradient,
hence the name conjugate gradient method.

## Conjugate gradient method
Let  $\hat{r}_k$ be the residual at the $k$-th step:

$$
\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k.
$$

Note that $\hat{r}_k$ is the negative gradient of $f$ at
$\hat{x}=\hat{x}_k$,
so the gradient descent method would be to move in the direction $\hat{r}_k$.
Here, we insist that the directions $\hat{p}_k$ are conjugate to each other,
so we take the direction closest to the gradient $\hat{r}_k$
under the conjugacy constraint.
This gives the following expression

$$
\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k.
$$

## Conjugate gradient method
We can also  compute the residual iteratively as

$$
\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1},
$$

which equals

$$
\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k),
$$

or

$$
(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k,
$$

which gives

$$
\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k},
$$


# Using `scipy` iterative solvers

The aim of this notebook is to show how the `scipy.sparse.linalg` module can be used to solve iteratively the Lippmann–Schwinger equation.

The problem at hand is a single ellipsoid in a periodic unit-cell, subjected to a macroscopic strain $\mathbf E$. Again, we strive to make the implementation dimension independent. We therefore introduce the dimension `dim` of the physical space, and the dimension `sym = (dim*(dim+1))//2` of the space of second-order, symmetric tensors.

We start by importing a few modules, including `h5py`, since input and output data are stored in `HDF5` format.

```python
import h5py as h5
import matplotlib.pyplot as plt
import numpy as np
```

```python
import janus
import janus.material.elastic.linear.isotropic as material
import janus.operators as operators
import janus.fft.serial as fft
import janus.green as green
```

```python
from scipy.sparse.linalg import cg, LinearOperator
```

```python
%matplotlib inline
```

```python
plt.rcParams['figure.figsize'] = (12, 8)
```

## Generating the microstructure

The microstructure is generated by means of the `gen_ellipsoid.py` script, which is listed below.

```python
%pfile gen_ellipsoid.py
```

The above script should be invoked as follows

```
python gen_ellipsoid.py tutorial.json
```

where the `tutorial.json` file holds the geometrical parameters.

```python
%pfile tutorial.json
```

The resulting microstructure is to be saved into the `example.h5` file.

```python
%run gen_ellipsoid.py tutorial.json
```

The microstructure is then retrieved as follows.

```python
with h5.File('./tutorial.h5', 'r') as f:
    phase = np.asarray(f['phase'])

```

We can retrieve `dim` and `sym` from the dimensions of the microstructure.

```python
dim = phase.ndim
sym = (dim*(dim+1))//2
```

And we can check visually that everything went all right.

```python
plt.imshow(phase);
```

## Creating the basic objects for the simulations

We first select the elastic properties of the inclusion, the matrix, and the reference material. For the latter, we select a material which is close to the matrix, but not equal, owing to the $(\mathbf C_{\mathrm m}-\mathbf C_{\mathrm{ref}})^{-1}$ factor in the Lippmann–Schwinger equation.

```python
mu_i, nu_i = 10., 0.3  # Elastic properties of the ellipsoidal inclusion
mu_m, nu_m = 1., 0.2  # Elastic properties of the matrix
mu_ref, nu_ref = 0.99*mu_m, nu_m  # Elastic properties of the reference material
```

We then define instances of the `IsotropicLinearElasticMaterial` class for all three materials.

```python
C_i = material.create(mu_i, nu_i, dim=dim)
C_m = material.create(mu_m, nu_m, dim=dim)
C_ref = material.create(mu_ref, nu_ref, dim=dim)
type(C_i)
```

We want to solve the Lippmann–Schwinger equation, which reads

\begin{equation}
\bigl(\mathbf C-\mathbf C_{\mathrm{ref}}\bigr)^{-1}:\boldsymbol\tau+\boldsymbol\Gamma_{\mathrm{ref}}[\boldsymbol\tau]=\mathbf E,
\end{equation}

where $\mathbf C=\mathbf C_{\mathrm i}$ in the inclusion, $\mathbf C=\mathbf C_{\mathrm m}$ in the matrix, and $\boldsymbol\Gamma_{\mathrm{ref}}$ is the fourth-order Green operator for strains. After suitable discretization, the above problem reads

\begin{equation}
\bigl(\mathbf C^h-\mathbf C_{\mathrm{ref}}\bigr)^{-1}:\boldsymbol\tau^h+\boldsymbol\Gamma_{\mathrm{ref}}^h[\boldsymbol\tau^h]=\mathbf E,
\end{equation}

where $\mathbf C^h$ denotes the local stiffness, discretized over a cartesian grid of size $N_1\times\cdots\times N_d$; in other words, it can be viewed as an array of size $N_1\times\cdot\times N_d\times s\times s$ and $\boldsymbol\Gamma_{\mathrm{ref}}^h$ is the discrete Green operator. The unknown discrete polarization field $\boldsymbol\tau^h$ ($N_1\times\cdots\times N_d\times s$ array) is constant over each cell of the cartesian grid. It can be assembled into a column vector, $x$. Likewise, $\mathbf E$ should be understood as a macroscopic strain field which is equal to $\mathbf E$ in each cell of the grid; it can be assembled into a column vector, $b$.

Finally, the operator $\boldsymbol\tau^h\mapsto\bigl(\mathbf C^h-\mathbf C_{\mathrm{ref}}\bigr)^{-1}:\boldsymbol\tau^h+\boldsymbol\Gamma_{\mathrm{ref}}^h[\boldsymbol\tau^h]$ is linear in $\boldsymbol\tau^h$ (or, equivalently, $x$); it can be assembled as a matrix, $A$. Then, the discrete Lippmann–Schwinger equation reads

\begin{equation}
A\cdot x=b,
\end{equation}

which can be solved by means of any linear solver. However, two observations should be made. First, the matrix $A$ is *full*; its assembly and storage might be extremely costly. Second, the matrix-vector product $x\mapsto A\cdot x$ can efficiently be implemented. This is the *raison d'être* of a library like `Janus`!

These observation suggest to implement $A$ as a `linearOperator`, in the sense of the `scipy` library ([see reference](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.LinearOperator.html#scipy-sparse-linalg-linearoperator)).

```python
class MyLinearOperator(LinearOperator):
    def __init__(self, phase, C_m, C_i, C_ref):
        dim = phase.ndim
        sym = (dim*(dim+1))//2
        alpha_i = 1./dim/(C_i.k-C_ref.k)
        beta_i = 1./2./(C_i.g-C_ref.g)
        alpha_m = 1./dim/(C_m.k-C_ref.k)
        beta_m = 1./2./(C_m.g-C_ref.g)

        T = np.array([operators.isotropic_4(alpha_i, beta_i, dim),
                      operators.isotropic_4(alpha_m, beta_m, dim)])
        self.tau2eps = operators.block_diagonal_operator(T[phase])
        self.green = green.filtered(C_ref.green_operator(),
                                      phase.shape, 1.,
                                      fft.create_real(phase.shape))
        self.arr_shape = phase.shape+(sym,)
        n = np.product(self.arr_shape)
        super().__init__(np.float64, (n, n))

    def _matvec(self, x):
        tau = x.reshape(self.arr_shape)
        eta = np.zeros_like(tau)
        self.tau2eps.apply(tau, eta)
        eta += self.green.apply(tau)
        y = eta.ravel()
        return y

    def _rmatvec(self, x):
        return self._matvec(x)

    def empty_arr(self):
        return np.empty(self.arr_shape)
```

The constructor of the above class first computes the local map $\boldsymbol\tau^h\mapsto\bigl(\mathbf C^h-\mathbf C_{\mathrm{ref}}\bigr)^{-1}$. Then, it implements the operator $\boldsymbol\tau^h\mapsto\bigl(\mathbf C^h-\mathbf C_{\mathrm{ref}}\bigr)^{-1}:\boldsymbol\tau^h$. The resulting operator is called `tau2eps`.

The constructor also implements a discrete Green operator, associated with the reference material. Several discretization options are offered in Janus. The *filtered* Green operator is a good option. **TODO Use the Willot operator instead**.

Finally, the operator $A$ is implemented in the `_matvec` method, where attention should be paid to the fact that `x` is a column-vector, while `green` and `tau2eps` both operates on fields that have the shape of a symmetric, second-order tensor field defined over the whole grid, hence the `reshape` operation. It is known that the operator $A$ is symmetric by construction. Therefore, the `_rmatvec` method calls `_matvec`.

## Solving the Lippmann–Schwinger equation

We are now ready to solve the equation. We first create an instance of the linear operator $A$.

```python
a = MyLinearOperator(phase, C_m, C_i, C_ref)
```

We then create the macroscopic strain $\mathbf E$ that is imposed to the unit-cell. In the present case, we take $E_{xy}=1$ (beware the Mandel–Voigt notation!).

```python
eps_macro = np.zeros((sym,), dtype=np.float64)
eps_macro[-1] = np.sqrt(2.)
```

We then populate the right-hand side vector, $b$. `b_arr` is the column-vector $b$, viewed as a discrete, second order 2D tensor field. It is then flattened through the `ravel` method.

```python
b_arr = a.empty_arr()
b_arr[...] = eps_macro
b = b_arr.ravel()
```

We know that the linear operator $A$ is definite. We can therefore use the conjugate gradient method to solve $A\cdot x=b$.

```python
x, info = cg(a, b)
assert info == 0
```

The resulting solution, $x$, must be reshaped into a $N_1\times\cdots\times N_d\times s$ array.

```python
tau = x.reshape(a.arr_shape)
```

We can plot the $\tau_{xy}$ component.

```python
plt.imshow(tau[..., -1]);
```

And compute the associated strain field.

```python
eps = a.tau2eps.apply(tau)
```

And plot the $\varepsilon_{xy}$ component.

```python
plt.imshow(eps[..., -1]);
```

```python
plt.plot(eps[63, :, -1])
```


# ニュートン法と最急降下法の比較
今回は、ニュートン法を用いた関数の最適化を行い、最急降下法との収束性の比較をしてみましょう。
まず、最急降下法およびニュートン法のアルゴリズムをそれぞれ実装し、その後教科書(福島 2011) の例(式4.19) に関して生成点列の挙動の確認を、その後、前回扱った関数についても同様に生成点列の挙動の確認をしてみましょう。

## 最急降下法
比較のために、まずは与えられた初期点をもとに、最急降下法による点列の生成を行う関数`sdm` を実装しましょう。
目的関数`fun`、その勾配`jac`、および目的関数のヘッセ行列`hess` は、それぞれPython 上の関数として与えられるもの(後に用意するもの)とします。
ここでは、反復回数は15 回としました。

```python
import numpy as np
from scipy.optimize import line_search

def sdm(xk, iter=15):
    sequence = [xk]
    for k in range(iter):
        dk = -jac(xk)
        alpha = line_search(fun, jac, xk, dk)[0]
        xk = xk + alpha * dk
        sequence.append(xk)
    return np.array(sequence)
```

## ニュートン法
続いて、与えられた初期点をもとに、ニュートン法による点列の生成を行う関数`newton` を実装しましょう。
最急降下法と同様に、反復回数は15 回とします。
ニュートン法は、教科書(福島 2011) にある通り、

0. $x^{(0)}\in\mathbb{R}^n$, $k:=0$.
1. $d^{(k)}:=-\nabla^2f(x^{(k)})^{-1}\nabla f(x^{(k)})$.
2. $x^{(k+1)}:=x^{(k)}+d^{(k)}$, $k:=k+1$. goto 1.

のステップを反復します。
ここでは、このプログラムをそのまま書き起こしてみましょう。

```python
import numpy as np
from scipy.optimize import line_search

def newton(xk, iter=15):
    sequence = [xk]
    for k in range(iter):
        dk = np.linalg.solve(hess(xk), -jac(xk))
        xk = xk + dk
        sequence.append(xk)
    return np.array(sequence)
```

## 教科書の例での生成点列の挙動
### 目的関数、勾配、ヘッセ行列の実装
まずは、教科書(福島 2011) の例(式4.19)、
\begin{align*}
f(x)=(x_0-1)^2+10(x_0^2-x_1)^2
\end{align*}
に対して、最急降下法とニュートン法をそれぞれ適用し、その生成点列の挙動を確認してみましょう。
目的関数を`fun`、その勾配を`jac`、そして目的関数のヘッセ行列を`hess` として、それぞれPython 上の関数として、それぞれ実装します。

```python
import numpy as np

def fun(x):
    return (x[0] - 1) ** 2 + 10 * (x[0] ** 2 - x[1]) ** 2

def jac(x):
    return np.array([2 * (x[0] - 1) + 40 * (x[0] ** 3 - x[0] * x[1]), -20 * (x[0] ** 2 - x[1])])

def hess(x):
    return np.array([[120 * (x[0] ** 2) - 40 * x[1] + 2, -40 * x[0]],
                      [-40 * x[0], 20]])
```

### 生成点列の観察と比較
それでは、原点$(0, 0)^\top$ を初期点に、最急降下法(SDM) とニュートン法(Newton) のそれぞれが生成する点列を観察してみましょう。

```python
%matplotlib inline
import matplotlib.pyplot as plt
X, Y = np.meshgrid(np.linspace(-0.25, 1.25, 100), np.linspace(-0.25, 1.25, 100))
Z = np.array([[fun(np.array([x, y])) for x, y in zip(vx, vy)] for vx, vy in zip(X, Y)])
plt.contour(X, Y, Z, cmap='plasma', levels=np.linspace(np.min(Z), np.max(Z), 20))
sequence = sdm(np.array([0., 0.]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='o', label='SDM')
sequence = newton(np.array([0., 0.]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='x', label='Newton')
plt.legend()
```

最急降下法が小刻みに振動しながら解を更新しているのに対し、Newton 法は極めて効率よく、2 回ほどの反復でほぼ最適解の$(1, 1)^\top$ 付近に到達していることが分かります。
したがって、この例では、Newton 法は最急降下法と比較して、比べ物にならないくらい効率的な挙動をすることが分かります。
しかし、Newton 法を用いれば、常に最急降下法よりも効率的に最適解を求めることができるのでしょうか・・・？

## 前回の例での生成点列の挙動
続いて、前回例として用いた２変数関数、
\begin{align*}
f(x_0, x_1):=\sin\left(\frac{1}{2}x_0^2-\frac{1}{4}x_1^2+3\right)\cos(2x_0+1-e^{x_1})
\end{align*}
を目的関数として考えましょう。
この関数は、前回確認した通り、下記のような複雑な形状をしていました。

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(8, 6), dpi=80)
ax = plt.axes(projection='3d')
X, Y = np.meshgrid(np.linspace(-1, 1, 100), np.linspace(-1, 1, 100))
Z = np.array([[np.sin((x ** 2) / 2 - (y ** 2 ) / 4 + 3) * np.cos(2 * x + 1 - np.exp(y)) for x, y in zip(vx, vy)] for vx, vy in zip(X, Y)])
ax.plot_surface(X, Y, Z, cmap='plasma')
```

### 目的関数、勾配、ヘッセ行列の実装
先程と同様に、目的関数を`fun`、その勾配を`jac`、そして目的関数のヘッセ行列を`hess` として、それぞれPython 上の関数として実装しましょう。
なお、数式の微分については、Anaconda に標準で含まれている[SymPy](https://www.sympy.org/en/index.html) という機能を用いてコンピュータに自動で計算させることができます。
(記号微分: 我々が普段微分の計算を行うときと同じように、パターンを置き換えて数式を微分する計算法)
ここでは詳細な説明は省きますが、下記が関数`F` とその各偏微分をSymPy を用いて計算するプログラムです。

```python
import numpy as np
from sympy import symbols, sin, cos, exp, diff

_x, _y = symbols('x, y')
F = sin((_x ** 2) / 2 - (_y ** 2) / 4 + 3) * cos(2 * _x + 1 - exp(_y))
FdX, FdY = diff(F, _x), diff(F, _y)
FdXdX, FdXdY = diff(FdX, _x), diff(FdX, _y)
FdYdX, FdYdY = diff(FdY, _x), diff(FdY, _y)

def fun(x):
    p = {_x: x[0], _y: x[1]}
    return float(F.subs(p))

def jac(x):
    p = {_x: x[0], _y: x[1]}
    return np.array([FdX.subs(p), FdY.subs(p)], dtype=np.float)

def hess(x):
    p = {_x: x[0], _y: x[1]}
    return np.array([[FdXdX.subs(p), FdXdY.subs(p)],
                      [FdYdX.subs(p), FdYdY.subs(p)]], dtype=np.float)
```

### 生成点列の観察と比較 (ニュートン法が収束する例)
それでは、最急降下法により生成した点列と、ニュートン法で生成した点列の、それぞれの挙動を、目的関数に関する等高線グラフの上に描画し、確認してみましょう。
まずは、初期点$x^{(0)}:=(-0.75, -0.25)^\top$ (解の十分近く) とした例です。

```python
%matplotlib inline
import matplotlib.pyplot as plt
X, Y = np.meshgrid(np.linspace(-1, 1, 100), np.linspace(-1, 1, 100))
plt.ylim(-1, 1)
plt.xlim(-1, 1)
Z = np.array([[np.sin((x ** 2) / 2 - (y ** 2 ) / 4 + 3) * np.cos(2 * x + 1 - np.exp(y)) for x, y in zip(vx, vy)] for vx, vy in zip(X, Y)])
plt.contour(X, Y, Z, cmap='plasma', levels=np.linspace(np.min(Z), np.max(Z), 15))
sequence = sdm(np.array([-0.75, -0.25]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='o', label='SDM')
sequence = newton(np.array([-0.75, -0.25]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='x', label='Newton')
plt.legend()
```

この場合、細かな解の更新を繰り返す最急降下法に比べて、ニュートン法はかなり効率的に解を更新していることがわかります。

### 生成点列の観察と比較 (ニュートン法が収束しない例)
では、前回と同様、初期点$(-0.3, 0.2)^\top$ を与えた場合はどうなるのでしょうか？

```python
%matplotlib inline
import matplotlib.pyplot as plt
X, Y = np.meshgrid(np.linspace(-1, 1, 100), np.linspace(-1, 1, 100))
plt.ylim(-1, 1)
plt.xlim(-1, 1)
Z = np.array([[np.sin((x ** 2) / 2 - (y ** 2 ) / 4 + 3) * np.cos(2 * x + 1 - np.exp(y)) for x, y in zip(vx, vy)] for vx, vy in zip(X, Y)])
plt.contour(X, Y, Z, cmap='plasma', levels=np.linspace(np.min(Z), np.max(Z), 15))
sequence = sdm(np.array([-0.3, 0.2]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='o', label='SDM')
sequence = newton(np.array([-0.3, 0.2]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='x', label='Newton')
plt.legend()
```

なんと、Newton 法で生成した点列は、画面の外(右上方向; 関数値が上がる方向) に吹き飛んでいることがわかります。
しかし最急降下法は、振動し効率が悪くはありますが、着実に最適解の方向へ点列を更新しています。

## 信頼領域法
Newton 法のような効率の良さと、最急降下法のような着実な収束性の両方を獲得するためにはどうすればよいのでしょうか？
ここでは、信頼領域法と呼ばれる最適化アルゴリズムの一種、`trust-exact` アルゴリズムを用いた最適化を行ってみましょう。
`trust-exact` アルゴリズムは、Anaconda に標準で付属しているSciPy に実装されており、
```python
from scipy.optimize import minimize
x = minimize(fun, x0, method='trust-exact', jac=jac, hess=hess).x
```
で、最適解`x` を求めることができます。
ここで、`fun` は目的関数、`jac` はその勾配、また`hess` は目的関数のヘッセ行列を求めるPython 関数とします。
それでは、最急降下法、ニュートン法、信頼領域法(`trust-exact`) の生成点列をそれぞれ求めて比較してみましょう。

```python
%matplotlib inline
import matplotlib.pyplot as plt
from scipy.optimize import minimize
X, Y = np.meshgrid(np.linspace(-1, 1, 100), np.linspace(-1, 1, 100))
plt.ylim(-1, 1)
plt.xlim(-1, 1)
Z = np.array([[np.sin((x ** 2) / 2 - (y ** 2 ) / 4 + 3) * np.cos(2 * x + 1 - np.exp(y)) for x, y in zip(vx, vy)] for vx, vy in zip(X, Y)])
plt.contour(X, Y, Z, cmap='plasma', levels=np.linspace(np.min(Z), np.max(Z), 15))
sequence = sdm(np.array([-0.3, 0.2]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='o', label='SDM')
sequence = newton(np.array([-0.3, 0.2]))
plt.plot(sequence[:, 0], sequence[:, 1], marker='x', label='Newton')
sequence = [np.array([-0.3, 0.2])]
minimize(fun, sequence[0], method='trust-exact', jac=jac, hess=hess, callback=lambda xk: sequence.append(xk))
sequence = np.array(sequence)
plt.plot(sequence[:, 0], sequence[:, 1], marker='v', label='trust-exact')
plt.legend()
```

結果から分かるように、信頼領域法(`trust-exact`) は、最急降下法と同様の最適解へ収束しますが、しかし最急降下法のように振動せず、かなり効率的に収束していることが分かります。

## 参考文献
  * 福島雅夫著『新版　数理計画入門』(朝倉書店; 2011)
  * 矢部博著『工学基礎　最適化とその応用』(数理工学社; 2006)
  * [Gradient descent - Wikipedia](https://en.wikipedia.org/wiki/Gradient_descent) (目的関数はこのページのものを使用しました。)


```python
%matplotlib inline
import matplotlib.pyplot as plt
import pycalphad.io.tdb_keywords
pycalphad.io.tdb_keywords.TDB_PARAM_TYPES.append('TH')
from pycalphad import Database, binplot, Model
import pycalphad.variables as v
from dask.distributed import Client
```

```python
TDB = """
 ELEMENT /-   ELECTRON_GAS              0.0000E+00  0.0000E+00  0.0000E+00!
 ELEMENT VA   VACUUM                    0.0000E+00  0.0000E+00  0.0000E+00!
 ELEMENT AL   FCC_A1                    2.6982E+01  4.5773E+03  2.8322E+01!
 ELEMENT SB   RHOMBOHEDRAL_A7           1.2175E+02  5.8702E+03  4.5522E+01!

TYPE_DEFINITION % SEQ * !
DEFINE_SYSTEM_DEFAULT SPECIE 2 !
DEFAULT_COMMAND DEF_SYS_ELEMENT VA !

FUN GHSERSB   298.15   -9242.858+156.154689*T-30.5130752*T*LN(T)
                          +.007748768*T**2-3.003415E-06*T**3
                          +100625*T**(-1);
              903.9 Y -11738.671+169.485713*T-31.38*T*LN(T)
                          +1.610442E+27*T**(-9);
             2000.00 N 91DIN !

 FUNCTION GHSERAL    2.98150E+02  -7976.15+137.093038*T-24.3671976*T*LN(T)
     -.001884662*T**2-8.77664E-07*T**3+74092*T**(-1);  7.00000E+02  Y
      -11276.24+223.048446*T-38.5844296*T*LN(T)+.018531982*T**2
     -5.764227E-06*T**3+74092*T**(-1);  9.33470E+02  Y
      -11278.378+188.684153*T-31.748192*T*LN(T)-1.230524E+28*T**(-9);
     2.90000E+03  N !

SPECIES AL1SB1 AL1SB1 !

PHASE LIQUID:L %  1  1.0 !
   CONSTITUENT LIQUID:L  : AL,SB,AL1SB1 : !

PHASE RHOMBOHEDRAL_A7 %  1  1.0 !
   CONSTITUENT RHOMBOHEDRAL_A7  : AL,SB : !

PHASE ALSB % 2 1.0 1.0 !
   CONSTITUENT ALSB : AL : SB : !

PARA G(RHOMBOHEDRAL_A7,SB;0)   298.15
 -9242.858+156.154689*T-30.5130752*T*LN(T)+7.748768E-3*T**2-3.003415E-6*T**3
 +100625*T**(-1);   HICKEL Y
 -11738.83+169.485872*T-31.38*T*LN(T)+1616.849E24*T**(-9);  2000.00 N !

PARA G(RHOMBOHEDRAL_A7,AL;0) 298.15 20000+GHSERAL#; 2000 N !

PARA TH(RHOMBOHEDRAL_A7,AL;0) 1 11111; 10000 N !
PARA TH(RHOMBOHEDRAL_A7,SB;0) 1 222222; 10000 N !

PARA G(LIQUID,AL;0) 2.98140E+02  +11005-11.8419*T+7.934E-20*T**7+GHSERAL#;
                 933  Y
      +10482.3-11.254*T+1.231E+28*T**(-9)+GHSERAL#;  6.00000E+03  N !

PARA G(LIQUID,SB;0)  298.15   +19822.329-21.923164*T-1.7485E-20*T**7
                                 +GHSERSB#;
                     903.78 Y +19914.189-22.029886*T-1.6168E+27*T**(-9)
                                 +GHSERSB#;
                    2000.00 N 91Din !
PARA G(LIQUID,AL1SB1;0) 298.15 -24797 + 6.6259*T + 0.82937*T*(1-LN(T)) + 2.6817E-3*T**2
                                 +GHSERSB# + GHSERAL#; 2000 N 90Cou !

PARA G(ALSB,AL:SB;0) 298.15 -24486 + 6.7006*T + 0.82783*T*(1-LN(T)) + 2.6797E-3*T**2
                                 +GHSERSB# + GHSERAL#; 2000 N 90Cou !

"""
```

```python
from sympy import Symbol

dbf = Database(TDB)

class CustomModel(Model):
    def __init__(self, dbe, comps, phase_name, parameters=None):
        super(self, CustomModel).__init__(dbe, comps, phase_name, parameters=parameters)
        phase = dbe.phases[self.phase_name]
        param_search = dbe.search
        th_param_query = (
            (where('phase_name') == phase.name) & \
            (where('parameter_type') == 'TH') & \
            (where('constituent_array').test(self._array_validity))
        )

        th = self.redlich_kister_sum(phase, param_search, th_param_query)
        #for name, value in self.models.items():
        #    self.models[name] = self.models[name].xreplace({Symbol('HICKEL'): th})

mod = Model(dbf, ['AL'], 'RHOMBOHEDRAL_A7')
mod.GM
```

    1.0*Piecewise((Piecewise((-8.77664e-7*T**3 - 0.001884662*T**2 - 24.3671976*T*log(T) + 137.093038*T - 7976.15 + 74092/T, (T >= 298.15) & (T < 700.0)), (-5.764227e-6*T**3 + 0.018531982*T**2 - 38.5844296*T*log(T) + 223.048446*T - 11276.24 + 74092/T, (T >= 700.0) & (T < 933.47)), (-31.748192*T*log(T) + 188.684153*T - 11278.378 - 1.230524e+28/T**9, (T >= 933.47) & (T < 2900.0)), (0, True)) + 20000, (T >= 298.15) & (T < 2000.0)), (0, True)) + 8.3145*T*Piecewise((RHOMBOHEDRAL_A70AL*log(RHOMBOHEDRAL_A70AL), RHOMBOHEDRAL_A70AL > 1.0e-13), (0, True))/RHOMBOHEDRAL_A70AL + 1.0*(24.9435*T*log(1 - exp(-exp(RHOMBOHEDRAL_A70AL*Piecewise((11111, (T >= 1.0) & (T < 10000.0)), (0, True)))/T)) + 12.47175*exp(RHOMBOHEDRAL_A70AL*Piecewise((11111, (T >= 1.0) & (T < 10000.0)), (0, True))))/RHOMBOHEDRAL_A70AL

```python
fig = plt.figure(figsize=(9,6))
scheduler = Client()

%time binplot(dbf, ['AL', 'SB', 'AL1SB1'] , sorted(dbf.phases.keys()), {v.X('SB'):(0,1,0.02), v.T: (300, 1300, 50), v.P:101325},  ax=fig.gca(), eq_kwargs={'scheduler': 'distributed'})

plt.show()
```

```python
dbf.phases
```

```python

```


```python
%load_ext autoreload
%autoreload 2

import os
os.chdir("..")
```

```python
from particlezoo import TemplateParser

tp = TemplateParser("standard.json")
```

```python
model = {
        "name": "Standard Model",
        "gauges": [
            {
                "name": "U(1)_Y",
                "group_type": "abelian",
                "coupling": "g_Y"
            },
            {
                "name": "SU(2)_L",
                "group_type": ["SU", "2"],
                "coupling": "g_L"
            },
            {
                "name": "SU(2)_R",
                "group_type": ["SU", "2"],
                "coupling": "g_R"
            }
        ],
        "fields": [
            {
                "name": "L",
                "description": "Left handed lepton",
                "spin": "1/2",
                "mass": "m_L",
                "representations": [
                    {"name": "U(1)_Y", "charge": "1/6"},
                    {"name": "SU(2)_L", "charge": "2", },
                    {"name": "SU(2)_R", "representation": [2]}
                ]
            }
        ]
    }

t = TemplateParser()
t.template = model

result = t.parse().build()
```

```python
result
```

    {'name': 'Standard Model', 'fields': [{'name': L, 'spin': 1/2, 'mass': <particlezoo.modeltemplates._models.NumericSymbol object at 0x7f5652057100>, 'representations': {'U(1)_Y': {'charge': '1/6', 'representation': 1/6}, 'SU(2)_L': {'charge': '2', 'representation': Matrix([[1]])}, 'SU(2)_R': {'representation': Matrix([[2]]), 'charge': 3}}}], 'gauges': [{'name': U(1)_Y, 'coupling': g_Y}, {'name': SU(2)_L, 'coupling': g_L}, {'name': SU(2)_R, 'coupling': g_R}]}

```python
from liesym import D
from sympy import *
```

```python
a = D(5)
a.get_irrep_by_dim(10, max_dd=5, with_symbols=True)
```

    [(Matrix([[1, 0, 0, 0, 0]]), 10)]

```python
a.tensor_product_decomposition([Matrix([[1, 0, 0, 0, 0]]),Matrix([[1, 0, 0, 0, 0]])])
```

    [Matrix([[0, 0, 0, 0, 0]]),
     Matrix([[0, 1, 0, 0, 0]]),
     Matrix([[2, 0, 0, 0, 0]])]

```python

```