[NumPyro](https://github.com/pyro-ppl/numpyro) is probabilistic programming language built on top of JAX. It is very similar to [Pyro](https://pyro.ai/), which is built on top of PyTorch.
However, the HMC algorithm in NumPyro
 [is much faster](https://stackoverflow.com/questions/61846620/numpyro-vs-pyro-why-is-former-100x-faster-and-when-should-i-use-the-latter).

Both Pyro flavors are usually also [faster than PyMc3](https://www.kaggle.com/s903124/numpyro-speed-benchmark), and allow for more complex models, since Pyro is integrated into Python.

# Installation

```
import numpy as np

np.set_printoptions(precision=3)
import matplotlib.pyplot as plt
import math
```

```
# When running in colab pro (high RAM mode), you get 4 CPUs.
# But we need to  force XLA to use all 4 CPUs
# This is generally faster than running in GPU mode

import os

os.environ["XLA_FLAGS"] = "--xla_force_host_platform_device_count=4"
```

```
# http://num.pyro.ai/en/stable/getting_started.html#installation

# CPU mode: often faster in colab!
!pip install numpyro

# GPU mode: as of July 2021, this does not seem to work
#!pip install numpyro[cuda111] -f https://storage.googleapis.com/jax-releases/jax_releases.html
```

    Collecting numpyro
      Downloading numpyro-0.7.2-py3-none-any.whl (250 kB)
         |████████████████████████████████| 250 kB 5.2 MB/s
    Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from numpyro) (4.41.1)
    Requirement already satisfied: jaxlib>=0.1.65 in /usr/local/lib/python3.7/dist-packages (from numpyro) (0.1.69+cuda110)
    Requirement already satisfied: jax>=0.2.13 in /usr/local/lib/python3.7/dist-packages (from numpyro) (0.2.17)
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from jax>=0.2.13->numpyro) (1.19.5)
    Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from jax>=0.2.13->numpyro) (0.12.0)
    Requirement already satisfied: opt-einsum in /usr/local/lib/python3.7/dist-packages (from jax>=0.2.13->numpyro) (3.3.0)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from jaxlib>=0.1.65->numpyro) (1.4.1)
    Requirement already satisfied: flatbuffers<3.0,>=1.12 in /usr/local/lib/python3.7/dist-packages (from jaxlib>=0.1.65->numpyro) (1.12)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py->jax>=0.2.13->numpyro) (1.15.0)
    Installing collected packages: numpyro
    Successfully installed numpyro-0.7.2

```
import jax

print("jax version {}".format(jax.__version__))
print("jax backend {}".format(jax.lib.xla_bridge.get_backend().platform))
print(jax.lib.xla_bridge.device_count())
print(jax.local_device_count())

import jax.numpy as jnp
from jax import random
```

    WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

    jax version 0.2.17
    jax backend cpu
    4
    4

```
import numpyro

# numpyro.set_platform('gpu')
import numpyro.distributions as dist
from numpyro.distributions import constraints
from numpyro.distributions.transforms import AffineTransform
from numpyro.infer import MCMC, NUTS, Predictive
from numpyro.infer import SVI, Trace_ELBO, init_to_value
from numpyro.diagnostics import hpdi, print_summary

from numpyro.infer.autoguide import AutoLaplaceApproximation

rng_key = random.PRNGKey(0)
rng_key, rng_key_ = random.split(rng_key)
```

# Example: 1d Gaussian with unknown mean.

We use the simple example from the [Pyro intro](https://pyro.ai/examples/intro_part_ii.html#A-Simple-Example). The goal is to infer the weight $\theta$ of an object, given noisy measurements $y$. We assume the following model:
$$
\begin{align}
\theta &\sim N(\mu=8.5, \tau^2=1.0)\\
y \sim &N(\theta, \sigma^2=0.75^2)
\end{align}
$$

Where $\mu=8.5$ is the initial guess.

## Exact inference

By Bayes rule for Gaussians, we know that the exact posterior,
given a single observation $y=9.5$, is given by

$$
\begin{align}
\theta|y &\sim N(m, s^s) \\
m &=\frac{\sigma^2 \mu + \tau^2 y}{\sigma^2 + \tau^2}
  = \frac{0.75^2 \times 8.5 + 1 \times 9.5}{0.75^2 + 1^2}
  = 9.14 \\
s^2 &= \frac{\sigma^2 \tau^2}{\sigma^2  + \tau^2}
= \frac{0.75^2 \times 1^2}{0.75^2 + 1^2}= 0.6^2
\end{align}
$$

```
mu = 8.5
tau = 1.0
sigma = 0.75
hparams = (mu, tau, sigma)

y = 9.5
m = (sigma**2 * mu + tau**2 * y) / (sigma**2 + tau**2)
s2 = (sigma**2 * tau**2) / (sigma**2 + tau**2)
s = np.sqrt(s2)
print(m)
print(s)
```

    9.14
    0.6

```
def model(hparams, y=None):
    prior_mean, prior_sd, obs_sd = hparams
    theta = numpyro.sample("theta", dist.Normal(prior_mean, prior_sd))
    y = numpyro.sample("y", dist.Normal(theta, obs_sd), obs=y)
    return y
```

## Ancestral sampling

```
def model2(hparams):
    prior_mean, prior_sd, obs_sd = hparams
    theta = numpyro.sample("theta", dist.Normal(prior_mean, prior_sd))
    yy = numpyro.sample("y", dist.Normal(theta, obs_sd))
    return theta, yy
```

```
with numpyro.handlers.seed(rng_seed=0):
    for i in range(5):
        theta, yy = model2(hparams)
        print([theta, yy])
```

    [DeviceArray(7.248, dtype=float32), DeviceArray(6.808, dtype=float32)]
    [DeviceArray(8.986, dtype=float32), DeviceArray(9.149, dtype=float32)]
    [DeviceArray(7.851, dtype=float32), DeviceArray(8.856, dtype=float32)]
    [DeviceArray(9.538, dtype=float32), DeviceArray(8.973, dtype=float32)]
    [DeviceArray(7.895, dtype=float32), DeviceArray(6.133, dtype=float32)]

## MCMC

See [the documentation](https://num.pyro.ai/en/stable/mcmc.html)

```
conditioned_model = numpyro.handlers.condition(model, {"y": y})
nuts_kernel = NUTS(conditioned_model)

mcmc = MCMC(nuts_kernel, num_warmup=200, num_samples=200, num_chains=4)
mcmc.run(rng_key_, hparams)

mcmc.print_summary()
samples = mcmc.get_samples()
```

    HBox(children=(FloatProgress(value=0.0, max=400.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=400.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=400.0), HTML(value='')))

    HBox(children=(FloatProgress(value=0.0, max=400.0), HTML(value='')))

                    mean       std    median      5.0%     95.0%     n_eff     r_hat
         theta      9.20      0.61      9.19      8.31     10.27    259.86      1.01

    Number of divergences: 0

```
print(type(samples))
print(type(samples["theta"]))
print(samples["theta"].shape)
```

    <class 'dict'>
    <class 'jax.interpreters.xla._DeviceArray'>
    (1000,)

```
nuts_kernel = NUTS(model)  # this is the unconditioned model
mcmc = MCMC(nuts_kernel, num_warmup=100, num_samples=1000)
mcmc.run(rng_key_, hparams, y)  # we need to specify the observations here

mcmc.print_summary()
samples = mcmc.get_samples()
```

    sample: 100%|██████████| 1100/1100 [00:02<00:00, 386.90it/s, 3 steps of size 1.58e+00. acc. prob=0.83]

                    mean       std    median      5.0%     95.0%     n_eff     r_hat
         theta      9.09      0.60      9.09      8.11     10.07    350.32      1.00

    Number of divergences: 0

## Stochastic variational inference

See [the documentation](https://num.pyro.ai/en/stable/svi.html)

```
# the guide must have the same signature as the model
def guide(hparams, y):
    prior_mean, prior_sd, obs_sd = hparams
    m = numpyro.param("m", y)  # location
    s = numpyro.param("s", prior_sd, constraint=constraints.positive)  # scale
    return numpyro.sample("theta", dist.Normal(m, s))

# The optimizer wrap these, so have unusual keywords
# https://jax.readthedocs.io/en/latest/jax.experimental.optimizers.html

# optimizer = numpyro.optim.Adam(step_size=0.001)
optimizer = numpyro.optim.Momentum(step_size=0.001, mass=0.1)
# svi = SVI(model, guide, optimizer, Trace_ELBO(), hparams=hparams, y=y) # specify static args to model/guide
svi = SVI(model, guide, optimizer, loss=Trace_ELBO())
nsteps = 2000
svi_result = svi.run(rng_key_, nsteps, hparams, y)  # or specify arguments here

print(svi_result.params)
print(svi_result.losses.shape)
plt.plot(svi_result.losses)
plt.title("ELBO")
plt.xlabel("step")
plt.ylabel("loss");
```

```
print([svi_result.params["m"], svi_result.params["s"]])
```

    [DeviceArray(9.084, dtype=float32), DeviceArray(0.586, dtype=float32)]

## Laplace (quadratic) approximation

See [the documentation](https://num.pyro.ai/en/stable/autoguide.html#autolaplaceapproximation)

```
guide_laplace = AutoLaplaceApproximation(model)
svi = SVI(model, guide_laplace, optimizer, Trace_ELBO(), hparams=hparams, y=y)
svi_run = svi.run(rng_key_, 2000)
params = svi_run.params
losses = svi_result.losses

plt.figure()
plt.plot(losses)
```

```
# Posterior is an MVN
# https://num.pyro.ai/en/stable/distributions.html#multivariatenormal
post = guide_laplace.get_posterior(params)
print(post)
m = post.mean
s = jnp.sqrt(post.covariance_matrix)
print([m, s])
```

    <numpyro.distributions.continuous.MultivariateNormal object at 0x7f4af21b00d0>
    [DeviceArray([9.118], dtype=float32), DeviceArray([[0.6]], dtype=float32)]

```
samples = guide_laplace.sample_posterior(rng_key_, params, (1000,))
print_summary(samples, 0.89, False)
```

                    mean       std    median      5.5%     94.5%     n_eff     r_hat
         theta      9.16      0.61      9.12      8.18     10.04    955.50      1.00

# Example: Beta-Bernoulli model

Example is from [SVI tutorial](https://pyro.ai/examples/svi_part_i.html)

The model is
$$
\begin{align}
\theta &\sim \text{Beta}(\alpha, \beta) \\
x_i &\sim \text{Ber}(\theta)
\end{align}
$$
where $\alpha=\beta=10$. In the code, $\theta$ is called
 `latent_fairness`.

```
alpha0 = 10.0
beta0 = 10.0

def model(data):
    f = numpyro.sample("latent_fairness", dist.Beta(alpha0, beta0))
    # loop over the observed data
    for i in range(len(data)):
        numpyro.sample("obs_{}".format(i), dist.Bernoulli(f), obs=data[i])
```

```
# create some data with 6 observed heads and 4 observed tails
data = jnp.hstack((jnp.ones(6), jnp.zeros(4)))
print(data)
N1 = jnp.sum(data == 1)
N0 = jnp.sum(data == 0)
print([N1, N0])
```

    [1. 1. 1. 1. 1. 1. 0. 0. 0. 0.]
    [DeviceArray(6, dtype=int32), DeviceArray(4, dtype=int32)]

## Exact inference

The posterior is given by
$$
\begin{align}
\theta &\sim \text{Ber}(\alpha + N_1, \beta  + N_0) \\
N_1 &= \sum_{i=1}^N [x_i=1] \\
N_0 &= \sum_{i=1}^N [x_i=0]
\end{align}
$$

```
alpha_q = alpha0 + N1
beta_q = beta0 + N0
print("exact posterior: alpha={:0.3f}, beta={:0.3f}".format(alpha_q, beta_q))

post_mean = alpha_q / (alpha_q + beta_q)
post_var = (post_mean * beta_q) / ((alpha_q + beta_q) * (alpha_q + beta_q + 1))
post_std = np.sqrt(post_var)
print([post_mean, post_std])
```

    exact posterior: alpha=16.000, beta=14.000
    [DeviceArray(0.533, dtype=float32), 0.08960287]

```
inferred_mean = alpha_q / (alpha_q + beta_q)
# compute inferred standard deviation
factor = beta_q / (alpha_q * (1.0 + alpha_q + beta_q))
inferred_std = inferred_mean * math.sqrt(factor)
print([inferred_mean, inferred_std])
```

    [DeviceArray(0.533, dtype=float32), DeviceArray(0.09, dtype=float32)]

## Variational inference

```
def guide(data):
    alpha_q = numpyro.param("alpha_q", alpha0, constraint=constraints.positive)
    beta_q = numpyro.param("beta_q", beta0, constraint=constraints.positive)
    numpyro.sample("latent_fairness", dist.Beta(alpha_q, beta_q))
```

```
# optimizer = numpyro.optim.Adam(step_size=0.001)
optimizer = numpyro.optim.Momentum(step_size=0.001, mass=0.1)
svi = SVI(model, guide, optimizer, loss=Trace_ELBO())
nsteps = 2000
svi_result = svi.run(rng_key_, nsteps, data)

print(svi_result.params)
print(svi_result.losses.shape)
plt.plot(svi_result.losses)
plt.title("ELBO")
plt.xlabel("step")
plt.ylabel("loss");
```

```
# grab the learned variational parameters
alpha_q = svi_result.params["alpha_q"]
beta_q = svi_result.params["beta_q"]
print("variational posterior: alpha={:0.3f}, beta={:0.3f}".format(alpha_q, beta_q))

post_mean = alpha_q / (alpha_q + beta_q)
post_var = (post_mean * beta_q) / ((alpha_q + beta_q) * (alpha_q + beta_q + 1))
post_std = np.sqrt(post_var)
print([post_mean, post_std])
```

    variational posterior: alpha=13.071, beta=11.132
    [DeviceArray(0.54, dtype=float32), 0.09927754]

## MCMC

```
nuts_kernel = NUTS(model)  # this is the unconditioned model
mcmc = MCMC(nuts_kernel, num_warmup=100, num_samples=1000)
mcmc.run(rng_key_, data)

mcmc.print_summary()
samples = mcmc.get_samples()
```

    sample: 100%|██████████| 1100/1100 [00:05<00:00, 207.57it/s, 3 steps of size 1.16e+00. acc. prob=0.88]

                           mean       std    median      5.0%     95.0%     n_eff     r_hat
      latent_fairness      0.53      0.09      0.53      0.37      0.68    380.27      1.00

    Number of divergences: 0

# Distributions

## 1d Gaussian

```
# 2 independent 1d gaussians (ie 1 diagonal Gaussian)
mu = 1.5
sigma = 2
d = dist.Normal(mu, sigma)
dir(d)
```

    ['__call__',
     '__class__',
     '__delattr__',
     '__dict__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__le__',
     '__lt__',
     '__module__',
     '__ne__',
     '__new__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__setattr__',
     '__sizeof__',
     '__str__',
     '__subclasshook__',
     '__weakref__',
     '_batch_shape',
     '_event_shape',
     '_validate_args',
     '_validate_sample',
     'arg_constraints',
     'batch_shape',
     'cdf',
     'enumerate_support',
     'event_dim',
     'event_shape',
     'expand',
     'expand_by',
     'has_enumerate_support',
     'has_rsample',
     'icdf',
     'infer_shapes',
     'is_discrete',
     'loc',
     'log_prob',
     'mask',
     'mean',
     'reparametrized_params',
     'rsample',
     'sample',
     'sample_with_intermediates',
     'scale',
     'set_default_validate_args',
     'shape',
     'support',
     'to_event',
     'tree_flatten',
     'tree_unflatten',
     'variance']

```
rng_key, rng_key_ = random.split(rng_key)
nsamples = 1000
ys = d.sample(rng_key_, (nsamples,))
print(ys.shape)
mu_hat = np.mean(ys, 0)
print(mu_hat)
sigma_hat = np.std(ys, 0)
print(sigma_hat)
```

    (1000,)
    1.5070927
    2.0493808

## Multivariate Gaussian

```
mu = np.array([-1, 1])
sigma = np.array([1, 2])
Sigma = np.diag(sigma)
d2 = dist.MultivariateNormal(mu, Sigma)
```

```
# rng_key, rng_key_ = random.split(rng_key)
nsamples = 1000
ys = d2.sample(rng_key_, (nsamples,))
print(ys.shape)
mu_hat = np.mean(ys, 0)
print(mu_hat)
Sigma_hat = np.cov(ys, rowvar=False)  # jax.np.cov not implemented
print(Sigma_hat)
```

    (1000, 2)
    [-1.013  1.009]
    [[ 0.977 -0.005]
     [-0.005  1.972]]

## Shape semantics

[Numpyro](http://num.pyro.ai/en/stable/distributions.html), [Pyro](https://pyro.ai/examples/tensor_shapes.html) and [TFP](https://www.tensorflow.org/probability/examples/Understanding_TensorFlow_Distributions_Shapes)
and [Distrax](https://github.com/deepmind/distrax)
all distinguish between 'event shape' and 'batch shape'.
For a D-dimensional Gaussian, the event shape is (D,), and the batch shape
will be (), meaning we have a single instance of this distribution.
If the covariance is diagonal, we can view this as D independent
1d Gaussians, stored along the batch dimension; this will have event shape () but batch shape (2,).

When we sample from a distribution, we also specify the sample_shape.
Suppose we draw N samples  from a single D-dim diagonal Gaussian,
and N samples from D 1d Gaussians. These samples will have the same shape.
However, the semantics of logprob differs.
We illustrate this below.

```
mu = np.array([-1, 1])
sigma = np.array([1, 2])
Sigma = np.diag(sigma)
d2 = dist.MultivariateNormal(mu, Sigma)
print(f"event shape {d2.event_shape}, batch shape {d2.batch_shape}")

nsamples = 3
ys2 = d2.sample(rng_key_, (nsamples,))
print("samples, shape {}".format(ys2.shape))
print(ys2)

# 2 independent 1d gaussians (same as one 2d diagonal Gaussian)
d3 = dist.Normal(mu, scale=np.sqrt(np.diag(Sigma)))  # scalar Gaussian needs std not variance
print(f"event shape {d3.event_shape}, batch shape {d3.batch_shape}")

ys3 = d3.sample(rng_key_, (nsamples,))
print("samples, shape {}".format(ys3.shape))
print(ys3)

print(np.allclose(ys2, ys3))

y = ys2[0, :]  # 2 numbers
print(d2.log_prob(y))  # log prob of a single 2d distribution on 2d input
print(d3.log_prob(y))  # log prob of two 1d distributions on 2d input
```

    event shape (2,), batch shape ()
    samples, shape (3, 2)
    [[-0.068  0.994]
     [-1.74  -1.018]
     [ 0.06   2.314]]
    event shape (), batch shape (2,)
    samples, shape (3, 2)
    [[-0.068  0.994]
     [-1.74  -1.018]
     [ 0.06   2.314]]
    True
    -2.6185904
    [-1.353 -1.266]

We can turn a set of independent distributions into a single product
distribution using the [Independent class](http://num.pyro.ai/en/stable/distributions.html#independent)

```
d4 = dist.Independent(d3, 1)  # treat the first batch dimension as an event dimensions
# now d4 is just like d2
print(f"event shape {d4.event_shape}, batch shape {d4.batch_shape}")
print(d4.log_prob(y))
```

    event shape (2,), batch shape ()
    -2.6185904

```

```


```python

def generateBinv():
    from sympy import symbols, Rational, Matrix
    from sympy.vector import Vector

    x,y,z = symbols('x y z')
    wholeFunc = Rational(0)
    alphaVec = []
    i = 0
    while i <= 3:
        j = 0
        while j <= 3:
            k = 0
            while k <= 3:
                a = symbols('a_{0}'.format(k+4*(j + 4*i)))
                alphaVec.append(a)
                wholeFunc += a*x**Rational(i)*y**Rational(j)*z**Rational(k)
                k += 1
            j += 1
        i += 1

    #print(wholeFunc)

    #order per corner f,fx,fy,fz,fxy,fxz,fyz,fxyz
    cornerVec = Matrix([wholeFunc,
                 wholeFunc.diff(x),
                 wholeFunc.diff(y),
                 wholeFunc.diff(z),
                 wholeFunc.diff(x,y),
                 wholeFunc.diff(x,z),
                 wholeFunc.diff(y,z),
                 wholeFunc.diff(x,y,z)])
    #print(cornerVec)
    #build B suchthat b = B.a

    def getBRows(vec,alpha):
        '''get the rows from cornervec evaluated at the specific points'''
        B = []
        for el in vec:
            row = []
            for a in alpha:
                k = Rational(el.diff(a))
                row.append(k)
            B.append(row)
        return Matrix(B)

    #x=0,y=0,z=0
    row0 = getBRows(cornerVec.subs({x:0,y:0,z:0}),alphaVec)
    row1 = getBRows(cornerVec.subs({x:0,y:0,z:1}),alphaVec)
    row2 = getBRows(cornerVec.subs({x:0,y:1,z:0}),alphaVec)
    row3 = getBRows(cornerVec.subs({x:0,y:1,z:1}),alphaVec)
    row4 = getBRows(cornerVec.subs({x:1,y:0,z:0}),alphaVec)
    row5 = getBRows(cornerVec.subs({x:1,y:0,z:1}),alphaVec)
    row6 = getBRows(cornerVec.subs({x:1,y:1,z:0}),alphaVec)
    row7 = getBRows(cornerVec.subs({x:1,y:1,z:1}),alphaVec)
    Binv = Matrix([row0,row1,row2,row3,row4,row5,row6,row7]).inv()
    string = "["
    for i in range(64):
        string += '['+('{:},'*63).format(*Binv[i,:64])+'{:}'.format(Binv[i,-1])+'],\n'
    string += ']'
    print(string)

def optimizeBvecFormation(order=1):
    from sympy import symbols, Matrix
    from sympy import cse, Function, Rational
    from scipy.special import binom

    def centralDiff(order, inFunc,var):
        #args = inFunc.args
        #func = inFunc.func
        #terms = []
        x,y,z,h = symbols('x y z h')
        outFunc = 0
        k = 0
        while k <= order:
            if var == 'x':
                outFunc += Rational(-1)**Rational(k) * Rational(int(binom(order,k))) * inFunc.subs({x:x + (Rational(order,2) - Rational(k))*h})
            if var == 'y':
                outFunc += Rational(-1)**Rational(k) * Rational(int(binom(order,k))) * inFunc.subs({y:y + (Rational(order,2) - Rational(k))*h})
            if var == 'z':
                outFunc += Rational(-1)**Rational(k) * Rational(int(binom(order,k))) * inFunc.subs({z:z + (Rational(order,2) - Rational(k))*h})
            k += 1
        return outFunc

    def centralDiff(order, inFunc,var):
        #args = inFunc.args
        #func = inFunc.func
        #terms = []
        x,y,z = symbols('xi yi zi', intergers=True)
        h = symbols('h')
        #print(var)
        if var == 'x':
            var = x
        if var == 'y':
            var = y
        if var == 'z':
            var = z
        outFunc = Rational(8)*(inFunc.subs({var:(var + h)}) - inFunc.subs({var:(var - h)}))/Rational(12)/h + (inFunc.subs({var:(var - Rational(2)*h)}) - inFunc.subs({var:(var + Rational(2)*h)}))/Rational(12)/h
        #print(inFunc)
        #print(Rational(8)*inFunc.subs({var:(var + h)})/Rational(12)/h)
        #print(-Rational(8)*inFunc.subs({var:(var - h)})/Rational(12)/h)
        #print(inFunc.subs({var:(var - Rational(2)*h)})/Rational(12)/h)
        #print(-inFunc.subs({var:(var + Rational(2)*h)})/Rational(12)/h)
        #outFunc = (inFunc.subs({var:var + h}) - inFunc.subs({var:var}))/h
        return outFunc

    vec = []

    xi,yi,zi,nz,ny = symbols('xi yi zi nz ny', intergers=True)
    func = Function('f')
    xf,yf,zf = Function('xf'),Function('yf'),Function('zf')
    vec = []
    #order
    #000,001,010,011,100,101,110,111
    i = 0
    while i <= 1:
        j = 0
        while j <= 1:
            k = 0
            while k <= 1:

                dx = (xf(xi+Rational(i+1)) - xf(xi+Rational(i-1)))/Rational(2)
                dy = (yf(yi+Rational(j+1)) - yf(yi+Rational(j-1)))/Rational(2)
                dz = (zf(zi+Rational(k+1)) - zf(zi+Rational(k-1)))/Rational(2)
                #f
                f = func((zi+Rational(k)) + nz*((yi+Rational(j)) + ny*(xi+Rational(i))))#x+Rational(i),y+Rational(j),z+Rational(k))
                vec.append(f)

                #fx
                fx = centralDiff(order,f,'x')/dx
                vec.append(fx)

                #fy
                fy = centralDiff(order,f,'y')/dy
                vec.append(fy)

                #fz
                fz = centralDiff(order,f,'z')/dz
                vec.append(fz)

                #fxy
                fxy = centralDiff(order,fx,'y')/dy
                vec.append(fxy)

                #fxz
                fxz = centralDiff(order,fx,'z')/dz
                vec.append(fxz)

                #fyz
                fyz = centralDiff(order,fy,'z')/dz
                vec.append(fyz)

                #fxyz
                fxyz = centralDiff(order,fxy,'z')/dz
                vec.append(fxyz)

                k += 1
            j += 1
        i += 1
    vec = Matrix(vec).subs({'h':1})
    #print(vec)
    cseFunc = cse(vec,optimizations='basic')
    lines = []
    for optLine in cseFunc[0]:
        line = "{0} = {1}".format(optLine[0],optLine[1].evalf())
        line = line.replace("xf","self.get_xvec").replace("yf","self.get_yvec").replace("zf","self.get_zvec").replace("f(","self.get_m(")
        line = line.replace("xi","i").replace("yi","j").replace("zi","k")
        #line = line.replace("(","[").replace(")","]")
        lines.append(line)
        print(line)

    #print(lines)
    out = "{0}".format(cseFunc[1][0].transpose()[0,:].evalf())
    out = out.replace("xf","self.get_xvec").replace("yf","self.get_yvec").replace("zf","self.get_zvec").replace("f(","self.get_m(")
    out = out.replace("xi","i").replace("yi","j").replace("zi","k")
    print(out)

    def index1(i):
        str = ''
        if i == -1:
            str += 'm'
        if i == 0:
            str += 'z'
        if i == 1:
            str += 'p'
        if i == 2:
            str += 'P'
        return str

    def index2(i):
        str = ''
        if i == -1:
            str += 'm'
        if i == 0:
            str += 'z'
        if i == 1:
            str += 'p'
        if i == 2:
            str += 'P'
        return str

    def index(i,j,k):
        str = ''
        str += index1(i) + index1(j) + index1(k)
        return str

    vec = []

    i = 0
    while i <= 1:
        j = 0
        while j <= 1:
            k = 0
            while k <= 1:
                #f
                vec.append(symbols('f_{0}'.format(index(i,j,k))))

                #x10 = symbols('x_{0}'.format(index1(i))) - symbols('x_{0}'.format(index1(i-1)))
                #x02 = symbols('x_{0}'.format(index1(i+1))) - symbols('x_{0}'.format(index1(i)))
                #y10 = symbols('y_{0}'.format(index1(j))) - symbols('y_{0}'.format(index1(j-1)))
                #y02 = symbols('y_{0}'.format(index1(j+1))) - symbols('y_{0}'.format(index1(j)))
                #z10 = symbols('z_{0}'.format(index1(k))) - symbols('z_{0}'.format(index1(k-1)))
                #z02 = symbols('z_{0}'.format(index1(k+1))) - symbols('z_{0}'.format(index1(k)))

                x12 = symbols('x_{0}'.format(index1(i+1))) - symbols('x_{0}'.format(index1(i-1)))
                y12 = symbols('y_{0}'.format(index1(j+1))) - symbols('y_{0}'.format(index1(j-1)))
                z12 = symbols('z_{0}'.format(index1(k+1))) - symbols('z_{0}'.format(index1(k-1)))

                #fx,fy,fz
                #f0 = symbols('f_{0}'.format(index(i,j,k)))
                #fmzz = symbols('f_{0}'.format(index(i-1,j,k)))
                #fpzz = symbols('f_{0}'.format(index(i+1,j,k)))
                #vec.append(((fmzz - f0)*x02 - (fpzz - f0)*x10)/(2*x10*x02))

                #fzmz = symbols('f_{0}'.format(index(i,j-1,k)))
                #fzpz = symbols('f_{0}'.format(index(i,j+1,k)))
                #vec.append(((fzmz - f0)*y02 - (fzpz - f0)*y10)/(2*y10*y02))

                #fzzm = symbols('f_{0}'.format(index(i,j,k-1)))
                #fzzp = symbols('f_{0}'.format(index(i,j,k+1)))
                #vec.append(((fzzm - f0)*z02 - (fzzp - f0)*z10)/(2*z10*z02))

                vec.append((symbols('f_{0}'.format(index(i+1,j,k))) - symbols('f_{0}'.format(index(i-1,j,k))) )/x12)
                vec.append((symbols('f_{0}'.format(index(i,j+1,k))) - symbols('f_{0}'.format(index(i,j-1,k))) )/y12)
                vec.append((symbols('f_{0}'.format(index(i,j,k+1))) - symbols('f_{0}'.format(index(i,j,k-1))) )/z12)

                #fxy,fxz,fyz
                vec.append((((symbols('f_{0}'.format(index(i+1,j+1,k))) - symbols('f_{0}'.format(index(i-1,j+1,k))) )/x12)-((symbols('f_{0}'.format(index(i+1,j-1,k))) - symbols('f_{0}'.format(index(i-1,j-1,k))) )/x12))/y12)
                vec.append((((symbols('f_{0}'.format(index(i+1,j,k+1))) - symbols('f_{0}'.format(index(i-1,j,k+1))) )/x12)-((symbols('f_{0}'.format(index(i+1,j,k-1))) - symbols('f_{0}'.format(index(i-1,j,k-1))) )/x12))/z12)
                vec.append((((symbols('f_{0}'.format(index(i,j+1,k+1))) - symbols('f_{0}'.format(index(i,j-1,k+1))) )/y12)-((symbols('f_{0}'.format(index(i,j+1,k-1))) - symbols('f_{0}'.format(index(i,j-1,k-1))) )/y12))/z12)

                #fxyz
                vec.append((((((symbols('f_{0}'.format(index(i+1,j+1,k+1))) - symbols('f_{0}'.format(index(i-1,j+1,k+1))) )/x12)-((symbols('f_{0}'.format(index(i+1,j-1,k+1))) - symbols('f_{0}'.format(index(i-1,j-1,k+1))) )/x12))/y12)-((((symbols('f_{0}'.format(index(i+1,j+1,k-1))) - symbols('f_{0}'.format(index(i-1,j+1,k-1))) )/x12)-((symbols('f_{0}'.format(index(i+1,j-1,k-1))) - symbols('f_{0}'.format(index(i-1,j-1,k-1))) )/x12))/y12))/z12)

                k += 1
            j += 1
        i += 1
    vec = Matrix(vec)

    cseFunc = cse(vec,optimizations='basic')
    #generate the indices
    lines = ['im = i - 1','iz = i','ip = i + 1','iP = i + 2',
            'jm = j - 1','jz = j','jp = j + 1','jP = j + 2',
            'km = k - 1','kz = k','kp = k + 1','kP = k + 2']
    i = -1
    while i <= 2:
        j = -1
        while j <= 2:
            k = -1
            while k <= 2:
                var = index(i,j,k)
                line = "{0} = self.index(i{1},j{2},k{3})".format(index(i,j,k),index2(i),index2(j),index2(k))
                lines.append(line)
                k += 1
            j += 1
        i += 1
    def replaceIndices(f):
        i = -1
        while i <= 2:
            j = -1
            while j <= 2:
                k = -1
                while k <= 2:
                    var = index(i,j,k)
                    f = f.replace(var,'[{0}]'.format(var))
                    k += 1
                j += 1
            i += 1
        return f
    def replaceIndices2(f):
        f = f.replace('x_m','self.xvec[im]')
        f = f.replace('x_z','self.xvec[iz]')
        f = f.replace('x_p','self.xvec[ip]')
        f = f.replace('x_P','self.xvec[iP]')
        f = f.replace('y_m','self.yvec[jm]')
        f = f.replace('y_z','self.yvec[jz]')
        f = f.replace('y_p','self.yvec[jp]')
        f = f.replace('y_P','self.yvec[jP]')
        f = f.replace('z_m','self.zvec[km]')
        f = f.replace('z_z','self.zvec[kz]')
        f = f.replace('z_p','self.zvec[kp]')
        f = f.replace('z_P','self.zvec[kP]')
        return f

    for expr in cseFunc[0]:
        f = str(expr[1])
        f = replaceIndices(f)
        f = replaceIndices2(f)
        f = f.replace('f_','self.m')
        line = '{0} = {1}'.format(expr[0], f)
        lines.append(line)

    bvec = str(cseFunc[1][0].transpose())
    bvec = bvec.replace('Matrix([','bvec = np.array(')
    bvec = bvec.replace('])',')')
    bvec = bvec.replace(',',',\n')
    bvec = replaceIndices(bvec)
    bvec = replaceIndices2(bvec)
    bvec = bvec.replace('f_','self.m')
    lines.append(bvec)
    code = ''
    for line in lines:
        code += line+'\n'
    print(code)

if __name__=='__main__':
    genFuncCalls()
    #generateBinv()
    #optimizeBvecFormation()
    #testResult()
```

```python
from sympy import *

x,y,z = symbols('x y z')
f = Function('f')
f1 = f(x+1,y,z)
f2 = f(f1.args[0] +1,y,z) + f1
e = f1.func
h = f1 - f(x-3,y,z)
g = Function('g')
g = f1 + f(x,y,z)
```

```python
g.subs({'x':x+6})
```

    f(x + 6, y, z) + f(x + 7, y, z)

```python
a,o = symbols('a o')
from sympy.solvers import solve
ne = a*exp(-x**2/(o*log(2)*Rational(7,2))**2)
c = cse(ne.diff(x,x,x,x,x).subs({o:10,a:1e13}))
print(c)
for k in c[0]:
    print("{0} = {1}".format(k[0],k[1]))
```

    ([(x0, log(2)), (x1, x**2/x0**2)], [18.1254818541641*x*(-4*x**4/(625*x0**4) + 196*x1/5 - 36015)*exp(-x1/1225)/x0**6])
    x0 = log(2)
    x1 = x**2/x0**2

```python
import numpy as np
f=lambdify(x,ne.diff(x,x,x,x,x).subs({o:10,a:1e13}),"numpy")
import pylab as plt
plt.plot(np.linspace(0,100,1000),f(np.linspace(0,100,1000)))
plt.show()
ne.diff(x,x,x,x,x).subs({x:o})
```

    -0.387942463307711*a/o**5

```python
def genFuncCalls():
    from sympy import symbols, Rational, Matrix,Function
    from sympy.vector import Vector
    from sympy import cse

    x,y,z = symbols('x y z')
    a = Function('a')
    wholeFunc = []
    alphaVec = []
    i = 0
    while i <= 3:
        j = 0
        while j <= 3:
            k = 0
            while k <= 3:
                afunc = a(Rational(k)+Rational(4)*(Rational(j) + Rational(4)*Rational(i)))
                alphaVec.append(afunc)
                wholeFunc.append(x**Rational(i)*y**Rational(j)*z**Rational(k))
                k += 1
            j += 1
        i += 1

    alphaVec = Matrix(alphaVec)
    wholeFunc = Matrix(wholeFunc)
    cornerVec = Matrix([wholeFunc.transpose()[:],
                 wholeFunc.diff(x).transpose()[:],
                 wholeFunc.diff(y).transpose()[:],
                 wholeFunc.diff(z).transpose()[:],
                 wholeFunc.diff(x,y).transpose()[:],
                 wholeFunc.diff(x,z).transpose()[:],
                 wholeFunc.diff(y,z).transpose()[:],
                 wholeFunc.diff(x,y,z).transpose()[:]])

    cseFunc = cse(cornerVec,optimizations='basic')
    #print(cornerVec.shape,alphaVec.shape)
    #print(Matrix(cornerVec.dot(alphaVec)))

    #print (cseFunc)
    lines = []
    for optLine in cseFunc[0]:
        line = "{0} = {1}".format(optLine[0],optLine[1])
        #line = line.replace("xf","self.get_xvec").replace("yf","self.get_yvec").replace("zf","self.get_zvec").replace("f(","self.get_m(")
        #line = line.replace("xi","i").replace("yi","j").replace("zi","k")
        #line = line.replace("(","[").replace(")","]")
        lines.append(line)
        print(line)

    #print(lines)
    out = "np.array({0},dtype=np.double)".format(cseFunc[1][0])
    #out = out.replace("xf","self.get_xvec").replace("yf","self.get_yvec").replace("zf","self.get_zvec").replace("f(","self.get_m(")
    #out = out.replace("xi","i").replace("yi","j").replace("zi","k")
    print(out)

genFuncCalls()
```

    x0 = z**2
    x1 = z**3
    x2 = y*z
    x3 = x0*y
    x4 = x1*y
    x5 = y**2
    x6 = x5*z
    x7 = x0*x5
    x8 = x1*x5
    x9 = y**3
    x10 = x9*z
    x11 = x0*x9
    x12 = x1*x9
    x13 = x*z
    x14 = x*x0
    x15 = x*x1
    x16 = x*y
    x17 = x16*z
    x18 = x0*x16
    x19 = x*x5
    x20 = x13*x5
    x21 = x0*x19
    x22 = x*x9
    x23 = x0*x22
    x24 = x**2
    x25 = x24*z
    x26 = x0*x24
    x27 = x1*x24
    x28 = x24*y
    x29 = x2*x24
    x30 = x0*x28
    x31 = x24*x5
    x32 = x25*x5
    x33 = x0*x31
    x34 = x24*x9
    x35 = x0*x34
    x36 = x**3
    x37 = x36*z
    x38 = x0*x36
    x39 = x1*x36
    x40 = x36*y
    x41 = x0*x40
    x42 = x36*x5
    x43 = x37*x5
    x44 = x0*x42
    x45 = x36*x9
    x46 = 2*x
    x47 = x46*z
    x48 = x0*x46
    x49 = x1*x46
    x50 = x46*y
    x51 = x2*x46
    x52 = x0*x50
    x53 = x1*x50
    x54 = x46*x5
    x55 = x47*x5
    x56 = x46*x9
    x57 = x47*x9
    x58 = 3*x24
    x59 = x58*z
    x60 = x0*x58
    x61 = x1*x58
    x62 = x58*y
    x63 = x3*x58
    x64 = x5*x58
    x65 = x58*x6
    x66 = x58*x7
    x67 = x58*x8
    x68 = x58*x9
    x69 = x11*x58
    x70 = 2*y
    x71 = x70*z
    x72 = x0*x70
    x73 = x1*x70
    x74 = 3*x5
    x75 = x74*z
    x76 = x0*x74
    x77 = x1*x74
    x78 = x*x74
    x79 = x14*x74
    x80 = x24*x70
    x81 = x24*x71
    x82 = x36*x70
    x83 = x36*x71
    x84 = x36*x74
    x85 = x38*x74
    x86 = 2*z
    x87 = 3*x0
    x88 = x87*y
    x89 = x5*x86
    x90 = x86*x9
    x91 = x87*x9
    x92 = x*x87
    x93 = x24*x86
    x94 = x36*x86
    x95 = x36*x87
    x96 = 4*x16
    x97 = x96*z
    x98 = 6*x19
    x99 = 6*x20
    x100 = x0*x98
    x101 = 6*x28
    x102 = 6*x29
    x103 = x0*x101
    x104 = 9*x31
    x105 = x0*x104
    x106 = 4*x13
    x107 = 6*x14
    x108 = 6*x18
    x109 = 6*x25
    x110 = 9*x26
    x111 = x109*x5
    x112 = 4*x2
    x113 = 6*x3
    x114 = 6*x6
    x115 = 9*x7
    np.array(Matrix([[1, z, x0, x1, y, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x, x13, x14, x15, x16, x17, x18, x1*x16, x19, x20, x21, x1*x19, x22, x13*x9, x23, x1*x22, x24, x25, x26, x27, x28, x29, x30, x1*x28, x31, x32, x33, x1*x31, x34, x25*x9, x35, x1*x34, x36, x37, x38, x39, x40, x2*x36, x41, x1*x40, x42, x43, x44, x1*x42, x45, x37*x9, x0*x45, x1*x45], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, z, x0, x1, y, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x46, x47, x48, x49, x50, x51, x52, x53, x54, x55, x0*x54, x1*x54, x56, x57, x0*x56, x1*x56, x58, x59, x60, x61, x62, x2*x58, x63, x4*x58, x64, x65, x66, x67, x68, x10*x58, x69, x12*x58], [0, 0, 0, 0, 1, z, x0, x1, x70, x71, x72, x73, x74, x75, x76, x77, 0, 0, 0, 0, x, x13, x14, x15, x50, x51, x52, x53, x78, x13*x74, x79, x15*x74, 0, 0, 0, 0, x24, x25, x26, x27, x80, x81, x0*x80, x1*x80, x64, x65, x66, x67, 0, 0, 0, 0, x36, x37, x38, x39, x82, x83, x0*x82, x1*x82, x84, x37*x74, x85, x39*x74], [0, 1, x86, x87, 0, y, x71, x88, 0, x5, x89, x76, 0, x9, x90, x91, 0, x, x47, x92, 0, x16, x51, x16*x87, 0, x19, x55, x79, 0, x22, x57, x22*x87, 0, x24, x93, x60, 0, x28, x81, x63, 0, x31, x5*x93, x66, 0, x34, x9*x93, x69, 0, x36, x94, x95, 0, x40, x83, x40*x87, 0, x42, x5*x94, x85, 0, x45, x9*x94, x45*x87], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, z, x0, x1, x70, x71, x72, x73, x74, x75, x76, x77, 0, 0, 0, 0, x46, x47, x48, x49, x96, x97, x0*x96, x1*x96, x98, x99, x100, x1*x98, 0, 0, 0, 0, x58, x59, x60, x61, x101, x102, x103, x1*x101, x104, 9*x32, x105, x1*x104], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, x86, x87, 0, y, x71, x88, 0, x5, x89, x76, 0, x9, x90, x91, 0, x46, x106, x107, 0, x50, x97, x108, 0, x54, x106*x5, x100, 0, x56, x106*x9, 6*x23, 0, x58, x109, x110, 0, x62, x102, 9*x30, 0, x64, x111, x105, 0, x68, x109*x9, 9*x35], [0, 0, 0, 0, 0, 1, x86, x87, 0, x70, x112, x113, 0, x74, x114, x115, 0, 0, 0, 0, 0, x, x47, x92, 0, x50, x97, x108, 0, x78, x99, 9*x21, 0, 0, 0, 0, 0, x24, x93, x60, 0, x80, x112*x24, x103, 0, x64, x111, x105, 0, 0, 0, 0, 0, x36, x94, x95, 0, x82, x112*x36, 6*x41, 0, x84, 6*x43, 9*x44], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, x86, x87, 0, x70, x112, x113, 0, x74, x114, x115, 0, 0, 0, 0, 0, x46, x106, x107, 0, x96, 8*x17, 12*x18, 0, x98, 12*x20, 18*x21, 0, 0, 0, 0, 0, x58, x109, x110, 0, x101, 12*x29, 18*x30, 0, x104, 18*x32, 27*x33]]),dtype=np.double)

```python

```


# scipy.integrat.solve_ivp
> # $ \frac{dx}{dt} = cos(y) \\
\frac{dy}{dt} = sin(x)\\
\text{where, }\begin{cases}y(0) & = 1\\
y'(0) & = 1
\end{cases}$

```python
import scipy as sci
import scipy.integrate
import numpy as np
import matplotlib.pyplot as plt
%matplotlib widget

def y(t,picard):
    x,y= picard
    fx = np.cos(y)
    fy = np.sin(x)
    return fx,fy
#sol = sci.integrate.solve_ivp(y,(0,10),(1,1))

sol = sci.integrate.solve_ivp(y,(0,10),(1,1), t_eval = np.linspace(0,10,100))
x,y = sol.y

fig = plt.figure()
ax = fig.add_subplot(2,1,1)
ax.plot(x,y)
ax.axis('scaled')
```

    (-1.2972498421129277,
     1.2970996248686584,
     0.2748287351423644,
     2.8662064155283975)

        Figure

```python
t = sol.t
ax = fig.add_subplot(212)
ax.plot(t,x)
ax.plot(t,y)
```

    [<matplotlib.lines.Line2D at 0x7fc1cdb26130>]

```python
def odefun(t,state):
    x,y,z = state
    return 10*(y-x), x*(28-z)-y, x*y-8/3*z
def sol():
    tspan = [0,30]
    state0 = [-0.1, 0.5, 1]
    state = sci.integrate.solve_ivp(odefun,tspan,state0)
    return state.t, state.y[0,:], state.y[1,:], state.y[2,:]
def plotfunc(t,x,y,z):
    fig = plt.figure()
    ax = fig.add_subplot(221)
    ax.plot(t,x)
    ax = fig.add_subplot(222)
    ax.plot(t,y)
    ax = fig.add_subplot(223)
    ax.plot(t,z)
    ax = fig.add_subplot(224)
    ax.plot(x,z)
```

```python
t,x,y,z = sol()
plotfunc(t,x,y,z)
```

        Figure

# [a mass on a spring](https://ericlab.tistory.com/71)

> $ x \Rightarrow \text{ function of length(distance) of spring}$
> $ x(t) \Rightarrow \text{ fucntion of time variavle length of spring}$
>> $Fk = -kx $
>> $Fg = ma $
>>> $ ma = -kx $
>>> $ \because ma = m \ddot{x} $
>>> $ \therefore m \ddot{x} = -kx$
> ## $ \therefore m \ddot{x} + kx = 0 $
> ## $ m \frac{d^2}{dt^2}x + kx = 0 $

```python
import sympy as sym

t,m,k = sym.symbols('t m k')
x = sym.symbols('x', cls=sym.Function)
f = sym.dsolve( m*x(t).diff(t,2) + k*x(t), x(t))
f
```

$\displaystyle x{\left(t \right)} = C_{1} e^{- t \sqrt{- \frac{k}{m}}} + C_{2} e^{t \sqrt{- \frac{k}{m}}}$

# damping
> damping
>> 물체의 속도( $\vec{v}$ )에 비례해서 운동을 방해하는 힘.
>> 일종의 피스톤 낙하산 날개..
>> $ Fd  = -d\vec{v} = -c\dot{x}$
> ## $\therefore Fd = -c \dot{x} \begin{cases}
m \ddot{x} &= -kx + -c\dot{x} \\
m \ddot{x} + c\dot{x} + kx & = 0 \\
\ddot{x} + \frac{c \dot{x}}{m} + \frac{kx}{m} & = 0 \\
\frac{d^2x}{dt^2} + \frac{c}{m}\frac{dx}{dt} + \frac{k}{m}x & = 0 \\
x'' + \frac{c}{m}x' + \frac{k}{m}x & = 0 \\
x(t)'' + \frac{c}{m}x(t)' + \frac{k}{m}x(t) & = 0 \\
\end{cases}$

```python
c = sym.symbols('c')
sym.dsolve(x(t).diff(t,2) + c/m * x(t).diff(t,1) + k/m*x(t))
```

$\displaystyle x{\left(t \right)} = C_{1} e^{\frac{t \left(- c - \sqrt{c^{2} - 4 k m}\right)}{2 m}} + C_{2} e^{\frac{t \left(- c + \sqrt{c^{2} - 4 k m}\right)}{2 m}}$

# ODEs
> ## $ \frac{dy}{dt} = f(t,y)
\begin{cases}
y & \text{ is a vector} \\
y(0) & = y_{0}
\end{cases} $
# example
> ## $ \frac{dC}{dt} = \frac {-C}{\tau} \begin{cases}
\tau & = 1 \\
C(0) & = 1 \\
t_{end} & = 5
\end{cases}
\quad \because C' + \frac{C}{\tau} = 0
$
> ## $ C(t) = C_1 e^{\frac{t}{\tau}} \begin{cases}
C(0) = 1 \\
\tau = 1 \\
\end{cases}
\quad \because C(t) = e^t
$

```python
t, tau = sym.symbols('t tau')
C = sym.symbols('C',cls=sym.Function)(t)
y = sym.dsolve(C.diff(t) + C/tau,C)
C1, _, _ = y.free_symbols
f = y.subs(C1,1).subs(tau,1)
yn = [sym.N(f.subs(t,i)) for i in np.arange(0,6)]
f
```

$\displaystyle C{\left(t \right)} = e^{- t}$

```python
def fA(t,C):
    tau = 1.0
    return -C/tau
t_span = np.array([0,5])
print(t_span)
C0 = np.array([1.0])
soln = sci.integrate.solve_ivp(fA, [0,9], C0, t_eval=[0,1,2,3,4,5,9])
t_exact = np.linspace(t_span[0],t_span[1],101)
C_exact = C0 * np.exp(-t_exact/1.0)
C_exact = C0 * np.exp(-t_exact/1.0)
print(C)
fig = plt.figure()
ax = fig.add_subplot()
ax.plot(t_exact, C_exact, '-', label='exact')
ax.plot(soln.t,soln.y[0],'o',label='solve_lvp')
#ax.plot(soln.t,soln.y[1],'x',label='y[1]')
ax.set_xlabel('time')
ax.set_ylabel('C')
ax.legend()

```

    [0 5]
    C(t)

    <matplotlib.legend.Legend at 0x7fc1c581f6a0>

        Figure

```python

```


```python
import sympy as sp
import numpy as np

sp.init_printing()
```

```python
xi, L, A, E = sp.symbols("xi L A E")
Nue = sp.Matrix([(1-xi)/2, (1+xi)/2])
Nue
```

```python
Bue=sp.diff(Nue,xi)*2/L
Bue
```

```python
KeTRUSS1D = sp.lambdify((A,E,L), A*E*sp.integrate(Bue*(Bue.T),(xi,-1,1))*L/2)
print(KeTRUSS1D(A,E,L))
```

    [[A*E/L -A*E/L]
     [-A*E/L A*E/L]]

Adatok megadása

```python
A1=60
A2=20
A3=30

E1=100e3
E2=200e3
E3=50e3

L1=1e3
L2=2e3
L3=3e3

FT=-15e3
```

Elem-csomópont összerendelések tárolása az `en` mátrixban

```python
en=np.array([
    [1,2],
    [2,3],
    [2,4]]) - 1
```

Elemi merevségi mátrixok

```python
Ke1 = KeTRUSS1D(A1,E1,L1)
Ke2 = KeTRUSS1D(A2,E2,L2)
Ke3 = KeTRUSS1D(A3,E3,L3)
```

Globális merevségi mátrix megadása során első lépésben egy zérus elemekkel
kitöltött mátrixot hozunk létre, majd a megfelelő helyekre betesszül az
egyes elemi merevségi mátrixainak elemeit.

```python
KG=np.zeros((4,4))
KG
```

    array([[ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.]])

Egy lehetséges leprogramozása ennek az alábbiakban látható,
ahol felhasználjuk az elem-csomópont összerendlés mátrixot:

[np.ix_](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ix_.html)

Az 1-es elem merevségi mátrixának elhelyezése
a globális merevségi mátrixban:

```python
elemSzam=1
KG[np.ix_(en[elemSzam-1],en[elemSzam-1])] += Ke1
KG
```

    array([[ 6000., -6000.,     0.,     0.],
           [-6000.,  6000.,     0.,     0.],
           [    0.,     0.,     0.,     0.],
           [    0.,     0.,     0.,     0.]])

Az 2-es elem merevségi mátrixának elhelyezése
a globális merevségi mátrixban:

```python
elemSzam=2
KG[np.ix_(en[elemSzam-1],en[elemSzam-1])] += Ke2
KG
```

    array([[ 6000., -6000.,     0.,     0.],
           [-6000.,  8000., -2000.,     0.],
           [    0., -2000.,  2000.,     0.],
           [    0.,     0.,     0.,     0.]])

Az 3-es elem merevségi mátrixának elhelyezése
a globális merevségi mátrixban:

```python
elemSzam=3
KG[np.ix_(en[elemSzam-1],en[elemSzam-1])] += Ke3
```

Tehát a globális merevségi mátrix:

```python
KG
```

    array([[ 6000., -6000.,     0.,     0.],
           [-6000.,  8500., -2000.,  -500.],
           [    0., -2000.,  2000.,     0.],
           [    0.,  -500.,     0.,   500.]])

A globális tehervektor:

```python
FG = np.zeros(4)
FG[1 - 1] += FT
```

A kondenzált merevségi mátrixot és a kondenzált tehervektort megkapjuk a kényszerekkel ellátott szabadságfokokhoz tartozó sorok és oszlopok törlésével:

(megtartjuk a maradék részt)

```python
szabad = [0,1]
```

A kondenzált merevségi mátrixot megkapjuk az **első** és **második** sorok/oszlopok megtartásával:

```python
print(np.ix_(szabad,szabad))
```

    (array([[0],
           [1]]), array([[0, 1]]))

```python
KGkond = KG[np.ix_(szabad,szabad)]
KGkond
```

    array([[ 6000., -6000.],
           [-6000.,  8500.]])

A kondezált tehervektort megkapjuk az **első** és **második** sor megtartásával:

```python
FGkond = FG[np.ix_(szabad)]
FGkond
```

    array([-15000.,      0.])

```python
# a teljes lineáris egyenletrendszer alulhatározott
np.linalg.solve(KG,FG)
```

    array([  1.12589991e+17,   1.12589991e+17,   1.12589991e+17,
             1.12589991e+17])

Megoldás az ismeretlen elmozdulásokra

```python
Umego = np.linalg.solve(KGkond,FGkond)
Umego
```

    array([-8.5, -6. ])

Tehát a teljes globális csomóponti elmozdulásvektor:

```python
UG = np.zeros(4)
UG[np.ix_(szabad)] += Umego
UG
```

    array([-8.5, -6. ,  0. ,  0. ])

Eredmények megjelenítése

[matplotlib.patches](https://matplotlib.org/api/patches_api.html)

```python
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
```

```python
nagy = 100
R = 200

plt.figure()
ax = plt.gca()
ax.set_xticklabels([])
ax.set_yticklabels([])
## eredeti konfiguráció
# rudak
ax.add_patch(mpatches.Rectangle((-R/2,-L1),R,L1,
                                fc = (1,1,1,1),ec = (0.5,0.5,0.5,1), ls = "--", lw = 2))
ax.add_patch(mpatches.Rectangle((-1.5*L1-R/2,-L1-L2),R,L2,
                                fc = (1,1,1,1),ec = (0.5,0.5,0.5,1), ls = "--", lw = 2))
ax.add_patch(mpatches.Rectangle((1.5*L1-R/2,-L1-L3),R,L3,
                                fc = (1,1,1,1),ec = (0.5,0.5,0.5,1), ls = "--", lw = 2))
# csomópontok
ax.add_patch(mpatches.Circle((0,0),R,
                             fc = (1,1,1,1),ec = (0.5,0.5,0.5,1),ls = "--", lw = 2))
ax.add_patch(mpatches.Rectangle((-2*L1,-L1 - R/2),4*L1,R,
                                fc = (1,1,1,1),ec = (0.5,0.5,0.5,1),ls = "--", lw = 2))
# terhelés
ax.add_patch(mpatches.Arrow(0,R+5*R,0,-5*R,
                            fc = (1,0,0,1),ec = (1,0,0,1), lw = 3, width = 2*R))

## elmozdult állapot
# rudak
ax.add_patch(mpatches.Rectangle((-R/2,-L1 + nagy*UG[1]),R,L1 + nagy*(UG[0] - UG[1]),
                                fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))
ax.add_patch(mpatches.Rectangle((-1.5*L1-R/2,-L1-L2 + nagy*UG[2]),R,L2 + nagy*(UG[1] - UG[2]),
                                fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))
ax.add_patch(mpatches.Rectangle((1.5*L1-R/2,-L1-L3  + nagy*UG[3]),R,L3 + nagy*(UG[1] - UG[3]),
                                fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))
# csomópontok
ax.add_patch(mpatches.Circle((0,0+nagy*UG[0]),R,
                             fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))
ax.add_patch(mpatches.Rectangle((-2*L1,-L1 + nagy*UG[1] - R/2),4*L1,R,
                                fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))
ax.add_patch(mpatches.Circle((-1.5*L1,-L1-L2 + nagy*UG[2]),R,
                             fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))
ax.add_patch(mpatches.Circle((1.5*L1,-L1-L3  + nagy*UG[3]),R,
                             fc = (1,1,1,1),ec = (0,0,0,1), lw = 2))

plt.axis("equal")
plt.show()
```

A teljes csomóponti terhelésvektor
$\mathbf{F}_\mathrm{TOT} = \mathbf{F}_\mathrm{REAK} + \mathbf{F}_\mathrm{KÜL}$:

```python
FTOT = np.dot(KG,UG)
FTOT
```

    array([-15000.,      0.,  12000.,   3000.])

```python
FREAK = FTOT - FG
FREAK
```

    array([     0.,      0.,  12000.,   3000.])

Az egyes elemekhez tartozó lokális elmozdulásvektorok:

```python
Ue1 = UG[en[0]]
Ue1
```

    array([-8.5, -6. ])

```python
Ue2 = UG[en[1]]
Ue2
```

    array([-6.,  0.])

```python
Ue3 = UG[en[2]]
Ue3
```

    array([-6.,  0.])

Az egyes elemekhez tartozó lokális tehervektor:

```python
Fe1 = np.dot(Ke1,Ue1)
Fe1
```

    array([-15000.,  15000.])

```python
Fe2 = np.dot(Ke2,Ue2)
Fe2
```

    array([-12000.,  12000.])

```python
Fe3 = np.dot(Ke3,Ue3)
Fe3
```

    array([-3000.,  3000.])

Tehát a normál igénybevételek:

```python
N1 = Fe1[0]
N1
```

```python
N2 = Fe2[0]
N2
```

```python
N3 = Fe3[0]
N3
```

Vagyis mindegyik rúd nyomó igénybevétel alatt van.

A rudakban ébredő feszültség:

```python
sig1 = N1/A1
sig1
```

```python
sig2 = N2/A2
sig2
```

```python
sig3 = N3/A3
sig3
```

A rudak alakváltozásai:

```python
eps1 = sig1/E1
eps1
```

```python
eps2 = sig2/E2
eps2
```

```python
eps3 = sig3/E3
eps3
```


### Nomenclature:
$d$ means a derivative ALONG the saturation line,
$\partial$ means a partial derivative AT the saturation line (or anywhere in the single phase region).

### References:
Krafcik and Velasco, DOI 10.1119/1.4858403
Thorade and Saadat, DOI 10.1007/s12665-013-2394-z
https://en.wikipedia.org/wiki/Product_rule
https://en.wikipedia.org/wiki/Quotient_rule
https://en.wikipedia.org/wiki/Triple_product_rule

### Clausius-Clapeyron
Clausius-Clapeyron p/T

\begin{equation}
\frac{dp}{dT} = \frac{s''-s'}{v''-v'}
\end{equation}

derivative of Clausius-Clapeyron
\begin{equation}
\begin{split}
\frac{d^2 p}{d T^2} &=
    \frac{\left(\frac{ds''}{dT}-\frac{ds'}{dT}\right) (v''-v')}{(v''-v')^2}
  - \frac{(s''-s') \left(\frac{dv''}{dT}-\frac{dv'}{dT}\right)}{(v''-v')^2} \\
 &= \frac{\left(\frac{ds''}{dT}-\frac{ds'}{dT}\right)}{(v''-v')}
  - \frac{dp}{dT} \frac{\left(\frac{dv''}{dT}-\frac{dv'}{dT}\right)}{(v''-v')}
\end{split}
\end{equation}

Clausius-Clapeyron T/p
\begin{equation}
\frac{dT}{dp} = \frac{v''-v'}{s''-s'}
\end{equation}

derivative of Clausius-Clapeyron
\begin{equation}
\begin{split}
\frac{d^2 T}{d p^2} &=
    \frac{\left(\frac{dv''}{dp}-\frac{dv'}{dp}\right) (s''-s')}{(s''-s')^2}
  - \frac{(v''-v') \left(\frac{ds''}{dp}-\frac{ds'}{dp}\right)}{(s''-s')^2} \\
 &= \frac{\left(\frac{dv''}{dp}-\frac{dv'}{dp}\right)}{(s''-s')}
  - \frac{dT}{dp} \frac{\left(\frac{ds''}{dp}-\frac{ds'}{dp}\right)}{(s''-s')}
\end{split}
\end{equation}

where
\begin{equation}
\begin{split}
\frac{ds}{dT} &= \left(\frac{\partial s}{\partial T}\right)_p  + \left(\frac{\partial s}{\partial p}\right)_T \frac{dp}{dT}\\
\frac{dv}{dT} &= \left(\frac{\partial v}{\partial T}\right)_p  + \left(\frac{\partial v}{\partial p}\right)_T \frac{dp}{dT}\\
\frac{dv}{dp} &= \left(\frac{\partial v}{\partial p}\right)_T  + \left(\frac{\partial v}{\partial T}\right)_p \frac{dT}{dp}\\
\frac{ds}{dp} &= \left(\frac{\partial s}{\partial p}\right)_T  + \left(\frac{\partial s}{\partial T}\right)_p \frac{dT}{dp}
\end{split}
\end{equation}

### Temporary Names

Introduce temporary names for some of the partial derivatives wrt $p$ and $T$:
\begin{equation}
\begin{split}
A &= \left(\frac{\partial \rho}{\partial T}\right)_p \\
B &= \left(\frac{\partial \rho}{\partial p}\right)_T \\
C &= \left(\frac{\partial s}{\partial T}\right)_p \\
E &= \left(\frac{\partial s}{\partial p}\right)_T \\
G &= \left(\frac{\partial h}{\partial T}\right)_p \\
K &= \left(\frac{\partial h}{\partial p}\right)_T
\end{split}
\end{equation}

Introduce temporary names for some of the partial derivatives wrt $\rho$ and $T$:
\begin{equation}
\begin{split}
W &= \left(\frac{\partial p}{\partial T}\right)_{\rho} \\
X &= \left(\frac{\partial p}{\partial \rho}\right)_T \\
Y &= \left(\frac{\partial s}{\partial T}\right)_{\rho} \\
Z &= \left(\frac{\partial s}{\partial \rho}\right)_T
\end{split}
\end{equation}

Introduce temporary names for some of the derivatives along the saturation line:
\begin{equation}
\begin{split}
M &= \frac{d \rho}{d h} = \frac{{d \rho}/{dT}}{{dh}/{dT}} \\
N &= \frac{d s}{d h} = \frac{{ds}/{dT}}{{dh}/{dT}}
\end{split}
\end{equation}

Now the rest is not too hard, but with intermediate steps it is quite long.

### First example: $d^2 \rho / dT^2$
The corresponding first derivative can be written in two ways:
\begin{equation}
\begin{split}
\frac{d \rho}{dT}
  &= \left(\frac{\partial \rho}{\partial T}\right)_p + \left(\frac{\partial \rho}{\partial p}\right)_T \frac{d p}{dT}
   = A + B \frac{dp}{dT}\\
  &= {\frac{dp}{dT} \left(\frac{\partial p}{\partial T}\right)_{\rho}} / {\left(\frac{\partial p}{\partial \rho}\right)_T}
   = {\frac{dp}{dT} W}/X
\end{split}
\end{equation}
Both ways can be used as a starting point for the second derivative:
\begin{equation}
\begin{split}
\frac{d^2 \rho}{dT^2}
  &= \frac{dA}{dT} + \frac{dB}{dT}\frac{dp}{dT} + B\frac{d^2p}{dT^2}\\
  &= \frac{\left(\frac{d^2p}{dT^2} - \frac{dW}{dT}\right)X - \left(\frac{dp}{dT}-W\right)\frac{dX}{dT}}{X^2}
\end{split}
\end{equation}
The first variant seems to be nicer, because the first deriv did not have a fraction in it.
The derivatives along the saturation line that appeared can be written as (again, using two different approaches):
\begin{equation}
\begin{split}
\frac{dA}{dT} &= \left(\frac{\partial A}{\partial T}\right)_p + \left(\frac{\partial A}{\partial p}\right)_T \frac{d p}{dT}\\
\frac{dB}{dT} &= \left(\frac{\partial B}{\partial T}\right)_p + \left(\frac{\partial B}{\partial p}\right)_T \frac{d p}{dT}\\
\frac{dW}{dT} &= \left(\frac{\partial W}{\partial T}\right)_{\rho} + \left(\frac{\partial W}{\partial \rho}\right)_T \frac{d\rho}{dT}\\
\frac{dX}{dT} &= \left(\frac{\partial X}{\partial T}\right)_{\rho} + \left(\frac{\partial X}{\partial \rho}\right)_T \frac{d\rho}{dT}\\
\end{split}
\end{equation}
and then the partial derivatives of A, B, W and X wrt p and T can be written as:
\begin{equation}
\begin{split}
\left(\frac{\partial A}{\partial T}\right)_p &= \left(\frac{\partial^2 \rho}{\partial T^2}\right)_p \\
\left(\frac{\partial A}{\partial p}\right)_T &= \left(\frac{\partial^2 \rho}{\partial p \partial T}\right) \\
\left(\frac{\partial B}{\partial T}\right)_p &= \left(\frac{\partial^2 \rho}{\partial T \partial p}\right) = \left(\frac{\partial A}{\partial p}\right)_T \\
\left(\frac{\partial B}{\partial p}\right)_T &= \left(\frac{\partial^2 \rho}{\partial p^2}\right)_T \\
\left(\frac{\partial W}{\partial T}\right)_{\rho} &= \left(\frac{\partial^2 p}{\partial T^2}\right)_{\rho} \\
\left(\frac{\partial W}{\partial \rho}\right)_T &= \left(\frac{\partial^2 p}{\partial \rho \partial T}\right) \\
\left(\frac{\partial X}{\partial T}\right)_{\rho} &= \left(\frac{\partial^2 p}{\partial T \partial \rho}\right) = \left(\frac{\partial W}{\partial \rho}\right)_T \\
\left(\frac{\partial X}{\partial \rho}\right)_T &= \left(\frac{\partial^2 p}{\partial {\rho}^2}\right)_T
\end{split}
\end{equation}
So this time, the second way looks maybe nicer, because it uses partial derivatives wrt to density and temperature. On the other hand, all second partial derivatives are already implemented in Coolprop and the pT derivatives are internally rewritten as dT derivatives.

### Second example: $d^2s/dT^2$
The corresponding first derivative can be written in two ways:
\begin{equation}
\begin{split}
\frac{ds}{dT}
  &= \left(\frac{\partial s}{\partial T}\right)_p + \left(\frac{\partial s}{\partial p}\right)_T \frac{dp}{dT}
   = C + E\frac{dp}{dT}\\
  &= \left(\frac{\partial s}{\partial T}\right)_{\rho} + \left(\frac{\partial s}{\partial \rho}\right)_T \frac{d \rho}{dT}
   = Y + Z\frac{d \rho}{dT}
\end{split}
\end{equation}
Both can be used as starting point for the second derivatives.
\begin{split}
\frac{d^2 s}{dT^2}
  &= \frac{dC}{dT} + \frac{dE}{dT}\frac{dp}{dT} + E\frac{d^2p}{dT^2}\\
  &= \frac{dY}{dT} + \frac{dZ}{dT}\frac{dp}{dT} + Z\frac{d^2 \rho}{dT^2}
\end{split}
Now, which one is nicer to work with? Not sure.
Just as shown above, the derivatives along the saturation line that appeared can be written as (again, using two different approaches):
\begin{equation}
\begin{split}
\frac{dC}{dT} &= \left(\frac{\partial C}{\partial T}\right)_p + \left(\frac{\partial C}{\partial p}\right)_T \frac{d p}{dT}\\
\frac{E}{dT} &= \left(\frac{\partial E}{\partial T}\right)_p + \left(\frac{\partial E}{\partial p}\right)_T \frac{d p}{dT}\\
\frac{dY}{dT} &= \left(\frac{\partial Y}{\partial T}\right)_{\rho} + \left(\frac{\partial Y}{\partial \rho}\right)_T \frac{d\rho}{dT}\\
\frac{dZ}{dT} &= \left(\frac{\partial Z}{\partial T}\right)_{\rho} + \left(\frac{\partial Z}{\partial \rho}\right)_T \frac{d\rho}{dT}\\
\end{split}
\end{equation}
Rewriting the the partial derivatives works just like shown above.

### Third example: $d^2{\rho}/dp^2$
The corresponding first derivative can be written in many ways, here is one:
\begin{equation}
\frac{d \rho}{dp}
  = \left(\frac{\partial \rho}{\partial p}\right)_T + \left(\frac{\partial \rho}{\partial T}\right)_p \frac{dT}{dp}
  = B + A\frac{dT}{dp}
\end{equation}

The second derivative is then:
\begin{equation}
\frac{d^2 \rho}{dp^2}
  = \frac{dB}{dp} + \frac{dA}{dp}\frac{dT}{dp} + A\frac{d^2T}{dp^2}
\end{equation}
The derivatives wrt p along the saturation line can be written as:
\begin{equation}
\begin{split}
\frac{dB}{dp} &= \left(\frac{\partial B}{\partial p}\right)_T + \left(\frac{\partial B}{\partial T}\right)_p \frac{d T}{dp}\\
\frac{dA}{dp} &= \left(\frac{\partial A}{\partial p}\right)_T + \left(\frac{\partial A}{\partial T}\right)_p \frac{d T}{dp}
\end{split}
\end{equation}
Rewriting the partial derivatives works just like shown above.

### Fourth example: $d^2s/dp^2$
The corresponding first derivative can be written in many ways, here is one:
\begin{equation}
\frac{ds}{dp}
  = \left(\frac{\partial s}{\partial p}\right)_T + \left(\frac{\partial s}{\partial T}\right)_p \frac{dT}{dp}
  = E + C\frac{dT}{dp}
\end{equation}

The second derivative is then:
\begin{equation}
\frac{d^2 s}{dp^2}
  = \frac{dE}{dp} + \frac{dC}{dp}\frac{dT}{dp} + C\frac{d^2T}{dp^2}
\end{equation}
Rewriting the derivatives along the sat line and the partial derivatives works just like shown above.

### Fifth example: $d^2h/dT^2$
First:
\begin{equation}
\frac{dh}{dT}
  = \left(\frac{\partial h}{\partial T}\right)_p + \left(\frac{\partial h}{\partial p}\right)_T \frac{dp}{dT}
  = G + K\frac{dp}{dT}
\end{equation}

Second:
\begin{equation}
\frac{d^2 h}{dT^2}
  = \frac{dG}{dT} + \frac{dK}{dT}\frac{dp}{dT} + K\frac{d^2p}{dT^2}
\end{equation}

### Sixth example: $d^2{\rho}/dh^2$
This is a bit different, but also not difficult.
First:
\begin{equation}
  M = \frac{d\rho}{dh} = \frac{{d\rho}/{dT}}{{dh}/{dT}}
\end{equation}

Intermediate:
\begin{equation}
\frac{dM}{dT}
  = \frac{\frac{d^2 \rho}{dT^2}\frac{dh}{dT}-\frac{d \rho}{dT}\frac{d^2h}{dT^2}}{\left(\frac{dh}{dT}\right)^2}
\end{equation}

Second:
\begin{equation}
\frac{d^2 \rho}{dh^2}
  = \frac{dM}{dh}
  = \frac{{dM}/{dT}}{{dh}/{dT}}
  = \frac{\frac{d^2 \rho}{dT^2}\frac{dh}{dT}-\frac{d \rho}{dT}\frac{d^2h}{dT^2}}{\left(\frac{dh}{dT}\right)^3}
\end{equation}

### Seventh example: $d^2s/dh^2$
First:
\begin{equation}
  N = \frac{ds}{dh} = \frac{{ds}/{dT}}{{dh}/{dT}}
\end{equation}

Intermediate:
\begin{equation}
\frac{dN}{dT}
  = \frac{\frac{d^2s}{dT^2}\frac{dh}{dT}-\frac{ds}{dT}\frac{d^2h}{dT^2}}{\left(\frac{dh}{dT}\right)^2}
\end{equation}

Second:
\begin{equation}
\frac{d^2s}{dh^2}
  = \frac{dN}{dh}
  = \frac{{dN}/{dT}}{{dh}/{dT}}
  = \frac{\frac{d^2s}{dT^2}\frac{dh}{dT}-\frac{ds}{dT}\frac{d^2h}{dT^2}}{\left(\frac{dh}{dT}\right)^3}
\end{equation}

### Eighth example: $d^2s/(dh dp)$
First:
\begin{equation}
  N = \frac{ds}{dh} = \frac{{ds}/{dT}}{{dh}/{dT}}
\end{equation}

Intermediate:
\begin{equation}
\frac{dN}{dT}
  = \frac{\frac{d^2s}{dT^2}\frac{dh}{dT}-\frac{ds}{dT}\frac{d^2h}{dT^2}}{\left(\frac{dh}{dT}\right)^2}
\end{equation}

Second:
\begin{equation}
\frac{d^2s}{dh dp}
  = \frac{dN}{dp}
  = \frac{{dN}/{dT}}{{dp}/{dT}}
  = \frac{\frac{d^2s}{dT^2}\frac{dh}{dT}-\frac{ds}{dT}\frac{d^2h}{dT^2}}{\left(\frac{dh}{dT}\right)^2 \left(\frac{dp}{dT}\right)}
\end{equation}

```
# load some bits and pieces
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

import CoolProp as CP
from CoolProp.CoolProp import PropsSI

# Check: CoolProp version
print(CP.__version__)
print(CP.__gitrevision__)

# Constants
eps = 1e-9
kilo = 1e3
Mega = 1e6
golden = (1 + 5 ** 0.5) / 2
nPoints = 1000
width = 12.5

# helper function: get slope from two sorted arrays
def numSlopeAr(xAr, yAr):
 deltaX = np.ones(len(xAr))
 deltaY = np.ones(len(yAr))
 slopeAr = np.ones(len(yAr))
 for index in range(1,len(xAr)-1):
     deltaX[index] = xAr[index-1] - xAr[index+1]
     deltaY[index] = yAr[index-1] - yAr[index+1]
     slopeAr[index] = deltaY[index]/deltaX[index]
 # inaccurate, but who cares?
 slopeAr[0]=slopeAr[1]
 slopeAr[-1]=slopeAr[-2]
 return slopeAr

# helper function: draw tangent from array
def drawTangent(xAr, yAr, slopeAr, index: int):
 xVal = xAr[index]
 yVal = yAr[index]
 slopeVal = slopeAr[index]
 # line length as percentage of length of x axis
 xRange = 0.15*abs(max(xAr)-min(xAr))
 xRangeAr = np.arange(xVal-xRange, xVal+xRange)
 tang = yVal + slopeVal*(xRangeAr-xVal)
 plt.plot(xVal,yVal,'om',xRangeAr,tang,'-k')
```

    5.1.1
    12f006445f234e572e64cc820146ab5d2c2a9d10

```
# All calculations happen in this cell
# All following cells are used for plotting only
# Run this cell twice to get rid of error message

# Set FluidName
FluidName = 'Propane'

# Triple and critical data
T_crt = PropsSI('TCRIT',FluidName)
T_trp = PropsSI('TTRIPLE',FluidName)
p_crt = PropsSI('PCRIT',FluidName)
p_trp = PropsSI('PTRIPLE',FluidName)
d_crt = PropsSI('RHOCRIT',FluidName)
d_trp_liq = PropsSI('D','T',T_trp,'Q',0,FluidName)
d_trp_vap  = PropsSI('D','T',T_trp,'Q',1,FluidName)
print("T_crt = " + str(T_crt))
print("T_trp = " + str(T_trp))

# Properties at saturation, liq and vap
# All the way to the crt point or keep some distance?
T_sat = np.linspace(T_trp, T_crt, num=nPoints)
p_sat = CP.CoolProp.PropsSI('P','T',T_sat,'Q',0,FluidName)
d_sat_liq = CP.CoolProp.PropsSI('D','T',T_sat,'Q',0,FluidName)
d_sat_vap = CP.CoolProp.PropsSI('D','T',T_sat,'Q',1,FluidName)
v_sat_liq = 1/d_sat_liq
v_sat_vap = 1/d_sat_vap
s_sat_liq = CP.CoolProp.PropsSI('S','T',T_sat,'Q',0,FluidName)
s_sat_vap = CP.CoolProp.PropsSI('S','T',T_sat,'Q',1,FluidName)
h_sat_liq = CP.CoolProp.PropsSI('H','T',T_sat,'Q',0,FluidName)
h_sat_vap = CP.CoolProp.PropsSI('H','T',T_sat,'Q',1,FluidName)
u_sat_liq = CP.CoolProp.PropsSI('U','T',T_sat,'Q',0,FluidName)
u_sat_vap = CP.CoolProp.PropsSI('U','T',T_sat,'Q',1,FluidName)

# Clausius-Clapeyron
# at crt point, vap=liq, so D becomes zero
Ds_vl = (s_sat_vap-s_sat_liq)
Dv_vl = (v_sat_vap-v_sat_liq)
dp_dT_q =  Ds_vl/Dv_vl
dT_dp_q = Dv_vl/Ds_vl # = 1/dp_dT_q
dp_dT_qn = numSlopeAr(T_sat,p_sat)
dT_dp_qn = numSlopeAr(p_sat,T_sat)

# derivs wrt pT, single-phase AT the sat liq line
dd_dp_Tl = CP.CoolProp.PropsSI('d(Dmass)/d(P)|T','D',d_sat_liq,'T',T_sat,FluidName)
dd_dT_pl = CP.CoolProp.PropsSI('d(Dmass)/d(T)|P','D',d_sat_liq,'T',T_sat,FluidName)
ds_dp_Tl = CP.CoolProp.PropsSI('d(Smass)/d(P)|T','D',d_sat_liq,'T',T_sat,FluidName)
ds_dT_pl = CP.CoolProp.PropsSI('d(Smass)/d(T)|P','D',d_sat_liq,'T',T_sat,FluidName)
dh_dp_Tl = CP.CoolProp.PropsSI('d(Hmass)/d(P)|T','D',d_sat_liq,'T',T_sat,FluidName)
dh_dT_pl = CP.CoolProp.PropsSI('d(Hmass)/d(T)|P','D',d_sat_liq,'T',T_sat,FluidName)
dv_dp_Tl = -dd_dp_Tl/d_sat_liq**2
dv_dT_pl = -dd_dT_pl/d_sat_liq**2

# derivs wrt pT, single-phase AT the sat vap line
dd_dp_Tv = CP.CoolProp.PropsSI('d(Dmass)/d(P)|T','D',d_sat_vap,'T',T_sat,FluidName)
dd_dT_pv = CP.CoolProp.PropsSI('d(Dmass)/d(T)|P','D',d_sat_vap,'T',T_sat,FluidName)
ds_dp_Tv = CP.CoolProp.PropsSI('d(Smass)/d(P)|T','D',d_sat_vap,'T',T_sat,FluidName)
ds_dT_pv = CP.CoolProp.PropsSI('d(Smass)/d(T)|P','D',d_sat_vap,'T',T_sat,FluidName)
dh_dp_Tv = CP.CoolProp.PropsSI('d(Hmass)/d(P)|T','D',d_sat_vap,'T',T_sat,FluidName)
dh_dT_pv = CP.CoolProp.PropsSI('d(Hmass)/d(T)|P','D',d_sat_vap,'T',T_sat,FluidName)
dv_dp_Tv = -dd_dp_Tv/d_sat_vap**2
dv_dT_pv = -dd_dT_pv/d_sat_vap**2

# derivs wrt dT, single-phase AT the sat liq line
dp_dd_Tl = CP.CoolProp.PropsSI('d(P)/d(Dmass)|T','D',d_sat_liq,'T',T_sat,FluidName)
dp_dT_dl = CP.CoolProp.PropsSI('d(P)/d(T)|Dmass','D',d_sat_liq,'T',T_sat,FluidName)
ds_dd_Tl = CP.CoolProp.PropsSI('d(Smass)/d(D)|T','D',d_sat_liq,'T',T_sat,FluidName)
ds_dT_dl = CP.CoolProp.PropsSI('d(Smass)/d(T)|D','D',d_sat_liq,'T',T_sat,FluidName)
dh_dd_Tl = CP.CoolProp.PropsSI('d(Hmass)/d(D)|T','D',d_sat_liq,'T',T_sat,FluidName)
dh_dT_dl = CP.CoolProp.PropsSI('d(Hmass)/d(T)|D','D',d_sat_liq,'T',T_sat,FluidName)
dp_dv_Tl = -d_sat_liq**2 * dp_dd_Tl
dp_dT_vl = dp_dT_dl

# derivs wrt dT, single-phase AT the sat vap line
dp_dd_Tv = CP.CoolProp.PropsSI('d(P)/d(Dmass)|T','D',d_sat_vap,'T',T_sat,FluidName)
dp_dT_dv = CP.CoolProp.PropsSI('d(P)/d(T)|Dmass','D',d_sat_vap,'T',T_sat,FluidName)
ds_dd_Tv = CP.CoolProp.PropsSI('d(Smass)/d(D)|T','D',d_sat_vap,'T',T_sat,FluidName)
ds_dT_dv = CP.CoolProp.PropsSI('d(Smass)/d(T)|D','D',d_sat_vap,'T',T_sat,FluidName)
dh_dd_Tv = CP.CoolProp.PropsSI('d(Hmass)/d(D)|T','D',d_sat_vap,'T',T_sat,FluidName)
dh_dT_dv = CP.CoolProp.PropsSI('d(Hmass)/d(T)|D','D',d_sat_vap,'T',T_sat,FluidName)
dp_dv_Tv = -d_sat_vap**2 * dp_dd_Tv
dp_dT_vv = dp_dT_dv

# two ways for derivs along the sat line: using derivs wrt dT or pT

# derivs wrt T ALONG the sat liq line (that is, at q=const=0)
dd_dT_ql = (dp_dT_q - dp_dT_dl)/dp_dd_Tl
dv_dT_ql = dv_dT_pl + dv_dp_Tl*dp_dT_q
ds_dT_ql = ds_dT_dl + ds_dd_Tl*dd_dT_ql
#ds_dT_ql = ds_dT_pl + ds_dp_Tl*dp_dT_q
dh_dT_ql = dh_dT_pl + dh_dp_Tl*dp_dT_q

# derivs wrt T ALONG the sat vap line (that is, at q=const=1)
dd_dT_qv = (dp_dT_q - dp_dT_dv)/dp_dd_Tv
dv_dT_qv = (dp_dT_q - dp_dT_dv)/dp_dd_Tv
dv_dT_qv = dv_dT_pv + dv_dp_Tv*dp_dT_q
ds_dT_qv = ds_dT_dv + ds_dd_Tv*dd_dT_qv
dh_dT_qv = dh_dT_pv + dh_dp_Tv*dp_dT_q

# derivs wrt p ALONG the sat liq line (that is, at q=const=0)
dd_dp_ql = dd_dp_Tl + dd_dT_pl*dT_dp_q
ds_dp_ql = ds_dp_Tl + ds_dT_pl*dT_dp_q
dv_dp_ql = dv_dp_Tl + dv_dT_pl*dT_dp_q

# derivs wrt p ALONG the sat vap line (that is, at q=const=1)
dd_dp_qv = dd_dp_Tv + dd_dT_pv*dT_dp_q
ds_dp_qv = ds_dp_Tv + ds_dT_pv*dT_dp_q
dv_dp_qv = dv_dp_Tv + dv_dT_pv*dT_dp_q

# derivs wrt h ALONG the sat line
dd_dh_ql = dd_dT_ql/dh_dT_ql
dd_dh_qv = dd_dT_qv/dh_dT_qv
ds_dh_ql = ds_dT_ql/dh_dT_ql
ds_dh_qv = ds_dT_qv/dh_dT_qv

# derivs of Clausius-Clapeyron
# d2p_dT2_q = ((ds_dT_qv-ds_dT_ql)*Dv_vl - Ds_vl*(dv_dT_qv-dv_dT_ql)) / Dv_vl**2
d2p_dT2_q = (ds_dT_qv-ds_dT_ql)/Dv_vl - dp_dT_q*(dv_dT_qv-dv_dT_ql)/Dv_vl
d2T_dp2_q = (dv_dp_qv-dv_dp_ql)/Ds_vl - dT_dp_q*(ds_dp_qv-ds_dp_ql)/Ds_vl
d2p_dT2_qn = numSlopeAr(T_sat,dp_dT_q)
d2T_dp2_qn = numSlopeAr(p_sat,dT_dp_q)

# second derivs AT the sat line
# AT the liq line
# wrt pT
d2d_dT2_pl = np.empty(nPoints)
d2d_dp2_Tl = np.empty(len(T_sat))
d2d_dpTl   = np.empty(len(T_sat))
d2s_dT2_pl = np.empty(len(T_sat))
d2s_dp2_Tl = np.empty(len(T_sat))
d2s_dpTl   = np.empty(len(T_sat))
d2h_dT2_pl = np.empty(len(T_sat))
d2h_dp2_Tl = np.empty(len(T_sat))
d2h_dpTl   = np.empty(len(T_sat))
# wrt dT
d2s_dT2_dl = np.empty(len(T_sat))
d2s_dd2_Tl = np.empty(len(T_sat))
d2s_ddTl   = np.empty(len(T_sat))
# AT the vap line
# wrt pT
d2d_dT2_pv = np.empty(len(T_sat))
d2d_dp2_Tv = np.empty(len(T_sat))
d2d_dpTv   = np.empty(len(T_sat))
d2s_dT2_pv = np.empty(len(T_sat))
d2s_dp2_Tv = np.empty(len(T_sat))
d2s_dpTv   = np.empty(len(T_sat))
d2h_dT2_pv = np.empty(len(T_sat))
d2h_dp2_Tv = np.empty(len(T_sat))
d2h_dpTv   = np.empty(len(T_sat))
# wrt dT
d2s_dT2_dv = np.empty(len(T_sat))
d2s_dd2_Tv = np.empty(len(T_sat))
d2s_ddTv   = np.empty(len(T_sat))

HEOS = CP.AbstractState("HEOS", FluidName)
for idx in range(0,len(T_sat)):
    # AT the liq line
    HEOS.update(CP.QT_INPUTS, 0, T_sat[idx])
    # wrt pT
    d2d_dT2_pl[idx] = HEOS.second_partial_deriv(CP.iDmass, CP.iT, CP.iP, CP.iT, CP.iP)
    d2d_dp2_Tl[idx] = HEOS.second_partial_deriv(CP.iDmass, CP.iP, CP.iT, CP.iP, CP.iT)
    d2d_dpTl[idx] = HEOS.second_partial_deriv(CP.iDmass, CP.iP, CP.iT, CP.iT, CP.iP)
    d2s_dT2_pl[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iT, CP.iP, CP.iT, CP.iP)
    d2s_dp2_Tl[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iP, CP.iT, CP.iP, CP.iT)
    d2s_dpTl[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iP, CP.iT, CP.iT, CP.iP)
    d2h_dT2_pl[idx] = HEOS.second_partial_deriv(CP.iHmass, CP.iT, CP.iP, CP.iT, CP.iP)
    d2h_dp2_Tl[idx] = HEOS.second_partial_deriv(CP.iHmass, CP.iP, CP.iT, CP.iP, CP.iT)
    d2h_dpTl[idx] = HEOS.second_partial_deriv(CP.iHmass, CP.iP, CP.iT, CP.iT, CP.iP)
    # wrt dT
    d2s_dT2_dl[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iT, CP.iDmass, CP.iT, CP.iDmass)
    d2s_dd2_Tl[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iDmass, CP.iT, CP.iDmass, CP.iT)
    d2s_ddTl[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iDmass, CP.iT, CP.iT, CP.iDmass)

    # AT the vap line
    HEOS.update(CP.QT_INPUTS, 1, T_sat[idx])
    # wrt pT
    d2d_dT2_pv[idx] = HEOS.second_partial_deriv(CP.iDmass, CP.iT, CP.iP, CP.iT, CP.iP)
    d2d_dp2_Tv[idx] = HEOS.second_partial_deriv(CP.iDmass, CP.iP, CP.iT, CP.iP, CP.iT)
    d2d_dpTv[idx] = HEOS.second_partial_deriv(CP.iDmass, CP.iP, CP.iT, CP.iT, CP.iP)
    d2s_dT2_pv[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iT, CP.iP, CP.iT, CP.iP)
    d2s_dp2_Tv[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iP, CP.iT, CP.iP, CP.iT)
    d2s_dpTv[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iP, CP.iT, CP.iT, CP.iP)
    d2h_dT2_pv[idx] = HEOS.second_partial_deriv(CP.iHmass, CP.iT, CP.iP, CP.iT, CP.iP)
    d2h_dp2_Tv[idx] = HEOS.second_partial_deriv(CP.iHmass, CP.iP, CP.iT, CP.iP, CP.iT)
    d2h_dpTv[idx] = HEOS.second_partial_deriv(CP.iHmass, CP.iP, CP.iT, CP.iT, CP.iP)
    # wrt dT
    d2s_dT2_dv[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iT, CP.iDmass, CP.iT, CP.iDmass)
    d2s_dd2_Tv[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iDmass, CP.iT, CP.iDmass, CP.iT)
    d2s_ddTv[idx] = HEOS.second_partial_deriv(CP.iSmass, CP.iDmass, CP.iT, CP.iT, CP.iDmass)

# calculate 2nd derivs ALONG the sat line (analytically and numerically)

# A=dd_dT_p and B=dd_dp_T
# liq side
dA_dT_ql = d2d_dT2_pl + d2d_dpTl*dp_dT_q
dB_dT_ql = d2d_dpTl + d2d_dp2_Tl*dp_dT_q
d2d_dT2_qla = dA_dT_ql + dB_dT_ql*dp_dT_q + dd_dp_Tl*d2p_dT2_q
d2d_dT2_qln = numSlopeAr(T_sat, dd_dT_ql)
# vap side
dA_dT_qv = d2d_dT2_pv + d2d_dpTv*dp_dT_q
dB_dT_qv = d2d_dpTv + d2d_dp2_Tv*dp_dT_q
d2d_dT2_qva = dA_dT_qv + dB_dT_qv*dp_dT_q + dd_dp_Tv*d2p_dT2_q
d2d_dT2_qvn = numSlopeAr(T_sat, dd_dT_qv)
#print(d2d_dT2_qla - d2d_dT2_qln)
#print(d2d_dT2_qva - d2d_dT2_qln)

# C=ds_dT_p and E=ds_dp_T
# liq side
dC_dT_ql = d2s_dT2_dl + d2s_ddTl*dd_dT_ql
dE_dT_ql = d2s_ddTl + d2s_dd2_Tl*dd_dT_ql
d2s_dT2_qla = dC_dT_ql + dE_dT_ql*dd_dT_ql + ds_dd_Tl*d2d_dT2_qla
d2s_dT2_qln = numSlopeAr(T_sat, ds_dT_ql)
# vap side
dC_dT_qv = d2s_dT2_dv + d2s_ddTv*dd_dT_qv
dE_dT_qv = d2s_ddTv + d2s_dd2_Tv*dd_dT_qv
d2s_dT2_qva = dC_dT_qv + dE_dT_qv*dd_dT_qv + ds_dd_Tv*d2d_dT2_qva
d2s_dT2_qvn = numSlopeAr(T_sat, ds_dT_qv)
#print(d2s_dT2_qla - d2s_dT2_qln)
#print(d2s_dT2_qva - d2s_dT2_qvn)

# B=dd_dp_T and A=dd_dT_p
# liq side
dB_dp_ql = d2d_dp2_Tl + d2d_dpTl*dT_dp_q
dA_dp_ql = d2d_dpTl + d2d_dT2_pl*dT_dp_q
d2d_dp2_qla = dB_dp_ql + dA_dp_ql*dT_dp_q + dd_dT_pl*d2T_dp2_q
d2d_dp2_qln = numSlopeAr(T_sat, ds_dp_ql)
# vap side
dB_dp_qv = d2d_dp2_Tv + d2d_dpTv*dT_dp_q
dA_dp_qv = d2d_dpTv + d2d_dT2_pv*dT_dp_q
d2d_dp2_qva = dB_dp_qv + dA_dp_qv*dT_dp_q + dd_dT_pv*d2T_dp2_q
d2d_dp2_qvn = numSlopeAr(T_sat, ds_dp_qv)
#print(d2d_dp2_qla - d2d_dp2_qln)
#print(d2d_dp2_qva - d2d_dp2_qvn)

# E=ds_dp_T and C=ds_dT_p
# liq side
dE_dp_ql = d2s_dp2_Tl + d2s_dpTl*dT_dp_q
dC_dp_ql = d2s_dpTl + d2s_dT2_pl*dT_dp_q
d2s_dp2_qla = dE_dp_ql + dC_dp_ql*dT_dp_q + ds_dT_pl*d2T_dp2_q
d2s_dp2_qln = numSlopeAr(p_sat, ds_dp_ql)
# vap side
dE_dp_qv = d2s_dp2_Tv + d2s_dpTv*dT_dp_q
dC_dp_qv = d2s_dpTv + d2s_dT2_pv*dT_dp_q
d2s_dp2_qva = dE_dp_qv + dC_dp_qv*dT_dp_q + ds_dT_pv*d2T_dp2_q
d2s_dp2_qvn = numSlopeAr(p_sat, ds_dp_qv)
#print(d2s_dp2_qla - d2s_dp2_qln)
#print(d2s_dp2_qva - d2s_dp2_qvn)

# G=dh_dT_p and K=dh_dp_T
# liq side
dG_dT_ql = d2h_dT2_pl + d2h_dpTl*dp_dT_q
dK_dT_ql = d2h_dpTl + d2h_dp2_Tl*dp_dT_q
d2h_dT2_qla = dG_dT_ql + dK_dT_ql*dp_dT_q + dh_dp_Tl*d2p_dT2_q
d2h_dT2_qln = numSlopeAr(T_sat, dh_dT_ql)
# vap side
dG_dT_qv = d2h_dT2_pv + d2h_dpTv*dp_dT_q
dK_dT_qv = d2h_dpTv + d2h_dp2_Tv*dp_dT_q
d2h_dT2_qva = dG_dT_qv + dK_dT_qv*dp_dT_q + dh_dp_Tv*d2p_dT2_q
d2h_dT2_qvn = numSlopeAr(T_sat, dh_dT_qv)
#print(d2h_dT2_qla - d2h_dT2_qln)
#print(d2h_dT2_qva - d2h_dT2_qvn)

# liq side
d2d_dh2_qla = (d2d_dT2_qla*dh_dT_ql - dd_dT_ql*d2h_dT2_qla) / dh_dT_ql**3
d2d_dh2_qln = numSlopeAr(h_sat_liq, dd_dh_ql)
# vap side
d2d_dh2_qva = (d2d_dT2_qva*dh_dT_qv - dd_dT_qv*d2h_dT2_qva) / dh_dT_qv**3
d2d_dh2_qvn = numSlopeAr(h_sat_vap, dd_dh_qv)
#print(d2d_dh2_qla - d2d_dh2_qln)
#print(d2d_dh2_qva - d2d_dh2_qvn)

# liq side
d2s_dh2_qla = (d2s_dT2_qla*dh_dT_ql - ds_dT_ql*d2h_dT2_qla) / dh_dT_ql**3
d2s_dh2_qln = numSlopeAr(h_sat_liq, ds_dh_ql)
# vap side
d2s_dh2_qva = (d2s_dT2_qva*dh_dT_qv - ds_dT_qv*d2h_dT2_qva) / dh_dT_qv**3
d2s_dh2_qvn = numSlopeAr(h_sat_vap, ds_dh_qv)
#print(d2s_dh2_qla - d2s_dh2_qln)
#print(d2s_dh2_qva - d2s_dh2_qvn)
```

    T_crt = 369.89
    T_trp = 85.525

```
# vapor pressure, Clausius-Clapeyron

# pick a point and set figure size
mySatPoint = int(nPoints-50)
print("T_sat = " + str(T_sat[mySatPoint]))
print("p_sat = " + str(p_sat[mySatPoint]))
plt.figure(figsize=(width,width*3/2/golden))

plt.subplot(3,2,1)
plt.plot(T_sat, p_sat, color='blue')
#plt.yscale('log')
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))
plt.grid(b=True, linestyle=':')
plt.minorticks_on()
plt.xlabel('Temperature in K')
plt.ylabel('Pressure in Pa')
drawTangent(T_sat, p_sat, dp_dT_q, mySatPoint)
#print(dp_dT_q - dp_dT_qn)

plt.subplot(3,2,2)
plt.plot(p_sat, T_sat, color='blue')
#plt.yscale('log')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.grid(b=True, linestyle=':')
plt.minorticks_on()
plt.xlabel('Pressure in Pa')
plt.ylabel('Temperature in K')
drawTangent(p_sat, T_sat, dT_dp_q, mySatPoint)
#print(dT_dp_q - dT_dp_qn)

plt.subplot(3,2,3)
plt.plot(T_sat, dp_dT_q, color='blue')
#plt.yscale('log')
plt.grid(b=True, linestyle=':')
plt.minorticks_on()
plt.xlabel('Temperature in K')
plt.ylabel('dp/dT in Pa/K')
drawTangent(T_sat, dp_dT_q, d2p_dT2_q, mySatPoint)
#print(d2p_dT2_q - d2p_dT2_qn)

plt.subplot(3,2,4)
plt.plot(p_sat, dT_dp_q, color='blue')
plt.yscale('log')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.grid(b=True, linestyle=':')
plt.minorticks_on()
plt.xlabel('Pressure in Pa')
plt.ylabel('dT/dp in K/Pa')
drawTangent(p_sat, dT_dp_q, d2T_dp2_q, mySatPoint)
#print(d2T_dp2_q - d2T_dp2_qn)

plt.subplot(3,2,5)
plt.plot(T_sat, d2p_dT2_q, color='blue')
plt.grid(b=True, linestyle=':')
plt.minorticks_on()
plt.xlabel('Temperature in K')
plt.ylabel('d2p/dT2 in Pa/K2')

plt.subplot(3,2,6)
plt.plot(p_sat, abs(d2T_dp2_q), color='blue')
plt.yscale('log')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.grid(b=True, linestyle=':')
plt.minorticks_on()
plt.xlabel('Pressure in Pa')
plt.ylabel('|d2T/dp2| in K/Pa2')
```

```
# Pick a point and set figure size
mySatPoint = int(nPoints-50)
print("T_sat = " + str(T_sat[mySatPoint]))
print("p_sat = " + str(p_sat[mySatPoint]))
plt.figure(figsize=(width,width*3/2/golden))

# d versus T
plt.subplot(3,2,1)
plt.plot(T_sat, d_sat_liq, color='blue')
plt.plot(T_sat, d_sat_vap, color='red')
#plt.plot(T_sat, (d_sat_liq+d_sat_vap)/2, color='black')
plt.xlabel('Temperature in K')
plt.ylabel('Density in kg/m³')
drawTangent(T_sat, d_sat_liq, dd_dT_ql, mySatPoint)
drawTangent(T_sat, d_sat_vap, dd_dT_qv, mySatPoint)

# s versus T
plt.subplot(3,2,2)
plt.plot(T_sat, s_sat_liq, color='blue')
plt.plot(T_sat, s_sat_vap, color='red')
plt.xlabel('Temperature in K')
plt.ylabel('Specific entropy in J/(kg·K)')
drawTangent(T_sat, s_sat_liq, ds_dT_ql, mySatPoint)
drawTangent(T_sat, s_sat_vap, ds_dT_qv, mySatPoint)

# d versus p
plt.subplot(3,2,3)
plt.plot(p_sat, d_sat_liq, color='blue')
plt.plot(p_sat, d_sat_vap, color='red')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Pressure in Pa')
plt.ylabel('Density in kg/m³')
drawTangent(p_sat, d_sat_liq, dd_dp_ql, mySatPoint)
drawTangent(p_sat, d_sat_vap, dd_dp_qv, mySatPoint)
#print(dd_dp_ql - numSlopeAr(p_sat, d_sat_liq))
#print(dd_dp_qv - numSlopeAr(p_sat, d_sat_vap))

# s versus p
plt.subplot(3,2,4)
plt.plot(p_sat, s_sat_liq, color='blue')
plt.plot(p_sat, s_sat_vap, color='red')
#plt.yscale('log')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Pressure in Pa')
plt.ylabel('Specific entropy in J/(kg·K)')
drawTangent(p_sat, s_sat_liq, ds_dp_ql, mySatPoint)
drawTangent(p_sat, s_sat_vap, ds_dp_qv, mySatPoint)
#print(ds_dp_ql - numSlopeAr(p_sat, s_sat_liq))
#print(ds_dp_qv - numSlopeAr(p_sat, s_sat_vap))

# d versus h
plt.subplot(3,2,5)
plt.plot(h_sat_liq, d_sat_liq, color='blue')
plt.plot(h_sat_vap, d_sat_vap, color='red')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Specific enthalpy in J/kg')
plt.ylabel('Density in kg/m³')
drawTangent(h_sat_liq, d_sat_liq, dd_dh_ql, mySatPoint)
drawTangent(h_sat_vap, d_sat_vap, dd_dh_qv, mySatPoint)

# s versus h
plt.subplot(3,2,6)
plt.plot(h_sat_liq, s_sat_liq, color='blue')
plt.plot(h_sat_vap, s_sat_vap, color='red')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Specific enthalpy in J/kg')
plt.ylabel('Specific entropy in J/(kg·K)')
drawTangent(h_sat_liq, s_sat_liq, ds_dh_ql, mySatPoint)
drawTangent(h_sat_vap, s_sat_vap, ds_dh_qv, mySatPoint)
```

```
# Pick a point and set figure size
mySatPoint = int(nPoints-50)
print("T_sat = " + str(T_sat[mySatPoint]))
print("p_sat = " + str(p_sat[mySatPoint]))
plt.figure(figsize=(width,width*3/2/golden))

# dd_dT versus T
plt.subplot(3,2,1)
plt.plot(T_sat, dd_dT_ql, color='blue')
plt.plot(T_sat, dd_dT_qv, color='red')
plt.xlabel('Temperature in K')
plt.ylabel('dd/dT in kg/m³/K')
drawTangent(T_sat, dd_dT_ql, d2d_dT2_qla, mySatPoint)
drawTangent(T_sat, dd_dT_qv, d2d_dT2_qva, mySatPoint)

# ds_dT versus T
plt.subplot(3,2,2)
plt.plot(T_sat, ds_dT_ql, color='blue')
plt.plot(T_sat, ds_dT_qv, color='red')
plt.xlabel('Temperature in K')
plt.ylabel('ds/dT in J//(kg·K)/K')
drawTangent(T_sat, ds_dT_ql, d2s_dT2_qla, mySatPoint)
drawTangent(T_sat, ds_dT_qv, d2s_dT2_qva, mySatPoint)

# dd_dp versus p
plt.subplot(3,2,3)
#plt.plot(p_sat, dd_dp_ql, color='blue')
plt.plot(p_sat, dd_dp_qv, color='red')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Pressure in Pa')
plt.ylabel('dd/dp in kg/m³/Pa')
#drawTangent(p_sat, dd_dp_ql, d2d_dp2_qla, mySatPoint)
drawTangent(p_sat, dd_dp_qv, d2d_dp2_qva, mySatPoint)

# ds_dp versus p
plt.subplot(3,2,4)
plt.plot(p_sat, ds_dp_ql, color='blue')
#plt.plot(p_sat, ds_dp_qv, color='red')
#plt.yscale('log')
plt.ylim([0,10*min(ds_dp_ql)])
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Pressure in Pa')
plt.ylabel('ds/dp in J//(kg·K)/Pa')
drawTangent(p_sat, ds_dp_ql, d2s_dp2_qla, mySatPoint)
#drawTangent(p_sat, ds_dp_qv, d2s_dp2_qva, mySatPoint)

# dd_dh versus h
plt.subplot(3,2,5)
plt.plot(h_sat_liq, dd_dh_ql, color='blue')
#plt.plot(h_sat_vap, dd_dh_qv, color='red')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Specific enthalpy in J/kg')
plt.ylabel('dd/dh')
drawTangent(h_sat_liq, dd_dh_ql, d2d_dh2_qla, mySatPoint)
#drawTangent(h_sat_vap, dd_dh_qv, d2d_dh2_qva, mySatPoint)

# ds_dh versus h
plt.subplot(3,2,6)
plt.plot(h_sat_liq, ds_dh_ql, color='blue')
#plt.plot(h_sat_vap, ds_dh_qv, color='red')
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.xlabel('Specific enthalpy in J/kg')
plt.ylabel('ds/dh')
drawTangent(h_sat_liq, ds_dh_ql, d2s_dh2_qla, mySatPoint)
#drawTangent(h_sat_vap, ds_dh_qv, d2s_dh2_qva, mySatPoint)
```


# Stochastic Variational Inference
Notes from the Hoffmann *et al.* (2013) paper

In probabilistic modelling, we use hidden variables to encode hidden structure in observed data; we articulate the relationship between the hidden and observed variables with a factorized probability distribution (i.e. a graphical model) and we use inference algorithms to estimate the **posterior distribution**, the **conditional distribution** of hidden structure given the observations.

Consider a graphical model of hidden and observed random variables for which we want to compute the posterior. For many models of interest, this posterior is not tractable to compute and we must appeal to approximate methods. The two most prominent strategies in statistics and machine learning are Markov chain Monte Carlo (MCMC) sampling and variational inference

* **MCMC Sampling**:
    We construct a Markov chain over the hidden variables whose stationary distribution is the posterior of interest. We run the chain until it has (hopefully) reached equilibrium and collect samples to approximate the posterior.
* **Variational inference**:
    We define a flexible family of distribution over the hidden variables, indexed by free parameters. We then find the setting of the parameters (i.e. the member of the family) that is closest to the posterior. Thus we solve the inference problem by solving an optimization problem.

The aim here is to develop a general variational method that scales.

Form of stochastic variational inference:
1. Subsample one or more data points from the data
2. Analyze the subsample using the current variational parameters
3. Implement a closed-form update of the variational parameters.
4. Repeat.
While traditional algorithms require repeatedly analyzing the whole dataset before updating the variational parameters, this algorithm only requires that we analyze randomly sampled subsets.

SVI is a sochastic **optimization algorithm** for mean-field variational inference. It approximates the posterior distribution of a probabilistic model with hidden variables, and can handle massive data sets of observations.

A graphical model with observations $x_{1:N}$, local hidden variables $z_{1:N}$ and global hidden variables $\beta$. The distribution of each observation $x_n$ only depends on its corresponding local variable $z_n$ and the global variables $\beta$.

## 1. Define the class of models to which our algorithm applies.
We define *local* and *global* hidden variables, and requirements on the conditional distributions within the model.

The joint distribution factorizes into a global term and a product of local terms:
$$
p(x, z, \beta |\alpha) = p(\beta | \alpha)\prod_{n=1}^N p(x_n, z_n |\beta)
$$
Our goal is to approximate the posterior distribution of the hidden variables given the observations, $p(\beta, z|x)$.

**Assumption 1**: The $n$th observation $x_n$ and the $n$th local variable $z_n$ are conditionally independent, given global variables $\beta$, of all other observations and local hidden variables,
$$
p(x_n, z_n |x_{-n}, z_{-n},\beta,\alpha) = p(x_n, z_n|\beta,\alpha)
$$

**Assumption 2**: The *complete conditionals* in the model. A complete conditional is the conditional distribution of a hidden variable given the other hidden variables and the observations. We assume that these distributions are in the **exponential family**,

\begin{eqnarray}
p(\beta|x, z, \alpha) & = & h(\beta)\exp\{\eta_g(x, z, \alpha)^Tt(\beta)-a_g(\eta_g(x, z, \alpha))\}\\
p(z_{nj}|x_n, z_{n,-j}, \beta) & = & h(z_{nj})\exp\{\eta_l(x_n, z_{n,-j},\beta)^Tt(z_{nj}) - a_l(\eta_l(x_n, z_{n,-j},\beta))\}
\end{eqnarray}

The scala functions $h(\cdot)$ and $a(\cdot)$ are respectively the *base measure* and *log-normalizer*, the vector functions $\eta(\cdot)$ and $t(\cdot)$ are respectively the *natural parameter* and *sufficient statistics*. **(For details of this consult a basic statistic book on exponential distributions)**.

These are conditional distributions, so the natural parameter is a function of the variables that are being conditioned on. For the local variables $z_{nj}$, the complete conditional distribution is determined by the global variables $\beta$ and the other local variables in the $n$th context, i.e. the $n$th data point $x_n$ and the local variables $z_{n,-j}$.

These assumptions on the complete conditional imply a **conjugacy relationship** between the global variables $\beta$ and the local contexts $(z_n, x_n)$, and this relationship implies the distribution of the local context given the global variables must be in an exponential family,

\begin{equation}
p(x_n, z_n |\beta) = h(x_n, z_n)\exp\{\beta^Tt(x_n, z_n) - a_l(\beta)\}
\end{equation}

The prior distribution $p(\beta)$ must also be in an exponential family,
$$
p(\beta) = h(\beta)\exp\{\alpha^Tt(\beta)-a_g(\alpha)\}
$$
The sufficient statistics are $t(\beta) = (\beta, -a_l(\beta))$ and thus the hyperparameter $\alpha$ has two components $\alpha = (\alpha_1, \alpha_2)$. The first component $\alpha_1$ is a vector of the same dimension as $\beta$, the second component $\alpha_2$ is a scalar.

The two equations above imply that the complete conditional for the global variable is in the same exponential family as the prior with natural parameter

$$
\eta_g(x, z, \alpha) = (\alpha_1 + \sum_{n=1}^N t(z_n, x_n), \alpha_2+N).
$$

Analysing data with one of the model associated with this family of distributions (e.g. Bayesian mixture models, Latent Dirichlet allocation) amounts to computing the posterior distribution of the hidden variables given the observations,

$$
p(z, \beta |x ) = \frac{p(x, z, \beta)}{\int p(x, z, \beta)dz d\beta}.
$$
We then use this posterior to explore the hidden structure of our data or to make predictions about future data.

## 2. Mean field variational inference
An approximate inference strategy that seeks a tractable distribution over the hidden variables which is close to the posterior distribution. Derive the traditional variational inference algorithm for our class of models, which is a coordinate ascent algorithm. Closeness is measured with the KL divergence. We use the resulting distribution, called the *variational distribution* to approximate the posterior.

### The evidence lower bound
Variational inference minimizes the KL divergence from the variational distribution to the posterior distribution. It maximizes the *evidence lower bound* (ELBO), a lower bound on the logarithm of the marginal probability of the observations $\log p(x)$. The ELBO is equal to the negative KL divergence up to an additive constant.

We derive the ELBO by introducing a distribution over the hidden variables $q(\alpha, \beta)$ and using Jensen's inequality. (This implies $\log\mathbb{E}[f(y)]\ge \mathbb{E}[\log f(y)]$ for any random variable $y$).

This gives the following bound on the log marginal,

\begin{eqnarray}
\log p(x) & = & \log\int p(x, z, \beta)dz d\beta\\
& = & \log\int p(x, z, \beta)\frac{q(z, \beta)}{q(z, \beta)}dzd\beta\\
& = & \log\left(\mathbb{E}_q\left[\frac{p(x,z,\beta)}{q(z,\beta)}\right]\right)\\
&\ge & \mathbb{E}_q[\log p(x, z, \beta)]-\mathbb{E}[\log q(z, \beta)]\\
&\triangleq &\mathcal{L}(q).
\end{eqnarray}

The ELBO contains two terms. The first term is the expected log joint, $\mathbb{E}_q[\log p(x, z, \beta)]$. The second is the entropy of the variational distribution, $-\mathbb{E}_q[\log q(z, \beta)]$. Both of these terms depend on $q(z, \beta)$, the variational distribution of the hidden variables.

We restrict $q(z, \beta)$ to be in a family that is tractable, one for which the expectations in the ELBO can be efficiently computed. We then try to find the member of the family that maximizes the ELBO. Finally, we use the optimized distribution as a proxy for the posterior.

Solving this maximization problem is equivalent to finding the member of the family that is closest in KL divergence to the posterior:

\begin{eqnarray}
KL(q(z,\beta)||p(z, \beta|x)) & = & \mathbb{E}_q[\log q(z, \beta)] - \mathbb{E}_q[\log p(z, \beta|x)]\\
& = & \mathbb{E}_q[\log q(z, \beta)]-\mathbb{E}_q[\log p(x, \, \beta)] + \log p(x)\\
& = & -\mathcal{L}(q) + \mathrm{const}.
\end{eqnarray}
$\log p(x)$ is replaced by a constant because it does not depend on $q$.

### The mean-field variational family.
The simplest variational family of distributions. In this family, each hidden variable is independent and governed by its own parameter,

$$
q(z, \beta) = q(\beta |\lambda)\prod_{n=1}^N \prod_{j=1}^J q(z_{nj}|\phi_{nj})
$$

The global parameters $\lambda$ govern the global variables, the local parameters $\phi_n$ govern the local variables in the $n$th context. The ELBO is a function of these parameters.

We set $q(\beta|\lambda)$ and $q(z_{nj}|\phi_{nj})$ to be in the same exponential family as the complete conditional distributions $p(\beta|x, z)$ and $p(z_{nj}|x_n,z_{n,-j},\beta)$. The variational parameters $\lambda$ and $\phi_{nj}$ are the natural parameters to those families,

\begin{eqnarray}
q(\beta|\lambda) & = & h(\beta)\exp\{\lambda^Tt(\beta)-a_g(\lambda)\}\\
q(z_{nj}|\phi_{nj}) & = & h(z_{nj})\exp\{\phi_{nj}^Tt(z_{nj}) - a_{l}(\phi_{nj})\}
\end{eqnarray}


## Nonlinear Dimensionality Reduction
G. Richards (2016, 2018), based on materials from Ivezic, Connolly, Miller, Leighly, and VanderPlas.

Today we will talk about the concepts of
* manifold learning
* nonlinear dimensionality reduction

Specifically using the following algorithms
* local linear embedding (LLE)
* isometric mapping (IsoMap)
* t-distributed Stochastic Neighbor Embedding (t-SNE)

Let's start by my echoing the brief note of caution given in Adam Miller's notebook: "astronomers will often try to derive physical insight from PCA eigenspectra or eigentimeseries, but this is not advisable as there is no physical reason for the data to be linearly and orthogonally separable".  Moreover, physical components are (generally) positive definite.  So, PCA is great for dimensional reduction, but for doing physics there are generally better choices.

While NMF "solves" the issue of negative components, it is still a linear process.  For data with non-linear correlations, an entire field, known as [Manifold Learning](http://scikit-learn.org/stable/modules/manifold.html) and [nonlinear dimensionality reduction]( https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction), has been developed, with several algorithms available via the [`sklearn.manifold`](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.manifold) module.

For example, if your data set looks like this:

Then PCA is going to give you something like this.

Clearly not very helpful!

What you were really hoping for is something more like the results below.  For more examples see
[Vanderplas & Connolly 2009](http://iopscience.iop.org/article/10.1088/0004-6256/138/5/1365/meta;jsessionid=48A569862A424ECCAEECE2A900D9837B.c3.iopscience.cld.iop.org)

## Local Linear Embedding

[Local Linear Embedding](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.LocallyLinearEmbedding.html#sklearn.manifold.LocallyLinearEmbedding) attempts to embed high-$D$ data in a lower-$D$ space.  Crucially it also seeks to preserve the geometry of the local "neighborhoods" around each point.  In the case of the "S" curve, it seeks to unroll the data.  The steps are

Step 1: define local geometry
- local neighborhoods determined from $k$ nearest neighbors.
- for each point calculate weights that reconstruct a point from its $k$ nearest
neighbors via
$$
\begin{equation}
  \mathcal{E}_1(W) = \left|X - WX\right|^2,
\end{equation}
$$
where $X$ is an $N\times K$ matrix and $W$ is an $N\times N$ matrix that minimizes the reconstruction error.

Essentially this is finding the hyperplane that describes the local surface at each point within the data set. So, imagine that you have a bunch of square tiles and you are trying to tile the surface with them.

Step 2: embed within a lower dimensional space
- set all $W_{ij}=0$ except when point $j$ is one of the $k$ nearest neighbors of point $i$.
- $W$ becomes very sparse for $k \ll N$ (only $Nk$ entries in $W$ are non-zero).
- minimize
>$\begin{equation}
  \mathcal{E}_2(Y) = \left|Y - W Y\right|^2,
\end{equation}
$

with $W$ fixed to find an $N$ by $d$ matrix ($d$ is the new dimensionality).

Step 1 requires a nearest-neighbor search.

Step 2 requires an
eigenvalue decomposition of the matrix $C_W \equiv (I-W)^T(I-W)$.

LLE has been applied to data as diverse as galaxy spectra, stellar spectra, and photometric light curves.   It was introduced by [Roweis & Saul (2000)](https://www.ncbi.nlm.nih.gov/pubmed/11125150).

Skikit-Learn's call to LLE is as follows, with a more detailed example already being given above.

```python
import numpy as np
from sklearn.manifold import LocallyLinearEmbedding
X = np.random.normal(size=(1000,2)) # 1000 points in 2D
R = np.random.random((2,10)) # projection matrix
X = np.dot(X,R) # now a 2D linear manifold in 10D space
k = 5 # Number of neighbors to use in fit
n = 2 # Number of dimensions to fit
lle = LocallyLinearEmbedding(k,n)
lle.fit(X)
proj = lle.transform(X) # 100x2 projection of the data
```

See what LLE does for the digits data, using the 7 nearest neighbors and 2 components.

```python
# Execute this cell to load the digits sample
%matplotlib inline
import numpy as np
from sklearn.datasets import load_digits
from matplotlib import pyplot as plt
digits = load_digits()
grid_data = np.reshape(digits.data[0], (8,8)) #reshape to 8x8
plt.imshow(grid_data, interpolation = "nearest", cmap = "bone_r")
print(grid_data)
X = digits.data
y = digits.target
```

```python
#LLE
from sklearn.manifold import LocallyLinearEmbedding
k = 7 # Number of neighbors to use in fit
n = 2 # Number of dimensions to fit
lle = LocallyLinearEmbedding(k,n)
lle.fit(X)
X_reduced = lle.transform(X)

plt.scatter(X_reduced[:,0], X_reduced[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

## Isometric Mapping

is based on multi-dimensional scaling (MDS) framework.  It was introduced in the same volume of *Science* as the article above.  See [Tenenbaum, de Silva, & Langford (2000)](https://www.ncbi.nlm.nih.gov/pubmed/?term=A+Global+Geometric+Framework+for+Nonlinear+Dimensionality+Reduction).
Geodestic curves are used to recover non-linear structure.

In Scikit-Learn [IsoMap](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.Isomap.html) is implemented as follows:

```python
# Execute this cell
import numpy as np
from sklearn.manifold import Isomap
XX = np.random.normal(size=(1000,2)) # 1000 points in 2D
R = np.random.random((2,10)) # projection matrix
XX = np.dot(XX,R) # X is a 2D manifold in 10D space
k = 5 # number of neighbors
n = 2 # number of dimensions
iso = Isomap(k,n)
iso.fit(XX)
proj = iso.transform(XX) # 1000x2 projection of the data
```

Try 7 neighbors and 2 dimensions on the digits data.

```python
# IsoMap
from sklearn.manifold import Isomap
k = 7 # Number of neighbors to use in fit
n = 2 # Number of dimensions to fit
iso = Isomap(k,n)
iso.fit(X)
X_reduced = iso.transform(X)

plt.scatter(X_reduced[:,0], X_reduced[:,1], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

## t-SNE

[t-distributed Stochastic Neighbor Embedding (t-SNE)](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding) is not discussed in the book, Scikit-Learn does have a [t-SNE implementation](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) and it is well worth mentioning this manifold learning algorithm too.  SNE itself was developed by [Hinton & Roweis](http://www.cs.toronto.edu/~fritz/absps/sne.pdf) with the "$t$" part being added by [van der Maaten & Hinton](http://jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf).  It works like the other manifold learning algorithms.

 Try it on the digits data.  You'll need to import `TSNE` from `sklearn.manifold`, instantiate it with 2 components, then do a `fit_transform` on the original data.

```python
# t-SNE
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2)
X_reduced = tsne.fit_transform(X)

plt.scatter(X_reduced[:,0], X_reduced[:,1] , c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

You'll know if you have done it right if you understand Adam Miller's comment "Holy freakin' smokes.  That is magic.  (It's possible we just solved science)."

Personally, I think that some exclamation points may be needed in there!

What's even more illuminating is to make the plot using the actual digits to plot the points.  Then you can see why certain digits are alike or split into multiple regions.  Can you explain the patterns you see here?

```python
# Execute this cell
from matplotlib import offsetbox

#----------------------------------------------------------------------
# Scale and visualize the embedding vectors
def plot_embedding(X):
    x_min, x_max = np.min(X, 0), np.max(X, 0)
    X = (X - x_min) / (x_max - x_min)

    plt.figure()
    ax = plt.subplot(111)
    for i in range(X.shape[0]):
      #plt.text(X[i, 0], X[i, 1], str(digits.target[i]), color=plt.cm.Set1(y[i] / 10.), fontdict={'weight': 'bold', 'size': 9})
      plt.text(X[i, 0], X[i, 1], str(digits.target[i]), color=plt.cm.nipy_spectral(y[i]/9.))

    shown_images = np.array([[1., 1.]])  # just something big
    for i in range(digits.data.shape[0]):
        dist = np.sum((X[i] - shown_images) ** 2, 1)
        if np.min(dist) < 4e-3:
            # don't show points that are too close
            continue
        shown_images = np.r_[shown_images, [X[i]]]
        imagebox = offsetbox.AnnotationBbox(offsetbox.OffsetImage(digits.images[i], cmap=plt.cm.gray_r), X[i])
        ax.add_artist(imagebox)
    plt.xticks([]), plt.yticks([])

plot_embedding(X_reduced)
plt.show()
```

With the remainder of time in class today, play with the arguments of the algorithms that we have discussed this week and/or try running them on a different data set.  For example the iris data set or one of the other samples of data that are included with Scikit-Learn.  Or maybe have a look through some of these public data repositories:

- [https://github.com/caesar0301/awesome-public-datasets?utm_content=buffer4245d&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer](https://github.com/caesar0301/awesome-public-datasets?utm_content=buffer4245d&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer)
- [http://www.datasciencecentral.com/m/blogpost?id=6448529%3ABlogPost%3A318739](http://www.datasciencecentral.com/m/blogpost?id=6448529%3ABlogPost%3A318739)
- [http://www.kdnuggets.com/2015/04/awesome-public-datasets-github.html](http://www.kdnuggets.com/2015/04/awesome-public-datasets-github.html)