# Portfolio Value-at-Risk (VaR) estimation and evaluation
### Author: George Fatouros, gfatouros@innov-acts.com

### Introduction

* Value at Risk (VAR) is a rather simple yet valuable risk estimation measure, which helps traders and investors understand the risk of loss for their investments. Using VaR financial professionals can estimate how much their investments could lose in a specified time window (e.g., during a day or month) under normal market conditions.

* VaR can be calculated at different levels and granularities. For example, it can used to estimate the risk of a single asset or of a portfolio of assets. This is a very common use of VaR by commercial banks and investment organizations, which wish to estimate and understand the potential losses of their institutional portfolios.

* From a statistical perspective VaR quantifies the maximum potential losses over a specified timeframe with a degree of confidence. Confidence is therefore an important parameter of VaR calculation. For example, if for a given portfolio the 95% confidence of a one-day VAR is 100.000 EUR, this means that there is a 95% confidence that the portfolio will not lose more than 100.000 EUR within a day. 95% and 99% are the most common confidence internals used in VaR calculation.

$$P[r < VaR_a] = a $$ where r, a are the portfolio returns and the confidence probability respectively.

* Since the introduction of VaR as a standard risk metric, various types of models have been developed for its assessment. The main categories of which are the:
    * Non-Parametric: Assumptions regarding the distribution of returns are not required. The Historical Simulation (HS) is the main representative of this method, where the empirical distribution of past portfolio returns is used to calculate VaR.
    * Parametric: Portfolio returns and their distribution should be theoretically defined prior to VaR estimation. Some well-known methods of the parametric category are the Variance-Covariance Method (VC) and many GARCH-variants Methods, such as the Risk Metrics model.

    * Semi-Parametric: Some assumptions regarding the distribution of returns are made. The Monte Carlo Method is the main semi-parametric method which generates random scenarios for future portfolio returns, drawing their distribution based on some non-linear pricing models.

### Exploratory Data Analysis

```python
# import required packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import datetime as dt
import pandas_datareader as pdr

# set plotting parameters
%matplotlib inline
sns.set_style("whitegrid")
plt.rc("figure", figsize=(16, 8))
plt.rc("savefig", dpi=90)
plt.rc("font", family="sans-serif")
plt.rc("font", size=14)

```

#### The Data

* We download historical market data from yahoo finance for Apple, Google, Facebook, Nvidia and Tesla stocks. The dataset consists of daily market data such as open, high, low and close prices ranging from 2016-01-01 to 2021-12-31.

```python
# load historical data from yahoo finance
tickers = ['aapl', 'goog', 'fb', 'nvda', 'tsla']
data = pdr.DataReader(tickers, data_source='yahoo', start='2016-01-01', end='2021-12-31')

```

* For this analysis we will use only the adjusted close prices. The adjusted closing price amends a stock's closing price to reflect that stock's value after accounting for any corporate actions. It is often used when examining historical returns or doing a detailed analysis of past performance.

* The below plot of daily prices is misleading in understanding the performance of each stock because Google market price is traded at a much higher rate than the other stocks.

```python
prices = data['Adj Close']
prices.plot()
plt.title('Daily prices')
plt.ylabel('Adjusted Close Price')
plt.show()
```

* To alleviate this issue we rescale our data to log prices. The log prices plot illustrates that Google and Facebook stocks had similar performance, while Nvidia and Tesla had higher growth than the others.

* However, in order to be able to perform forecasting and reliable analysis in timeseries data these data should be stationary.

* A stationary time series is one whose properties do not depend on the time at which the series is observed. Thus, time series with trends, or with seasonality, are not stationary. Here, we can observe that all the time series have an up trend.

```python
np.log(prices).plot()
plt.ylabel('Adjusted Close log Price')
plt.title('Daily log prices')
plt.show()
```

* The most common technique for making a time series stationary or at least close to stationary is to use its rate of change from the one time step to the next.

* So instead of analyzing the prices $p$, we will transform them in terms of daily returns $r$ based on the following formula.

$$r_t = \frac{p_t-p_{t-1}}{p_{t-1}}$$

```python
returns = data['Adj Close'].pct_change().dropna()
returns.plot()
plt.title('Daily returns')
plt.ylabel('returns')
plt.show()
```

* As we can see from the above graph of returns, after the transformation the upward trend in the time series was eliminated and now they are moving around zero. This means that time series are now closer to stationarity and we can use them for our analysis.

* Note that the time series are still not stationary as they  have not constant variance over time, however, time series stationarity goes beyond the scope of this notebook. For more information on time series stability, see this article: https://machinelearningmastery.com/time-series-data-stationary-python/

```python
returns_summary = (returns*100).describe()
returns_summary.round(2)[1:]
```

      Symbols
      aapl
      goog
      fb
      nvda
      tsla

      mean
      0.15
      0.10
      0.10
      0.28
      0.27

      std
      1.85
      1.63
      2.04
      2.97
      3.61

      min
      -12.86
      -11.10
      -18.96
      -18.76
      -21.06

      25%
      -0.63
      -0.57
      -0.78
      -1.09
      -1.45

      50%
      0.11
      0.13
      0.11
      0.27
      0.15

      75%
      1.03
      0.89
      1.13
      1.72
      1.92

      max
      11.98
      10.45
      15.52
      29.81
      19.89

* Uncomment the bellow cell of code for more EDA

```python
# from pandas_profiling import ProfileReport
# profile = ProfileReport(returns, minimal=True)
# profile
```

### Historical Simulation (HS) VaR

* The Historical Simulation (HS) is the main representative of the non-parametric VaR method, where the empirical distribution of past portfolio returns is used to calculate VaR.

* This is probably the simplest VaR calculation method. It relies on significant volumes of historical market data (e.g., typically one trading year data for conventional assets and much more than that for hedge funds) to calculate the price changes for all the assets of the portfolio. Accordingly, it calculates the value of the portfolio for each one of the price changes. These historical values for the portfolio can be sorted and used to form a distribution. Then the VaR at a given confidence level (e.g., 99%) is computed from the respective quantile of the formed (e.g., 1% lowest value for the 99% case) in the series of simulated portfolio values.

```python
fig, [ax1, ax2, ax3] = plt.subplots(3, 1, sharey=True, )
fig.suptitle('Apple stock returns Distribution \n at different historical windows')

history1 = returns['aapl'][253:]
ax1 = history1.plot(kind='hist', bins=150, ax=ax1)
ax1.axvline(np.percentile(history1, 5), color='orange', label='5%')
ax1.axvline(np.percentile(history1, 1), color='red', label='1%')
ax1.set_title('1 year history', loc='left', fontsize=14)

history2 = returns['aapl'][2*253:]
ax2 = history2.plot(kind='hist', bins=150, ax=ax2)
ax2.axvline(np.percentile(history2, 5), color='orange', label='5%')
ax2.axvline(np.percentile(history2, 1), color='red', label='1%')
ax2.set_title('2 year history', loc='left', fontsize=14)

history3 = returns['aapl'][3*253:]
ax3 = history3.plot(kind='hist', bins=150, ax=ax3)
ax3.axvline(np.percentile(history3, 5), color='orange', label='5%')
ax3.axvline(np.percentile(history3, 1), color='red', label='1%')
ax3.set_title('3 year history', loc='left', fontsize=14)
plt.legend()
plt.show()
```

```python
def hs(returns, alpha=95, time_frame='day'):
    '''Calculates Historical VaR for daily data

    Parameters
    ----------
    returns : array_like with dims [TxN] where N the number of the assets forming the portfolio
        and T the number of the historical data.
        Input array or object that can be converted to an array.
        It is assumed that the values have daily frequency.
    alpha : {float}, optional, default=95
        Probability of occurrence for the amount of loss to compute, which must be between
        0 and 100 inclusive.
    time_frame : {str}, optional, default='day'
        The timeframe of the VaR estimation in days. The default calculates daily VaR.
        Input Options: {'day', 'month', 'quarter', 'year'}
     For the monthly VaR the daily VaR is multiplied by the square root of 21,
     for the quarterly VaR by 63 and for the annual VaR by 253.

    Returns
    -------
    historical VaR in returns terms: scalar or ndarray
        If returns is a single time-series, then the result
        is a scalar. If multiple time-series are given, the result is a vector
        with same length as number of the input series.
    '''
    if time_frame == 'day':
        time = 1
    elif time_frame == 'month':
        time = 21
    elif time_frame == 'quarter':
        time = 63
    elif time_frame == 'year':
        time = 253
    else:
        raise ValueError("'time_frame' input value must be one of {'day', 'month', 'quarter', 'year'}")

    q = 100 - alpha
    var = -np.sqrt(time) * np.percentile(returns, q, axis=0, method='inverted_cdf')
    return np.round(var, 4)
```

* Example: 95% Historical VaR of €1000 imvestment in each stock.

```python
# input params
investment = 1000
prob = 95
# Portfolio VaR calculation for different time horizons
varD = investment * hs(returns, alpha=prob, time_frame='day')
varM = investment * hs(returns, alpha=prob, time_frame='month')
varQ = investment * hs(returns, alpha=prob, time_frame='quarter')
varY = investment * hs(returns, alpha=prob, time_frame='year')
# put results in dataframe
res = np.concatenate((varD.reshape(1, len(varD)),
                varM.reshape(1, len(varM)),
                varQ.reshape(1, len(varQ)),
                varY.reshape(1, len(varY))), axis=0)
var = pd.DataFrame(res, columns=tickers)
var['HS VaR (€)'] = ['Daily', 'Monthly', 'Quarterly', 'Annual']
cols = tickers.copy()
cols.insert(0, "HS VaR (€)")
var = var[cols]
var
```

      HS VaR (€)
      aapl
      goog
      fb
      nvda
      tsla

      0
      Daily
      26.7
      25.5
      29.2
      42.5
      49.6

      1
      Monthly
      122.4
      116.8
      133.9
      194.6
      227.1

      2
      Quarterly
      212.0
      202.4
      231.9
      337.0
      393.4

      3
      Annual
      424.8
      405.5
      464.7
      675.3
      788.3

### Portfolio VaR

To compute the VaR of a portfolio, the correlation $\rho$ among the financial instruments should be taken into account. In this case, the VaR of a portfolio for a given day can be estimated by the following Equation:
\begin{equation} \label{VaR_p}
    VaR^{\alpha}_p = \sqrt{VRV^T}
\end{equation}

where $V$ is a vector of the weighted VaR estimates per instrument $V = [w_{aapl}VaR^{\alpha}_{apple}, w_{goog}VaR^{\alpha}_{goog}, \dots, w_{tsla}VaR^{\alpha}_{tsla}]$ and $R$ is the correlation matrix of stocks'   daily returns.

```python
def portfolio_var(returns, w, alpha=95, time_frame='day'):
    '''Calculate Historical VaR for daily data

    Parameters
    ----------
    returns : array_like with dims [TxN] where N the number of the assets forming the portfolio
        and T the number of the historical data.
        Input array or object that can be converted to an array.
        It is assumed that the values have daily frequency.
    w: list or 1D numpy array with dims [1xN]
        The weight of each asset/timeseries.
        The values of w must be floats between 0 and 1 with their sum equal to 1.

    alpha : {float}, optional, default=95
        Probability of occurrence for the amount of loss to compute, which must be between
        0 and 100 inclusive.
    time_frame : {str}, optional, default='day'
        The timeframe of the VaR estimation in days. The default calculates daily VaR.
        Input Options: {'day', 'month', 'quarter', 'year'}
     For the monthly VaR the daily VaR is multiplied by the square root of 21,
     for the quarterly VaR by 63 and for the annual VaR by 253.

    Returns
    -------
    Portfolio historical VaR in returns terms: scalar
    '''

    w = np.array(w)
    if w.sum() != 1:
        raise ValueError('The sum of w must be equal to 1')
    w = w.reshape(1, returns.shape[1])
    # calculate weighted VaR for each asset forming the portfolio, dims [1xN]
    V = w * hs(returns, alpha=alpha, time_frame=time_frame)
    # calculate correlation matrix with dims [NxN]
    R = returns.corr()
    # calculate portfolio VaR
    var = np.sqrt(V @ R @ V.T)[0][0]
    return np.round(var, 4)
```

* Example: 95% Historical VaR of €1000 investment in a balanced portfolio.

```python
#input params
investment = 1000
prob = 95
weights = np.array([.2, .2, .2, .2, .2])
# Portfolio VaR calculation for different time horizons
varD = investment * portfolio_var(returns, w=weights, alpha=prob, time_frame='day')
varM = investment * portfolio_var(returns, w=weights, alpha=prob, time_frame='month')
varQ = investment * portfolio_var(returns, w=weights, alpha=prob, time_frame='quarter')
varY = investment * portfolio_var(returns, w=weights, alpha=prob, time_frame='year')
# put results in dataframe
res = np.array([[varD], [varM],[varQ],[varY]])
var['portfolio'] = res
var
```

      HS VaR (€)
      aapl
      goog
      fb
      nvda
      tsla
      portfolio

      0
      Daily
      26.7
      25.5
      29.2
      42.5
      49.6
      26.3

      1
      Monthly
      122.4
      116.8
      133.9
      194.6
      227.1
      120.7

      2
      Quarterly
      212.0
      202.4
      231.9
      337.0
      393.4
      209.0

      3
      Annual
      424.8
      405.5
      464.7
      675.3
      788.3
      418.8

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

### Portfolio VaR evaluation

```python
test_day = '2017-01-01'
# select train set according to the selected history window
train = returns[returns.index < test_day]
test = returns[returns.index >= test_day]

# list to save results
hs95 = []
hs99 = []
for day in test.index:
    # calculate VaR for 95% and 99% Confidence
    hs95.append(portfolio_var(train, w=weights, alpha=95, time_frame='day'))
    hs99.append(portfolio_var(train, w=weights, alpha=99, time_frame='day'))
    # move training window
    train = train.append(test[test.index == day])[1:]

# true portfolio returns for the testing period
pnl = (weights * test).sum(axis=1).values
# save results to pandas df
df = pd.DataFrame({"HS95":hs95,"HS99":hs99, "PnL":pnl}, index=test.index)
```

```python
(investment * df).plot()
plt.title('Portfolio VaR')
plt.show()
```

```python
class Backtest:
    def __init__(self, actual, forecast, alpha):
        self.index = actual.index
        self.actual = actual.values
        self.forecast = forecast.values
        self.alpha = (100-alpha)/100

    def hit_series(self):
        return (self.actual < self.forecast) * 1

    def number_of_hits(self):
        return self.hit_series().sum()

    def hit_rate(self):
        return np.round(self.hit_series().mean() * 100, 2)

    def expected_hits(self):
        return self.actual.size * self.alpha
```

```python
bt95 = Backtest(df.PnL, df.HS95, 95)
bt99 = Backtest(df.PnL, df.HS99, 99)
data = [[bt95.expected_hits(), bt95.number_of_hits(), bt95.hit_rate()],
        [bt99.expected_hits(), bt99.number_of_hits(), bt99.hit_rate()]]
cols = ['Expected violations', 'Violations number', 'Violation rate(%)']
idx = ['HS95', 'HS99' ]
pd.DataFrame(data, columns=cols, index=idx)
```

      Expected violations
      Violations number
      Violation rate(%)

      HS95
      62.95
      84
      6.67

      HS99
      12.59
      18
      1.43

```python

```


# Base - Reaction-diffusion

- author: Marcelo de Gomensoro Malheiros
- revision: 2020-09
- license: MIT (attribution is not required but greatly appreciated)

## Turing model
- diffusion rates are based on the ratio $r$ and scale $s$
- depends on explicitly clipping lower values at $L_a$ and $L_b$
- depends on explicitly clipping higher values at $U_a$ and $U_b$

\begin{align}
\frac{\partial{a}}{\partial{t}} &= 16 - a b + r s \nabla^2 a
\\
\\
\frac{\partial{b}}{\partial{t}} &= a b - b - 12 + s \nabla^2 b
\end{align}

```python
import numpy as np
from scipy import ndimage

def turing_model(ma, mb, ka, kb, la, lb, da, db, dt, ia, ib, sa, sb, wrap):

    if wrap:
        ndimage.convolve(ma, ka, output=la, mode='wrap')
        ndimage.convolve(mb, kb, output=lb, mode='wrap')
    else:
        ndimage.convolve(ma, ka, output=la, mode='reflect')
        ndimage.convolve(mb, kb, output=lb, mode='reflect')

    na = ma + (16 - ma * mb      + da * la) * dt
    nb = mb + (ma * mb - mb - 12 + db * lb) * dt

    global a, b
    np.clip(na, ia, sa, out=a)
    np.clip(nb, ib, sb, out=b)
```

## Simulation

Parameter list, with default values shown:
- `ratio=5` - ratio between A and B diffusion rates
- `scale=1` - overall scale of the pattern
- `speed=100` - percent of default time step
- `start=0` - start iteration counter
- `stop=1000` - final iteration counter
- `time=None` - simulation time (in milliseconds), used instead of `stop` and taking into account `speed`
- `use_a=False` - use previous values for A?
- `use_b=False` - use previous values for B?
- `wrap=True` - use toroidal domain if true, non-flux boundary otherwise
- `seed=1` - random seed
- `ini_a=4` - initial constant values for A
- `ini_b=4` - initial constant values for B
- `var_a=0` - added randomness magnitude for A
- `var_b=1` - added randomness magnitude for B
- `shape=40` - domain dimension (single integer for square, or tuple)
- `inf_a=0` - lower bound for A
- `inf_b=0` - lower bound for B
- `sup_a=1000` - upper bound for A
- `sup_b=1000` - upper bound for B
- `axis=False` - show domain size?
- `cmap='inferno'` - Matplotlib colormap used
- `first=False` - show initial state?
- `info=False` - monitor concentration intervals (each 100 iterations, may be changed)
- `limit=None` - plot images with given dimension
- `show='a'` - reagents to show (can be the empty string, either `a` or `b`, or both)
- `size=2` - image output size
- `snap=4` - how many captures are shown (can also be a list of iteration counts)
- `detail=None` - show 1-D plot at given domain row (A in orange and B in blue)
- `extent=(0, 10)` - upper and lower values for 1-D plot
- `func=None` - auxiliary function
- `out=None` - output file name (PDF/PNG/JPG/...)
- `dpi=100` - resolution for output
- `interpolation='bilinear'` - used when converting a matrix to an image
- `detect=False` - detect constant-valued and stable pattern states, besides numerical problems (each 100 iterations, may be changed)
- `model=turing_model` - model function used

Global variables set after the simulation is run:
- `a` - final concentrations for A
- `b` - final concentrations for B
- `sim` - information about the simulation (can be used as an object or a dict)

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# turing: bounded non-linear Turing reaction-diffusion model - v6.3

kernel_a = kernel_b = np.array([[1, 4, 1], [4, -20, 4], [1, 4, 1]]) / 6

class Bunch(dict):
    def __init__(self, dictionary):
        dict.__init__(self, dictionary)
        self.__dict__.update(dictionary)

def turing(ratio=5, scale=1, speed=100, start=0, stop=1000, time=None, use_a=False, use_b=False, wrap=True,
           seed=1, ini_a=4, ini_b=4, var_a=0, var_b=1, shape=40, inf_a=0, inf_b=0, sup_a=1000, sup_b=1000,
           axis=False, cmap='inferno', first=False, info=False, limit=None, show='a', size=2, snap=4,
           detail=None, extent=(0, 10), func=None, out=None, dpi=100, interpolation='bilinear',
           detect=False, model=turing_model):

    # simulation init

    diff_a = ratio * scale
    diff_b = scale
    delta_t = 0.01 * speed / 100
    if time: stop = int(time * 100 / speed) + start

    global sim, a, b
    np.random.seed(seed)
    if type(shape) == int: shape = (shape, shape)
    if not use_a:
        a = np.full(shape, ini_a, dtype=float)
        if var_a != 0: a += np.random.random_sample(shape) * var_a
    if not use_b:
        b = np.full(shape, ini_b, dtype=float)
        if var_b != 0: b += np.random.random_sample(shape) * var_b
    lap_a = np.empty_like(a)
    lap_b = np.empty_like(b)
    is_nan = is_stable = is_uniform = last_a = False
    if info:
        high_a = high_b = - float('inf')
        low_a = low_b = float('inf')
        if info is True: info = 100
    if detect is True: detect = 100

    # plotting helper functions

    def draw(matrix, row):
        if axis: axes[row, col].axis('on')
        axes[row, col].imshow(matrix, cmap=cmap, interpolation=interpolation)
        axes[row, col].set_anchor('N')
        if limit:
            axes[row, col].set_xbound(0, limit[1] - 1)
            axes[row, col].set_ybound(0, limit[0] - 1)

    def plot():
        axes[0, col].set_title(iteration)
        row = 0
        if detail:
            if 'a' in show:
                t = a.copy()
                t[detail - 1,:] = t[detail + 1,:] = a.min()
                draw(t, row); row += 1
            if 'b' in show:
                t = b.copy()
                t[detail - 1,:] = t[detail + 1,:] = b.min()
                draw(t, row); row += 1
        else:
            if 'a' in show: draw(a, row); row += 1
            if 'b' in show: draw(b, row); row += 1

        if detail:
            axes[row, col].axis('on')
            axes[row, col].get_xaxis().set_visible(False)
            axes[row, col].grid()
            axes[row, col].plot((inf_a,) * shape[1], color='orange', linestyle='--')
            axes[row, col].plot((sup_a,) * shape[1], color='orange', linestyle='--')
            axes[row, col].plot(a[detail], color='orange')
            axes[row, col].plot((inf_b,) * shape[1], color='blue', linestyle='--')
            axes[row, col].plot((sup_b,) * shape[1], color='blue', linestyle='--')
            axes[row, col].plot(b[detail], color='blue')
            axes[row, col].set_anchor('N')
            axes[row, col].set_ybound(extent[0], extent[1])

    # plotting init

    axes = ax = ay = col = fig = rows = 0
    if 'a' in show: rows += 1
    if 'b' in show: rows += 1
    if detail: rows += 1
    if type(snap) == int:
        if snap > 100: print("too many captures, check 'snap' parameter"); return
        if first: snap = np.linspace(start, stop, snap, dtype=int)
        else: snap = np.linspace(start, stop, snap + 1, dtype=int)[1:]
    cols = len(snap)
    if show:
        fig, axes = plt.subplots(rows, cols, squeeze=False, figsize=(cols * size, rows * size))
        for ay in axes:
            for ax in ay: ax.axis('off')
    if first and show:
        iteration = start
        plot()
        col += 1
    if type(limit) == int: limit = (limit, limit)

    # simulation loop

    for iteration in range(start + 1, stop + 1):
        if func: func(iteration, seed)

        if detect and iteration % detect == 0: last_a = a.copy()
        if a.shape != shape:
            shape = a.shape
            lap_a = np.empty_like(a)
            lap_b = np.empty_like(b)

        model(a, b, kernel_a, kernel_b, lap_a, lap_b, diff_a, diff_b, delta_t, inf_a, inf_b, sup_a, sup_b, wrap)

        if info and iteration % info == 0:
            high_a = max(a.max(), high_a)
            high_b = max(b.max(), high_b)
            low_a = min(a.min(), low_a)
            low_b = min(b.min(), low_b)
        if detect and iteration % detect == 0:
            if a.ptp() < 0.001 or b.ptp() < 0.001: is_uniform = True
            elif np.isnan(np.sum(a)): is_nan = True
            elif type(last_a) != bool and np.allclose(a, last_a, atol=0.00001, rtol=0): is_stable = True
            last_a = a.copy()

        if is_stable or iteration in snap:
            if show: plot()
            col += 1
        if is_stable or is_uniform or is_nan: break

    # finalization

    if info:
        min_a, max_a, min_b, max_b = a.min(), a.max(), b.min(), b.max()
        print('A [{:.2f}, {:.2f}] <{:.2f}, {:.2f}> '.format(min_a, max_a, low_a, high_a),
              'B [{:.2f}, {:.2f}] <{:.2f}, {:.2f}> '.format(min_b, max_b, low_b, high_b), end='  ')
        if is_stable: print('stability of A at {}'.format(iteration))
        elif is_uniform: print('uniformity of A or B at {}'.format(iteration))
        elif is_nan: print('NaN found in A at {}'.format(iteration))
        else: print()

    if col == 0 or not show: plt.close()
    else:
        plt.show()
        if out: fig.savefig(out, bbox_inches='tight', dpi=dpi)

    del axes, ax, ay, col, cols, draw, fig, last_a, lap_a, lap_b, plot, rows
    sim = Bunch(locals())
```

## Simple validation

```python
turing()
```

## Usage examples

```python
# show A, B and 1D detail view (dashed lines are the lower and upper bounds for concentrations)
# show initial state (iteration 0)
# enforce a lower bound for B and an upper bound for A
turing(show='ab', detail=12, extent=(-1,7), first=True, snap=5, stop=2000, scale=2, inf_b=1.0, sup_a=5.5)
```

```python
# set non-zero 'start' iteration count
# 'time' adjusts number of iterations accordingly to speed
turing(ratio=15, stop=2000)
turing(ratio=15, start=1000, time=2000, speed=50)
```

```python
# skip output, but still stop at a stable pattern
turing(detect=True, info=True, ini_a=4, ini_b=4, var_b=2, stop=10000, show='')
```

    A [2.67, 7.31] <2.67, 8.13>  B [0.00, 8.49] <0.00, 8.51>   stability of A at 8700

```python
# stop prematurely when concentrations turn constant
turing(detect=True, ini_a=3, ini_b=4, var_b=0.1)
print('is uniform?', sim.is_uniform)
```

    is uniform? True

```python
# fields available for simulation object
for i in sorted(sim):
    print(i + '=' + repr(sim[i]))
```

    axis=False
    cmap='inferno'
    delta_t=0.01
    detail=None
    detect=100
    diff_a=5
    diff_b=1
    dpi=100
    extent=(0, 10)
    first=False
    func=None
    inf_a=0
    inf_b=0
    info=False
    ini_a=3
    ini_b=4
    interpolation='bilinear'
    is_nan=False
    is_stable=False
    is_uniform=True
    iteration=200
    limit=None
    model=<function turing_model at 0x7fc1e07a3c20>
    out=None
    ratio=5
    scale=1
    seed=1
    shape=(40, 40)
    show='a'
    size=2
    snap=array([ 250,  500,  750, 1000])
    speed=100
    start=0
    stop=1000
    sup_a=1000
    sup_b=1000
    time=None
    use_a=False
    use_b=False
    var_a=0
    var_b=0.1
    wrap=True


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Bistatic Radar Range Equation
***

The power at the receiving radar for a bistatic configuration is given by (Equation 4.60)

 \begin{equation}
     P_{radar} = \frac{P_t\, G_t(\theta, \phi)\, G_r(\theta, \phi)\, \sigma(\theta, \phi)\, \lambda^2}{(4\pi)^3\, r_t^2\, r_r^2} \hspace{0.5in} \text{(W)}
 \end{equation}
 ***

Begin by getting the library path

```python
import lib_path
```

Set the minimum and maximum range product (m)

```python
range_product_min = 1e6

range_product_max = 1e7
```

Import the `linspace` routine from `scipy`

```python
from numpy import linspace
```

Set up the range product array

```python
range_product = linspace(range_product_min, range_product_max, 2000)
```

Set up the transmit antenna gain (dB), receive antenna gain (dB),  bistatic target RCS (dBsm), peak transmit power (W), and the operating frequency (Hz)

```python
transmit_antenna_gain = 25.0

receive_antenna_gain = 30.0

bistatic_target_rcs = -10.0

peak_power = 50e3

frequency = 1e9
```

Set up the input args

```python
kwargs = {'transmit_target_range': 1.0,

          'receive_target_range': range_product,

          'peak_power': peak_power,

          'transmit_antenna_gain': 10 ** (transmit_antenna_gain / 10.0),

          'receive_antenna_gain': 10 ** (receive_antenna_gain / 10.0),

          'frequency': frequency,

          'bistatic_target_rcs': 10 ** (bistatic_target_rcs / 10.0)}
```

Import the `power_at_radar` routine from `bistatic_radar_range`

```python
from Libs.radar_range.bistatic_radar_range import power_at_radar
```

Calculate the power at the receiving radar

```python
power_at_radar = power_at_radar(**kwargs)
```

Import the `matplotlib` routines and `log10` from `scipy` for displaying the results

```python
from matplotlib import pyplot as plt

from numpy import log10
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(range_product / 1.0e6, power_at_radar, '')

# Set the plot title and labels

plt.title('Bistatic Power at the Receiver', size=14)

plt.xlabel('Range Product (km$^2$)', size=12)

plt.ylabel('Power at Receiver (W)', size=12)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)
```


# 机器学习练习 3 - 多类分类

该代码涵盖了基于Python的解决方案，用于Coursera机器学习课程的第三个编程练习。 有关详细说明和方程式，请参阅[exercise text](ex3.pdf)。

代码修改并注释：黄海广，haiguang2000@qq.com

对于此练习，我们将使用逻辑回归来识别手写数字（0到9）。 我们将扩展我们在练习2中写的逻辑回归的实现，并将其应用于一对一的分类。 让我们开始加载数据集。 它是在MATLAB的本机格式，所以要加载它在Python，我们需要使用一个SciPy工具。

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.io import loadmat

```

```python
data = loadmat('ex3data1.mat')
data
```

    {'X': array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
            [ 0.,  0.,  0., ...,  0.,  0.,  0.],
            [ 0.,  0.,  0., ...,  0.,  0.,  0.],
            ...,
            [ 0.,  0.,  0., ...,  0.,  0.,  0.],
            [ 0.,  0.,  0., ...,  0.,  0.,  0.],
            [ 0.,  0.,  0., ...,  0.,  0.,  0.]]),
     '__globals__': [],
     '__header__': b'MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Sun Oct 16 13:09:09 2011',
     '__version__': '1.0',
     'y': array([[10],
            [10],
            [10],
            ...,
            [ 9],
            [ 9],
            [ 9]], dtype=uint8)}

```python
data['X'].shape, data['y'].shape
```

    ((5000, 400), (5000, 1))

好的，我们已经加载了我们的数据。图像在martix X中表示为400维向量（其中有5,000个）。 400维“特征”是原始20 x 20图像中每个像素的灰度强度。类标签在向量y中作为表示图像中数字的数字类。

训练集的第二部分是一个5000维度的向量y，包含样本的标签。此处‘0’代表‘10’。

# sigmoid 函数
g 代表一个常用的逻辑函数（logistic function）为S形函数（Sigmoid function），公式为： \\[g\left( z \right)=\frac{1}{1+{{e}^{-z}}}\\]
合起来，我们得到逻辑回归模型的假设函数：
	\\[{{h}_{\theta }}\left( x \right)=\frac{1}{1+{{e}^{-{{\theta }^{T}}X}}}\\]

```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
```

代价函数：
$J\left( \theta  \right)=\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]}$

```python
def cost(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    first = np.multiply(-y, np.log(sigmoid(X * theta.T)))
    second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))
    reg = (learningRate / (2 * len(X))) * np.sum(np.power(theta[:,1:theta.shape[1]], 2))
    return np.sum(first - second) / len(X) + reg
```

如果我们要使用梯度下降法令这个代价函数最小化，因为我们未对${{\theta }_{0}}$ 进行正则化，所以梯度下降算法将分两种情形：
\begin{align}
  & Repeat\text{ }until\text{ }convergence\text{ }\!\!\{\!\!\text{ } \\
 & \text{     }{{\theta }_{0}}:={{\theta }_{0}}-a\frac{1}{m}\sum\limits_{i=1}^{m}{[{{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}}]x_{_{0}}^{(i)}} \\
 & \text{     }{{\theta }_{j}}:={{\theta }_{j}}-a\frac{1}{m}\sum\limits_{i=1}^{m}{[{{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}}]x_{j}^{(i)}}+\frac{\lambda }{m}{{\theta }_{j}} \\
 & \text{          }\!\!\}\!\!\text{ } \\
 & Repeat \\
\end{align}

以下是原始代码是使用for循环的梯度函数：

```python
def gradient_with_loop(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)

    parameters = int(theta.ravel().shape[1])
    grad = np.zeros(parameters)

    error = sigmoid(X * theta.T) - y

    for i in range(parameters):
        term = np.multiply(error, X[:,i])

        if (i == 0):
            grad[i] = np.sum(term) / len(X)
        else:
            grad[i] = (np.sum(term) / len(X)) + ((learningRate / len(X)) * theta[:,i])

    return grad
```

向量化的梯度函数

```python
def gradient(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)

    parameters = int(theta.ravel().shape[1])
    error = sigmoid(X * theta.T) - y

    grad = ((X.T * error) / len(X)).T + ((learningRate / len(X)) * theta)

    # intercept gradient is not regularized
    grad[0, 0] = np.sum(np.multiply(error, X[:,0])) / len(X)

    return np.array(grad).ravel()
```

现在我们已经定义了代价函数和梯度函数，现在是构建分类器的时候了。 对于这个任务，我们有10个可能的类，并且由于逻辑回归只能一次在2个类之间进行分类，我们需要多类分类的策略。 在本练习中，我们的任务是实现一对一全分类方法，其中具有k个不同类的标签就有k个分类器，每个分类器在“类别 i”和“不是 i”之间决定。 我们将把分类器训练包含在一个函数中，该函数计算10个分类器中的每个分类器的最终权重，并将权重返回为k X（n + 1）数组，其中n是参数数量。

```python
from scipy.optimize import minimize

def one_vs_all(X, y, num_labels, learning_rate):
    rows = X.shape[0]
    params = X.shape[1]

    # k X (n + 1) array for the parameters of each of the k classifiers
    all_theta = np.zeros((num_labels, params + 1))

    # insert a column of ones at the beginning for the intercept term
    X = np.insert(X, 0, values=np.ones(rows), axis=1)

    # labels are 1-indexed instead of 0-indexed
    for i in range(1, num_labels + 1):
        theta = np.zeros(params + 1)
        y_i = np.array([1 if label == i else 0 for label in y])
        y_i = np.reshape(y_i, (rows, 1))

        # minimize the objective function
        fmin = minimize(fun=cost, x0=theta, args=(X, y_i, learning_rate), method='TNC', jac=gradient)
        all_theta[i-1,:] = fmin.x

    return all_theta
```

这里需要注意的几点：首先，我们为theta添加了一个额外的参数（与训练数据一列），以计算截距项（常数项）。 其次，我们将y从类标签转换为每个分类器的二进制值（要么是类i，要么不是类i）。 最后，我们使用SciPy的较新优化API来最小化每个分类器的代价函数。 如果指定的话，API将采用目标函数，初始参数集，优化方法和jacobian（渐变）函数。 然后将优化程序找到的参数分配给参数数组。

实现向量化代码的一个更具挑战性的部分是正确地写入所有的矩阵，保证维度正确。

```python
rows = data['X'].shape[0]
params = data['X'].shape[1]

all_theta = np.zeros((10, params + 1))

X = np.insert(data['X'], 0, values=np.ones(rows), axis=1)

theta = np.zeros(params + 1)

y_0 = np.array([1 if label == 0 else 0 for label in data['y']])
y_0 = np.reshape(y_0, (rows, 1))

X.shape, y_0.shape, theta.shape, all_theta.shape
```

    ((5000, 401), (5000, 1), (401,), (10, 401))

注意，theta是一维数组，因此当它被转换为计算梯度的代码中的矩阵时，它变为（1×401）矩阵。 我们还检查y中的类标签，以确保它们看起来像我们想象的一致。

```python
np.unique(data['y'])#看下有几类标签
```

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], dtype=uint8)

让我们确保我们的训练函数正确运行，并且得到合理的输出。

```python
all_theta = one_vs_all(data['X'], data['y'], 10, 1)
all_theta
```

    array([[ -2.38335105e+00,   0.00000000e+00,   0.00000000e+00, ...,
              1.30434941e-03,  -7.35542490e-10,   0.00000000e+00],
           [ -3.18549977e+00,   0.00000000e+00,   0.00000000e+00, ...,
              4.45902426e-03,  -5.08371406e-04,   0.00000000e+00],
           [ -4.79997891e+00,   0.00000000e+00,   0.00000000e+00, ...,
             -2.87612799e-05,  -2.47948035e-07,   0.00000000e+00],
           ...,
           [ -7.98390460e+00,   0.00000000e+00,   0.00000000e+00, ...,
             -8.96271728e-05,   7.22810041e-06,   0.00000000e+00],
           [ -4.57356699e+00,   0.00000000e+00,   0.00000000e+00, ...,
             -1.33528780e-03,   9.97862678e-05,   0.00000000e+00],
           [ -5.40564903e+00,   0.00000000e+00,   0.00000000e+00, ...,
             -1.16550490e-04,   7.87549333e-06,   0.00000000e+00]])

我们现在准备好最后一步 - 使用训练完毕的分类器预测每个图像的标签。 对于这一步，我们将计算每个类的类概率，对于每个训练样本（使用当然的向量化代码），并将输出类标签为具有最高概率的类。

```python
def predict_all(X, all_theta):
    rows = X.shape[0]
    params = X.shape[1]
    num_labels = all_theta.shape[0]

    # same as before, insert ones to match the shape
    X = np.insert(X, 0, values=np.ones(rows), axis=1)

    # convert to matrices
    X = np.matrix(X)
    all_theta = np.matrix(all_theta)

    # compute the class probability for each class on each training instance
    h = sigmoid(X * all_theta.T)

    # create array of the index with the maximum probability
    h_argmax = np.argmax(h, axis=1)

    # because our array was zero-indexed we need to add one for the true label prediction
    h_argmax = h_argmax + 1

    return h_argmax
```

现在我们可以使用predict_all函数为每个实例生成类预测，看看我们的分类器是如何工作的。

```python
y_pred = predict_all(data['X'], all_theta)
correct = [1 if a == b else 0 for (a, b) in zip(y_pred, data['y'])]
accuracy = (sum(map(int, correct)) / float(len(correct)))
print ('accuracy = {0}%'.format(accuracy * 100))
```

    accuracy = 94.46%

在下一个练习中，我们将介绍如何从头开始实现前馈神经网络。

# 神经网络模型图示

```python

```

```python

```


# 拉普拉斯特征映射(Laplacian Eigenmap, LE)

## 符号定义

|符号|含义|
|:-:|:-:|
|$\pmb{x}$|样本点|
|$X_i$|样本点$\pmb{x_i}$的k近邻点集合|
|$N$|总样本数|
|$N_i$|样本点$\pmb{x_i}$的k近邻点集合的样本总数|
|$\pmb{X_i^{(j)}}$|样本点$\pmb{x_i}$的k近邻点集合中的第j个样本点|
|$W$|邻接矩阵|
|$\pmb{y}$|样本点降维后坐标|

## 概念
LE和LLE类似，均是从“局部”的角度来构建数据与数据之间的联系和区别。不同点在于：LLE认为任意一个数据点均可以由其最近邻的k个点线性表示，并且这种线性表示在降维后依然成立；LE认为任意一个数据点和其最近邻的k个点在降维后也应当尽可能相近，即在降维后尽可能保证数据间的结构不发生改变。

显然，LLE的优化目标和LE的优化目标不同。

对于LLE来说，降维前后数据点之间的距离变化并不是很重要，只需要降维后依然能够满足降维前的局部线性关系即可。在实际例子中会发现，降维后，某些数据点的k个最近邻点发生了变化（数据之间的结构发生了变化），但是该数据点依然可以由变换前的k个最近邻点的降维结果以相同的权重线性组合得到。

对于LE来说，降维前后数据点之间的线性关系是否保持并不是很重要，只要降维后相近的数据点依然尽可能相近即可。即降维前后，任意数据点的k个最近邻点不会发生变化。

上述分析暗示了LE的核心步骤：

1. 在高维空间中对数据间的结构进行描述
2. 在低维空间中寻找满足这一结构的集合

对于第一步，LE希望关系的描述有如下特点：
* 数据点A和数据点B关系的描述是唯一的。即无论是从数据点A出发对AB关系进行描述还是从数据点B出发对AB关系进行描述均是相同的（无向图）。
* 对关系的描述是局部的。即仅描述数据点A与其周围最为接近的数个数据点之间的关系，不会考虑数据点A与所有数据点之间的关系。

若将关系表示为邻接矩阵，则上述的第一条要求使得邻接矩阵为对称矩阵，上述的第二条要求使得关系矩阵为稀疏矩阵。

对于第二步，实际上就是解决一个优化问题。其优化目标就是：找到一个集合，使得集合中的元素与高维空间中的数据点一一对应，并且这些元素之间的空间关系与高维空间中数据点之间的关系尽可能相似。

## 推导

### 高维空间中的邻接矩阵

k近邻能很方便的找到任意数据点k个最近邻的数据点，这些最近邻的点即为空间中与当前数据点最为接近的数据点的集合。因此用k近邻对定义两个数据点之间是否存在关系是非常合适的。

但是k近邻存在一个明显的漏洞： **若数据点A是数据点B的最近邻点，那么数据点B不一定是数据点A的最近邻点。** 即两个数据点之间的关系不是相互的。但是根据上一节的分析，对于LE来说，希望关系是相互的。

这一漏洞有两种处理方法：

1. 仅考虑互为近邻的关系，即仅认为互为近邻的关系才是有效关系，而其他关系全部置零
2. 对于使用矩阵表示的关系，可以简单的使用原关系矩阵和其转置的和的$\frac{1}{2}$来表示

上述均是在寻找存在“局部”关系的点，对于关系的具体计算也有多种方式

1. 使用径向基函数（RBF）进行描述。较近的点取值大，较小的点取值较小。例如类似于下式的形式，其中$\gamma>0$为需要给定的参数。
$$
\begin{equation}
    \pmb{w_{ij}} = \exp^{-\gamma||\pmb{x_i}-\pmb{x_j}||_2^2}
\end{equation}
$$
2. 简单使用$\{0, 1\}$描述，对于存在关系取$1$，不存在关系则取$0$

通过上述两步，即可完成高维空间中局部关系的描述，并得到描述这些关系的邻接矩阵。

### 低维表示求解

对于上一步获得的邻接矩阵$W$，希望在降维后依然能够保持，因此可以定义如下的损失函数

$$
\begin{equation}
    \mathcal{L} = \sum_{ij}||\pmb{y_i}-\pmb{y_j}||_2^2w_{ij}
\end{equation}
$$

上式实际上衡量了在高维空间中存在关系的数据点降维后的距离。当在高维空间中存在关系的两个点尽可能接近时，损失越小。而且不存在关系($w_{ij}=0$)的两个数据点之间的距离并不会影响损失。

考虑到邻接矩阵为对称矩阵，因此有

$$
\begin{equation}
    \begin{split}
    \mathcal{L}
    &= \sum_{ij}||\pmb{y_i}-\pmb{y_j}||_2^2w_{ij} \\
    &= \sum_{i=1}^N\sum_{j=1}^N||\pmb{y_i}-\pmb{y_j}||_2^2w_{ij} \\
    &= \sum_{i=1}^N\sum_{j=1}^N(\pmb{y_i}^T\pmb{y_i}-2\pmb{y_i}^T\pmb{y_j}+\pmb{y_j}^T\pmb{y_j})w_{ij} \\
    &= \sum_{i=1}^N(\sum_{j=1}^Nw_{ij})\pmb{y_i}^T\pmb{y_i}+\sum_{j=1}^N(\sum_{i=1}^Nw_{ij})\pmb{y_j}^T\pmb{y_j} - 2\sum_{i=1}^N\sum_{j=1}^N\pmb{y_i}^T\pmb{y_j}w_{ij} \\
    &= 2\sum_{i=1}^N(\sum_{j=1}^Nw_{ij})\pmb{y_i}^T\pmb{y_i} - 2\sum_{i=1}^N\sum_{j=1}^N\pmb{y_i}^T\pmb{y_j}w_{ij} \\
    &= 2\sum_{i=1}^N(\sqrt{\sum_{j=1}^Nw_{ij}}\pmb{y_i})^T(\sqrt{\sum_{j=1}^Nw_{ij}}\pmb{y_i}) - 2\sum_{i=1}^N\pmb{y_i}^T(\sum_{j=1}^N\pmb{y_j}w_{ij}) \\
    &= 2tr(Y^TDY) - 2tr(Y^TWY) \\
    &= 2tr(Y^TLY)
    \end{split}
\end{equation}
$$

其中$L$矩阵即为拉普拉斯矩阵
$$
\begin{equation}
    D = diag(\sum_{i=1}^Nw_{i1}, \sum_{i=1}^Nw_{i2}, \cdots, \sum_{i=1}^Nw_{iN})
\end{equation}
$$
$$
\begin{equation}
    L = D-W
\end{equation}
$$

因此优化目标为

$$
\begin{equation}
    \begin{split}
    \arg\min\limits_{Y}\mathcal{L}
    &\Rightarrow \arg\min\limits_{Y} tr(Y^TLY)
    \end{split}
\end{equation}
$$

为了消除数据缩放对结果的影响，实际使用时会使用$Y^TDY=\pmb{I}$的限制条件，因此最终的优化目标为

$$
\begin{equation}
    \arg\min\limits_{Y} tr(Y^TLY), s.t. Y^TDY=\pmb{I}
\end{equation}
$$

考虑使用拉格朗日乘子法对上式进行求解

$$
\begin{equation}
    \left\{
    \begin{split}
    &\frac{\partial{\mathcal{L}}}{\partial{Y}}-\lambda\frac{\partial{(Y^TDY-\pmb{I})}}{\partial{Y}} = 0 \\
    & Y^TDY-\pmb{I}=0
    \end{split}
    \right.
\end{equation}
$$

$$
\begin{equation}
    \begin{split}
    & LY+L^TY-2\lambda DY=0 \\
    \Rightarrow &LY=\lambda DY
    \end{split}
\end{equation}
$$

上式即为一个广义特征值以及特征向量的求解问题。

将上式回代到原损失计算式有
$$
\begin{equation}
    \begin{split}
    \mathcal{L}
    &= 2tr(Y^TLY) \\
    &= 2tr(\lambda Y^TDY) \\
    &= 2tr(\lambda \pmb{I})
    \end{split}
\end{equation}
$$

考虑到损失应当尽可能小，因此选择前d个最小广义特征值对应的广义特征向量。

由于矩阵$L$和矩阵$D$均为对称半正定矩阵，因此其特征值必大于等于0

下面讨论特征值等于0的情况

若广义特征值等于0，则有

$$
\begin{equation}
    \begin{split}
    L\pmb{y} &= \pmb{0} \\
    \Rightarrow D\pmb{y} &= W\pmb{y} \\
    \end{split}
\end{equation}
$$

考虑到$D = diag(\sum_{i=1}^Nw_{i1}, \sum_{i=1}^Nw_{i2}, \cdots, \sum_{i=1}^Nw_{iN})$，当$\pmb{y}$为全1矩阵时，上式显然成立。

若不添加任何限制条件，上述的损失显然存在非常严重的问题： **对没有局部关系的数据点之间的距离没有计算损失。** 这一问题导致若所有的数据点降维后均为同一个点，则$\mathcal{L}=0$，从损失的角度来看有最优解。但是，显然，若任意数据点降维后均为同一个点，这样的降维是没有意义的，此时对应的实际上就是广义特征值为0的情况。

因此在实际使用时，选择舍弃掉0这个广义特征值对应的广义特征向量，并选用后续的d个广义特征值对应的广义特征向量作为最终的解。

## 流程

* 定义数据集$X=\{\pmb{x_1}, \pmb{x_2}, \cdots, \pmb{x_N}\}$、邻域大小$k$以及降维后维度$d$
* 计算邻接矩阵
* 计算拉普拉斯矩阵和度矩阵
$$
\begin{equation}
    L = D-W
\end{equation}
$$
$$
\begin{equation}
    D = diag(\sum_{i=1}^Nw_{i1}, \sum_{i=1}^Nw_{i2}, \cdots, \sum_{i=1}^Nw_{iN})
\end{equation}
$$
* 计算下式的前d+1个最小广义特征值对应的特征向量，并取$[1, d+1]$个作为最终的解
$$
\begin{equation}
    L\pmb{y}=\lambda D\pmb{y}
\end{equation}
$$

```python
import numpy as np
from scipy import linalg
import matplotlib.pyplot as plt
from sklearn import manifold
from sklearn.neighbors import NearestNeighbors
```

```python
class MyLE(object):

    def __init__(self, input_data, reduction_dims, n_neighbors):

        self.input_data = input_data
        self.reduction_dims = reduction_dims
        self.n_neighbors = n_neighbors

        self.samples_num, self.input_dims = self.input_data.shape

        # affinity matrix
        self.affinity_mat = np.zeros((self.samples_num, self.samples_num))

        # reduction result
        self.output_mat = np.zeros((self.samples_num, reduction_dims))

    def __call__(self):
        self.cal_affinity_mat()
        self.cal_reduction_result()
        return self.output_mat

    def cal_affinity_mat(self):
        nbrs = NearestNeighbors(n_neighbors=self.n_neighbors).fit(self.input_data)
        self.affinity_mat = nbrs.kneighbors_graph(self.input_data).toarray()

        self.affinity_mat = 0.5 * (self.affinity_mat + self.affinity_mat.T)

    def cal_reduction_result(self):

        # cal matrix D
        d_mat = np.diag(np.sum(self.affinity_mat, axis=0).reshape(-1))

        # cal matrix L
        l_mat = d_mat - self.affinity_mat

        # cal top k eighvalues and eighvectors
        _, topk_e_vecs = self._topk_eigh(l_mat, d_mat, self.reduction_dims)

        self.output_mat = np.array(topk_e_vecs)
        return self.output_mat

    def _topk_eigh(self, mat1, mat2, k, skip_num=1):
        e_vals, e_vecs = linalg.eigh(mat1, mat2)

        sorted_ind = np.argsort(e_vals)

        topk_e_vals = e_vals[sorted_ind[skip_num:k+skip_num]]
        topk_e_vecs = e_vecs[:, sorted_ind[skip_num:k+skip_num]]
        return topk_e_vals, topk_e_vecs
```

```python
# ---------------------- create data ----------------------
n_samples = 3000

# Create our sphere.
np.random.seed(1024)
p = np.random.rand(n_samples) * (2 * np.pi - 0.55)
t = np.random.rand(n_samples) * np.pi

# Sever the poles from the sphere.
indices = (t < (np.pi - (np.pi / 8))) & (t > ((np.pi / 8)))
samples_color = p[indices]
x, y, z = (
    np.sin(t[indices]) * np.cos(p[indices]),
    np.sin(t[indices]) * np.sin(p[indices]),
    np.cos(t[indices]),
)
raw_samples = np.array([x, y, z]).T

#  ---------------------- sklearn LE ----------------------
sklearn_le = manifold.SpectralEmbedding(n_components=2, n_neighbors=20)
sklearn_le_result = sklearn_le.fit_transform(raw_samples)

# ----------------------- My LE -----------------------
my_le = MyLE(raw_samples, reduction_dims=2, n_neighbors=20)
my_le_result = my_le()

# ----------------------
# draw func
fig = plt.figure(figsize=(20, 20))

ax = fig.add_subplot(221, projection="3d")
ax.set_title("Original data", fontsize=15)
ax.scatter(raw_samples[:, 0], raw_samples[:, 1], raw_samples[:, 2], c=samples_color, cmap=plt.cm.Spectral)

ax = fig.add_subplot(222)
plt.xticks([])
plt.yticks([])
plt.title("Projection of data using Sklearn LE", fontsize=15)
ax.scatter(sklearn_le_result[:, 0], sklearn_le_result[:, 1], c=samples_color, cmap=plt.cm.Spectral)

ax = fig.add_subplot(223, projection="3d")
ax.set_title("Original data", fontsize=15)
ax.scatter(raw_samples[:, 0], raw_samples[:, 1], raw_samples[:, 2], c=samples_color, cmap=plt.cm.Spectral)

ax = fig.add_subplot(224)
plt.xticks([])
plt.yticks([])
plt.title("Projection of data using My LE", fontsize=15)
ax.scatter(my_le_result[:, 0], my_le_result[:, 1], c=samples_color, cmap=plt.cm.Spectral)

plt.show()
```


```python
%matplotlib inline
```

Introduction to PyTorch
============

PyTorch's tensor library
------------------------------------

The most of PyTorch operations are running on tensors.
A tensor is an multidimensional array.
Lets have a look on some basic tensor operations.
But first, lets import some important PyTorch libraries:
- torch - a Tensor library similar to NumPy, with strong GPU support
- torch.autograd - a "tape-based" (about this - later on) automatic differentiation library
- torch.nn - a neural networks library deeply integrated with autograd
- torch.optim - an optimization package to be used with torch.nn with standard optimization methods such as SGD, RMSProp, LBFGS, Adam etc.

We also set a seed to be able to reproduce the same results later.

```python
import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.optim as optim

torch.manual_seed(123)
```

    <torch._C.Generator at 0x7fe41fd172f0>

Creating Tensors
----------------

Tensors can be created from Python lists with the torch.Tensor() function.

```python
# Create a torch.Tensor object from python list
v = [1, 2, 3]
print(type(v))
v_tensor = torch.Tensor(v)
print(v_tensor)

# Create a torch.Tensor object of size 2x3 from 2x3 matrix
m2x3 = [[1, 2, 3], [4, 5, 6]]
m2x3_tensor = torch.Tensor(m2x3)
print(m2x3_tensor)

# Create a 3D torch.Tensor object of size 3x3x3.
m3x3x3 = [[[1, 2, 3], [4, 5, 6], [7, 8, 9]],
          [[10, 11, 12],[13, 14, 15], [16, 17, 18]],
            [[19, 20, 21],[22, 23, 24], [25, 26, 27]]]
m3x3x3_tensor = torch.Tensor(m3x3x3)
print(m3x3x3_tensor)

#Create a 4Dtensor from random data and given dimensions (in this case 3x4x5x6) with torch.randn()
m4x3x3x3_tensor = torch.randn((4, 3, 3, 3))
m4x3x3x3_tensor.shape
print(m4x3x3x3_tensor)
```

    <type 'list'>

     1
     2
     3
    [torch.FloatTensor of size 3]

     1  2  3
     4  5  6
    [torch.FloatTensor of size 2x3]

    (0 ,.,.) =
       1   2   3
       4   5   6
       7   8   9

    (1 ,.,.) =
      10  11  12
      13  14  15
      16  17  18

    (2 ,.,.) =
      19  20  21
      22  23  24
      25  26  27
    [torch.FloatTensor of size 3x3x3]

    (0 ,0 ,.,.) =
      0.4934 -0.2766  0.2439
     -1.2116 -0.1520  0.1509
     -0.6251 -0.4416  0.3208

    (0 ,1 ,.,.) =
     -0.3273 -0.5305 -0.0172
      0.4719  0.5671  2.7930
      0.3229  0.8552  0.7492

    (0 ,2 ,.,.) =
     -1.7119  0.6025 -0.7018
     -1.3130  0.1574  2.0114
      0.1004  0.8222 -0.0176

    (1 ,0 ,.,.) =
      1.2481 -0.0710  2.1627
      1.5215 -1.0547  1.7822
      1.9736 -0.3101 -0.8211

    (1 ,1 ,.,.) =
      0.1315 -0.6948 -0.5823
      1.0035 -1.4613  0.8985
      0.6210 -0.9679  0.6740

    (1 ,2 ,.,.) =
     -1.2828 -0.5097  0.1464
     -0.4860 -0.7529  1.6989
      0.4991 -2.1702  0.5130

    (2 ,0 ,.,.) =
     -1.9029  0.8260 -0.6644
      1.6663 -0.5704  1.3906
     -1.4855  0.2987 -0.3029

    (2 ,1 ,.,.) =
      0.3354  1.0599  0.1941
     -0.9295  0.5329 -1.1307
      0.1024  2.5200 -1.2324

    (2 ,2 ,.,.) =
     -0.8294  0.4342 -0.7374
      0.1591 -1.3560  0.5513
      0.3732  1.4246 -0.6860

    (3 ,0 ,.,.) =
     -0.4791  2.2610  1.3191
      1.0130  1.6650  0.8469
      0.2564  0.8049 -0.3107

    (3 ,1 ,.,.) =
     -0.9208 -0.1507 -0.6977
      1.5227  2.4308  1.5695
     -0.0657 -0.4277  0.2537

    (3 ,2 ,.,.) =
      0.7337  1.1388  0.5516
     -0.6187 -0.6503  0.3017
      0.6113  0.4344 -0.3423
    [torch.FloatTensor of size 4x3x3x3]

What is a multidimensional tensor?
-------------------
Since we frequently deal with n > 3 dimensional tensors, its understanding is very important.
The best way to think of a higher (n) dimensional object (and tensor in particular) is as of a container which keeps a series of n-1 dimensional objects "inside" of it. We can "pull out" these "inner" objects by indexing in to higher dimensional tensor container.
Let's have a look on some examples:

- For a vector v (dim(v)=1), indexing into it ("pulling out of it") returns its "slice" - a scalar s (dim(s)=0).

- For a matrix, indexing into it returns its "slice" - a (row or column) vector.

- 3D tensor can be seen as a cube or 3D rectangular consisting of horizontally "stacked" matrices. So if we index into a such tensor it will give us its slice which is a matrix!

- We can't easily visualize 5D (or n-D) tensors, but the idea is actually the same. If we index in to them, we will pull out an object of dimension n-1.

- E.g. a 4D tensor can be seen as a list of cubes or 3D reactangulars. If we index in to a 4D tensor, we will get 3D rectangulars.

```python
# Index into v_tensor and get a scalar
print(v_tensor[0])

# Index into m2x3_tensor and get a vector
print(m2x3_tensor[0])

# Index into m3x3x3_tensor and get a matrix
print(m3x3x3_tensor[0])

# Index into m4x3x3x3_tensor and get a 3D rectangular of size 4x5x6
print(m4x3x3x3_tensor[0])
```

    1.0

     1
     2
     3
    [torch.FloatTensor of size 3]

     1  2  3
     4  5  6
     7  8  9
    [torch.FloatTensor of size 3x3]

    (0 ,.,.) =
     -0.1115  0.1204 -0.3696
     -0.2404 -1.1969  0.2093
     -0.9724 -0.7550  0.3239

    (1 ,.,.) =
     -0.1085  0.2103 -0.3908
      0.2350  0.6653  0.3528
      0.9728 -0.0386 -0.8861

    (2 ,.,.) =
     -0.4709 -0.4269 -0.0283
      1.4220 -0.3886 -0.8903
     -0.9601 -0.4087  1.0764
    [torch.FloatTensor of size 3x3x3]

Operations with Tensors
----------------------

You can operate on tensors in the ways you would expect.
See the documentation <http://pytorch.org/docs/torch.html> for a complete list of operations.

Simple mathematical operations: Addition, Multiplication

```python
x = torch.Tensor([1, 2, 3])
y = torch.Tensor([4, 5, 6])
print(x)
print(y)

w = torch.matmul(x, y)
print(w)
```

     1
     2
     3
    [torch.FloatTensor of size 3]

     4
     5
     6
    [torch.FloatTensor of size 3]

    32.0

Helpful operation: Concatenation

```python
# By default, it concatenates along the axis with 0 (rows). It's "stacking" the rows.

x_1 = torch.randn(2, 5)
print(x_1)
y_1 = torch.randn(3, 5)
print(y_1)
z_1 = torch.cat([x_1, y_1])
print(z_1)

# Concatenate columns:
x_2 = torch.randn(2, 3)
print(x_2)
y_2 = torch.randn(3, 5)
print(y_2)
# second arg specifies which axis to concat along. Here we select 1 (columns). It's attaching the columns.
z_2 = torch.cat([x_2, y_2], 1)
print(z_2)

# If your tensors are not compatible, torch will complain.  Uncomment to see the error
torch.cat([x_1, x_2])
```

Reshaping Tensors
----------------

We can use the .view() method to reshape a tensor. Often we will need to reshape our data before passing it
to a neuronal network.

Let's assume we have 64000 RGB images with the size of 28x28 pixels.
We can define an array fo shape (64000, 3, 28, 28) to hold them, where 3 is number of color channels:

```python
x = torch.randn(64000, 3, 28, 28)
# Now we want to add a batch dimension of size 32. We can then infer the second dimension by placing -1:
x_rehsaped = x.view(32, -1, 3, 28, 28)
print(x_rehsaped.shape)
```

    torch.Size([32, 2000, 3, 28, 28])

Computation Graphs and Automatic Differentiation
---------------------------------------------

A computation graph is a specification of what parameters with which operations are involved in the computation to give the output.

The fundamental class of Pytorch autograd.Variable keeps track of how it was created.

```python
# Variables wrap tensor objects
x = autograd.Variable(torch.Tensor([1, 2, 3]), requires_grad=True)
# You can access the data with the .data attribute
print(x.data)

y = autograd.Variable(torch.Tensor([4, 5, 6]), requires_grad=True)

# With autograd.Variable you can also perform all the same operations you did with tensors
z = x + y
print(z.data)

#  w knows also that it's result of addition of z lements (AddBackward)
operation = z.grad_fn
print(operation)

```

     1
     2
     3
    [torch.FloatTensor of size 3]

     5
     7
     9
    [torch.FloatTensor of size 3]

    <AddBackward1 object at 0x7fe41361ffd0>

The autograd.Variable knows which operation has created it. But how does that help compute a gradient?

```python
# Lets sum up all the entries in z
s = z.sum()
print(s)
print(s.grad_fn)
```

    Variable containing:
     21
    [torch.FloatTensor of size 1]

    <SumBackward0 object at 0x7fe413507cd0>

Gradient
-------
So now, what is the derivative of this sum with respect to the first component of x? Remember, that x is a tensor of 3 elements: $x = (x_0, x_1, x_2)$

In math, we want a partial derivative of $s$ with respect to $x_0$: $\frac{\partial s}{\partial x_0}$

Well, $s$ knows that it was created as a $sum$ of the tensor $z$ elements $(z_0, z_1, z_2)$. $z$ knows
that it was the sum $x + y$. So

\begin{align}s = \overbrace{x_0 + y_0}^\text{$z_0$} + \overbrace{x_1 + y_1}^\text{$z_1$} + \overbrace{x_2 + y_2}^\text{$z_2$}\end{align}

And so $s$ contains enough information to determine that the derivative of $s$ with respect to $x_0$ is 1!

*Reminder:* If you compute the partial derivative with respekt to one variable, you handle all other variables as constants. Therefore they all $(x_1, x_2, y_0, y_1, y_2)$ get zeroes, and the derivative of $f(x_0) = x_0$ is 1.

First we need to run backpropagation and calculate gradients with respect to every variable.
*Note:* if you run backward multiple times, the gradient will increment.
That is because Pytorch *accumulates* the gradient into the .grad
property, since for many models this is very convenient.
Lets now have Pytorch compute the gradient, and see that we were right with our guess of 1:

```python
# calling .backward() on any variable will run backprop, starting from it.
s.backward(retain_graph=True)

```

```python
print(x)
print(x.grad)
print(y.grad)
```

    Variable containing:
     1
     2
     3
    [torch.FloatTensor of size 3]

    Variable containing:
     3
     3
     3
    [torch.FloatTensor of size 3]

    Variable containing:
     3
     3
     3
    [torch.FloatTensor of size 3]

How NOT to break the computational graph
----------------------------------

Let's create two torch tensors and add them up:

```python
x = torch.randn((2, 2))
y = torch.randn((2, 2))
z = x + y  # These are Tensor types, and backprop would not be possible

print(z)

```

     0.1730  1.8913
     0.1251  0.1286
    [torch.FloatTensor of size 2x2]

Now we wrap the torch tensors in autograd.Variable. The var_z contains the information for backpropagation:

```python
var_x = autograd.Variable(x, requires_grad=True)
var_y = autograd.Variable(y, requires_grad=True)
# var_z contains enough information to compute gradients, as we saw above
var_z = var_x + var_y
print(var_z.grad_fn)
```

    <AddBackward1 object at 0x7fe41352c110>

But what happens if we extract the wrapped tensor object out of var_z and re-wrap the tensor in a new autograd.Variable?

```python
var_z_data = var_z.data
new_var_z = autograd.Variable(var_z_data)
print(new_var_z.grad_fn)
```

    None

The variable chain is not existing anymore, since we have extracted only data and the whole operations chain was lost.
If we try now to compute backward on new_var_z, it will throw an error:

```python
new_var_z.backward(retain_graph=True)
```

CUDA
----
Check wether GPU accelaration with **CUDA** is available

```python
# let us run this cell only if CUDA is available
if torch.cuda.is_available():
    # creates a LongTensor and transfers it
    # to GPU as torch.cuda.LongTensor
    a = torch.LongTensor(10).fill_(3).cuda()
    print(type(a))
    b = a.cpu()
    # transfers it to CPU, back to
    # being a torch.LongTensor
```

Linear Model
=======

```python
import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
```

```python
x = [i for i in range(20)] #list comprehention
x_train = np.array(x, dtype=np.float32)
x_train = x_train.reshape(-1, 1)
print(x)
print(x_train.shape)

y = [(5*i + 2) for i in x] #list comprehention
y_train = np.array(y, dtype=np.float32)
y_train = y_train.reshape(-1, 1)
print(y)
print(y_train.shape)

```

    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
    (20, 1)
    [2, 7, 12, 17, 22, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97]
    (20, 1)

Create Model Class
-----------------

```python

class LinearRegressor(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressor, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        out = self.linear(x)
        return out

input_dim = 1
output_dim = 1

model = LinearRegressor(input_dim, output_dim)

model
```

    LinearRegressor(
      (linear): Linear(in_features=1, out_features=1)
    )

Loss & Optimizer
---------------

```python
loss_function = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
optimizer
loss_function
```

    MSELoss(
    )

```python
epochs = 500

for epoch in range(epochs):
    epoch += 1
    #Convert inputs and outputs to torch variable
    inputs = Variable(torch.from_numpy(x_train))

    real_outputs = Variable(torch.from_numpy(y_train))

    # Reset Gradients
    optimizer.zero_grad()

    # Forward - compute the output
    pred_outputs = model(inputs)

    # Loss
    loss = loss_function(pred_outputs, real_outputs)

    # Backword - compute gradients
    loss.backward()

    # Update parameters
    optimizer.step()

    print('epoch {}, loss {}'.format(epoch, loss.data[0]))
```

    epoch 1, loss 3330.96435547
    epoch 2, loss 1881.8626709
    epoch 3, loss 1063.4017334
    epoch 4, loss 601.130065918
    epoch 5, loss 340.03616333
    epoch 6, loss 192.568328857
    epoch 7, loss 109.277175903
    epoch 8, loss 62.2333488464
    epoch 9, loss 35.6621856689
    epoch 10, loss 20.6541633606
    epoch 11, loss 12.1769609451
    epoch 12, loss 7.38844966888
    epoch 13, loss 4.68333339691
    epoch 14, loss 3.15493202209
    epoch 15, loss 2.29113745689
    epoch 16, loss 1.80271029472
    epoch 17, loss 1.52630400658
    epoch 18, loss 1.36964297295
    epoch 19, loss 1.28061759472
    epoch 20, loss 1.22979319096
    epoch 21, loss 1.20054602623
    epoch 22, loss 1.18348562717
    epoch 23, loss 1.17330884933
    epoch 24, loss 1.16701996326
    epoch 25, loss 1.16292822361
    epoch 26, loss 1.16007828712
    epoch 27, loss 1.15793061256
    epoch 28, loss 1.15617823601
    epoch 29, loss 1.15465211868
    epoch 30, loss 1.15325260162
    epoch 31, loss 1.15192627907
    epoch 32, loss 1.15064108372
    epoch 33, loss 1.14937984943
    epoch 34, loss 1.14813303947
    epoch 35, loss 1.14689469337
    epoch 36, loss 1.14566218853
    epoch 37, loss 1.14443290234
    epoch 38, loss 1.14320623875
    epoch 39, loss 1.14198327065
    epoch 40, loss 1.14075922966
    epoch 41, loss 1.13953721523
    epoch 42, loss 1.13831853867
    epoch 43, loss 1.13709974289
    epoch 44, loss 1.13588225842
    epoch 45, loss 1.13466560841
    epoch 46, loss 1.13345181942
    epoch 47, loss 1.13223946095
    epoch 48, loss 1.13102686405
    epoch 49, loss 1.12981712818
    epoch 50, loss 1.12860763073
    epoch 51, loss 1.12740015984
    epoch 52, loss 1.12619328499
    epoch 53, loss 1.12498831749
    epoch 54, loss 1.12378358841
    epoch 55, loss 1.12258088589
    epoch 56, loss 1.12137913704
    epoch 57, loss 1.12017941475
    epoch 58, loss 1.11898028851
    epoch 59, loss 1.11778283119
    epoch 60, loss 1.11658644676
    epoch 61, loss 1.11539149284
    epoch 62, loss 1.11419713497
    epoch 63, loss 1.11300492287
    epoch 64, loss 1.11181366444
    epoch 65, loss 1.11062431335
    epoch 66, loss 1.10943520069
    epoch 67, loss 1.10824799538
    epoch 68, loss 1.10706198215
    epoch 69, loss 1.10587668419
    epoch 70, loss 1.10469269753
    epoch 71, loss 1.10351061821
    epoch 72, loss 1.10232949257
    epoch 73, loss 1.10114979744
    epoch 74, loss 1.09997189045
    epoch 75, loss 1.09879422188
    epoch 76, loss 1.09761846066
    epoch 77, loss 1.09644293785
    epoch 78, loss 1.09526956081
    epoch 79, loss 1.09409844875
    epoch 80, loss 1.09292626381
    epoch 81, loss 1.09175646305
    epoch 82, loss 1.09058868885
    epoch 83, loss 1.0894215107
    epoch 84, loss 1.08825469017
    epoch 85, loss 1.08709037304
    epoch 86, loss 1.085927248
    epoch 87, loss 1.0847645998
    epoch 88, loss 1.08360350132
    epoch 89, loss 1.08244431019
    epoch 90, loss 1.08128559589
    epoch 91, loss 1.08012807369
    epoch 92, loss 1.07897210121
    epoch 93, loss 1.07781684399
    epoch 94, loss 1.07666349411
    epoch 95, loss 1.07551205158
    epoch 96, loss 1.07435965538
    epoch 97, loss 1.0732101202
    epoch 98, loss 1.07206165791
    epoch 99, loss 1.07091379166
    epoch 100, loss 1.06976830959
    epoch 101, loss 1.06862294674
    epoch 102, loss 1.06747913361
    epoch 103, loss 1.06633603573
    epoch 104, loss 1.0651961565
    epoch 105, loss 1.06405568123
    epoch 106, loss 1.06291675568
    epoch 107, loss 1.06177866459
    epoch 108, loss 1.06064271927
    epoch 109, loss 1.05950784683
    epoch 110, loss 1.05837345123
    epoch 111, loss 1.05724036694
    epoch 112, loss 1.05610907078
    epoch 113, loss 1.0549788475
    epoch 114, loss 1.05385005474
    epoch 115, loss 1.05272185802
    epoch 116, loss 1.05159509182
    epoch 117, loss 1.0504693985
    epoch 118, loss 1.04934501648
    epoch 119, loss 1.04822170734
    epoch 120, loss 1.04710018635
    epoch 121, loss 1.04597961903
    epoch 122, loss 1.04486060143
    epoch 123, loss 1.04374194145
    epoch 124, loss 1.04262447357
    epoch 125, loss 1.04150927067
    epoch 126, loss 1.04039406776
    epoch 127, loss 1.03928053379
    epoch 128, loss 1.03816819191
    epoch 129, loss 1.03705775738
    epoch 130, loss 1.03594756126
    epoch 131, loss 1.03483831882
    epoch 132, loss 1.03373169899
    epoch 133, loss 1.03262424469
    epoch 134, loss 1.03151917458
    epoch 135, loss 1.03041529655
    epoch 136, loss 1.02931237221
    epoch 137, loss 1.02821099758
    epoch 138, loss 1.02711033821
    epoch 139, loss 1.02601122856
    epoch 140, loss 1.02491283417
    epoch 141, loss 1.02381563187
    epoch 142, loss 1.02271962166
    epoch 143, loss 1.0216255188
    epoch 144, loss 1.02053201199
    epoch 145, loss 1.01943993568
    epoch 146, loss 1.01834869385
    epoch 147, loss 1.01725888252
    epoch 148, loss 1.01616990566
    epoch 149, loss 1.0150822401
    epoch 150, loss 1.01399600506
    epoch 151, loss 1.01291131973
    epoch 152, loss 1.01182675362
    epoch 153, loss 1.01074433327
    epoch 154, loss 1.00966215134
    epoch 155, loss 1.00858151913
    epoch 156, loss 1.00750219822
    epoch 157, loss 1.0064227581
    epoch 158, loss 1.00534558296
    epoch 159, loss 1.00427126884
    epoch 160, loss 1.00319552422
    epoch 161, loss 1.00212168694
    epoch 162, loss 1.00104928017
    epoch 163, loss 0.99997740984
    epoch 164, loss 0.998907446861
    epoch 165, loss 0.997838199139
    epoch 166, loss 0.996771931648
    epoch 167, loss 0.995703995228
    epoch 168, loss 0.994637966156
    epoch 169, loss 0.993573665619
    epoch 170, loss 0.992510199547
    epoch 171, loss 0.991447746754
    epoch 172, loss 0.990386784077
    epoch 173, loss 0.989326655865
    epoch 174, loss 0.988267123699
    epoch 175, loss 0.987210392952
    epoch 176, loss 0.986153483391
    epoch 177, loss 0.985097527504
    epoch 178, loss 0.984043121338
    epoch 179, loss 0.982990145683
    epoch 180, loss 0.981939017773
    epoch 181, loss 0.980888009071
    epoch 182, loss 0.979837536812
    epoch 183, loss 0.978789806366
    epoch 184, loss 0.977740943432
    epoch 185, loss 0.976695179939
    epoch 186, loss 0.975649952888
    epoch 187, loss 0.974604964256
    epoch 188, loss 0.973562121391
    epoch 189, loss 0.972519397736
    epoch 190, loss 0.971478819847
    epoch 191, loss 0.9704387784
    epoch 192, loss 0.969401478767
    epoch 193, loss 0.968362212181
    epoch 194, loss 0.967327296734
    epoch 195, loss 0.966291606426
    epoch 196, loss 0.965257465839
    epoch 197, loss 0.964224636555
    epoch 198, loss 0.963192462921
    epoch 199, loss 0.962162315845
    epoch 200, loss 0.961131751537
    epoch 201, loss 0.960103809834
    epoch 202, loss 0.959075450897
    epoch 203, loss 0.958049118519
    epoch 204, loss 0.957024395466
    epoch 205, loss 0.95599925518
    epoch 206, loss 0.954976439476
    epoch 207, loss 0.953954398632
    epoch 208, loss 0.952932834625
    epoch 209, loss 0.951912999153
    epoch 210, loss 0.95089495182
    epoch 211, loss 0.949876189232
    epoch 212, loss 0.948859095573
    epoch 213, loss 0.947844982147
    epoch 214, loss 0.946830153465
    epoch 215, loss 0.945816338062
    epoch 216, loss 0.944804489613
    epoch 217, loss 0.943793416023
    epoch 218, loss 0.942782759666
    epoch 219, loss 0.941774070263
    epoch 220, loss 0.940765857697
    epoch 221, loss 0.939758956432
    epoch 222, loss 0.938753008842
    epoch 223, loss 0.937748432159
    epoch 224, loss 0.936745524406
    epoch 225, loss 0.935741782188
    epoch 226, loss 0.934740841389
    epoch 227, loss 0.933740496635
    epoch 228, loss 0.932740986347
    epoch 229, loss 0.931742370129
    epoch 230, loss 0.930745720863
    epoch 231, loss 0.92974960804
    epoch 232, loss 0.928754210472
    epoch 233, loss 0.927759647369
    epoch 234, loss 0.926767170429
    epoch 235, loss 0.925775527954
    epoch 236, loss 0.924784362316
    epoch 237, loss 0.923794746399
    epoch 238, loss 0.922805786133
    epoch 239, loss 0.921818375587
    epoch 240, loss 0.920831859112
    epoch 241, loss 0.919846057892
    epoch 242, loss 0.918861567974
    epoch 243, loss 0.917877852917
    epoch 244, loss 0.91689568758
    epoch 245, loss 0.915914237499
    epoch 246, loss 0.914934277534
    epoch 247, loss 0.913955032825
    epoch 248, loss 0.912976861
    epoch 249, loss 0.91199952364
    epoch 250, loss 0.911023259163
    epoch 251, loss 0.910047829151
    epoch 252, loss 0.909074306488
    epoch 253, loss 0.908101379871
    epoch 254, loss 0.907129406929
    epoch 255, loss 0.906159043312
    epoch 256, loss 0.905188858509
    epoch 257, loss 0.904220223427
    epoch 258, loss 0.903252482414
    epoch 259, loss 0.902285277843
    epoch 260, loss 0.901319384575
    epoch 261, loss 0.900355517864
    epoch 262, loss 0.899390697479
    epoch 263, loss 0.898429691792
    epoch 264, loss 0.897467255592
    epoch 265, loss 0.896506667137
    epoch 266, loss 0.895547032356
    epoch 267, loss 0.89458835125
    epoch 268, loss 0.893631160259
    epoch 269, loss 0.892674803734
    epoch 270, loss 0.891718745232
    epoch 271, loss 0.890764534473
    epoch 272, loss 0.889811813831
    epoch 273, loss 0.888859570026
    epoch 274, loss 0.887908160686
    epoch 275, loss 0.886957943439
    epoch 276, loss 0.886007785797
    epoch 277, loss 0.885059535503
    epoch 278, loss 0.884112656116
    epoch 279, loss 0.883166611195
    epoch 280, loss 0.882221400738
    epoch 281, loss 0.881277084351
    epoch 282, loss 0.880333602428
    epoch 283, loss 0.879391491413
    epoch 284, loss 0.8784506917
    epoch 285, loss 0.877509474754
    epoch 286, loss 0.876570820808
    epoch 287, loss 0.875633537769
    epoch 288, loss 0.874695897102
    epoch 289, loss 0.873760223389
    epoch 290, loss 0.87282449007
    epoch 291, loss 0.871890425682
    epoch 292, loss 0.870957255363
    epoch 293, loss 0.870025336742
    epoch 294, loss 0.869093596935
    epoch 295, loss 0.868164181709
    epoch 296, loss 0.867234230042
    epoch 297, loss 0.866306006908
    epoch 298, loss 0.865379035473
    epoch 299, loss 0.864453196526
    epoch 300, loss 0.863526701927
    epoch 301, loss 0.862603843212
    epoch 302, loss 0.861680030823
    epoch 303, loss 0.860757827759
    epoch 304, loss 0.85983645916
    epoch 305, loss 0.858916163445
    epoch 306, loss 0.857997298241
    epoch 307, loss 0.85707873106
    epoch 308, loss 0.856161415577
    epoch 309, loss 0.855246067047
    epoch 310, loss 0.854329943657
    epoch 311, loss 0.853415310383
    epoch 312, loss 0.852502346039
    epoch 313, loss 0.851590156555
    epoch 314, loss 0.850677967072
    epoch 315, loss 0.849767506123
    epoch 316, loss 0.848858356476
    epoch 317, loss 0.847949504852
    epoch 318, loss 0.84704208374
    epoch 319, loss 0.846135795116
    epoch 320, loss 0.845229744911
    epoch 321, loss 0.844325423241
    epoch 322, loss 0.843421339989
    epoch 323, loss 0.842518806458
    epoch 324, loss 0.841616034508
    epoch 325, loss 0.840716183186
    epoch 326, loss 0.839816212654
    epoch 327, loss 0.838917732239
    epoch 328, loss 0.838019549847
    epoch 329, loss 0.837123036385
    epoch 330, loss 0.836227238178
    epoch 331, loss 0.835331439972
    epoch 332, loss 0.834438204765
    epoch 333, loss 0.833545207977
    epoch 334, loss 0.832652568817
    epoch 335, loss 0.831760764122
    epoch 336, loss 0.830871462822
    epoch 337, loss 0.829982161522
    epoch 338, loss 0.829094231129
    epoch 339, loss 0.828205943108
    epoch 340, loss 0.827320098877
    epoch 341, loss 0.826433956623
    epoch 342, loss 0.825549960136
    epoch 343, loss 0.824666500092
    epoch 344, loss 0.823783993721
    epoch 345, loss 0.822902083397
    epoch 346, loss 0.822021186352
    epoch 347, loss 0.821141421795
    epoch 348, loss 0.820262610912
    epoch 349, loss 0.819384694099
    epoch 350, loss 0.818507790565
    epoch 351, loss 0.817631900311
    epoch 352, loss 0.816756844521
    epoch 353, loss 0.815882503986
    epoch 354, loss 0.815009713173
    epoch 355, loss 0.814137279987
    epoch 356, loss 0.813265919685
    epoch 357, loss 0.812395095825
    epoch 358, loss 0.811525523663
    epoch 359, loss 0.810657143593
    epoch 360, loss 0.809789180756
    epoch 361, loss 0.808922946453
    epoch 362, loss 0.808056950569
    epoch 363, loss 0.807192504406
    epoch 364, loss 0.806328296661
    epoch 365, loss 0.805464625359
    epoch 366, loss 0.804603278637
    epoch 367, loss 0.803742587566
    epoch 368, loss 0.802882015705
    epoch 369, loss 0.802022814751
    epoch 370, loss 0.801163971424
    epoch 371, loss 0.800305664539
    epoch 372, loss 0.799450039864
    epoch 373, loss 0.798594295979
    epoch 374, loss 0.797739744186
    epoch 375, loss 0.796885728836
    epoch 376, loss 0.796033501625
    epoch 377, loss 0.7951810956
    epoch 378, loss 0.794329583645
    epoch 379, loss 0.79347962141
    epoch 380, loss 0.792630374432
    epoch 381, loss 0.79178249836
    epoch 382, loss 0.790934920311
    epoch 383, loss 0.790088713169
    epoch 384, loss 0.789242625237
    epoch 385, loss 0.788398146629
    epoch 386, loss 0.78755402565
    epoch 387, loss 0.786711573601
    epoch 388, loss 0.785869300365
    epoch 389, loss 0.785028576851
    epoch 390, loss 0.78418803215
    epoch 391, loss 0.783348798752
    epoch 392, loss 0.782510399818
    epoch 393, loss 0.781673669815
    epoch 394, loss 0.780836701393
    epoch 395, loss 0.780000209808
    epoch 396, loss 0.779165744781
    epoch 397, loss 0.778331875801
    epoch 398, loss 0.77749890089
    epoch 399, loss 0.776666760445
    epoch 400, loss 0.775835394859
    epoch 401, loss 0.775004982948
    epoch 402, loss 0.77417576313
    epoch 403, loss 0.773346424103
    epoch 404, loss 0.772519230843
    epoch 405, loss 0.771692216396
    epoch 406, loss 0.770866215229
    epoch 407, loss 0.770041763783
    epoch 408, loss 0.769217371941
    epoch 409, loss 0.768394172192
    epoch 410, loss 0.767570853233
    epoch 411, loss 0.766750276089
    epoch 412, loss 0.765929281712
    epoch 413, loss 0.765109777451
    epoch 414, loss 0.764290452003
    epoch 415, loss 0.763472735882
    epoch 416, loss 0.762655556202
    epoch 417, loss 0.761839568615
    epoch 418, loss 0.761023163795
    epoch 419, loss 0.760208964348
    epoch 420, loss 0.759395718575
    epoch 421, loss 0.758583366871
    epoch 422, loss 0.757771253586
    epoch 423, loss 0.756960272789
    epoch 424, loss 0.756150126457
    epoch 425, loss 0.755340695381
    epoch 426, loss 0.754532814026
    epoch 427, loss 0.753725349903
    epoch 428, loss 0.752918124199
    epoch 429, loss 0.752112329006
    epoch 430, loss 0.751306772232
    epoch 431, loss 0.750502943993
    epoch 432, loss 0.749699950218
    epoch 433, loss 0.748897194862
    epoch 434, loss 0.748095393181
    epoch 435, loss 0.747294843197
    epoch 436, loss 0.746495068073
    epoch 437, loss 0.745696365833
    epoch 438, loss 0.744898676872
    epoch 439, loss 0.744100928307
    epoch 440, loss 0.74330496788
    epoch 441, loss 0.742509245872
    epoch 442, loss 0.741714060307
    epoch 443, loss 0.740920722485
    epoch 444, loss 0.740127801895
    epoch 445, loss 0.739335119724
    epoch 446, loss 0.738543987274
    epoch 447, loss 0.737753987312
    epoch 448, loss 0.73696398735
    epoch 449, loss 0.736175179482
    epoch 450, loss 0.735387027264
    epoch 451, loss 0.734600186348
    epoch 452, loss 0.73381459713
    epoch 453, loss 0.733029007912
    epoch 454, loss 0.732244610786
    epoch 455, loss 0.731460571289
    epoch 456, loss 0.73067766428
    epoch 457, loss 0.729895353317
    epoch 458, loss 0.729114890099
    epoch 459, loss 0.728334486485
    epoch 460, loss 0.727554738522
    epoch 461, loss 0.72677564621
    epoch 462, loss 0.725997924805
    epoch 463, loss 0.725220680237
    epoch 464, loss 0.72444498539
    epoch 465, loss 0.723669946194
    epoch 466, loss 0.722895383835
    epoch 467, loss 0.722121119499
    epoch 468, loss 0.72134822607
    epoch 469, loss 0.720576047897
    epoch 470, loss 0.719805121422
    epoch 471, loss 0.719034731388
    epoch 472, loss 0.718264997005
    epoch 473, loss 0.717495799065
    epoch 474, loss 0.716728031635
    epoch 475, loss 0.715961396694
    epoch 476, loss 0.715195059776
    epoch 477, loss 0.714429616928
    epoch 478, loss 0.713664770126
    epoch 479, loss 0.712901413441
    epoch 480, loss 0.712138295174
    epoch 481, loss 0.711375832558
    epoch 482, loss 0.710614562035
    epoch 483, loss 0.709854722023
    epoch 484, loss 0.709094166756
    epoch 485, loss 0.708335280418
    epoch 486, loss 0.707577288151
    epoch 487, loss 0.706820130348
    epoch 488, loss 0.706063747406
    epoch 489, loss 0.705307602882
    epoch 490, loss 0.704552829266
    epoch 491, loss 0.703798532486
    epoch 492, loss 0.703045547009
    epoch 493, loss 0.702292978764
    epoch 494, loss 0.701541304588
    epoch 495, loss 0.700790584087
    epoch 496, loss 0.700040221214
    epoch 497, loss 0.699291229248
    epoch 498, loss 0.698542535305
    epoch 499, loss 0.697794914246
    epoch 500, loss 0.697048068047

```python

```

```python

```

```python

```


# Variational Principle  using Symbolic Mathematics in Python

## 1. Introduction

The variational principle tells us that we can use a trial wavefunction to solve the Schrodinger equation using the following theorem:

$${{\int {{\Psi ^*}\hat H{\rm{ }}\Psi } d\tau } \over {\int {{\Psi ^*}\Psi } d\tau }} \ge {E_0}$$

We will use Sympy to solve the particle in a box problem by guessing a trial wavefunction using variational principle

```python
import sympy as sym

```

This exercise is a bit more self-guided than the other notebooks we have done. One of the most useful things you can do is **open last week's notebook to remember the commands in sympy**. Also, remember that google is your friend:

1. [Sympy tutorial](https://docs.sympy.org/latest/tutorial/index.html)
2. [Stack Overflow](https://stackoverflow.com/search?q=sympy+)
3. [Stack Exchange](https://stackexchange.com/)

## 2. Particle in a box

The wave function that we pick for a particle in a box needs to have the following properties

1. single valued
1. normalizable
1. function and its first derivative are continuous
1. boundary condition that the wave function goes to zero at the ends of the box

Particle in a box: a is a classical particle, red is real part, blue is imaginary part.

This particle only expericnes kinetic energy between the box, so the Hamiltonian for this system is

$$\hat H = {{ - {\hbar ^2}} \over {2m}}{{{d^2}} \over {d{x^2}}} + \left\{ {\matrix{{V(x) = 0} & {0 < x < a}  \cr {V(x) = \infty } & {x < 0\text{ }{\rm{ or}}\;x > a}  \cr } } \right.$$

For our purposes, that means we can consider the Hamiltonian to be

$$\hat H = {{ - {\hbar ^2}} \over {2m}}{{{d^2}} \over {d{x^2}}}$$

as long as we keep the limits of integration to be $(0,a)$

### 2.1 Trial Wave function

Although the particle in box has a well known solution

[https://en.wikipedia.org/wiki/Particle_in_a_box](https://en.wikipedia.org/wiki/Particle_in_a_box)

(or check your favorite pchem book)

We are going to guess a trial wave function:

$$\Phi (x) = \left( {{x \over a} - {{{x^3}} \over a^3}} \right) + \alpha \left( {{{{x^5}} \over {{a^5}}} - {1 \over 2}\left( {{{{x^9}} \over {{a^9}}} + {{{x^9}} \over {{a^9}}}} \right)} \right)$$

```python

```

### 2.2 Exercise: Variational Theorem

We are going to follow the following plan:

1. Solve for the energy of the trial wave function above

$${E_{trial}} = {{\int\limits_0^a {\Phi (x){{ - {\hbar ^2}} \over {2m}}{{{d^2}} \over {d{x^2}}}\Phi (x)dx} } \over {\int\limits_0^a {\Phi {{(x)}^2}dx} }}$$

Your answer will be a function of $ m,a,\text{and } \alpha$ We will use $\alpha$ as the parameter we vary to minimize the energy and make a new trial wave function.

2. Minimize the trial energy
We will use a first derivative of the trial energy $${d \over {d\alpha }}{E_{trial}}(\alpha )$$ to find the value of $\alpha$ that gives you the lowest energy

3. Plot your new wavefunction compared to the ground state particle in a box: $${\psi _{true}}(x) = {\left( {{2 \over a}} \right)^{1/2}}\sin {{n\pi x} \over a}$$ Plot as a function of $x/a$ from $0$ to $1$. Assuming this has $m=m_e$, and $a=a_0$ use atomic (theorist) units to plot the function.

4. Compare your trial energy to the actual energy (using atomic units)
$${E_{true}}(n = 1) = {{{\hbar ^2}{\pi ^2}} \over {2m{a^2}}}$$

```python

import sympy as sym
m,a,x,alpha  = sym.symbols('m,a,x,alpha')
sym.Rational(1/2)
sym.init_printing()
```

Your descriptions/explanations here

```python
phi = ((x/a)-(x**3/a**3)+ alpha*((x**5/a**5)-1/2*(x**9/a**9+x**9/a**9)))
num1 = (phi*sym.Rational(-1/2))
num2= sym.diff(sym.diff(phi,x),x)
den1=(phi**2)

```

```python
Numerator =sym.integrate((num1*num2),(x,0,a))
```

```python
Denominator= sym.integrate((den1),(x,0,a))
```

```python
Numerator/Denominator
```

```python
E_trial=(Numerator/Denominator).subs(a,1)
E_trial
```

```python
sym.solveset(sym.diff(E_trial,alpha))
```

```python
E_trial.subs(alpha, -0.257747652303185)
```

```python
E_trial.subs(alpha, -4.73345207773701)
```

```python
Newphi=phi.subs(alpha,-0.257747652303185)
Newphi
```

```python
Newphi2=Newphi.subs(a,1)
Newphi2
```

```python
sym.plot(Newphi2,(x,0,1))
```

```python
sym.plot((2**1/2)*sym.sin(sym.pi*x),(x,0,1))
```

```python
sym.plot((2**1/2)*sym.sin(sym.pi*x),Newphi2,(x,0,1))
```

```python
E_true=sym.pi**2/(2*m)
E_true

```

The trial energy is greater than the actual energy

### 2.3 Exercise: New trial wavefunction

Determine the minimum energy of the particle in a box using a new trial wavefunction $$x^\alpha(x-a)^\alpha$$

1. Find the minimum energy, $E_{trial}$
2. Plot the new trial wavefunction and compare it to the true solution and the wavefunction you found above
3. Compare you new energy to the trial energy you found above
4. Which wavefunction is better? How do you know?

```python
# Plug in 1 for a, after you solve for values, first find the first dericative of part 1 function in terms of alpa
#after you solve both plug them into E trial and see which gives you the lower E value

phhi= (x**alpha)*(x-a)**alpha
phhi

```

```python
num11 = (phhi*sym.Rational(1/2))
num21= sym.diff(sym.diff(phhi,x),x)
den11=(phhi**2)
Numerator2 =sym.integrate((num11*num21),(x,0,a))
Numerator2

```

```python
Denominator2= sym.integrate((den11),(x,0,a))
Denominator2
```

$$\frac{a a^{4 \alpha} e^{2 i \pi \alpha} \Gamma\left(2 \alpha + 1\right) {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)}}{\Gamma\left(2 \alpha + 2\right)}$$

```python
Numerator2/Denominator2
```

$$\frac{a^{- 4 \alpha} \alpha e^{- 2 i \pi \alpha} \Gamma\left(2 \alpha + 2\right) \int_{0}^{a} \frac{x^{2 \alpha} \left(- a + x\right)^{2 \alpha} \left(a^{2} \alpha - a^{2} - 4 a \alpha x + 2 a x + 4 \alpha x^{2} - 2 x^{2}\right)}{x^{2} \left(- a + x\right)^{2}}\, dx}{2 a \Gamma\left(2 \alpha + 1\right) {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)}}$$

```python
E_trial2=(Numerator2/Denominator2).subs(a,1)
E_trial2
```

$$\frac{\alpha e^{- 2 i \pi \alpha} \Gamma\left(2 \alpha + 2\right) \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx}{2 \Gamma\left(2 \alpha + 1\right) {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)}}$$

```python
sym.solveset(sym.diff(E_trial2,alpha))
```

$$\left\{\alpha \mid \alpha \in \mathbb{C} \wedge \left(- 2 \alpha {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)} \operatorname{polygamma}{\left (0,2 \alpha + 1 \right )} \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx + 2 \alpha {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)} \operatorname{polygamma}{\left (0,2 \alpha + 2 \right )} \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx + \alpha {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)} \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 x^{2} - 4 x + 2 \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right) \log{\left (x \right )} + 2 \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right) \log{\left (x - 1 \right )} + 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx - 2 i \pi \alpha {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)} \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx - \alpha \frac{d}{d \alpha} {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)} \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx + {{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)} \int_{0}^{1} \frac{x^{2 \alpha} \left(x - 1\right)^{2 \alpha} \left(4 \alpha x^{2} - 4 \alpha x + \alpha - 2 x^{2} + 2 x - 1\right)}{x^{2} \left(x - 1\right)^{2}}\, dx\right) \Gamma\left(2 \left(\alpha + 1\right)\right) = 0 \right\} \setminus \left\{\alpha \mid \alpha \in \mathbb{C} \wedge e^{2 i \pi \alpha} \Gamma\left(2 \alpha + 1\right) {{{}_{2}F_{1}\left(\begin{matrix} - 2 \alpha, 2 \alpha + 1 \\ 2 \alpha + 2 \end{matrix}\middle| {1} \right)}}^{2} = 0 \right\}$$

I'm not sure what I did wrong here :(
I tried

### 2.4 Exercise: Design your own wavefunction!

**Now you get to make your own wavefunction!**

The only guidance I would give you is that it make sense mathematically and that it include $\alpha$ so that you can minimize the energy.

Remember that $a$ and $x$ are both length units, and that trigonometric, logarithmic, and exponential functions are all unitless

Using your new wavefunction:

1. Find the minimum energy, $E_{trial}$
2. Plot the new trial wavefunction and compare it to the true solution and the wavefunction you found above
3. Compare you new energy to the trial energy you found above
4. Which wavefunction is better? How do you know?

```python
# Your code here
myphi = ((x/a)-(x**8/a**8)+ alpha*((x**8/a**8)-1/2*(x**8/a**8)))
myphi
```

```python
mynum = (myphi*sym.Rational(-1/2))
myden=(myphi**2)
mynum2=sym.diff(sym.diff(myphi,x),x)
MyNumerator =sym.integrate((mynum*mynum2),(x,0,a))
MyDenominator= sym.integrate((myden),(x,0,a))
MyNumerator/MyDenominator

```

```python
MyE_trial=(MyNumerator/MyDenominator).subs(a,1)
MyE_trial
```

```python
sym.solveset(sym.diff(MyE_trial,alpha))
```

```python
MyE_trial.subs(alpha, -0.200898343265497)
```

```python
MyE_trial.subs(alpha, -10.6624896348766)
```

```python
ThatNewNewphi=myphi.subs(alpha,-10.6624896348766)
ThatNewNewphi
```

```python
ThatNewNewphi2=ThatNewNewphi.subs(a,1)
ThatNewNewphi2
```

```python
sym.plot(ThatNewNewphi2,(x,0,0.75))
```

Your descriptions/explanations here

```python
sym.plot((2**1/2)*sym.sin(sym.pi*x),(x,0,1))
```

```python
sym.plot((2**1/2)*sym.sin(sym.pi*x), ThatNewNewphi2,(x,0,1))
```

My wavefunction had a lower energy. For that reason, and that reason alone, I'm going to say that my wavefunction is better :)

# Reading Homework

Read the following sections in Kramer

- 4.2.3 Born-Oppenheimer approximation
- 4.3.2 Secular equation
- All of 4.5

For each subsection
- write down the subchapter name
- what was the most important idea
- draw an idea digram of the main idea

**Make sure to upload this to the assignment repository**

Example idea diagram:

```python

```


```python
import numpy as np
import sympy
import itertools
from sympy import *

init_printing(use_latex="mathjax")
```

$\varphi_1,\dots$ are the irreps. Let $\rho$ be $\varphi_k$

input $\dim G$

```python
dim_G = 24
```

input the character table M: first row ($i=0$) lists the numbers of elements in the conjugacy class $j$.
The next rows give are the characters of the congugacy class $j$.
So the first colunm is the characters of $1_G$ and therefore give us the dimension if the representation.

```python
M = Matrix([ [1, 6, 3, 8, 6], [1, 1, 1, 1, 1], [1, -1, 1, 1, -1],
             [2, 0, 2, -1, 0], [3, -1, -1, 0, 1], [3, 1, -1, 0, -1]])
M

```

$$\left[\begin{matrix}1 & 6 & 3 & 8 & 6\\1 & 1 & 1 & 1 & 1\\1 & -1 & 1 & 1 & -1\\2 & 0 & 2 & -1 & 0\\3 & -1 & -1 & 0 & 1\\3 & 1 & -1 & 0 & -1\end{matrix}\right]$$

$\sum_{j\in\text{ conjugasy classes of }G} \mid \chi_\rho (j) \mid^2 \chi_{\varphi_i}(j)$ where $\rho=\varphi_k$

```python
def sum_in_lmbda(M, k, i):
    n = 0
    for j in range(0, M.cols):
        n += M[0, j] * M[i, j] * Abs(M[k, j]) **2
    return n

```

$\lambda_{\varphi_i}^\rho = \frac{1}{\mid G \mid d_{\varphi_i}}\sum_{j\in\text{ conjugasy classes of }G} \mid \chi_\rho (j) \mid^2 \chi_{\varphi_i}(j)$ where $\rho=\varphi_k$

```python
def lmbda(M, k, i, dim_G):
    return sum_in_lmbda(M, k, i) / (dim_G * M[i, 0])
```

$L^\rho = \{ \lambda_{\varphi_i}^\rho \}_i$ and $\rho = \varphi_k$ not including $i=1$ i.e. without $\lambda^\rho_{\varphi_1}$

```python
def L_rho(M, k, dim_G):
    l = [lmbda(M, k, i, dim_G) for i in range(2, M.rows)]
    print('L_rho:', [simplify(x) for x in l])
    return l
```

$\gamma^{\varphi_k} = gap(\rho=\varphi_k) = 1 - \max_i\{L^{\rho}\}$

```python
def gamma_k(M, k, dim_G):
    return 1 - max(L_rho(M, k, dim_G))
```

$\gamma = (\dots, \gamma^{\varphi_k} ,\dots)$

```python
def gamma(M, dim_G):
    return Matrix(M.rows - 1, 1, lambda k, _: gamma_k(M, k + 1, dim_G))
g = gamma(M, dim_G)
g
```

    L_rho: [0, 0, 0, 0]
    L_rho: [0, 0, 0, 0]
    L_rho: [1, 1/2, 0, 0]
    L_rho: [0, 1/2, 1/3, 1/3]
    L_rho: [0, 1/2, 1/3, 1/3]

$$\left[\begin{matrix}1\\1\\0\\\frac{1}{2}\\\frac{1}{2}\end{matrix}\right]$$

```python
simplify(g)
```

$$\left[\begin{matrix}1\\1\\0\\\frac{1}{2}\\\frac{1}{2}\end{matrix}\right]$$

Calculating gap from the character table - for example the group S_4.ipynb


# Lorenz-63 model
In this Notebook we'll demonstrate how to construct multivariate models. To this end, we shall consider the Lorenz-63 model, defined as
\begin{equation}
    \begin{cases}
        \mathrm{d}X^1_t = -S (X^1_t - X^2_t) \mathrm{d}t + \mathrm{d}W^1_t, \\
        \mathrm{d}X^2_t = (RX^1_t - X^2_t - X^1_t \cdot X^2_t) \mathrm{d}t + \mathrm{d}W^2_t, \\
        \mathrm{d}X^3_t = (X^1_t \cdot X^2_t - BX^3_t) \mathrm{d}t + \mathrm{d}W^3_t,
    \end{cases}
\end{equation}
where $\{W^i_t\}$ are 1-dimensional Wienere processes and $(S, R, B) \in \mathbb{R}^3$. We assume that we collect discrete observations of the above system via the following set of equations
\begin{equation}
    \begin{cases}
        Y^1_t = k_o X^1_t + V^1_t, \\
        Y^2_t = k_o X^3_t + V^2_t,
    \end{cases}
\end{equation}
where $k_o > 0$, and $V^i_t \sim \mathcal{N}(0, \sigma^2)$ where $\sigma^2 = 0.1$. We proceed in the same manner as the Stochastic volatility example and import the necessary packages.

```python
from pyfilter.timeseries import StateSpaceModel, EulerMaruyma, LinearGaussianObservations
```

Next, we define the inital distributions as
\begin{equation}
X^1_0 \sim \mathcal{N}(-5.9162, 10), \quad X^2_0 \sim \mathcal{N}(-5.52332, 10), \quad X^3_0 \sim \mathcal{N}(24.5723, 10),
\end{equation}
which means that the model equations are given by

```python
import torch
import math

def finit(s, r, b):
    return torch.tensor([-5.91652, -5.52332, 24.5723], device=s.device)

def ginit(s, r, b):
    return math.sqrt(10) * torch.ones(3, device=s.device)

def f(x, s, r, b):
    x1 = -s * (x[0] - x[1])
    x2 = r * x[0] - x[1] - x[0] * x[2]
    x3 = x[0] * x[1] - b * x[2]

    return x1, x2, x3

def g(x, s, r, b):
    return torch.ones(3, device=x.device)
```

For this example we'll simulate data, we do this by setting $(S, R, B) \triangleq (10, 28, 8/3)$ and run the below script

```python
import matplotlib.pyplot as plt
from torch.distributions import Independent, Normal, Uniform, MultivariateNormal

dt = torch.tensor(1e-2)
hidden = EulerMaruyma((finit, ginit), (f, g), (10., 28., 8/3), ndim=3, dt=dt)

mat = torch.tensor([
    [0.8, 0., 0.],
    [0., 0., 0.8]
])

simmodel = LinearGaussianObservations(hidden, mat, torch.ones(2) / math.sqrt(10))

x, y = simmodel.sample(5000)

fig, ax = plt.subplots(2, figsize=(16, 9))

ax[1].plot(x.numpy())
ax[0].plot(y.numpy())
```

Where the upper plot is the observble data, and the lower is the hidden processes. Next, we shall perform the inference. We use the NESS to target the parameters. However, since the model has purely additive noise (rather than multiplicative) in the observational process, we may use the `Unscented Kalman Filter` to target the latent process. Furthermore, we use the same priors as in the original paper describing the NESS algorithm. Such that we get

```python
from pyfilter.algorithms import NESS
from pyfilter.filters import UKF
from pyfilter.algorithms.kernels import AdaptiveShrinkageKernel

hidden = EulerMaruyma((finit, ginit), (f, g), (Uniform(5, 40), Uniform(10, 50), Uniform(1, 20)), ndim=3, dt=dt)
ssm = LinearGaussianObservations(hidden, mat, torch.ones(2) / math.sqrt(10))

algs = list()
for i in range(4):
    filt_ = UKF(ssm.copy())
    alg = NESS(filt_, 2000, threshold=0.75)

    alg.initialize().fit(y)
    algs.append(alg)
```

    NESS: 100%|████████████████████████████████████████████████████████████████████████| 5000/5000 [01:30<00:00, 55.14it/s]
    NESS: 100%|████████████████████████████████████████████████████████████████████████| 5000/5000 [01:40<00:00, 49.78it/s]
    NESS: 100%|████████████████████████████████████████████████████████████████████████| 5000/5000 [01:41<00:00, 49.13it/s]
    NESS: 100%|████████████████████████████████████████████████████████████████████████| 5000/5000 [01:43<00:00, 48.15it/s]

And finally we plot the posterior distributions together with the true parameters

```python
from pyfilter.utils import normalize

fig, ax = plt.subplots(3, figsize=(16, 9))

for r, alg in enumerate(algs):
    weights = normalize(alg._w_rec)
    for i, (p, ptrue) in enumerate(zip(alg.filter.ssm.hidden.theta, simmodel.hidden.theta)):
        xrange, xvals = p.get_plottable(weights=weights)

        ax[i].plot(xrange, xvals)
        ax[i].axvline(ptrue.values, color='r')

plt.show()
```