# Dimensional Reduction
G. Richards
(2016, 2018, 2020)
based on materials from Connolly, Leighly, VanderPlas, Geron, and Ivezic Ch. 7.0-7.4

**This SHOULD NOT be necessary anymore, but I'm leaving it here for now (2020) just in case anyone runs into problems.  Before class starts, you may need to do the following:**

> find . -name “sdss_corrected_spectra.py” -print

> ./anaconda3/lib/python3.8/site-packages/astroML/datasets/sdss_corrected_spectra.py

> emacs -nw  ./anaconda3/lib/python3.8/site-packages/astroML/datasets/sdss_corrected_spectra.py

> #DATA_URL = 'http://www.astro.washington.edu/users/vanderplas/spec4000.npz'

> DATA_URL = 'http://staff.washington.edu/jakevdp/spec4000.npz'

Just in case that doesn't work, I've put "spec4000.npz" in PHYS_440_540/data.  Copy this to your "astroML_data" directory.

## Curse of Dimensionality

You want to buy a car.  Right now--you don't want to wait.  But you are picky and have certain things that you would like it to have.  Each of those things has a probability between 0 and 1 of being on the the car dealer's lot.  You want a red car which has a probability of being on the lot of $p_{\rm red}$; you want good gas mileage, $p_{\rm gas}$; you want leather seats, $p_{\rm leather}$; and you want a sunroof, $p_{\rm sunroof}$.  The probability that the dealer has a car on the lot that meets all of those requirements is
$$p_{\rm red} \, p_{\rm gas} \, p_{\rm leather} \, p_{\rm sunroof},$$
or $p^n$ where $n$ is the number of features (assuming equal probability for each).

If the probability of each of these is 50%, then the probability of you driving off with your car of choice is only $0.5*0.5*0.5*0.5 = 0.0625$.  Not very good.  Imagine if you also wanted other things.  This is the [Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality).

Let's illustrate the curse of dimensionality with two figures from [here.](https://medium.freecodecamp.org/the-curse-of-dimensionality-how-we-can-save-big-data-from-itself-d9fa0f872335)

In the first example we are trying to find which box hold some treasure, which gets harder and harder with more dimensions, despite there just being 5 boxes in each dimension:

In the next example we inscribe a circle in a square.  The area outside of the circle grows larger and larger as the number of dimensions increase:

Can also think about longest linear distance.  Goes up by 41% when add 2nd dimension (and 73% larger in 3-D than 1-D).

Mathematically we can describe this as: the more dimensions that your data span, the more points needed to uniformly sample the space.

For $D$ dimensions with coordinates $[-1,1]$, the fraction of points in a unit hypersphere (with radius $r$, as illustrated above) is
$$f_D = \frac{V_D(r)}{(2r)^D} = \frac{\pi^{D/2}}{D2^{D-1}\Gamma(D/2)}$$
which goes to $0$ as $D$ goes to infinity!  Actually, as you can see from the plot below, it is effectively 0 much earlier than that!

```python
# Execute this cell
# from Andy Connolly
%matplotlib inline
import numpy as np
import scipy.special as sp
from matplotlib import pyplot as plt

def unitVolume(dimension, radius=1.):
    return 2*(radius**dimension *np.pi**(dimension/2.))/(dimension*sp.gamma(dimension/2.))

dim = np.linspace(1,100)

#------------------------------------------------------------
# Plot the results
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(dim,unitVolume(dim)/2.**dim)

ax.set_yscale('log')
ax.set_xlabel('$Dimension$')
ax.set_ylabel('$Volume$')

plt.show()
```

Note that this works in the opposite direction too: let's say you want to find "rare" objects in 10 dimensions, where we'll define rare as <1% of the population.  Then you'll need to accept objects from 63% of the distribution in all 10 dimensions!  So are those really "rare" or are they just a particular 1% of the population?

```python
import numpy as np
#p^10 = 0.01, solve for p
p = 10**(np.log10(0.01)/10.0)
print(p)
```

What fraction of each dimension do you need to cover to split your data 50-50 in 2D?  Try it.

```python
import numpy as np
p = 10**(np.log10(____)/____)
print(p)
```

N.B.  Dimensionality isn't just measuring $D$ parameters for $N$ objects.  It could be a spectrum with $D$ values or an image with $D$ pixels, etc.  In the book the examples used just happen to be spectra of galaxies from the SDSS project.  But we can insert the data of our choice instead.

For example: the SDSS comprises a sample of 357 million sources:
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size) and normalizing the data range $-1$ to $1$

yields a probability of having one of the 357 million sources reside within a unit hypersphere of 1 in 1.4$\times 10^5$.

See also [this article](https://towardsdatascience.com/the-curse-of-dimensionality-50dc6e49aa1e).

## Principal Component Analysis (PCA)

In [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) we seek to take a data set like the one shown below and apply a transform to the data such that the new axes are aligned with the maximal variance of the data.  As can be seen in the Figure, this is basically just the same as doing regression by minimizing the square of the perpendicular distances to the new axes.  Note that we haven't made any changes to the data, we have just defined new axes.

```python
# Execute this cell
# Ivezic, Figure 7.2
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse

#------------------------------------------------------------
# Set parameters and draw the random sample
np.random.seed(42)
r = 0.9

sigma1 = 0.25
sigma2 = 0.08
rotation = np.pi / 6
s = np.sin(rotation)
c = np.cos(rotation)

X = np.random.normal(0, [sigma1, sigma2], size=(100, 2)).T
R = np.array([[c, -s],[s, c]])
X = np.dot(R, X) #Same data, now rotated by R matrix.

#------------------------------------------------------------
# Plot the diagram
fig = plt.figure(figsize=(5, 5), facecolor='w')
ax = plt.axes((0, 0, 1, 1), xticks=[], yticks=[], frameon=False)

# draw axes
ax.annotate(r'$x$', (-r, 0), (r, 0),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))
ax.annotate(r'$y$', (0, -r), (0, r),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))

# draw rotated axes
ax.annotate(r'$x^\prime$', (-r * c, -r * s), (r * c, r * s),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))
ax.annotate(r'$y^\prime$', (r * s, -r * c), (-r * s, r * c),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))

# scatter points
ax.scatter(X[0], X[1], s=25, lw=0, c='k', zorder=2)

# draw lines
vnorm = np.array([s, -c])
for v in (X.T):
    d = np.dot(v, vnorm)
    v1 = v - d * vnorm
    ax.plot([v[0], v1[0]], [v[1], v1[1]], '-k')

# draw ellipses
for sigma in (1, 2, 3):
    ax.add_patch(Ellipse((0, 0), 2 * sigma * sigma1, 2 * sigma * sigma2,
                         rotation * 180. / np.pi,
                         ec='k', fc='gray', alpha=0.2, zorder=1))

ax.set_xlim(-1, 1)
ax.set_ylim(-1, 1)

plt.show()
```

Note that the points are correlated along a particular direction which doesn't align with the initial choice of axes.  So, we should rotate our axes to align with this correlation.

We'll choose the rotation to maximize the ability to discriminate between the data points:
*   the first axis, or **principal component**, is direction of maximal variance
*   the second principal component is orthogonal to the first component and maximizes the residual variance
*   ...

PCA is a dimensional reduction process because we can generally account for nearly "all" of the variance in the data set with fewer than the original $K$ dimensions.  See more below.

We start with a data set $\{x_i\}$ which consists of $N$ objects for which we measure $K$ features.  We start by subtracting the mean for each feature in $\{x_i\}$ and write $X$ as a $N\times K$ matrix.

The covariance of this matrix is
$$C_X=\frac{1}{N-1}X^TX.$$

There are off-diagonal terms if there are correlations between the measurements (e.g., maybe two of the features are temperature dependent and the measurements were taken at the same time).

If $R$ is a projection of the data that is aligned with the maximal variance, then we have $Y= X R$ with covariance

$$ C_{Y} = R^T X^T X R = R^T C_X R.$$

$r_1$ is the first principal component of $R$, which can be derived using Langrange multipliers with the following cost function:

$$ \phi(r_1,\lambda_1) = r_1^TC_X r_1 - \lambda_1(r_1^Tr_1-1). $$

If we take derivative of $\phi(r_1,\lambda)$ with respect to $r_1$ and set it to 0, then we have

$$ C_Xr_1 - \lambda_1 r_1 = 0. $$

$\lambda_1$ (the largest eigenvalue of the matrix) is the root of the equation $\det(C_X -
\lambda_1 {\bf I})=0$ for which the eigenvalue is

$$ \lambda_1 =  r_1^T C_X r_1.$$

The columns of the full matrix, $R$ are the eigenvectors (known here as principal components).

The diagonal values of $C_Y$ are the variance contained within each component.

We aren't going to go through the linear algebra more than that here.  But it would be a good group project for someone.  See the end of 7.3.1 starting at the bottom on page 294 or go through [Karen Leighly's PCA lecture notes](http://seminar.ouml.org/lectures/principal-components-analysis/) if you want to walk through the math in more detail.

### Preparing data for PCA

   * Subtract the mean of each dimension (to "center" the data)
   * Divide by the variance in each dimension (to "whiten" the data)
   * (For spectra and images) normalize each row to yield an integral of unity.

Below is a typical call to the PCA algorithm.  Note that this example is somewhat backwards.  We are starting with `X` and then we are making it higher dimensional--to create a mock high-$D$ data set.  Then we are applying PCA as a dimensionality reduction technique.

```python
#Example call from 7.3.2
import numpy as np
from sklearn.decomposition import PCA

X = np.random.normal(size=(100,3)) # 100 points in 3D
R = np.random.random((3,10)) # projection matrix
X = np.dot(X,R) # X is now 10-dim, with 3 intrinsic dims
pca = PCA(n_components=4) # n_components can be optionally set
pca.fit(X)
eigenvalues = pca.transform(X) # compute the subspace projection of X, 4 eigenvalues for each of the 100 samples
mean = pca.mean_ # length 10 mean of the data
eigenvectors = pca.components_ # 4x10 matrix of components, multiply each by respective eigenvalue to reconstruct

#Reconstruction of object1
#Xreconstruct[0] = mean + eigenvectors*eigenvalues[0]

print(eigenvalues.shape)
print(eigenvectors.shape)
```

To illustrate what is happening here is a PCA reconstruction of handwritten "3s" from [Hastie et al.](https://web.stanford.edu/~hastie/ElemStatLearn/) :

[Scikit-Learn's decomposition module](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) has a number of [PCA type implementations](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA).
Let's work through an example using spectra of galaxies take during the Sloan Digital Sky Survey.  In this sample there are 4000 spectra with flux measurements in 1000 bins.  15 example spectra are shown below and our example will use half of the spectra chosen at random.

```python
%matplotlib inline
# Example from Andy Connolly
# See Ivezic, Figure 7.4
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA
#from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.utils import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)
print(len(spectra), len(wavelengths))

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 2000 # We'll just look at 2000 random spectra
n_components = 5 # Do the fit with 5 components, which is the mean plus 4
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0) # Compute the mean spectrum, which is the first component
# spec_mean = spectra[:50].mean(0)

# use Randomized PCA for speed
#pca = RandomizedPCA(n_components - 1)
pca = PCA(n_components - 1,svd_solver='randomized')
pca.fit(spectra[ind])
pca_comp = np.vstack([spec_mean,pca.components_]) #Add the mean to the components
evals = pca.explained_variance_ratio_
print(evals)
```

Now let's plot the components (eigenvectors).  See also Ivezic, Figure 7.4.  The left hand panels are just the first 5 spectra for comparison with the first 5 PCA components, which are shown on the right.  They are ordered by the size of their eigenvalues.

```python
#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel('wavelength (Angstroms)')
    ax.plot(wavelengths, pca_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel('wavelength (Angstroms)')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')
    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

Now let's make "scree" plots.  These plots tell us how much of the variance is explained as a function of the each eigenvector.  Our plot won't look much like Ivezic, Figure 7.5, so I've shown it below to explain where "scree" comes from.

```python
# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))

ax = fig.add_subplot(121)
ax.plot(np.arange(n_components-1), evals)
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("eigenvalue ")

ax = fig.add_subplot(122)
ax.plot(np.arange(n_components-1), evals.cumsum())
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("cumulative eigenvalue")

plt.show()
```

How much of the variance is explained ([explained_variance_ratio_](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)) by the first two components?  How about all of the components?

```python
print("The first component explains {:.3f} of the variance in the data.".format(___.___[0]))
print("The second component explains {:.3f} of the variance in the data.".format(___.___[1]))
print("All components explain {:.3f} of the variance in the data.".format(sum(___.___)))
```

This is why PCA enables dimensionality reduction.

How many components would we need to explain 99.5% of the variance?

```python
for num_feats in np.arange(1,20, dtype = int):
    pca = PCA(___=___)
    pca.___(spectra[ind])
    if (sum(___.___)>___):
        break
print("{:d} features are needed to explain 99.5% of the variance".format(____))
```

Note that we would need 1000 components to encode *all* of the variance.

There is a MUCH easier way to do this.  Just give it a number of components between 0 and 1 and it will interpret that as a percentage of the variance.

```python
pca995 = PCA(n_components=0.995)
pca995.fit(spectra[ind])
print("{:d} features are needed to explain 99.5% of the variance".format(pca995.n_components_))
```

If you ever use sklearn's PCA, note that if you give it a dataset that is too big, it won't do the full PCA, but rather an approximated one using the `svd_solver="randomized"`, but you can force it to use something else.

## Interpreting the PCA

- The output eigenvectors are ordered by their associated eigenvalues
- The eigenvalues reflect the variance within each eigenvector
- The sum of the eigenvalues is total variance of the system
- Projection of each spectrum onto the first few eigenspectra is a compression of the data

Once we have the eigenvectors, we can try to reconstruct an observed spectrum, ${x}(k)$, in the eigenvector basis, ${e}_i(k)$, as

$$ \begin{equation}
  {x}_i(k) = {\mu}(k) + \sum_j^R \theta_{ij} {e}_j(k).
\end{equation}
$$

That would give a full (perfect) reconstruction of the data since it uses all of the eigenvectors.  But if we truncate (i.e., $r<R$), then we will have reduced the dimensionality while still reconstructing the data with relatively little loss of information.

For example, we started with 4000x1000 floating point numbers.  If we can explain nearly all of the variance with 8 eigenvectors, then we have reduced the problem to 4000x8+8x1000 floating point numbers!

Execute the next cell to see how the reconstruction improves by adding more components.

```python
# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

### Caveats I

PCA is a linear process, whereas the variations in the data may not be.  So it may not always be appropriate to use and/or may require a relatively large number of components to fully describe any non-linearity.

Note also that PCA can be very impractical for large data sets which exceed the memory per core as the computational requirement goes as $\mathscr{O}(D^3$) and the memory requirement goes as $\mathscr{O}(2D^2)$.

### Missing Data
We have assumed so far that there is no missing data (e.g., bad pixels in the spectrum, etc.).  But often the data set is incomplete.  Since PCA encodes the flux correlation with wavelength (or whatever parameters are in your data set), we can actually use it to determine missing values.

An example is shown below.  Here, black are the observed spectra.  Gray are the regions where we have no data.  Blue is the PCA reconstruction, including the regions where there are no data.  Awesome, isn't it?

```python
# Execute this cell
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import ticker

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(8, 10))
fig.subplots_adjust(hspace=0)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k', lw=2,
            label='True spectrum')

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'N EV=%i' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1, prop=dict(size=14))

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()
```

The example that we have been using above is "spectral" PCA.  Some examples from the literature include:
- [Francis et al. 1992](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1992ApJ...398..476F&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf)
- [Connolly et al. 1995](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1995AJ....110.1071C&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf)
- [Yip et al. 2004](http://iopscience.iop.org/article/10.1086/425626/meta;jsessionid=31BB5F11B85D2BF4180834DC71BA0B85.c3.iopscience.cld.iop.org)

One can also do PCA on features that aren't ordered (as they were for the spectra).  E.g., if you have $D$ different parameters measured for your objects.  The classic example in astronomy is
[Boroson & Green 1992](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1992ApJS...80..109B&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf).

### Caveats II

One of the things that I don't like about PCA is that the eigenvectors are defined relative to the mean.  So they can be positive or negative and they often don't look anything like the original data itself.  Whereas it is often the case that you might expect that the components would look like, well, the physical components.  For example, quasars are fundamentally galaxies.  So, part of their flux comes from the galaxy that they live in.  But PCA doesn't return any component that looks like a typical galaxy.

## Non-negative Matrix Factorization (NMF)

This is where [Non-negative Matrix Factorizaiton (NMF)](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization) comes in.  Here we are treating the data as a linear sum of positive-definite components.

NMF assumes  any data matrix can be factored into two matrices, $W$ and $Y$, with
$$\begin{equation}
X=W Y,
\end{equation}
$$
where both $W$ and $Y$ are nonnegative.

So, $WY$ is an approximation of $X$. Minimizing the reconstruction error $|| (X - W Y)^2 ||$,
nonnegative bases can be derived through an iterative process.

Note, however, that the iterative process does not guarantee nonlocal minima (like $K$-means and EM), but using
random initialization and cross-validation can be used to find the global minimum.

An example from the literature is [Allen et al. 2008](http://arxiv.org/abs/0810.4231)

In Scikit-Learn the [NMF implementation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html) looks like:

```python
# Execute this cell
import numpy as np
from sklearn.decomposition import NMF
X = np.random.random((100,10)) # 100 points in 10-D
nmf = NMF(n_components=3)
nmf.fit(X)
proj = nmf.transform(X) # project to 3 dimension
comp = nmf.components_ # 3x10 array of components
err = nmf.reconstruction_err_ # how well 3 components capture the data
```

An example (and comparison to PCA) is given below.

```python
# Execute the next 2 cells
# Example from Figure 7.4
# Author: Jake VanderPlas
# License: BSD
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import NMF
#from sklearn.decomposition import RandomizedPCA
from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)
```

```python
#----------------------------------------------------------------------
# Compute PCA, and NMF components
def compute_PCA_NMF(n_components=5):
    spec_mean = spectra.mean(0)

    # PCA: use randomized PCA for speed
    #pca = RandomizedPCA(n_components - 1)
    pca = PCA(n_components - 1,svd_solver='randomized')
    pca.fit(spectra)
    pca_comp = np.vstack([spec_mean, pca.components_])

    # NMF requires all elements of the input to be greater than zero
    spectra[spectra < 0] = 0
    nmf = NMF(n_components)
    nmf.fit(spectra)
    nmf_comp = nmf.components_

    return pca_comp, nmf_comp

n_components = 5
decompositions = compute_PCA_NMF(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 10))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['PCA components', 'NMF components']

for i, comp in enumerate(decompositions):
    for j in range(n_components):
        ax = fig.add_subplot(n_components, 3, 3 * j + 1 + i)

        ax.yaxis.set_major_formatter(plt.NullFormatter())
        ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
        if j < n_components - 1:
            ax.xaxis.set_major_formatter(plt.NullFormatter())
        else:
            ax.set_xlabel('wavelength (Angstroms)')

        ax.plot(wavelengths, comp[j], '-k', lw=1)

        # plot zero line
        xlim = [3000, 7999]
        ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
        ax.set_xlim(xlim)

        if j == 0:
            ax.set_title(titles[i])

        if titles[i].startswith('PCA') or titles[i].startswith('ICA'):
            if j == 0:
                label = 'mean'
            else:
                label = 'component %i' % j
        else:
            label = 'component %i' % (j + 1)

        ax.text(0.03, 0.94, label, transform=ax.transAxes,
                ha='left', va='top')

        for l in ax.get_xticklines() + ax.get_yticklines():
            l.set_markersize(2)

        # adjust y limits
        ylim = plt.ylim()
        dy = 0.05 * (ylim[1] - ylim[0])
        ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

plt.show()
```

## Independent Component Analysis (ICA)

For data where the components are statistically independent (or nearly so) [Independent Component Analysis (ICA)](https://en.wikipedia.org/wiki/Independent_component_analysis) has become a popular method for separating mixed components.  The classical example is the so-called "cocktail party" problem.  This is illustrated in the following figure from Hastie, Tibshirani, and Friedman (Figure 14.27 on page 497 in my copy, so they have clearly added some stuff!).  Think of the "source signals" as two voices at a party.  You are trying to concentrate on just one voice.  What you hear is something like the "measured signals" pattern.  You could run the data through PCA and that would do an excellent job of reconstructing the signal with reduced dimensionality, but it wouldn't actually isolate the different physical components (bottom-left panel).  ICA on the other hand can (bottom-right panel).

.](../images/HastieFigure14_37.png)

[Hastie et al.](https://web.stanford.edu/~hastie/ElemStatLearn/): "ICA applied to multivariate data looks for a sequence of orthogonal projections such that the projected data look as far from Gaussian as possible. With pre-whitened data, this amounts to looking for
components that are as independent as possible."

In short you want to find components that are maximally non-Gaussian since the sum of 2 random variables will be more Gaussian than either of the components (remember the Central Limit Theorem).  Hastie et al. illustrate this as follows:

ICA is a good choice for a complex system with relatively indepent components.  For example a galaxy is roughly a linear combination of cool stars and hot stars, and a quasar is just a galaxy with others component from an accretion disk and emission line regions.  Ideally we want "eigenvectors" that are aligned with those physical traits/regions as opposed to mathematical constructs.

The basic call to the [FastICA algoirthm](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.FastICA.html) in Scikit-Learn looks like:

```python
# Execute this cell
import numpy as np
from sklearn.decomposition import FastICA
X = np.random.normal(size=(100,2)) # 100 objects in 2D
R = np.random.random((2,5)) # mixing matrix
X = np.dot(X,R) # Simulation of a 5D data space
ica = FastICA(2) # Now reproject to 2-D
ica.fit(X)
proj = ica.transform(X) # 100x2 projection of the data
comp = ica.components_ # 2x5 matrix of independent components
## sources = ica.sources_ # 100x2 matrix of sources
```

Execute the next 2 cells to produce a plot showing the ICA components.

```python
%matplotlib inline
#Example from Andy Connolly
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import FastICA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 500
n_components = 5
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0)
# spec_mean = spectra[:50].mean(0)

ica = FastICA(n_components - 1)
ica.fit(spectra[ind])
ica_comp = np.vstack([spec_mean,ica.components_]) #Add the mean to the components
```

```python
#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'ICA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, ica_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

As with PCA and NMF, we can similarly do a reconstruction:

```python
# Execute this cell
#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
evecs = data['evecs']
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 2, 4, 8]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

Ivezic, Figure 7.4 compares the components found by the PCA, ICA, and NMF algorithms.  Their differences and similarities are quite interesting.

If you think that I was pulling your leg about the cocktail problem, try it yourself!
Load the code instead of running it and see what effect changing some things has.

```python
%load ../code/plot_ica_blind_source_separation.py
```

Let's revisit the digits sample and see what PCA, NMF, and ICA do for it.

```python
## Execute this cell to load the digits sample
%matplotlib inline
import numpy as np
from sklearn.datasets import load_digits
from matplotlib import pyplot as plt
digits = load_digits()
grid_data = np.reshape(digits.data[0], (8,8)) #reshape to 8x8
plt.imshow(grid_data, interpolation = "nearest", cmap = "bone_r")
print(grid_data)
X = digits.data
y = digits.target
```

Do the PCA transform, projecting to 2 dimensions and plot the results.

```python
# PCA
from sklearn.decomposition import ___
pca = PCA(n_components = ___)
pca.___(___)
X_reduced = pca.transform(___)

plt.scatter(X_reduced[:,___], X_reduced[:,___], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

Similarly for NMF and ICA

```python
# NMF
from sklearn.decomposition import ___
nmf = NMF(___)
nmf.___(___)
X_reduced = nmf.___(___)

plt.scatter(___, ___, c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

```python
# ICA
from sklearn.decomposition import ___
ica = FastICA(___)
ica.___(___)
X_reduced = ica.___(___)

plt.scatter(___, ___, c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

Take a second to think about what ICA is doing.  What if you had digits from digital clocks instead of handwritten?

I wasn't going to introduce [Neural Networks](https://en.wikipedia.org/wiki/Artificial_neural_network) yet, but it is worth noting that Scikit-Learn's [`Bernoulli Restricted Boltzman Machine (RBM)`](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) is discussed in the [(unsupervised) neural network](http://scikit-learn.org/stable/modules/neural_networks_unsupervised.html) part of the User's Guide and is relevant here as the data input must be either binary or values between 0 and 1, which is the case that we have here.

We could think about doing dimensional reduction of the digits data set in another way.  There are 64 pixels in each of our images.  Presumably all of them aren't equally useful.  Let's figure out exactly which pixels are the most relevant.  We'll use Scikit-Learn's [`RandomForestRegressor`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html).  We won't get to regression until next week, but you don't need to understand the algorithm to do this, just look at the inputs and outputs.  Which pixels are the most important?  As a bonus see if you can plot digit images with those pixels highlighted.

```python
from sklearn.ensemble import RandomForestRegressor
RFreg = RandomForestRegressor()# Complete or leave blank as you see fit
RFreg.fit(X,y)# Do Fitting
importances = RFreg.feature_importances_# Determine "importances"
pixelorder = np.argsort(importances)[::-1] #Rank importances (highest to lowest)
print(pixelorder)

plt.figure()
plt.imshow(np.reshape(importances,(8,8)),interpolation="nearest")
plt.show()
```


# Exam project

```python
#import package
import random
import sympy as sm
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.cm as cm
import ipywidgets as widgets
from scipy import optimize
import pandas as pd
from scipy import optimize

#setting nice printing and seed
sm.init_printing(use_unicode=True)
np.random.seed(1)
```

# 1. Human capital accumulation

We use a mixture of methods to solve the questions is assigment 1. In the first question we solve the functions using sympy, and implement the functions manually. In question 2 and 3 the equations becomes more complicated to solve, and therefore we use numerical optimizers.

```python
# We need to define all the symbols that we wish to work with in sympy, as sm.symbols
#Initialize symbols
rho = sm.symbols('rho')
beta = sm.symbols('beta')
gamma = sm.symbols('gamma')
w = sm.symbols('w')
b = sm.symbols('b')
delta = sm.symbols('Delta')
h1 = sm.symbols('h_1')
h2 = sm.symbols('h_2')
c1 = sm.symbols('c1')
c2 = sm.symbols('c2')
l1 = sm.symbols('l1')
l2 = sm.symbols('l2')
exp = sm.symbols('{E_1}[{v_2}(h_2)]')
u = sm.symbols("u")
```

**Question 1**

To solve the model for period 2, firstly we have to define the human capital, budget constraint and utility function :

```python
#define the utility of working, desutility of working and the total utility equation
## Utility of working
u2 = (c2**(1-rho))/(1-rho)
## Disutility of working
dis_u2 = gamma*l2
## Total utility
total_u2 = u2 - dis_u2

#define the consumption in period, as a function of wage, human capital and work/not work
c2_f = w*h2*l2

#show the total utility function
total_u2
```

```python
#substitute the budget constraint in the utility function
total_u2_sub2 = total_u2.subs(c2,c2_f)
total_u2_sub2
```

```python
#solve the equation for human capital level
h2_sol = sm.solve(sm.Eq(total_u2_sub2,u),h2)[0]
h2_sol
```

Now that we have the equations of this model, we will define two functions. The first one will return the utility level given the parameters. The second one will return the human capital level.

The budget constraint is not the same whether the agent works or lives with the benefits. Therefore we had to use an if else statement to distinguish both case in the utility function.

```python
# define the total utility function
def utility_work(h2,l2,w,rho,gamma,b):
    """This function takes 6 required arguements, and returns utility given the values. Build on the basis of total_u2_sub2
        args:
            h2 = Human capital level
            l2 = dummy variable for work or not. Should either be 1 or 0
            w = wage for working
            rho = Some parameter
            gamma = some parameter determening the disutility of working
            b = uenemployment benefts

            return the utility """
    #If worker works
    if l2 == 1 :
        #Estimate solition based on total_u2_sub2
        solution = (((h2*l2*w)**(1-rho))/(1-rho))-gamma*l2
    #If worker does not work
    else :
        #Estimate solution bases og total_u2_sub2, but replace consumbtion with benefits (b)
        solution = ((b**(1-rho))/(1-rho))-gamma*l2
    return solution

def hum_cap_u(l2,gamma,b,w,rho):
    """This function takes 5 arguments and return the human capital level. It is build on h2_sol
        args:
            l2 = dummy for work or not. Should either be 0 or 1
            gamma = some parameter determening the disutility of not working
            b = unemployment benefits
            w = wages
            rho = some parameter
            returns human capital level given the parameters"""
    # We use h2_sol, but instead of the utility we plug in the utility of not working, as a "cutoff"-point for
    #when the workerwill work.
    #
    #Calculate the utility of not working
    nw_u = utility_work(h2,0,w,rho,gamma,b)

    #Calculate the human capital level where the utility is higher for working than not
    hum_cap = ((-gamma*l2*rho+gamma*l2-rho*nw_u+nw_u)**-(1/(rho-1)))/(l2*w)

    return hum_cap
```

Now we wish to plot the model, to see how utility depends on human capital level

```python
# set the given parameters of the model
rho = 2
beta = 0.96
gamma = 0.1
w = 2
b = 1
delta = 0.1
```

```python
# define a plot with the utility given the human capital
def plot_func(h2,w,rho,gamma,b):
    """This function takes 5 arguments and plots utility based on human captial, for both working and not working
        args:
            h2 = human capital
            w = wage
            rho = some parameter
            gamma = parameter determening the disutility of working
            b = unemployment benefits
            does not return anything, but prints a plot"""
    #Calculate the utility of working
    work = utility_work(h2,1,w,rho,gamma,b)

    #calculate the utility of not working
    not_work = utility_work(h2,0,w,rho,gamma,b)

    #Calculate the "cutoff"-point
    cutoff = hum_cap_u(1,gamma,b,w,rho)

    #plot work and not work
    plt.plot(h2,work,label = "Utility of working")
    plt.plot(h2,not_work,color="red",label="Utility of not working")
    plt.legend()

    plt.xlabel("Human capital")
    plt.ylabel("Utility")

    plt.show

    #Print the cutoff point
    print("The worker will supply labour if human capital is above",round(cutoff[0],2), "and no labour if it is below." )
    return
```

To plot the utility we define two vectors. One for human capital level, and one for benefits.

```python
#define vectors
h_vec = np.linspace(0.1,1.5,100)
b_vec = np.linspace(1,1,100)
plot_func(h_vec,w,rho,gamma,b_vec)
```

Here we can see a positive relation between human capital and the utility. This utility is negative, but the important interpertation is that it rises with level of human capital. The red line represents the utility from not working, therefore the human capital is lower than the level where the red line crosses the blue one, the agent will chose not to work.

Given this plot, we can assume that the agent will chose to work for a human capital level higher than 0.56. If it is not the case the agent should take the benefits and get -1 as utility from it.

```python
#The human capital level of the cutoff
cutoff = hum_cap_u(1,gamma,b,w,rho)
print(round(cutoff,2),"human capital")
```

    0.56 human capital

Another way to show this, and emphasize the cutoff, is to plot a graph with the labour supply on the y-axis. To do that we create a list (size 100). We assign 0 or 1 to this list depending on the value inside the human capital vector h_vec. If the value is higher than the cutoff we assign 1, which means the agent will work, or 0 in the other case.

```python
#Create the list
willwork = list(range(100))

#Use a for loop to fill the list with (1) if the human capital i is above the cutoff or (0) if it is equal or below
for i in range(len(h_vec)):
    #if utility of work higher than utility of not working
    if h_vec[i] > cutoff :
        #Assing 1
        willwork[i] = 1
    else :
        #Otherwise assign 0
        willwork[i] = 0
```

From these lists, now we plot the labour supply given the human capital :

```python
#We now wish to plot the cutoff in labour supply given the human capital
##Make a vector of the cutoff value
cutoff_vec = np.linspace(cutoff,cutoff,100)

#plot the labour supply as a function of human capital
plt.plot(h_vec,willwork)

#Plot cutoff line
plt.plot(cutoff_vec,willwork,linestyle="dashed",color="red",label=("Cutoff human capital level"))
plt.legend()
plt.xlabel("Human capital")
plt.ylabel("Labour supply")

plt.show
```

If the reader wished to examine the "cutoff"-point for themselves, we have provided an interavtive slide below, where we show if the worker will supply labour at a given human capital level

```python
#the labour supply as a function of human captal is then:
def labour_supply(hum_cap,w,rho,gamma,b):
    """This function ins used for interaction between human capital and the cutoff point
        args:
            hum_cap = human capital level
            w = wage
            rho = some parameter
            gamma = parameter determening disutility of not working
            b = benefits
            returns laboursupply as either 1 or 0 """
    #calculate cutoff
    cutoff = hum_cap_u(1,gamma,b,w,rho)

    #if cutoff is lower than hum cap, print that the worker will work and assign 1
    if hum_cap > cutoff :
        print("with a human capital level of",hum_cap,"the worker wil supply 1 labour (l = 1)")
        l = 1
    #if cutoff is higher or equal to hum cap, print that the worker will not work and assign 0
    else :
        print("with a human capital level of",hum_cap,"the worker will supply 0 labour (l=0)")
        l = 0
    return l
```

```python
#Create a slider with small float steps
human_capital = widgets.FloatSlider(min=0.1,max=1.5,value=0.1,step=0.01)
#Call interact on the function and fix all arguments except human capital
widgets.interact(labour_supply,hum_cap=human_capital,w=widgets.fixed(w),rho=widgets.fixed(rho),gamma=widgets.fixed(gamma),b=widgets.fixed(b))
```

    interactive(children=(FloatSlider(value=0.1, description='hum_cap', max=1.5, min=0.1, step=0.01), Output()), _…

    <function __main__.labour_supply(hum_cap, w, rho, gamma, b)>

**Question 2**

```python
#We redefine some of the symbols that we have used earlier
rho = sm.symbols('rho')
beta = sm.symbols('beta')
gamma = sm.symbols('gamma')
w = sm.symbols('w')
b = sm.symbols('b')
delta = sm.symbols('Delta')
```

Now that we have solved for period two we can do the same with the first period. We will begin, as before, by defining the utility, disutility and total utility equations and use them to create a utility function.

```python
#define the utility, disutility and total utility equations for period 1
##Utility of working
u1 = (c1**(1-rho))/(1-rho)
##Utility of not working
dis_u1 = gamma*l1

##Total utility
total_u1 = u1 - dis_u1 + beta*exp

#show total utility
total_u1
```

```python
#set the parameters of the model as their numeric values
rho = 2
beta = 0.96
gamma = 0.1
w = 2
b = 1
delta = 0.1

#define the utility function for period 2
def utility_2(h1,w=1,l1=1,rho=2,gamma=0.1,beta=0.96,delta=0.1,b=1):
    """This function takes 8 arguments and return the utility of either working or not working
        args:
            h1 = human capital leve
            w = wage
            l1 = work dummy. Should be either 0 or 1
            rho = some parameter
            gamma = parameter determening disutility of working
            beta = discout factor
            delta = human capital accumulation parameter
            b = benefits
            return the utility """
    #If the worker works
    if l1 == 1:
        #calculate human capital leve in period 2 as a function of human capital level in period 1
        h2 = h1+1 +(0.5*delta)
        #Substitute c for its components
        c = w*h1*1

        #Calculate expected utility in period 2
        u_p2 = utility_work(h2,1,w,rho,gamma,b)

        #Calculate the utility in period 1
        solution = ((c**(1-rho))/(1-rho))-gamma*1 +beta*u_p2
    else :
        #calculate human capital leve in period 2 as a function of human capital level in period 1
        h2 = h1 + 0 + (0.5*delta)
        #Substitute c for benefits when not working
        c = b

        #Calculate expected utility in period 2
        u_nw_p2 = utility_work(h2,0,w,rho,gamma,b)

        #Calculate the utility in period 1
        solution = ((c**(1-rho))/(1-rho))-gamma*0 +beta*u_nw_p2

    return solution
```

We create two lists for the utility values when working and not working :

```python
#Store the utility values given all the human capital values if agent works inside x
x = utility_2(h_vec,w,1,rho,gamma,beta,delta,b)

#Store the utility values given both h_vec and b_vec if the agent does not work
y = utility_2(h_vec,w,0,rho,gamma,beta,delta,b_vec)
```

The level of utility for not working in both periods is :

```python
#The utility of not working
y[0]
```

The function has become to complicated so solve symbolically, so we use a numerical optimzer to find the human capital level, where it makes sense to work. This is given if we mazimixe the utility function towards the cutoff point of -1.96

```python
#We define objective function
def objective(h):
    #This is given as the utility function
    return utility_2(h,w,1,rho,gamma,beta,delta,b)
#Definin constraints
def constraint(h):
    #The constraints are the cutoff point + the utility, as the utilities are negative
    return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
```

```python
#Define constraints as a dictionary
cons = ({"type": "ineq","fun":constraint})
#Initial valie of human capital leve set to 0.1
h_init = 0.1
#Use mimimize to optimze and print solution
sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
sol
```

         fun: -1.9599999941585289
         jac: array([4.28598835])
     message: 'Optimization terminated successfully.'
        nfev: 21
         nit: 7
        njev: 7
      status: 0
     success: True
           x: array([0.35172508])

```python
#The cutoff level of human capital is given as
sol.x[0]
```

We wish to plot this similar to in question 1, so we do below.

```python
#define a plot with the utility given the human capital
##define cutoff human capital as a vector to a line can be plottet
cutoff = np.linspace(sol.x[0],sol.x[0],100)
##Plot utility of working
plt.plot(h_vec,x, label = "Utility of working",alpha= 0.8)
##Plot utility of not working
plt.plot(h_vec,y, color="green", label = "Utility of not working",alpha= 0.8)
##Plot the cutoff level of human capital
plt.plot(cutoff,x,color = "red", linestyle = "dashed",label="Labour cutoff",alpha= 0.4)
##Plot a dot in the intersection bestween the cutoff human capital and the utility fuction for working
plt.scatter(cutoff[0],y[0],color="purple",alpha = 1)
plt.xlabel("Human capital level")
plt.ylabel("Utility")
plt.legend()
plt.show()
```

As before, we will plot the cutoff with the same way. We create a list and assign 0 or 1 given the human capital value (if it is higher or lower than the cutoff). After that we will plot these both list to get the labour supply given the human capital.

```python
#Create the list
willwork_p1 = list(range(100))

#Use a for loop to fill the list with (1) if the human capital i is above the cutoff or (0) if it is equal or below
for i in range(len(h_vec)):
    #If utiliy of working is strictly higher than utility of not working, assign 1
    if utility_2(h_vec[i],w,1,rho,gamma,beta,delta,1) > utility_2(h_vec[i],w,0,rho,gamma,beta,delta,1) :
        willwork_p1[i] = 1
    else :
        #Else assign 0
        willwork_p1[i] = 0
```

```python
#plot the cutoff in labour supply given the human capital
##Plot labour supply
plt.plot(h_vec,willwork_p1)
## plot the cutoff human capital level as a red dashed line
plt.plot(cutoff,np.linspace(0,1,100),linestyle = "dashed",color="red",label=("Cutoff human capital level"))
plt.legend()
plt.xlabel("Human capital")
plt.ylabel("Labour supply")
```

This means that the worker will chose to supply her labour if her human capital level is above 0.35. If the human capital is under, she won't chose to work and live from the benefits.

## Question 3

We will examine if workers will ever work if their wage is lower than the unemployment benefits. In this case the unemployment benefits are given as 1. It is theoretically possible that a worker would chose to work, if the accumulation of human capital can more than make up for the loss in utility. To test this we chose a list of the possible wages below 1, and test if there is a human capital level, where the utility of working is higher than not working

```python
#We define a list of different wage levels to examine
wages = np.linspace(0,0.99,100)
```

```python
#parameters
rho = 2
beta = 0.96
gamma = 0.1
b = 1
delta = 0.1
h_vec = np.linspace(0.1,1.5,100)
```

We create a for loop to test if the utility gained from working, even if wages are lower than b, is higher than the utility gotten when not working in both periods (which was -1.96).

```python
#For-loop that tests if utility from working is higher than not working for a wage lower than the benefits

#Initate. Looping over alle possibel wages under 1 2 decials (that is the highest we test is 0.99)
for wage in wages:
    #define a name for outputting
    name = "wage_" + str(round(wage,2))

    #Calculate the cutoff. This depends on the wage so it has to be done inside loop.
    ## this is a list as we use h_vec which contain all levels of human capital
    cutoff_w = utility_2(h1 = h_vec, w = wage, l1 = 1, rho = 2, gamma = 0.1, beta= 0.96, delta = 0.1, b = 1)

    #Check if the maximum utility given the wage is higher that -1.96, the utility of not working
    if max(cutoff_w) > -1.96:
        #If it is print the name
        print(name)
        #This will be the lowest wage where it is profitable to work
        ##Therefore break, as any higher wage will be redundant to calculate
        break
```

    wage_0.6

    C:\Users\augus\Anaconda3\lib\site-packages\ipykernel_launcher.py:16: RuntimeWarning: divide by zero encountered in reciprocal
      app.launch_new_instance()
    C:\Users\augus\Anaconda3\lib\site-packages\ipykernel_launcher.py:33: RuntimeWarning: divide by zero encountered in reciprocal

It turns out that it could be profitable for the worker to work even if the wage is 0.6 while the unemployment benefits are 1 if the human capital is high.

To illustrate this result, we plot the utility given the human capital for a wage equal to 2 and a wage equal to 0.6. We also print the cutoff to show when it is better to work.

```python
#define a plot with the utility given the human capital

#plot the utility function as wage level 2
plt.plot(h_vec,x, label = "Wage = 2",alpha= 0.6)
#plot the utility of not working
plt.plot(h_vec,y, color = "red", linestyle = "dashed",alpha= 0.5, label = "Utility of not working")
#Plot the utility of working when wages are 0.6
plt.plot(h_vec,cutoff_w, color = "green", label = "Wage = 0.6")
plt.xlabel("Human capital level")
plt.ylabel("Utility")
#Set new limits
plt.ylim(-5,0)
plt.legend()
plt.show()
```

It is evident that at very high levels of human capital, it can be rational to work, even if the wage is lower than the
unemployment benefits. From our loop before we know that the lowest wage where this is true is 0.6. Therefore we
use optimization to find the human capital level at which i makes sense to work if your wage is 0.6.

```python
#define the objective. Againg this is the utility function
def objective(h):
    #Insert 0.6 as wage into function
    return utility_2(h,0.6,1,rho,gamma,beta,delta,b)

#Define the constraint function. This is again 1.96 + the utility as utility is negative
def constraint(h):
    #Insert 0.6 as wage into function
    return (1.96 + utility_2(h,0.6,1,rho,gamma,beta,delta,b))
```

```python
cons = ({"type": "ineq","fun":constraint})
h_init = 0.1
sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
sol
```

         fun: -1.960000000389105
         jac: array([1.01749761])
     message: 'Optimization terminated successfully.'
        nfev: 27
         nit: 9
        njev: 9
      status: 0
     success: True
           x: array([1.47461354])

```python
#The human capital where it makes sense to work, at a wage of 0.6 is:
sol.x[0]
```

This is not the limit 1.5, so to be sure we test that the a wage of 0.59 does not have an assosiated human capital level where the worker would work.

```python
# I test that there is no wage value below 0.6 where the worker would work
w = 0.59
#We only use 2 decimal so 0.59 is the closest to 0.6

#Same as before, define objective function
def objective(h):
    return utility_2(h,w,1,rho,gamma,beta,delta,b)
#Define cnstriants
def constraint(h):
    return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
#Make it into a dictionary
cons = ({"type": "ineq","fun":constraint})
#Intial value of human capital
h_init = 0.1

#Solve
sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)

#Check if the solutions is lower than the limit of 1.5
if sol.x[0] > 1.5:
    print("No level of human capital where the worker would work, at human capital level of",str(w))
else:
    print("The worker til work if she has a human capital level of", str(sol.x[0]))
```

    No level of human capital where the worker would work, at human capital level of 0.59

So the worker will work at a wage of 0.6, if she has a human capital level of 1.47 or higher. But not if she has a wage of 0.59

We will plot the utility function at different wage levels in order to show how the workes calculation changes based on the wages.

```python

# Define wages to plot - chosen for nice visual and intuitive representateion
wages = [0.6,0.7,0.8,0.9,0.99]

# And the parameters again
rho = 2
beta = 0.96
gamma = 0.1
b = 1
delta = 0.1
h_vec = np.linspace(0.1,1.5,100)
```

```python
#Initiate a for loop, for each wage in the defines wages list
for wage in wages:
    #Calculate the utility function of working at each wage level
    wage_vec = utility_2(h1 = h_vec, w = wage, l1 = 1, rho = 2, gamma = 0.1, beta= 0.96, delta = 0.1, b = 1)
    #plot the function ans assing a name corresponding to the wage level
    plt.plot(h_vec,wage_vec, label = "Wage ="+str(wage))
#plot the utility at wage 2
plt.plot(h_vec,x, label = "Wage = 2",alpha= 0.6)
#plot utility of not working
plt.plot(h_vec,y, color = "red", linestyle = "dashed",alpha= 0.5, label = "Utility of not working")
plt.xlabel("Human capital level")
plt.ylabel("Utility")
plt.ylim(-5,-0.5)
plt.legend()
plt.show()
```

The respective "cutoff"-levels of human capital can be calculated using numerical optimazation

```python
#Initiate a for-loop for each of the wages in our wages list
for wage in wages:
    #assing wage to w
    w = wage
    #Then define an objective function with the given wage
    def objective(h):
        return utility_2(h,w,1,rho,gamma,beta,delta,b)
    #Then define the constraints with the given wage
    def constraint(h):
        return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
    #Constriants as a dictionary
    cons = ({"type": "ineq","fun":constraint})
    #Define initial guess
    h_init = 0.1

    #Solve the optimazation
    sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
    #Print the cutoff level of human capital and the wage level
    print("Human capital level =",round(sol.x[0],2),"with wages =", wage)
```

    Human capital level = 1.47 with wages = 0.6
    Human capital level = 1.23 with wages = 0.7
    Human capital level = 1.05 with wages = 0.8
    Human capital level = 0.91 with wages = 0.9
    Human capital level = 0.81 with wages = 0.99

# 2. AS-AD Model :

To solve assignment 2 we primarily use the sympy functionality. Towards the end we use some some numerical optimization.

```python
#All the symbold we wish to use for symbolic calculations are defined as sm.symbols
alpha = sm.symbols('alpha')
b = sm.symbols('b')
gamma = sm.symbols('gamma')
h = sm.symbols('h')
phi = sm.symbols('phi')
yt = sm.symbols('y_t')
pit = sm.symbols('pi_t')
pit_m1=sm.symbols('pi_t-1')
yt_m1 = sm.symbols('y_t-1')
vt = sm.symbols('v_t')
st = sm.symbols('s_t')
st_m1 = sm.symbols('s_t-1')
rt = sm.symbols('r_t')
it = sm.symbols('i_t')
exp_pit = sm.symbols('E[pi_t]')
exp_pit_p1 = sm.symbols('E[pi_t+1]')
```

**Question 1**

Now that we have the symbols let's write the SRAS curve with them :

```python
# Write the SRAS base equation
SRAS_pit=pit_m1-phi*gamma*yt_m1+st-phi*st_m1+gamma*yt
SRAS_pit
```

We do the same with the AD curve :

```python
# Write the AD base equation
AD_pit=(1/(h*alpha))*(vt-(1+b*alpha)*yt)
AD_pit
```

Now we will equalize SRAS and AD to find the equilibrium values. To do that we substract one from the other (which is the same as equalizing them):

```python
# SRAS equation minus the AD equation
AS_minus_AD= SRAS_pit-AD_pit
AS_minus_AD
```

We solve this equation for $y_t$ :

```python
# Solve the equation for yt
eq_y = sm.solve(sm.Eq(AS_minus_AD,0),yt)
eq_y[0]
```

We get the equation of the equilibrium output $y_t$ which depends on $\alpha , \phi ,\gamma , h, b, \pi_{t-1} ,s_{t-1} ,y_{t-1} ,v_t$ and $s_t $.

Now that we have the equilibrium equation of the output, let's insert it in the SRAS curve :

```python
# Substitute the new output equation to replace yt
eq_pi= SRAS_pit.subs(yt,eq_y[0])
eq_pi
```

We obtain the equation of the equilibrium inflation which depends on $\gamma ,\phi , s_t ,\pi_{t-1} ,y_{t-1}$ and $s_{t-1}$

**Question 2**

We want to find the output and inflation for the equilibrium and see the evolution when there is a demand disturbance.
To do that we will create a dictionary for the parameters of this question, and create two functions for the output and inflation from the equation we found previously. We will use these functions to get the output and inflation levels in the equilibrium and after a shock and plot the results.

First of all we create a dictionary to enter the value of our parameters ($\alpha$, $h$, $b$, $\phi$, $\gamma$) :

```python
# Initialization of the dictionary
pms = {}

# Enter the parameters and their values
pms['alpha']=5.76
pms['h'] = 0.5
pms['b'] = 0.5
pms['phi'] = 0
pms['gamma'] = 0.075
pms
```

    {'alpha': 5.76, 'h': 0.5, 'b': 0.5, 'phi': 0, 'gamma': 0.075}

From the equation of the output we found in question one ( "eq_y" ), we create a function by using lambdify.

```python
# Our eq_y is a list, therefore we need to consider only the first element of it to use the lambdify function.
equil_output= sm.lambdify((vt, st, pit_m1, st_m1, yt_m1, alpha, h, b, phi, gamma),eq_y[0])

# Define the function
def equili_output (vt, st, pit_m1, st_m1, yt_m1, alpha=pms['alpha'], h=pms['h'], b=pms['b'], phi=pms['phi'], gamma=pms['gamma']):
    """
    This function take five parameters plus five others which are by default define in the dictionary called "pms" and
    return the output of the economy.

        args :  vt (float) : the demand disturbance in period t
                st (float) : the supply disturbance in period t
                pit_m1 (float) : the inflation in period t-1
                st_m1 (float) : the supply disturbance in period t-1
                yt_m1 (float) : the output in period t-1

        return (float) : The output yt of the economy for the period t.

    """
    return equil_output (vt, st, pit_m1, st_m1, yt_m1, alpha, h, b, phi, gamma)
```

Now let's enter values of the variables at the equilibrium, which means $v_t = s_t = \pi_{t-1} = s_{t-1} = y_{t-1} = 0$ :

```python
equili_output(0,0,0,0,0)
```

The output is zero ($y_t = 0$) which makes sense given that all the other variables are equal to 0.

Now let's see what would be the impact of a positive demand disturbance on the production. For that let's just set $v_t = 0.1$ instead of 0 :

```python
equili_output(0.1,0,0,0,0)
```

When there is a positive disturbance of the demand $v_t$ we can observe an increase in the output. Indeed an increase in demand will stimulate the economy, the supply will adapt to the demand, increase as well, leading to more production.

Now let's see the impact of the demand disturbance on the **inflation** :

As before we use the inflation equation we found in question one ( "eq_pi" ) to create a function.

```python
# Use lambdify on the inflation equation eq_pi
equil_pi= sm.lambdify((st, yt, pit_m1, st_m1, yt_m1, gamma, phi),SRAS_pit)

# Define the function
def equili_pi (st, yt, pit_m1, st_m1, yt_m1, gamma=pms['gamma'], phi=pms['phi']):
    """
    This function take five parameters plus two others which are by default define in the dictionary called "pms" and
    return the inflation of the economy.

        args :  st (float) : the supply disturbance in period t
                yt (float) : the output in period t
                pit_m1 (float) : the inflation in period t-1
                st_m1 (float) : the supply disturbance in period t-1
                yt_m1 (float) : the output in period t-1

        return (float) : The inflation pit of the economy for period t.

    """
    return equil_pi (st, yt, pit_m1, st_m1, yt_m1, gamma, phi)
```

```python
equili_pi(0,0,0,0,0)
```

At the equilibrium, when $v_t = s_t = \pi_{t-1} = s_{t-1} = y_{t-1} = 0$ the inflation is zero ($\pi_t = 0$).
Regarding the demand disturbance, it is not involved in the inflation equation, however its impact increased the output in period t. This increase in output will impact the inflation :

```python
equili_pi(0,0.0244140625,0,0,0)
```

The positive disturbance on the demand also increases the inflation.
We can summarize the effect of this disturbance as follows:
   1. The positive shock on $v_t$ leads to a higher demand
   2. The supply and the demand adjust on a higher production and therefor a higher output $y_t$
   3. Finally, a higher output leads to inflation, an increase in $\pi_t$

**Question 3**

In this question we introduce a demand and a supply shock inside the demand and supply disturbance. Now both these variables become a function of shocks and some parameters.

In theses conditions we want to know how the economy will evolve, starting from the equilibrium, if there is a demand shock in the first period. To answer this question we will first adapt our model to the new definitions of $v_t$ and $s_t$.

Now we have two new parameters ($\delta$, $\omega$) and $v_t$ and $s_t$ which became functions. We have to set up this new configuration:

```python
#add the new parameters into the dictionary
pms["delta"] = 0.80
pms["omega"] = 0.15

#define the new symbols
delta = sm.symbols("delta")
omega = sm.symbols("omega")
vt_m1 = sm.symbols("v_t-1")
xt = sm.symbols("x_t")
ct = sm.symbols("c_t")

#define the new vt and st equation
vt_ar = delta*vt_m1 + xt
st_ar = omega*st_m1 + ct
```

To deal with these changes, we have to replace our $v_t$ in the output equation by the new one which is vt_ar :

```python
#The output equilibrium equation (our previous equation):
eq_y[0]
```

```python
#substitute vt by vt_ar
eq_y_ar = eq_y[0].subs(vt,vt_ar)
eq_y_ar
```

```python
#substitute st by st_ar
eq_y_ar2 = eq_y_ar.subs(st, st_ar)
eq_y_ar2
```

We have to do the same in the SRAS equation, by replacing $s_t$ by st_ar:

```python
#The SRAS equation (our previous equation):
SRAS_pit
```

```python
#substitute st by st_ar
eq_pi_ar = SRAS_pit.subs(st,st_ar)
eq_pi_ar
```

Now that we changed our equation, we also have to make these changes inside our previous functions :

```python
#create a new output function, with the new parameters
## This function is the aurtoregressive equivilant of the output function defines earlier
equil_output2= sm.lambdify((vt_m1, pit_m1, st_m1, yt_m1, ct, xt, alpha, h, b, phi, omega, gamma, delta),eq_y_ar2)
def equili_output_ar (vt_m1, pit_m1, st_m1, yt_m1, ct, xt, alpha=pms['alpha'], h=pms['h'], b=pms['b'], phi=pms['phi'], omega=pms['omega'], gamma=pms['gamma'], delta=pms['delta']):
    return equil_output2 (vt_m1, pit_m1, st_m1, yt_m1, ct, xt, alpha, h, b, phi, omega, gamma, delta)
```

```python
#create a new inflation function, with the new parameters
## This function is the aurtoregressive equivilant of the inflation function defines earlier
equil_piii= sm.lambdify((yt_m1, yt, pit_m1, st_m1, ct, omega, gamma, phi),eq_pi_ar)
def equili_pi_ar (yt_m1, yt, pit_m1, st_m1, ct, omega=pms['omega'], gamma=pms['gamma'], phi=pms['phi']):
    return equil_piii (yt_m1, yt, pit_m1, st_m1, ct, omega, gamma, phi)
```

```python
#test at the initial state
equili_pi_ar(0,0,0,0,0)
```

Now let's enter the new output value to get the new inflation value:

```python
equili_pi_ar(0,equili_output_ar(0,0.1,0,0,0,0,0), 0, 0, 0)
```

```python
equili_output_ar(0,0,0,0,0.1,0)
```

In order to examine how an output shock would affect the system we initiate a for-loop for the 100 periods we wish to model, and let the parameters evolve according to the functions

```python
#define the number of time period
T = 100

#Define shock and intial parameters
shock = 0.1
pi_m = 0
yn = 0
y_m = 0

#Define list to input our results in
list_output = list(range(100))
list_inflation = list(range(100))

```

```python
#create a loop to see the evolution of output and inflation
for i in range(100):

    #If it is the second period
    if i == 1 :
        #Calculate output
        y = equili_output_ar(0,pi_m,0,y_m,0,shock)
        #record output in this period
        list_output[i] = y
        #define output for next period
        yn = y

        #calculate inflation
        p = equili_pi_ar(y_m,yn,pi_m,0,0)
        #record inflation in this period
        list_inflation[i] = p
        #define lagged inflation
        pi_m = p
        #define lagged output
        y_m = y

    #If the period is not the first or second
    elif i> 1 :
        #Calculate output
        y = equili_output_ar(0,pi_m,0,y_m,0,0)
        #Record output
        list_output[i] = y
        #define output for next period
        yn = y

        #calculate inflation
        p = equili_pi_ar(y_m,yn,pi_m,0,0)
        #record inflation in this period
        list_inflation[i] = p

        #define lagged inflation
        pi_m = p
        #define lagged output
        y_m = y

    #If it is the intial period, that is where the shock is 0
    else :
        #Calculate output before any shocks
        y = equili_output_ar(0,0,0,0,0,0)
        #Record output
        list_output[i] = y
        #define output for next period
        yn = y

        #calculate inflation
        p = equili_pi_ar(y_m,yn,pi_m,0,0)
        #record inflation in this period
        list_inflation[i] = p

        #define lagged inflation
        pi_m = p
        #define lagged output
        y_m = y
```

```python
#Here we plot both output and inflation over 100 periods - They converge to 0 (conditional mean), after about 60 periods
plt.plot(list_output)
plt.plot(list_inflation)
```

**Question 4**

Now we wish to simulate the ASAD in a 1000 periods, with gaussian shock to both inflation and output

```python
#We define new given paramters
pms['sigma_x'] = 3.492
pms['sigma_c'] = 0.2
pms["delta"] = 0.80
pms["omega"] = 0.15

```

```python
#We make a function tha simulates our ASAD model for 1000 periods. The amound of periods is fixed, but could be generalized
def simulation(v_m=0, s_m=0) :
    """This function takes 0 requried arguments, and two optional. It return two list: one of simulated output and one of
        inflation."""

    #Set the seed
    np.random.seed(1)

    #define paramters. This could be generalized
    pi_m = 0
    yn = 0
    y_m = 0

    #Drawing 1000 random numbers for both inflation and output shocks
    xt = np.random.normal(loc=0,scale=pms['sigma_x'],size=1000)
    ct = np.random.normal(loc=0,scale=pms['sigma_c'],size=1000)

    #Defining list space
    list_output = list(range(1000))
    list_inflation = list(range(1000))

    #Initating a for-loop
    for i in range(1000):
        #for each period in a 1000 periods

        #If it is the first period
        if i == 0 :
            #Calculate the output
            y = equili_output_ar(v_m,pi_m,s_m,y_m,ct[i],xt[i])
            #record it
            list_output[i] = y
            #Define output in period
            yn = y

            #Calculate inflaiton
            p = equili_pi_ar(y_m,yn,pi_m,s_m,ct[i])
            #record it
            list_inflation[i] = p

            #Define lagged inflation
            pi_m = p
            #Define lagged output
            y_m = y

            #calculate autoregressive parameters, these are given by the shock + the parameter last period times a paramter
            v_m = pms["delta"] * v_m + xt[i]
            s_m = pms["omega"] * s_m + ct [i]

        #If it is any other period
        else :
            #Calculate the output
            y = equili_output_ar(v_m,pi_m,s_m,y_m,ct[i],xt[i])
            #record it
            list_output[i] = y
            #Define output in period
            yn = y

            #Calculate inflaiton
            p = equili_pi_ar(y_m,yn,pi_m,s_m,ct[i])
            #record it
            list_inflation[i] = p

            #Define lagged inflation
            pi_m = p
            #Define lagged output
            y_m = y
             #calculate autoregressive parameters, these are given by the shock + the parameter last period times a paramter
            v_m = pms["delta"] * v_m + xt[i]
            s_m = pms["omega"] * s_m + ct[i]

    return list_inflation, list_output
```

```python
#We store the inflation and output is an array called data, and in two seperat lists
data = simulation()
inflation = data[0]
output = data[1]
```

### Statistics for the simulated model

```python
#We use numpy to calculate the statistics for the simulation
#Variance
print(np.var(inflation))
print(np.var(output))
```

    1.018330396633222
    1.8330329923516997

```python
#Mean - this is only for plotting
mean_pi = np.linspace(np.mean(inflation),np.mean(inflation),1000)
mean_y = np.linspace(np.mean(output),np.mean(output),1000)

```

```python
#Plot the output across time
plt.plot(output,label="Output")
#Plot the mean
plt.plot(mean_y,color="r",label="Mean of output",linestyle="dashed")
plt.xlabel("Period")
plt.ylabel("Change in output")
plt.legend()
plt.show()
```

```python
#Plot the inflation across time
plt.plot(inflation,label="Inflation")
#Plot the mean of inflation
plt.plot(mean_pi,color="r",label="Mean of inflation",linestyle="dashed")
plt.xlabel("Period")
plt.ylabel("Change in inflaiton")
plt.legend()
plt.show()
```

```python
#Correlatation between inflation and output
#pearsons R is calculated using numpy
np.corrcoef(inflation,output)[0,1]
```

```python
#Plot inflation and output across time
##Here we plot the two variables as time series. Output is faded so inflation is visible
plt.plot(inflation,label="Inflation",alpha=1)
plt.plot(output,color="r",label="Output",alpha=0.2)
plt.xlabel("Period")
plt.ylabel("Change")
plt.legend()
plt.show()
```

In order to calculate the autocorrelation, we define the two lists as a dataframe and use pandas to make lagged variables

```python
#create a dataframe in order to make lagged variables
##defin dictionary of coloumn names
d = {"inflation" : inflation, "output":output}
#make data frame
df = pd.DataFrame(data=d)
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>inflation</th>
      <th>output</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.074830</td>
      <td>1.406367</td>
    </tr>
    <tr>
      <th>1</th>
      <td>-0.350344</td>
      <td>0.878997</td>
    </tr>
    <tr>
      <th>2</th>
      <td>-0.304003</td>
      <td>0.245455</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.412872</td>
      <td>-0.643367</td>
    </tr>
    <tr>
      <th>4</th>
      <td>-0.684640</td>
      <td>0.527190</td>
    </tr>
  </tbody>
</table>
</div>

```python
#Make lagged inflation
df["l_inflation"] = df["inflation"].shift(1)
#define an object for later
l_inflation = df["l_inflation"]

#Make lagged output
df["l_output"] = df["output"].shift(1)
#define an object for later
l_output = df["l_output"]
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>inflation</th>
      <th>output</th>
      <th>l_inflation</th>
      <th>l_output</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.074830</td>
      <td>1.406367</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>-0.350344</td>
      <td>0.878997</td>
      <td>0.074830</td>
      <td>1.406367</td>
    </tr>
    <tr>
      <th>2</th>
      <td>-0.304003</td>
      <td>0.245455</td>
      <td>-0.350344</td>
      <td>0.878997</td>
    </tr>
    <tr>
      <th>3</th>
      <td>-0.412872</td>
      <td>-0.643367</td>
      <td>-0.304003</td>
      <td>0.245455</td>
    </tr>
    <tr>
      <th>4</th>
      <td>-0.684640</td>
      <td>0.527190</td>
      <td>-0.412872</td>
      <td>-0.643367</td>
    </tr>
  </tbody>
</table>
</div>

Now we can test for autocorrelation the inflation and the output :

```python
#Autocorrelation of the inflation
np.corrcoef(inflation[1:999],l_inflation[1:999])[0,1]
```

The autocorrelation is very high (closed to 1) which means that the inflation heavily depend on its own level in the past. This result seems to make sense, indeed in the inflation equation there is a variable for the past inflation. This impact is positive and can be known in some circumstances as "self sustained inflation".

```python
#Autocorrelation of the output
np.corrcoef(output[1:999],l_output[1:999])[0,1]
```

The output autocorrelation is high as well.

**Question 5**

We want to plot the evolution of the correlation between inflation and output regarding the level of phi. To do that we will fist change our inflation function to include phi as a paramater that we are able to change.

```python
#create a new inflation function, phi as a variable
equil_piii= sm.lambdify((yt_m1, yt, pit_m1, st_m1, ct, omega, gamma, phi),eq_pi_ar)
def equili_pi_phi (yt_m1, yt, pit_m1, st_m1, ct, phi, omega=pms['omega'], gamma=pms['gamma']):
    return equil_piii (yt_m1, yt, pit_m1, st_m1, ct, omega, gamma, phi)
```

Now we will create a function where we can set phi that will determine the correlation between the two variables. To find this correlation we will use a loop and the list of output and the list of inflation :

```python
#define correlation function
def corr_func(phi,v_m=0, s_m=0):
    """This function takes 1 requires argument, that is phi. It return the correlation between inflation
        and output for that phi level"""

    #Set seed
    np.random.seed(1)
    #define parameters
    pi_m = 0
    yn = 0
    y_m = 0

    #Simulate random shocks
    xt = np.random.normal(loc=0,scale=pms['sigma_x'],size=1000)
    ct = np.random.normal(loc=0,scale=pms['sigma_c'],size=1000)

    #Make list for output and inflation
    list_output = list(range(1000))
    list_inflation = list(range(1000))

    #Intiate for loop for 1000 periods
    for i in range(1000):
        #If it is the first period
        if i == 0 :
            #Calculate output
            y = equili_output_ar(v_m,pi_m,s_m,y_m,ct[i],xt[i])
            #record it
            list_output[i] = y
            #define output in this period
            yn = y

            #Calculate inflation
            p = equili_pi_phi(y_m,yn,pi_m,s_m,ct[i],phi)
            #Record it
            list_inflation[i] = p

            #Define lagged inflation
            pi_m = p
            #Define lagged output
            y_m = y
            #define autoregressive parameters
            v_m = pms["delta"] * v_m + xt[i]
            s_m = pms["omega"] * s_m + ct [i]
        #if it is not the first period
        else :
            #Calculate output
            y = equili_output_ar(v_m,pi_m,s_m,y_m,ct[i],xt[i])
            #record it
            list_output[i] = y
            #define output in this period
            yn = y

            #Calculate inflation
            p = equili_pi_phi(y_m,yn,pi_m,s_m,ct[i],phi)
            #Record it
            list_inflation[i] = p

            #Define lagged inflation
            pi_m = p
            #Define lagged output
            y_m = y
            #define autoregressive parameters
            v_m = pms["delta"] * v_m + xt[i]
            s_m = pms["omega"] * s_m + ct[i]

    #use numpy to calculate correlation
    correlation = np.corrcoef(list_inflation,list_output)[0,1]

    return correlation
```

```python
#Define relevant levels of phi
phi_vec = np.linspace(0,1,1000)

#Make a list with space for corrlations
corr_list = list(range(1000))
#Initiate a for-loop
for i in range(1000) :
    #for each phi simulate the model with the same shock, an calculate the shocks
    ## this is a nested for-loop and can take som time
    corr = corr_func(phi = phi_vec[i])

    #recod the correlation
    corr_list[i] = corr

#print the correlation to illutstrate that it worked
corr_list[0:9]

```

```python
#Plot the correlation according Phi
plt.plot(phi_vec,corr_list)
plt.xlabel("Phi")
plt.ylabel("Correlation between output and inflation")
plt.show()
```

As we can see with this graph, as phi is increasing, the correlation between output and inflation is also increasing. Below is a slider where the reade can try to se the correlation for themselves

```python
#Define a slider
phi_slider = widgets.FloatSlider(min=0,max=1,step = 0.01)
#call interact on the correlation function
widgets.interact(corr_func,phi=phi_slider,v_m = widgets.fixed(0),s_m = widgets.fixed(0))
```

    interactive(children=(FloatSlider(value=0.0, description='phi', max=1.0, step=0.01), Output()), _dom_classes=(…

    <function __main__.corr_func(phi, v_m=0, s_m=0)>

**Question 5**

We wish to optimize the correlation towards 0.31. Therefore we use a numerical optimizer and resticts it to only go towards 0.31. We have written out own brute force optimizer, that we use. This means that our optimiation can take up to 10-15 seconds on a resonable pc.

```python
#Define the autoregresive parameters for the first period
v_m = 0
s_m = 0
#Define the objectitve, which is the correlation
def objective(phi):
    return float(corr_func(phi, v_m, s_m))
#Define the constraints, which is 0.31 - the correlation
def constraint(phi):
    return float((0.31 - corr_func(phi, v_m, s_m)))
#We have used float because we had some unknown issue that it resolved.
```

```python
#define the phi vector space
phi_vec = np.linspace(0.0,1,1000)

#Intiate a for loop - this
for phi in phi_vec:
    # x in the objective function given the phi level
    x = objective(phi)
    # y is the constraint function given the phi level
    y = constraint(phi)

    #Check if the the constrain is sufficiantly equal to 0 (3 decimals)
    if round(y,3) == 0:
        #If it is break the loop and print the phi and correlation values
        print("Optimization sucessful")
        print("Phi level =",phi,"and correlation optimized to",x)
        break

```

    Optimization sucessful
    Phi level = 0.9879879879879879 and correlation optimized to 0.3098953977987362

## 3. Exchange economy

Consider an **exchange economy** with

1. 3 goods, $(x_1,x_2,x_3)$
2. $N$ consumers indexed by \\( j \in \{1,2,\dots,N\} \\)
3. Preferences are Cobb-Douglas with log-normally distributed coefficients

    $$ \begin{eqnarray*}
    u^{j}(x_{1},x_{2},x_{3}) &=&
    \left(x_{1}^{\beta_{1}^{j}}x_{2}^{\beta_{2}^{j}}x_{3}^{\beta_{3}^{j}}\right)^{\gamma}\\
     &  & \,\,\,\beta_{i}^{j}=\frac{\alpha_{i}^{j}}{\alpha_{1}^{j}+\alpha_{2}^{j}+\alpha_{3}^{j}} \\
     &  & \,\,\,\boldsymbol{\alpha}^{j}=(\alpha_{1}^{j},\alpha_{2}^{j},\alpha_{3}^{j}) \\
     &  & \,\,\,\log(\boldsymbol{\alpha}^j) \sim \mathcal{N}(\mu,\Sigma) \\
    \end{eqnarray*} $$

4. Endowments are exponentially distributed,

$$
\begin{eqnarray*}
\boldsymbol{e}^{j} &=& (e_{1}^{j},e_{2}^{j},e_{3}^{j}) \\
 &  & e_i^j \sim f, f(z;\zeta) =  1/\zeta \exp(-z/\zeta)
\end{eqnarray*}
$$

Let $p_3 = 1$ be the **numeraire**. The implied **demand functions** are:

$$
\begin{eqnarray*}
x_{i}^{\star j}(p_{1},p_{2},\boldsymbol{e}^{j})&=&\beta^{j}_i\frac{I^j}{p_{i}} \\
\end{eqnarray*}
$$

where consumer $j$'s income is

$$I^j = p_1 e_1^j + p_2 e_2^j +p_3 e_3^j$$

We define parameters:

```python
#parameters:
N = 100
mu = np.array([3,2,1])
Sigma = np.array([[0.25, 0, 0], [0, 0.25, 0], [0, 0, 0.25]])
gamma = 0.8
zeta = 1

#seed
seed = 1986
np.random.seed(seed)

#preferances
alphas = np.exp(np.random.multivariate_normal(mu, Sigma, size=N))
betas = alphas/np.reshape(np.sum(alphas,axis=1),(N,1))

beta1= betas[:,0]
beta2= betas[:,1]
beta3=betas[:,2]

alpha1= alphas[:,0]
alpha2= alphas[:,1]
alpha2= alphas[:,2]

#endowments
e1 = np.random.exponential(zeta,size=N)
e2 = np.random.exponential(zeta,size=N)
e3 = np.random.exponential(zeta,size=N)
```

#### Question 1: Plot the histograms of the budget shares for each good across agents.

We make a histogram and plot the betas:

```python
plt.hist(betas, bins = 60, label= ("Good1", 'Good2', 'Good3'))
plt.legend()
plt.xlabel('Budget share')
plt.ylabel('Number of consumers')
plt.show
```

Consider the **excess demand functions:**

$$ z_i(p_1,p_2) = \sum_{j=1}^N x_{i}^{\star j}(p_{1},p_{2},\boldsymbol{e}^{j}) - e_i^j$$

**Question 2:** Plot the excess demand functions.

We define the demand and excess demand functions for each of the three goods. We have also converted the excess demand functions into one value by summing them up, to use later to find the equilibrium.

```python
# demand functions:
# We can eliminate p3 from the demand function, as it is equal to 1 (with p3*e3=e3)

def demand_good1 (beta1, p1, e1, p2, e2, e3):
    I= p1*e1 + p2*e2 + e3
    return beta1*(I/p1)

def demand_good2 (beta2, p1, e1, p2, e2, e3):
    I= p1*e1 + p2*e2 + e3
    return beta2*(I/p2)

def demand_good3 (beta3, p1, e1, p2, e2, e3):
    I= p1*e1 + p2*e2 + e3
    return beta2*(I)
```

```python
# Excess demand functions:

def excess_demand_good1(beta1, p1, e1, p2, e2, e3):

    #demand
    demand = demand_good1(beta1, p1, e1, p2, e2, e3)

    #supply
    supply= e1

    #excess demand
    excess_demand= demand-supply

    return excess_demand
```

```python
def excess_demand_good2(beta2, p1, e1, p2, e2, e3):

    #demand
    demand= demand_good2(beta2, p1, e1, p2, e2, e3)

    #supply
    supply= e2

    #excess demand
    excess_demand= demand- supply

    return excess_demand
```

```python
def excess_demand_good1_value(beta1, p1, e1, p2, e2, e3):

    excess_demand_good1_value = np.sum(excess_demand_good1(beta1, p1, e1, p2, e2, e3))

    return excess_demand_good1_value
```

```python
def excess_demand_good2_value(beta2, p1, e1, p2, e2, e3):

    #demand
    excess_demand_value2= np.sum(excess_demand_good2(beta2, p1, e1, p2, e2, e3))

    return excess_demand_value2
```

We make two 3D plots for the excess demands of goods 1 and 2.

```python
# We make a 3D plot of excess demand for good 1:
# a. prices
p1_vec = np.linspace(0.1,10,100)
p2_vec = np.linspace(0.1,10,100)

# b. plot
p1_grid,p2_grid = np.meshgrid(p1_vec,p2_vec,indexing='ij')
excess_demand_good1_grid = excess_demand_good1(beta1,p1_grid,e1,p2_grid,e2,e3)

fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(p1_grid,p2_grid,excess_demand_good1_grid,cmap=cm.jet)

# c. add labels
ax.set_xlabel('$p_1$')
ax.set_ylabel('$p_2$')
ax.set_zlabel('Excess demand good2')

# d. invert xaxis
ax.invert_xaxis()

# e. remove background
ax.xaxis.pane.fill = False
ax.yaxis.pane.fill = False
ax.zaxis.pane.fill = False

# f. add colorbar
fig.colorbar(cs);
plt.show()
```

```python
# We make a 3d plot of excess demand for good 2:

# a. plot
p1_grid,p2_grid = np.meshgrid(p1_vec,p2_vec,indexing='ij')
excess_demand_good2_grid = excess_demand_good2(beta2,p1_grid,e1,p2_grid,e2,e3)

fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(p1_grid,p2_grid,excess_demand_good1_grid,cmap=cm.jet)

# b. add labels
ax.set_xlabel('$p_1$')
ax.set_ylabel('$p_2$')
ax.set_zlabel('Excess demand Good2')

# c. invert xaxis
ax.invert_xaxis()

# d. remove background
ax.xaxis.pane.fill = False
ax.yaxis.pane.fill = False
ax.zaxis.pane.fill = False

# e. add colorbar
fig.colorbar(cs);
plt.show()
```

**Quesiton 3:** Find the Walras-equilibrium prices, $(p_1,p_2)$, where both excess demands are (approximately) zero, e.g. by using the following tâtonnement process:

1. Guess on $p_1 > 0$, $p_2 > 0$ and choose tolerance $\epsilon > 0$ and adjustment aggressivity parameter, $\kappa > 0$.
2. Calculate $z_1(p_1,p_2)$ and $z_2(p_1,p_2)$.
3. If $|z_1| < \epsilon$ and $|z_2| < \epsilon$ then stop.
4. Else set $p_1 = p_1 + \kappa \frac{z_1}{N}$ and $p_2 = p_2 + \kappa \frac{z_2}{N}$ and return to step 2.

Just like in the lecture, we use a while loop to determine the equilibrium prices, for which excess demands are both zero. We set out guesses for the prices to one.

```python
def find_equilibrium(betas, p1, e1, p2, e2, e3, kappa=0.5,eps=1e-8,maxiter=10000):

    t = 0
    while True:

        # a. step 1: excess demand
        Z1 = excess_demand_good1_value(beta1, p1, e1, p2, e2, e3)
        Z2 = excess_demand_good2_value(beta2, p1, e1, p2, e2, e3)

        # b: step 2: stop?
        if np.abs(Z1) < eps and np.abs(Z2) < eps or t >= maxiter:
            print(f'{t}: p1= {p1} -> excess demand Z1= {Z1}')
            print(f'{t}: p2= {p2} -> excess demand Z2= {Z2}')
            break

        # c. step 3: update p1
        p1 = p1 + kappa*Z1/N
        p2 = p2 + kappa*Z2/N

        t += 1

    return p1, p2
```

```python
p1,p2= find_equilibrium(betas, 1, e1, 1, e2, e3)
```

    1844: p1= 6.149742400548352 -> excess demand Z1= 9.996033889514422e-09
    1844: p2= 2.685539126759021 -> excess demand Z2= 4.059743818896067e-09

After 1844 iterations, we find the equilibrium prices of around 6.15 for p1 and 2.67 for p2. P3 is normalized to 1.

We check to make sure that demand is really smaller than epsilon.

```python
eps=1e-8
Z1 = excess_demand_good1_value(beta1, p1, e1, p2, e2, e3)
Z2 = excess_demand_good2_value(beta2, p1, e1, p2, e2, e3)
print(Z1,Z2)
assert(np.abs(Z1) < eps)
assert(np.abs(Z2) < eps)
```

    9.996033889514422e-09 4.059743818896067e-09

**Question 4:** Plot the distribution of utility in the Walras-equilibrium and calculate its mean and variance.

We define the utility function, insert the above calculated equilibrium prices in the function and plot it as a histogram.

```python
# Defining the utility function
def utility(betas, p1, e1, p2, e2, e3, gamma):
    """This function takes 7 arguments, and teruns the for each person
        utility given the parameters"""

    #calulater the demand for each good given the parameters
    x1= demand_good1(betas[:,0], p1, e1, p2, e2, e3)
    x2= demand_good2(betas[:,1], p1, e1, p2, e2, e3)
    x3= demand_good3(betas[:,2], p1, e1, p2, e2, e3)

    #Calculate the utility
    utility= (x1**beta1+x2**beta2+x3**beta3)**gamma

    return utility
```

```python
# Inserting equilibrium prices
utility_dist = utility(betas, 6.149742400548352, e1, 2.685539126759021, e2, e3, gamma)
```

```python
# Making a histogram
plt.hist(utility_dist, bins= 20)
plt.xlabel('Utility')
plt.ylabel('Number of consumers')
plt.show
```

Lastly, we calculate mean and variance.

```python
np.mean(utility_dist)
```

```python
np.var(utility_dist)
```