## EIGENVECTORS AND EIGENVALUES FROM A MATRIX

```python
import numpy as np
from numpy import linalg as LA
import matplotlib.pyplot as plt
from plotnine import *
import scipy.linalg as la
from sympy.interactive.printing import init_printing
from sympy.matrices import Matrix, eye, zeros, ones, diag, GramSchmidt
%matplotlib inline
plt.style.use('ggplot')
```

```python
print(plt.style.available)
```

    ['seaborn-dark', 'seaborn-darkgrid', 'seaborn-ticks', 'fivethirtyeight', 'seaborn-whitegrid', 'classic', '_classic_test', 'fast', 'seaborn-talk', 'seaborn-dark-palette', 'seaborn-bright', 'seaborn-pastel', 'grayscale', 'seaborn-notebook', 'ggplot', 'seaborn-colorblind', 'seaborn-muted', 'seaborn', 'Solarize_Light2', 'seaborn-paper', 'bmh', 'tableau-colorblind10', 'seaborn-white', 'dark_background', 'seaborn-poster', 'seaborn-deep']

```python
# import jtplot module in notebook
#from jupyterthemes import jtplot

# choose which theme to inherit plotting style from
# onedork | grade3 | oceans16 | chesterish | monokai | solarizedl | solarizedd
#jtplot.style(theme='oceans16')

# set "context" (paper, notebook, talk, poster)
# scale font-size of ticklabels, legend, etc.
# remove spines from x and y axes and make grid dashed
#jtplot.style(context='notebook', fscale=1.2, spines=False, ticks=True, gridlines='-')

# turn on X- and Y-axis tick marks (default=False)
# turn off the axis grid lines (default=True)
# and set the default figure size
# jtplot.style(ticks=True, grid=True)

# reset default matplotlib rcParams

```

```python
init_printing(use_unicode=True, wrap_line=True)
```

Let **A** be a square matrix.
S non-zero vector **v** is an eigenvector for A with eigenvalue $\lambda$ if:
$$Av = \lambda v$$
Rearranging the equations, we see that **v** is a solution of the homogenous system of equations:
$$(A - \lambda I)v = 0$$
where I is the identity matrix of size n.

Non-trivial solutions exist only if the matrix $A- \lambda I$ is singular => $det(A - \lambda I) = 0$.

Eigenvalues of A are roots of the CHARACTERISTIC POLYNOMIAL:
$$p(\lambda) = det(A - \lambda I)$$

***
Scipy package helps in computing the eigenvalues and eigenvectors of a square matrix A.

**scipy.linalg.eig** - computes eigenvalues and eigenvectors of a square matrix A
***

**!!! In numpy linalg and scipy the eigenvectors are normalized so their Euclidean norms are 1**

### How it works step by step

```python
# crate an array (vectors are in the columns, so x coord go first, y in the second)
M = np.array([[3,4], [0,5]])
M
```

    array([[3, 4],
           [0, 5]])

```python
# USING SCIPY the formula results a tuple in the form (eigvals, eigvecs)
results = la.eig(M)
results
```

    (array([3.+0.j, 5.+0.j]), array([[1.        , 0.89442719],
            [0.        , 0.4472136 ]]))

```python
# print the eigenvalues of A (these are real numbers so we can print them accordingly)
results[0].real
```

    array([3., 5.])

```python
# print the corresponding eigenvectors
results[1]
```

    array([[1.        , 0.89442719],
           [0.        , 0.4472136 ]])

```python
# unpacking the tuple
eigvals, eigvecs = LA.eig(M)

print("The eigenvalues are: ", eigvals.real)
print("The eigenvectors are: \n", eigvecs)
```

    The eigenvalues are:  [3. 5.]
    The eigenvectors are:
     [[1.         0.89442719]
     [0.         0.4472136 ]]

**Check the eigenvector/eigenvalue condition $Au = \lambda u$ where $u$ is the eigenvector and $\lambda$ is its eigenvalue**
- from the previous results, we have lambda1 = 3 and lambda2 = 5
- we can index them to extract each individually
- **we multiply the eigenvector by M (original matrix) and check that it is the same as multiplying the same eigenvector by its eigenvalue**

$u$ - one of the eigenvectors resulting from the calculations
$\lambda$ -corresponding eigenvalue for the previous vector
$M$ - matrix given as input for the calculations

```python
# take the second eigenvector
u = eigvecs[:, 1]
u
```

    array([0.89442719, 0.4472136 ])

```python
# take the second lambda
lambdaM = eigvals[1].real
lambdaM
```

```python
# multiply the input matrix with the eigenvector selected
np.dot(M, u)
```

    array([4.47213595, 2.23606798])

We see that the result of multiplying the input matrix with the eigenvector and the eigenvalue with the eigenvector are the same. So $Au = \lambda u$.

```python
# multiply same eigenvector by its corresponding eigenvalue
lambdaM*u
```

    array([4.47213595, 2.23606798])

### Function to calculate the eigenvalues and vectors and display them nicely

```python
# helper function
def isSquareMatrix(matrix):
    """
    checks is a matrix is square (length of all rows must equal the number of rows)
    """
    return matrix.shape[0] == matrix.shape[1]
```

```python
# helper function
def characteristic_polynomial(matrix):
    """
    display the characteristic polynomial of a matrix
    """

    m = Matrix(matrix)

    return m.charpoly()
```

```python
def solve_sympy(matrix):
    """
    matrix: a numpy array in a square shape
    """

    # for sympy calculations, transform the array into a Matrix object
    m = Matrix(matrix)
    print(m)

    # check that the matrix is square
    if isSquareMatrix(matrix):

        # calculate the eigenvalues and eigenvectors using sympy
        sympy_eigvals = m.eigenvals()
        sympy_eigvect = m.eigenvects()

        # calculate the characteristic polynomical
        char_poly = m.charpoly()

    # error for matrix input
    else:
        print("Your input matrix is not square or is empty.")

    print("Eigenvalues:")
    for i in sympy_eigvals.items():
        print(i[0])

    for i in sympy_eigvect:
        print("Eigenvalue: ", i[0], "\nCorresponding eigenvector: ", i[2])
    print("Characteristic polynomial: ", char_poly)

    # loop through the results and return the eigenvectors to plot
    l = []
    for i in range(len(sympy_eigvect)):
        l.append(sympy_eigvect[i][2][0])

    to_plot = np.array(l).astype(np.float64).T

    #if len(sympy_eigvect) ==2:
    #    p = sympy_eigvect[0][2][0], sympy_eigvect[1][2][0]
    #    to_plot = np.array(p).astype(np.float64).T
    #else:
    #    p = sympy_eigvect[0][2][0]
    #    to_plot = np.array(p).astype(np.float64).T

    return sympy_eigvect, char_poly, to_plot
```

```python
def solve_scipy(matrix):
    """
    matrix: a numpy array in a square shape
    """

    # check that the matrix is square
    if isSquareMatrix(matrix):

        # calculate the eigenvalues and eigenvectors - using scipy
        eigvals, eigvecs = la.eig(matrix)

    # error for matrix input
    else:
        print("Your input matrix is not square or is empty.")

    print("The eigenvalues are: ", eigvals.real, "\nThe eigenvectors are: \n", eigvecs)
    return eigvals, eigvecs
```

```python
def plot_vectors(original, eigen):
    """
    Create an overaly of the vectors in 2D space.
    original -> given matrix
    eigen -> calculated eigen vectors
    """

    # set the values in this order: 1st row for x, 2nd row for y
    #for origin
    X = np.array((0))
    Y = np.array((0))

    # for axes
    U = original[0,:]
    V = original[1,:]
    U1 = eigen[0,:]
    V1 = eigen[1,:]

    # check the limits for x and y axes and add 1
    # this can be condensed into a one liner if you want to make it square - here only for display purposes
    xlimit = max(np.amax(abs(U))+1, np.amax(abs(U1))+1)
    ylimit = max(np.amax(abs(V))+1, np.amax(abs(V1))+1)

    # choose the bigger number between x and y limits to have a square graph
    lim = xlimit if xlimit > ylimit else ylimit

    # set figure size
    fig, ax = plt.subplots(figsize=(20,10))#(1, 2, sharex=True, sharey=True, figsize=(20, 10))

    # IMPORTANT FOR ASPECT RATIO
    ax.set_aspect('equal')

    #ax[0]
    # plot the vectors
    plt.quiver(X, Y, U, V,
                   color =["r", "black"],
                   units='xy', scale=1, width = 0.03, alpha=0.7)
    #ax[1]
    plt.quiver(X, Y, U1, V1,
                   color =["green", "orange"],
                   units='xy', scale=1, headlength=5.5, width=0.05, alpha=0.7)

    # set the x,y limits automatically and limit them at 1 interval ticks
    plt.xlim(-lim, lim)
    plt.ylim(-lim, lim)
    plt.xticks(np.arange(-lim, lim, step=1))
    plt.yticks(np.arange(-lim, lim, step=1))
    #ax[0]

    # annotate the vectors
    plt.annotate("v1", xy = (original[0,:][0], original[1,:][0]),
                 color = "red", weight = "semibold", size = 12)
  #ax[0]
    plt.annotate("v2", xy = (original[0,:][1], original[1,:][1]),
                 color = "black", weight = "semibold", size =12)
  #ax[1]
    if len(eigen[0])>1:
        plt.annotate("eigen_v1", xy = ((eigen[0,:][0]), (eigen[1,:][0])-0.5),
                     color = "green", weight = "bold", size = 13)
      #ax[1]
        plt.annotate("eigen_v2", xy = (eigen[0,:][1], eigen[1,:][1]),
                     color = "orange", weight = "bold", size =13)
    else:
        plt.annotate("eigen_v1", xy = ((eigen[0,:][0]), (eigen[1,:][0])-0.5),
                     color = "green", weight = "bold", size = 13)
    plt.show()
```

## COURSERA QUIZ

1. **For the matrix A = [[1,0],[0,2]], what is the caracteristic polynomical and the solutions to it?**

```python
A = Matrix(2,2, (1, 0, 0, 2))
A1 = np.array([[1,0], [0, 2]])

eveca, epa, to_plota = solve_sympy(A1)

plot_vectors(A1, to_plota)
```

### 3. For the matrix A = [[3,4], [0,5]], what is the characteristic polynomial and the solutions to it?

```python
B1 = np.array([[3,4], [0,5]])

evecb1, epb1, to_plotb1 = solve_sympy(B1)

plot_vectors(B1, to_plotb1)
```

### For matrix A = [[1,0], [-1,4]] what is the charatceristic polynomial and the solutions?

```python
C1 = np.array([[1,0], [-1,4]])

evec_c1, ep_c1, to_plot_c1 = solve_sympy(C1)

plot_vectors(C1, to_plot_c1)
```

```python
to_plot_c1
```

    array([[3., 0.],
           [1., 1.]])

### For matrix A = [[-3, 8], [2, 3]] what is the characteristic polynomical and the solutions?

```python
D1 = np.array([[-3, 8], [2, 3]])

evec_d1, ep_d1, to_plot_d1 = solve_sympy(D1)

plot_vectors(D1, to_plot_d1)
```

### For the matrix A = [[1,0], [-1,4]] what is the characteristic polynomial and the solutions?

```python
E1 = np.array([[ 1, 0], [-1, 4]])

evec_E1, ep_E1, to_plot_e1 = solve_sympy(E1)

plot_vectors(E1, to_plot_e1)
```

```python
F = np.array([[5, 4], [ -4, -3]])
evec_F, ep_F, to_plot_f = solve_sympy(F)

plot_vectors(F, to_plot_f)
```

```python
G = np.array([[-2, -3], [1,1]])
evec_G, ep_G, to_plot_g = solve_sympy(G)
```


## Chapter 4 -  Training Models

### Linear Regression

Linear regression is usually the first machine learning model in modern statistical learning. Although dull, it is widely used and serves as a good jumping-off point for newer approaches - many fancy statistical learning approaches can be seen as generalisations or extensions of linear regression. Hence, having a good understanding of linear regression before studying more complex learning methods cannot be overstated.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

import statsmodels.api as sm
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, SGDRegressor
```

```python
# Ingest, preprocessing
df = pd.read_csv('Advertising.csv', index_col=0)

X1 = df.iloc[:,:1]
df1 = df.iloc[:,[0,1,2,3]]

X2 = df.iloc[:,:3]
df2 = df.iloc[:, [0,3]]

y = df.iloc[:, 3]
```

```python
# # For testing
# print('df')
# display(df.head())
# print('X1')
# display(X1.head())
# print('df1')
# display(df1.head())
# print('X2')
# display(X2.head())
# print('df2')
# display(df2.head())
# print('y')
# display(y.head())
```

```python
# Plot each of the variables against each other
sns.pairplot(df)
```

Univariate / Simple Linear Regression

Given $n$ samples, the model assumes there is approximately a linear relationship between $X$ and $Y$. Mathematically, it is in the form:
$$
y \approx \beta_0 + \beta_1x
$$
$\beta_0$ and $\beta_1$ are model coefficients or parameters that need to be estimated. The estimated values are $\hat{\beta_0}$ and $\hat{\beta_1}$ and the predicted value of $y$, $\hat{y}$ is

$$\hat{y} = \hat{\beta_0} + \hat{\beta_1}x$$

To obtain the parameter estimates, we find these values such that the mean squared error or MSE is minimised. Let the observations be represented in an $n$ by $p$ matrix, $\mathbf X$. The MSE is then calculated as

$$\begin{align}\text{MSE} (\mathbf X) &= \frac 1 n \sum_{i=1}^n \begin{pmatrix} \hat{y^{(i)}} - y^{(i)}\end{pmatrix}^2\\&= \frac 1 n \sum_{i=1}^n \begin{pmatrix} \hat{\beta_0} + \hat{\beta_1}x^{(i)}- y^{(i)}\end{pmatrix}^2\end{align}$$

Note that minimising the MSE is the same as minimising the residual sum of squares, RSS. This is because $\frac 1n \text{RSS} = \text{MSE}$

The closed-form solution of univariate linear regression is:

| $$\hat{\beta_1}$$| $$
\hat{\beta_1} =
\frac{\sum^n_{i=1}\begin{bmatrix}
\begin{pmatrix}
x_i-\bar{x}
\end{pmatrix}
\begin{pmatrix}
y_i-\bar{y}
\end{pmatrix}
\end{bmatrix}}
{\sum^n_{i=1}\begin{pmatrix}x_i - \bar{x}\end{pmatrix}^2}
$$  |
|:-|:-|
|  $$\hat{\beta_0}$$    | $$
\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}
$$|

where $\bar{x} = \frac {\sum_i x}{n}$ and $\bar{y} = \frac {\sum_i y}{N}$, the respective sample means. They are also considered the least-squares coefficient estimates for univariate linear regression.

```python
Xarray1 = np.c_[np.ones((X1.shape[0],1)), X1] # Add x0=1
```

```python
# Obtaining the coefficients using the closed-form solution
df_train1 = df.copy()
df_train1['TV_mean'] = df_train1['TV'].mean()
df_train1['sales_mean'] = df_train1['sales'].mean()
df_train1['model_beta1_xmxbar'] = df_train1['TV'] - df_train1['TV_mean']
df_train1['model_beta1_ymybar'] = df_train1['sales'] - df_train1['sales_mean']
df_train1['model_beta1_numer'] = df_train1['model_beta1_xmxbar'] * df_train1['model_beta1_ymybar']
df_train1['model_beta1_denom'] = df_train1['model_beta1_xmxbar']**2
df_train1[['model_beta1_numer', 'model_beta1_denom']].head()
beta1 = df_train1['model_beta1_numer'].sum()/df_train1['model_beta1_denom'].sum()
beta0 = df_train1['sales'].mean()-beta1*df_train1['TV'].mean()
Theta_hat11 = [beta0, beta1]
print(Theta_hat11)
```

    [7.0325935491276965, 0.047536640433019736]

```python
# Predict
np.dot(Xarray1[:2], Theta_hat11)
```

    array([17.97077451,  9.14797405])

```python
# For univariate linear regression, obtain the coefficients using the normal equations
Theta_hat12 = np.dot(np.dot(np.linalg.inv(np.dot(Xarray1.T, Xarray1)), Xarray1.T),y)
Theta_hat12 = list(Theta_hat12)
print(Theta_hat12)
```

    [7.032593549127698, 0.047536640433019736]

```python
# Predict
np.dot(Xarray1[:2], Theta_hat12)
```

    array([17.97077451,  9.14797405])

```python
# Obtaining the coefficients using statsmodels
reg11 = sm.OLS(y, Xarray1)
results11 = reg11.fit()
Theta_hat13 = list(results11.params)
print(Theta_hat13)
```

    [7.032593549127698, 0.047536640433019764]

```python
# Predict
print(results11.predict(Xarray1[:2]))
```

    [17.97077451  9.14797405]

```python
# Obtaining the coefficients using sklearn
reg = LinearRegression()
reg.fit(Xarray1, y)
Theta_hat14 = reg.coef_.copy()
Theta_hat14[0] = reg.intercept_
print(Theta_hat14)
```

    [7.03259355 0.04753664]

```python
# Predict
print(reg.predict(Xarray1[:2]))
```

    [17.97077451  9.14797405]

Multiple Linear Regression

The multiple linear regression model extends the univariate model. It is in the form:

$$
y \approx \beta_0 + \beta_1x_{1} + \beta_2x_{2}+ \cdots + \beta_px_{p}
$$

where $p$ is the number of features, $x_{j}$ is the value of the $j$th feature for the sample and $\beta_j$ is the corresponding model coefficient for that feature. Letting $\Theta = (\beta_1, \cdots, \beta_p)^T$ and $\mathbf x = (x_1, \cdots, x_p)^T$, we can use the more concise notation:

$$
y = \Theta^T \mathbf x
$$
Where $\Theta$ is the parameter column vector containing $\theta_0$ and the feature weights $\theta_1$ to $\theta_n$. and $\mathbf x_i$ is the feature column vector containing $x_{ti}$ for the $i$th sample.

To obtain $\hat{\Theta}$, we use the same approach as univariate linear regression. Find values $\hat{\beta_0}, \cdots, \hat{\beta_p}$ to minimise the MSE (and consequently RSS):

$$\begin{align}\text{MSE} (\mathbf X) &= \frac 1 n \sum_{i=1}^n \begin{bmatrix} \hat{y^{(i)}} - y^{(i)})\end{bmatrix}^2\\&= \frac 1 n \sum_{i=1}^n \begin{bmatrix} \hat{\beta_0} + \hat{\beta_1}x^{(i)}_1 + \hat{\beta_2}x^{(i)}_2 + \cdots + \hat{\beta_p}x^{(i)}_p - y^{(i)})\end{bmatrix}^2\end{align}$$

The closed-form solution for multiple linear regression is the normal equation. Let $\mathbf y = (y^{(1)}, \cdots, y^{(n)})^T$:

$$\hat{\Theta} = (\mathbf X ^T \mathbf X)^{-1} \mathbf X^T \mathbf y$$

```python
Xarray2 = np.c_[np.ones((X1.shape[0],1)), X2] # Add x0 = 1
```

```python
# For multiple linear regression, obtain the coefficients using the normal equations
Theta_hat21 = np.dot(np.dot(np.linalg.inv(np.dot(Xarray2.T, Xarray2)), Xarray2.T),y)
float_formatter = "{:.6f}".format
np.set_printoptions(formatter={'float_kind':float_formatter})
print(Theta_hat21)
```

    [2.938889 0.045765 0.188530 -0.001037]

```python
# Predict
print(np.dot(Xarray2[:2], Theta_hat21))
```

    [20.523974 12.337855]

```python
# Obtaining the coefficients using statsmodels
reg21 = sm.OLS(y, Xarray2)
results21 = reg21.fit()
Theta_hat22 = list(results21.params)
print(["{:.6f}".format(t) for t in Theta_hat22])
```

    ['2.938889', '0.045765', '0.188530', '-0.001037']

```python
# Predict
results21.predict(Xarray2[:2])
```

    array([20.523974, 12.337855])

```python
# Obtaining the coefficients using sklearn
reg2 = LinearRegression()
reg2.fit(Xarray2, y)
Theta_hat23 = reg2.coef_.copy()
Theta_hat23[0] = reg2.intercept_
print(Theta_hat23)
```

    [2.938889 0.045765 0.188530 -0.001037]

```python
# Predict
print(reg2.predict(Xarray2[:2]))
```

    [20.523974 12.337855]

### Gradient Descent - Batch Gradient Descent

Gradient descent is a generic optimization problem capable of finding optimal solutions to many problems. The idea is to tweak parameters iteratively to minimize a cost function.

- Note that when using gradient descent, the features must have a similar scale or it will take longer to converge.

Hence, to use gradient descent on a linear regression problem, we revisit the cost function, the MSE: $\text{MSE} (\Theta) = \frac 1 n \sum_{i=1}^n \begin{bmatrix} \Theta^T \mathbf x^{(i)} - y^{(i)})\end{bmatrix}^2$ and now compute the gradient w.r.t. the parameters $\beta_j$. Each expression is the partial derivative for every parameter $\beta_j$.
$$\frac{\partial}{\partial \beta_j}\text{MSE}(\Theta) = \frac 2n \sum_{i=1}^n \begin{bmatrix} \Theta^T \mathbf x^{(i)} - y^{(i)})\end{bmatrix} x_j^{(i)}$$

Combining them all, we obtain the gradient vector of the cost function:

$$\nabla_\beta \text{MSE}(\Theta) = \begin{bmatrix}\frac{\partial}{\partial \beta_0}\text{MSE}(\Theta)\\\frac{\partial}{\partial \beta_1}\text{MSE}(\Theta)\\\ \vdots\\\frac{\partial}{\partial \beta_p}\text{MSE}(\Theta)\end{bmatrix}= \begin{bmatrix}\frac 2n \sum_{i=1}^n \begin{bmatrix} \Theta^T \mathbf x^{(i)} - y^{(i)})\end{bmatrix} x_0^{(i)}\\\frac 2n \sum_{i=1}^n \begin{bmatrix} \Theta^T \mathbf x^{(i)} - y^{(i)})\end{bmatrix} x_1^{(i)}\\\ \vdots\\\frac 2n \sum_{i=1}^n \begin{bmatrix} \Theta^T \mathbf x^{(i)} - y^{(i)})\end{bmatrix} x_n^{(i)}\end{bmatrix} = \frac 2n \mathbf X^T (X\Theta - y)$$

In each step of gradient descent, we calculate $\nabla_\beta \text{MSE}(\Theta)$, the direction of movement for the gradients after going through the batch once. Then, we multiply this by the learning rate $\eta$, and update the parameter vector:
$$\Theta^{\text{new}} = \Theta - \eta \cdot \nabla_\theta \text{MSE}(\Theta)$$

```python
# GD for the multivariate case
eta = 0.00001 # learning rate
n_iterations = 50000 # number of iterations for GD

# Initialise the parameter vector
n, p = Xarray2.shape[0], Xarray2.shape[1]
Theta_hat2gd = np.random.normal(0,1,p)
print(Theta_hat2gd)
```

    [0.787003 0.068653 1.612421 -1.582863]

```python
for itn in range(n_iterations):
    gradients = 2/n * np.dot(Xarray2.T,np.dot(Xarray2, Theta_hat2gd) - np.array(y))
    Theta_hat2gd = Theta_hat2gd - eta * gradients
```

```python
print(Theta_hat2gd)
```

    [1.073751 0.050858 0.209926 0.010291]

And it can be verified that the values obtained via gradient descent is the same as that of the normal equations.

### Gradient Descent - Stochastic Gradient Descent

Batch gradient descent uses the whole training set when running every epoch. In contrast, Stochastic gradient descent picks a random sample from the training set at every epoch and computes the gradients based only on that single instance.

By convention, we iterate by rounds of $m$ iterations, and each round is called an epoch.

```python
# SGD for the univariate case, using SKLearn
# Train
scaler = StandardScaler()

reg_sgd1 = SGDRegressor(max_iter=1000000, penalty=None, eta0=0.0025)
Xarray1_s = scaler.fit_transform(Xarray1)
reg_sgd1.fit(Xarray1_s, y)
```

    SGDRegressor(alpha=0.0001, average=False, early_stopping=False, epsilon=0.1,
                 eta0=0.0025, fit_intercept=True, l1_ratio=0.15,
                 learning_rate='invscaling', loss='squared_loss', max_iter=1000000,
                 n_iter_no_change=5, penalty=None, power_t=0.25, random_state=None,
                 shuffle=True, tol=0.001, validation_fraction=0.1, verbose=0,
                 warm_start=False)

```python
# SGD for the multivariate case, using SKLearn
# Train
scaler2 = StandardScaler()
reg_sgd = SGDRegressor(max_iter=5000000, penalty=None, eta0=0.0025)
Xarray2_s = scaler2.fit_transform(Xarray2)
reg_sgd.fit(Xarray2_s, y)
```

    SGDRegressor(alpha=0.0001, average=False, early_stopping=False, epsilon=0.1,
                 eta0=0.0025, fit_intercept=True, l1_ratio=0.15,
                 learning_rate='invscaling', loss='squared_loss', max_iter=5000000,
                 n_iter_no_change=5, penalty=None, power_t=0.25, random_state=None,
                 shuffle=True, tol=0.001, validation_fraction=0.1, verbose=0,
                 warm_start=False)

```python
# Univariate Case
print(np.dot(Xarray1[:3], Theta_hat11)) # Closed-form solution
print(np.dot(Xarray1[:3], Theta_hat12)) # Closed-form solution, normal equations
print(results11.predict(Xarray1[:3])) # statsmodels
print(reg.predict(Xarray1[:3])) # sklearn LinearRegression
print(reg_sgd1.predict(Xarray1_s[:3])) # Stochastic Gradient Descent (SGDRegressor from sklearn)
```

    [17.970775 9.147974 7.850224]
    [17.970775 9.147974 7.850224]
    [17.970775 9.147974 7.850224]
    [17.970775 9.147974 7.850224]
    [17.826052 9.074389 7.787103]

```python
# Multivariate Case
print(np.dot(Xarray2[10:14],Theta_hat21)) # Normal equations
print(np.dot(Xarray2[10:14],Theta_hat22)) # statsmodels
print(reg2.predict(Xarray2[10:14])) # sklearn LinearRegression
print(np.dot(Xarray2[10:14],Theta_hat2gd)) # Batch Gradient Descent
print(reg_sgd.predict(Xarray2_s[10:14])) # Stochastic Gradient Descent (SGDRegressor from sklearn)
```

    [7.032299 17.285129 10.577121 8.826300]
    [7.032299 17.285129 10.577121 8.826300]
    [7.032299 17.285129 10.577121 8.826300]
    [5.902096 17.072383 10.330784 7.701954]
    [7.004401 17.085039 10.565385 8.734636]


# Solution {-}
Factor spectral density function $S(s)$:

```python
from sympy import symbols, factor

s = symbols('s')

S = (-s**2 + 1)/(s**4 - 8*s**2 + 16)
S
```

$\displaystyle \frac{1 - s^{2}}{s^{4} - 8 s^{2} + 16}$

```python
factor(S, s)
```

$\displaystyle - \frac{\left(s - 1\right) \left(s + 1\right)}{\left(s - 2\right)^{2} \left(s + 2\right)^{2}}$


# Introduction to some of the functions in SymPy

```python
import sympy as smp
from sympy import *
```

```python
x = smp.symbols('x')
```

```python
x**3
```

$\displaystyle x^{3}$

```python
x , y = smp.symbols('x y')
```

```python
f = x**2 + y **2
```

```python
f.subs(x,10) # substituting the value of x
```

$\displaystyle y^{2} + 100$

```python
f.subs(y,3)  # substituting the value of y
```

$\displaystyle x^{2} + 9$

```python
smp.sin(x)
```

$\displaystyle \sin{\left(x \right)}$

```python
smp.exp(1/x) # exponent
```

$\displaystyle e^{\frac{1}{x}}$

```python
smp.log(x) #Natural log
```

$\displaystyle \log{\left(x \right)}$

```python
smp.log(x,10) # log with a base 10
```

$\displaystyle \frac{\log{\left(x \right)}}{\log{\left(10 \right)}}$

```python
x**(smp.Rational(7,2)) # keeping the power odf x in fraction
```

$\displaystyle x^{\frac{7}{2}}$

# Lets test our Fortune with Limits.

```python
smp.sin(x/2+smp.sin(x))
```

$\displaystyle \sin{\left(\frac{x}{2} + \sin{\left(x \right)} \right)}$

```python
smp.limit(smp.sin(x/2+smp.sin(x)),x,smp.pi)
```

$\displaystyle 1$

```python
2*smp.exp(1/x)/(smp.exp(1/x)+1)
```

$\displaystyle \frac{2 e^{\frac{1}{x}}}{e^{\frac{1}{x}} + 1}$

```python
smp.limit(2*smp.exp(1/x)/(smp.exp(1/x)+1),x,0,dir ='+')
```

$\displaystyle 2$

```python
smp.limit(2*smp.exp(1/x)/(smp.exp(1/x)+1),x,0,dir ='-')
```

$\displaystyle 0$

```python
(smp.cos(x)-1)/x
```

$\displaystyle \frac{\cos{\left(x \right)} - 1}{x}$

```python
smp.limit(((smp.cos(x)-1)/x),x,smp.oo)   # Here smp.oo defins a limit of x with indinity..
```

$\displaystyle 0$

# Derivatives.........

```python
((1 + smp.sin(x))/(1-smp.cos(x)))**2
```

$\displaystyle \frac{\left(\sin{\left(x \right)} + 1\right)^{2}}{\left(1 - \cos{\left(x \right)}\right)^{2}}$

```python
smp.diff(((1 + smp.sin(x))/(1-smp.cos(x)))**2,x)
```

$\displaystyle \frac{2 \left(\sin{\left(x \right)} + 1\right) \cos{\left(x \right)}}{\left(1 - \cos{\left(x \right)}\right)^{2}} - \frac{2 \left(\sin{\left(x \right)} + 1\right)^{2} \sin{\left(x \right)}}{\left(1 - \cos{\left(x \right)}\right)^{3}}$

```python
smp.log(x,5)**(x/2)
```

$\displaystyle \left(\frac{\log{\left(x \right)}}{\log{\left(5 \right)}}\right)^{\frac{x}{2}}$

```python
smp.diff(smp.log(x,5)**(x/2),x)
```

$\displaystyle \left(\frac{\log{\left(x \right)}}{\log{\left(5 \right)}}\right)^{\frac{x}{2}} \left(\frac{\log{\left(\frac{\log{\left(x \right)}}{\log{\left(5 \right)}} \right)}}{2} + \frac{1}{2 \log{\left(x \right)}}\right)$

```python
# Let's Start with an abstract approach........../////

f,g = smp.symbols('f g', cls = smp.Function)
g = g(x)
f = f(x+g)
```

```python
f
```

$\displaystyle f{\left(x + g{\left(x \right)} \right)}$

```python
smp.diff(f,x) #Derivative of f with respect to x
```

$\displaystyle \left(\frac{d}{d x} g{\left(x \right)} + 1\right) \left. \frac{d}{d \xi_{1}} f{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x + g{\left(x \right)} }}$

# Intergration {Indefinate}

```python
smp.cos(x)*smp.cot(x)
```

$\displaystyle \cos{\left(x \right)} \cot{\left(x \right)}$

```python
smp.integrate(smp.cos(x)*smp.cot(x),x)  #By defalt SciPy do not give the conatant term. Hence, we have to keep this in mind
```

$\displaystyle \frac{\log{\left(\cos{\left(x \right)} - 1 \right)}}{2} - \frac{\log{\left(\cos{\left(x \right)} + 1 \right)}}{2} + \cos{\left(x \right)}$

```python
smp.csc(x)  #Cosec is coined as csc
```

$\displaystyle \csc{\left(x \right)}$

```python
4*smp.sec(3*x)*smp.tan(3*x)
```

$\displaystyle 4 \tan{\left(3 x \right)} \sec{\left(3 x \right)}$

```python
smp.integrate(4*smp.sec(3*x)*smp.tan(3*x),x)
```

$\displaystyle \frac{4}{3 \cos{\left(3 x \right)}}$

```python
2/smp.sqrt(1-x**2) - 1/(x)**(smp.Rational(1,4))
```

$\displaystyle \frac{2}{\sqrt{1 - x^{2}}} - \frac{1}{\sqrt[4]{x}}$

```python
smp.integrate(2/smp.sqrt(1-x**2) - 1/(x)**(smp.Rational(1,4)),x)
```

$\displaystyle - \frac{4 x^{\frac{3}{4}}}{3} + 2 \operatorname{asin}{\left(x \right)}$

# Initial Value Function

#1 Given [dy/dx = 8x+csc^2(x)] with y(pi/2)= -7 solve for y(x)

```python
integral = 8*x + (smp.csc(x))**2  # in the give equation we Take Integral Both Sides.
```

```python
integral = smp.integrate((integral),x)  # now, we will try to calculate the constat term {C}.
```

```python
integral.subs(x,smp.pi/2)   # substuiting the value of x with pi/2 we get pi^2
```

$\displaystyle \pi^{2}$

```python
C = -integral.subs(x,smp.pi/2) - 7
y = integral + C
```

```python
y.subs(x,smp.pi/2) # Verification
```

$\displaystyle -7$

```python
y   # as function of x
```

$\displaystyle 4 x^{2} - \pi^{2} - 7 - \frac{\cos{\left(x \right)}}{\sin{\left(x \right)}}$

#2 let me just define it--

```python
(1+smp.sqrt(x))**(smp.Rational(1,3))  / x**(smp.Rational(1,2))
```

$\displaystyle \frac{\sqrt[3]{\sqrt{x} + 1}}{\sqrt{x}}$

```python
smp.integrate((1+smp.sqrt(x))**(smp.Rational(1,3))  / x**(smp.Rational(1,2)),x)
```

$\displaystyle \frac{3 \sqrt{x} \sqrt[3]{\sqrt{x} + 1}}{2} + \frac{3 \sqrt[3]{\sqrt{x} + 1}}{2}$

#3

```python
x*(1-x**2)**smp.Rational(1/4)
```

$\displaystyle x \sqrt[4]{1 - x^{2}}$

```python
smp.integrate(x*(1-x**2)**smp.Rational(1/4),x)
```

$\displaystyle \frac{2 x^{2} \sqrt[4]{1 - x^{2}}}{5} - \frac{2 \sqrt[4]{1 - x^{2}}}{5}$

```python
(2*x-1)*smp.cos((3*(2*x-1)**2+6)**smp.Rational(1/2)) /(3*(2*x-1)**2+6)**smp.Rational(1/2)

```

$\displaystyle \frac{\left(2 x - 1\right) \cos{\left(\sqrt{3 \left(2 x - 1\right)^{2} + 6} \right)}}{\sqrt{3 \left(2 x - 1\right)^{2} + 6}}$

```python
smp.integrate((2*x-1)*smp.cos((3*(2*x-1)**2+6)**smp.Rational(1/2)) /(3*(2*x-1)**2+6)**smp.Rational(1/2),x)
```

$\displaystyle \frac{\sin{\left(\sqrt{3 \left(2 x - 1\right)^{2} + 6} \right)}}{6}$

# Integration{Definate}

```python
smp.exp(x)/(smp.exp(2*x)+9)**smp.Rational(1,2)
```

$\displaystyle \frac{e^{x}}{\sqrt{e^{2 x} + 9}}$

```python
smp.integrate(smp.exp(x)/(smp.exp(2*x)+9)**(smp.Rational(1,2)),(x,0,smp.log(4)))  #asin(x) is inverse of sin(x)
```

$\displaystyle - \operatorname{asinh}{\left(\frac{1}{3} \right)} + \operatorname{asinh}{\left(\frac{4}{3} \right)}$

```python
x**10 * smp.exp(x)
```

$\displaystyle x^{10} e^{x}$

```python
t = smp.symbols('t')
```

```python
smp.integrate(x**10 * smp.exp(x),(x,1,t))
```

$\displaystyle \left(t^{10} - 10 t^{9} + 90 t^{8} - 720 t^{7} + 5040 t^{6} - 30240 t^{5} + 151200 t^{4} - 604800 t^{3} + 1814400 t^{2} - 3628800 t + 3628800\right) e^{t} - 1334961 e$

# Improper Integrals

```python
(16*smp.atan(x))/ (1+x**2)
```

$\displaystyle \frac{16 \operatorname{atan}{\left(x \right)}}{x^{2} + 1}$

```python
smp.integrate((16*smp.atan(x))/ (1+x**2),(x,0,smp.oo))   #Very Cool
```

$\displaystyle 2 \pi^{2}$

# Summation Functions

```python
n = smp.symbols('n')
```

```python
6/4**n
```

$\displaystyle 6 \cdot 4^{- n}$

```python
smp.Sum(6/4**n, (n,0,smp.oo))
```

$\displaystyle \sum_{n=0}^{\infty} 6 \cdot 4^{- n}$

```python
#Bro Solve it ///////

smp.Sum(6/4**n, (n,0,smp.oo)).doit()

#///// WE have TO TELL SCIPY TO doit()
```

$\displaystyle 8$

```python
smp.Sum(2**(n+1)/5**n, (n,0,smp.oo))
```

$\displaystyle \sum_{n=0}^{\infty} 2^{n + 1} \cdot 5^{- n}$

```python
smp.Sum(2**(n+1)/5**n, (n,0,smp.oo)).doit()
```

$\displaystyle \frac{10}{3}$

```python
smp.Sum(smp.atan(n)/n**smp.Rational(11,10), (n,1,smp.oo))
```

$\displaystyle \sum_{n=1}^{\infty} \frac{\operatorname{atan}{\left(n \right)}}{n^{\frac{11}{10}}}$

```python
smp.Sum(smp.atan(n)/n**smp.Rational(11,10), (n,1,smp.oo)).doit()  # Some time it does not work..
```

$\displaystyle \sum_{n=1}^{\infty} \frac{\operatorname{atan}{\left(n \right)}}{n^{\frac{11}{10}}}$

```python
#Lets try this!! {.n()}
smp.Sum(smp.atan(n)/n**(smp.Rational(11,10)), (n,1,smp.oo)).n()
#////////Here, we are actually dealing with approximations./////////////////\\\\\\\\\\\\\\\\\\\\\\\\\
```

$\displaystyle 15.3028821020457$

```python
smp.Sum((1+smp.cos(n))/n**2, (n,1,smp.oo)).n()
```

$\displaystyle 1.969$

```python
# Happy Learning
```


```python
import sympy
sympy.init_printing()
import verify_3
```

# Section 1 - Linear Alebra Types

You can define both a vector and a matrix by creating an instance of Matrix from an array

```python
def demo_1a():

    vector = sympy.Matrix([1,2,3])
    matrix = sympy.Matrix([[1,2],[3,5]])
    display(vector)
    display(matrix)
demo_1a()
```

$\displaystyle \left[\begin{matrix}1\\2\\3\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 5\end{matrix}\right]$

Transpositions

```python
def demo_1b():

    vector = sympy.Matrix([1,2,3])
    matrix = sympy.Matrix([[1,2],[3,5]])
    display(vector)
    display(vector.transpose())
    display(matrix)
    display(matrix.transpose())
demo_1b()
```

$\displaystyle \left[\begin{matrix}1\\2\\3\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & 2 & 3\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 5\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & 3\\2 & 5\end{matrix}\right]$

Access individual entries

```python
def demo_1c():

    matrix = sympy.Matrix([[1,2],[3,5]])
    display(matrix)
    display(matrix[3]) # Read
    display(matrix[1,0])
    matrix[3] = 7 # Write
    display(matrix)
demo_1c()
```

__Exercise a__

Create the column vector $\left[x,y\right]^T$

```python
def exercise_1a():

    x = sympy.Symbol('x')
    y = sympy.Symbol('y')

    # Enter answer here
    answer = 0
    display(answer)
    print(verify_3.verify_1a(answer))
exercise_1a()
```

Slightly more interesting example, Jacobian matrix for polar coordinates

```python
def demo_1d():

    r = sympy.Symbol('r', positive=True)
    q = sympy.Symbol('theta', positive=True)
    x = r*sympy.cos(q)
    y = r*sympy.sin(q)
    jac = sympy.Matrix([[var1.diff(var2) for var1 in [x,y]] for var2 in [r,q]])
    display(jac)

demo_1d()
```

$\displaystyle \left[\begin{matrix}\cos{\left(\theta \right)} & \sin{\left(\theta \right)}\\- r \sin{\left(\theta \right)} & r \cos{\left(\theta \right)}\end{matrix}\right]$

# Section 2 - Multiplication

To multiply matrices, use the operator *

```python
def demo_2a():

    matrix_a = sympy.Matrix([[1,1],[1,1]])
    matrix_b = sympy.Matrix([[1,-1],[-1,1]])
    display(matrix_a)
    display(matrix_b)
    display(matrix_a*matrix_b)
demo_2a()
```

$\displaystyle \left[\begin{matrix}1 & 1\\1 & 1\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1 & -1\\-1 & 1\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}0 & 0\\0 & 0\end{matrix}\right]$

Similarly, matrices and vectors

```python
def demo_2b():

    matrix = sympy.Matrix([[1,2],[3,4]])
    vector = sympy.Matrix([1,1])
    display(matrix)
    display(vector)
    display(matrix*vector)
demo_2b()
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}1\\1\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}3\\7\end{matrix}\right]$

When multiplying vectors, and order and orientation is important

```python
def demo_1c():

    vector_a = sympy.Matrix([1,2])
    vector_b = sympy.Matrix([3,4])
    display(vector_a)
    display(vector_b)
    display(vector_a.transpose()*vector_b)
    display(vector_a*vector_b.transpose())
demo_1c()
```

$\displaystyle \left[\begin{matrix}1\\2\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}3\\4\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}11\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}3 & 4\\6 & 8\end{matrix}\right]$

__Exercise a__

Calculate the product $R \cdot b$ where $R$ is the 2D rotation matrix and $b = [1,0]$

```python
def exercise_2a():

    q = sympy.Symbol('theta', positive=True)# Rotation angle
    b = sympy.Matrix([1,0])
    R = sympy.Matrix([[sympy.cos(q), -sympy.sin(q)],[sympy.sin(q), sympy.cos(q)]])
    display(R,b)

    # Enter answer here
    answer = 0
    display(answer)
    print(verify_3.verify_2a(answer))
exercise_2a()
```

# Section 3 - Inversion and Null Spaces

To get the matrix use .inv()

```python
def demo_3a():

    matrix = sympy.Matrix([[1,2],[3,4]])
    display(matrix)
    display(matrix.inv())
demo_3a()
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}-2 & 1\\\frac{3}{2} & - \frac{1}{2}\end{matrix}\right]$

__Exercise a__

Solve the following system of equations $A \cdot x = b$

```python
def exercise_3a():

    A = sympy.Matrix([[1,2],[3,4]])
    b = sympy.Matrix([5,6])

    sol = 0
    print(verify_3.verify_3a(sol))
exercise_3a()
```

    False

If the matrix is not invertible, we can find its null space

```python
def demo_3b():

    matrix = sympy.Matrix([[1,1],[1,1]])
    display(matrix)
    display(matrix.nullspace())
demo_3b()
```

$\displaystyle \left[\begin{matrix}1 & 1\\1 & 1\end{matrix}\right]$

$\displaystyle \left[ \left[\begin{matrix}-1\\1\end{matrix}\right]\right]$

# Section 4 - Determinant

Taking the determinant

```python
def demo_4a():

    matrix = sympy.Matrix([[1,2],[3,4]])
    display(matrix)
    display(matrix.det())
demo_4a()
```

__Exercise a__

Find x for which the following matrix is singular

```python
def exercise_4a():

    x = sympy.Symbol('x', positive=True)
    matrix = sympy.Matrix([[1,2-x],[3,4]])

    answer = 0
    print(verify_3.verify_4a(answer))

exercise_4a()
```

    False

# Section 5 - Eigenvalue system

To find the eigenvalues

```python
def demo_5a():

    matrix = sympy.Matrix([[1,2],[3,4]])
    display(matrix.eigenvals())

demo_5a()
```

The first number is the eigenvalue, and the second number is the multiplicity.

To get the eigenvectors

```python
def demo_5b():

    matrix = sympy.Matrix([[1,2],[3,4]])
    display(matrix.eigenvects())

demo_5b()
```

$\displaystyle \left[ \left( \frac{5}{2} - \frac{\sqrt{33}}{2}, \  1, \  \left[ \left[\begin{matrix}- \frac{2}{- \frac{3}{2} + \frac{\sqrt{33}}{2}}\\1\end{matrix}\right]\right]\right), \  \left( \frac{5}{2} + \frac{\sqrt{33}}{2}, \  1, \  \left[ \left[\begin{matrix}- \frac{2}{- \frac{\sqrt{33}}{2} - \frac{3}{2}}\\1\end{matrix}\right]\right]\right)\right]$

In some cases it is also useful to find the characteristic polynomial

```python
def demo_5c():

    x = sympy.Symbol('x')
    matrix = sympy.Matrix([[1,2],[3,4]])
    display(matrix.charpoly('x'))
demo_5c()
```

__Exercise a__

A series of aligned point masses, with springs connecting the two adjacent pairs of masses. Their equations of motion can be described by

$\ddot{\vec{x}} = \vec{M} \cdot \vec{x}$. Find the eigensystem (eigenvectors + eigenvalues) for the matrix $M$. What does each of the modes represent?

```python
def exercise_5a():

    M = sympy.Matrix([[-1,1,0],
                      [1,-2,1],
                      [0,1,-1]])

    display(M.eigenvects())

    answer = 0
    print(verify_3.verify_5a(answer))
exercise_5a()
```

$\displaystyle \left[ \left( -3, \  1, \  \left[ \left[\begin{matrix}1\\-2\\1\end{matrix}\right]\right]\right), \  \left( -1, \  1, \  \left[ \left[\begin{matrix}-1\\0\\1\end{matrix}\right]\right]\right), \  \left( 0, \  1, \  \left[ \left[\begin{matrix}1\\1\\1\end{matrix}\right]\right]\right)\right]$

    False

# Section 6 - Matrix functions

sympy interprets functions on matrices as matrix functions (rather than operations on the

```python
def demo_6a():

    matrix = sympy.Matrix([[0,-1],[1,0]])
    display(matrix)
    display(sympy.exp(matrix))

demo_6a()
```

$\displaystyle \left[\begin{matrix}0 & -1\\1 & 0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}\cos{\left(1 \right)} & - \sin{\left(1 \right)}\\\sin{\left(1 \right)} & \cos{\left(1 \right)}\end{matrix}\right]$

```python
def demo_6b():

    matrix = sympy.Matrix([[0,-1],[1,0]])
    display(matrix)
    display(matrix**2)

demo_6b()
```

$\displaystyle \left[\begin{matrix}0 & -1\\1 & 0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}-1 & 0\\0 & -1\end{matrix}\right]$

__Exercise a__

In this problem we will explore the generator to the Lorentz boost. Calculate $\exp \left(\psi Z\right)$ ,where $Z$ is the anti diagonal unit matrix and $\tanh \psi = \beta$ is the rapidity.

```python
def exercise_6a():

    psi = sympy.Symbol('psi', positive=True) # Rapidity
    gen = sympy.Matrix([[0,-1],[-1,0]])

    # Enter anser here
    temp = sympy.exp(psi*gen)
    temp.simplify()
    display(temp)
    answer = sympy.Rational(0,1)
    print(verify_3.verify_6a(answer))

exercise_6a()
```

$\displaystyle \left[\begin{matrix}\frac{e^{\psi}}{2} + \frac{e^{- \psi}}{2} & - \frac{e^{\psi}}{2} + \frac{e^{- \psi}}{2}\\- \frac{e^{\psi}}{2} + \frac{e^{- \psi}}{2} & \frac{e^{\psi}}{2} + \frac{e^{- \psi}}{2}\end{matrix}\right]$

    False

# Section 7 - Worked example

## Tidal Field

In this section we derive the tidal field of a point mass. Close to some distant point the potential behaves roughly as $\varphi \approx \frac{1}{2} \vec{x} T \vec{x}$ where $T$ is the tidal tensor.

```python
def demo_7a():

    x = sympy.Symbol('x', real=True)
    y = sympy.Symbol('y', real=True)
    z = sympy.Symbol('z', real=True)
    G = sympy.Symbol('G', positive=True) # Gravitation constant
    M = sympy.Symbol('M', positive=True) # Mass
    a = sympy.Symbol('a', positive=True) # Distance

    potential = G*M/sympy.sqrt(x**2+y**2+z**2)

    display(potential)

    hessian = sympy.Matrix([[potential.diff(var1).diff(var2).simplify()
                             for var1 in [x,y,z]] for var2 in [x,y,z]])

    display(hessian)

demo_7a()
```

## Two Body Problem

```python
def demo_7b():

    r = sympy.Function('r', positive=True)
    s = sympy.Function('s', positive=True)
    qf = sympy.Function('theta', real=True)
    q = sympy.Symbol('theta', real=True)
    t = sympy.Symbol('t', real=True)
    pos = sympy.Matrix([r(t)*sympy.cos(qf(t)),r(t)*sympy.sin(qf(t))])
    G = sympy.Symbol('G', positive=True) # Gravitation constant
    M = sympy.Symbol('M', positive=True) # Mass
    l = sympy.Symbol('l', positive=True) # Angular momentum
    u = sympy.Symbol('u', real=True) # Energy
    xi = sympy.Symbol('xi', positive=True) # Auxiliary variable
    energy = -G*M/r(t)+(r(t).diff(t)**2+qf(t).diff(t)**2*r(t)**2)/2-u
    eom = pos.diff(t,2)+G*M*pos/(pos.transpose()*pos)**sympy.Rational(3,2)

    v = pos.diff(t)
    LRL = l*sympy.Matrix([v[1],-v[0]])-G*M*pos/sympy.sqrt((pos.transpose()*pos)[0])
    LRL = sympy.Matrix([LRL[0].simplify(), LRL[1].simplify()])
    LRL = LRL.subs(r(t),xi).subs(xi,r(t))

    print('conservation of angular momentum')
    temp = eom.transpose()*sympy.Matrix([sympy.sin(qf(t)), -sympy.cos(qf(t))])
    temp = temp[0].simplify()
    display(temp)
    temp = (r(t)**2*qf(t).diff(t)).diff(t).subs(sympy.solve(temp,qf(t).diff(t,2),dict=True)[0])
    display(temp)

    print('conservation of the LRL vector')
    temp = LRL
    display(LRL)
    temp = temp.diff(t)
    temp = temp.subs(sympy.solve(eom,[r(t).diff(t,2), qf(t).diff(t,2)],dict=True)[0])
    temp = temp.subs(qf(t).diff(t),l/r(t)**2)
    temp = sympy.Matrix([temp[0].simplify(), temp[1].simplify()])
    display(temp)

    print('trajectory')
    temp = energy
    temp = temp.subs(r(t).diff(t), r(q).diff(q)*qf(t).diff(t))
    temp = temp.subs(qf(t).diff(t), l/r(q)**2)
    temp = temp.subs(r(t), r(q))
    energy_r_q = temp
    temp = temp.subs(r(q),1/s(q))
    temp = temp.doit()
    temp = temp.diff(q)
    temp = temp/s(q).diff(q)
    temp = temp.simplify()
    temp = sympy.dsolve(temp,s(q))
    temp = 1/sympy.solve(temp,s(q))[0]
    temp = temp.subs(sympy.solve(temp.diff(q).subs(q,0),sympy.Symbol('C1'),dict=True)[0])
    display(temp)
    temp = temp.subs(sympy.solve(energy_r_q.subs(r(q),temp).doit(),sympy.Symbol('C2'),dict=True)[1])
    display(temp)

demo_7b()
```

## Sound Waves

```python
def demo_7c():

    x = sympy.Symbol('x', real=True) # Position
    t = sympy.Symbol('t', real=True) # Time
    rho = sympy.Symbol('rho') # Density
    v = sympy.Symbol('v') # Velocity
    p = sympy.Symbol('p') # Pressure
    s = sympy.Symbol('s') # Entropy
    J = sympy.Symbol('J') # Mass flux
    xi = sympy.Symbol('xi') # Auxiliary variable
    gamma = sympy.Symbol('gamma', positive=True) # Adiabatic index

    # Hydrodynamic equations
    continuity_equation = sympy.Derivative(rho,t)+sympy.Derivative(J,x)
    momentum_conservatoin = sympy.Derivative(v,t)+v*sympy.Derivative(v,x)+sympy.Derivative(p,x)/rho
    entropy_conservation = sympy.Derivative(s,t)+v*sympy.Derivative(s,x)
    hydro_eqns = [continuity_equation,
                  momentum_conservatoin,
                  entropy_conservation]
    print('equations of motion')
    display(hydro_eqns)

    # Perturbation
    rho_0 = sympy.Symbol('rho_0', positive=True) # Ambient density
    p_0 = sympy.Symbol('p_0', positive=True) # Ambient pressure
    delta_rho = sympy.Symbol(r'\delta \rho', positive=True) # Density fluctuation amplitude
    delta_p = sympy.Symbol(r'\delta p', positive=True) # Pressure fluctuation amplitude
    delta_v = sympy.Symbol(r'\delta v', positive=True) # Velocity fluctuation amplitude
    omega = sympy.Symbol('omega', real=True) # Frequency
    k = sympy.Symbol('k', positive=True) # Wavenumber
    epsilon = sympy.Symbol('epsilon') # Perturbative parameter
    temp = hydro_eqns
    mode = sympy.exp(sympy.I*(omega*t-k*x))
    temp = [itm.subs({rho:rho_0 + epsilon*delta_rho*mode,
                      p:p_0 + epsilon*delta_p*mode,
                      v:epsilon*delta_v*mode,
                      J:(rho_0 + epsilon*delta_rho*mode)*(epsilon*delta_v*mode), # sympy spoonfeed
                      s:sympy.log(p_0 + epsilon*delta_p*mode) - gamma*sympy.log(rho_0 + epsilon*delta_rho*mode)})
            for itm in temp]
    temp = [itm.doit() for itm in temp]
    temp = [itm.diff(epsilon).subs(epsilon,0) for itm in temp]
    temp = [itm/mode for itm in temp]
    temp = [sympy.expand(itm) for itm in temp]
    temp = sympy.Matrix([[itm.diff(var) for itm in temp] for var in [delta_rho, delta_p, delta_v]])
    print('Solutions are only possible if this matrix is singular')
    display(temp)
    print('The determinant is')
    temp = temp.det().simplify()
    display(temp)
    print('And the frequencies are')
    display(sympy.solve(temp,omega))
    print('These solutions represent a left going wave, a right going wave and a density perturbation that does not move')

demo_7c()
```

```python

```


```python
import numpy as np
import numpy.linalg as LA
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt
```

# Logistic Regression

## Maximim Log-likelihood

Here we will use logistic regression to conduct a binary classification.  The logistic regression process will be formulated using Maximum Likelihood estimation.  To begin, consider a random variable $y \in\{0,1\}$.  Let the $p$ be the probability that $y=1$. Given a set of $N$ trials, the liklihood of the sequence $y_{1}, y_{2}, \dots, y_{N}$ is given by:

$\mathcal{L} = \Pi_{i}^{N}p^{y_{i}}(1-p)^{1 - y_{i}}$

Given a set of labeled training data, the goal of maximum liklihood estimation is to determine a probability distribution that best recreates the empirical distribution of the training set.

The log-likelihood is the logarithmic transformation of the likelihood function.  As logarithms are strictly increasing functions, the resulting solution from maximizing the likelihood vs. the log-likelihood are the equivalent.  Given a dataset of cardinality $N$, the log-likelihood (normalized by $N$) is given by:

$l = \frac{1}{N}\sum_{i=1}^{N}\Big(y_{i}\log(p) + (1 - y_{i})\log(1 - p)\Big)$

## Logistic function

Logistic regression performs binary classification based on a probabilistic interpretation of the data.  Essentially, the process seeks to assign a probability to new observations.  If the probability associated with the new instance of data is greater than 0.5, then the new observation is assigned to 1 (for example).  If the probability associated with the new instance of the data is less than 0.5, then it is assigned to 0.  To map the real numerical values into probabilities (which must lie between 0 and 1), logistic regression makes use of the logistic (sigmoid) function, given by:

$\sigma(t) = \frac{1}{1 + e^{-t}}$

Note that by setting $t=0$, $\sigma(0) = 0.5$, which is the decision boundary.  We should also note that the derivative of the logistic function with respect to the parameter $t$ is:

$\frac{d}{dt}\sigma(t) = \sigma(t)(1 - \sigma(t))$

## Logistic Regression and Derivation of the Gradient

Let's assume the training data consists of $N$ observations, where observation $i$ is denoted by the pair $(y_{i},\mathbf{x}_{i})$, where $y \in \{0,1\}$ is the label for the feature vector $\mathbf{x}$.  We wish to compute a linear decision boundary that best seperates the labeled observations.  Let $\mathbf{\theta}$ denote the vector of coefficients to be estimated.  In this problem, the log likelihood can be expressed as:

$l =  \frac{1}{N}\sum_{i=1}^{N}\Big(y_{i}\log\big(\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big) + (1 - y_{i}) \log\big( 1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big)\Big)$

The gradient of the objective with respect to the $j^{th}$ element of $\mathbf{\theta}$ is:
$$
\begin{aligned}
\frac{d}{d\theta^{(j)}} l &=  \frac{1}{N}\sum_{i=1}^{N}\Bigg( \frac{d}{d\theta^{(j)}} y_{i}\log\big(\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big) + \frac{d}{d\theta^{(j)}}(1 - y_{i}) \log\big( 1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\big)\Bigg) \\
&= \frac{1}{N}\sum_{i=1}^{N}\Bigg(\frac{y_{i}}{\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} - \frac{1 - y_{i}}{1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} \Bigg)\frac{d}{d\theta^{(j)}}\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\\
&= \frac{1}{N}\sum_{i=1}^{N}\Bigg(\frac{y_{i}}{\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} - \frac{1 - y_{i}}{1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})} \Bigg)\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big(1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big)x_{i}^{(j)}\\
&=  \frac{1}{N}\sum_{i=1}^{N}\Bigg(\frac{y_{i} - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})}{\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big(1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big)}\Bigg)\sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big(1 - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Big)x_{i}^{(j)}\\
&= \frac{1}{N}\sum_{i=1}^{N}\Bigg(y_{i} - \sigma(\mathbf{\theta}^{T}\mathbf{x}_{i})\Bigg)x_{i}^{(j)}
\end{aligned}
$$

where the last equation has the familiar form of the product of the prediciton error and the $j^{th}$ feature.  With the gradient of the log likelihood function, the parameter vector $\mathbf{\theta}$ can now be calculated via gradient ascent (as we're maximizing the log likelihood):

$$
\begin{equation}
    \mathbf{\theta}^{(j)}(k+1) = \mathbf{\theta}^{(j)}(k) + \alpha \frac{1}{N}\sum_{i=1}^{N}\Bigg( y_{i} - \sigma(\mathbf{\theta}^{T}(k)\mathbf{x}_{i}))\Bigg)x_{i}^{(j)}
\end{equation}
$$

```python
# Supporting Methods

#logistic function
def sigmoid(a):
    return 1/(1 + np.exp(-a))

#ll function
def log_likelihood(x, y, theta):
    logits = np.dot(x, theta)
    log_like = np.sum(y * logits - np.log(1 + np.exp(logits)))
    return log_like
```

```python
#Load the data
iris = datasets.load_iris()
x = iris.data[:,2:]                    #features will be petal width and petal length
y = (iris.target==2).astype(int).reshape(len(x),1)    #1 of iris-virginica, and 0 ow

#Prepare Data for Regression
#pad x with a vector of ones for computation of intercept
x_aug = np.concatenate( (x,np.ones((len(x),1))) , axis=1)
```

```python
#sklearn logistic regression
log_reg = LogisticRegression(penalty='none')
log_reg.fit(x,y.ravel())
log_reg.get_params()
coefs = log_reg.coef_.reshape(-1,1)
intercept = log_reg.intercept_
theta_sklearn = np.concatenate((coefs, intercept.reshape(-1,1)), axis=0)
print("sklearn coefficients:")
print(theta_sklearn)
print("sklearn log likelihood: ", log_likelihood(x_aug, y, theta_sklearn))
```

    sklearn coefficients:
    [[  5.75452053]
     [ 10.44681116]
     [-45.27248307]]
    sklearn log likelihood:  -10.281754052558687

```python
#Perform Logistic Regression
num_iterations = 40000
alpha = 1e-2

theta0 = np.ones((3,1))
theta = []
theta.append(theta0)

k=0
while k < num_iterations:

    #compute prediction error
    e = y - sigmoid(np.dot(x_aug, theta[k]))

    #compute the gradient of the log-likelihood
    grad_ll = np.dot(x_aug.T, e)

    #gradient ascent
    theta.append(theta[k] + alpha * grad_ll)

    #update iteration step
    k += 1

    if k % 4000 == 0:
        #print("iteration: ", k, " delta: ", delta)
        print("iteration: ", k, " log_likelihood:", log_likelihood(x_aug, y, theta[k]))

theta_final = theta[k]
print("scratch coefficients:")
print(theta_final)
```

    iteration:  4000  log_likelihood: -11.529302688612685
    iteration:  8000  log_likelihood: -10.800986140073512
    iteration:  12000  log_likelihood: -10.543197464480874
    iteration:  16000  log_likelihood: -10.425775111214602
    iteration:  20000  log_likelihood: -10.36535749825992
    iteration:  24000  log_likelihood: -10.331961877189842
    iteration:  28000  log_likelihood: -10.312622172168293
    iteration:  32000  log_likelihood: -10.301055609225223
    iteration:  36000  log_likelihood: -10.293975480174714
    iteration:  40000  log_likelihood: -10.289566343598294
    scratch coefficients:
    [[  5.5044475 ]
     [ 10.17562424]
     [-43.60242548]]

```python
#Plot the data and the decision boundary

#create feature data for plotting
x_dec_bnd = np.linspace(0,7,100).reshape(-1,1)
#classification boundary from sklearn
y_sklearn = (theta_sklearn[2] * np.ones((100,1)) + theta_sklearn[0] * x_dec_bnd) / -theta_sklearn[1]
#classification boundary from scratch
y_scratch = (theta_final[2] * np.ones((100,1)) + theta_final[0] * x_dec_bnd) / -theta_final[1]

y_1 = np.where(y==1)[0] #training data, iris-virginica
y_0 = np.where(y==0)[0] #training data, not iris-virginica
plt.plot(x[y_0,0],x[y_0,1],'bo',label="not iris-virginica")
plt.plot(x[y_1,0],x[y_1,1],'k+',label="iris-virginica")
plt.plot(x_dec_bnd,y_sklearn,'r',label="sklearn dec. boundary")
plt.plot(x_dec_bnd,y_scratch,'g',label="scratch dec. boundary")
plt.xlabel('petal length')
plt.ylabel('petal width')
plt.title('Logistic Regression Classification')
plt.xlim((3,7))
plt.ylim((0,3.5))
plt.legend()
plt.show()
```


# Bose Einstein Coefficients

A collection of $n$ objects that can have one of $k$ different classes are selected. The classes have equal probability. How many ways are there to select the objects?

Alternatively: how many ways are there to partition $n$ objects into $k$ buckets?

If $n_i$ is the number of objects in bucket $i$, then $\sum_i n_i = n$.

## Stars and Bars

This is a neat way to approach counting problems.

Dividing objects into $k$ classes requires $k-1$ separators. The division can be visualized of a sequence of $n+k-1$ objects and dividers.

Consider 10 objects divided over 4 classes.

That makes 10 objects and 3 dividers that are allocated to $n+k-1$ slots:

``[ , , , , , , , , , , , , ]``

For example:

``[|,* ,* ,* ,|,* ,| ,* ,* ,* ,* ,* ,* ]``

Corresponds to an allocation of $\{0,3,1,6\}$ across the 4 classes.

The amount of ways to distribute $n$ objects over $n+k-1$ is:

\begin{equation}
\left(\begin{array}{c}n+k-1\\n\end{array}\right)
\end{equation}

Which is the result.

The Bose Einstein Coefficient is also known as the multichoose coefficient. It counts the ways that $n$ indistinct objects can be assigned to $k$ classes.

# Rising Factorials

What if the the objects that are distributed are distinct, so that they can have an internal ordering? For example, consider the case of $n$ books being distributed over $k$ library shelves.

In that case, there are

\begin{equation}
\left(\begin{array}{c}n+k-1\\n\end{array}\right)
\end{equation}

Ways of dividing the books up between the shelves, and

\begin{equation}
n!
\end{equation}

ways for the books to be ordered internally.

For example:

``[|,1 ,3 ,6 ,|,4 ,| ,5 ,7 ,9 ,8 ,11 ,10 ]``

The amount of combinations is:

\begin{equation}
n! {n + k - 1 \choose n} = \frac{(n+k-1)!}{(k-1)!} = (n+k-1)^\underline{n} = k^\overline{n}
\end{equation}

Combinatorics - Stars and Bars, Bose Einstein Coefficients.ipynb


# Circular motion

## A simple one-link manipulator
Consider the simple manipulator below, consisting of a homogenous rod of mass $m=0.1$kg and length $l=1$m. The rod is rotating about the single joint with a constant angular velocity of $\omega=2$rad/s.

<!---  -->

### What is the centripetal acceleration at the endpoint?

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Parameters. SI units, of course
m = 0.1
l = 1.0
omega = 2.0

# The formular for centripetal acceleration is
a_c = omega**2 * l
print "The centripetal acceleration at the endpoint is %1.2f m/s^2" % a_c
```

    The centripetal acceleration at the endpoint is 4.00 m/s^2

### What is the centripetal acceleration at the center of mass?

```python
# %load com-acceleration.py
# The center of mass for a homogeneous rod is of course in the middle of the rod
from IPython.display import display, Math, Latex
display(Latex(r"The centripetal acceleration at the center of mass is %1.2f $m/s^2$" % (omega**2*l/2)))
```

The centripetal acceleration at the center of mass is 2.00 $m/s^2$

## Derivation of the acceleration for uniform circular motion
Define the x- and y axis as in the figure below. The angle $\theta$ to be the angle to the x-axis

<!---  -->

We have defined unit vectors in a polar coordinate system attached to the moving link
\begin{align}
e_r &= \begin{bmatrix}\cos\theta\\\sin\theta\end{bmatrix}\\
e_\theta &= \begin{bmatrix}-\sin\theta\\\cos\theta\end{bmatrix}
\end{align}
With these unit vectors it is easy to express the position $p$ of the endpoint with respect to the origin, which is located at the joint:
$$ p(t) = l e_r(t) = l \begin{bmatrix}\cos\theta\\\sin\theta\end{bmatrix}. $$

In order to find the velocity of the endpoint we take the time-derivative of the position
\begin{align}
   v(t) &= \frac{d}{dt} p(t) = \left( \frac{d}{dt} l \right) e_r + l \frac{d}{dt} e_r\\
        &= l \frac{d}{dt} \begin{bmatrix}\cos\theta\\\sin\theta\end{bmatrix}\\
        &= l \frac{d}{d\theta}\begin{bmatrix}\cos\theta\\\sin\theta\end{bmatrix} \frac{d\theta}{dt}\\
        &= l \omega \begin{bmatrix}-\sin\theta\\\cos\theta\end{bmatrix} = l \omega e_\theta.
\end{align}
We see that the *speed*, the magnitude of the velocity, $|v| = l \omega $ is proportional to both the distance from the center of rotation and to the angular velocity. The velocity vector is tangential to the path of the endpoint, and if $\omega$ is positive, then the direction of the velocity is in the positive direction of $e_\theta$.

### Exercise: derive the centripetal acceleration by differentiating the velocity
Also: Discuss the magnitude and direction of the centripetal acceleration.

```python
# %load centripetal-acceleration-solution.py
display(Latex("""
The acceleration becomes
\\begin{align}
a(t) &= \\frac{d}{dt} v(t) = \\frac{d}{dt} l \\omega e_\\theta = \\frac{d}{dt} l \\omega \\begin{bmatrix} -\\sin\\theta \\\\ \\cos\\theta \\end{bmatrix} \\\\
     &= \\frac{d}{dt} \left( l \\ omega \\right) e_\\theta + l \\omega \\frac{d}{dt} \\begin{bmatrix} -\\sin\\theta \\\\ \\cos\\theta \\end{bmatrix} \\\\
     &= 0 + l \\omega^2 \\begin{bmatrix} -\\cos\\theta \\\\ -\\sin\\theta \\end{bmatrix} = - l \omega^2 e_r.
\\end{align}
"""))
```

The acceleration becomes
\begin{align}
a(t) &= \frac{d}{dt} v(t) = \frac{d}{dt} l \omega e_\theta = \frac{d}{dt} l \omega \begin{bmatrix} -\sin\theta \\ \cos\theta \end{bmatrix} \\
     &= \frac{d}{dt} \left( l \ omega \right) e_\theta + l \omega \frac{d}{dt} \begin{bmatrix} -\sin\theta \\ \cos\theta \end{bmatrix} \\
     &= 0 + l \omega^2 \begin{bmatrix} -\cos\theta \\ -\sin\theta \end{bmatrix} = - l \omega^2 e_r.
\end{align}

```python

```

```python

```


# Market Equilibrium under different market forms

Import various packages

```python
import numpy as np
import scipy as sp
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm
from scipy import optimize,arange
from numpy import array
import ipywidgets as widgets # Import a package for interactive plots
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# Model Description

We consider the standard economic promlem for a Monopoly firm maximizing it's profits.

The aggregate market demand is given by
$$Q(p)=A-\alpha\cdot p$$
which corresponds to the inverse market demand function
$$p(Q)=\frac{A}{\alpha}-\frac{1}{\alpha}\cdot Q$$

and the Monopoly profits are given $$\pi(q)=p\cdot q-c(q)=\left(\frac{A}{\alpha}-\frac{1}{\alpha}\cdot q\right)\cdot q-c\cdot q$$

where $q=Q$, $p(Q)$ is a linear market demand curve and $c(q)$ is the firms cost-function with constant cost $c$.

# Market Equilibrium

## Analytical Solution

Using Sympy, we seek to find an analytical expression for the market equilibrium when one firm has monopoly power, i.e. solve the monopoly firm's maximization problem

\\[ \max_{q}\pi(q)=\max_{q} \left(\frac{A}{\alpha}-\frac{1}{\alpha}\cdot q\right)\cdot q-c\cdot q \\]

Which has the standard solution given by:
$$q^{M\ast}=\frac{A-\alpha\cdot c}{2}\wedge p^{\ast}=\frac{A+\alpha\cdot c}{2\cdot\alpha}$$

```python
sm.init_printing(use_unicode=True) # sets printing on
# Defining variables;
A = sm.symbols('A')
q = sm.symbols('q')
c = sm.symbols('c')
alpha=sm.symbols('alpha')

```

```python
Pi = (A/alpha-q/alpha)*q-c*q # Define the firms profit function
F = sm.diff(Pi,q) # Take the first order condition
sm.solve(F,q)[0]
```

```python
Mq = sm.solve(F,q)[0] # Solves F for market quantity
# And the market price is given by;
Mp=(A-Mq)*1/alpha
sm.Matrix([Mq,Mp]) # Prints the market quantity and price

```

```python
#For later use, We turn the above solution into a Python function
Mq_func = sm.lambdify((A,alpha,c),Mq)
Mp_func = sm.lambdify((A,alpha,c),Mp)
```

## Numerical Solution

As a brief introduction to solving the problem numerically, we use a solver like fsolve to solve the first-order condition given the following parameter values:

Remember, the first-order condition is given by:
$$\frac{A}{\alpha}-c-\frac{2q}{\alpha}=0$$

```python
A = 4
alpha = 2
c = 1
output = optimize.fsolve(lambda q: 2-q-1,0)
print(f'analytical solution for market quantity is: {Mq_func(A,alpha,c):.2f}')
print(f' Solution with fsolve for market quantity is: {output}')
print(f'analytical solution for market price is: {Mp_func(A,alpha,c):.2f}')
```

    analytical solution for market quantity is: 1.00
     Solution with fsolve for market quantity is: [1.]
    analytical solution for market price is: 1.50

However for later use, It is perhaps more efficent to make Python maximize the firm's profits directly. However, as scipy only has minimization procedueres. We continue to minimize $-\pi(q)$, i.e. minimizing negative profits is the same as maximizing profits.

Below we first define functions for market demand and costs in python

```python
def demand(Q):
    return A/alpha-1/alpha*Q

def cost(q,c):  # c is constant marginal cost
    return c*q
```

```python
def minus_profits(q,*args):         # we want to see profits as a function of q when we maximize profits or
    return -(demand(q)*q-cost(q,c)) # minimize minus_profits; hence c is specified as "*args", when calling fmin
                                    # we specify the c in the "args=(c,)"
x0 = 0 # Initial guess
c = 1.0 # Specify the value of the constant cost 'c'
A=4.0 # Specify the value of the Constant in the market demand function Q(p)
alpha=2.0 # Specify the  slope coefficient in Q(p)

output = optimize.fmin(minus_profits,x0,args=(c,)) # note the comma in "args(c,)"; it needs to be there!
price=A/alpha-1/alpha*output
print(output,price)
```

    Optimization terminated successfully.
             Current function value: -0.500000
             Iterations: 25
             Function evaluations: 50
    [1.] [1.5]

Hence, the optimal output is 1, which yields the maximum profits of $-\cdot(-0.5)=0.5$

For the specified parameter values, we have plotted the monopoly firm's profit function below.

```python
# Define the expression whose roots we want to find

A = 4.0 # Specify the value of the Constant in the market demand function Q(p)
alpha = 2.0 # Specify the  slope coefficient in Q(p)
c = 1.0 # Specify the value of the constant cost 'c'

func = lambda q : (A/alpha-q/alpha)*q-c*q # Defines the profit function using a lambda function.

# Plot the profit function

q = np.linspace(0, 2, 200) # Return evenly spaced numbers over a specified interval from 0 to 2 .

plt.plot(q, func(q)) # -minus_profits(q) could have been used instead of func(q). But we wanted to show the lambda function.
plt.axhline(y=0.5,linestyle='dashed',color='k') # creates a horizontal line in the plot at func(q)=0.5
plt.axvline(x=1,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=0.5
plt.xlabel("Quantity produced q ")
plt.ylabel("Firm Profits")
plt.grid()
plt.title('The Monopoly Firms profit function')
plt.show()
```

And we can plot the market equilibrium price and output in a standard diagram as shown below.

```python
# Define marginal Revenue:
def MR(Q):
    return A/alpha-2/alpha*Q

plt.plot(q, demand(q))
plt.plot(q, MR(q))
plt.axhline(y=c,color='k') # creates a horizontal line in the plot at func(q)=0.5
plt.axvline(x=output,ymin=0,ymax=0.73,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=0.5
plt.axhline(y=price,xmin=0, xmax=0.5,linestyle='dashed',color='k')
plt.xlabel("Quantity produced q ")
plt.ylabel("Price")
plt.grid()
plt.title('The Demand Function')
plt.show()
```

Both plottet side by side.

```python
f = plt.figure(figsize=(13,6))
ax = f.add_subplot(121)
ax2 = f.add_subplot(122)
ax.plot(q, func(q))
ax.set_title('The Monopoly Firms profit function')
ax.set_xlabel('Quantity Produced q')
ax.set_ylabel('Firm Profits')
ax.axhline(y=0.5,linestyle='dashed',color='k') # creates a horizontal line in the plot at func(q)=0.5
ax.axvline(x=1,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=0.5
ax.set_ylim(0,)  # set a lower limit for y-axis at zero.
ax.grid()

ax2.plot(q, demand(q),label='Demand')
ax2.plot(q,MR(q), label='Marginal Revenue')
ax2.legend(loc='upper right') # Place the graph descriptions in the upper right corner
ax2.grid()
ax2.axhline(y=c,color='k', label='Marginal Cost') # creates a horizontal line in the plot at func(q)=0.5
ax2.axvline(x=output,ymin=0,ymax=0.71,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=0.5
ax2.axhline(y=price,xmin=0, xmax=0.5,linestyle='dashed',color='k')
ax2.set_xlabel("Quantity produced q ")
ax2.set_ylabel("Market Price p")
ax2.set_ylim(0,)
ax2.set_title('The Market Equilibrium')
```

We see that when the monopoly firm is producing $q=1$, they get the maximum profits of 0.5. In the right figure, we see that the monopoly firm maximises profits in the point, where the marginal revenue curve intersect the marginal cost curve(black line). The two curves intersect at $q=1$, and the market price is $p=1.5$ as given by the demand curve.

**Extention at the exam**: We have extended the above plot with fixed parameter values with an interactive feature such that you can change the two graphs by changing the parameters $(A,\alpha,c)$.

```python
def interactive_figure(A,alpha,c):

    """This function makes a intertive figure of the profit function, demand function and marginal revenue function
    with A,alpha and c as free variables"""

    # a. Specify the functions.

    func = lambda q : (A/alpha-q/alpha)*q-c*q # Defines the profit function using a lambda function.

    def demand(Q): # defies demand function
        return A/alpha-1/alpha*Q

    def MR(Q): # defines the marginal revenue function
        return A/alpha-2/alpha*Q

    # b. Create values for quantity.

    q = np.linspace(0, A-alpha*c, 200) # Return evenly spaced numbers over a specified interval from 0 to the point, where profits equal zero .
    qM = np.linspace(0, A/2, 200) # Return evenly spaced numbers over the interval from 0 to where MR(Q) intersect the x-axis.
    qD = np.linspace(0, A, 200) # Return evenly spaced numbers over the interval from 0 to where DM(Q) intersect the x-axis.

    # c. plot the figures
    f = plt.figure(figsize=(13,6))
    ax = f.add_subplot(121)
    ax2 = f.add_subplot(122)
    ax.plot(q, func(q))
    ax.set_title('The Monopoly Firms profit function')
    ax.set_xlabel('Quantity Produced q')
    ax.set_ylabel('Firm Profits')
    ax.axhline(y=np.max(func(q)),linestyle='dashed',color='k') # creates a horizontal line in the plot at func(q)=0.5
    ax.axvline(x=(A-alpha*c)/2,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=0.5
    ax.set_xlim(0,)
    ax.set_ylim(0,np.max(func(q))*1.05)  # set a lower limit for y-axis at zero and max-limit at max profit.
    ax.grid()

    ax2.plot(qD, demand(qD),label='Demand')
    ax2.plot(qM,MR(qM), label='Marginal Revenue')
    ax2.legend(loc='upper right') # Place the graph descriptions in the upper right corner
    ax2.grid()
    ax2.axhline(y=c,color='k', label='Marginal Cost') # creates a horizontal line in the plot at func(q)=0.5
    ax2.axvline(x=(A-alpha*c)/2,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=0.5
    ax2.axhline(y=(A+c*alpha)/(2*alpha),linestyle='dashed',color='k')
    ax2.set_xlabel("Quantity produced q ")
    ax2.set_ylabel("Market Price p")
    ax2.set_xlim(0,A/2) # set x axis to go from zero to where MR(Q) intersect the x-axis
    ax2.set_ylim(0,np.max(demand(q))*1.05) # set y axis to go from zero to max of demand(q).
    ax2.set_title('The Market Equilibrium')
```

And the interactive figure is illustrated below.

```python
widgets.interact(interactive_figure,
    A=widgets.FloatSlider(description="A",min=0.1,max=10,step=0.1,value=4), #
    alpha=widgets.FloatSlider(description="alpha",min=0.1,max=5,step=0.1,value=2),
    c=widgets.FloatSlider(description="c",min=0,max=1.999,step=0.1,value=1)
);
```

    interactive(children=(FloatSlider(value=4.0, description='A', max=10.0, min=0.1), FloatSlider(value=2.0, descr…

# Extentions: Solving for market equilibrium in a duopoly setting

## Market Equilibrium with Cournot Competition

Consider the inverse demand funcion with identical goods
$$p(Q)=\frac{A}{\alpha}-\frac{1}{\alpha}\cdot Q=\frac{A}{\alpha}-\frac{1}{\alpha}\cdot(q_1+q_2)$$

where $q_1$ is firm 1's output and $q_2$ is firm 2's output. $Q=q_1+q_2$.

Both firms have identical cost-function $c(q_i)=c\cdot q_i$. So given cost and demand, each firm have the following profit function:
$$\pi_{i}(q_{i},q_{j})=p_i(q_i,q_j)q_i-c(q_i)$$,
$i,j\in\{0,1\},i\neq j$, which they seek to maximize.

As this is the standard Cournot problem with two firms competing in quantities, we know that in equilibrium both firms produces the same Cournot output level given by:
$$q_1^{C}=q_2^{C}=\frac{A-\alpha c}{3}$$

### Analytical Solution

We can use **sympy** to find an analytical expression for the market equilibrium/ the Cournot Nash Equilibrium, i.e. solving for the pair $(q_1^{C},q_2^{C})$ in which both firms play a best-response to the other firms equilibrium strategy. Hence
$$\max_{q_{i}} \pi_{i}(q_i,q_j^{\ast})=\max \left(\frac{A}{\alpha}-\frac{1}{\alpha}\cdot(q_i+q_j^{\ast})-c\right)q_i $$

```python
# Defining variables;
A = sm.symbols('A') # Constant in Q(p)
q1 = sm.symbols('q1') # Firm 1's output
q2 = sm.symbols('q2') # Firm 2's output
c = sm.symbols('c') # Contant cost
alpha=sm.symbols('alpha') # Slope coefficient in Q(p)

Pi1 = (A/alpha-1/alpha*(q1+q2))*q1-c*q1 # Firm 1's profit function

Pi2 = (A/alpha-1/alpha*(q1+q2))*q2-c*q2 # Frim 2's profit function

F1 = sm.diff(Pi1,q1) # Take the first order condition for firm 1
F2 = sm.diff(Pi2,q2) # Take the first order condition for firm 2
sm.Matrix([F1,F2]) # Prints the first order conditions
```

```python
Cq2 = sm.solve(F2,q2)[0] # Solves Firm 2's FOC for q2.
Cq2
```

```python
Cq1 = sm.solve(F1,q2)[0] # Solves Firm 1's FOC for q2.
Cq1
```

```python
CON=sm.solve(Cq1-Cq2,q1)[0] # In Eq Cq1=Cq2, so solve  Cq1=Cq2=0 for q1
CON
```

Given the standard symmetry argument, we know that both firms produce the same in equilibrium. Hence
$$q_1^{C}=q_2^{C}=\frac{A-\alpha c}{3}$$
as given above.

The total market quantity and price are found below.

```python
MCQ = 2*CON  #  market quantiy
# And the market price is given by;
MCP=(A-MCQ)*1/alpha
sm.Matrix([MCQ,MCP]) # Prints the market quantity and price
```

These can again by turned into python-functions to compare the analytical solution with the numerical solution

```python
CON_func = sm.lambdify((A,alpha,c),CON) # Cournot quantity
MCP_func = sm.lambdify((A,alpha,c),MCP) # Market price
```

### Numerical Solution

```python
def demand(q1,q2,b): # Define demand
    return A/alpha-1/alpha*(q1+b*q2) # b is in place to allow for potential heterogeneous goods.

def cost(q,c):
    if q == 0:
     cost = 0
    else:
     cost = c*q
    return cost
```

```python
def profit(q1,q2,c1,b): # Define a function for profits
    return demand(q1,q2,b)*q1-cost(q1,c1)
```

Define reaction functions.

As we know scipy has various methods to optimize function. However as they are defined as minimization problems, maximizing a function $f(x)$ is the same as minimzing $-f(x)$.

```python
def reaction(q2,c1,b):
    q1 = optimize.brute(lambda q: -profit(q,q2,c1,b), ((0,1,),)) # brute minimizes the function;
                                                                 # when we minimize -profits, we maximize profits
    return q1[0]
```

A solution method which can be used to solve many economic problems to find the Nash equilibrium, is to solve for the equilibirum as fixed point.

Hence we are looking for a fixed point in which the following is true.
$$\left(\begin{matrix}
           q_{1}^{\ast} \\
           q_{2}^{\ast}
         \end{matrix}\right)=\left(\begin{matrix}
           r_{1}(q_2^{\ast}) \\
           r_{2}(q_1^{\ast})
         \end{matrix}\right)
        $$

where $r_1(q_2)$ is firm 1's reaction-function to firm 2's production level and vice versa.

Numerically this can be solved by defining a vector function:
$$f(q)=\left(\begin{matrix}
           r_{1}(q_2^{\ast}) \\
           r_{2}(q_1^{\ast})
         \end{matrix}\right)$$
and solve for a point $q^{\ast}=(q_1^{\ast},q_2^{\ast})$ such that $f(q^{\ast})=q^{\ast}$

We then define a function defined as $x-f(x)$ and look for the solution $x^{\ast}-f(x^{\ast})=0$

```python
def vector_reaction(q,param): # vector parameters = (b,c1,c2)
    return array(q)-array([reaction(q[1],param[1],param[0]),reaction(q[0],param[2],param[0])])
```

```python
param = [1.0,1.0,1.0] # Specify the parameters (b,c1,c2)
q0 = [0.3, 0.3] # Initial guess for quantities
alpha=2
A=4
ans = optimize.fsolve(vector_reaction, q0, args = (param))
print(ans)
```

    [0.6666581 0.6666581]

```python
A = 4
alpha = 2
c = 1
print(f'analytical solution for Cournot quantity is: {CON_func(A,alpha,c):.2f}')
print(f'analytical solution for market price is: {MCP_func(A,alpha,c):.2f}')
print(f' Solution with fsolve for market quantity is: {ans}')

```

    analytical solution for Cournot quantity is: 0.67
    analytical solution for market price is: 1.33
     Solution with fsolve for market quantity is: [0.6666581 0.6666581]

And we see that the numerical solution for the market quantity is fairly close to the analytical solution at $q_1^{C}=q_2^{C}=\frac{2}{3}$

Below we illustrate the equilibrium quantities visually by plotting the two firms reaction functions/best-response functions. The equilibrium quantities is found in the point in which they intersect.

```python
# Define the expression whose roots we want to find

A = 4.0 # Specify the value of the Constant in the market demand function Q(p)
alpha = 2.0 # Specify the  slope coefficient in Q(p)
c = 1 # Specify the value of the constant cost 'c'

func1 = lambda q : 1/2*(A-alpha*c-q) # Defines the best-response function for firm 1using a lambda function.
func2 = lambda q : A-alpha*c-2*q

# Plot the profit function

q = np.linspace(0, 5, 200) # Return evenly spaced numbers over a specified interval from 0 to 2 .

plt.clf()
plt.plot(q, func1(q),'-', color = 'r', linewidth = 2)
plt.plot(q,func2(q),'-', color = 'b', linewidth = 2)
plt.title("Cournot Nash Equilibrium",fontsize = 15)
plt.xlabel("$q_1$",fontsize = 15)
plt.ylabel("$q_2$",fontsize = 15,rotation = 90)
plt.axvline(x=CON_func(A,alpha,c),ymin=0,ymax=1/3,linestyle='dashed',color='k') # creates a vertical line in the plot at  q=2/3
plt.axhline(y=CON_func(A,alpha,c),xmin=0,xmax=1/3,linestyle='dashed',color='k') # creates a horizontal line in the plot at  q=2/3

plt.annotate('$R_2(q_1)$', xy=(1,0.5),  xycoords='data', # here we define the labels and arrows in the graph
              xytext=(30, 50), textcoords='offset points', size = 20,
              arrowprops=dict(arrowstyle="->", linewidth = 2,
                              connectionstyle="arc3,rad=.2"),
              )
plt.annotate('$R_1(q_2)$', xy=(0.5,1),  xycoords='data', # here we define the labels and arrows in the graph
              xytext=(30, 50), textcoords='offset points', size = 20,
              arrowprops=dict(arrowstyle="->", linewidth = 2,
                              connectionstyle="arc3,rad=.2"),
              )
plt.xlim(0,2) # sets the x-axis
plt.ylim(0,2) # Sets the y-axis
```

We see that when both firms have symmetric cost $c_1=c_2=c$ and produce homogeneous goods, both firms produce the same in the Cournot Nash equilibrium. We see that both firms individually produce less than if they have monopoly power due to the small increase in competition in the market. Hence when no collusion is possible as this is a one period static problem, the total market output is larger than the monopoly outcome and the associated market price is lower. However assuming no externalities and the standard economic assumptions, the market outcome is still inefficient seen from a social planners perspective as it is not equal to the social optimum, where all firms produce in the point in which marginal costs equal the market price.

## Market Equilibrium with Betrand Competition with differentiated goods

Lastly we will investiate the market outcome in the duopoly setting with two firms is competing in prices rather than quanties. This competition type is called bertrand competion, and we will consinder the Betrand model with differentiated products and the standard Betrand Model of Duopoly with identical firms producing homogeneous products with the same cost-functions with constant marginal costs c.

The market demand function is the same given by
$$Q(p)=A-\alpha\cdot p$$

However from the perspective of firm i, the consumers demand for firm i's good is:
$$q_i(p_i,p_j)=A-p_i+b\cdot p_j$$, $i,j\in\{1,2\}, i\neq j$, where b indicates that the goods are imperfect substitutes.

The profit of firm i when choosing the price $p_i$ and firm j chooses the price $p_j$ is given by:
$$\pi_i(p_i,p_j)=q_i(p_i,p_j)[p_i-c]$$

And the price pair $(p_1^{\ast},p_2^{\ast})$ constitute a Nash equilibrium if, for each firm i, the choosen price $p_i^{\ast}$ solve the firms maximization problem, i.e.
$$\max_{0\leq p_i<\infty}\pi_i(p_i,p_j^{\ast})=\max_{0\leq p_i<\infty}\left[A-p_i+b\cdot p_j^{\ast}\right][p_i-c]$$

### Analytical Solution with differentiated goods

We can use **sympy** to find an analytical expression for the market equilibrium/ the Bertrand Nash Equilibrium, i.e. solving for the pair $(p_1^{B},p_2^{B})$ for which both firms play a best-response to the other firms equilibrium strategy.

As this is the betrand problem with differentiated goods, we know already that both firms will try to underbid eachother in prices. Hence in equilibrium, $p_1^{B}=p_2^{B}$ and it is assumed that each firm produce half of the  market demand, i.e.
$$p_1^{B}=p_2^B=\frac{A+c}{2-b}$$

```python
# Defining variables;
A = sm.symbols('A') # Constant in Q(p)
p1 = sm.symbols('p1') # Firm 1's price
p2 = sm.symbols('p2') # Firm 2's price
c = sm.symbols('c') # Contant cost
b = sm.symbols('b') # constant reflecting that the goods are differentiated
alpha=sm.symbols('alpha') # Slope coefficient in Q(p)

Pi1 = (A-p1+b*p2)*(p1-c) # Firm 1's profit function

Pi2 = (A-p2+b*p1)*(p2-c) # Firm2's profit function

F1 = sm.diff(Pi1,p1) # Take the first order condition for firm 1
F2 = sm.diff(Pi2,p2) # Take the first order condition for firm 2
sm.Matrix([F1,F2]) # Prints the first order conditions
```

We can then use the first order conditions to find the best-response functions by using sympy's solve function to solve $$F1=0$$ for $p_1$.

```python
BR1 = sm.solve(F1,p1)[0] # Solves Firm 1's FOC for p1.
BR2 = sm.solve(F2,p2)[0] # Solves Firm 2's FOC for p2.
sm.Matrix([BR1,BR2]) # Prints the best-response functions
```

However to solve the function in an easier way, we solve firm 2's FOC for $p_1$. Call this BR12. We know that both firm's FOC must hold in equilibrium. Hence the equilibrium price is found by solving $BR1=BR12\Leftrightarrow BR1-BR12=0$ which can be solved sympy's solve function:

```python
BR12=sm.solve(F2,p1)[0] # Solves Firm 2's FOC for p1.
MP=sm.solve(BR1-BR12,p2)[0] # In Eq p1=p2, so solve  BR1-BR12=0 for p2
MP

```

Hence both firms charge the price $p_1^{B}=p_2^{B}=-\frac{A+c}{b-2}=\frac{A+c}{2-b}$

Turned into a function

```python
MP_func = sm.lambdify((A,b,c),MP) # Market price

A = 4
b = 0.5 # parameter different from 1 to indicate imperfect substitutes
c = 1
print(f'analytical solution for Bertrand price is: {MP_func(A,b,c):.2f}')

```

    analytical solution for Bertrand price is: 3.33

### Numerical Solution

```python
A = 4
b=0.5
c = 1
B1 = 1/2*(A+b*p2+c)
B2 = 1/b*(2*p2-A-c)
SOL = optimize.fsolve(lambda p: 1/2*(A+b*p+c)-1/b*(2*p-A-c),0)
print(f' Solution with fsolve for Bertrand price is: {SOL:}')

```

     Solution with fsolve for Bertrand price is: [3.33333333]

Thus when the firms products are imperfect substitutes, the market price is still above the firms marginal cost. Hence the market outcome is pareto inefficient, which differs from the result from the standard Betrand model with homogeneous products.

## Market Equilibrium with Betrand Competition and homogeneous goods

The market equilibrium with betrand competition, homogeneous goods and identical firms has some interesting, nice properties. Most importantly, the equilibrium is pareto efficient equal to perfect market outcome.

The market demand function is the same given by
$$Q(p)=A-\alpha\cdot p$$

Both firm compete in prices, and seek to maximize the profit function:
$$\max_{p_i}\pi_i(p_i,p_j)=
\begin{cases}
Q(p)\cdot (p_i-c)  &p_j>p_i\\
\frac{Q(p)\cdot (p_i-c)}{2}  &p_i=p_j\\
0 &p_i>p_j
\end{cases}
$$
It is a standard assumption that the market is divided evenly between the two firms if the set the same price, but there is no reason why it couldn't be different in practice.

Both firms have the symmetric Best-Response functions:
$$BR_i(p_j)=
\begin{cases}
p_i=p_m  &p_j>p_m\\
p_i=p_j-\epsilon  &p_i>p_j>c\\
p_i=c &p_j<c
\end{cases}$$
where $p_m$ is the monopoly price found in the first problem.

Then with simple economic reasoning it can be shown/proven that the only strategies/set of prices $\{p_1^{B},p_2^{B}\}$ that can constitute a Nash Equilibrium is $(p_1^{B},p_2^{B})=(c,c)$. Because in all other cases at least one firm has an incentive to deviate. We will not prove this, but below we show it numerically.

So both firms will produce half of the total market output in equilibrium, i.e. both firms produce the monopoly output:
$$q^{B\ast}=\frac{A-\alpha\cdot c}{2}$$

### Numerical Solution

```python

def total_demand(p): # total demand Q(p)
    return A-alpha*p
```

```python

def profit(p1,p2,c1): # Define profit function depending p_1 and p_2 with af if, elif else statement"
    if p1 > p2:
        profits = 0 # firm 2 takes all the market
    elif p1 == p2:
        profits = 0.5*total_demand(p1)*(p1-c1) # The firms split the market in two
    else:
        profits = total_demand(p1)*(p1-c1) # firm 1 takes all the market
    return profits

def reaction(p2,c1):  # Reaction function
    if p2 > c1:
        reaction = c1+0.8*(p2-c1)
    else:
        reaction = c1
    return reaction
```

```python
def vector_reaction(p,param): # vector param = (c1,c2)
    return array(p)-array([reaction(p[1],param[0]),reaction(p[0],param[1])])

param = [2.0,2.0] # c1 = c2 =2
alpha=2
A=4
p0 = [0.5, 0.5] # initial guess: p1 = p2 = 0.5

Psol = optimize.fsolve(vector_reaction, p0, args = (param))
print(Psol) # Bertrand prices
```

    [2. 2.]

As should become clear from this little numerical demostration. The two firms price setting decision is in practical sense invariant to the shape of the demand curve - as long as it is downwards slopping. The two firms will regardless of the demand function try to underbid the other firm in terms of price as long as the market price is above the marginal cost c. Hence as long as there is positive profits to get in the market, both firm will try to get the entire market by setting a price just below the other. This proces will continue until the market price reach the firms marginal cost, which are assumed identical. Thus the betrand price solely depends on the value of the costs $c_1=c_2$, as the firms compete the market price $p^{B}$ down to $c_1=c_2$. Because only then, no firm has an incentive to deviate, i.e. the pair of prices ${(p_1^{B},p_2^{B}})={c}$ constitute a Nash equilibrium. This is also known as the bertrand paradox.

# Conclusion

We see that the assumption about the market structure have a critical impact on the market equilibrium. We have shown that under the standard assumptions, when there is only one firm in the market, which utilizes it's monopoly power, the market equilibrium output is inefficiently low and the equilibrium price is ineffciently high from a social welfare perspective. When the number of firms increases to two, we show that the market inefficiency decreases but at different degrees depending on competition type. If the firms compete in quantities (Cournot), the market output is still lower than the social optimum. However there is still some competition between the firms, which results in a lower market price and higher market output compared to the monopoly case. Lastly, we show that when the two firms compete in prices(Bertrand) the market equilibrium is pareto efficient. As both firms seek to undercut the other firm resulting in both firms asking a price equal to their marginal costs (assumed identical). Hence even though there are only two firms, the market equilibrium is efficient as it is identical to social optimum with a market price equal to the marginal costs. However when allowing for the two firms to produce different goods that are imperfect substitutes, both firms raise their prices above marginal cost. Thus they earn positive profit and the market outcome is once again inefficient.