**Competing in different settings**

In this project we consider 2 firms who compete in the same duopolistic market. We will look at three possible competition forms, which are characterized by

**Cournot**

- Firms compete in quantities, and decide upon these independently and simultaneously

- Firms profit maximize given the other's choice of quantity

- Both firms have market power, thus one's decision of output affects the price of the other's goods

**Bertrand**

- Firms compete in prices, and decide upon these simultaneously and as if they were in a perfect competition setting

- We assume that consumers seek the lowest price

**Stackelberg**

- Firms compete in quantities, and decide upon these sequentially.

- One firm is assumed to be the leader, and the other is the follower.

In the following we will analyse a sitaution with linear demand with $a=101$, $b=0.1$ and cost equal to 1 for all firms under all three situations.

```python
#Import packages
from scipy import optimize,arange
from numpy import array
import numpy as np
import matplotlib.pyplot as plt
import random
import sympy as sm
from math import *
%matplotlib inline
from IPython.display import Markdown, display
import pandas as pd
```

**Bertand**

First we wanna solve the case of Bertand competetion, i.e a sitatuion, where the firms compete in prices. Here we consider two companies who are selling differentiated products. The quantity each firm produces is dependent on the on it's own price $p_1$ and the competitor's price $p_2$.
$$x_1 = a - p_1 +bp_2$$
$$x_2 = a - p_2 + bp_1$$
The individual companies profits can be defined as:
$$\pi_1 = (p_1 - c_1)(a - p_1 +bp_2)$$
$$\pi_2 = (p_2 - c_2)(a - p_2 + bp_1).$$
For the following parts we assume that both companies have identical marginal costs and thus $$c_1 = c_2 = c.$$

```python
#Variables
a = sm.symbols('a')
p1 = sm.symbols('p1')
p2 = sm.symbols('p2')
c = sm.symbols('c')
b = sm.symbols('b')
alpha=sm.symbols('alpha')

#Frist we define the profit funtioncs for the two firms:
pi1 = (p1-c)*(a-p1+b*p2)
pi2 = (p2-c)*(a-p2+b*p1)
```

In order to get the equilibrium price we first have to determine the first order conditions (FOC) by maximizing the profit functions of the individual firms with repsect to the prices they offer.
$$\frac{\partial\pi_1}{\partial p_1}= a - 2p_1 + bp_2 + c=0$$
$$\frac{\partial\pi_2}{\partial p_2}= a - 2p_2 + bp_1 + c=0$$

By solving the first order conditions for the respective prices we arrive at the reaction functions.
$$p_1^{*} = \frac{a+ b*p_2 + c}{2}$$
$$p_2^{*} = \frac{a+ b*p_1 + c}{2}$$

By setting them equal, since it is the best repsonse for both firms to set their price equal to the competitior's price, we arrive at our equilibrium price:
$$p^{*} = \frac{-(a+c)}{b-2}$$

```python
#The first order conditions are then:
FOC1 = sm.diff(pi1, p1)
FOC2 = sm.diff(pi2, p2)

#Reactionsfuncions
RF1 = sm.solve(FOC1, p1)[0]
RF2 = sm.solve(FOC2, p2)[0]

#Equilibrium price
RF12 = sm.solve(FOC2,p1)[0]
EP1 = sm.solve(RF1-RF12,p2)[0]
print(EP1) #Equilibrium price both companies will charge
RF21 = sm.solve(FOC1,p2)[0]
EP2 = sm.solve(RF2-RF21,p1)[0]
```

    -(a + c)/(b - 2)

Using the lambdify feature from the sympy package we can now make a function to solve the bertrand competition for chosen values. In this case we set $a$ equal to 101, $b$ equal to 0.1 and $c$ equal to 1 (the numbers were decided on arbitrarily).

```python
#make it a function to solve it
bertrand = sm.lambdify((a,b,c), EP1)
price1=bertrand(101,0.1,1)
quantity1 = 101 - 0.9*bertrand(101,0.1,1)
profit1 = (bertrand(101,0.1,1)-1)*(101 - 0.9*bertrand(101,0.1,1))

#Defining a way to print the solution in a fancy way
def fancy(string):
    """ A function that prints strings in a fancy way
    args:
    string : a string

    returns : a string i an fancy way"""
    display(Markdown(string))

fancy(f'The equilibrium price is $p$ = {price1:.1f}, the equilibrium quantity is $x$ = {quantity1:.1f} and the profits for both firms are $π$ = {profit1:.1f}')
```

The equilibrium price is $p$ = 53.7, the equilibrium quantity is $x$ = 52.7 and the profits for both firms are $π$ = 2775.6

**Stackelberg**

We now turn to the Stackelberg, i.e a sequantiacal competion in quantities, thus one company is the leader and the other is a follower.

```python
#values
p = sm.symbols('p')
x1 = sm.symbols('x1')
x2 = sm.symbols('x2')

#Giving values to our parameters
a = 101
c = 1
b = 0.1
```

Each firm chooses a quantitiy $x_{1}, x_{2}\geq0$ where the cost of production is $c_{1}=c_{2}$. Each firm set the price $p = a-b\left(x_{1}+x_{2}\right)$. The profit function for firm $i$ is then written as $\pi_{i}\left(x_{i}\right)=p\left(x_{i}\right)-c_{i}x_{i}$ Assume firm 1 is the leader and 2 is the follower. The game is solved by backward induction.

We will first define the inverse demand function $p$:

```python
#Defining price
p = a - b*(x1+x2)
```

Then, we set up the profit function for firm 1 and firm 2:

```python
#Profit function of firm 1
profit_1 = p*x1 - c*x1
profit_2 = p*x2 - c*x2
```

We will say that firm 1 decides first on its output. Firm 2 observes the decision of firm 1 and decides how to reacht. We can solve the problem backwards and find optimal output of firm 2 for each possible output level of firm 1.
It means that we maximize the profit of firm 2 by differentiating it by $x_2$.

```python
#Differentiating the profit of firm 2 by q2

diff_profit_2 = sm.diff(profit_2, x2)
diff_profit_2
```

$\displaystyle - 0.1 x_{1} - 0.2 x_{2} + 100$

```python
#Finding optimal q2 by isolating q2 from the differentiation
opt_2 = sm.solve(diff_profit_2, x2)
opt_2[0]
```

$\displaystyle 500.0 - 0.5 x_{1}$

Then, we can substitute $q_2$ into the profit function of firm 1. Thanks to that, we have a single variable function.

```python
#Substituting q2 into the profit function of firm 1
profit_1_subs = profit_1.subs(x2,opt_2[0])
profit_1_subs
```

$\displaystyle x_{1} \left(51.0 - 0.05 x_{1}\right) - x_{1}$

Once again, we need to differentiate the profit of firm 1 by $q_1$ to find the optimal value of $q_1$.

```python
#Differentiating the profit of firm 1 by q1

diff_profit_1 = sm.diff(profit_1_subs, x1)
diff_profit_1
```

$\displaystyle 50.0 - 0.1 x_{1}$

```python
#Finding the optimal value of q1

opt_1 = sm.solve(diff_profit_1, x1)
opt_1[0]
```

$\displaystyle 500.0$

Now, we can find the optimal value of $q_2$.

```python
opt_2_subs = opt_2[0].subs(x1,opt_1[0])
opt_2_subs
```

$\displaystyle 250.0$

We can also find the equilibrium market price, profit of firm 1 and profit of firm 2:

```python
#Finding equilibrium market price
p = a - b*(opt_1[0] + opt_2_subs)

#Finding profit of firm 1
profit_1 = p*opt_1[0] - c*opt_1[0]

#Finding profit of firm 2
profit_2 = p*opt_2_subs - c*opt_2_subs

fancy(f'The equilibrium price is $p$ = {p:.0f}, and the profits for both firms are $π_1$ = {profit_1:.0f} and $π_2$ = {profit_2:.0f}')
```

The equilibrium price is $p$ = 26, and the profits for both firms are $π_1$ = 12500 and $π_2$ = 6250

We now want to make a figure that show how the beste responses relate.

```python
#Defining the x-axis
qu_1 = np.linspace(0, 1000, 1000)

#Defining the lambda functions for our best responses functions
BR2 = lambda qu_1 : 500 - qu_1/2
BR1 = lambda qu_1 : 50 - qu_1/10

#Plotting the figure
plt.figure(figsize=(15,10))
plt.plot(qu_1, BR2(qu_1), label = "Best Response of Firm 1")
plt.plot(BR1(qu_1),qu_1 , label = "Best Response of Firm 2")
plt.grid(True)
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.title('Best response functions')
plt.legend()

#To include a quantity of 0 for 1 firm, we set the axis like this
plt.xlim(0,1000) # sets the x-axis
plt.ylim(-100,600) # Sets the y-axis

plt.show()
```

**ADRIAN - this should be changed. This tells that when firm chooses a quantity of 500 in the first stage, firm 2 will choose a quantity half the size of firm 1 in second stage. In a case where firm 1 chooses to produce 1000 units, firm 2 will then produce 0.**

**Cournot**

For the Cournot case we are gonna set up some functions to solve the model in order to make an extention with n firms later on. For the solution with we have found inspiration in the following article: http://janboone.github.io/competition_policy_and_regulation/Collusion_Cournot/Collusion_Cournot.html

Consider the linear invese demand curve $p\left(x_{1},x_{2}\right)=a-x_{1}-bx_{j}$ where $b\in\left(0,1\right)$ is the elasticity of substitution between the goods, i.e. indicates that the goods are heterogenous for $0<b<1$.

We write the inverse demand:

```python
#Variables
a = sm.symbols('a')
x1 = sm.symbols('x1')
x2 = sm.symbols('x2')
c1 = sm.symbols('c1')
c2 = sm. symbols('c2')
c = sm.symbols('c')
b = sm.symbols('b')
pi = sm.symbols('pi')
```

```python
def price(x1,x2,a,b):
    """A function that computes price
    args:
    x1 : quantity for firm 1
    x2 : quantity for firm 2
    a :total demand
    b : elasticity of substitution

    returns a float
    """
    return a-b*(x1+x2)

```

Firms have constant marginal costs, where total costs are given by $c\left(x_{1,2}\right)=cx_{1,2}$

```python
def cost(x,c):
    """A function that computes total cost
    args:
    c : cost for firm i
    x : quantity for firm i

    returns a float
    """
    cost = c*x
    return cost
```

Firms maximize profits given the choice of the other firm such that $$\pi\left(x_{1},x_{2}\right)=p_{1}\left(x_{1},x_{2}\right)x_{1}-c\left(x_{1}\right)$$

```python
def profit(x1,x2,c1,a,b):
    """A function that computes profit for firm 1
    args:
    x1 : quantity for firm 1
    x2 : quantity for firm 2
    c1 : cost for firm 1
    a : total demand
    b : elasticity of substitution

    returns a float
    """
    return price(x1,x2,a,b)*x1-cost(x1,c1)
```

In a Cournot setting, firm $1$ chooses $x_{1}$ taking $x_{2}$ as given. Thus, the Nash equilibrium for $x_{1}^{*},x_{2}^{*}$ holds for $$x_{1}^{*}=\text{arg}\underset{x_{1}}{\text{ max }}\pi\left(x_{1},x_{2}^{*}\right)\;\;\;\;\;\;\text{for each }i\neq j$$ Hence, taking the first order condition to find the Nash equilibrium for $x_{1}^{*},x_{2}^{*}$ yields $$\frac{\partial\pi\left(x_{1},x_{2}\right)}{\partial x_{1,2}}=0$$

We differentiate the profit function. First we need to make it into a expression in order to use `SymPy.diff`

```python
#Profit for firm i
pi_1 = profit(x,x2,c1,a,b) #taking the defined function from above

#FOC is then
foc1 = sm.diff(pi_1,x1)
foc1
```

$\displaystyle a - b x_{1} - b \left(x_{1} + x_{2}\right) - c_{1}$

```python
#Profit for firm 2
pi_2 = profit(x2,x1,c2,a,b)

#FOC is then
foc2 = sm.diff(pi_2,x2)
foc2
```

$\displaystyle a - b x_{2} - b \left(x_{1} + x_{2}\right) - c_{2}$

We now set this equal to 0:

```python
#For firm 1
top1 = sm.solve(sm.Eq(foc1,0),x1)[0]
top1
```

$\displaystyle \frac{a - b x_{2} - c_{1}}{2 b}$

```python
#For firm 2
top2 = sm.solve(sm.Eq(foc2,0),x2)[0]
top2
```

$\displaystyle \frac{a - b x_{1} - c_{2}}{2 b}$

Thus, we are able to find the optimal production level for a firm $1$ given the choice of $x_{2}$ by creating the function 'BR'. In this case, we use the function `fmin`. This allows one to look for the global maximum of $\pi$. Likewise, we keep in mind that `optimize` is bound to minimize problems, and since we want to maximize the profit function, we then minimize $-\pi\left(x_{i},x_{j}\right)$.

```python
def BR(x2,c1,b,a):
    """A function that computes the optimal quantity
    args
    x2 : quantity for firm 2
    c1 : cost for firm 1
    b : elasticity of substitution
    a : total demand

    returns a float
    """
    x1 = optimize.fmin(lambda x: -profit(x,x2,c1,a,b), 0.1, disp=0)

    return x1[0]
```

Now, we must find an equilibrium for price and quantity given their best response. For this reason, consider the vector function $f\left(x_{1},x_{2}\right)$ which takes firm $1$'s best response $r_{1}\left(x_{2}\right)$ to $2$ 's production level $x_{2}$ into consideration. We can write this as $$f\left(x_{1},x_{2}\right)=\left[\begin{array}{c}
r_{1}\left(x_{2}^{*}\right)\\
r_{2}\left(x_{1}^{*}\right)
\end{array}\right]$$

We are then looking for the quantities produced is equal to best response given what the other firm produce, i.e the point where:

$$\left(\begin{array}{c}
x_{1}^{*}\\
x_{2}^{*}
\end{array}\right)=\left(\begin{array}{c}
r_{1}(x_{2}^{*})\\
r_{2}(x_{1}^{*})
\end{array}\right)$$

Thus, we create a function 'vec_reaction' given by the difference between an array of the quantaites subtracted by the best response functions and pass $b,a, c_{i},c_{j}$ as a vector of parameters.

```python
def vec_reaction(x,param):
    """A function that computes the difference between the quantities and the beste response functions
    args
    x : quantaties
    param : list of parameters
    """
    return array(x)-array([BR(x[1],param[2],param[0],param[1]),BR(x[0],param[3],param[0], param[1])])
```

Now, moving on to find the actual Cournot equilibrium, we are able to set the values of each paramter. Using the `fsolve` from `scipy.optimize` to find $x$. Also, giving `fsolve` the inital guess of the quantaites, while passing the paramater vector 'param' as arguments 'args' to 'fsolve', this yields:

```python
param = [0.1,101,1,1] #parameters as with the other models
x0 = [0.1, 0.1] #initial guess

EQ = optimize.fsolve(vec_reaction, x0, args = (param))

fancy(f'The Nash equilibrium quantity for both firms are $x^* =[x_1^*,x_2^*]$ = [{EQ[0]:.0f},{EQ[1]:.0f}] where the equilibrium price for both firms are $p^*$ = {price(EQ[0],EQ[1],101,0.1):.0f} with a profit of $\pi^*$= {profit(EQ[0],EQ[1],1,101,0.1):.0f}')

```

The Nash equilibrium quantity for both firms are $x^* =[x_1^*,x_2^*]$ = [333,333] where the equilibrium price for both firms are $p^*$ = 34 with a profit of $\pi^*$= 11111

We now want to show, how the quantity produced changes, when the cost of firm 1 changes.

```python
range_c = np.linspace(1,0,20)
range_x = [optimize.fsolve(vec_reaction, x0, args = ([0.1,101, c, 1])) for c in range_c]

plt.style.use('seaborn')
plt.clf()
plt.plot(range_c,array(range_x)[:,0], label = '$x_1$')
plt.plot(range_c,array(range_x)[:,1], label = '$x_2$')
plt.xlabel('$c_1$')
plt.ylabel('$x$')
plt.title('Quantity produced by each firm as function of cost for firm 1')
plt.legend()
plt.show ()
```

In the figure, we see, that as the cost of firm 1 goes up, the quantity produced by firm 1 falls, while the quantity produced by firm 2 goes up.

**Compare all three cases**

We have now tried to solve a situation with two firms competing with three different models, giving them the same parameter values. This leads to the following results:

```python
fancy(f'For Bertrand: The equilibrium price is $p$ = {price1:.0f}, the equilibrium quantity is $q$ = {quantity1:.0f} and the profits for both firms are $π$ = {profit1:.0f}')
fancy(f'For Stackelberg: the equilibrium price is $p$ = {p:.0f}, and the profits for both firms are $π_1$ = {profit_1:.0f} and $π_2$ = {profit_2:.0f}')
fancy(f'For Cournot: the equilibrium equilibrium quantity for both firms are $x^* =[x_1^*,x_2^*]$ = [{EQ[0]:.0f},{EQ[1]:.0f}] where the equilibrium price for both firms are $p^*$ = {price(EQ[0],EQ[1],101,0.1):.0f} with a profit of $\pi$= {profit(EQ[0],EQ[1],1,101,0.1):.0f}')

```

For Bertrand: The equilibrium price is $p$ = 54, the equilibrium quantity is $q$ = 53 and the profits for both firms are $π$ = 2776

For Stackelberg: the equilibrium price is $p$ = 26, and the profits for both firms are $π_1$ = 12500 and $π_2$ = 6250

For Cournot: the equilibrium equilibrium quantity for both firms are $x^* =[x_1^*,x_2^*]$ = [333,333] where the equilibrium price for both firms are $p^*$ = 34 with a profit of $\pi$= 11111

**Cournot with N > 2 firms**

In order to do an extention to one of the models, we now want to investigate the case of N > 2 firms. First we set up the situation theoretically and use Python to show this. Afterwards we will use Python to solve the model.
In the following theoretical set-up we will follow the set-up in this note, adapting it to our situation: http://people.exeter.ac.uk/dgbalken/micro07/CournotHandout.pdf

Following the notation from ealier we now add:

- Firm $i$'s output: $x_i$

- Total output in in industry: $X=x_1+x_2+...+x_n$

- The opponent's output: $x_{-i} = X-x_i$

Each firm will then max profit given the expectation of $x_{-i}$:

$$\Pi_i(x_{-i},x_i)=(p(x_{-i}+x_i)-c_i)x_i$$

as with the n = 2 case we use the invese demand funcktion in a linear case:

$$p = a-b(x_{-i}+x_i)=a-bX$$

The FOC of is then:
$$\frac{\partial \Pi_i}{\partial x_i} = \frac{\partial p}{\partial x_i} \cdot x_i + p-c_i = 0$$

where $\frac{\partial p}{\partial x_i} = -b$ giving us

$$\frac{\partial\Pi_i}{\partial x_i} : -bx_i + p-c_i = 0$$

inserting the inverse demand function:

$$\frac{\partial\Pi_i}{\partial x_i} : -bx_i + (a-b(x_{-i}+x_{i}))-c_i = 0$$

solving for $x_i$ to get the reaction function for firm $i$ given its expectation for the amount the rest of the firms produce:

$$x_i(x_{-i}) = \frac{a-c}{2b}-\frac{1}{2}x_{-i}$$

which is the case for all firms $i=1,..,n$. We have $X=x_i+x_{-i}$ so the FOC found earlier can be rewritten to:
$$ -bx_i + (a-bX-c_i) = 0$$

which for the n firms:
$$-bX + n(a-bX)-n\bar c = 0$$

where we use the average cost level: $\bar c = \frac{c_1+c_2+...+c_n}{n}$

we can now rewrite this and get the quantity produced and the price in the market:

$$bX + n(bX) = n(A-\bar c)$$

$$(n+1)bX  = n(A-\bar c)$$

Quantity is then:

$$X = \frac{n}{n+1} \frac{a-\bar c}{b}$$

Price is:
$$p = a-bX = a-b[\frac{n}{n+1} \frac{a-\bar c}{b}] = \frac{1}{n+1}a + \frac{n}{n+1}\bar c$$

Using the found functions we can now try to set a program that is able to solve cases with N firms.

```python
def Round(a):
    """A function that prints floats rounded as strings
    args:
    a : a float

    returns a string
    """
    return str("{:.2f}".format(a))
```

First we define a function, that calculates the prices $p = \frac{1}{n+1}a + \frac{n}{n+1}\bar c $ rewriting using the definition of the average cost level gives: $p= \frac{1}{n+1}a + \frac{c_1+c_2+...+c_n}{n+1}$ which we use in our function.

```python
def price_eq(a, n, c):
    """A function that computes price
    args:
    a : total demand
    n : number of firms
    c : cost of firm i

    returns a float
    """

    price = max((a+sum(c))/(1+n), 0)
    return price
```

We now want to define a function that returns the quantity produced for firm $i$.
this is given by:
$$x_i = \frac{a-bX-c_i}{b}=\frac{a-c_i}{b}-\frac{n}{n+1}\frac{a-\bar c}{b} = \frac{1}{n+1}\frac{a}{b}+ \frac{n(\bar c -c_i)-c_i}{(n+1)b} = \frac{1}{n+1}\frac{1}{b}[a- n(\bar c -c_i)-c_i]$$

use that $\bar c = \frac{c_1+c_2+...+c_n}{n}$
$$x_i = \frac{1}{n+1}\frac{1}{b}[a+ (n-1)c_i-(c_1+c_2+...+c_n-c_i)]$$
which we now use to define a function, that returns the quantity:

```python
def xi_eq(a, b, n, c, i): #Returns equilibrium quantity for firm i
    """A function that computes quantity for firm i
    args:
    a : total demand
    b : elasticity of substitution between the good
    n : number of firms
    c : cost of firm i
    i : the number of the firm

    Returns a float"""

    x_i = (1/b)*((a+(n-1)*c[i]-sum(c))/(1+n))
    return x_i
```

We can then define a function, that returns the quantity for the entire number of firms

```python
def X_eq(a, b, n, c):
    """A function that computes quantity for the total amount of firms
    args:
    a : total demand
    b : elasticity of substitution between the good
    n : number of firms
    c : cost of firm i

    returns a float
    """

    X = []
    for i in range(0, n, 1):
        X.append(xi_eq(a, b, n, c, i))

    return sum(X)
```

Based on the assumtion that $$c_i = \bar c = c $$ the profit for firm $i$ can be found by:
$$\Pi_{i}=(p-c)x_{i}=\left(\frac{a+nc}{n+1}-c\right)\frac{1}{n+1}\frac{1}{b}(a-c)$$

Without identical marginal costs $$c_i \neq c $$ $$\bar c \neq c$$ the profit for firm $i$ would be:
$$\Pi_{i}=(p-c_i)x_{i}=\left(\frac{a+n\bar c}{n+1}-c_i\right)\frac{1}{n+1}\frac{1}{b}(a+n(\bar c - c_i)-c_1) = \frac{1}{(n+1)^2 * b}(a+n(\bar c - c_1)-c_i)^2$$

By maximizing the previously derived profit function we find the equilibrium profit.

```python
def profit_eq(a, b, n, c, i):
    """A function that computes profit for firm i and sets profit to zero, if there is no production
    args:
    a : total demand
    b : elasticity of substitution between the good
    n : number of firms
    c : cost of firm i
    i : the number of the firm

    returns a float
    """

    if xi_eq(a, b, n, c, i) == 0:
        profit = 0
    else:
        profit = max((1/b)*(((a-n*c[i]+(sum(c)-c[i]))/(1+n))**2), 0)

    return profit

```

We are now ready to define a function, that can solve a general case of cournot competetion with n firms. We choose the the cost for each firm will be drawn from a uniform destribution using random.uniform, given it a seed number such that we can compare with the solution we later solve with Python. Note that adding a seed number means, that every firms will get the same random drawn cost values. This is done in order to compare with the solution later solved with Python.

```python
def EQ_N(a, b, n, c_low, c_high,seed):
    """A function that draws number of firms and compute quantaties, price and average profit
    args:
    a : total demand
    b : elasticity of substitution between the good
    N : the maximal number of firms that can be drawn
    c_low : the lower bound of the interval for the cost
    c_high : the upper bound of the interval for the cost

    Returns a float"""

    #drawing of costs for each firm:
    c = []
    np.random.seed(seed)
    c = np.random.uniform(c_low,c_high,size=n)
    #the price:
    price = price_eq(a, n, c)

    #quantities and profits
    xi_list = []
    profiti_list = []

    #the quantity for the entire number of firms
    X = X_eq(a, b, n, c)

    return X, price, c

```

First we solve with 1000 firms and use the same parameters as with the other three models:

```python
a = 101
b = 0.1
c_low = 1
c_high = 5
seed = 1986
X,price, c_vec = EQ_N(a,b,1000,c_low,c_high,seed)
print('\n The number of firms are 1000 ' +
                  '\n The total amount produced is ' + Round(X) +',' + ' while the price is ' + Round(price) +
                   '\n')

```

     The number of firms are 1000
     The total amount produced is 978.80, while the price is 3.12

**Quantity and pice as functions of n**

We now want to show how the amout produced depends on the number of firms n. We therefor create a list with a range for n:

```python
#list with n values
n_vec = list(range(100,1000))

#array with solutions for different n
output_vec1 = array([EQ_N(a,b,n,c_low,c_high,seed) for n in n_vec])

#plot the the  quantity as function of number of firms
fig,ax = plt.subplots()
ax.plot(n_vec,output_vec1[:,0], label = '$X$')
ax.title.set_text('Quantatity as a function of number of firms')
plt.legend()
plt.show()

```

Furthermore we can show how the price varies with number of firms n:

```python
#define solution in order to show price for different n
def solution(a,b,c_vec):
    n= len(c_vec) #the number of firms will be equal to the length of the c vector
    q = n/(n+1) * (a-np.mean(c_vec))/b
    p = a-b*q
    return q,p

output_vec = array([solution(a,b,c_vec) for c_vec in output_vec1[:,2]])

#plot the price as function of number of firms
plt.style.use('seaborn')
plt.clf()
plt.plot(n_vec,output_vec[:,1], label = '$price$')
plt.title('Price as a function of number of firms')
plt.legend()
plt.show()
```

**Solving with python**\
Up untill now we have solved the model analytically and then used python to show the results. However the model can also be solved using python, which will be done in the following section:

First we set up the same number og firms and parameters values as in the previous solution:

```python
N = 1000 # number of firms
#np.random.seed(1986)
c_low = 1
c_high = 5
c_vec = np.random.uniform(c_low,c_high,size=N)
```

We then use the profit function: $\Pi_i = (p(x_{-i}+x_{i})-c_i)x_i$ with $p=a-b(x_{-i}+x_i)$ to find the foc with sympy:

```python
c_i = sm.symbols('c_i')
x_i = sm.symbols('x_i') # x for firm i
x_minus = sm.symbols('x_{-i}') # x for the the opponents

#The profit of firm 1 is then:
Pi_i = x_i*((a_1-b_1*(x_i+x_minus))-c_i)

#giving focs:
foc = sm.diff(Pi_i,x_i)
foc

```

$\displaystyle a - b x_{i} - b \left(x_{i} + x_{-i}\right) - c_{i}$

In order to use this in our solutionen, we rewrite $x_{i}+x_{-i} = \sum x_{i}$ using np.sum and then define a function for the foc:

```python
def focs(a,b,x_vec,c_vec):
    """A function that defines the foc for firm i
     args:
      a : total demand
      b : elasticity of substitution between the good
      c_vec : a vector with the cost for the firms
      x_vec : the quantatiy of firm i
     """
    # use the foc from the sympy.diff
    return -b*x_vec+a-b*np.sum(x_vec)-c_vec
```

We are now ready to set up a function that solves the case with N firms:

```python
def solve(a,b,c_vec):
    """A function that solve the cournot model with N firms
     args:
    a : total demand
    b : elasticity of substitution between the good
    c_vec : a vector with the cost for the firms

    Returns two floats
    """
    obj = lambda x_vec : focs(a,b,x_vec,c_vec)

    res = optimize.root(obj, x0 = [90 for n in range(N)])

    x_vec_star = res.x
    Q = np.sum(x_vec_star)

    # Use inverse demand to find price
    p = a- b*Q
    return Q, p
```

We then use the profit function: $\Pi_i = (p(x_{-i}+x_{i})-c_i)x_i$ with $p=a-b(x_{-i}+x_i)$ to find the foc with sympy:

```python
#Choose the same parameters as in previous models
a = 101
b = 0.1
X,p = solve(a,b,c_vec)
print('\n The number of firms are 1000 ' +
                  '\n The total amount produced is ' + Round(Q) +',' + ' while the price is ' + Round(p) +
                   '\n')
```

     The number of firms are 1000
     The total amount produced is 978.80, while the price is 3.12

```python
#Compare with the solution from the analytical solution
X,price, c_vec = EQ_N(a,b,1000,c_low,c_high,1986)
print('\n The number of firms are 1000 ' +
                  '\n The total amount produced is ' + Round(X) +',' + ' while the price is ' + Round(price) +
                   '\n')
```

     The number of firms are 1000
     The total amount produced is 978.80, while the price is 3.12

We see, that we get the same result for both the analytical solution and the solution using Python to solve.

```python
#my try with the figure
#list with n values
#n_vec = list(range(100,1000))

#array with solutions for different n
#output_vec1 = array([solve(a,b,c_vec) for N in n_vec])

#plot the the  quantity as function of number of firms
#fig,ax = plt.subplots()
#ax.plot(n_vec,output_vec1[:,0], label = '$X$')
#ax.title.set_text('Quantatity as a function of number of firms')
#plt.legend()
#plt.show()

```

**Conclusion**

In this paper we examine two firms who have the possibility of competing in Bertrand, Stackelberg and Cournot. We use the same parameter values for all competition types, and find that the firms benefit the most from competing in a Cournot setting as this yields the highest total profits of XX. This is intuitive, as in contrast to Bertrand, the firms will undercut each other until their price is equal to their marginal costs to obtain efficiency. Therefore, this setting yields a price as in perfect competition, whereas Cournot yields an oligopoly equilibrium.

As Cournot yields the best outcome in terms of profits for two firms, we extend this setting for $n$ firms. We find that for $n\rightarrow\infty$, the totalt amount produces increases, while the price falls because competing becomes more intense as the competition setting moves from oligopoly to infinitely many firms competing in quantities.

```python

```


# Taylor problem 1.39

last revised: 07-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

The goal of this notebook is to practice Python while considering some visualizations of problem 1.39 to see how they might help check the results, interpret the behavior, or suggest new ideas on how to extend the problem.  Suggestions are very welcome! We'll use the notebook Matplotlib backend for Jupyter (`%matplotlib notebook`) to allow for reading points off the plot.

**Problem statement:** A ball is thrown with initial speed $v_0$ up an inclined plane.  The plane is inclined at an angle $\phi$ above the horizontal, and the ball's initial velocity is at an angle $\theta$ above the plane.  Choose axes with $x$ measured up the slope, $y$ normal to the slope, and $z$ across it.  Write down Newton's second law using these axes and find the ball's position as a function of time.  Show that the ball lands a distance

$
\begin{align}
    R = \frac{2 v_0^2 \sin\theta \cos(\theta+\phi)}{g \cos^2\phi}
\end{align}
$

from its launch point.  Show that for given $v_0$ and $\phi$, the maximum possible range up the inclined plane is

$
\begin{align}
   R_{\textrm{max}} = \frac{v_0^2}{g(1+\sin\phi)}  \;.
\end{align}
$

### Gameplan:
* Accept the formulas for $R$ and $R_\textrm{max}$ as given and make plots.  Interpret.
* Use a fixed value of phi first, then loop through phi.
* Add formatting to plots (and widgets!).
* Check the formulas using sympy.
* Solve the ODE numerically and make some spot comparisons to the formulas.

### Make plots of $R$ and $R_\textrm{max}$

We are given expressions for $R$ and $R_\textrm{max}$, so we'll first turn them into functions.  Then make some plots.

```python
import numpy as np
from numpy import pi
```

```python
def Range_R(theta, phi, g=9.8, v0=1.0):
    """
    Distance R up the plane given angles theta and phi.
    Given in Taylor problem 1.39.
    """
    return (2.*v0**2 * np.sin(theta) * np.cos(theta+phi))/(g*np.cos(phi)**2)
```

```python
def Rmax(phi, g=9.8, v0=1.0):
    """
    Maximum distance up the plane for fixed angle phi.
    Given in Taylor problem 1.39.
    """
    return v0**2/(g*(1. + np.sin(phi)))
```

```python
# set up for plotting but now using the notebook backend
%matplotlib notebook
import matplotlib.pyplot as plt
import matplotlib as mpl

# The following can be uncommented if we want to adjust the font size.
plt.rcParams.update({'font.size': 10})
```

```python
# Make a plot of range R versus theta for several phi values
phi_vec = [0., pi/8., pi/4., 3.*pi/8.]
theta = np.arange(0,pi/2.,0.01)

# Label the figure and axis with _R to distinguish it
fig_R = plt.figure(figsize=(3,3))
ax_R = fig_R.add_subplot(1,1,1)
ax_R.set_title('Distance to landing')
ax_R.set_xlabel(r'$\theta$')
ax_R.set_ylabel(r'$R(\theta,\phi)$')

# start with phi=0 to see if it looks right
ax_R.plot(theta, Range_R(theta,phi=0.))
fig_R.tight_layout()  # make the spacing of subplots nicer

```

    <IPython.core.display.Javascript object>

Point to the curve to read off coordinates and try out the various controls (hover over a button to reveal a tooltip).

Remember that this is not a trajectory; each trajectory is just one point on this curve, specified by $\theta$ and $\phi=0$.  Does the shape and features of the curve (e.g., symmetry about the peak) agree with your intuition?  Can you prove (or verify) it should be that way from the analytic formula?

It's awkward to work in radians when our intuition (mine, at least) is better in degrees.  Let's define some functions to convert (no doubt there are built-in functions, but these are easy and good practice for us).

```python
def rad_to_deg(theta_rad):
    """Take as input an angle in radians and return it in degrees."""
    return 180./np.pi * theta_rad

def deg_to_rad(theta_deg):
    """Take as input an angle in degrees and return it in radians."""
    return np.pi/180. * theta_deg

```

Now make a plot with $\theta$ in degrees with a list of $\phi$ values, specified in radians but converted to degrees for the plot legend.

```python
# Now make a plot of range R versus theta for several phi values
phi_vec = [0., pi/8., pi/4., 3.*pi/8., 0.99*pi/2]
theta = np.arange(0,pi/2.,0.01)
theta_deg = rad_to_deg(theta)

fig_R = plt.figure(figsize=(4,4))
ax_R = fig_R.add_subplot(1,1,1)
ax_R.set_title('Distance to landing')
ax_R.set_xlabel(r'$\theta$ (degrees)')
ax_R.set_ylabel(r'$R(\theta,\phi)$')

#ax_R.plot(theta, Range_R(theta,phi=0.))

ax_R.set_ylim(0.,0.15)
for phi in phi_vec:
    label_phi = fr'$\phi = {rad_to_deg(phi):.1f}\!^\circ$'
    ax_R.plot(theta_deg, Range_R(theta,phi), label=label_phi)
ax_R.legend()

fig_R.tight_layout()  # make the spacing of subplots nicer

```

    <IPython.core.display.Javascript object>

Hmmm, it would be nice to have a picture of the incline next to this.  How would we do that?

```python
# make a plot of range R versus theta for several phi values
phi_vec = [0., pi/8., pi/4., 3.*pi/8., 0.99*pi/2]
theta = np.arange(0,pi/2.,0.01)
theta_deg = rad_to_deg(theta)

fig_R = plt.figure(figsize=(6,3))
ax_R = fig_R.add_subplot(1,2,1)
ax_R.set_title('Distance to landing')
ax_R.set_xlabel(r'$\theta$')
ax_R.set_ylabel(r'$R(\theta,\phi)$')

ax_R.set_ylim(0.,0.15)
for phi in phi_vec:
    label_phi = fr'$\phi = {rad_to_deg(phi):.1f}\!^\circ$'
    ax_R.plot(theta_deg, Range_R(theta,phi), label=label_phi)
ax_R.legend()

# now add another subplot with the inclined plane.
ax_plane = fig_R.add_subplot(1,2,2)
ax_plane.set_title('Inclined plane')
ax_plane.set_xlim(0.,1.1)
ax_plane.set_ylim(0.,1.1)
for phi in phi_vec:
    label_phi = fr'$\phi = {rad_to_deg(phi):.1f}\!^\circ$'
    x_pts = [0., 1.]
    y_pts = [0., np.tan(phi)]
    ax_plane.plot(x_pts, y_pts, label=label_phi, lw=2)
ax_plane.axis('off')
ax_plane.legend()

fig_R.tight_layout()  # make the spacing of subplots nicer

```

    <IPython.core.display.Javascript object>

Ok, now we can look at the $R(\theta,\phi)$ plot and make observations, which we can then try to back-up from the formula. **Try to answer these questions.**
* Can you generalize the argument about symmetry to any $\phi$?
* What are the constraints on $R$ from the figure?  Do they match the formula?
* Do the changes in the curves with increasing $\phi$ make sense?  Can they be extracted from the formula?

Now plot the $R_{max}$ formula. **Evaluate whether it is consistent with the plots of R.**

```python
# make a plot of R_max versus phi
fig_Rmax = plt.figure(figsize=(3,3), num='Taylor 1.39')
ax_Rmax = fig_Rmax.add_subplot(1,1,1)
ax_Rmax.set_title(r'Maximum Distance given $\phi$')
ax_Rmax.set_xlabel(r'$\phi$ (degrees)')
ax_Rmax.set_ylabel(r'$R_{max}$')

phi = np.arange(0,pi/2.,.1)
ax_Rmax.plot(rad_to_deg(phi), Rmax(phi))

fig_Rmax.tight_layout()  # make the spacing of subplots nicer
```

    <IPython.core.display.Javascript object>

**Spotcheck some points on this curve against the previous graph of $R$.  What should you compare?**

### Widgetizing with ipywidgets and interact

The widgets don't seem to play well with the notebook back end, so we'll switch to `%matplotlib inline`.  You may need to restart the kernel.

```python
%matplotlib inline

from ipywidgets import interact, fixed
import ipywidgets as widgets

# A simple function evaluation (all in radians at this point).
interact(Range_R, theta=np.pi/4., phi=(0,np.pi/2), g=fixed(9.8));
```

    interactive(children=(FloatSlider(value=0.7853981633974483, description='theta', max=2.356194490192345, min=-0…

```python
def plot_range_R_versus_theta(phi_deg=0):
    """Make a plot of range R versus theta for given phi values in degrees"""

    phi_rad = deg_to_rad(phi_deg)
    theta_deg = np.arange(0,90.,0.1)
    theta_rad = deg_to_rad(theta_deg)
    range_R = Range_R(theta_rad,phi_rad)

    fig_R = plt.figure(figsize=(6,3), num='Taylor 1.39')
    ax_R = fig_R.add_subplot(1,2,1)
    ax_R.set_title('Distance to landing')
    ax_R.set_xlabel(r'$\theta$')
    ax_R.set_ylabel(r'$R(\theta,\phi)$')

    label_phi = fr'$\phi = {phi_deg:.1f}\!^\circ$'
    ax_R.plot(theta_deg, range_R, label=label_phi)
    ax_R.set_ylim(bottom=0., top=1.1*range_R.max())  # set limit *after* plotting to get auto-scaling
    ax_R.legend()

    ax_plane = fig_R.add_subplot(1,2,2)
    ax_plane.set_title('Inclined plane')
    ax_plane.set_xlim(0.,1.1)
    ax_plane.set_ylim(0.,1.1)
    label_phi = fr'$\phi = {phi_deg:.1f}\!^\circ$'
    x_pts = [0., 1.]
    y_pts = [0., np.tan(phi_rad)]
    ax_plane.plot(x_pts, y_pts, label=label_phi, lw=2)
    ax_plane.axis('off')
    ax_plane.legend()

    fig_R.tight_layout()  # make the spacing of subplots nicer

interact(plot_range_R_versus_theta, phi_deg=(0.,90.));
```

    interactive(children=(FloatSlider(value=0.0, description='phi_deg', max=90.0), Output()), _dom_classes=('widge…

```python
# to avoid the jiggling and do some formatting
phi_deg_widget = widgets.FloatSlider(min=0., max=90.0, step=0.1, value=0.,
                                     description=r'$\phi$ in degrees',
                                     readout_format='.0f',
                                     continuous_update=False
                                    )
interact(plot_range_R_versus_theta, phi_deg=phi_deg_widget);
```

    interactive(children=(FloatSlider(value=0.0, continuous_update=False, description='$\\phi$ in degrees', max=90…

### Ok, how about using sympy?

See if we can reproduce the algebra of solving for final t, x, and y.

```python
import sympy as sy
theta, phi, v0, m, g = sy.symbols('theta phi v0 m g')
x, y, t = sy.symbols('x y t')
half = sy.Rational(1,2)
```

```python
tf = sy.solve(v0*sy.sin(theta)*t - half*g*sy.cos(phi)*t**2, t)
tf
```

    [0, 2*v0*sin(theta)/(g*cos(phi))]

```python
x = v0*sy.cos(theta)*t - half*g*sy.sin(phi)*t**2
xf = x.subs(t,tf[1])
R = sy.trigsimp(xf)
R
```

    2*v0**2*sin(theta)*cos(phi + theta)/(g*cos(phi)**2)

```python
thetamax = sy.solve(sy.trigsimp(sy.diff(R,theta)),theta)
thetamax
```

    [-phi/2 + pi/4, -phi/2 + 3*pi/4]

```python
Rmax = R.subs(theta,thetamax[0])
sy.simplify(Rmax)
```

    v0**2*(-sin(phi) + 1)/(g*cos(phi)**2)

### Ok, now as a differential equation

We'll use odeint for simplicity.  Treat it as coupled 2nd-order differential equations for $x(t)$, $y(t)$, $v_x(t)$, and $v_y(t)$:

$
\begin{align}
   \frac{d}{dt}\left(\begin{array}{c}
                          \mathbf{x} \\
                          \mathbf{v}
                      \end{array}\right)
               = \left(\begin{array}{c}
                          \mathbf{v} \\
                          \mathbf{F}/m
                       \end{array}\right)
      \qquad \Longrightarrow \qquad
   \frac{d}{dt}\left(\begin{array}{c}
                          x \\
                          y \\
                          v_x \\
                          v_y
                      \end{array}\right)
               = \left(\begin{array}{c}
                          v_x \\
                          v_y \\
                          F_x/m \\
                          F_y/m
                       \end{array}\right)
\end{align}
$

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
```

```python
def ode_rhs(u_vec, t, *params):
    """
    Right-hand side (rhs) of the differential equation, with
    u_vec = [x, y, v_x, v_y] and params = [g, phi].
    """
    x, y, v_x, v_y = u_vec  # We don't actually use x or y here, but could!
    g, phi = params
    return [v_x, v_y, -g*np.sin(phi), -g*np.cos(phi)]
```

```python
g = 9.8
phi = np.pi/8.
theta = np.pi/4.
v0 = 1.

analytic_range = Range_R(theta,phi,g,v0)
print("Analytic range = {}".format(analytic_range))

u0_vec = [0, 0, v0*np.cos(theta), v0*np.sin(theta)]

t_max = 1.  # integration time
t_pts = np.arange(0, t_max, 0.01)

# absolute and relative tolerances for ode solver
abserr = 1.0e-8
relerr = 1.0e-6

# Integrate the differential equation
x, y, v_x, v_y = odeint(ode_rhs, u0_vec, t_pts, args=(g, phi),
              atol=abserr, rtol=relerr).T
```

    Analytic range = 0.06469904807392131

We'll just check one case here, but you can do more!

```python
fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(1,1,1)
ax.plot(x, y)
ax.set_ylim(0,.15)
ax.set_xlim(0,.15)
ax.set_xlabel('x: distance up incline')
ax.set_ylabel(r'y: distance $\perp$ to incline')
ax.axvline(analytic_range, color="red")
```

**Did it work?**

```python

```


# Introducción

A representación gráfica é unha das ferramentas máis potentes para explorar datos, e hoxe en día unha das tarefas máis importantes tanto para enxeñeiros como científicos. De feito, os gráficos úsanse tanto para expoñer como para entender datos e para comprobar visualmente resultados calculados.

O módulo de Python máis amplamente usado para a representación gráfica é **Matplotlib** (<http://matplotlib.org/>), que será fundamentalmente o que usaremos neste curso. Este módulo é moi potente e posúe moitisimas funcionalidades. Esta práctica trata de introducir algúns dos elementos máis básicos. Para ter unha idea de todas as posibilidades que ofrece este módulo, pódese visitar a *Matplotlib Gallery* (http://matplotlib.org/gallery.html).

## Obxectivos

- Uso de Matplotlib no plano e no espazo tridimensional
- Representación gráfica de vectores
- Representación de rexións no plano
- Visualización de obxectos (superficies) no espazo tridimensional

# Uso de Matplotlib

Para usar **Matplotlib** (con versión 3.1.0), necesitamos importar o seu módulo. Posto que usaremos a partir da interface `pyplot`, tamén será necesario importar `pyplot`:

```python
import matplotlib as mp
import matplotlib.pyplot as plt
mp.__version__
```

    '3.1.0'

Evidetemente para realizar cálculos numéricos de forma eficiente, tamén será preciso importar o módulo **Numpy**:

```python
import numpy as np
```

Matplotlib é unha librería de Python que pode ser empregar fora dos *notebooks* Jupyter para representar gráficamente tanto funcións como entidaeds xeométricas (xa sexa por pantalla ou gardalas nun ficheiro). Para que as representacións gráficas aparezan incrustadas neste documento, usaremos o comando *máxico*
`%matplotlib inline`. Polo contrario se é necesario interactuar coas gráficas se podería usar tamén a opción `%matplotlib notebook`

```python
%matplotlib inline
```

Para incrementar o tamaño da fonte que aparecen nos eixes ou no texto das gráficas, pódese incrementar o tamaño da fonte por defecto usando:
```python
matplotlib.rcParams.update({'font.size': 16})
```

# Representación gráfica de vectores en $\mathbb{R}^2$ e $\mathbb{R}^3$

No que segue, en primeiro lugar, representárase gráficamente tres vectores en $\mathbb{R}^2$, cada un dunha cor diferente e gardaremos o resultado desta visualización nun arquivo de extensión `.pdf`:

```python
# Matriz cos datos dos vectores
V = np.array([[1,1],[-1,1],[2,-2]])

# Representación gráfica de vectores 2D
plt.quiver([0], [0], V[:,0], V[:,1], color=['r','b','g'], angles='xy', scale_units='xy', scale=1)

# Definición dos limites dos eixes
plt.xlim([-2,2])
plt.ylim([-2,2])

# Etiquetas dos eixes
plt.xlabel('x')
plt.ylabel('y')

# Escritura a ficheiro e visualización
plt.savefig("grafica-con-vectores.pdf")
plt.show()
```

A representación gráfica de elementos tridimensionais debe facerse empregando un módulo diferente de **Matplotlib**, que é **Axes3D**. Deste xeito, volveríamos a empregar a función `quiver`:

```python
from mpl_toolkits.mplot3d import Axes3D

# Datos dos vectores
V = np.array([[1,1,0],[-1,0,1],[1,0,-2]])

# Inicialización da representación 3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

# Representación dos vectores
ax.quiver([0], [0], [0], V[:,0], V[:,1], V[:,2])
# Axuste dos límites dos eixes
ax.set_xlim([-2,2])
ax.set_ylim([-2,2])
ax.set_zlim([-2,2])

# Etiquetas dos eixes
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
plt.show()
```

Usando as funcións `np.dot` e `np.cross` pódese calcular o produto escalar e o produto vectorial de dous vectores.

### Exercicio 3.1
Dados os vectores $\vec{u}=(1,0,1)^t$ e $\vec{v}=(-1,2,0)^t$. Calcula a representa graficamente o vector que resulta ao proxectar $\vec{u}$ sobre $\vec{v}$.

```python
## O TEU CÓDIGO AQUÍ
```

### Exercicio 3.2
Tendo en conta os vectores anteriores, calcula a representa graficamente o vector $\vec{u}\times\vec{v}$.

```python
## O TEU CÓDIGO AQUÍ
```

### Exercicio 3.3
Tendo en conta os vectores anteriores e o vector $\vec{w}=(0,1,1)^t$, calcula o valor do produto mixto e representa graficamente os vértices do paralepípedo ao que se lle está a calcular o volume.

```python
## O TEU CÓDIGO AQUÍ
```

# Representación gráfica de rexións no plano

Da mesma forma a como se pode empregar o módulo **Matplotlib** para facer representacións gráficas de vectores, tamén se pode empregar **Sympy** para representar rexións no plano. O primeiro paso é importar o módulo de **Sympy** (coa version 1.4) e definir as coordenadas cartesianas $x,y,z$:

```python
import sympy as sp
x, y, z = sp.symbols('x:z', real=True)
sp.__version__
```

    '1.4'

Para representar graficamente un conxunto de puntos ou ben unha rexión empregarase a función `plot_implicit`. Por xemplo, no caso de querer representar os puntos ca circunferencia centrada na orixe de coordenadas e de radio $\sqrt{5}$:

```python
eq = sp.Eq(x**2 + y**2, 5)
p1 = sp.plot_implicit(eq)
print(type(eq))
```

Para visualizar rexións tamén se poden empregar desigualdades como argumento da función `plot_implicit`:

```python
ineq = y > x**2
p2 = sp.plot_implicit(ineq)
```

As condicións lóxicas usadas como argumento de `plot_implicit` tamén poden incluir operadores lóxicos `sp.And` ou `sp.Or` (ou ben `&` e `|`). Nembargantes, existe un *bug* na versión de **Sympy** 1.4.0 co uso de operadores lóxicos, que xa foi solucionado na versión estable actual 1.5.1 (véxase https://github.com/sympy/sympy/issues/17710):

```python
p3 = sp.plot_implicit(y > x)
p3 = sp.plot_implicit((y > x) & (y < -x)) # Implementación cun bug na versión 1.4.0

```

### Exercicio 3.4
Representa graficamente as seguintes rexións ou conxuntos no plano:
- O conxunto de puntos que satisfan $y=2x^2-3$
- A rexión do plano dos puntos que cumpren $|y|<2$
- A rexión do plano dos puntos que están no interior da elipse de radios $5$ e $3$ (nas direccións dos eixes coordenados) e centrada no punto $(2,3)$

```python
## O TEU CÓDIGO AQUÍ
```

# Representación gráfica de superficies no espazo tridimensional

Nin **Sympy** nin tampouco o módulo **Matplotlib** posúen unha forma sinxela de representar superficies tridimensionais. Para esta tarefa, existen outras librerías moito máis especializadas en visualización de datos científicos como **Mayavi** (que non estudaremos neste curso). Para máis información, pódese consultar https://docs.enthought.com/mayavi/mayavi/.

Nembargantes, a partir de **Matplotlib** e do uso da librería **Scikit-image** (versión 0.16.2), ódese implementar a función `isosurface` a partir da cal se poden representar gráficamente iso-superficies das ecuacións implicitas de calquera das superficies que estudiaremos no Tema 1 deste curso.

En primeiro lugar, nos aseguramos que temos dispoñible o módulo **Scikit-image**

```python
!pip -q install scikit-image
```

Esta é a implementación da función `isosurface` (descargable dende https://github.com/gpeyre/numerical-tours) que imos a usar no resto deste documento:

```python
import numpy as np
import matplotlib.pyplot as plt
from skimage import measure
from mpl_toolkits.mplot3d import Axes3D

def isosurface(M,v,step,title=""):
    """
    returns the isosurface of value v of M, subsetting M with the steps argument
    """
    # Get the vertices and faces of a given isosurface
    sel = np.arange(0,np.shape(M)[0],step)
    verts, faces = measure.marching_cubes_classic(M[np.ix_(sel,sel,sel)], v, spacing=(1,1,1))
    # Plot in 3D with a fixed scaling in all Cartesian axis
    fig = plt.figure(figsize = (10,10))
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_trisurf(verts[:, 0], verts[:,1], faces, verts[:, 2], lw=.1, cmap="jet")
    scaling = np.array([getattr(ax, 'get_{}lim'.format(dim))() for dim in 'xyz'])
    ax.auto_scale_xyz(*[[np.min(scaling), np.max(scaling)]]*3)
    ax.axis("off")
    plt.show()
```

Agora podemos usar a funcion `isosurface` para pintar obxectos tridimensionais como esferas, elipsoides, hiperboloides, etc:

```python
# Representar a esfera de radio 1
p = np.linspace(-1.5,1.5,100)
x, y, z = np.meshgrid(p, p, p)
values = x*x + y*y + z*z
isosurface(values, 1., 1)
```

To plot two functions on one graph, we use `plt.plot` twice:

```python
# Representar un toro de radio exterior (R) e interior (r)
p = np.linspace(-5.,5,100)
R = 2; r = 1
x, y, z = np.meshgrid(p, p, p)
values = (R - np.sqrt(x**2 + y**2))**2 + z**2 - r**2
isosurface(values, 0., 1)
```

```python
# Hiperboloide de dúas follas
p = np.linspace(-5.,5,100)
x, y, z = np.meshgrid(p, p, p)
values = z**2 - y**2 -x**2 - 1.
isosurface(values, 0., 1)
```

### Exercicio 3.5
Representa graficamente as seguintes superficies:
- Un elipsoide
- Un hiperboloide dunha folla que o seu eixe coincida co eixe $x$
- Un cono elíptico que o seu eixe coincida co eixe $y$
- Un paraboloide elíptico que teña o seu eixe na recta $z=y=0$
- Un paraboloide hiperbólico

```python
## O TEU CÓDIGO AQUÍ
```


# 머신러닝 개요와 선형회귀

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl

style_name = 'bmh' #bmh
mpl.style.use(style_name)
np.set_printoptions(precision=4, linewidth =150)

style = plt.style.library[style_name]
style_colors = [ c['color'] for c in style['axes.prop_cycle'] ]
```

## 머신러닝?

- 넓은 의미 : 컴퓨터를 이용한 문제 해결에 있어서 지식기반, 규칙 기반 방법이 아닌 데이터와 범용 알고리즘으로 퍼포먼스를 개선하는 방법

> Samuel, A. L. (1959) : "Programming computers to learn from experience should eventually eliminate the need for much of this detailed programming effort"
 - “Some Studies in Machine Learning Using the Game of Checkers” in IBM Journal of Research and Development (Volume:3, Issue: 3), p. 210, 기계학습, 오일석, 한빛미디어

> Tom Mitchell (1998) : Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.
-“Machine Learning”, Coursera, Andrew Ng

- 좁은 의미 : 주어진 데이터를 가장 잘 표현하는 함수를 찾는 것

## 아이가 그림을 그린다면?

- 아이는 T, P, E를 반복하면서 적합한 선을 그리는 법을 배워간다.

- Task T  : 선을 그린다.

- Measure P : 선이 점에 잘 맞는가?

- Experience E : 만족할 때 까지 계속 선을 그린다. 빨간점에서 선이 멀어지면 좋지 않다.

## 수업 범위

### 데어터과학 벤다이어그램

- 데이터 과학보다는 머신러닝에 집중

- 도메인 전문 지식은 다루지 않음

http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram

## 머신러닝 분류

### 지도학습Supervised

#### 회귀 문제Regression : 선형 회귀Linear Regression

- 정답 : 연속된 실수
- 예 : 대지면적에 따른 집값, 시간에 따른 트랜지스터 집적 개수, 기온에 따른 빙과류 판매량

#### 분류 문제Classification : 로지스틱 회귀Logistic Regression

- 정답 : (0,1) 또는 (0, 1, …, K)
- 예 : 개-고양이 분류, 양성종양-악성종양 분류, …

http://scikit-learn.org/stable/auto_examples/linear_model/plot_iris_logistic.html#sphx-glr-auto-examples-linear-model-plot-iris-logistic-py

### 비지도학습Unsupervised

- K-means, PCA, kNN ......

## 직선찾기

### 데이터 준비

- 앤드류 응 교수의 코세라 강의 '머신러닝'에서 사용하는 데이터를 'data_1d.npz'로 만들어 사용

- 데이터 파일 생성 스크립트 : save_data_1d.ipynb

```python
# 데이터 로드
D = np.load('data_1d.npz')

# X : (N,2), y : (N,1)
X = D['X']
y = D['y']
```

```python
plt.figure(figsize=(10,5))
plt.plot(X[:,1], y, 'o')
plt.xlabel('x', fontsize=20)
plt.ylabel('y', fontsize=20)
plt.show()
```

### 개념

- 아이의 직선 그리기처럼……

- 모든 점을 지나는 직선은 그릴 수 없으니 선을 그려보고 점과 선과의 거리를 최대한 줄여보자

- Task T  : 선을 그린다.
    - $y=w_0 + w_1 x$
    - $w_0$와 $w_1$을 결정하기

- Measure P : 선과 점의 수직거리의 총합 $\to$ 작을 수 록 좋다!!

- Experience E : 선을 그리고 P를 측정하기를 반복

- 위 그림의 두 직선에 해당하는 에러량

- 두 직선에서 다른 것은 직선의 위치와 기울기를 결정하는 계수 벡터 $\color{Green}{\mathbf{w}_g}$, $\color{Red}{\mathbf{w}_r}$ 이고 에러를 최소화하기 위해 우리가 결정해야 할 것

$$
\color{Green}{\text{ERROR}} = \frac{1}{2N} \sum_{n=1}^N \left\{ \color{Green}{h(x_n, \mathbf{w}_{g})}  - \color{RoyalBlue}{y_n} \right\}^2
$$

$$
\color{Red}{\text{ERROR}} = \frac{1}{2N} \sum_{n=1}^N \left\{ \color{Red}{h(x_n, \mathbf{w}_{r})}  - \color{RoyalBlue}{y_n} \right\}^2
$$

- 이상의 개념에 포함된 수학과 구현을 수업을 통해 배우게 됨

- 우선 개요에서는 어떤 방식으로 문제를 푸는지 구체적인 데이터를 예로 들어 소개
    - 경사하강법을 이용한 방법
    - 정규방적식을 이용한 방법

### 경사하강법을 이용한 선형회귀

- 다음 코드는 가장 기본적인 선형회귀를 경사하강법을 사용하여 수행한 것

- 핵심코드는 다음 3줄로 여기에 행렬과 벡터의 연산, 미분, 최적화 내용이 들어 있음

```python
for i in range(num_iters) :
    #           이 부분에 미분과 선형대수의 개념이 있음
    c = (1/n) * np.dot(X.T, np.dot(X,w) - y)
    # 이 부분에 최적화의 개념이 있음
    w -= alpha * c
```

```python
num_iters = 1500
n = X.shape[0]
w = np.random.rand(2).reshape(2,1)
alpha = 0.01

# 여기서 우리의 경험 E를 반복하면서 태스크 T를 개선해 나간다.
for i in range(num_iters) :
    #           이 부분에 미분과 선형대수의 개념이 있음
    c = (1/n) * np.dot(X.T, np.dot(X,w) - y)
    # 이 부분에 최적화의 개념이 있음
    w -= alpha * c

x = np.linspace(5, 23, 100)
pred = w[0]+ w[1]*x

plt.figure(figsize=(10,5))
plt.plot(X[:,1], y, 'o', x, pred)
plt.xlabel('x', fontsize=20)
plt.ylabel('y', fontsize=20)
plt.show()

print("Regression coef. w0={:f}, w1={:f}".format(w[0,0], w[1,0]))
```

## 곡선 찾기

### 데이터 준비

- 년도별 트랜지스터의 집적도를 나타내는 moore의 법칙 데이터 활용

- Transistor count from wiki : https://en.wikipedia.org/wiki/Transistor_count

```python
plt.rcParams["figure.figsize"] = (10,5)
x = np.linspace(1971, 2019, 100)

# 데이터 로드
# https://en.wikipedia.org/wiki/Transistor_count
D = np.load('moore.npz')

# X : (N,2), y : (N,1)
X = D['X']
y = D['y']

#########################################################################
# 그림을 그리면 대충 지수적으로 증가하는 것 처럼 보임
# y에 로그 취해서 선형화된 그림과 비교
plt.rcParams["figure.figsize"] = (15,4)

log_y = np.log(y)
fig, axs = plt.subplots(1, 2)
axs[0].plot(X, y, 'o')
axs[0].set_xlabel('year')
axs[0].set_ylabel('Transistor count')

axs[1].plot(X, log_y, 'o')
axs[1].set_xlabel('year')
axs[1].set_ylabel('Log(Transistor count)')
plt.show()
```

### 개념

- 데이터를 직선으로 피팅하기 부적합

- 데이터가 지수적 증가 형태를 보이므로 로그를 취해서 대충 선형으로 변환

- 이후 직선찾기 개념을 그대로 이용하고 지수를 취해서 다시 곡선으로 바꿈

- 직접 곡선 기저함수를 사용하는 법도 4주차에 다룸

### 정규방정식을 이용한 선형회귀

- 다만 위와 다르게 이후 회귀 과정은 반복법을 쓰지 않고 극대, 극소의 개념을 써서 한번에 답을 찾는다.

- 회귀 후 예측 곡선으로 2018년에 집적도 예측

```python
#########################################################################
# y에 로그를 취해서 선형으로 만든 것을 바탕으로 직선을 찾기
# 정규방정식을 이용
plt.rcParams["figure.figsize"] = (10,5)

# 여기서 더 이상 반복하지않고 한번에 답을 찾는다.
X              = np.hstack((np.ones(X.shape[0]).reshape(-1,1), X.reshape(-1,1))) #design matrix
popt_exp_model = np.linalg.solve(np.dot(X.T, X), np.dot(X.T , log_y.reshape(-1,1))).reshape(-1) #normal eq.
y_exp_model    = np.exp(popt_exp_model[1]*x + popt_exp_model[0])

########################################################################
# 선형화된 데이터와 직선 그리기
plt.plot(X[:,1], log_y, 'o')
plt.plot(x, popt_exp_model[0] + popt_exp_model[1]*x)
plt.xlabel('year')
plt.ylabel('Log(Transistor count)')
plt.show()

########################################################################
# 데이터와 회귀 곡선 그리기
plt.plot(X[:,1], y, 'o')
plt.plot(x, y_exp_model, label="y_exp_model")
plt.xlabel('year')
plt.ylabel('Transistor count')
plt.legend()
plt.show()

print("w : {}".format(popt_exp_model))
print( "Prediction 2018 : {:,d}".format( int(y_exp_model[-1]) ) )
```

$$
\begin{align}
&\log(\text{transistor count}) = 0.351x - 685 \\[5pt]
&e^{\log(\text{transistor count})} = e^{0.351x - 685} \\[5pt]
&\text{transistor count} = e^{-685}e^{0.351x} = (3.22 \times 10^{-298}) e^{0.351x}
\end{align}
$$

```python
print(np.exp(-6.8500e+02))
print(np.log(3E-298))
```

    3.223145390850647e-298
    -685.0717454235576

### Excel의 회귀 결과와 비교

- 엑셀로 선형회귀 작업을 할 수 있음

- 데이터 선택 → 분산형 그래프 삽입 → 차트도구:디자인에서 차트레이아웃3 선택 → 직선 더블클릭 → ‘지수’ 선택

- 회귀 결과는 우리 코딩 결과와 일치

## 목표

- 다음 식을 적어도 “합리적 수준”에서 이해하고 구현하기 위해……

$$
\begin{align}
\mathbf{w} &= \mathbf{w} - \alpha \frac{\partial}{\partial \mathbf{w}} \left( \frac{1}{2N} \sum_{n=1}^N \{ h(x_n, \mathbf{w})- y_n \}^2 \right) \\[5pt]
\mathbf{w} &= \left(\mathbf{X}^{\text{T}}\mathbf{X}\right)^{-1} \mathbf{X}^{\text{T}} \mathbf{y}
\end{align}
$$

- 파이썬 기본 문법을 익히고,

- 함수, 미분, 행렬과 벡터의 연산, 약간의 최적화 이론을 공부

```python
%%html
<link href='https://fonts.googleapis.com/earlyaccess/notosanskr.css' rel='stylesheet' type='text/css'>
<!--https://github.com/kattergil/NotoSerifKR-Web/stargazers-->
<link href='https://cdn.rawgit.com/kattergil/NotoSerifKR-Web/5e08423b/stylesheet/NotoSerif-Web.css' rel='stylesheet' type='text/css'>
<!--https://github.com/Joungkyun/font-d2coding-->
<link href="http://cdn.jsdelivr.net/gh/joungkyun/font-d2coding/d2coding.css" rel="stylesheet" type="text/css">
<style>
    h1     { font-family: 'Noto Sans KR' !important; color:#348ABD !important;   }
    h2     { font-family: 'Noto Sans KR' !important; color:#467821 !important;   }
    h3, h4 { font-family: 'Noto Sans KR' !important; color:#A60628 !important;   }
    p:not(.navbar-text) { font-family: 'Noto Serif KR', 'Nanum Myeongjo'; font-size: 12pt; line-height: 200%;  text-indent: 10px; }
    li:not(.dropdown):not(.p-TabBar-tab):not(.p-MenuBar-item):not(.jp-DirListing-item):not(.p-CommandPalette-header):not(.p-CommandPalette-item):not(.jp-RunningSessions-item)
            { font-family: 'Noto Serif KR', 'Nanum Myeongjo'; font-size: 12pt; line-height: 200%; }
    table  { font-family: 'Noto Sans KR' !important;  font-size: 11pt !important; }
    li > p  { text-indent: 0px; }
    li > ul { margin-top: 0px !important; }
    sup { font-family: 'Noto Sans KR'; font-size: 9pt; }
    code, pre  { font-family: D2Coding, 'D2 coding' !important; font-size: 12pt !important; line-height: 130% !important;}
    .code-body { font-family: D2Coding, 'D2 coding' !important; font-size: 12pt !important;}
    .ns        { font-family: 'Noto Sans KR'; font-size: 15pt;}
    .summary   {
                   font-family: 'Georgia'; font-size: 12pt; line-height: 200%;
                   border-left:3px solid #FF0000;
                   padding-left:20px;
                   margin-top:10px;
                   margin-left:15px;
               }
    .green { color:#467821 !important; }
    .comment { font-family: 'Noto Sans KR'; font-size: 10pt; }
</style>
```

<link href='https://fonts.googleapis.com/earlyaccess/notosanskr.css' rel='stylesheet' type='text/css'>

<link href='https://cdn.rawgit.com/kattergil/NotoSerifKR-Web/5e08423b/stylesheet/NotoSerif-Web.css' rel='stylesheet' type='text/css'>

<link href="http://cdn.jsdelivr.net/gh/joungkyun/font-d2coding/d2coding.css" rel="stylesheet" type="text/css">


# Introduction to the Harmonic Oscillator

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week week we are going to begin studying molecular dynamics, which uses classical mechanics to study molecular systems. Our "hydrogen atom" in this section will be the 1D harmomic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2}  \omega_0^2 m {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

In classical mechanics, our goal is to determine the equations of motion, $x(t),y(t)$, that describe our system.

In this notebook we will use sympy to solve an second order, ordinary differential equation.

## 1. Solving differential equations with sympy

Soliving differential equations can be tough, and there is not always a set plan on how to proceed. Luckily for us, the harmonic osscillator is the classic second order diffferential eqations.

Consider the following second order differential equation

$$ay(t)''+by(t)'=c$$

where $y(t)'' = {{{d^2}y} \over {dt^2}}$, and $y(t)' = {{{d}y} \over {dt}}$

We can rewrite this as a homogeneous linear differential equations

$$ay(t)''+by(t)'-c=0$$

The goal here is to find $y(t)$, similar to our classical mechanics problems. Lets use sympy to solve this equation

### Second order ordinary differential equation

First we import the sympy library

```python
import sympy as sym
```

Next we initialize pretty printing

```python
sym.init_printing()
```

Next we will set our symbols

```python
t,a,b,c=sym.symbols("t,a,b,c")
```

Now for somehting new. We can define functions using `sym.Function("f")`

```python
y=sym.Function("y")
y(t)
```

Now, If I want to define a first or second derivative, I can use `sym.diff`

```python
sym.diff(y(t),(t,1)),sym.diff(y(t),(t,2))
```

My differential equation can be written as follows

```python
dfeq=a*sym.diff(y(t),(t,2))+b*sym.diff(y(t),(t,1))-c
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

The two constants $C_1$ and $C_2$ can be determined by setting boundry conditions.
First, we can set the condition $y(t=0)=y_0$

The next intial condition we will set is $y'(t=0)=v_0$

To setup the equality we want to solve, we are using `sym.Eq`. This function sets up an equaility between a lhs aand rhs of an equation

```python
# sym.Eq example
alpha,beta=sym.symbols("alpha,beta")
sym.Eq(alpha+2,beta)
```

Back to the actual problem

```python
y0,v0=sym.symbols("y_0,v_0")
ics=[sym.Eq(sol.args[1].subs(t, 0), y0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

We can use this result to first solve for $C_2$ and then solve for $C_1$.
Or we can use sympy to solve this for us.

```python
solved_ics=sym.solve(ics)
solved_ics
```

Substitute the result back into $y(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

We can plot this result too. Assume that $a,b,c=1$ and that the starting conditions are $y_0=0,v_0=0$

We will use two sample problems:

* case 1 : initial position is nonzero and initial velocity is zero
* case 2 : initial position is zero and initialvelocity is nonzero

```python
# Print plots
%matplotlib inline
```

#### Initial velocity set to zero

```python
case1 = sym.simplify(full_sol.subs({y0:0, v0:0, a:1, b:1, c:1}))
case1
```

```python
sym.plot(case1.rhs)
sym.plot(case1.rhs,(t,-2,2))
```

#### Initial velocity set to one

```python
case2 = sym.simplify(full_sol.subs({y0:0, v0:1, a:1, b:1, c:1}))
case2
```

```python
sym.plot(case2.lhs,(t,-2,2))
```

## Calculate the phase space

As we will see in lecture, the state of our classical systems are defined as points in phase space, a hyperspace defined by ${{\bf{r}}^N},{{\bf{p}}^N}$. We will convert our sympy expression into a numerical function so that we can plot the path of $y(t)$ in phase space $y,y'$.

```python
case1
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
yfunc=sym.lambdify(t,case1.rhs,'numpy')
vfunc=sym.lambdify(t,case1.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(yfunc(tlst),vfunc(tlst))
plt.xlabel('$y$')
plt.ylabel("$y'$")
plt.show()
```

### Exercise 1.1

Change the initial starting conditions and see how that changes the plots. Make three different plots with different starting conditions

```python
# Your code here
case3 = sym.simplify(full_sol.subs({y0:1, v0:1, a:1, b:1, c:1}))
case3
```

```python
sym.plot(case3.rhs)
sym.plot(case3.rhs,(t,-2,2))
```

```python
case4 = sym.simplify(full_sol.subs({y0:0, v0:0, a:2, b:1, c:1}))
case4
```

```python
sym.plot(case4.rhs)
sym.plot(case4.rhs,(t,-2,2))
```

```python
case5 = sym.simplify(full_sol.subs({y0:0, v0:0, a:1, b:2, c:1}))
case5
```

```python
sym.plot(case5.rhs)
sym.plot(case5.rhs,(t,-2,2))
```

## 2. Harmonic oscillator

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= - \omega_0^2 m x $$

$$ a = - \omega_0^2 x $$

$$ x(t)'' = - \omega_0^2 x $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

Your goal is determine and plot the equations of motion of a 1D harmomnic oscillator

### Exercise 2.1

1. Use the methodology above to determine the equations of motion $x(t), v(t)$ for a harmonic ocillator
1. Solve for any constants by using the following initial conditions: $x(0)=x_0, v(0)=v_0$
1. Show expressions for and plot the equations of motion for the following cases:
    1. $x(0)=0, v(0)=0$
    1. $x(0)=0, v(0)>0$
    1. $x(0)>0, v(0)=0$
    1. $x(0)<0, v(0)=0$
1. Plot the phasespace diagram for the harmonic oscillator

```python
# Your code here
x=sym.Function("x")
x(t)
```

```python
t,a,m,x_p,w_0=sym.symbols("t,a,m,x_p,w_0")
t,a,m,x_p,w_0
```

```python
sym.diff(x(t),(t,1)),sym.diff(x(t),(t,2))
```

```python
dfeq2=sym.diff(x(t),(t,2))+((w_0**2)*x(t))
dfeq2
```

```python
sol2 = sym.dsolve(dfeq2)
sol2
```

```python
x0,v0=sym.symbols("x_0,v_0")
ics2=[sym.Eq(sol2.args[1].subs(t, 0), x0),
     sym.Eq(sol2.args[1].diff(t).subs(t, 0), v0)]
ics2
```

```python
solved_ics2=sym.solve(ics2)
solved_ics2
```

```python
full_sol2 = sol2.subs(solved_ics2[0])
full_sol2
```

```python
caseA = sym.simplify(full_sol2.subs({x0:0, v0:0, w_0:1}))
caseA
```

```python
sym.plot(caseA.rhs)
sym.plot(caseA.rhs,(t,-2,2))
```

```python
caseB = sym.simplify(full_sol2.subs({x0:0, v0:1, w_0:1}))
caseB
```

```python
sym.plot(caseB.rhs)
sym.plot(caseB.rhs,(t,-2,2))
```

```python
caseC = sym.simplify(full_sol2.subs({x0:1, v0:0, w_0:1}))
caseC
```

```python
sym.plot(caseC.rhs)
sym.plot(caseC.rhs,(t,-2,2))
```

```python
caseD = sym.simplify(full_sol2.subs({x0:-1, v0:0, w_0:1}))
caseD
```

```python
sym.plot(caseD.rhs)
sym.plot(caseD.rhs,(t,-2,2))
```

```python
# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,caseA.rhs,'numpy')
vfunc=sym.lambdify(t,caseA.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python
# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,caseB.rhs,'numpy')
vfunc=sym.lambdify(t,caseB.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python
# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,caseC.rhs,'numpy')
vfunc=sym.lambdify(t,caseC.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python
# Make numerical functions out of symbolic expressions
xfunc=sym.lambdify(t,caseD.rhs,'numpy')
vfunc=sym.lambdify(t,caseD.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(xfunc(tlst),vfunc(tlst))
plt.xlabel('$x$')
plt.ylabel("$x'$")
plt.show()
```

```python

```


```
%matplotlib inline
```

```
from __future__ import division
from __future__ import print_function
```

```
import numpy as np

import matplotlib as mpl
import matplotlib.pyplot as plt

import imageio

from scipy import ndimage
from scipy import stats
from scipy.spatial import distance

from IPython import display
```

# K-Means clustering

K-Means is the most basic data clustering approach. Its operation consists of assigning each data sample to its *cluster* ensuring that samples added to a cluster are more similar to themselves than to the data in other clusters.

Alternatively, K-Means can be seen as a lossy compressions scheme, in which each sample is replaced with a prototype in a way that minimizes the total sum of errors.

Formally, given a set of $N$ data samples $X = \{x^{(1)}, x^{(2)}, \ldots, x^{(N)}\}$ and a desired number of clusters $K$ K-Means finds $K$ cluster *centroids* or *prototypes* $\{\mu^{(j)}, j\in 1,2,\ldots,K\}$. Then, it assigns each sample $x^{(i)}$ to the cluster $c^{(i)} \in \{1, 2, \ldots, K\}$ whose centroids is the closest to $x^{(i)}$: $c^{(i)} = \underset{j}\arg\min ||x^{(i)} - \mu^{(j)}||$. To find the centroids, K-means attempts to solve the following optimization problem:

\begin{equation}
    \underset{\mu, c}\arg\min \sum_i||x^{(i)} - \mu^{(c^{(i)})}||^2_2
\end{equation}

A typical algorithm used to solve the optimization problem iterates between optimizing over $\mu$ and optimizing over $c$:

- Initialize $\mu$ to e.g. K randomly selected data samples
- Loop:
    - Assign samples to clusters (optimize over $c$)
    - Move cluster centroids (optimize over $\mu$): $\mu^{(j)} \gets \frac{\sum_i [c^{(i)}=j] x^{(i)}}{\sum_i [c^{(i)}=j]}$, where $[\cdot]$ is the indicator function.

The code below applies the K-Means algorithm to optimize the color palette of an image. It replaces each 16-bit pixel (two 8-bit colors) by a 4-bit color identifier.

```
!wget -q https://github.com/janchorowski/nn_assignments/raw/nn17_fall/lecture_notebooks/11-Rosa_Gold_Glow_2_small_noblue.png
```

```
img = imageio.imread('11-Rosa_Gold_Glow_2_small_noblue.png')/255.

plt.subplot(121)
plt.imshow(img)
plt.grid(False)

im_flat = img.reshape([-1, 3])
plt.subplot(122)
plt.scatter(im_flat[:,0], im_flat[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
plt.axis('square')
None
```

```
# use K-means to quantize the colors
```

```
N, D = im_flat.shape
```

```
K = 16
initial_center_ids = np.random.choice(N, (K,), replace=False)
centers = im_flat[initial_center_ids, :]

old_loss = np.inf
for i in range(10000):
    center_to_cluster_dists = distance.cdist(centers, im_flat)
    cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
    for k in range(K):
        centers[k] = im_flat[cluster_assignment==k].mean(axis=0)
    loss = ((centers[cluster_assignment] - im_flat)**2).sum()
    if not i%10:
        print ("Iteration: %d Loss: %g" % (i, loss))
    if loss >= old_loss:
        break
    old_loss = loss

plt.subplot(131)
plt.imshow(img)
plt.grid(False)

plt.subplot(132)
plt.scatter(im_flat[:,0], im_flat[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
plt.scatter(centers[:,0], centers[:,1], marker='x')
plt.axis('square')

plt.subplot(133)
center_to_cluster_dists = distance.cdist(centers, im_flat)
cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
plt.imshow(centers[cluster_assignment].reshape(img.shape))
plt.grid(False)
```

# On-line K-means

The default K-means algorithm works in batch mode: all data samples are processed at each iteration. We can, however simply transform k-means to an on-line algorithm. The main idea is to process a stream of data samples. For each sample, we find its closest centroid and move it in the direction of the sample by a small amount. This is called **winner takes all** approach: the winning (closest) centroid takes all the update.

The code below recreates the compression example using on-line training.

```
# An on-line version of k-Means

K = 16
initial_center_ids = np.random.choice(N, (K,), replace=False)
centers = im_flat[initial_center_ids, :]

def make_plot():
    plt.subplot(131)
    plt.imshow(img)
    plt.grid(False)

    plt.subplot(132)
    plt.scatter(im_flat[:,0], im_flat[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
    plt.scatter(centers[:,0], centers[:,1], marker='x')
    plt.axis('square')

    plt.subplot(133)
    center_to_cluster_dists = distance.cdist(centers, im_flat)
    cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
    plt.imshow(centers[cluster_assignment].reshape(img.shape))
    plt.grid(False)
    plt.show()

alpha = 1e-2
for i in range(1000000):
    sampled_x = im_flat[np.random.randint(N)]
    cluster_id = np.argmin(((centers - sampled_x)**2).sum(axis=1))
    # centers[cluster_id] = centers[cluster_id] - alpha * (centers[cluster_id] - sampled_x)
    # or equivalently
    centers[cluster_id] = (1 - alpha) * centers[cluster_id] + alpha * sampled_x

    if not i%100000:
        center_to_cluster_dists = distance.cdist(centers, im_flat)
        cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
        loss = ((centers[cluster_assignment] - im_flat)**2).sum()
        display.clear_output()
        print ("Iteration: %d Loss: %g" % (i, loss))
        make_plot()

```

# Kohonen maps

The on-line *winner takes all* approach yields yet another generalization: **winner takes most**. In the very successful Kohonen map data representation we define a neighborhood relation (a topology) on the centroids. Then, we change a centroid *and its neighbors*. Please note that the topology is chosen a-priori, without seeing any data.

In the example below, the topology is a linear one: the centroids are arranged into a chain and we define matrix $C$ to control which centroids are changed during algorithm iterations.

In the code given below, for each data sample the closest centroid is found. Then, we look at its neighborhood and change all the centroids, proportionally to the appropriate row of the *C* matrix.

Typically, when the algorithm finishes the chain of centorids spans all colors and movement along it roughly corresponds to pixel intensity changes. Thus, the map has found a nonlinear, 1D representation of the data (the pixel intensities).

There are many more variations of the self organizing map, you can play with demos for several ones here: http://www.demogng.de/js/demogng.html?model=GNG&showAutoRestart&autoRestartTime=8

```
# Kohonen map

# First create a topology between units.
# This has nothing to do with the data we want to quantize!

NY, NX = np.meshgrid(np.arange(1), np.arange(24))
centroid_topology = np.stack([NY.ravel(), NX.ravel()], axis=1)

K = centroid_topology.shape[0]

C = np.zeros((K, K))
for ni in range(K):
    for nj in range(K):
        C[ni, nj] = np.maximum(0.0,
                               1.0 - np.abs(centroid_topology[ni] - centroid_topology[nj]).sum()/2.0)

def plot_lines(C, locations, **kwargs):
    segments = []
    for ni, nj in zip(*np.nonzero(C)):
        if ni < nj:
            segments.append(locations[ni])
            segments.append(locations[nj])
            segments.append(np.zeros((2,)) + np.nan)
    segments = np.array(segments)
    plt.plot(segments[:,0], segments[:,1], **kwargs)

plt.scatter(centroid_topology[:,0], centroid_topology[:,1])
plot_lines(C, centroid_topology, color='k')
```

```
# Make it a loop
# C[0, -1] = 0.5
# C[-1, 0] = 0.5
# C
```

    array([[1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5],
           [0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. ,
            0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5,
            1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. , 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5, 0. ],
           [0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. , 0.5],
           [0.5, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,
            0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. ]])

```
# Kohonen map

# First create a topology between units.
# This has nothing to do with the data we want to quantize!

def make_plot():
    plt.subplot(131)
    plt.imshow(img)
    plt.grid(False)

    plt.subplot(132)
    plt.scatter(im_flat[:,0], im_flat[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
    plt.scatter(centers[:,0], centers[:,1], marker='x')
    plot_lines(C, centers[:,:2], color='k')
    plt.axis('square')

    plt.subplot(133)
    center_to_cluster_dists = distance.cdist(centers, im_flat)
    cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
    plt.imshow(centers[cluster_assignment].reshape(img.shape))
    plt.grid(False)
    plt.show()

initial_center_ids = np.random.choice(N, (K,), replace=False)
centers = im_flat[initial_center_ids, :]

alpha = 2e-2
for i in range(3000000):
    sampled_x = im_flat[np.random.randint(N)]
    cluster_id = np.argmin(((centers - sampled_x)**2).sum(axis=1))
    cluster_alpha = (alpha * C[cluster_id]).reshape([-1, 1])
    centers = (1 - cluster_alpha) * centers + cluster_alpha * sampled_x

    if not i%100000:
        center_to_cluster_dists = distance.cdist(centers, im_flat)
        cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
        loss = ((centers[cluster_assignment] - im_flat)**2).sum()
        display.clear_output(wait=True)
        print ("Iteration: %d Loss: %g" % (i, loss))
        make_plot()
```

```
plt.scatter(im_flat[:,0], im_flat[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
plt.scatter(centers[:,0], centers[:,1], marker='x')
plot_lines(C, centers[:,:2], color='k')
plt.axis('square')
```

# Fitting a Gaussian mixture using EM

K-means is a *clustering* or *quantization* algorithm - each data sample is assigned to a cluster (also called prototypes or centroids) and the prototypes are chosen to minimize the reconstruction error. Thus, K-means is typically used to:
1. describe the data,in terms of clusters/prototypes,
2. compress the data, replacing each sample by the nearest prototype.

K-means does not implement an easy way to generate new data as it does not define a data generating probability distribution and only relies on a data reconstruction objective.

However, with a slight change we can add the probabilistic interpretation to the K-means algorithm. Consider the following data generation scheme, called a mixture of Gaussians:
1. Sample a prototype $z\in \{1, 2,\ldots, K\}$ with $p(z=j) = \pi_j$
2. Sample the new data point $x \in \mathbb{R}^d$ from $p(x|z=j) = \mathcal{N}(\mu_j, \Sigma_j)$

We will collectively call the parameters of the model $\Theta=(\pi, \mu, \Sigma)$ and to make it clear we will use $\Theta$ as a subscript for probability distributions. We will fit them using the maximum likelihood principle:

\begin{equation}
\begin{split}
L(\Theta) &= \prod_i p(x^{(i)}) = \prod_i \sum_j p_\Theta(x^{(i)}, z=j) = \prod_i \sum_j p_\Theta(x^{(i)}|z=j) p_\Theta(z=j) \\
\log L(\Theta) &= \sum_i \log p(x^{(i)})= \sum_i \log\left(\sum_j p_\Theta(x^{(i)}, z=j)\right) = \sum_i \log\left(\sum_j p_\Theta(x^{(i)}|z=j) p_\Theta(z=j)\right)
\end{split}
\end{equation}

However, this form of the likelihood function is difficult to optimize and we will instead introduce the [Expectation Maximization (EM)](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm) algorithm.

## Intuitions about EM Algorithm

First, observe that if we knew for each data sample $x^{(i)}$ the mixture component from which it was generated, the likelihood of the joint distribution over the observed variables $x$ and the latent variables $z$ is easy to optimize:

\begin{equation}
\begin{split}
\prod_i p_\Theta(x^{(i)}, z=z^{(i)}) &= \prod_i p_\Theta(x^{(i)}|z=z^{(i)}) p_\Theta(z=z^{(i)}) \\
\log \prod_i p_\Theta(x^{(i)}, z=z^{(i)}) &= \sum_i \left( \log  p_\Theta(x^{(i)}|z=z^{(i)}) + \log p_\Theta(z=z^{(i)})\right)
\end{split}
\end{equation}

However, we do not know the value of $z$. On the other hand, we can estimate $z$ by computing a posterior distribution of $z$ given $x$ for our current estimate of the parameters $\Theta$:

\begin{equation}
p_\Theta(z=j|x) = \frac{p_\Theta(x,z)}{p_\Theta(x)} = \frac{p_\Theta(x|z=j)p_\Theta(z=j)}{\sum_{l}p_\Theta(x|z=l)p_\Theta(z=l)}
\end{equation}

Then a heuristic procedure for fitting Gaussian Mixtures would repeat these two steps:
1. *Estimate* the posterior $p_\Theta(z=j | x^{(i)})$. Let $w^{(i)}_{j} = p_\Theta(z=j | x^{(i)})$.
2. *Maximize* the likelihood of $\Theta$ under our current guess about $z$. This is simple fitting of Gaussian mixtures in which we treat sample $x^{(i)}$ as belonging to cluster $j$ with weight $w^{(i)}_{j}$:
  - $\pi_j = \frac{\sum_{i=1}^N w^{(i)}_{j}}{N}$
  - $\mu_j = \frac{\sum_i w^{(i)}_{j}x^{(i)}}{\sum_i w^{(i)}_{j}}$
  - $\Sigma_j = \frac{\sum_i w^{(i)}_{j} (x^{(i)} - \mu_j)(x^{(i)} - \mu_j)^T}{\sum_i w^{(i)}_{j}}$
Please note how this procedure is similar to the K-Means algorithm.

## Justification of the EM algorithm

We will now prove that iterations of the E-M steps indeed improve the likelihood of our model. Intuitively, the EM algorithm first constructs a guess about the distribution of $z$, then uses this guess to optimize the parameters $\Theta$.  We will first formalize how we can measure the quality of an approximating distribution.

Formally, the [Kulback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence), or simply the KL-divergence is defined as:

\begin{equation}
KL(q(z) || p(z)) = -\sum_z q(z) \log\frac{p(z)}{q(z)} = -\mathbb{E}_{z \sim q}\left[ \log\frac{p(z)}{q(z)} \right]
\end{equation}

It has the following properties:
1. $KL(q(z) || p(z)) =0$ iff $q(z) = p(z)$
2. $KL(q(z) || p(z)) \geq 0$, which can be proven using the [Jensen's inequality](https://en.wikipedia.org/wiki/Jensen%27s_inequality): $KL(q(z) || p(z)) = \mathbb{E}_{z \sim q}\left[ -\log\frac{p(z)}{q(z)} \right] \geq -\log \mathbb{E}_{z \sim q}\left[\frac{p(z)}{q(z)} \right] = -\log \sum_z p(z) = -\log 1 = 0$.

Now consider any distribution $q_{\phi}(z|x)$.

\begin{equation}
\begin{split}
KL(q_\phi(z|x) || p_\Theta(z|x)) &= -\mathbb{E}_{z \sim q}\left[ \log\frac{p_\Theta(z|x)}{q_\phi(z|x)}\right] \\
&= -\mathbb{E}_{z \sim q}\left[ \log\frac{p_\Theta(z|x)p_\Theta(x)}{q_\phi(z|x)p_\Theta(x)}\right] \\
&= -\mathbb{E}_{z \sim q}\left[\log\frac{p_\Theta(x, z)}{q_\phi(z|x)}\right] + \mathbb{E}_{z \sim q}\left[ \log p_\Theta(x) \right] \\
&= -\mathbb{E}_{z \sim q}\left[\log\frac{p_\Theta(x, z)}{q_\phi(z|x)}\right] + \log p_\Theta(x)
\end{split}
\end{equation}

From there we conclude, that:
$$
\log p_\Theta(x) = KL(q_\phi(z|x) || p_\Theta(z|x)) + \mathbb{E}_{z \sim q}\left[\log\frac{p_\Theta(x, z)}{q_\phi(z|x)}\right]
$$

And the full data likelihood is just a sum over all the samples:

$$
\log L(\Theta) = \sum_i \log p_\Theta(x^{(i)}) = \sum_i KL(q_\phi(z^{(i)}|x^{(i)}) || p_\Theta(z^{(i)}|x^{(i)})) + \sum_i \mathbb{E}_{z^{(i)} \sim q}\left[\log\frac{p_\Theta(x^{(i)}, z^{(i)})}{q_\phi(z^{(i)}|x^{(i)})}\right]
$$

Now the two steps the EM algorithm is doing are:
1. Expectation: set $\sum_i KL(q_\phi(z^{(i)}|x^{(i)}) || p_\Theta(z^{(i)}|x^{(i)})) = 0$ by setting $q_\phi(z^{(i)}|x^{(i)}) = p_\Theta(z^{(i)}|x^{(i)})$. This step introduces the surrogate distribution $q$ to model our current belief about $z$. This is easy to compute using the Bayes theorem.
2. Maximization: keeping the approximating distribution $q(z|x)$ constant, maximize the term $\mathbb{E}_{z^{(i)} \sim q}\left[\log\frac{p_\Theta(x^{(i)}, z^{(i)})}{q_\phi(z^{(i)}|x^{(i)})}\right]$ over $\Theta$.

We will first show that EM iterations indeed increase the log-likelihood. Let $\Theta^t$ denote the model parameters at iteration $t$:

\begin{equation}
\begin{split}
\log L(\Theta^t) &= \sum_i \log p_{\Theta^t}(x^{(i)}) \\
&= \sum_i KL(q(z^{(i)}|x^{(i)}) || p_{\Theta^t}(z^{(i)}|x^{(i)})) + \sum_i \mathbb{E}_{z^{(i)} \sim q(z^{(i)}|x^{(i)})}\left[\log\frac{p_{\Theta^t}(x^{(i)}, z^{(i)})}{q(z^{(i)}|x^{(i)})}\right] \;\text{ for any } q(z^{(i)}|x^{(i)}) \\
&= \sum_i \mathbb{E}_{z^{(i)} \sim q_{\phi^t}(z^{(i)}|x^{(i)})}\left[\log\frac{p_{\Theta^t}(x^{(i)}, z^{(i)})}{q_{\phi^t}(z^{(i)}|x^{(i)})}\right] \text{ because we set } q_{\phi^t}(z^{(i)}|x^{(i)}) = p_{\Theta^t}(z^{(i)}|x^{(i)}) \;\text{ and thus KL is 0} \\
&\leq \sum_i \mathbb{E}_{z^{(i)} \sim q_{\phi^t}(z^{(i)}|x^{(i)})}\left[\log\frac{p_{\Theta^{t+1}}(x^{(i)}, z^{(i)})}{q_{\phi^t}(z^{(i)}|x^{(i)})}\right] \;\text{ because we maximize over } \Theta^{t+1} \\
&\leq \sum_i KL(q_{\phi^t}(z^{(i)}|x^{(i)}) || p_{\Theta^{t+1}}(z^{(i)}|x^{(i)})) + \sum_i \mathbb{E}_{z^{(i)} \sim q_{\phi^t}(z^{(i)}|x^{(i)})}\left[\log\frac{p_{\Theta^{t+1}}(x^{(i)}, z^{(i)})}{q_{\phi^t}(z^{(i)}|x^{(i)})}\right] \;\text{ because KL is } \geq 0 \\
&= \log L(\Theta^{t+1})
\end{split}
\end{equation}

Finally, please note that the second step is just regular maximum likelihood estimation. Recall that $\Theta=(\pi, \mu, \Sigma)$ and let $w^{(i)}_j=q_\phi(z^{(i)}=j|x^{(i)})$:

\begin{equation}
\begin{split}
&\text{arg max}_\Theta \mathbb{E}_{z^{(i)} \sim q}\left[\log\frac{p_\Theta(x^{(i)}, z^{(i)})}{q_\phi(z^{(i)}|x^{(i)})}\right] =\\
&\text{arg max}_{\Theta}\sum_j q_\phi(z^{(i)}=j|x^{(i)}) \log\frac{p_\Theta(x^{(i)}, z^{(i)}=j)}{q_\phi(z^{(i)}=j|x^{(i)})} =\\
&\text{arg max}_{\Theta}\sum_j  w^{(i)}_j \log\frac{p_\Theta(x^{(i)}, z^{(i)}=j)}{w^{(i)}_j} =\\
&\text{arg max}_{\Theta}\sum_j  w^{(i)}_j \log p_\Theta(x^{(i)}, z^{(i)}=j) = \quad [w_j\text{ are constant}]\\
&\text{arg max}_{\pi, \mu, \Sigma}\sum_j  w^{(i)}_j \log p(x^{(i)}|z^{(i)}=j;\mu,\Sigma) + \sum_j  w^{(i)}_j \log p(z^{(i)}=j;\pi)
\end{split}
\end{equation}

The maximizations over $\mu, \Sigma$ and over $\pi$ can be performed separately. It is fairly straightforward to show that tey result in:
  - $\pi_j = \frac{\sum_{i=1}^N w^{(i)}_{j}}{N}$
  - $\mu_j = \frac{\sum_i w^{(i)}_{j}x^{(i)}}{\sum_i w^{(i)}_{j}}$
  - $\Sigma_j = \frac{\sum_i w^{(i)}_{j} (x^{(i)} - \mu_j)(x^{(i)} - \mu_j)^T}{\sum_i w^{(i)}_{j}}$

## EM demonstration

The code below implements the EM algorithm to fit a Gaussian mixture to the pixel intensities. Note that typically the means of the Gaussians are initialized using a vanilla K-Means iteration.

```
def make_ellipse(mean, cov, ax, **kwargs):
    """Plot an ellipse."""
    kwargs.setdefault('alpha', 0.5)
    v, w = np.linalg.eigh(cov)
    u = w[0] / np.linalg.norm(w[0])
    angle = np.arctan2(u[1], u[0])
    angle = 180 * angle / np.pi  # convert to degrees
    v = 2. * np.sqrt(2.) * np.sqrt(v)
    ell = mpl.patches.Ellipse(mean, v[0], v[1],
                              180 + angle, **kwargs)
    ell.set_clip_box(ax.bbox)
    ax.add_artist(ell)
```

```
# first run a k-means to initialize
K = 16
initiali_center_ids = np.random.choice(N, (K,), replace=False)
centers = im_flat[initiali_center_ids, :]

old_loss = np.inf
for i in range(10000):
    center_to_cluster_dists = distance.cdist(centers, im_flat)
    cluster_assignment = np.argmin(center_to_cluster_dists, axis=0)
    for k in range(K):
        centers[k] = im_flat[cluster_assignment==k].mean(axis=0)
    loss = ((centers[cluster_assignment] - im_flat)**2).sum()
    if not i%10:
        print ("Iteration: %d Loss: %g" % (i, loss))
    if loss >= old_loss:
        break
    old_loss = loss
```

    Iteration: 0 Loss: 344.154
    Iteration: 10 Loss: 103.688
    Iteration: 20 Loss: 84.9733
    Iteration: 30 Loss: 81.3583
    Iteration: 40 Loss: 79.4319
    Iteration: 50 Loss: 78.2768
    Iteration: 60 Loss: 76.8871
    Iteration: 70 Loss: 76.2338
    Iteration: 80 Loss: 75.718
    Iteration: 90 Loss: 75.5033
    Iteration: 100 Loss: 75.433
    Iteration: 110 Loss: 75.3438
    Iteration: 120 Loss: 74.7627
    Iteration: 130 Loss: 74.4229
    Iteration: 140 Loss: 74.3691
    Iteration: 150 Loss: 74.3351
    Iteration: 160 Loss: 74.3146

```
im_flat_2D = im_flat[:, :2]

if 0:
    # We can start the EM iteration by randomly selecting centers
    K = 16
    initial_center_ids = np.random.choice(N, (K,), replace=False)
    mus = im_flat_2D[initial_center_ids, :]
else:
    # but it is often better to start with K-means centroids
    mus = centers[:, :2]

# initialize the covariances and cluster probabilities
sigmas = np.array([np.cov(im_flat_2D.T)] * K)
pis = np.ones((K,)) / K

# we can optionally foce some minimum covariance which will result in broader Gaussians
# use 1e-6 to get results similar to those of the sklearn's implementation below
reg_covar = 1e-6

def make_plot():
    plt.subplot(121)
    plt.imshow(img)
    plt.grid(False)

    plt.subplot(122)
    plt.scatter(im_flat_2D[:,0], im_flat_2D[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
    plt.scatter(centers[:,0], centers[:,1], marker='x')
    plt.axis('square')

    for mean, cov in zip(centers, sigmas):
        make_ellipse(mean, cov, plt.gca(), edgecolor='none', alpha=0.7)
    None
    plt.show()

old_log_likelihood = np.inf
for i in range(100):
    # E step
    log_prob_z_and_x = []
    for k in range(K):
        log_prob_z_and_x.append(stats.multivariate_normal.logpdf(im_flat_2D, mean=mus[k,:], cov=sigmas[k]) +
                                np.log(pis[k]))
    log_prob_z_and_x = np.array(log_prob_z_and_x)

    prob_z_given_x = np.exp(log_prob_z_and_x - np.max(log_prob_z_and_x, 0, keepdims=True))
    Q = prob_z_given_x = prob_z_given_x / prob_z_given_x.sum(0, keepdims=True)

    log_likelihood = np.log(np.exp(log_prob_z_and_x).sum(0)).sum()
    if not i%5:
        display.clear_output()
        print ("Iteration: %d log-likelihood: %g" % (i, log_likelihood))
        make_plot()

    if log_likelihood >= old_log_likelihood:
        pass
        #break
    old_log_likelihood = log_likelihood

    # M step
    w = prob_z_given_x
    pis = w.mean(1)

    for k in range(K):
        w_sum = sum(w[k])
        mus[k] = (im_flat_2D * w[k, :, None]).sum(0) / w_sum
        deviation_from_center = im_flat_2D - mus[k]
        sigmas[k] = deviation_from_center.T.dot(deviation_from_center * w[k, :, None]) / w_sum
        sigmas[k][np.arange(2), np.arange(2)] += reg_covar
```

## Gaussian mixtures in sklearn

The Gaussian mixtures are implemented in many popular toolboxes, notably in `sklearn`.

```
from sklearn import mixture
```

```
gmm = mixture.GaussianMixture(
    n_components=16, covariance_type='full', reg_covar=1e-6)
gmm.fit(im_flat_2D)
```

    GaussianMixture(covariance_type='full', init_params='kmeans', max_iter=100,
                    means_init=None, n_components=16, n_init=1,
                    precisions_init=None, random_state=None, reg_covar=1e-06,
                    tol=0.001, verbose=0, verbose_interval=10, warm_start=False,
                    weights_init=None)

```
plt.subplot(121)
plt.imshow(img)
plt.grid(False)

plt.subplot(122)
plt.scatter(im_flat_2D[:,0], im_flat_2D[:,1], c=im_flat, marker='.', edgecolors='face', alpha=0.3)
plt.scatter(gmm.means_[:,0], gmm.means_[:,1], marker='x')
plt.axis('square')

for mean, cov in zip(gmm.means_, gmm.covariances_):
    make_ellipse(mean, cov, plt.gca(), edgecolor='none', alpha=0.3)
None
```

```

```