# Generating Fractals

Here we will be generating fractals and learning about a few things as well

1. Complex numbers (very breifly)
2. Root finding (this is a good primer for gradient decent for those of you with ML on the mind)
3. Thinking iteratively ( very important nor numerical mathematics)

## Complex Numbers

Before we start talking about fractals, we should probably introduce the idea of a complex number before hand, as the fractas we will be generating rely heavily on them. I'll note that this may get kind of deep into some mathematics, and if you want to skip this section you totally can. There are som quirks to complex numbers, but luckily Python will handle them for you, and realistically they will behave like any other number in Python (for our purposes).

First and foremost, let us introduce the complex number $i$
$$
\begin{equation}
i  = \sqrt{-1} \implies i^2 = -1
\end{equation}
$$

where $i$ is known as the imaginary number, or some impossible number that when multiplied by itself, returns a negative number. At first this might be concerning, but we must remember that when it comes to math, we made it all up anyways, so why not make up another number for fun? However, that line of thinking might be more concerning, so let's move on.

More often than not, we will write a complex number as $z$, which is written as the sum of an imaginary numers real and complex parts,

$$
z = a + ib
$$
where $a$ is the real part of the complex number $z$, and $b$ is the complex or imaginary part.

If you're curious, you can get more details about complex numbers using the drop down below. For our purposes however, all that you really need to know is the following:

1. A complex number $a + ib$ can be thought of as the classical $x,y$ ordered pair we're used to. In this case our $x$ axis is the real part of the number, and our $y$ axis is the imaginary part of the number

2. These $(a, b)$ ordered pairs can be used to define the "complex plane", which we will use to plot our fractals

 More details on complex numbers (not required)

There are a lot of very useful properties that come from imaginary numbers, and they're used all the time in fiels such as pure mathematics, signal processing, physics, chemistry, fluid dynamics... in principle we could go on forever. The largest reason for this is due to something known as [eulers formula](https://en.wikipedia.org/wiki/Euler%27s_formula), where we instead think of complex numbers on the "complex plane", with the real part of the function $a$ representing our $x$ axis, and the imaginary part of our function $b$ on the $y$ axis. Euler's formula states that we can write the exponential of a complex number as follows

$$
e^{ix} = r (\cos \theta + i\sin \theta)
$$
where $r$ is the radius of tihs circle on the complex plane (more on this later).
For those of you mathematically inclined and want to prove this, the easiest way is to write out the Taylor series for each $e^{ix}, i\sin \theta$ and $\cos \theta$, and you may be surprised what you see.

Long story short, using Euler's formula, we can essentially write any complex number in terms of sine and cosine, this is useful for a whole lot of reasons, but for fractals specificially, this implies periodic boundary conditions -- we expect to see repeating patterns. Indeed, as cosine and sine have $N$ roots repeating every $\pi$, we expect that our complex functions may also have (up to) that many roots!

### Finding Roots Of Complex Polynomials

We all remember polynomial equations of real numbers, for example the quadratic function

$$
x^2 - C = 0
$$
where we can all read quite readily that the solution to this equation is $x = \pm \sqrt{C}$. But what about if we have some complex polynomial like

$$
z^2 + C = 0,
$$

To be honest, I pulled a fast one on you. It' just as easy! In this case, we have two roots which are $\pm i \sqrt{C} $, where we just bring our friend the imaginary unit along for the ride. In principle what we do is we once again factor this into the real and imaginary part of the solution, but often it's easier to think of this (in a way that will make mathmaticians cry) we simply get rid of the part we don't like, and call it $i$. For quadratic complex polynomial equations, we can simply use the quadratic formula and sprinkle in the imaginary unit where ever we need it. This is a bit of an over simplification, but for our purposes it should be fine.

Where this can get a little spicier is when the roots aren't obvious enough to be read off. For example, the equation

$$
z^3 = 1,
$$

is cubic, which means we have three distinct roots which satisfy this equation. In this case, it is easier to look at our friend Euler's formula to find these roots. So let's rewrite our complex equation above using euler's formula. First, the left hand side

$$
z = r e^{ix} \implies z^3 = r^3 e^{3ix}
$$

and the right hand side:

$$
1 = re^{ix} = r (\cos \theta + i\sin \theta)
$$

Where, as one has no complex component, we know that this must be one. Therefore, $r$ is equal to one in this equation, angles are those where cosine is one and sine is zero, or  $\theta = 2\pi k$ where $k$ is an integer. Therefore, we have

$$
e^{3ix} = e^{2\pi k}
$$

or by taking the natural logarithm of each side,

$$
ix = \frac{2 \pi k}{3}
$$

And going back to our original equation:

$$
z = e^{ix} = e^{\frac{2 \pi k}{3}}
$$

Where we can take our first three roots as $k = -1, 0, 1$ and obtain

$$
z = 1, e^{2\pi i/3}, e^{-2\pi i/3}
$$

We also note we have periodic roots at integer values of $k$, but we won't worry about those. Knowing these roots are useful in terms of understanding the behaviour of our fractals. We may expect that different roots may cause different basins of convergence, or result in rotations of our fractal. This is also important with respect to establishing the domains in which our fractals may exist.

# Root Finding With Complex Numbers

If you were crazy enough to read the drop down menu, you may have noticed that finding roots to complex equations were more work that the quadratic formula. And if you take anything away from these notebooks it should be that we Data Scientists (and regular scientists) are _super_ lazy. Wouldn't it be nice if we could use our computer to solve these for us? Luckily, the answer is a resounding yes! And even more better is that our Newton Raphson formula from before generalizes to the complex domain without us having to do anything. Convenient!

One thing to be aware of however in Python is we will need to define complex numbers. It's quite simple. For a complex number $ a + ib$, we can define that in python like

```python3
a = b = 1
complex_number = complex(a, b)
```

and we can then throw that number at our root finding routines as we did before and find ourselves solutions.

## My First Fractal: Mandelbrot

Before we go into root finding for fractals, let's start with a relatively simple one to generate the Mandelbrot.  Rather than have me drone on for a year, here's a YouTube video that does a better job than I could explaining that set.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('NGMRB4O922I')
```

# Your Task

Write two functions, the first function will be to calculate the Mandelbrot set as follows

### Mandelbrot Function
1. Initialize $z$ and the number of iterations $n$ as zero,
    - Also intialize the maximum iterations as 80
2. while `abs(z) <= 2 and n < max_iter` do the following
    * $z = z^2 + c$
    * n += 1
3. On exit, return the number of iterations $n$

```python
# note the **kwargs is not strictly necessary, and won'tdo anything here, but will be useful
# if you want to use some of the provided functions later.

def mandelbrot(c, max_iter = 80, **kwargs):
    '''
    here c is any complex number, and max_iter is the maximum numberof iterations through your loop
    you want to go
    '''
    z = # YOUR CODE HERE
    n = # YOUR CODE HERE
    while CONDITION: # YOUR CODE HERE
        z = SOMETHING #YOUR CODE HERE
        n += 1

    return n

```

### Iteration Function

You will also need a function which iterates across the complex plane to see what pixel you should generate there. This function itself will require three functions

1. A scale function to convert pixel location into complex coordinates
2. A scale function to convert the number of itereations in your `mandelbrot` function into an RGB color scale
3. A function which iterates over pixels with a given height and width for your image.

In principle we will have the following pseudocode for point 3 which will encompass the other functions

---
```python
def CreateImage(mandelbrot, height, width, domain):

    ImageMap = np.zeros([width, height])

    for x in range(0, width):
        for y in range(0, height):

            c = scale_function_coordinate(x, y, width, height, real_max, real_min, complex_max, complex_min)

            m = mandelbrot(c)

            color = color_function(m)

            X[x,y] = color

   return ImageMap

```
---
Where let's outline those functions explicitly

##### Pixel Scale Function

Given a pixel coordinate $x$ and $y$, we need to convert this location into a complex number within our domain. the formula for this is as follows

$$
R = R_{min} + \frac{x}{\text{Image Width}} \times (R_{max} - R_{min})
$$

Where $R$ is your value in the real coordinate, $R_{min}$ is the smallest value in the real domain, and $R_{max}$ is the largest value in the real domain, $x$ is the current pixel, and Image Width is the width of the image in pixels you want to create. A similar formula for the complex domain is as follows

$$
C = C_{min} + \frac{x}{\text{Image Height}} \times (C_{max} - C_{min})
$$

Where $C$ is your value in the complex coordinate, $C_{min}$ is the smallest value in the complex domain,  $C_{max}$ is the largest value in the complex domain, $x$ is the current pixel, and Image Height is the height of the image in pixels you want to create.

In this case, if we have a known image size in advance, and we know the domain in which our fractal will exist, we can calculate the complex value $c$ at this place in our image. Please fill in the function below

```python
import numpy as np
def scale_function_coordinate(x, y, width, height, r_max, r_min, c_max, c_min):
    '''
    x --> x coordnate of pixel
    y --> y coordinate of pixel

    width --> width of image
    height --> height of image

    r_max, r_min --> maximum and minimum numbers on thereal axis
    c_max, c_min --> maximum and minimum numbers on the complex axis
    '''

    R = None # YOUR CODE HERE
    C = None # YOUR CODE HERE

    return complex(R, C) # complex is a built in function for complex numbers

```

##### Color Function

Now we need to be able to convert the number of iterations to an RGB coordinate. RGB colors can take values between 0 and 255, so we need to find a way to scale our number of iterations to become some pretty colors so we can observe the changes. Rather than boring you with this, I'll just provide the function

```python
def color(number_of_iterations):
    return 255 - int(m * 255/max_iter)

```

## Putting it All Together

If that worked out well for you, you should be able to fill in the following to create your image functions!

```python
# Boundaries for the mandelbrot function
bounds = [-2, 1, -1, 1]

def CreateImage(function, width, height, bounds):
    r_max, r_min, c_max, c_min = bounds

    if width > 1000:
        print(f'width of {width} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    if height > 1000:
        print(f'height of {height} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    X = np.zeros(width, height)

    for x in range(0, width):
        for y in range(0, height):
            c = scale_function_coordinate(x, y, width, height, r_max, r_min, c_max, c_min)

            # Note here for changes later for root finding
            m = function(c)
            color = color(m)
            X[width, height] = color

    return X

# When you're ready uncomment this line to see if it worked

# plt.imshow(X)
```

If all that worked out, running the above cell should produce what you see below!

```python
import sys
sys.path.append('scripts/')
import fractalfuncs as FF
import matplotlib.pyplot as plt

bounds = [-2, 1, -1, 1]
X = FF.CreateImageMap(function = FF.mandelbrot, function_args = {}, bounds = bounds)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

# Using Rootfinding

Now, rather than using the mandelbrot set, let's try and use our root finding techniques to find roots instead! If we're in a stable region,it should be pretty easy! If not, it will get spicy and diverge. We will use that divergence to create our fractals instead. Here the fractal properties not only come from the mathematical formulation of our complex set, but also the convergence properties of our root finder: different root finding techniques will result in different fractals.

## Your Task

Copy and paste your NewtonRaphson root finder from the Root Finding portion of this, and use it in this assignment. **NOTE** instead of returning the root, you will have to modify your NewtonRaphson function to return the number of iterations it took.

To use root finding, we will do exactly what we did for the mandelbrot set above, however, we will now modify your image generation function to use $c$ as an initial guess at your solution, and see if your NewtonRaphson root finder can find a solution or not. You will need to modify the cell below for use with your own function. Remember that you will also need to pass the derivative and the function you are evaluating (Hint: `**kwargs` can be handy here)

```python
def CreateImageRootFinding(YOUR ARGUMENTS HERE, width, height, bounds):
    r_max, r_min, c_max, c_min = bounds

    if width > 1000:
        print(f'width of {width} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    if height > 1000:
        print(f'height of {height} is too large. Your computer only has so many pixels.')
        print("try zooming in with a smaller boundary to observe more detail")
        return

    X = np.zeros(width, height)

    for x in range(0, width):
        for y in range(0, height):
            INITIAL_GUESS = scale_function_coordinate(x, y, width, height, r_max, r_min, c_max, c_min)

            # Note you may need to wrap this in try/except to prevent accidental zero division/other nastiness
            m = MY_ROOT_FINDER(INITIAL_GUESS)
            color = color(m)
            X[width, height] = color

```

## First Fractal With Root Finding

The mandelbrot set works well for what it is, but alas, if you try to use thet function in root finding, you will find that your fractal is dreadfully boring. A more interesting function is

$$
f(z) = z^3 - 1
$$

Whose derivative is

$$
f^\prime(z) = 3z^2
$$

### Sanity Check

See if you can reproduce the image below with your own function

```python
def function(z):
    return z**3 - 1

def derivative(z):
    return 3 * z ** 2

bounds = [-1,1,-1,1]

newton_args = dict(fprime = derivative, f = function, max_iter = 200, prec = 1e-6)
X = FF.CreateImageMap(FF.NewtonRaphsonFact, newton_args, bounds, height=250, width=250)

plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

Using the function above, try playing around with the following:

1. Different powers of $z$
2. Change the constant (-1) term. Larger/Smaller positive/negative. What if this term is complex?

What do you observe about the fractal at higher powers and different values of the constant

# Other Functions To Try

Once you've got that working, you should try these functions as well and see what fractals you observe!

$$
\begin{aligned}
f(z) &= \sin(z), x \in \left[-\frac{\pi}{2} - \frac{1}{2}, -\frac{\pi}{2} + \frac{1}{2}\right], y\in \left[-0.3, 0.3\right] \\
f(z) &= \cosh(z) - 1, x \in \left[-0.2, 0.2\right], y \in \left[-\pi, -\pi -\frac{\pi}{8}\right]\\
f(z) &= z^3 - 3^z, x\in [-10, 10], y\in[-10, 10]
\end{aligned}
$$

Note that if you don't know how to calculate a derivative, that's okay, you can use wolfrapmalpha, or alternatively, you can take them numerically with a function i've provided. It can be used as follows

```python
# Only if you haven't imported it already
import sys
sys.path.append('scripts/')
import fractalfuncs as FF
def myfunction(z):
    return z**2 # for example

def myderivative(z):
    return FF.nderiv(myfunction, z)
```
I note that numerical derivatives are always worse than analytic ones, but that's okay for now. If anyone is interested I can talk about how that works later as well.

## Bored of that?

If you're bored, you can also try other root finding techniques instead of your newton solver! Here are some suggestions

1. [Secant Method](https://en.wikipedia.org/wiki/Secant_method#:~:text=In%20numerical%20analysis%2C%20the%20secant,difference%20approximation%20of%20Newton's%20method.)
2. [Halley's Method](https://en.wikipedia.org/wiki/Halley%27s_method#:~:text=In%20numerical%20analysis%2C%20Halley's%20method,Householder's%20methods%2C%20after%20Newton's%20method.)
3. [Schroder's Method](https://mathworld.wolfram.com/SchroedersMethod.html)

Note that these fractals are getting created with based on convergence properties of the above solvers. For example, the cells below outline the same function we used originially, just with their different methods of solving!

```python
def function(z):
    return z**3 - 1

def derivative(z):
    return 3 * z ** 2

def secondder(z):
    return 6 * z

bounds = [-1,1,-1,1]
secant_args  = dict(function = function, mult = 0.5)
X = FF.CreateImageMap(FF.secantfact, secant_args, bounds, height=100, width=100)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

```python
bounds = [-1,1,-1,1]
schroder_args = dict(derivative = derivative, function = function,
                     secondder=secondder, prec = 1e-6, max_iter = 50)
X = FF.CreateImageMap(FF.schroderfact, schroder_args, bounds, height=100, width=100)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

```python
bounds = [-1,1,-1,1]
halley_args = dict(derivative = derivative, function = function,
                     seconder=secondder, prec = 1e-6, max_iter = 50)
X = FF.CreateImageMap(FF.halleyfact, halley_args, bounds, height=100, width=100)
plt.imshow(X, extent=bounds)
plt.xlabel("Real Axis", size = 12)
plt.ylabel("Imaginary Axis", size = 12)
plt.show()
```

Where you'll notice each root finding technique has different convergence criteria, so the fractals generated are all slightly different. If you try the other fractals listed, you'll notice tht their fractal patterns will show even more variation


### **Claim**

In the limit, maximizing the likelihood of a parametric distribution whose family contains the true unknown distribution is equivalent to minimizing the forward Kullback-Leibler divergence between them.

---

Let $P(X)$ be an unknown distribution producing observations $x$ with a pdf $p(x)$. We wish to approximate the pdf $p(x)$ of with a parametric family $q_{\theta}(x)$. We assume that the true distribution is within the family described by $q_{\theta}(x)$.

The parameters $\theta_{ML}$ of the approximating distribution $q_{\theta_{ML}}$ are defined with the maximum likelihood estimator

$$\theta_{ML} = \max_{\theta} \frac{1}{m}\sum_{i=1}^m \log q_{\theta}(x_i)$$

where $x_i \sim P$ and $i \in \{1, \dots, m\}$.

The approximating distribution $q_{\theta_{KL}}$ is defined by minimizing the KL divergence between $p(x)$ and $q_{\theta}(x)$.

$$\theta_{KL} = \min_{\theta} KL(P||Q_{\theta}) = \min_{\theta} \int_x p(x) \log \frac{p(x)}{q_{\theta}(x)} dx$$

where here we use the forward KL divergence.

If we take the limit as $m \rightarrow \infty$ then $\theta_{ML}$ found by maximizing the likelihood is the same as $\theta_{KL}$ found by minimizing the forward KL divergence.

\begin{align}
\lim_{m \rightarrow \infty} \theta_{ML} &= \lim_{m \rightarrow \infty} \max_{\theta} \frac{1}{m}\sum_{i=1}^m \log q_{\theta}(x_i) \\
&= \max_{\theta} \int_x p(x) \log q_{\theta}(x) dx \\
&= \min_{\theta} \bigg [ -\int_x p(x) \log q_{\theta}(x) dx \bigg] \\
&= \min_{\theta} \bigg [ \int_x p(x) \log p(x) dx -\int_x p(x) \log q_{\theta}(x) dx \bigg] \\
&= \min_{\theta} \bigg [ \int_x p(x) \log \frac{p(x)}{q_{\theta}(x)} dx \bigg] \\
&= \min_{\theta} KL(P||Q_{\theta}) \\
&= \theta_{KL}
\end{align}

```python

```

Maximizing Liklihood and Minimizing KL Divergence.ipynb


# How

## Calculate the derivative of an expression.

We can calculate the derivative of an expression using `sympy.diff` which takes,
an expression, a variable and a degree.

````{tip}
```
sympy.diff(expression, variable, degree=1)
```
````

The default value of `degree` is 1.

For example to compute $\frac{d (4 x ^ 3 + 2 x + 1}{dx}$:

```python
import sympy as sym

x = sym.Symbol("x")
expression = 4 * x ** 3 + 2 * x + 1
sym.diff(expression, x)
```

$\displaystyle 12 x^{2} + 2$

To compute the second derivative: $\frac{d ^ 2 (4 x ^ 3 + 2 x + 1}{dx ^ 2}$

```python
sym.diff(expression, x, 2)
```

$\displaystyle 24 x$

## Calculate the indefinite integral of an expression.

We can calculate the indefinite integral of an expression using
`sympy.integrate`. Which takes an expression and a variable.

````{tip}
```
sympy.integrate(expression, variable)
```
````

For example to compute $\int 4x^3 + 2x + 1 dx$:

```python
sym.integrate(expression, x)
```

$\displaystyle x^{4} + x^{2} + x$

## Calculate the definite integral of an expression.

We can calculate the definite integral of an expression using
`sympy.integrate`. The first argument is an expression but instead of passing a
variable as the second argument we pass a tuple with the variable and the upper
and lower bounds of integration.

````{tip}
```
sympy.integrate(expression, (variable, lower_bound, upper_bound))
```
````

For example to compute $\int_0^4 4x^3 + 2x + 1 dx$:

```python
sym.integrate(expression, (x, 0, 4))
```

$\displaystyle 276$

## Use $\infty$

In `sympy` we can access $\infty$ using `sym.oo`:

````{tip}
```
sympy.oo
```
````

For example:

```python
sym.oo
```

$\displaystyle \infty$

## Calculate limits

We can calculate limits using `sympy.limit`. The first argument is the
expression, then the variable and finally the expression the variable tends to.

````{tip}
```
sympy.limit(expression, variable, value)
```
````

For example to compute $\lim_{h \to 0} \frac{4 x ^ 3 + 2 x + 1 - 4(x - h)^3 - 2(x - h) - 1}{h}$:

```python
h = sym.Symbol("h")
expression = (4 * x ** 3 + 2 * x + 1 - 4 * (x - h) ** 3 - 2 * (x - h) - 1) / h
sym.limit(expression, h, 0)
```

$\displaystyle 12 x^{2} + 2$


# Integer Programming

* All decision variables are integers

\begin{align}
\text{maximize}\  & \mathbf{c}^T\mathbf{x} \\
\text{subject to } & \\
& A\mathbf{x} &&\leq \mathbf{b} \\
& \mathbf{x} &&\geq 0 \\
& \mathbf{x} \in \mathbb{Z}^n
\end{align}

* Binary integer programming: Variables are restricted to be either 0 or 1

# Binary Knapsack Problem
* Combinatorial optimization problem
* Problem of packing the most valuable or useful items without overloading the luggage.
    * A set of items ($N$ items), each with a weight($w$) and a value($v$)
    * Fixed capacity
    * Maximize the total value possible

## Problem Formulation
\begin{align}
\text{maximize}\  & \sum_{i=0}^{N-1}v_{i}x_{i} \\
\text{subject to } & \\
& \sum_{i=0}^{N-1}w_{i}x_{i} & \leq C \\
& x_i \in \{0,1\} & \forall i=0,\dots,N-1
\end{align}

## Coding in Python

## Creating the data (weights and values)

```python
w = [4,2,5,4,5,1,3,5]
v = [10,5,18,12,15,1,2,8]
C = 15
N = len(w)
```

## Step 2: Importing gurobipy package

```python
from gurobipy import *
```

## Step 3: Create an optimization model

```python
knapsack_model = Model('knapsack')
```

## Step 4: Add multiple binary decision variables

Adds multiple decision variables and stores them in the model.
```python
addVars(*indices,
        lb=0.0,
        ub=float('inf'),
        obj=0.0,
        vtype=GRB.CONTINUOUS,
        name=""
```

```python
x = knapsack_model.addVars(N, vtype = GRB.BINARY, name="x")
```

## Step 5: Add the constraints

$$\sum_{i=1}^{N} w_{i}x_{i} \leq C$$

```python
# \sum_{i=1}^{N} w_{i}*x_{i} <= C
knapsack_model.addConstr(sum(w[i]*x[i] for i in range(N)) <= C)
```

## Step 6: Define the objective function

$$\sum_{i=1}^{N} v_{i}x_{i}$$

```python
# \sum_{i=1}^{N} v_{i}*x_{i}
obj_fn = sum(v[i]*x[i] for i in range(N))
knapsack_model.setObjective(obj_fn, GRB.MAXIMIZE)
```

## Step 7: Solve the model and output the solution

```python
knapsack_model.optimize()
print('Optimization is done. Objective Function Value: %.2f' % knapsack_model.objVal)
# Get values of the decision variables

for v in knapsack_model.getVars():
    print('%s: %g' % (v.varName, v.x))
```

```python

```


# Modeling and Simulation in Python

Chapter 9

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()
```

The following displays SymPy expressions and provides the option of showing results in LaTeX format.

```python
from sympy.printing import latex

def show(expr, show_latex=False):
    """Display a SymPy expression.

    expr: SymPy expression
    show_latex: boolean
    """
    if show_latex:
        print(latex(expr))
    return expr
```

### Analysis with SymPy

Create a symbol for time.

```python
t = symbols('t')
```

If you combine symbols and numbers, you get symbolic expressions.

```python
expr = t + 1
```

The result is an `Add` object, which just represents the sum without trying to compute it.

```python
type(expr)
```

    sympy.core.add.Add

`subs` can be used to replace a symbol with a number, which allows the addition to proceed.

```python
expr.subs(t, 2)
```

`f` is a special class of symbol that represents a function.

```python
f = Function('f')
```

    f

The type of `f` is `UndefinedFunction`

```python
type(f)
```

    sympy.core.function.UndefinedFunction

SymPy understands that `f(t)` means `f` evaluated at `t`, but it doesn't try to evaluate it yet.

```python
f(t)
```

`diff` returns a `Derivative` object that represents the time derivative of `f`

```python
dfdt = diff(f(t), t)
```

```python
type(dfdt)
```

    sympy.core.function.Derivative

We need a symbol for `alpha`

```python
alpha = symbols('alpha')
```

Now we can write the differential equation for proportional growth.

```python
eq1 = Eq(dfdt, alpha*f(t))
```

And use `dsolve` to solve it.  The result is the general solution.

```python
solution_eq = dsolve(eq1)
```

We can tell it's a general solution because it contains an unspecified constant, `C1`.

In this example, finding the particular solution is easy: we just replace `C1` with `p_0`

```python
C1, p_0 = symbols('C1 p_0')
```

```python
particular = solution_eq.subs(C1, p_0)
```

In the next example, we have to work a little harder to find the particular solution.

### Solving the quadratic growth equation

We'll use the (r, K) parameterization, so we'll need two more symbols:

```python
r, K = symbols('r K')
```

Now we can write the differential equation.

```python
eq2 = Eq(diff(f(t), t), r * f(t) * (1 - f(t)/K))
```

```python
diff(f(t),t)
```

And solve it.

```python
solution_eq = dsolve(eq2)
```

The result, `solution_eq`, contains `rhs`, which is the right-hand side of the solution.

```python
general = solution_eq.rhs
```

We can evaluate the right-hand side at $t=0$

```python
at_0 = general.subs(t, 0)
```

Now we want to find the value of `C1` that makes `f(0) = p_0`.

So we'll create the equation `at_0 = p_0` and solve for `C1`.  Because this is just an algebraic identity, not a differential equation, we use `solve`, not `dsolve`.

The result from `solve` is a list of solutions.  In this case, [we have reason to expect only one solution](https://en.wikipedia.org/wiki/Picard%E2%80%93Lindel%C3%B6f_theorem), but we still get a list, so we have to use the bracket operator, `[0]`, to select the first one.

```python
solutions = solve(Eq(at_0, p_0), C1)
type(solutions), len(solutions)
```

    (list, 1)

```python
value_of_C1 = solutions[0]
```

Now in the general solution, we want to replace `C1` with the value of `C1` we just figured out.

```python
particular = general.subs(C1, value_of_C1)
```

The result is complicated, but SymPy provides a method that tries to simplify it.

```python
particular = simplify(particular)
```

Often simplicity is in the eye of the beholder, but that's about as simple as this expression gets.

Just to double-check, we can evaluate it at `t=0` and confirm that we get `p_0`

```python
particular.subs(t, 0)
```

This solution is called the [logistic function](https://en.wikipedia.org/wiki/Population_growth#Logistic_equation).

In some places you'll see it written in a different form:

$f(t) = \frac{K}{1 + A e^{-rt}}$

where $A = (K - p_0) / p_0$.

We can use SymPy to confirm that these two forms are equivalent.  First we represent the alternative version of the logistic function:

```python
A = (K - p_0) / p_0
```

```python
logistic = K / (1 + A * exp(-r*t))
```

To see whether two expressions are equivalent, we can check whether their difference simplifies to 0.

```python
simplify(particular - logistic)
```

This test only works one way: if SymPy says the difference reduces to 0, the expressions are definitely equivalent (and not just numerically close).

But if SymPy can't find a way to simplify the result to 0, that doesn't necessarily mean there isn't one.  Testing whether two expressions are equivalent is a surprisingly hard problem; in fact, there is no algorithm that can solve it in general.

### Exercises

**Exercise:** Solve the quadratic growth equation using the alternative parameterization

$\frac{df(t)}{dt} = \alpha f(t) + \beta f^2(t) $

```python
# Solution goes here
beta= symbols('Beta')
```

```python
# Solution goes here
eq3 = Eq(diff(f(t), t), alpha * f(t) + beta *f(t)**2)
```

```python
# Solution goes here
solution_2 = dsolve(eq3)
```

**Exercise:**  Use [WolframAlpha](https://www.wolframalpha.com/) to solve the quadratic growth model, using either or both forms of parameterization:

    df(t) / dt = alpha f(t) + beta f(t)^2

or

    df(t) / dt = r f(t) (1 - f(t)/K)

Find the general solution and also the particular solution where `f(0) = p_0`.

Results from the website

alternate1:   f'(t) = -(r f(t) (f(t) - K))/K
alternate2:   f'(t) = r (f(t) - f(t)^2/K)

General solution:  f(t) = (ξ e^(c_1 ξ + r t))/(e^(c_1 ξ + r t) - 1)


(nm_linear_algebra_intro)=
# Linear algebra introduction

## Linear (matrix) systems

We can re-write a system of simultaneous (linear) equations in a matrix form. For example, let's consider:

\\[\begin{eqnarray*}
  2x + 3y &=& 7 \\\\\\
   x - 4y &=& 3
\end{eqnarray*}\\]

It can be rewritten in a matrix form:

\\[
\left(
  \begin{array}{rr}
    2 & 3 \\\\\\
    1 & -4  \\\\\\
  \end{array}
\right)\left(
  \begin{array}{c}
    x \\\\\\
    y \\\\\\
  \end{array}
\right) = \left(
  \begin{array}{c}
    7 \\\\\\
    3 \\\\\\
  \end{array}
\right)
\\]

We understand that this system always has the form of

\\[
\left(
  \begin{array}{rr}
    a & b \\\\\\
    c & d  \\\\\\
  \end{array}
\right)\left(
  \begin{array}{c}
    x \\\\\\
    y \\\\\\
  \end{array}
\right) = \left(
  \begin{array}{c}
    e \\\\\\
    f \\\\\\
  \end{array}
\right),
\\]

where \\(a,b,c,d,e,f\\) are arbitrary constants.

Let's call the matrix which stores the coefficients of our system of linear equations to be \\(A\\)

\\[
A=
\left(
  \begin{array}{rr}
    a & b \\\\\\
    c & d  \\\\\\
  \end{array}
\right)
\\]

and the matrix that contains our variables to be \\(\mathbf{x}\\)

\\[
\mathbf{x}=
\left(
  \begin{array}{c}
    x \\\\\\
    y  \\\\\\
  \end{array}
\right).
\\]

The matrix that contains the results of our system of linear equation will be called \\(\mathbf{b}\\)

\\[
\mathbf{b}=
\left(
  \begin{array}{c}
    e \\\\\\
    f  \\\\\\
  \end{array}
\right).
\\]

This system of equations can be represented as the matrix equation
\\[A\pmb{x}=\pmb{b}.\\]

More generally, consider an arbitrary system of \\(n\\) linear equations for \\(n\\) unknowns

\\[
\begin{eqnarray*}
  A_{11}x_1 + A_{12}x_2 + \dots + A_{1n}x_n &=& b_1 \\\\\\
  A_{21}x_1 + A_{22}x_2 + \dots + A_{2n}x_n &=& b_2 \\\\\\
  \vdots &=& \vdots \\\\\\
  A_{n1}x_1 + A_{n2}x_2 + \dots + A_{nn}x_n &=& b_n
\end{eqnarray*}
\\]

where \\(A_{ij}\\) are the constant coefficients of the linear system, \\(x_j\\) are the unknown variables, and \\(b_i\\)
are the terms on the right hand side (RHS). Here the index \\(i\\) is referring to the equation number
(the row in the matrix below), with the index \\(j\\) referring to the component of the unknown
vector \\(\pmb{x}\\) (the column of the matrix).

This system of equations can be represented as the matrix equation \\(A\pmb{x}=\pmb{b}\\):

\\[
\left(
  \begin{array}{cccc}
    A_{11} & A_{12} & \dots & A_{1n} \\\\\\
    A_{21} & A_{22} & \dots & A_{2n} \\\\\\
    \vdots & \vdots & \ddots & \vdots \\\\\\
    A_{n1} & A_{n2} & \dots & A_{nn} \\\\\\
  \end{array}
\right)\left(
         \begin{array}{c}
           x_1 \\\\\\
           x_2 \\\\\\
           \vdots \\\\\\
           x_n \\\\\\
         \end{array}
       \right)  = \left(
                   \begin{array}{c}
                     b_1 \\\\\\
                     b_2 \\\\\\
                     \vdots \\\\\\
                     b_n \\\\\\
                   \end{array}
                 \right)
\\]

We can easily solve the above \\(2 \times 2\\) example of two equations and two unknowns using substitution (e.g. multiply the second equation by 2 and subtract the first equation from the resulting equation to eliminate \\(x\\) and hence allowing us to find \\(y\\), then we could compute \\(x\\) from the first equation). We find:

\\[ x=\frac{37}{11}, \quad y=\frac{1}{11}.\\]

```{margin} Note
Cases where the matrix is non-square, i.e. of shape \\(m \times n\\) where \\(m\ne n\\) correspond to the over- or under-determined systems where you have more or less equations than unknowns.
```

Example systems of \\(3\times 3\\) are a little more complicated but doable. In this notebook, we consider the case of \\(n\times n\\), where \\(n\\) could be billions (e.g. in AI or machine learning).

## Matrices in Python

We can use `numpy.arrays` to store matrices. The convention for one-dimensional vectors is to call them column vectors and have shape \\(n \times 1\\). We can extend to higher dimensions through the introduction of matrices as two-dimensional arrays (more generally vectors and matrices are just two examples of {ref}`tensors <tensor_review>`).

We use subscript indices to identify each component of the array or matrix, i.e. we can identify each component of the vector \\(\pmb{v}\\) by \\(v_i\\), and each component of the matrix \\(A\\) by \\(A_{ij}\\).

The dimension or shape of a vector/matrix is the number of rows and columns it posesses, i.e. \\(n \times 1\\) and \\(m \times n\\) for the examples above. Here is an example of how we can extend our use of the `numpy.array` to two dimensions in order to define a matrix \\(A\\).

```python
import numpy as np

A = np.array([[10., 2., 1.],
              [6., 5., 4.],
              [1., 4., 7.]])

print(A)
```

    [[10.  2.  1.]
     [ 6.  5.  4.]
     [ 1.  4.  7.]]

Check total size of the array storing matrix \\(A\\). It will be \\(3\times3=9\\):

```python
print(np.size(A))
```

    9

Check the number of dimensions of matrix \\(A\\):

```python
print(np.ndim(A))
```

    2

Check the shape of the matrix \\(A\\):

```python
print(np.shape(A))
```

    (3, 3)

Transpose matrix \\(A\\):

```python
print(A.T)
```

    [[10.  6.  1.]
     [ 2.  5.  4.]
     [ 1.  4.  7.]]

Get the inverse of matrix \\(A\\):

```python
import scipy.linalg as sl

print(sl.inv(A))
```

    [[ 0.14285714 -0.07518797  0.02255639]
     [-0.28571429  0.51879699 -0.2556391 ]
     [ 0.14285714 -0.28571429  0.28571429]]

Get the determinant of matrix \\(A\\):

```python
print(sl.det(A))
```

    133.00000000000003

````{margin}
Normal `*` operator does operations element-wise, which we do not want!!!
```python

print(A*sl.inv(A))
```
[[ 1.42857143 -0.15037594  0.02255639]
 [-1.71428571  2.59398496 -1.02255639]
 [ 0.14285714 -1.14285714  2.        ]]

````

Multiply \\(A\\) with its inverse using the `@` matrix multiplication operator. Note that due to roundoff errors the off diagonal values are not exactly zero:

```python
print(A @ sl.inv(A))
```

    [[ 1.00000000e+00 -1.66533454e-16  0.00000000e+00]
     [ 1.11022302e-16  1.00000000e+00  2.22044605e-16]
     [-2.77555756e-17 -1.66533454e-16  1.00000000e+00]]

Another way of multiplying matrices is to use `np.dot` function:

```python
print(np.dot(A, sl.inv(A)))
print("\n")
print(A.dot(sl.inv(A)))
```

    [[ 1.00000000e+00 -1.66533454e-16  0.00000000e+00]
     [ 1.11022302e-16  1.00000000e+00  2.22044605e-16]
     [-2.77555756e-17 -1.66533454e-16  1.00000000e+00]]

    [[ 1.00000000e+00 -1.66533454e-16  0.00000000e+00]
     [ 1.11022302e-16  1.00000000e+00  2.22044605e-16]
     [-2.77555756e-17 -1.66533454e-16  1.00000000e+00]]

Initialise vector and matrix of zeros:

```python
print(np.zeros(3))
print("\n")
print(np.zeros((3,3)))
```

    [0. 0. 0.]

    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]

Initialise identity matrix:

```python
print(np.eye(3))
```

    [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]

### Matrix objects

Note that NumPy has a matrix object.  We can cast the above two-dimensional arrays into matrix objects and then the star operator does yield the expected matrix product:

```python
A = np.array([[10., 2., 1.],
              [6., 5., 4.],
              [1., 4., 7.]])

print(type(A))
print(type(np.mat(A)))
```

    <class 'numpy.ndarray'>
    <class 'numpy.matrix'>

```python
print(np.mat(A)*np.mat(sl.inv(A)))
```

    [[ 1.00000000e+00 -1.66533454e-16  0.00000000e+00]
     [ 1.11022302e-16  1.00000000e+00  2.22044605e-16]
     [-2.77555756e-17 -1.66533454e-16  1.00000000e+00]]

### Slicing
We can use slicing to extract components of matrices:

```python
# Single entry, first row, second column
print(A[0,1])

# First row
print(A[0,:])

# last row
print(A[-1,:])

# Second column
print(A[:,1])

# Extract a 2x2 sub-matrix
print(A[1:3,1:3])
```

    2.0
    [10.  2.  1.]
    [1. 4. 7.]
    [2. 5. 4.]
    [[5. 4.]
     [4. 7.]]

## Exercises

### Solving a linear system

Let's quickly consider the \\(2 \times 2\\) case from the beginning of the notebook that we claimed the solution for to be

\\[x=\frac{37}{11} \quad\text{and}\quad y=\frac{1}{11}.\\]

To solve the matrix equation

\\[ A\pmb{x}=\pmb{b}\\]

we can simply multiply both sides by the inverse of the matrix \\(A\\) (if \\(A\\) is [invertible](https://en.wikipedia.org/wiki/Invertible_matrix)):

\\[
\begin{align}
A\pmb{x} & = \pmb{b}\\\\\\
\implies A^{-1}A\pmb{x} & = A^{-1}\pmb{b}\\\\\\
\implies I\pmb{x} & = A^{-1}\pmb{b}\\\\\\
\implies \pmb{x} & = A^{-1}\pmb{b}
\end{align}
\\]

so we can find the solution \\(\pmb{x}\\) by multiplying the inverse of \\(A\\) with the RHS vector \\(\pmb{b}\\).

```python
A = np.array([[2., 3.],
              [1., -4.]])

# Check first whether the determinant of A is non-zero
print("Det A = ", sl.det(A))

b = np.array([7., 3.])

# Compute A inverse and multiply by b
print("A^-1 @ b =", sl.inv(A) @ b)
```

    Det A =  -11.0
    A^-1 @ b = [3.36363636 0.09090909]

We can solve the system using `scipy.linalg.solve`:

```python
print("A^-1 @ b =", sl.solve(A,b))
```

    A^-1 @ b = [3.36363636 0.09090909]

Check if the solutions match:

```python
print(np.allclose(np.array([37./11., 1./11.]), sl.solve(A,b)))
```

    True

### Matrix multiplication

Let
\\[
A = \left(
  \begin{array}{ccc}
    1 & 2 & 3 \\\\\\
    4 & 5 & 6 \\\\\\
    7 & 8 & 9 \\\\\\
  \end{array}
\right)
\mathrm{\quad\quad and \quad\quad}
b = \left(
  \begin{array}{c}
    2 \\\\\\
    4 \\\\\\
    6 \\\\\\
  \end{array}
\right)
\\]

We will store \\(A\\) and \\(b\\) in NumPy arrays. We will create NumPy array \\(I\\) containing the identity matrix \\(I_3\\) and perform \\(A = A+I\\). Then we will substitute third column of \\(A\\) with \\(b\\). We will solve \\(Ax=b\\).

```python
A = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])
b = np.array([2, 4, 6])
print("A =", A)
print("b = ",b)

print("Size of A: ", A.size," and shape of A: ",A.shape)
print("Size of b: ", b.size," and shape of b: ",b.shape)

I = np.eye(3)
print("I = ",I)
A = A + I
print("A = ",A)

A[:, 2] = b
print("A = ",A)

x = sl.solve(A,b)
print("x = ", x)
```

    A = [[1 2 3]
     [4 5 6]
     [7 8 9]]
    b =  [2 4 6]
    Size of A:  9  and shape of A:  (3, 3)
    Size of b:  3  and shape of b:  (3,)
    I =  [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]
    A =  [[ 2.  2.  3.]
     [ 4.  6.  6.]
     [ 7.  8. 10.]]
    A =  [[2. 2. 2.]
     [4. 6. 4.]
     [7. 8. 6.]]
    x =  [3.80647894e-17 7.77156117e-17 1.00000000e+00]

## Matrix properties

Consider \\(N\\) linear equations in \\(N\\) unknowns, \\(A\pmb{x}=\pmb{b}\\).

this system has a unique solution provided that the determinant of \\(A\\), \\(\det(A)\\), is non-zero. In this case the matrix is said to be non-singular.

If \\(\det(A)=0\\) (with \\(A\\) then termed a singular matrix), then the linear system does not have a unique solution, it may have either infinite or no solutions.

For example, consider

\\[
\left(
  \begin{array}{rr}
    2 & 3 \\\\\\
    4 & 6  \\\\\\
  \end{array}
\right)\left(
  \begin{array}{c}
    x \\\\\\
    y \\\\\\
  \end{array}
\right) = \left(
  \begin{array}{c}
    4 \\
    8 \\
  \end{array}
\right).
\\]

The second equation is simply twice the first, and hence a solution to the first equation is also automatically a solution to the second equation.

We only have one linearly-independent equation, and our problem is under-constrained - we effectively only have one eqution for two unknowns with infinitely many possibly solutions.

If we replaced the RHS vector with \\((4,7)^T\\), then the two equations would be contradictory - in this case we have no solutions.

Note that a set of vectors where one can be written as a linear sum of the others are termed linearly-dependent. When this is not the case the vectors are termed linearly-independent.

```{admonition} The following properties of a square \\(n\times n\\) matrix are equivalent:

* \\(\det(A)\ne 0\implies\\) A is non-singular
* the columns of \\(A\\) are linearly independent
* the rows of \\(A\\) are linearly independent
* the columns of \\(A\\) span \\(n\\)-dimensional space (we can reach any point in \\(\mathbb{R}^N\\) through a linear combination of these vectors)
* \\(A\\) is invertible, i.e. there exists a matrix \\(A^{-1}\\) such that \\(A^{-1}A = A A^{-1}=I\\)
* the matrix system \\(A\pmb{x}=\pmb{b}\\) has a unique solution for every vector \\(b\\)

```


## Definition

Given the parametric equation of a logarithmic spiral:

$$
x(t) = ae^{bt} \cos(t)
\\
y(t) = ae^{bt} \sin(t)
$$

Then the curvature $\kappa$ of the spiral is given by:

$$
\kappa (t) = \frac{ e^{-bt} } {a\sqrt{1 + b^2}}
$$

where $t$ is the tangential angle.

## Objective

Given the general definition of the curvature of a parametric curve in the xy plane:

$$
\kappa = \left \| \frac{d\mathbf{T}}{dt} \right \| = \frac{| \dot{x}\ddot{y} - \dot{y}\ddot{x} |}{{(\dot{x}^2 + \dot{y}^2)} ^ { \frac{3}{2} }}
$$

Recover the above curvature equation of a logarithmic spiral using sympy.

```python
from sympy import *
init_printing()

# Define variables
x, y, t = symbols('x y t')
a, b = symbols('a b')

# Spiral equation
x = a * exp(b*t) * cos(t)
y = a * exp(b*t) * sin(t)

x, y
```

```python
# Get derivatives
xd = diff(x, t)
yd = diff(y, t)
xdd = diff(xd, t)
ydd = diff(yd, t)

# Define the true curvature equation
true_curvature = exp(- b * t) / (a * sqrt(1 + b**2))
true_curvature
```

```python
# Compute curvature
curvature = Abs(xd * ydd - yd * xdd) / (xd**2 + yd**2)**(3/2)
curvature
```

```python
curvature = curvature.simplify()
curvature
```

```python
args = {'a': 14, 'b': 5, 't': 0.2}
true_curvature.subs(args).evalf()
```

```python
curvature.subs(args).evalf()
```

```python
expand(true_curvature - curvature)
```

The newly computed `curvature` expression give the same result as the `true_curvature` expression given in the textbooks.

Analysis/Notebooks/Spiral Dataset/Spiral Curvature Calculus.ipynb


# 2/ Linearity

```python
# setup SymPy
from sympy import *
init_printing()
x, y, z, t = symbols('x y z t')
alpha, beta = symbols('alpha beta')
```

## Simplest linear function

```python
b, m = symbols('b m')

def f(x):
    return m*x
```

```python
f(1)
```

```python
f(2)
```

```python
f(1+2)
```

```python
f(1) + f(2)
```

```python
expand(f(x+y)) ==  f(x) + f(y)
```

    True

## What about vector inputs?

```python
m_1, m_2 = symbols('m_1 m_2')

def T(vec):
    """A function that takes a 2D vector and returns a number."""
    return m_1*vec[0] + m_2*vec[1]
```

```python
u_1, u_2 = symbols('u_1 u_2')
u = Matrix([u_1,u_2])
v_1, v_2 = symbols('v_1 v_2')
v = Matrix([v_1,v_2])
```

```python
T(u)
```

```python
T(v)
```

```python
T(u) + T(v)
```

```python
expand( T(u+v) )
```

```python
simplify( T(alpha*u + beta*v) - alpha*T(u) - beta*T(v) )
```

# Linear transformations

A linear transformation is function that takes vectors as inputs, and produces vectors as outputs:

$$
   T: \mathbb{R}^n \to \mathbb{R}^m.
$$

See [page 136](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=43) in v2.2 of the book.

```python
m_11, m_12, m_21, m_22 = symbols('m_11 m_12 m_21 m_22')

def T(vec):
    """A linear transformations R^2 --> R^2."""
    out_1 = m_11*vec[0] + m_12*vec[1]
    out_2 = m_21*vec[0] + m_22*vec[1]
    return Matrix([out_1, out_2])
```

```python
T(u)
```

```python
T(v)
```

```python
T(u+v)
```

## Linear transformations as matrix-vector products

See [page 133](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=40) in v2.2 of the book.

```python
def T_impl(vec):
    """A linear transformations implemented as matrix-vector product."""
    M_T = Matrix([[m_11, m_12],
                  [m_21, m_22]])
    return M_T*vec
```

```python
T_impl(u)
```

```python

```


# Mathematical Induction

The simple case of 'weak' induction.

1. Prove the Base Case k=b
2. Prove that, assuming it holds for some case $k$, it also holds for $k+1$

- It holds for $k=b$. (step 1)
- If it holds for $k=b$ then it should hold for $k=b+1$. (step 2)

- Therefore, it holds for $k=b+1$. (step 1 + step 2)

- If it holds for $b+1$ then it holds for $b+2$. (step 2)

- If it holds for $b+2$ then it holds for $b+3$. (step 2)
.
.
.

And so on.

### Example:

The proposition is that:

\begin{equation}
P(n) = \sum_{i=0}^{n-1} b^i = \frac{1-b^n}{1-b}
\end{equation}

#### Base Case: Prove that it Holds for n = 1

\begin{equation}
P(1) = \sum_{i=0}^{1-1} b^i = b^0 = \frac{1-b^1}{1-b} = 1
\end{equation}

This proves the base case.

#### Step Case: Prove that if it holds for n=k then it also holds for n=k+1

Assume the following to be true:
\begin{equation}
P(n) = \frac{1-b^n}{1-b}
\end{equation}

\begin{equation}
P(k+1) = P(k) + b^{k} =  \frac{1-b^k}{1-b} + \frac{(1-b)b^{k}}{1-b} = \frac{1-b^k+b^k+b^{k+1}}{1-b} = \frac{1-b^{k+1}}{1-b}
\end{equation}

This proves the step case.


The Relative Frequency of any random variable is the number of occurance in the total number of observation.
The Relative Frequency is calculated as:

\begin{equation}
Relative Frequency = \frac{Frequency}{Total\ number\  of\ observations}
\end{equation}

E.g. We have a samples are like { 5,7,11,19,23,5,18,7,18,23 }. If we calculate the relative frequency of each unique values then it will be as:

${R.F of (5)} = {\frac {2}{10}}$

${R.F of (7)} = {\frac {2}{10}}$

${R.F of (11)} = {\frac {1}{10}}$

${R.F of (18)} = {\frac {2}{10}}$

${R.F of (19)} = {\frac {1}{10}}$

${R.F of (23)} = {\frac {2}{10}}$

Whereas, Probability is actually the limiting case of the  Relative Frequency  when the sample approaches(limits) towards population.

Going to demonstrate that probability is actually the limiting case of relative frequency when the sample slowly approaches population.

Lets first take an example of binomial random variable where we are taking a sample of observations (responses) arrising due to repititive conduction of a binomial experiment. Which means that we are going to perform some experiment 'N' number of times where each each conduction of experiment (trial) will result in an obervation or response of a trial out of two possible responses. Lets say that our binomial trial is the toss of a coin where each toss is going to give us one of the two responses : Either Heads or Tails.

As we are in statistical domain, we will evaluate the relative frequencies of heads as well as tails as:

\begin{equation}
r = \frac{h}{N}
\end{equation}

Where, r = Relative Frequency.
       h = Number of Binomial Experiments in which heads was the outcome.
       N = Total number of times binomial experiment has been conducted.

```python
# Importing the Numpy library alias np and matplotlib library alias plt
import numpy as np
import matplotlib.pyplot as plt
```

Going to conduct a binomial experiment (having two results either Head or Tail) for N=10 times and considering that the coin used in coin tossing is an unbiased coin, i.e. p=q=0.5

```python
N = 10
psuccess = 0.5
qfailure = 1-psuccess
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    4

We tossed an unbiased coin 10 times, we got only 4 heads. So, the relative frequency is given by:

\begin{equation}
r = \frac{h}{N} = \frac{4}{10} = 0.4
\end{equation}

Let's toss an unbiased coin 100 times and see what happens.

```python
N = 100
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    54

We tossed a coin 100 times, we got 54 heads. So, the relative frequency is given by:

\begin{equation}
r = \frac{h}{N} = \frac{54}{100} = 0.54
\end{equation}

Let's toss an unbiased coin 10000 times and see what happens.

```python
N = 10000
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    4972

We tossed a coin 10000 times, we got 4972 heads. So, the relative frequency is given by:

\begin{equation}
r = \frac{h}{N} = \frac{4972}{10000} = 0.4972
\end{equation}

Let's toss a coin 100000 times and see what happens.

```python
N = 100000
ExperimentOutcomes = np.random.binomial(N,psuccess)
```

```python
ExperimentOutcomes
```

    49991

We tossed a coin 100000 times, we got 49991 heads. So the relative frequency is given by:

\begin{equation}
r = \frac{h}{N} = \frac{49991}{100000} = 0.49991
\end{equation}

The theoretical answer to the probability of getting heads in a single toss is 0.5 and we can observe that as we are increasing the number of tosses, we are approaching the theoretical value of probability in terms of relative frequency.

This phenomena can also be shown by plotting relative frequencies of heads with increasing sample sizes which are slowly approaching population.

```python
#we are trying to get the values from the lower limit=10, higher limit=500
#in the interval of 10.
Ns = np.arange(10,500,10)
```

```python
Ns
```

    array([ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100, 110, 120, 130,
           140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250, 260,
           270, 280, 290, 300, 310, 320, 330, 340, 350, 360, 370, 380, 390,
           400, 410, 420, 430, 440, 450, 460, 470, 480, 490])

We have generated different sample sizes at the gap of 10 tosses.

```python
ExperimentOutcomes = np.random.binomial(Ns,psuccess)
```

We have tossed an unbiased coin at different values of total tosses(N) and recorded the number of heads(h) during several number of total tosses.

```python
RelativeFrequencies = ExperimentOutcomes/Ns
```

We have now calculated relative frequencies by dividing the number of heads(h) which we observed during different values of N.

```python
plt.stem(Ns,RelativeFrequencies)
```

As we can observe from above plot that as we are tossing a coin more number of times, the relative frequency of heads is approaching towards psuccess = 0.5.

Now, let's change the psuccess to 0.65 and plot the similar graph once again.

```python
psuccess = 0.65
ExperimentOutcomes = np.random.binomial(Ns,psuccess)
```

```python
RelativeFrequencies = ExperimentOutcomes/Ns
```

```python
plt.stem(Ns,RelativeFrequencies)
```

As we can observe from above plot that as we are tossing a coin more number of times, the relative frequency of heads is approaching towards psuccess = 0.65.

Now, let's change the psuccess to 0.35 and plot the similar graph once again.

```python
psuccess = 0.35
ExperimentOutcomes = np.random.binomial(Ns,psuccess)
```

```python
RelativeFrequencies = ExperimentOutcomes/Ns
```

```python
plt.stem(Ns,RelativeFrequencies)
```

As we can observe from above plot that as we are tossing a coin more number of times, the relative frequency of heads is approaching towards psuccess = 0.35.

```python

```

1. Relative Frequency and Probability and their relation.ipynb


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
plt.rcParams['font.size'] = 13
plt.rcParams['axes.spines.right'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['xtick.top'] = False
```

### A linear mapping from input to output space

Linear regression assumes that some recorded output values $y_i$ for $i=1,...,N_\mathrm{samples}$ depend linearly on some input values $\mathbf{x}_i$, and that any deviations are due to noise. Given $\mathbf{x}$-$y$ data, your task is then to rediscover the form of this linear mapping, which in the simplest case corresponds to fitting a line to the data. The question is merely how to select the best line?

```python
# Generate example data
# Initialize
sigma = 0.5                              # noise std
nSamples = 21                            # number of samples
x = np.linspace(0, 5, nSamples)          # input values
lineFun = lambda w0, w1, x: w0 + w1*x    # linear mapping

# Generate y-data
w0 = 1    # intercept
w1 = 0.5  # slope
y = lineFun(w0, w1, x)
y += np.random.randn(nSamples)*sigma

# Plot our generated data
plt.figure(figsize=(7.5, 3))
plt.plot(x, y, 'ko')
plt.xlabel('x')
plt.ylabel('y');
```

### Least squares approach

One approach for evaluating how good a line fits is by summing up the squared distances to each $y$-value from the line, and to compare this value for various lines. A lower value would indicate a better fit and a higher a worse fit. Usually, this sum is further divided by the number of samples, so as to get a mean squared error (MSE).

```python
# Calculate the MSE for a test line
w0Test = 1.0
w1Test = 0.5
yHat = lineFun(w0Test, w1Test, x)              # Model predictions, the predicted line
mseFun = lambda y, yHat: np.mean((y-yHat)**2)  # MSE function
mseTest = mseFun(y, yHat)                      # MSE value for our test line

# Plot the test line (blue), the distace to each y_i (red) and the y-values (black)
plt.figure(figsize=(7.5, 3))
for xi, yi, yHati in zip(x, y, yHat):
    plt.stem([xi], [yi], 'r', bottom=yHati)
plt.plot(x, yHat, 'b-')
plt.plot(x, y, 'ko')
plt.xlabel('x')
plt.ylabel('y');
plt.title('MSE: ' + '%1.3f' % mseTest);
```

From the example above we learn that each line (a unique combination of $w_0$ and $w_1$) obtains a slightly different MSE value. Our task is therefore to find the $w_0$ and $w_1$ combination with the lowest value. Naively, we can try to do this by simply testing various combinations and plotting the MSE value as a function of both $w_0$ and $w_1$.

```python
# Get w0 and w1 combinations over a grid
nGrid = 21
W0, W1 = np.meshgrid(np.linspace(w0-2, w0+2, nGrid), np.linspace(w1-1, w1+1, nGrid))

# Get the MSE for each combination
mseVals = np.zeros([nGrid, nGrid])
for i in range(nGrid):
    for j in range(nGrid):
        yHat = lineFun(W0[i, j], W1[i, j], x)
        mseVals[i, j] = mseFun(y, yHat)

# Plot the surface
fig = plt.figure(figsize=(15, 5))
ax = plt.subplot(1, 2, 1, projection='3d')
ax.plot_surface(W0, W1, mseVals, cmap=cm.coolwarm, linewidth=0, antialiased=False)
ax.set_xlabel('$w_0$')
ax.set_ylabel('$w_1$')
ax.set_zlabel('MSE')
ax = plt.subplot(1, 2, 2)
ax.contourf(W0, W1, mseVals, 50, cmap=cm.coolwarm)
ax.set_xlabel('$w_0$')
ax.set_ylabel('$w_1$');
```

The take home message so far is thus that each line (unique combination of $w_0$ and $w_1$) corresponds to one point on a MSE surface, and that the best line in a MSE sense is at the bottom of the surface where the MSE minimum is located.

```python
# Test various w0 and w1 values to see how the corresponding line fits at various locations on MSE surface
w0Test = 0.5
w1Test = 1.0

# Evaluate the MSE value for our current test line
yHat = lineFun(w0Test, w1Test, x)
mseFun = lambda y, yHat: np.mean((y-yHat)**2)
mseTest = mseFun(y, yHat)

# Plot
fig = plt.figure(figsize=(15, 5))
ax = plt.subplot(1, 2, 1)
for xi, yi, yHati in zip(x, y, yHat):
    ax.stem([xi], [yi], 'r', bottom=yHati)
ax.plot(x, yHat, 'b-')
ax.plot(x, y, 'ko')
ax.set_xlabel('x')
ax.set_ylabel('y');
ax.set_title('MSE: ' + '%1.3f' % mseTest);
ax = plt.subplot(1, 2, 2)
ax.contourf(W0, W1, mseVals, 50, cmap=cm.coolwarm)
ax.set_xlabel('$w_0$')
ax.set_ylabel('$w_1$');
ax.plot(w0Test, w1Test, 'ko', ms=10);
```

###  How to find the minimum in practice

Before moving on we need a more general description of our optimization problem, and we obtain it by describing the linear mapping using matrix notation. The line equation used above ($\hat{y}_i = w_0+w_1 x_i$) can then be written as:

\begin{equation}
 \hat{y}_i = \mathbf{x}_i^T \mathbf{w}, \quad \text{where}: \mathbf{x}_i^T = [1, x_i] \; \text{and} \; \mathbf{w}^T = [w_0, w_1],
\end{equation}

and subsequently, it is possible to write the predictions for all samples as $\hat{\mathbf{y}} = \mathbf{X}\mathbf{w}$, where $\mathbf{x}_i^T$, ..., $\mathbf{x}_{N_\mathrm{samples}}^T$ make up the rows in $\mathbf{X}$. Similarly, the MSE can also be expressed with matrix notation as:

\begin{equation}
 MSE = \frac{1}{N_\mathrm{samples}} (\mathbf{y}-\mathbf{X}\mathbf{w})^T(\mathbf{y}-\mathbf{X}\mathbf{w})
\end{equation}

Now, lets assume that we start with an initial guess ($\tilde{w}$) for both $w_0$ and $w_1$, and then we will try to iteratively improve it. This does, however, require a way of choosing how to iteratively change $\tilde{w}$, but we can use the gradient for this. The gradient points in the direction that a function increases the fastest. So, by moving in the opposite direction we should effectively find new values for $w_0$ and $w_1$ that correspond to a lower MSE value. We this move on to find the gradient by differentiating the MSE function with respect to $\mathbf{w}$:

\begin{equation}
 \frac{MSE}{d\mathbf{w}} = \frac{-2}{N_\mathrm{samples}} \mathbf{X}^T(\mathbf{y}-\mathbf{X}\mathbf{w})
\end{equation}

The idea of moving in the opposite direction to the gradient is fundamental to all gradient based optimization techniques, and the simplest idea of just taking small steps in the opposite direction is called gradient descent.

```python
# Plot the MSE surface first
fig = plt.figure(figsize=(7.5, 5))
plt.contourf(W0, W1, mseVals, 50, cmap=cm.coolwarm)
plt.xlabel('$w_0$')
plt.ylabel('$w_1$');

# Gradient descent
eta = 0.5e-2                                 # step length
wTilde = np.array([2.5, 1.25])              # initial guess
X = np.vstack([np.ones(nSamples), x]).T     # create the X matrix where each row is one sample
plt.plot(wTilde[0], wTilde[1], 'o', ms=6, c=[1, 1, 1]) # Plot our initial location on the MSE surface
for i in range(10):
    gradient = -2./nSamples * np.dot(X.T, y-np.dot(X, wTilde))  # Calculate the gradient
    wTilde -= eta*gradient                                      # Move in the opposite direction of the gradient
    plt.plot(wTilde[0], wTilde[1], 'o', ms=6, c=[1, 1, 1])                 # Plot our new location on the MSE surface
```

Gradient descent is a stupidly simple approach, but it often converges quite slowly (as seen above). Luckily, we don't actually have to use it to find our MSE optimal line. We can actually take a shortcut by remembering that the gradient is zero at a the minimum. Thus, we can solve for the optimal $w_0$ and $w_1$ values by setting the gradient to zero:

\begin{align}
 \frac{-2}{N_\mathrm{samples}} \mathbf{X}^T(\mathbf{y}-\mathbf{X}\mathbf{w}) &= 0 \\
 \mathbf{X}^T \mathbf{y} &= \mathbf{X}^T \mathbf{X}\mathbf{w}\\
 (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y} &= \mathbf{w}
\end{align}

where the last line now corresponds to the classical expression for solving a linear regression problem.

```python
# Optimal solution
wOpt = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y)
yHat = lineFun(wOpt[0], wOpt[1], x)
mseFun = lambda y, yHat: np.mean((y-yHat)**2)
mseTest = mseFun(y, yHat)

fig = plt.figure(figsize=(15, 5))
ax = plt.subplot(1, 2, 1)
for xi, yi, yHati in zip(x, y, yHat):
    ax.stem([xi], [yi], 'r', bottom=yHati)
ax.plot(x, yHat, 'b-')
ax.plot(x, y, 'ko')
ax.set_xlabel('x')
ax.set_ylabel('y');
ax.set_title('MSE: ' + '%1.3f' % mseTest);
ax = plt.subplot(1, 2, 2)
ax.contourf(W0, W1, mseVals, 50, cmap=cm.coolwarm)
ax.set_xlabel('$w_0$')
ax.set_ylabel('$w_1$');
ax.plot(wOpt[0], wOpt[1], 'o', ms=6, c=[1, 1, 1]);
```

### What if x is multi-dimensional?

The general solution derived above works even if $\mathbf{x}_i$ is multi-dimensional. The only difference is that we now try to fit a plane or a hyper-plane to the data instead of just a line.

```python
# Initialize
w0 = 0.5        # True parameter values for w0, w1, and w2
w1 = 0.5
w2 = 0.5
sigma = 1.      # noise std
nGrid = 11      # x-grid resolution
X1, X2 = np.meshgrid(np.linspace(-5, 5, nGrid), np.linspace(-5, 5, nGrid))    # x1 ans x2 values on a grid
planeFun = lambda w0, w1, w2, x1, x2: w0 + w1*x1 + w2*x2                      # linear mapping

# Generate noisy y-data
y = planeFun(w0, w1, w2, X1, X2).ravel()
y += np.random.randn(nGrid**2)*sigma

# Find the least squares solution
X = np.vstack([np.ones(nGrid**2), X1.ravel(), X2.ravel()]).T
wOpt = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y)
yHat = planeFun(wOpt[0], wOpt[1], wOpt[2], X1, X2)

# Plot the found least squares plane and the data points
fig = plt.figure(figsize=(7.5, 5))
ax = fig.add_subplot(111, projection='3d')
ax.plot(X1.ravel(), X2.ravel(), y, 'ko')
ax.plot_surface(X1, X2, yHat, cmap=cm.coolwarm, linewidth=0, antialiased=False, alpha=0.5)
ax.set_xlabel('x_1')
ax.set_ylabel('x_2')
ax.set_zlabel('y');
```

### Maximum likelihood, a second approach

Another approach of finding an optimal line would be to assume that each observed $y$-value consist of two parts, signal and noise, such that

\begin{equation}
 y_i = \mathbf{x}_i^T \mathbf{w} + \epsilon_i
\end{equation}

where $\epsilon$ is a normally distributed random noise term with zero mean. If the noise is identical and all data points (samples) are independent, then we can score various mappings (unique $\mathbf{w}$ vectors) by how likely it is that each particular mapping would have generated the observed data. That is, the probability to observe any one single data point for the mapping $\mathbf{w}$ is

\begin{equation}
 P(y_i|\mathbf{w}) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( -\frac{(y_i - \mathbf{x}_i^T \mathbf{w})^2}{2 \sigma^2} \right),
\end{equation}

and subsequently, the likelihood for observing all independent data points are:

\begin{equation}
 l(\mathbf{y}|\mathbf{w}) = \prod_i^{N_\mathrm{samples}} \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( -\frac{(y_i - \mathbf{x}_i^T \mathbf{w})^2}{2 \sigma^2} \right).
\end{equation}

It is however a bit cumbersome to work with likelihoods, but luckily we can work with the simpler log-likelihood instead. Taking logs of the likelihood thus gives:

\begin{align}
 ll(\mathbf{y}|\mathbf{w}) &= \sum_i^{N_\mathrm{samples}} \ -\frac{(y_i - \mathbf{x}_i^T \mathbf{w})^2}{2 \sigma^2} - N_\mathrm{samples} \log( \sqrt{2 \pi \sigma^2} ) \\
 ll(\mathbf{y}|\mathbf{w}) &= \frac{-1}{2 \sigma^2} (\mathbf{y}-\mathbf{X}\mathbf{w})^T(\mathbf{y}-\mathbf{X}\mathbf{w}) - N_\mathrm{samples} \log( \sqrt{2 \pi \sigma^2} )
\end{align}

The log-likelihood represents a function which we would like to maximize, in contrast to the MSE, as we want to find the $\mathbf{w}$ that is most likely to have generated the data. However, the gradient is zero at both a maximum and a minimum, and we can thus find the $\mathbf{w}$ at the maximum by setting the gradient of the log-likelihood function to zero.

\begin{align}
 \frac{ll(\mathbf{y}|\mathbf{w})}{d\mathbf{w}} = \frac{-1}{\sigma^2} \mathbf{X}^T(\mathbf{y}-\mathbf{X}\mathbf{w}) = &0 \\
  \frac{-1}{\sigma^2} \mathbf{X}^T(\mathbf{y}-\mathbf{X}\mathbf{w}) &= 0 \\
 \mathbf{X}^T \mathbf{y} &= \mathbf{X}^T \mathbf{X}\mathbf{w}\\
 (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y} &= \mathbf{w}
\end{align}

Amazingly, we can now see that the maximum likelihood solution is the same as the least squares solution.

### Orthogonal projections, a third approach

If we go back and look at how we described our linear mapping in matrix form

\begin{equation}
 \hat{\mathbf{y}} = \mathbf{X} \mathbf{w}
\end{equation}

we notice that all possible prediction vectors ($\hat{\mathbf{y}}$) live in the column space of $\mathbf{X}$, that is in the space spanned by the columns in $\mathbf{X}$. The intuitive explanation for this is that all prediction vectors are constructed as a linear combination of the columns in $\mathbf{X}$, and thus these must lie in the columns space of $\mathbf{X}$. This also means that if $\mathbf{y}$ is not in the column space of $\mathbf{X}$, then we can not describe it perfectly as a linear combination of the columns in $\mathbf{X}$. However, we can search for an orthogonal projection of $\mathbf{y}$ onto the column space, and this will represent the $\hat{\mathbf{y}}$ that is closest (euclidean norm) to $\mathbf{y}$, that is our best possible approximation. These ideas are illustrated below where the dashed black lines represent the columns of $\mathbf{X}$, the gray plane the column space, the blue line $\mathbf{y}$, the black line our orthogonal projection of $\mathbf{y}$, and the red line the error $\mathbf{y}-\hat{\mathbf{y}}$

```python
# Example X and y selected to get a descent figure
X = np.array([[1, 1, 1],[3, 1, 0.5]]).T
y = np.array([7, 14, 1])
wOpt = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y)
u = np.dot(X, wOpt)

# The columns space
nGrid = 11
X1 = np.zeros([nGrid, nGrid])
X2 = np.zeros([nGrid, nGrid])
X3 = np.zeros([nGrid, nGrid])
W1, W2 = np.meshgrid(np.linspace(-5, 5, nGrid), np.linspace(-5, 5, nGrid))
for i in range(nGrid):
    for j in range(nGrid):
        X1[i, j] = W1[i, j]*X[0, 0] + W2[i, j]*X[0, 1]
        X2[i, j] = W1[i, j]*X[1, 0] + W2[i, j]*X[1, 1]
        X3[i, j] = W1[i, j]*X[2, 0] + W2[i, j]*X[2, 1]

# Plot columns vectors, the column space, y, and the orthogonal projection of y onto the column space
fig = plt.figure(figsize=(10, 7.5))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X1, X2, X3, color='gray', linewidth=0, antialiased=False, alpha=0.5)
scaling = 5
ax.plot([0, scaling*X[0, 0]], [0, scaling*X[1, 0]], [0, scaling*X[2, 0]], 'k--', lw=2)
ax.plot([0, scaling*X[0, 1]], [0, scaling*X[1, 1]], [0, scaling*X[2, 1]], 'k--', lw=2)
ax.plot([0, y[0]], [0, y[1]], [0, y[2]], 'k-', lw=2)
ax.plot([0, u[0]], [0, u[1]], [0, u[2]], 'b-', lw=2)
ax.plot([y[0], u[0]], [y[1], u[1]], [y[2], u[2]], 'r-', lw=2)
ax.grid(False)
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$x_3$');
```

The question is now then how to find the orthogonal projection of $\mathbf{y}$. We do, however, know that the inner product between two vectors is zero if they are orthogonal. The error $\mathbf{y}-\hat{\mathbf{y}}$ should thus be orthogonal to all columns in $\mathbf{X}$. If we now write this is mathematical terms, we get:

\begin{align}
 <\mathbf{X}, (\mathbf{y} - \hat{\mathbf{y}})> = <\mathbf{X}, (\mathbf{y} - \mathbf{X} \mathbf{w})> &= 0, \\
 \mathbf{X}^T(\mathbf{y}-\mathbf{X}\mathbf{w}) &= 0, \\
 \mathbf{X}^T \mathbf{y} &= \mathbf{X}^T \mathbf{X}\mathbf{w}, \\
 (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y} &= \mathbf{w},
\end{align}

and amazingly see that this approach also yielded the exact same solution as the original least squares approach.


# Linear Equations
The equations in the previous lab included one variable, for which you solved the equation to find its value. Now let's look at equations with multiple variables. For reasons that will become apparent, equations with two variables are known as linear equations.

## Solving a Linear Equation
Consider the following equation:

\begin{equation}2y + 3 = 3x - 1 \end{equation}

This equation includes two different variables, **x** and **y**. These variables depend on one another; the value of x is determined in part by the value of y and vice-versa; so we can't solve the equation and find absolute values for both x and y. However, we *can* solve the equation for one of the variables and obtain a result that describes a relative relationship between the variables.

For example, let's solve this equation for y. First, we'll get rid of the constant on the right by adding 1 to both sides:

\begin{equation}2y + 4 = 3x \end{equation}

Then we'll use the same technique to move the constant on the left to the right to isolate the y term by subtracting 4 from both sides:

\begin{equation}2y = 3x - 4 \end{equation}

Now we can deal with the coefficient for y by dividing both sides by 2:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Our equation is now solved. We've isolated **y** and defined it as 3x-4/2

While we can't express **y** as a particular value, we can calculate it for any value of **x**. For example, if **x** has a value of 6, then **y** can be calculated as:

\begin{equation}y = \frac{3\cdot6 - 4}{2} \end{equation}

This gives the result 14/2 which can be simplified to 7.

You can view the values of **y** for a range of **x** values by applying the equation to them using the following R code:

```R
# Create a dataframe with an x column containing values from -10 to 10
df = data.frame(x = seq(-10, 10))

# Add a y column by applying the solved equation to x
df$y = (3*df$x - 4) / 2

#Display the dataframe
df
```

x	y

-10  	-17.0

 -9  	-15.5

 -8  	-14.0

 -7  	-12.5

 -6  	-11.0

 -5  	 -9.5

 -4  	 -8.0

 -3  	 -6.5

 -2  	 -5.0

 -1  	 -3.5

  0  	 -2.0

  1  	 -0.5

  2  	  1.0

  3  	  2.5

  4  	  4.0

  5  	  5.5

  6  	  7.0

  7  	  8.5

  8  	 10.0

  9  	 11.5

 10  	 13.0

We can also plot these values to visualize the relationship between x and y as a line. For this reason, equations that describe a relative relationship between two variables are known as *linear equations*:

```R
library(ggplot2)
library(repr)
options(repr.plot.width=4, repr.plot.height=4)
ggplot(df, aes(x,y)) + geom_point() + geom_line(color = 'blue')
```

In a linear equation, a valid solution is described by an ordered pair of x and y values. For example, valid solutions to the linear equation above include:
- (-10, -17)
- (0, -2)
- (9, 11.5)

The cool thing about linear equations is that we can plot the points for some specific ordered pair solutions to create the line, and then interpolate the x value for any y value (or vice-versa) along the line.

## Intercepts
When we use a linear equation to plot a line, we can easily see where the line intersects the X and Y axes of the plot. These points are known as *intercepts*. The *x-intercept* is where the line intersects the X (horizontal) axis, and the *y-intercept* is where the line intersects the Y (horizontal) axis.

Let's take a look at the line from our linear equation with the X and Y axis shown through the origin (0,0).

```R
ggplot(df, aes(x,y)) + geom_point() + geom_line(color = 'blue') +
       geom_hline(yintercept=0) + geom_vline(xintercept=0)
```

The x-intercept is the point where the line crosses the X axis, and at this point, the **y** value is always 0. Similarly, the y-intercept is where the line crosses the Y axis, at which point the **x** value is 0. So to find the intercepts, we need to solve the equation for **x** when **y** is 0.

For the x-intercept, our equation looks like this:

\begin{equation}0 = \frac{3x - 4}{2} \end{equation}

Which can be reversed to make it look more familar with the x expression on the left:

\begin{equation}\frac{3x - 4}{2} = 0 \end{equation}

We can multiply both sides by 2 to get rid of the fraction:

\begin{equation}3x - 4 = 0 \end{equation}

Then we can add 4 to both sides to get rid of the constant on the left:

\begin{equation}3x = 4 \end{equation}

And finally we can divide both sides by 3 to get the value for x:

\begin{equation}x = \frac{4}{3} \end{equation}

Which simplifies to:

\begin{equation}x = 1\frac{1}{3} \end{equation}

So the x-intercept is 11/3 (approximately 1.333).

To get the y-intercept, we solve the equation for y when x is 0:

\begin{equation}y = \frac{3\cdot0 - 4}{2} \end{equation}

Since 3 x 0 is 0, this can be simplified to:

\begin{equation}y = \frac{-4}{2} \end{equation}

-4 divided by 2 is -2, so:

\begin{equation}y = -2 \end{equation}

This gives us our y-intercept, so we can plot both intercepts on the graph:

```R
ggplot(df, aes(x,y)) + geom_line(color = 'blue') +
       geom_hline(yintercept=0) + geom_vline(xintercept=0) +
       annotate("text", x = 3, y = -2, label = "y-intercept")+
       annotate("text", x = 5, y = 1, label = "x-intercept")
```

The ability to calculate the intercepts for a linear equation is useful, because you can calculate only these two points and then draw a straight line through them to create the entire line for the equation.

## Slope
It's clear from the graph that the line from our linear equation describes a slope in which values increase as we travel up and to the right along the line. It can be useful to quantify the slope in terms of how much **x** increases (or decreases) for a given change in **y**. In the notation for this, we use the greek letter Δ (*delta*) to represent change:

\begin{equation}slope = \frac{\Delta{y}}{\Delta{x}} \end{equation}

Sometimes slope is represented by the variable ***m***, and the equation is written as:

\begin{equation}m = \frac{y_{2} - y_{1}}{x_{2} - x_{1}} \end{equation}

Although this form of the equation is a little more verbose, it gives us a clue as to how we calculate slope. What we need is any two ordered pairs of x,y values for the line - for example, we know that our line passes through the following two points:
- (0,-2)
- (6,7)

We can take the x and y values from the first pair, and label them x1 and y1; and then take the x and y values from the second point and label them x2 and y2. Then we can plug those into our slope equation:

\begin{equation}m = \frac{7 - -2}{6 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{7 + 2}{6 - 0} \end{equation}

That gives us the result 9/6 which is 11/2 or 1.5 .

So what does that actually mean? Well, it tells us that for every change of **1** in x, **y** changes by 11/2 or 1.5. So if we start from any point on the line and move one unit to the right (along the X axis), we'll need to move 1.5 units up (along the Y axis) to get back to the line.

You can plot the slope onto the original line with the following R code to verify it fits:

```R
line = data.frame(x = c(0,1.5), y = c(-2,0))
ggplot() + geom_line(data = df, aes(x,y),color = 'blue') +
       geom_hline(yintercept=0) + geom_vline(xintercept=0) +
       geom_line(data = line, aes(x,y), color = 'red', size = 3)
```

### Slope-Intercept Form
One of the great things about algebraic expressions is that you can write the same equation in multiple ways, or *forms*. The *slope-intercept form* is a specific way of writing a 2-variable linear equation so that the equation definition includes the slope and y-intercept. The generalised slope-intercept form looks like this:

\begin{equation}y = mx + b \end{equation}

In this notation, ***m*** is the slope and ***b*** is the y-intercept.

For example, let's look at the solved linear equation we've been working with so far in this section:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Now that we know the slope and y-intercept for the line that this equation defines, we can rewrite the equation as:

\begin{equation}y = 1\frac{1}{2}x + -2 \end{equation}

You can see intuitively that this is true. In our original form of the equation, to find y we multiply x by three, subtract 4, and divide by two - in other words, x is half of 3x - 4; which is 1.5x - 2. So these equations are equivalent, but the slope-intercept form has the advantages of being simpler, and including two key pieces of information we need to plot the line represented by the equation. We know the y-intecept that the line passes through (0, -2), and we know the slope of the line (for every x, we add 1.5 to y.

Let's recreate our set of test x and y values using the slope-intercept form of the equation, and plot them to prove that this  describes the same line:

```R
## Make a data frame with the x values
df = data.frame(x = seq(-10,10))

## Add the y values using the formula y = mx + b
m = 1.5
b = -2
df$y = m * df$x + b

## Plot the result
ggplot() + geom_line(data = df, aes(x,y),color = 'blue') +
       geom_hline(yintercept=0) + geom_vline(xintercept=0) +
       geom_line(data = line, aes(x,y), color = 'red', size = 3) +
       annotate("text", x = 3, y = -2, label = "y-intercept")
```


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

```

# Using a known function

 We will optimize the following equation:
\begin{equation} y = (x-5)^2 + 3 \end{equation}

Which we know the minimum is:

\begin{equation} x = 5 \end{equation}
\begin{equation} y = 3 \end{equation}

```python
X = np.linspace(0,10, 100) # Get 100 datapoints equally spaced from 0 to 10
y = 3 + np.power((X-5),2)
```

Let's plot our data to check the relation between X and y

```python
plt.plot(X,y,'red')
plt.xlabel("$x$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.xlim((0,10))
```

# Gradient Descent

## Cost Function & Gradients

 Remember that in optimization methods we optimize a given cost. In our case it will be the quadratic equation below. Just in case calculating gradients is not your thing, I already precalculated the gradient in this case.

Cost
\begin{equation}
y = (X-5)^2 + 3
\end{equation}

Gradient

\begin{equation}
2*(X-5)
\end{equation}

```python
def cost(X):
    return 3 + np.power((X-5), 2)
```

```python
def gradient(X):
    return 2*(X-5)
```

```python
def gradient_descent(X,learning_rate=0.01,iterations=100, print_all=True):
    costs = []
    for it in range(iterations):
        X = X -learning_rate*gradient(X)
        if it % 50 == 0 and print_all:
            print("The iteration is {} and the current value of X is {}".format(it, X))
        costs.append(cost(X))
    return X, costs

```

 Let's start from a random number between 0 and 1, with 200 iterations and a learning rate of 0.01.

```python
lr =0.01
n_iter = 200
X = np.random.random()*10
X, costs = gradient_descent(X,lr,n_iter)
print("Minimum is at (x,y) = ({}, {})".format(X, cost(X)))
```

    The iteration is 0 and the current value of X is 7.399074672684869
    The iteration is 50 and the current value of X is 5.873670256056753
    The iteration is 100 and the current value of X is 5.3181642176498185
    The iteration is 150 and the current value of X is 5.115865761356701
    Minimum is at (x,y) = (5.043055915557471, 3.001853811864492)

 Let's plot the cost history over iterations

```python
fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('Cost')
ax.set_xlabel('Iterations')
_=ax.plot(range(n_iter),costs,'b.')
```

 After around 120 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. And if the learning rate is bigger?

```python
new_lr =0.1
new_n_iter = 50
X_b = np.random.random()
X_new, costs_new = gradient_descent(X_b,new_lr,new_n_iter)
print("Minimum is at (x,y) = ({}, {})".format(X_new, cost(X_new)))

fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('Cost')
ax.set_xlabel('Iterations')
_=ax.plot(range(new_n_iter),costs_new,'b.')
```

Clearly with a higher learning rate we got to the optimal point in 10 iterations. Let's see how this changes with different learning rates

```python
learning_rates = [0.01,0.1,1,10]
iterations = 200
X_0 = np.random.random()
plt.figure(figsize=(20,5))
all_costs = []
for lr in learning_rates:
    X, costs = gradient_descent(X_0,lr,iterations, print_all=True)
    print("Minimum is at (x,y) = ({}, {})".format(X, cost(X)))
    all_costs.append(costs)

ax.set_ylabel('Cost')
ax.set_xlabel('Iterations')
plt.xlim((0,200))
plt.ylim((0,50))
plt.scatter(range(iterations), all_costs[0], color='red')
plt.scatter(range(iterations), all_costs[1], color='blue')
plt.scatter(range(iterations), all_costs[2], color='green')
plt.scatter(range(iterations), all_costs[3], color='black')
plt.legend(['LR = 0.01', 'LR = 0.1', 'LR = 1', 'LR = 10'])
plt.show()
```

```python

```


```python
from sympy import *
init_printing(use_latex='mathjax')
x, y, z = symbols('x,y,z')
r, theta = symbols('r,theta', positive=True)
```

## Matrices

El objeto `Matrix` de *Sympy* nos ayuda con pequeños problemas en álgebra lineal.

```python
rot = Matrix([[r*cos(theta), -r*sin(theta)],
              [r*sin(theta),  r*cos(theta)]])
rot
```

### Métodos estándar

```python
rot.det()
```

```python
rot.inv()
```

```python
rot.singular_values()
```

### Ejercicios

Encuentre la inversa de la siguiente matriz:

$$ \left[\begin{matrix}1 & x\\y & 1\end{matrix}\right] $$

```python
# Crea una matriz y usa el método `inv` para encontrar la inverso

```

### Operadores

Los operadores estándar *SymPy*  trabajan en matrices.

```python
rot * 2
```

```python
rot**2
```

```python
v = Matrix([[x], [y]])
v
```

```python
rot * v
```

### Ejercicio

En el último ejercicio encontraste la inversa de la siguiente matriz

```python
M = Matrix([[1, x], [y, 1]])
M
```

```python
M.inv()
```

Ahora verifica que esta es la verdadera inversa multiplicando la matriz por su inversa. ¿Recuperas la matriz identidad?

```python
# Multiplica `M` por su inversa. ¿Recuperas la matriz identidad?
```

### Ejercicio

¿Cuáles son los vectores y valores propios de `M`?

```python
# Encuentra los metodos para calcular vectores y valores propios. Usa estos metodos en `M`

```

### Acceso a elementos estilo NumPy

```python
rot[0, 0]
```

```python
rot[:, 0]
```

```python
rot[1, :]
```

### Mutación

Podemos cambiar elementos en la matriz.

```python
rot[0, 0] += 1
rot
```

```python
simplify(rot.det())
```

```python
rot.singular_values()
```

### Ejercicio

Juega con tu matriz `M`, manipulando elementos de forma similar a *NumPy*. Luego prueba los diversos métodos de los que hemos hablado (u otros). Averigua qué tipo de respuestas obtienes.

```python
# Juega con matrices

```