# HW07: Solving Linear Least Squares Problems
**Yuanxing Cheng, 810925466**

This homework studies the polynomial fitting problem. Given a set of $m$ different points, $(x_i , y_i )$, $i = 1, 2, \dots, m$, in the $xy\text{-coordinate}$ plane, determine a polynomial of degree $n − 1$.

$$p(x) = a_1 x^{n-1} + a_2 x^{n-2} + \cdots + a_{n-1} x + a_n$$

where $n\leq m$, such that $\displaystyle \sum_{i=1}^{m} \left|p(x_i) - y_i\right|^2$ is minimized.

This leads to a linear least squares problem, where the coefficient matrix is given by the $m \times n$ Vandermonde matrix

$$\left[\begin{array}{ccccc}
x_{1}^{n-1} & x_{1}^{n-2} & \cdots & x_{1} & 1 \\
x_{2}^{n-1} & x_{2}^{n-2} & \cdots & x_{2} & 1 \\
x_{3}^{n-1} & x_{3}^{n-2} & \cdots & x_{3} & 1 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
x_{m}^{n-1} & x_{m}^{n-2} & \cdots & x_{m} & 1 \\
\end{array}\right]$$

the right hand side is the $m\text{−dimensional}$ column vector
$\left[\begin{array}{cccc}
y_{1} & y_{2} & \cdots & y_{m} \\
\end{array}\right]'$, and the unknown is the $n\text{−dimensional}$ column vector $\left[\begin{array}{cccc}
a_{1} & a_{2} & \cdots & a_{n} \\
\end{array}\right]'$ containing the coefficients of p(x). Note that several MATLAB functions, e.g., `vander`, `polyval`, may be useful in this homework.

## Q1
Take the following $m$ equally spaced points on the interval $−1 \leq x \leq 1$,

$$\begin{align}
x_i=-1+ {{2(i-1)}\over{m-1}} && y_i = \frac{1} {1 + 25x_i^2}
\end{align}$$

Here $i = 1,2,\cdots, m$. Use the following three different approaches to solve the above least squares problem, respectively, to obtain the polynomial $p(x)$ of degree $n − 1$. Here take $m = 50$ and $n = 20$.

($\text{a}$) The Matlab backslash function `\`, which solves the least squares problem when the matrix is an $m \times n$ matrix ($m > n$).

($\text{b}$) The reduced $QR$ factorization. Here the $QR$ factorization can be computed via the MATLAB built-in function `qr`.

($\text{c}$) Solution of the normal equation.

Compare the coefficients of $p(x)$ obtained from the above three approaches. Which two approaches generate results more close to each other. You need to use format long to observe more digits in the coefficients for comparison.

$Answer$

The distance between using Matlab backslash function and reduced QR factorization is the lowest.

$Code$

```MATLAB
clear
clc

m = 50;
n = 20;

x = linspace(-1,1,m);
x = x';
y = 1./(1+25*x.^2);
A = vander(x);

A1 = A(:,m-n+1:m);
x1 = A1\y;

[Q,R] = qr(A1,0);
y2 = Q'*y;
x2 = backwardSubeps(R,y2);

A3 = A1'*A1;
L = chol(A3,'lower');
y3 = L^(-1)*A1'*y;
x3 = backwardSubeps(L',y3);

d12 = norm(x1-x2);
d23 = norm(x2-x3);
d31 = norm(x3-x1);

s1 = 'Matlab backslash function';
s2 = 'reduced QR factorization';
s3 = 'normal equation';

if d12 < d23
    if d12 < d31
        s = [s1,' and ',s2];
    else
        s = [s3,' and ',s1];
    end
else
    if d23 < d31
        s = [s2,' and ',s3];
    else
        s = [s3,' and ',s1];
    end
end

disp(['The distance between using ',s,' is the lowest.'])

```

## Q2
For the case $m = 50$ and $n = 20$, draw the fitting polynomial $y = p(x)$ obtained from one approach in Part 1 and compare it in the same figure with the plot of the original function $y = 1/(1 + 25x^2 )$. Do they match well?

$Answer$

Yes, all three methods fit very well.

$Code$

```MATLAB
clear
clc

m = 50;
n = 20;

x = linspace(-1,1,m);
x = x';
y = 1./(1+25*x.^2);
A = vander(x);

A1 = A(:,m-n+1:m);
x1 = A1\y;

[Q,R] = qr(A1,0);
y2 = Q'*y;
x2 = backwardSubeps(R,y2);

A3 = A1'*A1;
L = chol(A3,'lower');
y3 = L^(-1)*A1'*y;
x3 = backwardSubeps(L',y3);

s1 = 'Matlab backslash function';
s2 = 'reduced QR factorization';
s3 = 'normal equation';

t = linspace(-1,1,100);

subplot(3,1,1)
Y1 = polyval(x1,t);
plot(x,y,'bo')
hold on
plot(t,Y1,'r')
title(s1)
```

```MATLAB
subplot(3,1,2)
Y2 = polyval(x2,t);
plot(x,y,'bo')
hold on
plot(t,Y2,'r')
title(s2)

subplot(3,1,3)
Y3 = polyval(x3,t);
plot(x,y,'bo')
hold on
plot(t,Y3,'r')
title(s3)
```

## Q3
Repeat Part 2, but taking $m = n = 50$, which is now a polynomial interpolation problem. Does the interpolation polynomial match the function $y = 1/(1 + 25x^2)$ well in this case?

$Answer$

For the result from using Matlab backslash function and reduced QR factorization, they don't fit as well as before. And for the last method, it even doesn't work here because the matrix we now are using is no more full rank.

$Code$

```MATLAB
clear
clc

m = 50;
n = 50;

x = linspace(-1,1,m);
x = x';
y = 1./(1+25*x.^2);
A = vander(x);
```

```MATLAB
A1 = A(:,m-n+1:m);
x1 = A1\y;

[Q,R] = qr(A1,0);
y2 = Q'*y;
x2 = backwardSubeps(R,y2);

s1 = 'Matlab backslash function';
s2 = 'reduced QR factorization';

t = linspace(-1,1,100);

subplot(2,1,1)
Y1 = polyval(x1,t);
plot(x,y,'bo')
hold on
plot(t,Y1,'r')
title(s1)

subplot(2,1,2)
Y2 = polyval(x2,t);
plot(x,y,'bo')
hold on
plot(t,Y2,'r')
title(s2)
```

Computational/Intro to Numerical Computing/HW_Chap07.ipynb


# Exponentials, Radicals, and Logs
Up to this point, all of our equations have included standard arithmetic operations, such as division, multiplication, addition, and subtraction. Many real-world calculations involve exponential values in which numbers are raised by a specific power.

## Exponentials
A simple case of of using an exponential is squaring a number; in other words, multipying a number by itself. For example, 2 squared is 2 times 2, which is 4. This is written like this:

\begin{equation}2^{2} = 2 \cdot 2 = 4\end{equation}

Similarly, 2 cubed is 2 times 2 times 2 (which is of course 8):

\begin{equation}2^{3} = 2 \cdot 2 \cdot 2 = 8\end{equation}

In Python, you use the ****** operator, like this example in which **x** is assigned the value of 5 raised to the power of 3 (in other words, 5 x 5 x 5, or 5-cubed):

```python
x = 5**3
print(x)
```

    125

Multiplying a number by itself twice or three times to calculate the square or cube of a number is a common operation, but you can raise a number by any exponential power. For example, the following notation shows 4 to the power of 7 (or 4 x 4 x 4 x 4 x 4 x 4 x 4), which has the value:

\begin{equation}4^{7} = 16384 \end{equation}

In mathematical terminology, **4** is the *base*, and **7** is the *power* or *exponent* in this expression.

## Radicals (Roots)
While it's common to need to calculate the solution for a given base and exponential, sometimes you'll need to calculate one or other of the elements themselves. For example, consider the following expression:

\begin{equation}?^{2} = 9 \end{equation}

This expression is asking, given a number (9) and an exponent (2), what's the base? In other words, which number multipled by itself results in 9? This type of operation is referred to as calculating the *root*, and in this particular case it's the *square root* (the base for a specified number given the exponential **2**). In this case, the answer is 3, because 3 x 3 = 9. We show this with a **√** symbol, like this:

\begin{equation}\sqrt{9} = 3 \end{equation}

Other common roots include the *cube root* (the base for a specified number given the exponential **3**). For example, the cube root of 64 is 4 (because 4 x 4 x 4 = 64). To show that this is the cube root, we include the exponent **3** in the **√** symbol, like this:

\begin{equation}\sqrt[3]{64} = 4 \end{equation}

We can calculate any root of any non-negative number, indicating the exponent in the **√** symbol.

The **math** package in Python includes a **sqrt** function that calculates the square root of a number. To calculate other roots, you need to reverse the exponential calculation by raising the given number to the power of 1 divided by the given exponent:

```python
import math

# Calculate square root of 25
x = math.sqrt(25)
print (x)

# Calculate cube root of 64
cr = round(64 ** (1. / 3))
print(cr)
```

    5.0
    4

The code used in Python to calculate roots other than the square root reveals something about the relationship between roots and exponentials. The exponential root of a number is the same as that number raised to the power of 1 divided by the exponential. For example, consider the following statement:

\begin{equation} 8^{\frac{1}{3}} = \sqrt[3]{8} = 2 \end{equation}

Note that a number to the power of 1/3 is the same as the cube root of that number.

Based on the same arithmetic, a number to the power of 1/2 is the same as the square root of the number:

\begin{equation} 9^{\frac{1}{2}} = \sqrt{9} = 3 \end{equation}

You can see this for yourself with the following Python code:

```python
import math

print (9**0.5)
print (math.sqrt(9))
```

    3.0
    3.0

## Logarithms
Another consideration for exponential values is the requirement occassionally to determine the exponent for a given number and base. In other words, how many times do I need to multiply a base number by itself to get the given result. This kind of calculation is known as the *logarithm*.

For example, consider the following expression:

\begin{equation}4^{?} = 16 \end{equation}

In other words, to what power must you raise 4 to produce the result 16?

The answer to this is 2, because 4 x 4 (or 4 to the power of 2) = 16. The notation looks like this:

\begin{equation}log_{4}(16) = 2 \end{equation}

In Python, you can calculate the logarithm of a number using the **log** function in the **math** package, indicating the number and the base:

```python
import math

x = math.log(16, 4)
print(x)
```

    2.0

The final thing you need to know about exponentials and logarithms is that there are some special logarithms:

The *common* logarithm of a number is its exponential for the base **10**. You'll occassionally see this written using the usual *log* notation with the base omitted:

\begin{equation}log(1000) = 3 \end{equation}

Another special logarithm is something called the *natural log*, which is a exponential of a number for base ***e***, where ***e*** is a constant with the approximate value 2.718. This number occurs naturally in a lot of scenarios, and you'll see it often as you work with data in many analytical contexts. For the time being, just be aware that the natural log is sometimes written as ***ln***:

\begin{equation}log_{e}(64) = ln(64) = 4.1589 \end{equation}

The **math.log** function in Python returns the natural log (base ***e***) when no base is specified. Note that this can be confusing, as the mathematical notation *log* with no base usually refers to the common log (base **10**). To return the common log in Python, use the **math.log10** function:

```python
import math

# Natural log of 29
print (math.log(29))

# Common log of 100
print(math.log10(100))
```

    3.367295829986474
    2.0

## Solving Equations with Exponentials
OK, so now that you have a basic understanding of exponentials, roots, and logarithms; let's take a look at some equations that involve exponential calculations.

Let's start with what might at first glance look like a complicated example, but don't worry - we'll solve it step-by-step and learn a few tricks along the way:

\begin{equation}2y = 2x^{4} ( \frac{x^{2} + 2x^{2}}{x^{3}} ) \end{equation}

First, let's deal with the fraction on the right side. The numerator of this fraction is x2 + 2x2 - so we're adding two exponential terms. When the terms you're adding (or subtracting) have the same exponential, you can simply add (or subtract) the coefficients. In this case, x2 is the same as 1x2, which when added to 2x2 gives us the result 3x2, so our equation now looks like this:

\begin{equation}2y = 2x^{4} ( \frac{3x^{2}}{x^{3}} ) \end{equation}

Now that we've condolidated the numerator, let's simplify the entire fraction by dividing the numerator by the denominator. When you divide exponential terms with the same variable, you simply divide the coefficients as you usually would and subtract the exponential of the denominator from the exponential of the numerator. In this case, we're dividing 3x2 by 1x3: The coefficient 3 divided by 1 is 3, and the exponential 2 minus 3 is -1, so the result is 3x-1, making our equation:

\begin{equation}2y = 2x^{4} ( 3x^{-1} ) \end{equation}

So now we've got rid of the fraction on the right side, let's deal with the remaining multiplication. We need to multiply 3x-1 by 2x4. Multiplication, is the opposite of division, so this time we'll multipy the coefficients and add the exponentials: 3 multiplied by 2 is 6, and -1 + 4 is 3, so the result is 6x3:

\begin{equation}2y = 6x^{3} \end{equation}

We're in the home stretch now, we just need to isolate y on the left side, and we can do that by dividing both sides by 2. Note that we're not dividing by an exponential, we simply need to divide the whole 6x3 term by two; and half of 6 times x3 is just 3 times x3:

\begin{equation}y = 3x^{3} \end{equation}

Now we have a solution that defines y in terms of x. We can use Python to plot the line created by this equation for a set of arbitrary *x* and *y* values:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 3*df['x']**3

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that the line is curved. This is symptomatic of an exponential equation: as values on one axis increase or decrease, the values on the other axis scale *exponentially* rather than *linearly*.

Let's look at an example in which x is the exponential, not the base:

\begin{equation}y = 2^{x} \end{equation}

We can still plot this as a line:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the slope-intercept equation to x
df['y'] = 2.0**df['x']

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="magenta")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()
plt.show()
```

Note that when the exponential is a negative number, Python reports the result as 0. Actually, it's a very small fractional number, but because the base is positive the exponential number will always positive. Also, note the rate at which y increases as x increases - exponential growth can be be pretty dramatic.

So what's the practical application of this?

Well, let's suppose you deposit $100 in a bank account that earns 5% interest per year. What would the balance of the account be in twenty years, assuming you don't deposit or withdraw any additional funds?

To work this out, you could calculate the balance for each year:

After the first year, the balance will be the initial deposit ($100) plus 5% of that amount:

\begin{equation}y1 = 100 + (100 \cdot 0.05) \end{equation}

Another way of saying this is:

\begin{equation}y1 = 100 \cdot 1.05 \end{equation}

At the end of year two, the balance will be the year one balance plus 5%:

\begin{equation}y2 = 100 \cdot 1.05 \cdot 1.05 \end{equation}

Note that the interest for year two, is the interest for year one multiplied by itself - in other words, squared. So another way of saying this is:

\begin{equation}y2 = 100 \cdot 1.05^{2} \end{equation}

It turns out, if we just use the year as the exponent, we can easily calculate the growth after twenty years like this:

\begin{equation}y20 = 100 \cdot 1.05^{20} \end{equation}

Let's apply this logic in Python to see how the account balance would grow over twenty years:

```python
import pandas as pd

# Create a dataframe with 20 years
df = pd.DataFrame ({'Year': range(1, 21)})

# Calculate the balance for each year based on the exponential growth from interest
df['Balance'] = 100 * (1.05**df['Year'])

#Display the dataframe
print(df)

# Plot the line
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.Year, df.Balance, color="green")
plt.xlabel('Year')
plt.ylabel('Balance')
plt.show()
```

```python

```

```python

```

Basics Of Algebra by Hiren/01-04-Exponentials Radicals and Logarithms.ipynb


# I. Linear least squares approximation

Consider a function $y = f(x)$ which is defined by a set of values $y_0, y_1, \cdots, y_n$ at points $x_0, x_1, \cdots, x_n$.

```python
x = [-1, -0.7, -0.43, -0.14, -0.14, 0.43, 0.71, 1, 1.29, 1.57, 1.86, 2.14, 2.43, 2.71, 3]
y = [-2.25, -0.77, 0.21, 0.44, 0.64, 0.03, -0.22, -0.84, -1.2, -1.03, -0.37, 0.61, 2.67, 5.04, 8.90]
```

### I.I. Find a best fit polynomial

$$
P_m(x) = a_0 + a_1 x + \cdots + a_m x^m
$$

using the linear least squares approach. To this end

1. implement a function which constructs the design matrix using $1, x, \cdots, x^m$ as the basis functions.

2. construct explicitly the normal system of equations of the linear least squares problem at fixed $m$.

3. Solve the normal equations to find the coefficients of $P_m(x)$ for $m = 0, 1, 2, \dots$. For the linear algebra problem, you can either use library functions (`numpy.linalg.solve`) or your LU factorization code from week 1.

(20% of the total grade)

```python
# ... ENTER YOUR CODE HERE
import numpy as np
import matplotlib.pyplot as plt

def design_matrix(x, m):
    mat = np.empty(shape=[len(x), m])
    for i in range(m):
        mat[:, i] = np.power(x, i)
    return mat

def solve_normal_equation(x, y, m):
    A=design_matrix(x, m)
    AT_A=A.T@A
    b=A.T@y
    beta=np.linalg.solve(AT_A,b)
    return beta

def predict(xval, beta):
    term = sum(beta[j]*xval**j for j in range(len(beta)))
    return term
```

```python
solve_normal_equation(x, y, 4)
```

    array([ 0.49483598, -0.26983377, -2.01973166,  1.01540301])

### I.II

To find the optimal value of m, use the following criterion: take $m=0, 1, 2, \dots$, for each value of $m$ compute

$$
\sigma_m^2 = \frac{1}{n - m} \sum_{k=0}^n \left( P_m(x_k) - y_k \right)^2
$$

And take the value of $m$, at which $\sigma_m$ stabilizes or starts increasing.

(20% of the total grade)

```python
# ... ENTER YOUR CODE HERE ...
def compute_sigma(x, y, m):
    beta = solve_normal_equation(x, y, m)
    y_predict = np.array([predict(_, beta) for _ in x])
    n=len(x)
    sigma = np.sum(np.square(y_predict - np.array(y))) / (n - m)
    return sigma

sigma = []
m_range = range(10)
sigmas = [compute_sigma(x, y, m) for m in m_range]
plt.plot(m_range, sigmas)
plt.show()
```

Plot your polynomials $P_m(x)$ on one plot, together with the datapoints. Visually compare best-fit polynomials of different degrees. Is the visual comparison consistent with the optimal value of $m$?

```python
# ... ENTER YOUR CODE HERE
plt.scatter(x,y)
for m in range(5):
    beta = solve_normal_equation(x, y, m)
    y_predict = [predict(_, beta) for _ in x]
    #print(beta)
    plt.plot(x, y_predict, label=m)
    plt.legend()
plt.show()
```

### I.III. Linear least-squares using the QR factorization.

For the optimal value of $m$ from the previous part, solve the LLS problem using the QR factorization, withou ever forming the normal equations explicitly. For linear algebra, you can use standard library functions (look up `numpy.linalg.solve`, `numpy.linalg.qr` etc) or your code from previous weeks.

Compare the results with the results of solving the normal system of equations.

(20% of the grade)

```python
# ... ENTER YOUR CODE HERE ...
def solve_normal_equation_qr(x, y, m):
    A=design_matrix(x, m)
    Q, R = np.linalg.qr(A) #factorize
    y_rotated = Q.T@y # rotate y by Q.T
    beta=np.linalg.solve(R[:m,:], y_rotated[:m]) #solve R1*beta = f
    return beta
```

```python
# ... ENTER YOUR CODE HERE
plt.scatter(x,y)
for m in range(1,5):
    beta = solve_normal_equation_qr(x, y, m)
    y_predict = [predict(_, beta) for _ in x]
    print(beta)
    plt.plot(x, y_predict, label=m)
    plt.legend()
plt.show()
```

# II. Lagrange interpolation

### II.1

Consider the function, $f(x) = x^2 \cos{x}$. On the interval $x\in [\pi/2, \pi]$, interpolate the function using the Lagrange interpolating polynomial of degree $m$ with $m=1, 2, 3, 4, 5$. Use the uniform mesh. Plot the resulting interpolants together with $f(x)$.

(20% of the total grade)

```python
# ... ENTER YOUR CODE HERE ...
def f(x):
    return (x**2)*np.cos(x)

def lagrange_k(xval, xk, k):
    n = len(xk)
    term = 1.0
    for j in range(n):
        if j==k:
            continue
        num = xval - xk[j]
        den = xk[k] - xk[j]
        term *= num / den
    return term

def lagrange(xval, xk, yk):
    return sum(yk[j] * lagrange_k(xval, xk, j) for j in range(len(xk)))
```

```python
import numpy as np
import sympy
import matplotlib.pyplot as plt
#plt.style.use('fivethirtyeight')

#%matplotlib notebook

def plot_interp(func, nodes, add_legend=True):
    #tabulate
    yy = func(nodes)

    #interpolate and evaluate on a finer grid
    xn = np.linspace(np.pi/2, np.pi, 201)
    yn = [lagrange(_, nodes, yy) for _ in xn]

    #plot
    plt.plot(xn, yn, '-', alpha=0.7, label=r'interp, $n=%s$'%len(nodes))
    plt.plot(xn, func(xn), label=r'$f(x)$')
    plt.plot(nodes, yy, 'o', ms=7)
    if add_legend:
        plt.legend(loc='best')
        #s = r"$f(x) = %s $" % sympy.printing.latex(func(sympy.Symbol('x')))
        #plt.text(-1, 0.9, s, fontsize=22)
```

```python
num_points = 3
nodes = np.linspace(np.pi/2, np.pi, num_points)
plot_interp(f, nodes)
```

```python
#alternate plot
xv=np.linspace(np.pi/2, np.pi, 201)
fv=[f(x) for x in xv]
plt.plot(xv,fv, label="original")

###lagrange roots
for m in range(2, 5):
    xv_m = np.linspace(np.pi/2,np.pi, m)
    yv_m = [f(x) for x in xv_m]
    langv = [lagrange(_, xv_m, yv_m) for _ in xv]
    # remove this to see result
    plt.plot(xv, langv, label=m)

plt.legend()
plt.show()
```

### II.2.

Repeat the previous task using the Chebyshev nodes. Compare the quality of interpolation on a uniform mesh and Chebyshev nodes for $m=3$.

(20% of the total grade)

```python
# ... ENTER YOUR CODE HERE ...
def cheb_nodes(n):
    k = np.arange(n)
    return np.cos((2*k+1)*np.pi/(2*n))
```

```python
num_points = 3
rightrange = np.pi
leftrange = np.pi/2
nonscaled_nodes = cheb_nodes(num_points)
nodes = (rightrange-leftrange)/(1-(-1))*(nonscaled_nodes-(-1)) + leftrange
plot_interp(f, nodes)
```

```python
#alternate plot
xv=np.linspace(np.pi/2, np.pi, 201)
fv=[f(x) for x in xv]
plt.plot(xv,fv, label="original")

###lagrange roots
for m in range(2, 5):
    xv_m = (rightrange-leftrange)/(1-(-1))*(cheb_nodes(m)-(-1)) + leftrange
    yv_m = [f(x) for x in xv_m]
    langv = [lagrange(_, xv_m, yv_m) for _ in xv]
    # remove this to see result
    plt.plot(xv, langv, label=m)

plt.legend()
plt.show()
```

```python

```


## Linear Regression using Gradient Descent

In this notebook, I've built a simple linear regression model using gradient descent to fit a line ($y = mx + c$) through input data consisting of ($x, y$) values. The purpose is to better understand gradient descent by calculating the gradients by hand and implementing it without using out-of-the-box libraries/ methods.

### Problem Statement

**Given a set of $x$ and $y$ values, the problem statement here is to find a line "$y = mx + b$" that best fits the data**. This involes finding the values of $m$ and $b$ that minimizes an _Error/ Loss_ function for the given dataset. We will use _Mean Squared Error (denoted as E)_ as the error function to be minimized, and it is given by -

$E = \dfrac{1}{N} \sum_{i=1}^N (y_i - (mx_i + b))^2$

To find the optimial value of $m$ and $b$ that minimizes the error function using Gradient Descent, the values are randomly initialized first and then updated $number of steps$ times or until the error function value converges. Below equations are used to update the values of $m$ and $b$ at each step -

$
m = m - learning\_rate \times \dfrac{\partial E}{\partial m}
$

$
b = b - learning\_rate \times \dfrac{\partial E}{\partial b}
$

### Calculating Gradients for Mean Squared Error

The Gradient Descent update formulae given above needs the partial derivatives of error function $E$ with respect to both $m$ and $b$ ($\dfrac{\partial E}{\partial m}$ and $\dfrac{\partial E}{\partial b}$ respectively). These partial derivatives are calculated as follows -

$
\begin{align}
E &= \dfrac{1}{N} \sum_{i=1}^N (y_i - (mx_i + b))^2 \\
  &= \dfrac{1}{N} \sum_{i=1}^N y_i^2 - 2y_i(mx_i + b) + (mx_i + b)^2 \\
  &= \dfrac{1}{N} \sum_{i=1}^N y_i^2 - 2x_iy_im - 2y_ib + x_i^2m^2 + 2x_imb + b^2 \\
\end{align}
$

$
\begin{align}
\dfrac{\partial E}{\partial m} &= \dfrac{1}{N} \sum_{i=1}^N 0 - 2x_iy_i - 0 + 2x_i^2m + 2x_ib + 0 \\
                               &= \dfrac{-2}{N} \sum_{i=1}^N x_i(y_i - (mx_i + b))
\end{align}
$

$
\begin{align}
\dfrac{\partial E}{\partial b} &= \dfrac{1}{N} \sum_{i=1}^N 0 - 0 - 2y_i + 0 + 2x_im + 2b \\
                               &= \dfrac{-2}{N} \sum_{i=1}^N (y_i - (mx_i + b))
\end{align}
$

## Implementation

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

def plot_fit(points, m, b, step, error):
    '''
    Plots a scatterplot of points, with a line defined by m and b.
    Also shows the current step number in Gradient Descent and
    current error value.
    '''

    x_values = [p[0] for p in points]
    y_values = [p[1] for p in points]

    plt.scatter(x_values, y_values, color='skyblue')
    fit_xvalues = list(range(int(min(x_values)), int(max(x_values))))
    fit_yvalues = [m * fit_x + b for fit_x in fit_xvalues]
    plt.plot(fit_xvalues, fit_yvalues, color='blue')
    plt.title('Step: ' + str(step) + ', Error:' + str(round(error, 2)))
    plt.show()

def mean_squared_error(points, m, b):
    '''
    Calculates Mean Squared Error for the line defined by m & b
    with the given points.
    '''
    n = len(points)
    error = 0

    for i in range(n):
        x_i = points[i, 0]
        y_i = points[i, 1]

        error += (y_i - (m * x_i + b)) **2

    error = float(error) / n

    return error

def gradient_step(points, current_m, current_b, learning_rate):
    '''
    Calculates gradients and new values for m and b based on
    current values of m and b and given learning rate.
    '''
    gradient_m = 0
    gradient_b = 0

    n = len(points)

    for i in range(n):
        x_i = float(points[i, 0])
        y_i = float(points[i, 1])

        gradient_m += (-2 / n) * (x_i * (y_i - (current_m * x_i + current_b)))
        gradient_b += (-2 / n) * (y_i - (current_m * x_i + current_b))

    new_m = current_m - learning_rate * gradient_m
    new_b = current_b - learning_rate * gradient_b
    new_error = mean_squared_error(points, new_m, new_b)

    return [new_m, new_b, new_error]

def gradient_descent(points, initial_m, initial_b, learning_rate, num_steps):
    '''
    Runs Gradient Descent for the given points, starting with the initial
    m and b values, and the given learning rate for the given number of steps.
    '''
    m = initial_m
    b = initial_b

    error_vec = list()

    # Update m and b for each gradient descent step
    for step in range(num_steps):

        [m, b, error] = gradient_step(points, m, b, learning_rate)

        # Plot the current fit once every 100 steps to track progress
        if not step % 100:
            plot_fit(points, m, b, step, error)

        error_vec.append(error)

    # Plot the error function value for each step
    plt.plot(error_vec, color='red')
    plt.xlabel('Number of Steps')
    plt.ylabel('Mean Squared Error')
    plt.show()

    return [m, b]

def run():
    points = np.genfromtxt('data.csv', delimiter=',')

    initial_m = 0
    initial_b = 0

    learning_rate = 0.0001
    num_steps = 1000

    m, b = gradient_descent(points, initial_m, initial_b, learning_rate, num_steps)

if __name__ == '__main__':
    run()
```


## Symbolic Mathematics with [Sympy](http://docs.sympy.org/latest/tutorial/intro.html)

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
```

#### declare symbolic variables

```python
x = sym.Symbol('x')
y = sym.Symbol('y')
```

or

```python
x, y = sym.symbols('x y')
```

#### declare the matematical expression $xy\exp[-(x^2+y^2)]$

```python
expr = x*y*sym.exp(-(x**2+y**2))
```

#### print the `LaTeX` expression

```python
sym.latex(expr)
```

#### expand an expression

```python
expr.expand()
```

```python
sym.expand( (x-y)**3 )
```

```python
sym.expand_trig(sym.cos(2*x))
```

#### simplify an expression

```python
sym.simplify( sym.sin(x) * ( sym.sin(x)**2 + sym.cos(x)**2 ) / sym.tan(x) )
```

#### Task: declare and simplify the expression $\left(2 \cos^{2}{\left (u \right )} - 1\right) \sin{\left (v \right )} + 2 \sin{\left (u \right )} \cos{\left (u \right )} \cos{\left (v \right )}$

```python
x,y = sym.symbols('u v')
expr = (2*sym.cos(x)**2 - 1)*sym.sin(y) + 2*sym.sin(x)*sym.cos(x)*sym.cos(y)
simplified = sym.simplify(expr)
print( sym.latex(simplified) )
```

* Limit, Integral, and Derivative represent the unevaluated calculation
* limit, integrate, and diff perform the calculation

#### limits: calculate $\lim_{x \to \frac{\pi}{2}} \tan{\left (x \right )}$

```python
x = sym.Symbol('x')
```

```python
print( sym.latex( sym.Limit( sym.tan(x), x, sym.pi/2, dir='+' ) ), '=' )
sym.limit( sym.tan(x), x, sym.pi/2, dir='+' )
```

```python
sym.limit( sym.tan(x), x, sym.pi/2, dir='-' )
```

#### Task: using `sym.Integral`,  `sym.integrate`, and  `sym.latex` calculate and print the integral $\int x y e^{- x^{2} - y^{2}}\, dx$ and its solution

```python
x,y = sym.symbols('x y')
expr = x*y*sym.exp(-(x**2+y**2))
print( sym.latex( sym.Integral(expr, x) ), '=', sym.latex( sym.integrate(expr, x) ))
```

#### Task: calculate and print the integral $\int_{-2}^{2}\int_{-2}^{2} x y e^{- x^{2} - y^{2}}\, dx\, dy$ and its solution

```python
print( sym.latex( sym.Integral(expr, [x,-2,2], [y,-2,2] ) ), '=', sym.latex( sym.integrate(expr, [x,-2,2], [y,-2,2] ) ) )
```


# Daily Coding Problem 14: Find $\pi$ using a Monte Carlo Method

The area of a circle is defined as $\pi r^{2}$. Estimate $\pi$ to 3 decimal places using a Monte Carlo method.

## Visualising the Geometry

The equation of a circle whose centre is at the origin is:

\begin{equation}
r^{2} = x^{2} + y^{2}
\end{equation}
where $r$ is the radius of the circle. As a circle can be thought of as a polygon with an infinite number of sides, a line plot joining a significantly large number of points along the edge of the circle can approximate its shape. Both $x$- and $y$-values vary from $-r$ to $r$, assuming that the centre of the circle lies at the origin. Given the $x$-position, the equation of a circle can be rearranged to yield an equation for $y$:

\begin{equation}
y = \pm\sqrt{\left(r^{2} - x^{2}\right)}
\end{equation}

Simply plotting the positive $y$-values as a function of $x$ will result in a semi-circle. It is necessary to include the negative $y$-values and their corresponding $x$-positions to complete the circle.

This produces the circle below, inscribed in a square of side $2r$:

```python
import matplotlib.pyplot as plt
import numpy as np
import random

def CirclePoints(x, r):
    """Given list of x-coordinates, returns two lists of y-coordinates to plot circle around origin."""
    return np.sqrt(r**2 - x**2), -np.sqrt(r**2 - x**2)

def SquarePoints(r):
    """Given radius of circle, return vertices of square in which circle is inscribed."""
    x = [-r, -r, r, r, -r]
    y = [-r, r, r, -r, -r]
    return x, y

circle_radius = 1.0   # Define radius r of circle

x = np.linspace(-circle_radius, circle_radius, int(circle_radius)*100)   # x-coordinates

# y-coordinates for semi-circles at positive and negative y-values
y_pos, y_neg = CirclePoints(x, circle_radius)
y = np.append(y_pos, list(reversed(y_neg)))   # Merge y-coordinates

# Append x-coordinates with x-values for negative y-positions
x = np.append(x, list(reversed(x)))

# Draw circle
fig0 = plt.figure(figsize=(8,8), facecolor='w', edgecolor='k')
plt.plot(x, y, color='k')
plt.grid()
plt.axis('equal')
plt.xlabel("x (arbitrary units)")
plt.ylabel("y (arbitrary units)")

# Draw square in which circle is inscribed
x_sq, y_sq = SquarePoints(circle_radius)
plt.plot(x_sq, y_sq, color='b')

# Draw line to represent radius of circle
plt.plot([0.0, circle_radius], [0.0, 0.0], color='red')
plt.annotate(r"$r$", xy=(circle_radius/2, circle_radius/16.0), color='r', size=20)

plt.show()
```

## Determination of $\pi$

The area of the circle $A_{c}$ is given by $\pi r^{2}$. The circle is incribed by a square of area $A_{s} = 2r\times 2r = 4r^{2}$. Therefore, the ratio of $A_{c}$ to $A_{s}$ is given by:

\begin{equation}
\frac{A_{c}}{A_{s}} = \frac{\pi r^{2}}{4r^{2}} = \frac{\pi}{4}
\end{equation}

It is apparent from the diagram above that $A_{c} < A_{s}$, therefore it can be inferred that $\pi < 4$. The equation of the ratios can be rearranged to yield one for $\pi$:

\begin{equation}
\pi = 4\frac{A_{c}}{A_{s}}
\end{equation}

In order to lie within the square, a point must have coordinates such that $-r\le x\le r$ and $-r\le y\le r$. If this point also lies within the boundaries of the circle, from the equation of the circle above, it must have coordinates such that:

\begin{equation}
r\ge \sqrt{x^{2} + y^{2}}
\end{equation}

By generating a large number of points with coordinates that lie within the square, and determining the proportion of these that lie within the circle, the ratio $\frac{A_{c}}{A_{s}}$ can be approximated.

The following line can be commented out should the random seed not be predetermined:

```python
# Set random seed; comment out if random seed is not predetermined
random.seed(1)
```

```python
def WithinCircleCheck(x, y, r):
    """Determines whether point with coordinates (x,y) lies within circle of radius r and returns True if so."""
    return r > np.sqrt(x**2 + y**2)

n_points = 1000000   # Number of points to be generated

# Generate lists of x- and y-coordinates for points that lie within the square
points_x = []
points_y = []
for point in range(n_points):
    points_x.append(random.uniform(-circle_radius, circle_radius))
    points_y.append(random.uniform(-circle_radius, circle_radius))

points_x = np.array(points_x)
points_y = np.array(points_y)

# Boolean array to determine whether each generated point lies within circle
within_circle = WithinCircleCheck(points_x, points_y, circle_radius)

# Plot positions of first 100 generated points
if n_points < 100:
    n_plot = n_points
else:
    n_plot = 100

fig1 = plt.figure(figsize=(8,8), facecolor='w', edgecolor='k')
plt.plot(x, y, color='k')
plt.grid()
plt.axis('equal')
plt.xlabel("x (arbitrary units)")
plt.ylabel("y (arbitrary units)")
plt.plot(x_sq, y_sq, color='b')

plt.scatter(points_x[:n_plot], points_y[:n_plot], color='r')
plt.title("First {} Generated Points".format(n_plot))

plt.show()
```

As shown by the positions of the first few generated points above, a proportion of them lie within the circle. Using the equation for $\pi$ given above, its value can be determined by multiplying the proportion of these points by four:

```python
def Determine_pi(prop_in_circle):
    """Returns estimation of pi based on proportion of points that lie within circle."""
    return 4 * list(prop_in_circle).count(True) / float(len(prop_in_circle))

# Print value of pi to 3 decimal places
print("The value of pi = {:.3f}".format(Determine_pi(within_circle)))
```

    The value of pi = 3.141

## The Point of Diminishing Returns

It would be interesting to see how quickly the value converges.

```python
# Lists for pi values and points at sampling positions
pi_values = []
point_gen = []

# Take sample of 100 points and determine pi approximations
for point in range(0, n_points, n_points//n_plot):
    pi_values.append(Determine_pi(within_circle[:point+1]))
    point_gen.append(point+1)

# Plot evolution of approximate pi value with number of generated points
fig2 = plt.figure(figsize=(14,10), facecolor='w', edgecolor='k')
plt.subplot(2, 1, 1)
plt.plot(point_gen, pi_values, color='k')
plt.plot([point_gen[0], point_gen[-1]], [np.pi, np.pi], color='r')
# Show line for pi on plot
plt.annotate(r"$\pi$", xy=(point_gen[-1]/10, np.pi), xytext=(point_gen[-1]/5, np.pi+0.005),
             color='r', size=20, arrowprops=dict(facecolor='r', edgecolor='r', shrink=0.05))
plt.grid()
plt.ylim(3.13, 3.15)
plt.title(r"Improvement in Approximation of $\pi$ with Number of Generated Points")
plt.xlabel("Number of Generated Points")
plt.ylabel(r"Approximation of $\pi$")

# Plot deviation of approximate pi value from true value
plt.subplot(2, 1, 2)
pi_residual = np.array(pi_values) - np.pi
plt.plot(point_gen, pi_residual, color='k')
plt.grid()
plt.ylim(-0.0025, 0.0025)
plt.title(r"Deviation of Approximation of $\pi$ from True Value")
plt.xlabel("Number of Generated Points")
plt.ylabel(r"$\pi$ $-$ (Approximation of $\pi$)")

plt.tight_layout()
plt.show()
```

After about 100,000 generated points, the value of $\pi$ can be approximated to 2 decimal places with reasonable confidence. As expected, as the number of generated points increases, the deviation from the expected value of $\pi$ decreases. However, the magnitude of these improvements decreases with each successive generated point.

It is difficult to state with certainty where the point of diminishing returns lies. In the case of determining $\pi$ to 2 decimal places, it could be said that this point lies after 100,000 generations as no improvement is seen beyond this point. However, the problem asks for a solution to 3 decimal places. After approximately 450,000 points have been generated, the approximate value of $\pi$ lies within the boundaries $3.141\le \pi\le 3.143$. It can be argued that beyond this point, little improvement in the approximation is seen and its value is close enough to the true value to provide a valid answer.

```python

```


## Exercise 03.1

Compare the computed values of

$$
d_0 = a \cdot b + a  \cdot  c
$$

and

$$
d_1 = a \cdot (b + c)
$$

when $a = 100$, $b = 0.1$ and $c = 0.2$. Store $d_{0}$ in the variable `d0` and $d_{1}$ in the variable `d1`.

Try checking for equality, e.g. `print(d0 == d1)`.

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
assert d0 == 30.0
assert d1 != 30.0
assert d0 != d1
```

## Exercise 03.2

For the polynomial
\begin{align}
f(x, y) &= (x + y)^{6}
\\
&=  x^6 + 6x^{5}y + 15x^{4}y^{2} + 20x^{3}y^{3} + 15x^{2}y^{4} + 6xy^{5} + y^{6}
\end{align}
compute $f$ using: (i) the compact form $(x + y)^{6}$; and (ii) the expanded form for:

(a) $x = 10$ and $y = 10.1$

(b) $x = 10$ and $y = -10.1$

and compare the number of significant digits for which the answers are the same.
Store the answer for the compact version using the variable `f0`, and using the variable `f1` for the expanded version.

For case (b), compare the computed and analytical solutions and consider the relative error.
Which approach would you recommend for computing this expression?

#### (a) $x = 10$ and $y = 10.1$

```python
x = 10.0
y = 10.1

# YOUR CODE HERE
raise NotImplementedError()
```

```python
import math
assert math.isclose(f0, 65944160.60120103, rel_tol=1e-10)
assert math.isclose(f1, 65944160.601201, rel_tol=1e-10)
```

#### (b) $x = 10$ and $y = -10.1$

```python
x = 10.0
y = -10.1

# YOUR CODE HERE
raise NotImplementedError()
```

```python
import math
assert math.isclose(f0, 1.0e-6, rel_tol=1e-10)
assert math.isclose(f1, 1.0e-6, rel_tol=1e-2)
```

## Exercise 03.3

Consider the expression

$$
f = \frac{1}{\sqrt{x^2 - 1} - x}
$$

When $x$ is very large, the denominator approaches zero, which can cause problems.

Try rephrasing the problem and eliminating the fraction by multiplying the numerator and denominator by $\sqrt{x^2 - 1} + x$ and evaluate the two versions of the expression when:

(a) $x = 1 \times 10^{7}$

(b) $x = 1 \times 10^{9}$ (You may get a Python error for this case. Why?)

#### (a) $x = 1 \times 10^{7}$

```python
# YOUR CODE HERE
raise NotImplementedError()
```

#### (b) $x = 1 \times 10^{9}$

```python
# YOUR CODE HERE
raise NotImplementedError()
```


## Introduction to Sympy

What if you would like to perform some calculus alongside your other computations? For that, you would need a computer algebra system (CAS). Luckily, the sympy package can provide you with the tools to perform symbolic computations and then can help you numerically evaluate those results.

Let's get started...

```python
# Import sympy tools
import sympy as sp
sp.init_printing(use_latex=True)

# Other tools
import numpy as np
import matplotlib.pyplot as plt
```

Note that I imported sympy, but then also initialized the sympy printing facilities to utilize $\LaTeX$. The reason for this will be obvious soon, if you haven't already figured it out.

Rather than have x be a vector, I need it to be a symbol. In particular, I need it to be a symbol that one would like to manipulate similar to how one uses symbols in mathematical expressions. Unlike variables which point to specific data structures, a symbol is more like how a variable is used in mathematics, and can take on values at a later time.

```python
# Make x a symbol
x = sp.symbols('x')
```

```python
# Let's write an expression
y = sp.cos(x)
```

```python
# Just provide the expression by itself,
# and it will be printed with LaTeX!
y
```

So, you can define mathematical expressions in sympy, and they will be rendered using $\LaTeX$. Don't be confused by the naming conventions though. In this particular case, x is a standard python variable: it just so happens to reference the sympy _symbol_ 'x'. We are naming the python variable the same as the intended mathematical symbol just for parsimony. Similarly, y is a standard python variable, but it now references a sympy expression (which itself is composed of a sympy function defined in terms of the symbol x):

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
type(y)
```

    cos

```python
type(type(y))
```

    sympy.core.function.FunctionClass

```python
y.free_symbols
```

Additionally, we can perform symbolic math on that expression. For example, let's take the derivative with respect to x using the `diff()` function.

```python
dydx = y.diff(x)
```

```python
dydx
```

Now we have the derivative of the function with respect to x, and have solved it __analytically__ (rather than numerically) using sympy! The `diff()` function is a member function of the function class, but you can also use the general `diff()` function if desired:

```python
sp.diff(y,x)
```

Sympy has it's own builtin function which utilize matplotlib underneath for plotting expressions as well...

```python
sp.plot(dydx)
```

However, we may want more control over the x values sampled and plot itself. This can sometimes be better done by evaluating the function _numerically_ at the intended points. Let's do that now...

```python
x_vals = np.linspace(-2*np.pi,2*np.pi,101)
y_vals = np.array([dydx.evalf(subs=dict(x=x_val)) for x_val in x_vals])
print('The length of x is %d'%(len(x_vals)))
print('The length of y is %d'%(len(y_vals)))
```

    The length of x is 101
    The length of y is 101

Here we have used a python list comprehension to evaluate our derivative (dydx) at each of the 101 points in the $-2\pi$ to $2\pi$ range created by `linspace()`. The `evalf()` function allows us plug in specific numberic values for our symbols. In particular, we pass the subs= argument a python dictionary object which contains a mapping from a symbol to a particular value we would like to associate with that symbol. Multiple symbols can be passed into the function using the dictionary object, so that functions with more than one symbol can be evaluated numerically as well. We can now plot those specific values and maintain full control over the appearance of the plot itself since we are using matplotlib/pyplot directly:

```python
plt.plot(x_vals,y_vals)
plt.title(r'$y=%s$'%(sp.latex(dydx)))
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

You can also now see in the code above how the `sympy.latex()` function can be used to convert the expression we were storing in dydx into a string form recognized by $\LaTeX$ math mode, and therefore also the `pyplot.title()` function from matplotlib. It's usually much easier to control how you want your plots to look using numerical evaluation instead of using sympy's built-in plotting tools, so keep that in mind in the future.


# Task for lecture 19

## 1. Find the extrema in the function $f(x)=x^3-7.5x^2+18x-10$ analytically and determine if they are minimum or maximum.

```python
# import Python libraries
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
from sympy.interactive import printing
printing.init_printing()
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

```python
a, b = sym.symbols('a b')
V = b*(a - 2*b)*(a - 2*b)
Vdiff = sym.expand(sym.diff(V, b))
roots = sym.solve(Vdiff, b)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

For a function of one variable, if the maximum or minimum of a function is not at the limits of the domain and if at least the first and second derivatives of the function exist, a maximum and minimum can be found as the point where the first derivative of the function is zero.

```python
# Question 1
x = sym.symbols('x')
y = x**3 -7.5*x**2 + 18*x -10
dy = sym.diff(y, x)
dy
```

```python
roots = sym.solve(dy, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.0, \quad 3.0\right ]$$

If the second derivative on that point is positive, then it's a minimum, if it is negative, it's a maximum.
So, let's perform the second derivative.

```python
dy2 = sym.diff(dy, x)
dy2
```

```python
roots2 = sym.solve(dy2, x)
display(Math(sym.latex('Roots:') + sym.latex(roots2)))
```

$$Roots:\left [ 2.5\right ]$$

## 2. Find the minimum in the $f(x)=x^3-7.5x^2+18x-10$ using the gradient descent algorithm.

```python
# From https://en.wikipedia.org/wiki/Gradient_descent

cur_x = 6               # The algorithm starts at x=6
gamma = 0.01            # step size multiplier
precision = 0.00001
step_size = 1           # initial step size
max_iters = 10000       # maximum number of iterations
iters = 0               # iteration counter

f  = lambda x: x**3 - 7.5*x**2 + 18*x - 10  # lambda function for f(x)
df = lambda x: 3*x**2 - 15*x + 18    # lambda function for the gradient of f(x)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x -= gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local minimum at {} with function value {}.'.format(3, f(3)))
print('Local minimum by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

    True local minimum at 3 with function value 3.5.
    Local minimum by gradient descent at 3.000323195755751 with function value 3.5000001567170003.

courses/modsim2018/reginaldo/Task for lecture 19.ipynb


```python
import numpy as np
from numpy import linalg as la
```

**1. Function  Plot**

with function *meshgrid*, *contour* and *contourf*

$
f(x) = 10 (x_2 - x_1^2)^2 + (1-x_1)^2
$

```python
import matplotlib
import matplotlib.pyplot as plt

#f(x) = 10 (x2 - x1**2)**2 + (1-x1)**2
def rosen(x):
  f = 10 * (x[1] - x[0]**2)**2 + (1-x[0])**2
  return f

nx, ny = (240, 200)
xv = np.linspace(-1.2, 1.2, nx)
xh = np.linspace(-0.5,1.5, ny)
x0, x1 = np.meshgrid(xv, xh, sparse= True)

F = np.zeros((x1.shape[0],x0.shape[1])) # shape (200,240)

for i in range(F.shape[0]):
  for j in range(F.shape[1]):
    x = [x0[0,j], x1[i,0]]
    F[i, j] = rosen(x)

plt.figure('Contours')
plt.contour(x0[0,:], x1[:,0], F, 50)
plt.axis('scaled')
plt.colorbar()
plt.show()

plt.figure('Contours')
plt.contourf(x0[0,:], x1[:,0], F, 50)
plt.axis('scaled')
plt.colorbar()
plt.show()
```

**2. Gradient Computation**

with fucntion *symbols* and *diff*

$
∇f=\begin{bmatrix}
    -40x_1(x_2-x_1^2)+2x_1-2 \\
    20*(x_2-x_1^2)
   \end{bmatrix}
$

```python
from sympy import symbols, diff
x_1, x_2 = symbols('x_1 x_2', real= True)
g0 = diff((10 * (x_2 - x_1**2)**2 + (1-x_1)**2),x_1)
g1 = diff((10 * (x_2 - x_1**2)**2 + (1-x_1)**2),x_2)

def rosen_grad(x):
  g = np.zeros(2)
  g[0] = g0.subs({x_1:x[0], x_2:x[1]})
  g[1] = g1.subs({x_1:x[0], x_2:x[1]})
  return g
```

**3. Backtracking Line Search**

```python
def backtrack_linesearch(f, gk, pk, xk, alpha = 0.1, beta = 0.8): # Algorithm parameters alpha and beta
  t = 1
  while(f(xk + t*pk) > f(xk) + alpha * t * gk @ pk):
    t *= beta # reduce t incrementally
  return t

def steepest_descent_bt(f, grad, x0):
  tol = 1e-5 # converge to a gradient norm of 1e-5
  x = x0
  history = np.array( [x0] )
  while ( la.norm(grad(x)) > tol ):
    p = -grad(x)
    t = backtrack_linesearch(f, grad(x), p, x)
    x += t * p
    history = np.vstack( (history, x) )# The returned array formed by stacking the given arrays, will be at least 2-D.
  return x, history

# plot convergence behaviour
x_startpoint = np.array([-1.2, 1.0]) # start point

xstar, hist = steepest_descent_bt(rosen, rosen_grad, x_startpoint)
nsteps = hist.shape[0]

print('Optimal solution:',xstar)
print('The minima is ',rosen(xstar))
print('Iteration count:', nsteps)
```

    Optimal solution: [1.00000578 1.00001197]
    The minima is  3.514046547976252e-11
    Iteration count: 1311

**4. Convergence Behavior**

```python
fhist = np.zeros(nsteps)
for i in range(nsteps):
  fhist[i] = rosen(hist[i,:])

plt.figure('Convergence behaviour')
plt.semilogy(np.arange(0, nsteps), np.absolute(fhist))
plt.grid(True, which ="both")
plt.title('Convergence of Steepest Descent')
#plt.text(0,10e-10,'Y axis in Semilogy')    #text annotation
plt.xlabel('Iteration count')
plt.ylabel(r'$|f^k - f^*|$')
plt.show()

plt.figure('Contours Behavior')
plt.title('Contours Behavior')
plt.contour(x0[0,:], x1[:,0], F, 150)
plt.axis('scaled')
plt.plot(hist[:,0],hist[:,1],'r-')
plt.colorbar()
plt.show()
```

On a semilog plot, $|f^k-f^*|$ vs *k* looks like a straight piecewise line segment.From the figure *Contours Behaviour*, the rate of convergence is very fast, but the search path is different with the samples given by slides.


Suggestions for lab exercises.

# Variables and assignment

## Exercise 1

Remember that $n! = n \times (n - 1) \times \dots \times 2 \times 1$. Compute $15!$, assigning the result to a sensible variable name.

## Exercise 2

Using the `math` module, check your result for $15$ factorial. You should explore the help for the `math` library and its functions, using eg tab-completion, the spyder inspector, or online sources.

## Exercise 3

[Stirling's approximation](http://mathworld.wolfram.com/StirlingsApproximation.html) gives that, for large enough $n$,

\begin{equation}
  n! \simeq \sqrt{2 \pi} n^{n + 1/2} e^{-n}.
\end{equation}

Using functions and constants from the `math` library, compare the results of $n!$ and Stirling's approximation for $n = 5, 10, 15, 20$. In what sense does the approximation improve?

# Basic functions

## Exercise 1

Write a function to calculate the volume of a cuboid with edge lengths $a, b, c$. Test your code on sample values such as

1. $a=1, b=1, c=1$ (result should be $1$);
2. $a=1, b=2, c=3.5$ (result should be $7.0$);
3. $a=0, b=1, c=1$ (result should be $0$);
4. $a=2, b=-1, c=1$ (what do you think the result should be?).

## Exercise 2

Write a function to compute the time (in seconds) taken for an object to fall from a height $H$ (in metres) to the ground, using the formula
\begin{equation}
  h(t) = \frac{1}{2} g t^2.
\end{equation}
Use the value of the acceleration due to gravity $g$ from `scipy.constants.g`. Test your code on sample values such as

1. $H = 1$m (result should be $\approx 0.452$s);
2. $H = 10$m (result should be $\approx 1.428$s);
3. $H = 0$m (result should be $0$s);
4. $H = -1$m (what do you think the result should be?).

## Exercise 3

Write a function that computes the area of a triangle with edge lengths $a, b, c$. You may use the formula
\begin{equation}
  A = \sqrt{s (s - a) (s - b) (s - c)}, \qquad s = \frac{a + b + c}{2}.
\end{equation}

Construct your own test cases to cover a range of possibilities.

# Floating point numbers

## Exercise 1

Computers cannot, in principle, represent real numbers perfectly. This can lead to problems of accuracy. For example, if
\begin{equation}
  x = 1, \qquad y = 1 + 10^{-14} \sqrt{3}
\end{equation}
then it *should* be true that
\begin{equation}
  10^{14} (y - x) = \sqrt{3}.
\end{equation}
Check how accurately this equation holds in Python and see what this implies about the accuracy of subtracting two numbers that are close together.

## Exercise 2

The standard quadratic formula gives the solutions to

\begin{equation}
  a x^2 + b x + c = 0
\end{equation}

as

\begin{equation}
  x = \frac{-b \pm \sqrt{b^2 - 4 a c}}{2 a}.
\end{equation}

Show that, if $a = 10^{-n} = c$ and $b = 10^n$ then

\begin{equation}
  x = \frac{10^{2 n}}{2} \left( -1 \pm \sqrt{1 - 10^{-4n}} \right).
\end{equation}

Using the expansion (from Taylor's theorem)

\begin{equation}
  \sqrt{1 - 10^{-4 n}} \simeq 1 - \frac{10^{-4 n}}{2} + \dots, \qquad n \gg 1,
\end{equation}

show that

\begin{equation}
  x \simeq -10^{2 n} + \frac{10^{-2 n}}{4} \quad \text{and} \quad -\frac{10^{-2n}}{4}, \qquad n \gg 1.
\end{equation}

## Exercise 3

By multiplying and dividing by $-b \mp \sqrt{b^2 - 4 a c}$, check that we can also write the solutions to the quadratic equation as

\begin{equation}
  x = \frac{2 c}{-b \mp \sqrt{b^2 - 4 a c}}.
\end{equation}

## Exercise 4

Using Python, calculate both solutions to the quadratic equation

\begin{equation}
  10^{-n} x^2 + 10^n x + 10^{-n} = 0
\end{equation}

for $n = 3$ and $n = 4$ using both formulas. What do you see? How has floating point accuracy caused problems here?

## Exercise 5

The standard definition of the derivative of a function is

\begin{equation}
  \left. \frac{\text{d} f}{\text{d} x} \right|_{x=X} = \lim_{\delta \to 0} \frac{f(X + \delta) - f(X)}{\delta}.
\end{equation}

We can *approximate* this by computing the result for a *finite* value of $\delta$:

\begin{equation}
  g(x, \delta) = \frac{f(x + \delta) - f(x)}{\delta}.
\end{equation}

Write a function that takes as inputs a function of one variable, $f(x)$, a location $X$, and a step length $\delta$, and returns the approximation to the derivative given by $g$.

## Exercise 6

The function $f_1(x) = e^x$ has derivative with the exact value $1$ at $x=0$. Compute the approximate derivative using your function above, for $\delta = 10^{-2 n}$ with $n = 1, \dots, 7$. You should see the results initially improve, then get worse. Why is this?

# Prime numbers

## Exercise 1

Write a function that tests if a number is prime. Test it by writing out all prime numbers less than 50.

## Exercise 2

500 years ago some believed that the number $2^n - 1$ was prime for *all* primes $n$. Use your function to find the first prime $n$ for which this is not true.

## Exercise 3

The *Mersenne* primes are those that have the form $2^n-1$, where $n$ is prime. Use your previous solutions to generate all the $n < 40$ that give Mersenne primes.

## Exercise 4

Write a function to compute all prime factors of an integer $n$, including their multiplicities. Test it by printing the prime factors (without multiplicities) of $n = 17, \dots, 20$ and the multiplicities (without factors) of $n = 48$.

##### Note

One effective solution is to return a *dictionary*, where the keys are the factors and the values are the multiplicities.

## Exercise 5

Write a function to generate all the integer divisors, including 1, but not including $n$ itself, of an integer $n$. Test it on $n = 16, \dots, 20$.

##### Note

You could use the prime factorization from the previous exercise, or you could do it directly.

## Exercise 6

A *perfect* number $n$ is one where the divisors sum to $n$. For example, 6 has divisors 1, 2, and 3, which sum to 6. Use your previous solution to find all perfect numbers $n < 10,000$ (there are only four!).

## Exercise 7

Using your previous functions, check that all perfect numbers $n < 10,000$ can be written as $2^{k-1} \times (2^k - 1)$, where $2^k-1$ is a Mersenne prime.

## Exercise 8 (bonus)

Investigate the `timeit` function in python or IPython. Use this to measure how long your function takes to check that, if $k$ on the Mersenne list then $n = 2^{k-1} \times (2^k - 1)$ is a perfect number, using your functions. Stop increasing $k$ when the time takes too long!

##### Note

You could waste considerable time on this, and on optimizing the functions above to work efficiently. It is *not* worth it, other than to show how rapidly the computation time can grow!

# Logistic map

Partly taken from Newman's book, p 120.

The logistic map builds a sequence of numbers $\{ x_n \}$ using the relation

\begin{equation}
  x_{n+1} = r x_n \left( 1 - x_n \right),
\end{equation}

where $0 \le x_0 \le 1$.

## Exercise 1

Write a program that calculates the first $N$ members of the sequence, given as input $x_0$ and $r$ (and, of course, $N$).

## Exercise 2

Fix $x_0=0.5$. Calculate the first 2,000 members of the sequence for $r=1.5$ and $r=3.5$ Plot the last 100 members of the sequence in both cases.

What does this suggest about the long-term behaviour of the sequence?

## Exercise 3

Fix $x_0 = 0.5$. For each value of $r$ between $1$ and $4$, in steps of $0.01$, calculate the first 2,000 members of the sequence. Plot the last 1,000 members of the sequence on a plot where the $x$-axis is the value of $r$ and the $y$-axis is the values in the sequence. Do not plot lines - just plot markers (e.g., use the `'k.'` plotting style).

## Exercise 4

For iterative maps such as the logistic map, one of three things can occur:

1. The sequence settles down to a *fixed point*.
2. The sequence rotates through a finite number of values. This is called a *limit cycle*.
3. The sequence generates an infinite number of values. This is called *deterministic chaos*.

Using just your plot, or new plots from this data, work out approximate values of $r$ for which there is a transition from fixed points to limit cycles, from limit cycles of a given number of values to more values, and the transition to chaos.

# Mandelbrot

The Mandelbrot set is also generated from a sequence, $\{ z_n \}$, using the relation

\begin{equation}
  z_{n+1} = z_n^2 + c, \qquad z_0 = 0.
\end{equation}

The members of the sequence, and the constant $c$, are all complex. The point in the complex plane at $c$ is in the Mandelbrot set only if the $\|z_n\| < 2$ for all members of the sequence. In reality, checking the first 100 iterations is sufficient.

Note: the python notation for a complex number $x + \text{i} y$ is `x + yj`: that is, `j` is used to indicate $\sqrt{-1}$. If you know the values of `x` and `y` then `x + yj` constructs a complex number; if they are stored in variables you can use `complex(x, y)`.

## Exercise 1

Write a function that checks if the point $c$ is in the Mandelbrot set.

## Exercise 2

Check the points $c=0$ and $c=\pm 2 \pm 2 \text{i}$ and ensure they do what you expect. (What *should* you expect?)

## Exercise 3

Write a function that, given $N$

1. generates an $N \times N$ grid spanning $c = x + \text{i} y$, for $-2 \le x \le 2$ and $-2 \le y \le 2$;
2. returns an $N\times N$ array containing one if the associated grid point is in the Mandelbrot set, and zero otherwise.

## Exercise 4

Using the function `imshow` from `matplotlib`, plot the resulting array for a $100 \times 100$ array to make sure you see the expected shape.

## Exercise 5

Modify your functions so that, instead of returning whether a point is inside the set or not, it returns the logarithm of the number of iterations it takes. Plot the result using `imshow` again.

## Exercise 6

Try some higher resolution plots, and try plotting only a section to see the structure. **Note** this is not a good way to get high accuracy close up images!

# Equivalence classes

An *equivalence class* is a relation that groups objects in a set into related subsets. For example, if we think of the integers modulo $7$, then $1$ is in the same equivalence class as $8$ (and $15$, and $22$, and so on), and $3$ is in the same equivalence class as $10$. We use the tilde $3 \sim 10$ to denote two objects within the same equivalence class.

Here, we are going to define the positive integers programmatically from equivalent sequences.

## Exercise 1

Define a python class `Eqint`. This should be

1. Initialized by a sequence;
2. Store the sequence;
3. Define its representation (via the `__repr__` function) to be the integer length of the sequence;
4. Redefine equality (via the `__eq__` function) so that two `eqint`s are equal if their sequences have same length.

## Exercise 2

Define a `zero` object from the empty list, and three `one` objects, from a single object list, tuple, and string. For example

```python
one_list = Eqint([1])
one_tuple = Eqint((1,))
one_string = Eqint('1')
```

Check that none of the `one` objects equal the zero object, but all equal the other `one` objects. Print each object to check that the representation gives the integer length.

## Exercise 3

Redefine the class by including an `__add__` method that combines the two sequences. That is, if `a` and `b` are `Eqint`s then `a+b` should return an `Eqint` defined from combining `a` and `b`s sequences.

##### Note

Adding two different *types* of sequences (eg, a list to a tuple) does not work, so it is better to either iterate over the sequences, or to convert to a uniform type before adding.

## Exercise 4

Check your addition function by adding together all your previous `Eqint` objects (which will need re-defining, as the class has been redefined). Print the resulting object to check you get `3`, and also print its internal sequence.

## Exercise 5

We will sketch a construction of the positive integers from *nothing*.

1. Define an empty list `positive_integers`.
2. Define an `Eqint` called `zero` from the empty list. Append it to `positive_integers`.
3. Define an `Eqint` called `next_integer` from the `Eqint` defined by *a copy of* `positive_integers` (ie, use `Eqint(list(positive_integers))`. Append it to `positive_integers`.
4. Repeat step 3 as often as needed.

Use this procedure to define the `Eqint` equivalent to $10$. Print it, and its internal sequence, to check.

# Rational numbers

Instead of working with floating point numbers, which are not "exact", we could work with the rational numbers $\mathbb{Q}$. A rational number $q \in \mathbb{Q}$ is defined by the *numerator* $n$ and *denominator* $d$ as $q = \frac{n}{d}$, where $n$ and $d$ are *coprime* (ie, have no common divisor other than $1$).

## Exercise 1

Find a python function that finds the greatest common divisor (`gcd`) of two numbers. Use this to write a function `normal_form` that takes a numerator and divisor and returns the coprime $n$ and $d$. Test this function on $q = \frac{3}{2}$, $q = \frac{15}{3}$, and $q = \frac{20}{42}$.

## Exercise 2

Define a class `Rational` that uses the `normal_form` function to store the rational number in the appropriate form. Define a `__repr__` function that prints a string that *looks like* $\frac{n}{d}$ (**hint**: use `len(str(number))` to find the number of digits of an integer). Test it on the cases above.

## Exercise 3

Overload the `__add__` function so that you can add two rational numbers. Test it on $\frac{1}{2} + \frac{1}{3} + \frac{1}{6} = 1$.

## Exercise 4

Overload the `__mul__` function so that you can multiply two rational numbers. Test it on $\frac{1}{3} \times \frac{15}{2} \times \frac{2}{5} = 1$.

## Exercise 5

Overload the [`__rmul__`](https://docs.python.org/2/reference/datamodel.html?highlight=rmul#object.__rmul__) function so that you can multiply a rational by an *integer*. Check that $\frac{1}{2} \times 2 = 1$ and $\frac{1}{2} + (-1) \times \frac{1}{2} = 0$. Also overload the `__sub__` function (using previous functions!) so that you can subtract rational numbers and check that $\frac{1}{2} - \frac{1}{2} = 0$.

## Exercise 6

Overload the `__float__` function so that `float(q)` returns the floating point approximation to the rational number `q`. Test this on $\frac{1}{2}, \frac{1}{3}$, and $\frac{1}{11}$.

## Exercise 7

Overload the `__lt__` function to compare two rational numbers. Create a list of rational numbers where the denominator is $n = 2, \dots, 11$ and the numerator is the floored integer $n/2$, ie `n//2`. Use the `sorted` function on that list (which relies on the `__lt__` function).

## Exercise 8

The [Wallis formula for $\pi$](http://mathworld.wolfram.com/WallisFormula.html) is

\begin{equation}
  \pi = 2 \prod_{n=1}^{\infty} \frac{ (2 n)^2 }{(2 n - 1) (2 n + 1)}.
\end{equation}

We can define a partial product $\pi_N$ as

\begin{equation}
  \pi_N = 2 \prod_{n=1}^{N} \frac{ (2 n)^2 }{(2 n - 1) (2 n + 1)},
\end{equation}

each of which are rational numbers.

Construct a list of the first 20 rational number approximations to $\pi$ and print them out. Print the sorted list to show that the approximations are always increasing. Then convert them to floating point numbers, construct a `numpy` array, and subtract this array from $\pi$ to see how accurate they are.

# The shortest published Mathematical paper

A [candidate for the shortest mathematical paper ever](http://www.ams.org/journals/bull/1966-72-06/S0002-9904-1966-11654-3/S0002-9904-1966-11654-3.pdf) shows the following result:

\begin{equation}
  27^5 + 84^5 + 110^5 + 133^5 = 144^5.
\end{equation}

This is interesting as

> This is a counterexample to a conjecture by Euler ... that at least $n$ $n$th powers are required to sum to an $n$th power, $n > 2$.

## Exercise 1

Using python, check the equation above is true.

## Exercise 2

The more interesting statement in the paper is that

\begin{equation}
  27^5 + 84^5 + 110^5 + 133^5 = 144^5.
\end{equation}

> [is] the smallest instance in which four fifth powers sum to a fifth power.

Interpreting "the smallest instance" to mean the solution where the right hand side term (the largest integer) is the smallest, we want to use python to check this statement.

You may find the `combinations` function from the `itertools` package useful.

```python
import numpy
import itertools
```

The `combinations` function returns all the combinations (ignoring order) of `r` elements from a given list. For example, take a list of length 6, `[1, 2, 3, 4, 5, 6]` and compute all the combinations of length 4:

```python
input_list = numpy.arange(1, 7)
combinations = list(itertools.combinations(input_list, 4))
print(combinations)
```

    [(1, 2, 3, 4), (1, 2, 3, 5), (1, 2, 3, 6), (1, 2, 4, 5), (1, 2, 4, 6), (1, 2, 5, 6), (1, 3, 4, 5), (1, 3, 4, 6), (1, 3, 5, 6), (1, 4, 5, 6), (2, 3, 4, 5), (2, 3, 4, 6), (2, 3, 5, 6), (2, 4, 5, 6), (3, 4, 5, 6)]

We can already see that the number of terms to consider is large.

Note that we have used the `list` function to explicitly get a list of the combinations. The `combinations` function returns a *generator*, which can be used in a loop as if it were a list, without storing all elements of the list.

How fast does the number of combinations grow? The standard formula says that for a list of length $n$ there are

\begin{equation}
  \begin{pmatrix} n \\ k \end{pmatrix} = \frac{n!}{k! (n-k)!}
\end{equation}

combinations of length $k$. For $k=4$ as needed here we will have $n (n-1) (n-2) (n-3) / 24$ combinations. For $n=144$ we therefore have

```python
n_combinations = 144*143*142*141/24
print("Number of combinations of 4 objects from 144 is {}".format(n_combinations))
```

    Number of combinations of 4 objects from 144 is 17178876

### Exercise 2a

Show, by getting python to compute the number of combinations $N = \begin{pmatrix} n \\ 4 \end{pmatrix}$ that $N$ grows roughly as $n^4$. To do this, plot the number of combinations and $n^4$ on a log-log scale. Restrict to $n \le 50$.

With 17 million combinations to work with, we'll need to be a little careful how we compute.

One thing we could try is to loop through each possible "smallest instance" (the term on the right hand side) in increasing order. We then check all possible combinations of left hand sides.

This is computationally *very expensive* as we repeat a lot of calculations. We repeatedly recalculate combinations (a bad idea). We repeatedly recalculate the powers of the same number.

Instead, let us try creating the list of all combinations of powers once.

### Exercise 2b

1. Construct a `numpy` array containing all integers in $1, \dots, 144$ to the fifth power.
2. Construct a list of all combinations of four elements from this array.
3. Construct a list of sums of all these combinations.
4. Loop over one list and check if the entry appears in the other list (ie, use the `in` keyword).

# Lorenz attractor

The Lorenz system is a set of ordinary differential equations which can be written

\begin{equation}
  \frac{\text{d} \vec{v}}{\text{d} \vec{t}} = \vec{f}(\vec{v})
\end{equation}

where the variables in the state vector $\vec{v}$ are

\begin{equation}
  \vec{v} = \begin{pmatrix} x(t) \\ y(t) \\ z(t) \end{pmatrix}
\end{equation}

and the function defining the ODE is

\begin{equation}
  \vec{f} = \begin{pmatrix} \sigma \left( y(t) - x(t) \right) \\ x(t) \left( \rho - z(t) \right) - y(t) \\ x(t) y(t) - \beta z(t) \end{pmatrix}.
\end{equation}

The parameters $\sigma, \rho, \beta$ are all real numbers.

## Exercise 1

Write a function `dvdt(v, t, params)` that returns $\vec{f}$ given $\vec{v}, t$ and the parameters $\sigma, \rho, \beta$.

## Exercise 2

Fix $\sigma=10, \beta=8/3$. Set initial data to be $\vec{v}(0) = \vec{1}$. Using `scipy`, specifically the `odeint` function of `scipy.integrate`, solve the Lorenz system up to $t=100$ for $\rho=13, 14, 15$ and $28$.

Plot your results in 3d, plotting $x, y, z$.

## Exercise 3

Fix $\rho = 28$. Solve the Lorenz system twice, up to $t=40$, using the two different initial conditions $\vec{v}(0) = \vec{1}$ and $\vec{v}(0) = \vec{1} + \vec{10^{-5}}$.

Show four plots. Each plot should show the two solutions on the same axes, plotting $x, y$ and $z$. Each plot should show $10$ units of time, ie the first shows $t \in [0, 10]$, the second shows $t \in [10, 20]$, and so on.

This shows the *sensitive dependence on initial conditions* that is characteristic of chaotic behaviour.


# More Matrices
This notebook continues your exploration of matrices.

## Matrix Multiplication
Multiplying matrices is a little more complex than the operations we've seen so far. There are two cases to consider, *scalar multiplication* (multiplying a matrix by a single number), and *dot product matrix multiplication* (multiplying a matrix by another matrix.
### Scalar Multiplication
To multiply a matrix by a scalar value, you just multiply each element by the scalar to produce a new matrix:

\begin{equation}2 \times \begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\end{bmatrix} = \begin{bmatrix}2 & 4 & 6 \\8 & 10 & 12\end{bmatrix}\end{equation}

In Python, you perform this calculation using the **\*** operator:

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6]])
print(2 * A)
```

### Dot Product Matrix Multiplication
To mulitply two matrices together, you need to calculate the *dot product* of rows and columns. This means multiplying each of the elements in each row of the first matrix by each of the elements in each column of the second matrix and adding the results. We perform this operation by applying the *RC* rule - always multiplying ***R***ows by ***C***olumns. For this to work, the number of ***columns*** in the first matrix must be the same as the number of ***rows*** in the second matrix so that the matrices are *conformable* for the dot product operation.

Sounds confusing, right?

Let's look at an example:

\begin{equation}\begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\end{bmatrix} \cdot \begin{bmatrix}9 & 8 \\ 7 & 6 \\ 5 & 4\end{bmatrix}\end{equation}

Note that the first matrix is 2x3, and the second matrix is 3x2. The important thing here is that the first matrix has two rows, and the second matrix has two columns. To perform the multiplication, we first take the dot product of the first ***row*** of the first matrix (1,2,3) and the first ***column*** of the second matrix (9,7,5):

\begin{equation}(1,2,3) \cdot (9,7,5) = (1 \times 9) + (2 \times 7) + (3 \times 5) = 38\end{equation}

In our resulting matrix (which will always have the same number of ***rows*** as the first matrix, and the same number of ***columns*** as the second matrix), we can enter this into the first row and first column element:

\begin{equation}\begin{bmatrix}38 & ?\\? & ?\end{bmatrix} \end{equation}

Now we can take the dot product of the first row of the first matrix and the second column of the second matrix:

\begin{equation}(1,2,3) \cdot (8,6,4) = (1 \times 8) + (2 \times 6) + (3 \times 4) = 32\end{equation}

Let's add that to our resulting matrix in the first row and second column element:

\begin{equation}\begin{bmatrix}38 & 32\\? & ?\end{bmatrix} \end{equation}

Now we can repeat this process for the second row of the first matrix and the first column of the second matrix:

\begin{equation}(4,5,6) \cdot (9,7,5) = (4 \times 9) + (5 \times 7) + (6 \times 5) = 101\end{equation}

Which fills in the next element in the result:

\begin{equation}\begin{bmatrix}38 & 32\\101 & ?\end{bmatrix} \end{equation}

Finally, we get the dot product for the second row of the first matrix and the second column of the second matrix:

\begin{equation}(4,5,6) \cdot (8,6,4) = (4 \times 8) + (5 \times 6) + (6 \times 4) = 86\end{equation}

Giving us:

\begin{equation}\begin{bmatrix}38 & 32\\101 & 86\end{bmatrix} \end{equation}

In Python, you can use the *numpy.**dot*** function or the **@** operator to multiply matrices and two-dimensional arrays:

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6]])
B = np.array([[9,8],
              [7,6],
              [5,4]])
print(np.dot(A,B))
print(A @ B)
```

This is one case where there is a difference in behavior between *numpy.**array*** and *numpy.**matrix***, You can also use a regular multiplication (**\***) operator with a matrix, but not with an array:

```python
import numpy as np

A = np.matrix([[1,2,3]
               ,[4,5,6]])
B = np.matrix([[9,8],
               [7,6],
               [5,4]])
print(A * B)
```

Note that, unlike with multiplication of regular scalar numbers, the order of the operands in a multiplication operation is significant. For scalar numbers, the *commmutative law* of multiplication applies, so for example:

\begin{equation}2 \times 4 = 4 \times 2\end{equation}

With matrix multiplication, things are different, for example:

\begin{equation}\begin{bmatrix}2 & 4 \\6 & 8\end{bmatrix} \cdot \begin{bmatrix}1 & 3 \\ 5 & 7\end{bmatrix} \ne \begin{bmatrix}1 & 3 \\ 5 & 7\end{bmatrix} \cdot \begin{bmatrix}2 & 4 \\6 & 8\end{bmatrix}\end{equation}

Run the following Python code to test this:

```python
import numpy as np

A = np.array([[2,4],
              [6,8]])
B = np.array([[1,3],
              [5,7]])
print(A @ B)
print(B @ A)
```

## Identity Matrices
An *identity* matrix (usually indicated by a capital **I**) is the equivalent in matrix terms of the number **1**. It always has the same number of rows as columns, and it has the value **1** in the diagonal element positions I1,1, I2,2, etc; and 0 in all other element positions. Here's an example of a 3x3 identity matrix:

\begin{equation}\begin{bmatrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{bmatrix} \end{equation}

Multiplying any matrix by an identity matrix is the same as multiplying a number by 1; the result is the same as the original value:

\begin{equation}\begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\\7 & 8 & 9\end{bmatrix} \cdot \begin{bmatrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{bmatrix} = \begin{bmatrix}1 & 2 & 3 \\4 & 5 & 6\\7 & 8 & 9\end{bmatrix} \end{equation}

If you doubt me, try the following Python code!

```python
import numpy as np

A = np.array([[1,2,3],
              [4,5,6],
              [7,8,9]])
B = np.array([[1,0,0],
              [0,1,0],
              [0,0,1]])
print(A @ B)
```

## Matrix Division
You can't actually divide by a matrix; but when you want to divide matrices, you can take advantage of the fact that division by a given number is the same as multiplication by the reciprocal of that number. For example:

\begin{equation}6 \div 3 = \frac{1}{3}\times 6 \end{equation}

In this case, 1/3 is the reciprocal of 3 (which as a fraction is 3/1 - we "flip" the numerator and denominator to get the reciprocal). You can also write 1/3 as 3-1.

### Inverse of a Matrix
For matrix division, we use a related idea; we multiply by the *inverse* of a matrix:

\begin{equation}A \div B = A \cdot B^{-1}\end{equation}

The inverse of B is B-1 as long as the following equation is true:

\begin{equation}B \cdot B^{-1} = B^{-1} \cdot B = I\end{equation}

**I**, you may recall, is an *identity* matrix; the matrix equivalent of 1.

So how do you calculate the inverse of a matrix? For a 2x2 matrix, you can follow this formula:

\begin{equation}\begin{bmatrix}a & b\\c & d\end{bmatrix}^{-1} = \frac{1}{ad-bc}  \begin{bmatrix}d & -b\\-c & a\end{bmatrix}\end{equation}

What happened there?
- We swapped the positions of *a* and *d*
- We changed the signs of *b* and *c*
- We multiplied the resulting matrix by 1 over the *determinant* of the matrix (*ad-bc*)

Let's try with some actual numbers:

\begin{equation}\begin{bmatrix}6 & 2\\1 & 2\end{bmatrix}^{-1} = \frac{1}{(6\times2)-(2\times1)}  \begin{bmatrix}2 & -2\\-1 & 6\end{bmatrix}\end{equation}

So:

\begin{equation}\begin{bmatrix}6 & 2\\1 & 2\end{bmatrix}^{-1} = \frac{1}{10}  \begin{bmatrix}2 & -2\\-1 & 6\end{bmatrix}\end{equation}

Which gives us the result:

\begin{equation}\begin{bmatrix}6 & 2\\1 & 2\end{bmatrix}^{-1} = \begin{bmatrix}0.2 & -0.2\\-0.1 & 0.6\end{bmatrix}\end{equation}

To check this, we can multiply the original matrix by its inverse to see if we get an identity matrix. This makes sense if you think about it; in the same way that 3 x 1/3 = 1, a matrix multiplied by its inverse results in an identity matrix:

\begin{equation}\begin{bmatrix}6 & 2\\1 & 2\end{bmatrix} \cdot \begin{bmatrix}0.2 & -0.2\\-0.1 & 0.6\end{bmatrix} = \begin{bmatrix}(6\times0.2)+(2\times-0.1) & (6\times-0.2)+(2\times0.6)\\(1\times0.2)+(2\times-0.1) & (1\times-0.2)+(2\times0.6)\end{bmatrix} = \begin{bmatrix}1 & 0\\0 & 1\end{bmatrix}\end{equation}

Note that not every matrix has an inverse - for example, if the determinant works out to be 0, the inverse matrix is not defined.

In Python, you can use the *numpy.linalg.**inv*** function to get the inverse of a matrix in an *array* or *matrix* object:

```python
import numpy as np

B = np.array([[6,2],
              [1,2]])

print(np.linalg.inv(B))
```

Additionally, the *matrix* type has an ***I*** method that returns the inverse matrix:

```python
import numpy as np

B = np.matrix([[6,2],
              [1,2]])

print(B.I)
```

For larger matrices, the process to calculate the inverse is more complex. Let's explore an example based on the following matrix:

\begin{equation}\begin{bmatrix}4 & 2 & 2\\6 & 2 & 4\\2 & 2 & 8\end{bmatrix} \end{equation}

The process to find the inverse consists of the following steps:

1: Create a matrix of *minors* by calculating the *determinant* for each element in the matrix based on the elements that are not in the same row or column; like this:

\begin{equation}\begin{bmatrix}\color{blue}4 & \color{lightgray}2 & \color{lightgray}2\\\color{lightgray}6 & \color{red}2 & \color{red}4\\\color{lightgray}2 & \color{red}2 & \color{red}8\end{bmatrix}\;\;\;\;(2\times8) - (4\times2) = 8\;\;\;\;\begin{bmatrix}8 & \color{lightgray}? & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{lightgray}4 & \color{blue}2 & \color{lightgray}2\\\color{red}6 & \color{lightgray}2 & \color{red}4\\\color{red}2 & \color{lightgray}2 & \color{red}8\end{bmatrix}\;\;\;\;(6\times8) - (4\times2) = 40\;\;\;\;\begin{bmatrix}8 & 40 & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\end{bmatrix}\end{equation}

\begin{equation}\begin{bmatrix}\color{lightgray}4 & \color{lightgray}2 & \color{blue}2\\\color{red}6 & \color{red}2 & \color{lightgray}4\\\color{red}2 & \color{red}2 & \color{lightgray}8\end{bmatrix}\;\;\;\;(6\times2) - (2\times2) = 8\;\;\;\;\begin{bmatrix}8 & 40 & 8\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{lightgray}4 & \color{red}2 & \color{red}2\\\color{blue}6 & \color{lightgray}2 & \color{lightgray}4\\\color{lightgray}2 & \color{red}2 & \color{red}8\end{bmatrix}\;\;\;\;(2\times8) - (2\times2) = 12\;\;\;\;\begin{bmatrix}8 & 40 & 8\\12 & \color{lightgray}? & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{red}4 & \color{lightgray}2 & \color{red}2\\\color{lightgray}6 & \color{blue}2 & \color{lightgray}4\\\color{red}2 & \color{lightgray}2 & \color{red}8\end{bmatrix}\;\;\;\;(4\times8) - (2\times2) = 28\;\;\;\;\begin{bmatrix}8 & 40 & 8\\12 & 28 & \color{lightgray}?\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{red}4 & \color{red}2 & \color{lightgray}2\\\color{lightgray}6 & \color{lightgray}2 & \color{blue}4\\\color{red}2 & \color{red}2 & \color{lightgray}8\end{bmatrix}\;\;\;\;(4\times2) - (2\times2) = 4\;\;\;\;\begin{bmatrix}8 & 40 & 8\\12 & 28 & 4\\\color{lightgray}? & \color{lightgray}? & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{lightgray}4 & \color{red}2 & \color{red}2\\\color{lightgray}6 & \color{red}2 & \color{red}4\\\color{blue}2 & \color{lightgray}2 & \color{lightgray}8\end{bmatrix}\;\;\;\;(2\times4) - (2\times2) = 4\;\;\;\;\begin{bmatrix}8 & 40 & 8\\12 & 28 & 4\\4 & \color{lightgray}? & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{red}4 & \color{lightgray}2 & \color{red}2\\\color{red}6 & \color{lightgray}2 & \color{red}4\\\color{lightgray}2 & \color{blue}2 & \color{lightgray}8\end{bmatrix}\;\;\;\;(4\times4) - (2\times6) = 4\;\;\;\;\begin{bmatrix}8 & 40 & 8\\12 & 28 & 4\\4 & 4 & \color{lightgray}?\end{bmatrix} \end{equation}

\begin{equation}\begin{bmatrix}\color{red}4 & \color{red}2 & \color{lightgray}2\\\color{red}6 & \color{red}2 & \color{lightgray}4\\\color{lightgray}2 & \color{lightgray}2 & \color{blue}8\end{bmatrix}\;\;\;\;(4\times2) - (2\times6) = -4\;\;\;\;\begin{bmatrix}8 & 40 & 8\\12 & 28 & 4\\4 & 4 & -4\end{bmatrix} \end{equation}

2: Apply *cofactors* to the matrix by switching the sign of every alternate element in the matrix of minors:

\begin{equation}\begin{bmatrix}8 & -40 & 8\\-12 & 28 & -4\\4 & -4 & -4\end{bmatrix} \end{equation}

3: *Adjugate* by transposing elements diagonally:

\begin{equation}\begin{bmatrix}8 & \color{green}-\color{green}1\color{green}2 & \color{orange}4\\\color{green}-\color{green}4\color{green}0 & 28 & \color{purple}-\color{purple}4\\\color{orange}8 & \color{purple}-\color{purple}4 & -4\end{bmatrix} \end{equation}

4: Multiply by 1/determinant of the original matrix. To find this, multiply each of the top row elements by their corresponding minor determinants (which we calculated earlier in the matrix of minors), and then subtract the second from the first and add the third:

\begin{equation}Determinant = (4 \times 8) - (2 \times 40) + (2 \times 8) = -32\end{equation}

\begin{equation}\frac{1}{-32}\begin{bmatrix}8 & -12 & 4\\-40 & 28 & -4\\8 & -4 & -4\end{bmatrix} =  \begin{bmatrix}-0.25 & 0.375 & -0.125\\1.25 & -0.875 & 0.125\\-0.25 & 0.125 & 0.125\end{bmatrix}\end{equation}

Let's verify that the original matrix multiplied by the inverse results in an identity matrix:

\begin{equation}\begin{bmatrix}4 & 2 & 2\\6 & 2 & 4\\2 & 2 & 8\end{bmatrix} \cdot \begin{bmatrix}-0.25 & 0.375 & -0.125\\1.25 & -0.875 & 0.125\\-0.25 & 0.125 & 0.125\end{bmatrix}\end{equation}

\begin{equation}= \begin{bmatrix}(4\times-0.25)+(2\times1.25)+(2\times-0.25) & (4\times0.375)+(2\times-0.875)+(2\times0.125) & (4\times-0.125)+(2\times-0.125)+(2\times0.125)\\(6\times-0.25)+(2\times1.25)+(4\times-0.25) & (6\times0.375)+(2\times-0.875)+(4\times0.125) & (6\times-0.125)+(2\times-0.125)+(4\times0.125)\\(2\times-0.25)+(2\times1.25)+(8\times-0.25) & (2\times0.375)+(2\times-0.875)+(8\times0.125) & (2\times-0.125)+(2\times-0.125)+(8\times0.125)\end{bmatrix} \end{equation}

\begin{equation}= \begin{bmatrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{bmatrix} \end{equation}

As you can see, this can get pretty complicated - which is why we usually use a calculator or a computer program. You can run the following Python code to verify that the inverse matrix we calculated is correct:

```python
import numpy as np

B = np.array([[4,2,2],
              [6,2,4],
              [2,2,8]])

print(np.linalg.inv(B))
```

### Multiplying by an Inverse Matrix
Now that you know how to calculate an inverse matrix, you can use that knowledge to multiply the inverse of a matrix by another matrix as an alternative to division:

\begin{equation}\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix} \cdot \begin{bmatrix}6 & 2\\1 & 2\end{bmatrix}^{-1} \end{equation}

\begin{equation}=\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix} \cdot \begin{bmatrix}0.2 & -0.2\\-0.1 & 0.6\end{bmatrix}  \end{equation}

\begin{equation}=\begin{bmatrix}(1\times0.2)+(2\times-0.1) & (1\times-0.2)+(2\times0.6)\\(3\times0.2)+(4\times-0.1) & (3\times-0.2)+(4\times0.6)\end{bmatrix}\end{equation}

\begin{equation}=\begin{bmatrix}0 & 1\\0.2 & 1.8\end{bmatrix}\end{equation}

Here's the Python code to calculate this:

```python
import numpy as np

A = np.array([[1,2],
              [3,4]])

B = np.array([[6,2],
              [1,2]])

C = A @ np.linalg.inv(B)

print(C)
```

## Solving Systems of Equations with Matrices
One of the great things about matrices, is that they can help us solve systems of equations. For example, consider the following system of equations:

\begin{equation}2x + 4y = 18\end{equation}
\begin{equation}6x + 2y = 34\end{equation}

We can write this in matrix form, like this:

\begin{equation}\begin{bmatrix}2 & 4\\6 & 2\end{bmatrix} \cdot \begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}18\\34\end{bmatrix}\end{equation}

Note that the variables (***x*** and ***y***) are  arranged as a column in one matrix, which is multiplied by a matrix containing the coefficients to produce as matrix containing the results. If you calculate the dot product on the left side, you can see clearly that this represents the original equations:

\begin{equation}\begin{bmatrix}2x + 4y\\6x + 2y\end{bmatrix} =\begin{bmatrix}18\\34\end{bmatrix}\end{equation}

Now. let's name our matrices so we can better understand what comes next:

\begin{equation}A=\begin{bmatrix}2 & 4\\6 & 2\end{bmatrix}\;\;\;\;X=\begin{bmatrix}x\\y\end{bmatrix}\;\;\;\;B=\begin{bmatrix}18\\34\end{bmatrix}\end{equation}

We already know that ***A • X = B***, which arithmetically means that ***X = B ÷ A***. Since we can't actually divide by a matrix, we need to multiply by the inverse; so we can find the values for our variables (*X*) like this: ***X = A-1 • B***

So, first we need the inverse of A:

\begin{equation}\begin{bmatrix}2 & 4\\6 & 2\end{bmatrix}^{-1} = \frac{1}{(2\times2)-(4\times6)}  \begin{bmatrix}2 & -4\\-6 & 2\end{bmatrix}\end{equation}

\begin{equation}= \frac{1}{-20}  \begin{bmatrix}2 & -4\\-6 & 2\end{bmatrix}\end{equation}

\begin{equation}=\begin{bmatrix}-0.1 & 0.2\\0.3 & -0.1\end{bmatrix}\end{equation}

Then we just multiply this with B:

\begin{equation}X = \begin{bmatrix}-0.1 & 0.2\\0.3 & -0.1\end{bmatrix} \cdot \begin{bmatrix}18\\34\end{bmatrix}\end{equation}

\begin{equation}X = \begin{bmatrix}(-0.1 \times 18)+(0.2 \times 34)\\(0.3\times18)+(-0.1\times34)\end{bmatrix}\end{equation}

\begin{equation}X = \begin{bmatrix}5\\2\end{bmatrix}\end{equation}

The resulting matrix (*X*) contains the values for our *x* and *y* variables, and we can check these by plugging them into the original equations:

\begin{equation}(2\times5) + (4\times2) = 18\end{equation}
\begin{equation}(6\times5) + (2\times2) = 34\end{equation}

These of course simplify to:

\begin{equation}10 + 8 = 18\end{equation}
\begin{equation}30 + 4 = 34\end{equation}

So our variable values are correct.

Here's the Python code to do all of this:

```python
import numpy as np

A = np.array([[2,4],
              [6,2]])

B = np.array([[18],
              [34]])

C = np.linalg.inv(A) @ B

print(C)
```

```python

```

Vector and Matrices by Hiren/03-04-More Matrices.ipynb


### Agnostic smooth minimization

**Purpose of this demo**: Motivate optimization in general (hyper-parameter selection, non-convexity)

+ Disclaimer: I'm not expert in Python - I use Python/Matlab as tools to validate algorithms and theorems.
+ Thus, my implementations are not the most efficient ones + there might be bugs

**Problem definition:**.

\begin{align}
  f(x_1, x_2) = (x_1 + 2x_2 - 7)^2 + (2x_1 + x_2 - 5)^2
\end{align}

\begin{equation*}
\begin{aligned}
& \underset{x \in \mathbb{R}^2}{\text{min}}
& & f(x_1, x_2)
\end{aligned}
\end{equation*}

+ Any properties you might extract by just looking at the function?

+ Is it differentiable?

\begin{align}
  \frac{\partial f(x_1, x_2)}{\partial x_1} = 2(x_1 + 2x_2 - 7) + 4(2x_1 + x_2 - 5)
\end{align}

\begin{align}
  \frac{\partial f(x_1, x_2)}{\partial x_2} = 4(x_1 + 2x_2 - 7) + 2(2x_1 + x_2 - 5)
\end{align}

and as a vector:
\begin{align}
    \nabla f(x_1, x_2) = \begin{bmatrix} 2(x_1 + 2x_2 - 7) + 4(2x_1 + x_2 - 5) \\ 4(x_1 + 2x_2 - 7) + 2(2x_1 + x_2 - 5) \end{bmatrix}
\end{align}

+ Is it negative-valued, positive-valued, or both?

**3D plot**

```python
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
import random

from matplotlib import rc
#rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
## for Palatino and other serif fonts use:
rc('font',**{'family':'serif','serif':['Palatino']})
rc('text', usetex=True)

from numpy import linalg as la

from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

# Make data.
X = np.arange(-10, 10, 0.25)
Y = np.arange(-10, 10, 0.25)
X, Y = np.meshgrid(X, Y)
Z = (X + 2*Y - 7)**2 + (2*X + Y - 5)**2

ax.plot_wireframe(X, Y, Z, rstride=4, cstride=4)
ax.view_init(10, 90)
```

```python
# Returns the value of the objecive function
def f(x):
    return (x[0] + 2*x[1] - 7)**2 + (2*x[0] + x[1] - 5)**2
```

```python
def GD_Booth(x_new, eta, iters, epsilon, verbose, x_star):
    p = 2

    x_list, f_list = [la.norm(x_new - x_star, 2)], [f(x_new)]

    for i in range(iters):
        x_old = x_new

        # Compute gradient
        grad = np.zeros(p)
        grad[0] = 2*(x_old[0] + 2*x_old[1] - 7) + 4*(2*x_old[0] + x_old[1] - 5)
        grad[1] = 4*(x_old[0] + 2*x_old[1] - 7) + 2*(2*x_old[0] + x_old[1] - 5)

        # Perform gradient step
        x_new = x_old - eta * grad

        if (la.norm(x_new - x_old, 2) / la.norm(x_new, 2)) < epsilon:
            break

        # Keep track of solutions and objective values
        x_list.append(la.norm(x_new - x_star, 2))
        f_list.append(f(x_new))

        if verbose:
            print("iter# = "+ str(i) + ", ||x_new - x_star||_2 = " + str(la.norm(x_new - x_star, 2)) + ", f(x_new) = " + str(f(x_new)))

    print("Number of steps:", len(f_list))
    return x_new, x_list, f_list
```

```python
# Run algorithm
epsilon = 1e-6                # Precision parameter
iters = 100
eta = 0.1
x_init = np.random.randn(2)    # Initial estimate
# print(x_init)
#x_init[0] = 1
#x_init[1] = 1
print(x_init)
x_star = [1, 3]

x_GD, x_list, f_list = GD_Booth(x_init, eta, iters, epsilon, True, x_star)

# Plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

xs = range(len(x_list))
plt.plot(xs, x_list, '-o', color = '#3399FF', linewidth = 4, alpha = 0.7, markerfacecolor = 'b')
plt.yscale('log')
plt.xlabel('Iterations')
plt.ylabel(r"$\|x^\star - \widehat{x}\|_2$")

# Make room for the ridiculously large title.
plt.subplots_adjust(top=0.8)
plt.show()
```

**Problem definition: Weirder 2d non-convex problem**.

\begin{align}
  f(x_1, x_2) = \sum_{i = 1}^2 \tfrac{x_i^2}{4000} - \prod_{i = 1}^2\cos\left( \tfrac{x_i}{\sqrt{i}} \right) + 1
\end{align}

\begin{equation*}
\begin{aligned}
& \underset{x \in \mathbb{R}^2}{\text{min}}
& & f(x_1, x_2)
\end{aligned}
\end{equation*}

+ Any properties you might extract by just looking at the function?

+ Is it differentiable?

\begin{align}
  \frac{\partial f(x_1, x_2)}{\partial x_1} = \tfrac{x_1}{2000} + \sin\left(x_1\right) \cdot \cos\left(\tfrac{x_2}{\sqrt{2}}\right)
\end{align}

\begin{align}
  \frac{\partial f(x_1, x_2)}{\partial x_2} = \tfrac{x_2}{2000} + \cos\left(x_1\right) \cdot \frac{\sin\left(\tfrac{x_2}{\sqrt{2}}\right)}{\sqrt{2}}
\end{align}

and as a vector:
\begin{align}
    \nabla f(x_1, x_2) = \begin{bmatrix} \tfrac{x_1}{2000} + \sin\left(x_1\right) \cdot \cos\left(\tfrac{x_2}{\sqrt{2}}\right) \\
    \tfrac{x_2}{2000} + \cos\left(x_1\right) \cdot \frac{\sin\left(\tfrac{x_2}{\sqrt{2}}\right)}{\sqrt{2}} \end{bmatrix}
\end{align}

**3D plot**

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

# Make data.
X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
Z = (X**2)/4000 + (Y**2)/4000 - np.cos(X)*np.cos(Y/np.sqrt(2)) + 1

surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
```

```python
# Returns the value of the objecive function
def f(x):
    return (x[0]**2)/4000 + (x[1]**2)/4000 - np.cos(x[0])*np.cos(x[1]/np.sqrt(2)) + 1
```

```python
def GD_Griewank(x_new, eta, iters, epsilon, verbose, x_star):
    p = 2

    x_list, f_list = [la.norm(x_new - x_star, 2)], [f(x_new)]

    for i in range(iters):
        x_old = x_new

        # Compute gradient
        grad = np.zeros(p)
        grad[0] = x_old[0]/2000 + np.sin(x_old[0]) * np.cos(x_old[1]/np.sqrt(2))
        grad[1] = x_old[1]/2000 + np.cos(x_old[0]) * np.sin(x_old[1]/np.sqrt(2))/np.sqrt(2)

        # Perform gradient step
        x_new = x_old - eta * grad

        if (la.norm(x_new - x_old, 2) / la.norm(x_new, 2)) < epsilon:
            break

        # Keep track of solutions and objective values
        x_list.append(la.norm(x_new - x_star, 2))
        f_list.append(f(x_new))

        if verbose:
            print("iter# = "+ str(i) + ", ||x_new - x_star||_2 = " + str(la.norm(x_new - x_star, 2)) + ", f(x_new) = " + str(f(x_new)))

    print("Number of steps:", len(f_list))
    return x_new, x_list, f_list
```

```python
# Run algorithm
epsilon = 1e-6                # Precision parameter
iters = 1000
eta = 0.1
# x_init = np.random.randn(2)    # Initial estimate
# print(x_init)
x_init[0] = -3
x_init[1] = 3
# print(x_init)
x_star = [0, 0]

x_GD, x_list, f_list = GD_Griewank(x_init, eta, iters, epsilon, True, x_star)

# Plot
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

xs = range(len(x_list))
plt.plot(xs, x_list, '-o', color = '#3399FF', linewidth = 4, alpha = 0.7, markerfacecolor = 'b')
plt.yscale('log')
plt.xlabel('Iterations')
plt.ylabel(r"$\|x^\star - \widehat{x}\|_2$")

# Make room for the ridiculously large title.
plt.subplots_adjust(top=0.8)
plt.show()
```

### Non-convex Lipschitz continuous gradient function

\begin{align}
  f(x) = x^2 + 3\sin^2(x)
\end{align}

Then, its gradient and Hessian are calculated as:
\begin{align}
f'(x) = 2x + 6\sin(x) \cdot \cos(x), ~~f''(x) = 2 + 6\cos^2(x) - 6\sin^2(x)
\end{align}

```python
fig = plt.figure()
ax = fig.add_subplot(111)

# Make data.
x = np.arange(-5, 5, 0.25)
f = x**2 + 3*(np.sin(x))**2

ax.plot(x, f, color='green', marker='o', linestyle='dashed', linewidth=2, markersize=2)
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)

# Make data.
x = np.arange(-5, 5, 0.25)
hessian_f = 2 + 6*(np.cos(x))**2 - 6*(np.sin(x))**2

ax.plot(x, hessian_f, color='green', marker='o', linestyle='dashed', linewidth=2, markersize=2)
```

Which means that we can upper bound:
\begin{align}
\|\nabla^2 f(x)\|_2 \leq 8 := L
\end{align}