# Linear Production Model

This notebook demonstrates the use of linear programming to maximize profit for a simple model of a multiproduct production facility. The notebook uses [Pyomo](http://www.pyomo.org/) to represent the model with the [glpk](https://en.wikibooks.org/wiki/GLPK) solver to calculate solutions.

## Example: Production Plan for a Single Product Plant

Suppose you are thinking about starting up a business to produce Product X. You have determined there is a market for X of up to 40 units per week at a price of \$270 each. The production of each unit requires \$100 of raw materials, 1 hour of type A labor, and 2 hours of type B labor. You have an unlimited amount of raw material available to you, but only 80 hours per week of labor A at a cost of \$50/hour, and 100 hours per week of labor B at a cost of \$40 per hour. Ignoring all other expenses, what is the maximum weekly profit?

To get started on this problem, we sketch a flow diagram illustrating the flow of raw materials and labor through the production plant.

The essential decision we need to make is how many units or Product X to produce each week. That's our *decision variable* which we denote as $x$. The weekly revenues are then

$$ \mbox{Revenue} = \$270 x $$

The costs include the value of the raw materials and each form of labor. If we produce x units a week, then the total cost is

$$ \mbox{Cost} = \underbrace{\$100 x}_{\mbox{Raw Material}}
    + \underbrace{\$50 x}_{\mbox{Labor A}} + \underbrace{2\times\$40 x}_{\mbox{Labor B}} = \$230 x$$

We see immediately that the gross profit is just

$$\begin{eqnarray*}\mbox{Profit} & = & \mbox{Revenue} - \mbox{Cost} \\
& = & \$270x - \$230x \\
& = & \$40 x
\end{eqnarray*}$$

which means there is a profit earned on each unit of X produced, so let's produce as many as possible.

There are three constraints that limit how many units can be produced. There is market demand for no more than 40 units per week. Producing $x = 40$ units per week will require 40 hours per week of Labor A, and 80 hours per week of Labor B. Checking those constraints we see that we have enough labor of each type, so the maximum profit will be

$$\max \mbox{Profit} = $40 \mbox{ per unit} \times 40 \mbox{ units per week} = \$1600 \mbox{ per week}$$

What we conclude is that market demand is the 'most constraining constraint.' Once we've made that deduction, the rest is a straightforward problem that can be solved by inspection.

### Mathematical Model

Even though this example has a straightforward solution, it is useful to consider how it can be represented mathematically, and solved using typical tools for linear programming.

The mathematical representation consists of a single non-negative decision variable, $x$, an objective function, and a set of linear constraints.  Here we include all constraints even though we know only one of them -- the most 'constraining constraint' -- will be active.

\begin{align}
\max_{x \geq 0} &\ 40\ x  & \mbox{objective}\\
\mbox{subject to:}\qquad \\
x & \leq 40  & \mbox{demand constraint} \\
x & \leq 80  & \mbox{labor A constraint} \\
2\ x & \leq 100 & \mbox{labor B constraint}
\end{align}

All of these constraints must be satisfied, therefore the demand constraint is the 'most constrainting'. Again, the maximum value of $x$ is 40, so the maximum profit is $\$ 40 \times 40 = \$1,600$.

### Exercises

1. Suppose the demand could be increased to 50 units per month. What would be the increased profits?  What if the demand increased to 60 units per month?  How much would you be willing to pay for your marketing department for the increased demand?

2. Increase the cost of LaborB. At what point is it no longer financially viable to run the plant?

## Production Plan: Product Y

Your marketing department has developed plans for a new product called Y. The product sells at a price of \\$210 each, and they expect that you can sell all that you can make. It's also cheaper to make, requiring only \\$90 in raw materials, 1 hour of Labor type A at \\$50 per hour, and 1 hour of Labor B at \\$40 per hour. What is the potential weekly profit?

The analysis proceeds in the same form as Product X. In this case the revenue is given by

$$ \mbox{Revenue} = \$210 y $$

The total cost is

$$ \mbox{Cost} = \underbrace{\$90 x}_{\mbox{Raw Material}}
    + \underbrace{\$50 x}_{\mbox{Labor A}} + \underbrace{\$40 x}_{\mbox{Labor B}} = \$180 x$$

The gross profit is thn

\begin{eqnarray}\mbox{Profit} & = & \mbox{Revenue} - \mbox{Cost} \\
& = & \$210x - \$180x \\
& = & \$30 x
\end{eqnarray}

We see the profit per unit of Y is smaller. So a decision to produce Y instead of X must be based on the ability to make Y in larger quantities.

The mathematical formulation of this problem becomes

\begin{align}
\max_{y \geq 0} &\ 30\ y  & \mbox{objective}\\
\mbox{subject to:}\qquad \\
y & \leq 80  & \mbox{labor A constraint} \\
y & \leq 100 & \mbox{labor B constraint}
\end{align}

Compared to product X, we can manufacture and sell up 80 units per week for a total profit of \$2,400. This is very welcome news.

### Exercises

1. What is the limiting resource?  That is, which of the two types of labor limits the capacity of your plant to produce more units of Y?

2. What rate would you be willing to pay for the additional labor necessary to increase the production of Y?

## Production Plan: Mixed Product Strategy

So far we have learned that we can make \\$1,600 per week by manufacturing product X, and \\$2,400 per week manufacturing product Y.  Is it possible to do even better?

To answer this question, we consider the possibilty of manufacturing both products in the same plant. The marketing department assures us that product Y will not affect the sales of product X.  So the same constraints hold as before, but now we have two decision variables, $x$ and $y$.

Mathematical formulation

\begin{align}
\max_{x,y \geq 0} &\ 40\ x + 30\ y  & \mbox{objective}\\
\mbox{subject to:}\qquad \\
x & \leq 40  & \mbox{demand constraint} \\
x + y & \leq 80  & \mbox{labor A constraint} \\
2x + y & \leq 100 & \mbox{labor B constraint}
\end{align}

## Solution using scipy.optimize.linprog

The [scipy.optimize library](https://docs.scipy.org/doc/scipy/reference/optimize.html) includes a comprehensive collection of functions for finding roots and minimizing systems of one or more equations. Given a new problem, this library is often a good place to look for tools needed to solve a system of equations, or to minimize a particular objective.

In particular, the function `scipy.optimize.linprog` solves linear programming problems represented in the form

\begin{align}
\min_{x \geq 0} &\ c^T x \\
\mbox{subject to:}\qquad \\
A_{ub} & \leq b_{ub} \\
A_{eq} & = b_{eq} \\
\end{align}

where $c$ is a vector of coefficients in the objective function to be minimized, $A_{ub}$ and $b_{ub}$ are the coefficients and right-hand-sides of problem constraints written as upper bounds, and $A_{eq}$ and $b_{eq}$ are coefficients and right-hand-sides for equality constraints.

```python
from scipy.optimize import linprog

c = [-40, -30]

A_ub = [[1, 0],
        [1, 1],
        [2, 1]]

b_ub = [40,
        80,
        100]

results = linprog(c, A_ub, b_ub)

print(results.message)
if results.success:
    print('x =', results.x)
    print('objective = ', results.fun)
```

    Optimization terminated successfully.
    x = [19.99999999 59.99999998]
    objective =  -2599.999998967465

The mixed product strategy earns more profit than either of the single product srategies. Does this surprise you?  Before going further, try to explain why it is possible for a mixed product strategy to earn more profit than either of the possible single product strategies.

## What are the active constraints?

```python
%matplotlib inline
from pylab import *

figure(figsize=(6,6))
subplot(111, aspect='equal')
axis([0,100,0,100])
xlabel('Production Qty X')
ylabel('Production Qty Y')

# Labor A constraint
x = array([0,80])
y = 80 - x
plot(x,y,'r',lw=2)
fill_between([0,80,100],[80,0,0],[100,100,100],color='r',alpha=0.15)

# Labor B constraint
x = array([0,50])
y = 100 - 2*x
plot(x,y,'b',lw=2)
fill_between([0,50,100],[100,0,0],[100,100,100],color='b',alpha=0.15)

# Demand constraint
plot([40,40],[0,100],'g',lw=2)
fill_between([40,100],[0,0],[100,100],color='g',alpha=0.15)

legend(['Labor A Constraint','Labor B Constraint','Demand Constraint'])

# Contours of constant profit
x = array([0,100])
for p in linspace(0,3600,10):
    y = (p - 40*x)/30
    plot(x,y,'y--')

# Optimum
plot(20,60,'r.',ms=20)
annotate('Mixed Product Strategy', xy=(20,60), xytext=(50,70),
         arrowprops=dict(shrink=.1,width=1,headwidth=5))

plot(0,80,'b.',ms=20)
annotate('Y Only', xy=(0,80), xytext=(20,90),
         arrowprops=dict(shrink=0.1,width=1,headwidth=5))

plot(40,0,'b.',ms=20)
annotate('X Only', xy=(40,0), xytext=(70,20),
         arrowprops=dict(shrink=0.1,width=1,headwidth=5))

text(4,23,'Increasing Profit')
annotate('', xy=(20,15), xytext=(0,0),
         arrowprops=dict(width=0.5,headwidth=5))

savefig('figures/LPprob01.png', bbox_inches='tight')
```

## What is the incremental value of labor?

The optimal solution is found at the intersection of constraints corresponding to Labor A and Labor B.  These are the **active constraints**. Adding additional labor of either type will increase the profit.  By contrast, the demand constraint for product X is inactive at the optimum, therefore attempting to increase demand for product X would be wasted effort.

This leaves the important question of how much the profit will increase if additional resources are available. Let's see what happens if one additional hour of Labor A is available.

```python
from scipy.optimize import linprog

c = [-40, -30]

A_ub = [[1, 0],
        [1, 1],
        [2, 1]]

b_ub = [40,
        81,    #  <<<<<< 1 additional hour of Labor A
        100]

results = linprog(c, A_ub, b_ub)

print(results.message)
if results.success:
    print('x =', results.x)
    print('objective = ', results.fun)
```

    Optimization terminated successfully.
    x = [18.99999999 61.99999997]
    objective =  -2619.999998864241

An additional hour of Labor A increases the profit by \$20. The base cost of \$50/hour for Labor A is included in the objective function. Therefore we should be willing to pay up to \$50 + \$20 = \$70/hour for additional Labor A.

### Exercise

1. What is the incremental value of Labor B?

## Theory of Constraints

* For $n$ decisions you should expect to find $n$ 'active' constraints.
* Each inactive constraint has an associated 'slack.'  The associated resources have no incremental value.
* Each active constraint has an associated 'shadow price'.  This is additional value of additional resources.

```python

```

```python

```


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
x = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
```

```python

x = x.replace(" ", "")
x = x.capitalize()

print(x)
```

    Diclorodifeniltricloroetanodichlorodiphenyltrichloroethane

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
A, B, C, D = 'word', 15, 3.14, 'list'
print(A, B, C, D)
```

    word 15 3.14 list

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
w,h = (input("what is your weight and hight? ").split())
w= float(w)
h=float(h)
bmi = w/(h**2)
print(round(bmi,5))

```

    what is your weight and hight? 100 1.70
    34.60208

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<center><\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
if bmi <= 18.5:
    print("underweight")
elif bmi <= 24.9:
    print("normal weight")
elif bmi <= 29.9:
    print("pre-obesity")
elif bmi <= 34.9:
    print("Obesity class I")
elif bmi <= 39.9:
    print("Obesity class II")
else:
    print("Obesity class III")
```

    Obesity class I

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
def factorial(number):
    n = 1
    for i in range(number):
        n *=i+1
    return n
print(factorial(int(input("write the number you want to get det factorial of "))))
```

    write the number you want to get det factorial of 5
    120

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
def sum_of_numbers():

    while True:
        number = int(input("Enter a number: "))
        if number == -1:
            break
        number += number
    print("Sum of numbers: ", number+1)

sum_of_numbers()
```

    Enter a number: 2
    Enter a number: -2
    Enter a number: -1
    Sum of numbers:  0

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python
name, hours, salary_per_hour = input("Insert name, amount of hours worked and salary per hour, separated by commas: ").split(",")
total_salary = float(hours)*float(salary_per_hour)
print(total_salary)

```

    Insert name, amount of hours worked and salary per hour, separated by commas: aleks,12,45
    540.0

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate itens a, b, c, d and e:

```python
A, B, C = input("input a three numbers ").split()
A = float(A)
B = float(B)
C = float(C)
pi = 3.14159
```

    input a three numbers 2 3 4

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
triangle = (A*C)/2
print('Triangle: ', triangle )
```

    Triangle:  10.35

    b) the area of the circle of radius C. (pi = 3.14159)

```python
circle = pi*(C**2)
print('circle: ', circle)
```

    circle:  66.47604439999999

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
trapezoid = (A+B)*C/2
print('trapezoid: ', trapezoid)
```

    trapezoid:  10.0

    d) the area of the square that has side B.

```python
square = B**2
print('Squere: ', square)
```

    Squere:  44.89

    e) the area of the rectangle that has sides A and B.

```python
rectangle = A*B
print('rectangle: ', rectangle)
```

    rectangle:  6.0

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python
a = int(input("input a numbaer for a "))
while a == 0:
    print('a can\'t be zero')
    a = int(input("input a numbaer for a "))

b, c =[float(i) for i in input('input a number for b and c').split(',')]

d = b**2 - 4*a*c

if d < 0:
    print("No real roots")
else:
    print("x1 =", (-b + d**0.5)/(2*a), ", x2 =", (-b - d**0.5)/(2*a))
```

    input a numbaer for a 3
    input a number for b and c3,4
    No real roots

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
x1, y1, x2, y2 = [float(i) for i in input("insert coordinates x1, y1, x2, y2 ").split()]

distance = ((x2 - x1)**2 + (y2 - y1)**2)**0.5

print('distance =', distance)
```

    insert coordinates x1, y1, x2, y2 1 2 4 5
    distance = 4.242640687119285

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python
x, y = [float(x) for x in input("type in numbers for x and y ").split()]

if x == 0 and y == 0:
    print('origin')
elif x == 0:
    print('y axis')
elif y == 0:
	print('x axis')
elif x > 0 and y > 0:
    print('q1')
elif x < 0 and y > 0:
    print('q2')
elif x < 0 and y < 0:
    print('q3')
else:
    print('q4')
```

    type in numbers for x and y 3 4
    q1

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd

df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
df = df.iloc[:,:2]
df.head(20)

def country(countrycode):
    for i in range(len(df.iloc[:])):
        if str(countrycode) == df.iloc[i,1]:
            return(df.iloc[i,0])
    else:
        print("this is not a landcode")
print(country(43))
```

    Austria

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
number = [float(x) for x in input('type in six numbers ').split(',')]

sum_, positive=0, 0

for i in number:
    sum_ += i
    if i >0:
        positive += 1
print('The number of positive values is: ', positive, '\n', 'The avrage value is:', sum_/6)

```

    type in six numbers 1,2,3,4,5,6
    The number of positive values is:  6
     The avrage value is: 3.5

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python
a = int(input("input a number"))
for i in range(1,a+1):
    if i%2 ==0:
        print(i**2)
```

    input a number8
    4
    16
    36
    64

#### 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python
number = int(input("enter a number "))
even = ""
positive = ""

if number == 0:
    print("null")

elif number%2==0:
    even = 'even'
else:
    even = 'odd'

if number < 0:
    positive = 'negative'
else:
    posisitve = 'positive'

print(positive, even)

```

    enter a number -9
    negative odd

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python
number = [int(x) for x in input('Input six numbers: ').split(',')]
for i in range(len(number) - 1):
        for j in range(len(number) - 1 - i):
            if number[j] > number[j + 1]:
                number[j], number[j + 1] = number[j + 1], number[j]
print(numbers)
```

    Input six numbers: 1,3,4,2,1,5,4,32,6
    [1, 1, 2, 3, 4, 4, 5, 6, 32]

```python

```


# Basic Neural Network from Scratch

>## Objective:

To understand and build a basic one hidden layer Neutral Network from scratch

> ## Approach :

- **Getting the Dataset**
- **Logistic Regression**
- **Neural Network : Understanding**

       - Neural Network structure
       - Activation functions : Softmax and Tanh and their necessity
       - Weight factors and Bias
       - Forward Propagation
       - Back Propagation
       - Shape of Vectors
       - Differentiation

- **Implementation**

       - Model Function
       - Predict Function
       - Loss Function

- **Varying hidden layer nodes**
- **Conclusion**

**Importing Libraries**

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.linear_model
import matplotlib
%matplotlib inline
matplotlib.rcParams['figure.figsize'] = (10.0, 8.0)
```

**`Plot_decision_boundary` function is defined to conveniently see the decision boundary**

```python
# Helper function to plot a decision boundary.
# If you don't fully understand this function don't worry, it just generates the contour plot below.
def plot_decision_boundary(pred_func):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)
```

## `1.` Getting the Dataset

Dataset is generated using sklearn's built in methods

```python
# Generate a dataset and plot it
import sklearn
from sklearn import datasets
np.random.seed(0)
X, y = sklearn.datasets.make_moons(200, noise=0.20)
plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.Spectral)
```

The dataset generated has two classes, plotted as red and blue points. You can think of the blue dots as male patients and the red dots as female patients, with the x- and y- axis being medical measurements.

Your goal is to train a Machine Learning classifier that predicts the correct class (male or female) given the x- and y- coordinates. Note that the data is not linearly separable, a straight line cant be drawn that separates the two classes. This means that linear classifiers, such as Logistic Regression, won't be able to fit the data unless you hand-engineer non-linear features (such as polynomials) that work well for the given dataset.

In fact, that's one of the major advantages of Neural Networks. You don't need to worry about feature engineering. The hidden layer of a neural network will learn features for you.

## `2.` Logistic Regression:

To understand the problem better , try to implement a logistic regression first on the dataset and plot the decision boundary using the earlier defined function

**TASK : Build a `logistic regression` model and fit it on the dataset**

```python
### START CODE HERE (~ 2 Lines of code)

### END CODE
```

**TASK : Plot the `Decision boundary` using the earlier defined function**

```python
# For the argument of lambda x: use the predict function of the earlier defined classifier that was fit on dataset

### START CODE HERE (~2 Lines of code)
#(Write code where '#' is given)

plot_decision_boundary(lambda x: '# ')
plt.title("Logistic Regression")

### END CODE
```

The graph shows the decision boundary learned by our Logistic Regression classifier. It separates the data as good as it can using a straight line, but it's unable to capture the "moon shape" of our data.

## `3.`  Neural Networks :

### `3.1` Neural Networks Understanding

Let's now build a 3-layer neural network with one input layer, one hidden layer, and one output layer. The number of nodes in the input layer is determined by the dimensionality of our data, 2. Similarly, the number of nodes in the output layer is determined by the number of classes we have, also 2. (Because we only have 2 classes we could actually get away with only one output node predicting 0 or 1, but having 2 makes it easier to extend the network to more classes later on). The input to the network will be x- and y- coordinates and its output will be two probabilities, one for class 0 ("female") and one for class 1 ("male"). It looks something like this:

You can choose the dimensionality (the number of nodes) of the hidden layer. The more nodes we put into the hidden layer the more complex functions we will be able fit. But higher dimensionality comes at a cost. First, more computation is required to make predictions and learn the network parameters. A bigger number of parameters also means that the model become more prone to overfitting the data.

How to choose the size of the hidden layer? While there are some general guidelines and recommendations, it always depends on your specific problem and is more of an art than a science. Its best to play with the number of nodes in the hidden layer and see how it affects the output.

You also need to pick an activation function for the hidden layer. The activation function transforms the inputs of the layer into its outputs.  Common chocies for activation functions are tanh, the sigmoid function, or ReLUs. Here you'll use tanh, which performs quite well in many scenarios. A nice property of these functions is that their derivate can be computed using the original function value. For example, the derivative of $\tanh x$ is $1-\tanh^2 x$. This is useful because it allows to compute $\tanh x$ once and re-use its value later on to get the derivative.

The network is desired to output probabilities thus the activation function for the output layer will be the softmax, which is simply a way to convert raw scores to probabilities. If you're familiar with the logistic function you can think of softmax as its generalization to multiple classes.

### `3.2` Activation Functions

**Softmax :**

Where the function is defined as :

**Tanh :**

**Why are Activation Functions necessary?**

If there were no activation function or just linear activation function then it would work same as a linear classifier/regressor and the models applications would be very limited and thus a nonlinear activation function is what really allows to fit nonlinear hypotheses.

**Understanding the network**

The input $x$ contains two attributes namely $x1$ and $x2$ and has $m$ number of training examples , in this dataset $m$ = 200.
Neural networks are similar to logistic regression just that the latter is used for binary classification/regression and the former is used for multiclass classification/regression. Think of it this way that In logistic regression , attributes or inputs are given to a box or a node which multiplies the inputs with weight factors , adds bias and outputs the resultant similar to linear regression which is then transformed by a non linear activtion function , sigmoid in logistic regression to give the final output. Similarly , In neural networks this whole process happens alot of times , in this notebook with just one hidden layer the whole process occurs as many times as the number of nodes in the hidden layer , so each node's input-output can be visualised as logistic regression and many such singluar logistic regression units give birth to neural networks which shows generally why neural networks perform better than its singular counterpart. This whole process can be explained better with the following image :

### `3.3` Weight factors and Bias

**Bias** : The term bias is used to adjust the final output matrix as the y-intercept does. For instance, in the classic equation, $y = mx + c$, if $c$ = 0, then the line will always pass through 0. Adding the bias term provides more flexibility and better generalisation to our Neural Network model.

**Weight Factors** : Weight factors gives out the best linear combinations of attributes which would give the closest ouput

Weight factors and Bias values are first randomly given and then adjusted through `Back Propaagtion` and `Gradient Descent` to find the optimum combination

### `3.4` Forward Propagation : How the network makes predictions

The network makes predictions using forward propagation, which is just a bunch of matrix multiplications and the application of the activation function(s) as defined above. If $x$ is the 2-dimensional input to the network then predictions $\hat{y}$ (also two-dimensional) can be calculated as follows:

$$
\begin{aligned}
z_1 = xW_1 + b_1 \\
a_1 = \tanh(z_1) \\
z_2 = a_1W_2 + b_2 \\
a_2 = \hat{y} = \mathrm{softmax}(z_2)
\end{aligned}
$$

$z_i$ is the weighted sum of inputs of layer $i$ (bias included) so $z_1$ contains all the outputs from all the nodes in the first hidden layer so  $z_1[1]$ will be the output of first node of hidden layer , $z_1[2]$ the output of second node of hidden layer and so on. $a_i$ is the output of layer $i$ after applying the activation function, it forms a vector similar to $z_i$ and the values are tranformed from $z_i$ using an activation function.\
$W_1, b_1, W_2, b_2$ are parameters of the network, which are needed to learn from the training data. You can think of them as matrices transforming data between layers of the network. Looking at the matrix multiplications above you can figure out the dimensionality of these matrices. If 500 nodes are used for the hidden layer then $W_1 \in \mathbb{R}^{2\times500}$, $b_1 \in \mathbb{R}^{500}$, $W_2 \in \mathbb{R}^{500\times2}$, $b_2 \in \mathbb{R}^{2}$. Now you see why you'll have more parameters if we increase the size of the hidden layer.

### `3.5` Shapes of all the Vectors

Understand that all the variables like weight factors, bias , input x etc are all vectors. Vectorization is done to ease out processes like element wise multiplication and then its summation which would generally require a for loop but with a vectorization a simple dot product would suffice thus increasing the speed and also conveneient and short to code.
With neural networks , understanding the shapes of all the parameters is very important so lets look at the shapes :

Here , number of training examples are $m$=200 and **n_hid** is the number of nodes in hidden layer
1. $X$ has a shape of `(m,2)` : 2 is for the number of attributes i.e $x_1$ and $x_2$
2. $W_1$ has a shape of `(2,n_hid)` connecting each node of input layer and hidden layer and giving them weights : 2 is for the number of attributes i.e $x_1$ and $x_2$
3. $b_1$ has a shape of `(1,n_hid)` as there is one bias for each node in the hidden layer
4. $W_2$ has a shape of `(n_hid,2)` connecting each node from hidden layer to the output layer of 2 : 2 is for the output layer giving probabilities of `male` and `female`
5. $b_2$ has a shape of `(1,2)` as there is one bias for each node in the output layer thus 2 for 2 nodes (`male` and `female` probabilities)
6. $z_1$ has a shape of `(1,n_hid)` as it gives one output for each node and it does that $m$ such instances
7. $a_1$ has a shape of `(1,n_hid)` same as $z_1$ as only values inside are changed and not the shape
8. $ \hat{y}$ has a shape of `(1,2)` as its the final output and it has 2 probability values for each of **m** instances

### `3.6` Back Propagation : Learning the Parameters

Learning the parameters for the network means finding parameters ($W_1, b_1, W_2, b_2$) that minimize the error on the training data. But how to define the error? Especially for that a function that measures our error the loss function is called. Negative log neighbourhood would be used here . If you have $N$ training examples and $C$ classes then the loss for our prediction $\hat{y}$ with respect to the true labels $y$ is given by:

$$
\begin{aligned}
H(y,p) = - \sum_i y_i log(p_i)
\end{aligned}
$$
The formula looks complicated, but all it really does is sum over our training examples and the $p_{i}$ are the probabilities of the correct classes. So, the further away $y$ (the correct labels) and $\hat{y}$ (our predictions) are, the greater the loss will be. **Understand that if $p_{i}$ is the probability of the correct classes then $\hat{y}$ will always be equal to one. So this would be just summation of negative log of probabilities of the correct classes.**

Remember that the goal is to find the parameters that minimize the loss function. You can use gradient descent to find its minimum. Here the most basic version of Gradient Descent, also called batch gradient descent with a fixed learning rate is implemented.  Variations such as SGD (stochastic gradient descent) or minibatch gradient descent typically perform better in practice. So if you are serious you'll want to use one of these, and ideally you would also decay the learning rate over time but for now you can use this.

As an input, gradient descent needs the gradients (vector of derivatives) of the loss function with respect to the parameters: $\frac{\partial{L}}{\partial{W_1}}$, $\frac{\partial{L}}{\partial{b_1}}$, $\frac{\partial{L}}{\partial{W_2}}$, $\frac{\partial{L}}{\partial{b_2}}$. To calculate these gradients following formulas can be used,

Applying the backpropagation formula ,the following results are obtained :

$$
\begin{aligned}
\delta_3 = \hat{y} - y \\
\delta_2 = (1 - \tanh^2z_1) \circ \delta_3W_2^T \\
\frac{\partial{L}}{\partial{W_2}} = a_1^T \delta_3  \\
\frac{\partial{L}}{\partial{b_2}} = \delta_3\\
\frac{\partial{L}}{\partial{W_1}} = x^T \delta_2\\
\frac{\partial{L}}{\partial{b_1}} = \delta_2 \\
\end{aligned}
$$

$T$ signifies transpose of that matrix/vector.
**NOTE : in $\delta_3$ , $y$ would always be equal to 1 if $\hat{y}$ are the probabilities of the correct classes. So in the code its better to find the probabilities of the right classes and then subtract it by 1**
A More in-depth and better understanding can be given [here](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)

### `3.7` Differentiations :

The Softmax differentiation with the cross entropy loss can be given as :

$$
\begin{align}
L &= - \sum_i y_i log(p_i) \\
\frac{\partial L}{\partial o_i} &= - \sum_k y_k \frac{\partial log(p_k)}{\partial o_i } \\
&= - \sum_k y_k \frac{\partial log(p_k)}{\partial p_k} \times \frac{\partial p_k}{ \partial o_i} \\
&= - \sum y_k \frac{1}{p_k} \times \frac{\partial p_k}{\partial o_i} \\
\end{align}
$$

With the softmax derivative :

$$
\begin{align}
\frac{\partial L}{\partial o_i}  &= -y_i(1-p_i) - \sum_{k\neq i} y_k \frac{1}{p_k}(-p_k.p_i) \\
&= -y_i(1-p_i) + \sum_{k \neq 1} y_k.p_i \\
&= - y_i + y_ip_i + \sum_{k \neq 1} y_k.p_i \\
&= p_i\left( y_i +  \sum_{k \neq 1} y_k\right) - y_i \\
&= p_i\left( y_i +  \sum_{k \neq 1} y_k\right)  - y_i
\end{align}
$$

$y$ is a one hot encoded vector for the labels, so $\sum_k y_k = 1$ and $y_i +  \sum_{k \neq 1} y_k = 1$ , Thus simplifying it into :\

$\frac{\partial L}{\partial o_i} = p_i - y_i$ which is the same as $\delta_3 = p_i - y_i$

A much more intensive and indepth explanation can be provided [here](https://deepnotes.io/softmax-crossentropy) and [here](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)

**Forward and backward propagation can be understood with the help of following schematics**

**For single node :**

**For multiple :**

**These derivatives are calculated , multiplied by learning rate and then added to the weights so as these gradients tend to zero that is error is minimum, the final optimum weights are obtained**

**IMPORTANT** : In the following section the terms $dW1$ , $dW2$ , $db1$ ... etc are short forms of $\frac{\partial{L}}{\partial{W_1}}$ ,  $\frac{\partial{L}}{\partial{W_2}}$ , $\frac{\partial{L}}{\partial{b_1}}$ ... respectively and always assume the same for other gradient parameters also

## `4.` Implementation :

Now , that a proper understanding of neural networks has been obtained , its time to make a model implementing it

**Pre defining constant parameters**

```python
num_examples = len(X) # training set size
nn_input_dim = 2 # input layer dimensionality
nn_output_dim = 2 # output layer dimensionality

# Gradient descent parameters (I picked these by hand)
epsilon = 0.01 # learning rate for gradient descent
reg_lambda = 0.01 # regularization strength

```

### `4.1` Model function

**TASK : Build a function to implement the Neural Network**

```python
# This function learns parameters for the neural network and returns the model.
# - nn_hdim: Number of nodes in the hidden layer
# - num_passes: Number of passes through the training data for gradient descent
# - print_loss: If True, print the loss every 1000 iterations

# One completion of forward and backward pass happens in one pass and you'll be using 20000 such passes

### START CODE HERE : (Write code where '#' is given)

def build_model(nn_hdim, num_passes=20000, print_loss=False):

    # Initialize the parameters to random values.
    # With succesive gradient descents , the model will learn and will keep updating the parameters
    np.random.seed(0)
    W1 = np.random.randn(nn_input_dim, nn_hdim) / np.sqrt(nn_input_dim)
    b1 = np.zeros((1, nn_hdim))

    # Weights are divided by square root of dimensions to somehow normalise the weights and not have very high values
    # Seeing the earlier definition , define 'W2' and 'b2' in the same way
    # Take help of the earlier section where shapes are defined
    W2 = '#'
    # Understand that while weight factors cant be randomly assigned to zeros , however biases can.
    b2 = '#'

    # This is the model to return at the end
    model = {}

    # Gradient descent. For each batch...
    for i in range(0, num_passes):

        ## FORWARD PROPAGATION
        z1 = X.dot(W1) + b1
        a1 = np.tanh(z1)

        ## Seeing the earlier definition , define 'z2' and find 'probs' using softmax activation instead of tanh
        ## Take help of the section where forward propagation formulas are defined
        z2 = '#'
        exp_scores = '#'
        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

        ## BACK PROPAGATION
        delta3 = probs
        delta3[range(num_examples), y] -= 1
        dW2 = (a1.T).dot(delta3)
        db2 = np.sum(delta3, axis=0, keepdims=True)
        #keepdims =True keeps the earlier vectors dimensions which is very important

        ## Seeing the earlier definitions , define 'delta2' ,'dW1' ,'db1'
        ## Take help of the backward propagation formulas defined in the earlier sections

        delta2 = '#'
        dW1 = '#'
        db1 = '#'

        # Regularisation terms are added (b1 and b2 don't have regularization terms)
        dW2 += reg_lambda * W2
        dW1 += reg_lambda * W1

        # Gradient descent parameter update
        W1 += -epsilon * dW1
        b1 += -epsilon * db1

        ## Seeing the earlier definitions , define 'W2' , 'b2'
        ## Understand how parameters are updated , epsilon is the contant learning rate
        W2 = '#'
        b2 = '#'

        # Assigning new parameters to the model
        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

        # Optionally print the loss.
        # This is expensive because it uses the whole dataset, so don't want to do it too often.
        if print_loss and i % 1000 == 0:
             print("Loss after iteration %i: %f" %(i, calculate_loss(model)))

    return model
```

Now , The neural network is made which takes in input `nn_hdim` which is the number of nodes in hidden layer and returns the model keeping in mind the input and output parameters. The approach of this was to first randomly assign the parameters and with gradient descent , update it until it reaches the minimum.

Once model is made , its essential to make a `predict` function to predict the models outputs and then a `calculate_loss` function that calculates the loss between $\hat{y}$ and $y$

### `4.2` Predict function :

**TASK : Make a function `Predict` that takes in model and X as arguments and returns the maximum probabilities**

```python
# Helper function to predict an output (0 or 1)
def predict(model, x):
    W1, b1, W2, b2 = model['W1'], model['b1'], model['W2'], model['b2']

### START CODE HERE (Write code where '#' is given)

     # FORWARD PROPAGATION
    ## Write the full steps of forward propagration as before using the updated parameters
    z1 = '#'
    a1 = '#'
    z2 = '#'
    exp_scores = '#'
    probs = '#'

### END CODE
    return np.argmax(probs, axis=1)
# The maximum probabilites are returned
```

A good schematic to understand why the function returns the maximum probability is given below :

Now , that the output probabilites are calculated , The last step remaining is to calculate the loss between $\hat{y}$ and $y$ using `cross-entropy` function

**Remember it can also be written as summation of negative log of the correct class probabilities which will be implemented in the code**

### `4.3` Loss function:

**TASK : Make function to calculate loss between $\hat{y}$ and $y$**

```python
# Helper function to evaluate the total loss on the dataset
def calculate_loss(model):

    ### START CODE HERE : (Write code where '#' is given)

    ## Take parameters 'W1','b1','W2','b2' from 'model'
    ## Take help from the previously defined functions to understand how it can be done

    W1, b1, W2, b2 = '#'

    # Forward propagation to calculate our predictions
    ## Note : you can also use previously defined function predict instead

    # Write code to get all the parameters using forward propagation formulas
    z1 = '#'
    a1 = '#'
    z2 = '#'
    exp_scores = '#'
    probs = '#'

    ## Calculating the loss
    corect_logprobs = -np.log(probs['#', '#'])
    ## Remember the arguments are such that probs[,] gives out probabilities of correct classes similar to p_i in delta3 formula
    data_loss = np.sum(corect_logprobs)

    # Adding regulatization term to loss (optional)

    data_loss += reg_lambda/2 * (np.sum(np.square(W1)) + np.sum(np.square(W2)))
    return 1./num_examples * data_loss

### END CODE
```

**Now that all the helper functions and model functions are made its time to implement it**

**TASK : Using `build_model` function, take `nn_hdim=3` and `print_loss=True` and get the model in a variable called `model`**

```python
# Build a model with a 3-dimensional hidden layer
### START CODE HERE (~ 1 Line of code)

### END CODE HERE

```

**TASK : Plot the decision boundary using `plot_decision_boundary` function**

```python
# Plot the decision boundary
### START CODE HERE (~ 2 Lines of code)

## See the plot_decision_boundary implementation in logistic regression and write the arguments similarly
## Use predict function in lambda x : argument
## Give title to the plot

### END CODE HERE
```

## `5.` Varying the hidden layer size

In the example above you made a hidden layer size of 3. Let's now get a sense of how varying the hidden layer size affects the result.

**TASK : Using a `for_loop` and the functions build above. make models and plot decision boundaries of varying hidden layer sizes**

```python
plt.figure(figsize=(16, 32))
hidden_layer_dimensions = [1, 2, 3, 4, 5, 20, 50]
for i, nn_hdim in enumerate(hidden_layer_dimensions):
    ### START CODE HERE (FULL CODE)

    ## Use plt.subplots and plt.title to form subplots of different nn_hdim and give them title accordingly
    ## Make model using build_model and make plot decision boundary using plot_decision_boundary and predict functions
plt.show()

    ### END CODE
```

## `6.` Conclusion

After trying out different activation functions with different sizes of hidden layer ,get an idea of why alot of nodes cant be used in a single hidden layer and which activation functions work best . Based on this , make the adjustments and choose the best final model

Basic Neural Network from Scratch/Basic Neural Network from Scratch (To-Do Template).ipynb


# 3. Classification

 * Assign **labels** to **data points** (documents, images, users, etc.)
 * Training examples generalized to a **decision rule** (or more)

```python
import pandas as pd
import numpy as np
```

## Linear classifiers
 * Model expressed as a normal $w$ to a hyperplane separating two classes (and, optionally, a bias (offset) term $b$).
 * Linear classifiers are easy to understand, quick to train, work very well (especially in high dimensions) and make extremely efficient predictions.
 * In all future examples, use *homgeneous representation* and only define a model as $w$, with no separate bias term, for simplicity's sake.
     * Actually, the bias will simply be included in the model, as we replace $w = [w_1,\dots,w_d]$ with $\tilde w=[w_1,\dots,w_d,b]$.

## Perceptrons [book]
 * Linear binary classifier
 * Model: *weights* $w$ and a *threshold* $\theta$
 * Predicts:
     - +1 if $wx > \theta$
     - -1 if $wx < \theta$
     - "wrong" if $wx = \theta$ (!!!)
 * The weight vector symbolizes a hyperplane (s.t. $wx = \theta$)
 * **Only works for data which is linearly separable**
     - vanilla perceptron will not converge otherwise
     - perceptron will converge to *some* hyperplane
     - SVMs (see next section) try to find an optimal hyperplane, and also work well with data which are not perfectly linearly separable
 * The **Winnow Algorithm** is an improvement of the basic Perceptron which is guaranteed to converge (all features are 0 or 1).
 * Can be parallelized in a decent manner using multiple iterations of MapReduce
     - partition training data to mappers
     - map: emit all points which disagree with current $w$
     - reduce: perform all updates to $w$ in centralized way
     - repeat until convergence

### Multiclass Perceptrons
 * For $k$ classes, train $k$ Perceptrons, one for every class; (positive examples are examples of that class; negative examples are all others)
 * Classify by computing all $w_ix$ values, and picking the largest one, as long as it's larger than the threshold $\theta$. If it's not, then the new object does not belong to any known class.
 * In some cases (e.g. topic modelling), we can also have objects which may belong to more than one class (e.g. a news article can be about both sports and celebrities)

### Nonlinear Perceptrons
 * Transform data into another coordinate system first (e.g. to polar coordinates) and run Perceptron in *that* space. But more on this, later, when we talk about non-linear SVMs.

### Limitations
 * Perceptrons don't converge on non-linearily separable data
 * Perceptrons don't automatically choose a "best" separating hyperplane
 * Perceptrons can get (very) biased towards a class since they terminate as soon as they correctly classified all training data points (see textbook, Fig. 12.13)

## Support Vector Machines
 * Max-margin classifiers
 * Find $w$ so that the margin to closest representatives of each class is maximized. Can be shown (but not here) that this is the right thing to do in order to improve our generalization.
 * Margin size is $\frac{1}{||w||_2} = \mathit{margin}$
 * Want to maximize margin size, so we want to minimize $w^Tw$, while also making sure that the margin is actually correct (duh!): $\forall \> i, w^Tx_i\ge1$
 * There's always noise $\implies$ model as slack variables.
     - slack variables represent points we can't avoid mis-classifying
     - one slack variable for every point
         - $0$ if that point is "good" and can be classified OK by the SVM
         - $>0$ if that point is not correctly classified by our computed $w$, but we sort of have to live with it
         - in practice, most (if not all, if we're lucky) points $i$ have $\xi_i = 0$

### Convex optimization
 * This is the way we solve the SVM constrained minimization problem
 * Minimization because we want to "shrink" our model's magnitude (i.e. the margin size) and, as much as possible, the impact of the slack variables (data points which we cannot avoid misclassifying).
 * Constrained because we also want to ensure our model does the right thing and correctly classifies most points (for most points, we expect their $\xi_i$ to be 0, and $y_iw^Tx_i\ge1$, meaning that the point is snugly placed in the correct class; positive means the right side of the separation hyperplane; also greater than one means it's beyond the separation margin).
 * **Problem:** current approach has a run-time complexity of $\Omega(n^2)$, and simply doesn't work if the data don't fit in memory.

### Convex optimization, reformulated
 * Because linear algebra is awesome, we can reformulate the (primal) SVM definition into something a little more useful.
 * $\xi_i = \operatorname{max}(0, 1 - y_iw^Tx_i)$, from the constraint part; it's called hinge loss ($l_{\operatorname{hinge}}$)
 * Shove this into the minimization formula.
 * We get: $\min_w{\left (w^Tw + C\sum\limits_{i}\max(0, 1 - y_iw^Tx_i) \right )}$

### Convex optimization, reformulated, again
 * We can formulate a multicriteria optimization problem in two ways:
     * Minimize everything at once
     * Minimize one objective, while keeping the other under a certain bound
 * Our above (re)formulation belongs to the first category (one big-ass $\min$)
 * Let's write it in the second way.
 * Minimize the second part, as long as the first part ($w^Tw$) is below some threshold B.
 * We write that as:
     * $\min_w\sum\limits_i\max(0, 1 - y_i w^T x_i)$ s. t. $||w||_2 \lt \frac{1}{\sqrt{\lambda}}$
 * Interestingly enough, all 3 of our formulations produce the **same solutions**. TODO: ask algebra guru for more details about why.
 * However, the complexity varies. A lot. The last version is the best in that regard.

## Convex optimization, in general
 * Many supervized learning problems consists of two components: a loss function and a regularizer.
 * From a (handwavily explained) Bayesian standpoint, the former represents the evidence, while the latter, our prior knowledge (i.e. we *know* our model shouldn't be *too* complex) [citation needed]
 * As we saw before, there are two ways to state these problems mathematically.
     * Using a single "large" minimization (l + r)
     * Using a little minimization (l) plus a constraint (r)
 * We will focus on the second techniques, as it makes it possible to perform online learning.

SVM -> OCP -> online-to-batch conversion by averaging if we want to train a single SVM on fixed data set
If we also pick data at random -> SGD.

* TODO: strong convexity and its benefits
    * Using geometric examples can really help!

## Online Convex Programming (OCP)

It can be shown that going through the data in streaming fashion can lead to sensible solution, while allowing truly massive data sets to be processed.

Note that it is normal that some members of the training set are never used in a stochastic gradient descent algorithm.

```python
def shuffle_in_unison(a, b):
    """Helper used by the OCP implementation. Shuffles both numpy arrays in the same way."""
    assert len(a) == len(b)
    shuffled_a = np.empty(a.shape, dtype=a.dtype)
    shuffled_b = np.empty(b.shape, dtype=b.dtype)
    permutation = np.random.permutation(len(a))
    for old_index, new_index in enumerate(permutation):
        shuffled_a[new_index] = a[old_index]
        shuffled_b[new_index] = b[old_index]
    return shuffled_a, shuffled_b

```

```python
def svm_ocp(train_x, train_y):
    assert train_x.shape[0] == train_y.shape[0]
    assert train_y.shape[1] == 1

    # The regularization parameter.
    lbd = 0.005
    rows, features = train_x.shape

    # Initialize our model with zeros.
    model = np.zeros(features)
    model_sum = np.zeros(features)

    # If we don't shuffle our input, the learning gets *really* messed up.
    train_x_vals, train_y_vals = shuffle_in_unison(train_x.values,
                                                   train_y.values)

    for t, (x, y) in enumerate(zip(train_x_vals, train_y_vals)):
        eta = 1 / math.sqrt(t + 1)
        # eta = 1 / (lbd * (t + 2))
        prediction = np.dot(x, model)
        score = (y * prediction)[0]
        if score < 1:
            # We misclassified 'y'. Or weren't condifent enough.
            # Let's update the model.

            # Gradient descent step.
            model_prime = model + eta * y * x

            # Regularization step (stay inside the hypersphere with radius
            # lambda).
            model_prime_norm = np.linalg.norm(model_prime)
            regularizer = 1 / (math.sqrt(lbd) * model_prime_norm)
            model = model_prime * min(1, regularizer)

        model_sum += model

    # Online-to-batch conversion: we simply want to average over all iterations
    # of our model. This is a very simple and provably accurate method.
    # TODO(andrei) Consider skipping first 'k' elements.
    avg_model = model_sum / rows

    return avg_model
```

## PEGASOS vs. standard OCP/SGD

	PEGASOS	OCP/SGD

Updating	Batch updates	Single-point updates

Loss function	Strongly convex	Convex

Run-time complexity	Depends on number of dimensions and desired accuracy	Depends on number of data points

## Perceptrons vs. SVMs

(TODO)

	Perceptron	SVM

Scalability	Both can support millions of features

Supported data types	Numeric only	Numeric only, unless kernels are used

## Adapting to geometry (AdaGrad)
 * In some cases, gradient descent can take a long time to converge because of the distorted curvature of the loss function
 * We can "stretch" the dimensions to make our gradient "aim" better towards the minimum and thus converge faster
 * Transformation matrix $A$: $z = Aw$, $w=A^{-1}z$
 * Gradient descent in z-space:
     - $z_{t+1} = \operatorname{arg min}\limits_{z: A^{-1}z \in S} || z - \left(z_t - \eta\nabla_z g(z_t) \right) ||_2^2$
     - explained:
         - we still want to minimize, hence the $\operatorname{arg min}$
         - we still want to constrain based on the original space, so we impose the constraint on $w = A^{-1}z \in S$
         - the updated model is computed via gradient descent (duh) in the $\nabla_zg(z_t)$ part; $\eta$ just tweaks the learning speed as much as we want
     - We reach: $\operatorname{arg min}\limits_{w \in S} || w - \left( w_t - \eta A^{-1}\nabla_w f(w_t) \right) ||_A^2$ (which uses the Mahalanobis norm instead of the $\ell_2$ one)
 * All that's left is to pick an appropriate projection matrix $A$
     - positive definite
     - symmetric
     - all eigenvalues $\ge 0$
 * We could compute a nice $G_T$ which minimizes the regret upper bound...
 * But we obviously can't compute $G_T$ at time $t < T$ :(
 * See 06:24 for more details.
 * There's an alternate way of computing an optimal $G_t$ at every step
 * $G_t = \left( \sum\limits_{\tau = 1}^{t}\nabla f_{\tau}(w_{\tau})\nabla f_{\tau}(w_{\tau})^T \right)^\frac{1}{2}$
 * However, G is expensive to compute, and it also needs to be inverted.
 * Solution: $G_t := diag(G_t)$; this is much easier to work with and trivial to invert
 * **Bonus:** can prove that AdaGrad provides advantages if data is sparse (better upper bound for $\mathbb{E}[L(\hat{w})] - L(w^*)$)
 * Generalization of OCP and friends: Proximal gradient methods.

Note: AdaGrad is not covered in the textbook (as of January 2016)

### Mahalanobis distance

TODO: more info, since we need it to really grok ADAGRAD.

> The Mahalanobis distance is essentially the distance between a point and the centroid of a cluster, normalized by the standard deviation of the cluster in each dimension.
>
> -- Rajaraman, Anand, and Jeffrey D. Ullman. Mining of massive datasets. Vol. 77. Cambridge: Cambridge University Press, 2012.

\begin{equation}
    d_{\text{Mahalanobis}}(x, y) = \sqrt{\sum_{i=1}^{d}\left( \frac{p_i - c_i}{\sigma_i} \right)^{2}}
\end{equation}

### AdaGrad in exams

Not seen in any data mining exam from 2011 to 2014.

## Exploiting parallelism
 * Parallelize gradient computation (cluster)
     - needs to constantly sync value of $w$ after each iteration
 * Parallel stochastic gradient descent (cluster)
     - partition data
     - run SGD independently on each partition
     - average produced models
     - TODO: analysis of error rate
 * Hogwild! (multi-proc)
      - on local, multi-processor machine
      - no sychrnoization
      - works well, as long as the data is sparse!
 * Parameter server
      - keep track of $w$ on central machine, which receives $\Delta w$s every once in a while, after which it pushes the update to all the workers

```python

```


# Triple-six puzzle

Consider the following puzzle:

  * $0 \, 0 \, 0 = 6$
  * $1 \, 1 \, 1 = 6$
  * $2 \, 2 \, 2 = 6$
  * ...
  * $10 \, 10 \, 10 = 6$

Insert operators `+`, `-`, `*`, `/`, square root and factorial such that the equations hold, e.g., $(1 + 1 + 1)! = 6$.  For most, multiplle solutions are possible, e.g., trivially, $(\sqrt{1} + \sqrt{1} + \sqrt{1})! = 6$.  A few less trivial examples would be $3! + 3 - 3 = 6$, $3!*3/3 = 6$, or $3*3 - 3 = 6$.

We will use the sympy library to tackle the puzzle.

```python
import itertools
import sympy
```

## Representing expressions

We start by defining the binary operators `+`, `-`, `*` and `/` based on sympy primitives `sympy.Add`, `sympy.Mul`, and `sympy.Pow`.  For each operator, we set `evaluate` to `False` to ensure that we can create an expression that isn't symplified automatically, i.e., $x + x$ should not be simplified to $2x$.

```python
def Add(x, y):
    return sympy.Add(x, y, evaluate=False)
```

```python
def Sub(x, y):
    return Add(x, -y)
```

```python
def Mul(x, y):
    return sympy.Mul(x, y, evaluate=False)
```

```python
def Div(x, y):
    return Mul(x, sympy.Pow(y, -1))
```

```python
binary_operators = [Add, Sub, Mul, Div]
```

Next, we define the unary operators square root and factorial, again based on their sympy implementation.

```python
def Sqrt(x):
    return sympy.sqrt(x, evaluate=False)
```

```python
def Fac(x):
    return sympy.factorial(x, evaluate=False)
```

```python
unary_operators = [Sqrt, Fac]
```

Using these operators, we can build experssions in a symbol `x`.

```python
x = sympy.Symbol('x')
```

```python
expr = Sub(Mul(x, x), x)
expr
```

    -x + x*x

Substituing 3 for `x` yields 6.

```python
expr.subs(x, 3)
```

    6

We can construct more elaborate expressions as well.  This one will evalutate to 6 for `x` is 3.

```python
expr = Mul(Fac(x), Div(x, x))
expr
```

    (x/x)*factorial(x)

```python
expr.subs(x, 3) == 6
```

    True

## Counting expressions

Now that we have a way to represent expressions, we need a way to generate all relevant expressions systematically.

Unfortunately, the number of mathematical expressions satifying the equations is infinite.  This is easy to proof since, e.g.,$1! = 1$, so $(1 + 1 + 1)! = (1! + 1! + 1!)! = (1!! + 1!! + 1!!)! = \cdots = 6$.

This problem can be circumvented by limiting the number of consecutive unary operations to some number, say 2.  This would yield $x$, $x!$, $\sqrt{x}$, $x!!$, $\sqrt{x}!$, $\sqrt{x!}$, $\sqrt{\sqrt{x}}$.  In general, if the number of consecutive uniary opreations is $d_{unary}$, then the number of combinations is $c_{unary} = 2^{d_{unary} + 1} - 1$, so 7 for $n = 2$.

If we consider only a single binary operator, the number of such expression would be $7 \times 4 \times 7 \times 7$.  Now we can add the second binary operator.  The first expression will be either the first, or the second operated, so that gives rise to a factor of 2.  For the other operand, there are again 7 possibilities, and we can apply 7 combinations of unary operators.  In total, there are $7 \times 7 \times 4 \times 7 \times 2 \times 7 \times 4 \times 7$ expressions.

```python
d_unary = sympy.Symbol('d_unary')
n_unary = sympy.Symbol('n_unary')
c_unary = n_unary**(d_unary + 1) - 1
n_binary = sympy.Symbol('n_binary')
```

```python
nr_expressions = c_unary*c_unary*n_binary*c_unary*2*c_unary*n_binary*c_unary
nr_expressions.subs(((n_unary, 2), (n_binary, 4), (d_unary, 2)))
```

    537824

These expressions need to be evaluated for each number from 0 to 10 which gives a quite impressive number of evaluations.

```python
nr_evaluations = 11*nr_expressions
nr_evaluations
```

    22*n_binary**2*(n_unary**(d_unary + 1) - 1)**5

Obviously, many expressions will be equivalent due to the properties of the mathematical operations, e.g., associativity, $a + (b + c) = (a + b) + c = a + b + c$.

It is also worth pointing out that this will not (reasonably) work.  One of the expressions that will be generated is $((10!! \times 10!!)!! \times 10!!)!!$, and you don't even want to think about the number of digits required to represent that number, given that $100!$ has 158 digits!

```python
sympy.factorial(100)
```

    93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000

Since $x!!$ will not yield any solutions that are non-trial, we will exclude this case.  That means that the total number of expressinos is reduced to $6 \times 6 \times 4 \times 6 \times 2 \times 6 \times 4 \times 6$ expressions.

```python
print(f'The total number of expressions is {6*6*4*6*2*6*4*6}')
```

    The total number of expressions is 248832

## Generating expressions

Now we have all ingredients to construct expressions.  We can write a generator for that purpose.  We define a helper function to apply a list of function, e.g., `[f, g]` to an argument `x` to result in `f(g(x))`.

```python
def compose(func_list, x):
    expr = x
    for func in func_list:
        expr = func(expr)
    return expr
```

We need a function that will generate all expressions, and we implement it as a generator using `yield`.

```python
def expression_trees():
    u_ops = [[], [Sqrt], [Fac], [Sqrt, Sqrt], [Sqrt, Fac], [Fac, Sqrt]]
    x = sympy.Symbol('x')
    for u_ops_1 in u_ops:
        operand_11 = compose(u_ops_1, x)
        for u_ops_2 in u_ops:
            operand_12 = compose(u_ops_2, x)
            for b_op_1 in binary_operators:
                for u_ops_3 in u_ops:
                    operand_21 = compose(u_ops_3, b_op_1(operand_11, operand_12))
                    for u_ops_4 in u_ops:
                        operand_22 = compose(u_ops_4, x)
                        for b_op_2 in binary_operators:
                            for u_ops_5 in u_ops:
                                yield compose(u_ops_5, b_op_2(operand_21, operand_22))
                                yield compose(u_ops_5, b_op_2(operand_22, operand_21))
```

Let's verify that we really generate the expected number of expressions.

```python
counter = 0
for _ in expression_trees():
    counter += 1
print(f'{counter} expression generated')
```

    248832 expression generated

Finally, the following function will solve the puzzle for a given value of $x$.  It will print all expressions that evaluate to 6, and return the total number of such expressions.

```python
def solve_puzzle(value):
    counter = 0
    for i, expr in enumerate(expression_trees()):
        try:
            if expr.subs(x, value) == 6:
                counter += 1
                print(str(expr).replace('x', str(value)))
        except ValueError:
            pass
    print(f'{counter} solutions for {value:d}')
    return counter
```

```python
solve_puzzle(0)
```

Since it takes considerable time to generate all expressions, it is more efficient to do that only once, and evaluate it for all $x$ values at once.

```python
def solve_puzzle_all_values(max_value=10):
    counter = 0
    for i, expr in enumerate(expression_trees()):
        for value in range(max_value + 1):
            try:
                if expr.subs(x, value) == 6:
                    counter += 1
                    print(str(expr).replace('x', str(value)))
            except ValueError:
                pass
    print(f'{counter} solutions for 0 up to {max_value:d}')
    return counter
```


```python
import scipy.integrate
import math
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as dates
```

```python
#Modelo SEIR++:
# S -> Susceptibles
# E -> Expuestos
# I -> Infectados
# R -> Recuperados
# B -> Beta, Tasa de transmisión
# D -> sensación pública de riesgo

population = 12000000 #Aprox de la población
E0 = 3 # casos iniciales
infection_delay = 4 #aprocimación de cuantos días llevaban infectados los primeros casos antes de llegar

m_total = 3 #cantidad de medidas tomadas
m_days = [ 0 for i in range(m_total)]

# Días en los que se tomaron nuevas medidas por parte del gobierno.
m_days[0] = 18 #Días hasta las primeras medidas (especificamente cierre de vuelos internacionales)
m_days[1] = 23 #Días hasta la 2da medida mayor
m_days[2] = 60 #cuarentena total hipotética

daysTotal = 30 #Días analizados

gamma = 1/7  #valor cambiante
#gamma es la tasa de recuperación: 1/gamma será el período de tiempo medio (en días) de recuperación (pasa a ser no infector)
#es el tiempo medio que el paciente pasa estando infectado (en realidad es que pueda infectar a otros)

sigma = 1/5  #valor promedio relativamente fiel
# sigma es la velocidad con la que se transforma un expuesto en infector: 1/sigma es el período promedio de este cambio

r1 = 2.68 #Tasa básica de reproducción del COVID19 sin medidas tomadas, sería beta/gamma y algunas versiones del modelo usan r0 en vez de beta
# r es otro de los parámetros difíciles de estimar y que varía bastante
r0 = 8.4 #Valor máximo estimado con los resultados obtenidos en China
# en este caso tomamos r0 mucho mayor por causa de la cantidad de entradas de nuevos casos extranjeros y r1 será 2.68

beta0 = r0*gamma #Tasa de transmición inicial media sin medidas tomadas
beta1 = r1*gamma #Tasa de transmición luego del fin de los vuelos
#beta es la tasa de transmisión, 1/beta representa mayormente la probabilidad de contagio luego de un contacto Inf - Suscep
#beta es de los párametros más complicados de estimar por la existencia de asintomáticos y de los más importantes en el modelo

alfa = [0 for i in range(m_total)]
# fuerza de las medidas tomadas por el gobierno de 0 a 1 un valor que determina beta y es dificil de estimar
# esto se puede tomar como una función constante a trozos y graficar según los cambios en alfa
alfa[0] = 0.2 #rigor de las medidas iniciales
alfa[1] = 0.45 #segundas medidas
alfa[2] = 0.6 #cuarentena de 0.5 ~ 0.6?
#alfa[2] = 0.65 #ESTO ES UN EJEMPLO DE MEDIDAS EFECTIVAS

k = 1117.3
#intensidad de la reacción de los individuos, valor calculado en China extremadamente alto

presympt_time = 5.2
#Tiempo medio en que demora la aparición de los sintomas, tiempo de incubación

noSymptoms = 0.012
#porciento de personas asintomáticas

fRatio = 0.90 #De nuevo un valor optimista que depende del rigor de las medidas
#porciento de casos de infectados con síntomas detectados

findRatio = (1-noSymptoms) * fRatio
#porciento de casos de infectados detectados (se asume que todos son hospitalizados)

findTime = presympt_time + 3 # (Valor arbitrario)
#tiempo medio en días que se demora en hospitalizar a un paciente infectado

timeInHospital = 14 #tiempo que se mantiene un paciente infectado en el hospital
timeInfected = 1.0 / gamma #duración media de la infección

UCI_count = 200
#cantidad de unidades de cuidados intensivos disponibles en el país

CriticalRate = 0.047
#porciento de pacientes que necesitan cuidados intensivos

critical_lag = 5 #(Valor arbitrario)
#tiempo medio en días que un paciente hospitalizado pasa a estado crítico

death_delay = 7 #(Valor arbitrario)
#tiempo medio en días que un paciente en estado crítico muere

UCIFatalityRate = 0.023 # (Calculado en China)
#porciento de pacientes que utilizan UCIs y mueren

FatalityRate = CriticalRate * 0.95 # (Valor arbitrario)
#porciento de pacientes críticos que mueren sin acceso a UCI

Hosp = [0 for i in range(daysTotal)]
#array con la cantidad de casos hospitalizados en un día dado

Critical = [0 for i in range(daysTotal)]
#array con la cantidad de casos críticos en un día dado

first_case_date = dates.datetime.date(2020,3,6) #Fecha de llegada (no de detección) del primer caso registrado

###OPCIONES###
plt_detail = 1  # Plotear con cuadrículas (1), plotear "normal" (0)
plt_options = (0,1,1,1,0) # 0 para ocultar, 1 mostrar : muertes, recuperados, expuestos, infectados, hospitalizados
plt_method = 0  # 0 para predeterminado, 1 para Euler, 2 para Runge-Kutta de orden 2, 3 para Runge-Kutta orden 4
h = 0.1 # valor de h usado en los métodos numéricos
##############
```

```python
def GetParams():

    print("Elige entre la modelación por defecto(0) or personalizada(1)")
    op = int(input())

    print("Total de días a analizar: ")
    daysTotal = int(input())

    if op == 1:
        print("Población: ")
        _population = int(input())

        print("Cantidad de casos iniciales: ")
        _E0 = int(input())

        print("Fecha de llegada de los primeros casos (YYYY-MM-DD): ")
        print("Ejemplo: '2020-2-29'")
        y,m,d = input().split("-")
        _first_case_date = dates.datetime.date(int(y),int(m),int(d))

        print("Cantidad de UCIs (unidades de cuidados intensivos) :" )
        _UCI_count = int(input())

        print("Cantidad de medidas mayores tomadas por el gobierno: ")
        _m_total = int(input())
        _m_days = [0 for i in range(_m_total)]

        print("Ingresa el # del día (contando desde el caso inicial) de esas medidas, separados por un espacio: ")
        print("Ejemplo: '7 21 40'")
        md = input().split(" ")
        for i in range (0,_m_total) : _m_days[i] = int(md[i])

        print("Intruduce la severidad (0-10) de cada medida, separado por un espacio: ")
        print("Ejemplo: '1 2 7'")
        sev = input().split(" ")
        _alfa = [0 for i in range(_m_total)]
        for i in range (0,_m_total): alfa[i] = (int(sev[i])/10)

        print("Valor inicial estimado de r0: ")
        r0 = float(input())
        _beta0 = r0*gamma

        return _population,_E0,_beta0,_first_case_date,_UCI_count,_m_total,_m_days,_alfa,daysTotal

    return population,E0,beta0,first_case_date,UCI_count,m_total,m_days,alfa,daysTotal
```

```python
#Modelo por defecto

def model(seir, t, m_total, m_days, alfa, beta0):

    S,E,I,R = seir
    N = S + E + I + R

    def D(I,t):
        return CriticalRate*I

    def Beta (I,t,N):
        if t < m_days[0]:
            return beta0
        else:
            for i in range (1,m_total):
                if t < m_days[i]:
                    return beta1 * (1-alfa[i-1])* np.power((1-D(I,t)/N),k)
        return beta1 * (1-alfa[m_total-1])* np.power((1-D(I,t)/N),k)

    beta = Beta(I,t,N)

    dS = -1 * beta * S * I / N
    dE = beta * S * I / N - sigma * E
    dI = sigma * E - gamma * I
    dR = gamma * I

    return dS, dE, dI, dR
```

```python
#Método de solución predeterminado:

def Solve (params):  #Predeterminado

    population, m_total,m_days,alfa,beta0,daysTotal,E0 = params
    X = np.arange(daysTotal)
    params2 = m_total,m_days,alfa,beta0

    seir = population - E0, 0, E0, 0

    y_data_var = scipy.integrate.odeint(model, seir, X, params2)
    S, E, I, R = y_data_var.T

    return X, S, E, I, R
```

```python
# Modelo version 2 para métodos numéricos

def D(I,t):
        return CriticalRate*I

def Beta (I,t,N):
        if t < m_days[0]:
            return beta0
        else:
            for i in range (1,m_total):
                if t < m_days[i]:
                    return beta1 * (1-alfa[i-1])* np.power((1-D(I,t)/N),k)
        return beta1 * (1-alfa[m_total-1])* np.power((1-D(I,t)/N),k)

def dS(t, S, E, I, R):
    N = S+E+I+R
    return -1 * Beta(I,t,N) * S * I / N

def dE(t, S, E, I, R):
    N = S+E+I+R
    return Beta(I,t,N) * S * I / N - sigma * E

def dI(t, S, E, I, R):
    return sigma * E - gamma * I

def dR(t, S, E, I, R):
    return gamma * I
```

```python
#Métodos de solución numéricos

def Solve_Euler (params,h):  #Método de Euler

    population, m_total,m_days,alfa,beta0,daysTotal,E0 = params
    X = np.arange(daysTotal)
    S,E,I,R,t = population - E0, 0, E0, 0, 0

    steps = int (1/h)
    depth = daysTotal*steps

    S_ = np.arange(daysTotal)
    S_dat = [0 for i in range (depth)]
    E_ = np.arange(daysTotal)
    E_dat = [0 for i in range (depth)]
    I_ = np.arange(daysTotal)
    I_dat = [0 for i in range (depth)]
    R_ = np.arange(daysTotal)
    R_dat = [0 for i in range (depth)]

    for i in range(depth):

        F1 = dS(t, S, E, I, R)
        G1 = dE(t, S, E, I, R)
        H1 = dI(t, S, E, I, R)
        I1 = dR(t, S, E, I, R)

        if (i>0): t = i*h

        S_dat[i] = S + h * F1
        S = S_dat[i]
        E_dat[i] = E + h * G1
        E = E_dat[i]
        I_dat[i] = I + h * H1
        I = I_dat[i]
        R_dat[i] = R + h * I1
        R = R_dat[i]

    for j in range (daysTotal):
        S_ [j] =  (S_dat[j*steps])
        E_ [j] =  (E_dat[j*steps])
        I_ [j] =  (I_dat[j*steps])
        R_ [j] =  (R_dat[j*steps])

    return X, S_, E_, I_, R_

def Solve_Runge_Kutta_2 (params,h):  #Runge-Kutta de segundo orden

    population, m_total,m_days,alfa,beta0,daysTotal,E0 = params
    X = np.arange(daysTotal)
    S,E,I,R,t = population - E0, 0, E0, 0, 0

    steps = int (1/h)
    depth = daysTotal*steps

    S_ = np.arange(daysTotal)
    S_dat = [0 for i in range (depth)]
    E_ = np.arange(daysTotal)
    E_dat = [0 for i in range (depth)]
    I_ = np.arange(daysTotal)
    I_dat = [0 for i in range (depth)]
    R_ = np.arange(daysTotal)
    R_dat = [0 for i in range (depth)]

    for i in range(depth):

        F1 = dS(t, S, E, I, R)
        G1 = dE(t, S, E, I, R)
        H1 = dI(t, S, E, I, R)
        I1 = dR(t, S, E, I, R)

        if (i>0): t = i*h

        F2 = dS(t + h, S + h * F1, E + h * G1, I + h * H1, R + h * I1)
        G2 = dE(t + h, S + h * F1, E + h * G1, I + h * H1, R + h * I1)
        H2 = dI(t + h, S + h * F1, E + h * G1, I + h * H1, R + h * I1)
        I2 = dR(t + h, S + h * F1, E + h * G1, I + h * H1, R + h * I1)

        S_dat[i] = (S + (h / 2) * (F1 + F2))
        S = S_dat[i]
        E_dat[i] = (E + (h / 2) * (G1 + G2))
        E = E_dat[i]
        I_dat[i] = (I + (h / 2) * (H1 + H2))
        I = I_dat[i]
        R_dat[i] = (R + (h / 2) * (I1 + I2))
        R = R_dat[i]

    for j in range (daysTotal):
        S_ [j] =  (S_dat[j*steps])
        E_ [j] =  (E_dat[j*steps])
        I_ [j] =  (I_dat[j*steps])
        R_ [j] =  (R_dat[j*steps])

    return X, S_, E_, I_, R_

def Solve_Runge_Kutta_4 (params,h): #Runge Kutta de cuarto orden

    population, m_total,m_days,alfa,beta0,daysTotal,E0 = params
    X = np.arange(daysTotal)
    S,E,I,R,t = population - E0, 0, E0, 0, 0

    steps = int (1/h)
    depth = daysTotal*steps

    S_ = np.arange(daysTotal)
    S_dat = [0 for i in range (depth)]
    E_ = np.arange(daysTotal)
    E_dat = [0 for i in range (depth)]
    I_ = np.arange(daysTotal)
    I_dat = [0 for i in range (depth)]
    R_ = np.arange(daysTotal)
    R_dat = [0 for i in range (depth)]

    for i in range(depth):

        F1 = dS(t, S, E, I, R)
        G1 = dE(t, S, E, I, R)
        H1 = dI(t, S, E, I, R)
        I1 = dR(t, S, E, I, R)

        if (i>0):
            t = i*h

        F2 = dS(t + h / 2, S + h * F1 / 2, E + h * G1 / 2 , I + h * H1 / 2, R + h * I1 / 2)
        G2 = dE(t + h / 2, S + h * F1 / 2, E + h * G1 / 2 , I + h * H1 / 2, R + h * I1 / 2)
        H2 = dI(t + h / 2, S + h * F1 / 2, E + h * G1 / 2 , I + h * H1 / 2, R + h * I1 / 2)
        I2 = dR(t + h / 2, S + h * F1 / 2, E + h * G1 / 2 , I + h * H1 / 2, R + h * I1 / 2)

        F3 = dS(t + h / 2, S + h * F2 / 2, E + h * G2 / 2, I + h * H2 / 2, R + h * I2 / 2)
        G3 = dE(t + h / 2, S + h * F2 / 2, E + h * G2 / 2, I + h * H2 / 2, R + h * I2 / 2)
        H3 = dI(t + h / 2, S + h * F2 / 2, E + h * G2 / 2, I + h * H2 / 2, R + h * I2 / 2)
        I3 = dR(t + h / 2, S + h * F2 / 2, E + h * G2 / 2, I + h * H2 / 2, R + h * I2 / 2)

        F4 = dS(t + h, S + h * F3, E + h * G3, I + h * H3, R + h * I3)
        G4 = dE(t + h, S + h * F3, E + h * G3, I + h * H3, R + h * I3)
        H4 = dI(t + h, S + h * F3, E + h * G3, I + h * H3, R + h * I3)
        I4 = dR(t + h, S + h * F3, E + h * G3, I + h * H3, R + h * I3)

        S_dat[i] = (S + (h / 6) * (F1 + 2 * F2 + 2 * F3 + F4))
        S = S_dat[i]
        E_dat[i] = (E + (h / 6) * (G1 + 2 * G2 + 2 * G3 + G4))
        E = E_dat[i]
        I_dat[i] = (I + (h / 6) * (H1 + 2 * H2 + 2 * H3 + H4))
        I = I_dat[i]
        R_dat[i] = (R + (h / 6) * (I1 + 2 * I2 + 2 * I3 + I4))
        R = R_dat[i]

    for j in range (daysTotal):
        S_ [j] =  (S_dat[j*steps])
        E_ [j] =  (E_dat[j*steps])
        I_ [j] =  (I_dat[j*steps])
        R_ [j] =  (R_dat[j*steps])

    return X, S_, E_, I_, R_
```

```python
#Calendario simplificado

def calc_date(date0,days_passed):
    x = date0.toordinal() + days_passed
    ans = date0.fromordinal(x)
    return ans
```

```python
#Resolviendo y usando el modelo:

def BuildSolution (params):

    population,E0,beta0,first_case_date,UCI_count,m_total,m_days,alfa,daysTotal = params

    params2 = population,m_total,m_days,alfa,beta0,daysTotal,E0

    X, S, E, I, R = Solve(params2) # DEFAULT

    if (plt_method == 1):
        X, S, E, I, R = Solve_Euler(params2,h)
    if (plt_method == 2):
        X, S, E, I, R = Solve_Runge_Kutta_2(params2,h)
    if (plt_method == 3):
        X, S, E, I, R = Solve_Runge_Kutta_4(params2,h)

    death_count = 0 #cantidad de muertes
    hosp_total = 0 #cantidad total de hospitalizados
    rec_total = 0 #total de recuperados
    firstfoundcase = 0   #día del primer caso hospitalizado

    deaths = [0 for i in range (0,daysTotal)]  #muertes diarias
    Dprev = 0
    deaths = [0 for i in range (0,daysTotal)]
    Rprev = 0
    lag = int(findTime)+1
    clag = lag + critical_lag
    dlag = death_delay

    #Calcula la cantidad de hospitalizados, críticos y las muertes diarias
    #Los hospitalizados y críticos forman parte del número de infectados y las muertes de los "recuperados"

    Hosp = [0 for i in range(daysTotal)]
    #array con la cantidad de casos hospitalizados en un día dado
    Critical = [0 for i in range(daysTotal)]
    #array con la cantidad de casos críticos en un día dado

    for i in range (lag,daysTotal):
        Hosp[i] = I[i-lag] * findRatio
        hosp_total +=  Hosp[i]
        if (firstfoundcase == 0 and i > 2 and hosp_total > 0):
            firstfoundcase = i
        Critical[i] = I[i-clag] * CriticalRate
        needed_UCI = Critical[i-clag] - UCI_count #diferencia entre las UCIs actuales y las necesitadas
        if (needed_UCI > 0):
            precalc_FR = (UCIFatalityRate) * (UCI_count/Critical[i-clag]) + (FatalityRate) * (needed_UCI/Critical[i-clag])
        FR = UCIFatalityRate if Critical[i-clag] <= UCI_count else precalc_FR

        if i + dlag < daysTotal:
            deaths[i+dlag] = FR * (R[i] - Rprev)
            death_count += deaths[i+dlag]
            Rprev = R[i]
            Dprev = deaths[i+dlag]

    if firstfoundcase > infection_delay : firstfoundcase -= infection_delay
    # La idea de esto es que el primer caso ya estaba contagiado cuando arribo, se aproxima cuanto tiempo

    return X,S,E,I,R,death_count, hosp_total, rec_total, first_case_date, firstfoundcase,Hosp,daysTotal
```

```python
def BuildStadistics(data):

    X,S,E,I,R,death_count, hosp_total, rec_total, first_case_date, firstfoundcase,Hosp, daysTotal = data

    infected = E[0]    #total de infectados
    worstday = 0      #día con más infectados
    maxinfects = 0    #máximo número de infectados
    maxexposure = 0   #máximo número de población expuesta a la enfermedad (posibles contagios)
    maxexposure_day = 0  #día de mayor exposición al contagio

    for i in range (0,daysTotal):
        if I[i] > maxinfects:
            worstday = i
            maxinfects = I[i]
        if R[i] > infected: infected = R[i]
        if E[i] > maxexposure:
            maxexposure_day = i
            maxexposure = E[i]
    infected = infected + I [daysTotal-1]

    return X,S,E,I,R,infected,worstday,maxinfects,maxexposure,maxexposure_day,death_count,hosp_total,rec_total,firstfoundcase,first_case_date,Hosp,daysTotal
```

```python
def PlotSolution(data):

    X,S,E,I,R,infected,worstday,maxinfects,maxexposure,maxexposure_day,death_count,hosp_total,rec_total,firstfoundcase,first_case_date,Hosp,daysTotal = data

    fig = plt.figure(dpi=100, figsize=(16,9))
    ax = fig.add_subplot(111)

    if plt_options[0]==1:
        ax.plot(X, deaths, 'black', alpha=0.72, lw=2, label='Muertes(t)')  #muy pequeño en relación a los otros
    if plt_options[1]==1:
        ax.plot(X, R, 'g', alpha=0.72, lw=2, label='Recuperados(t)')       #muy grande en relación a los otros
    if plt_options[2]==1:
        ax.plot(X, E, 'y', alpha=0.72, lw=2, label='Expuestos (t)')
    if plt_options[3]==1:
        ax.plot(X, I, 'r', alpha=0.72, lw=2, label='Infectados (t)')
    if plt_options[4]==1:
        ax.plot(X, Hosp, 'b', alpha=0.72, lw=2, label='Hospitalizados (t)')

    ax.set_xlabel('Tiempo en días')
    ax.set_ylabel('Cantidad de Personas')
    ax.set_ylim(bottom=1.0)

    if plt_detail == 1 :
        ax.grid(ls='-')
    else:
        ax.grid(b=True, which='major', c='w', lw=2, ls='-')

    plt.legend(loc='best')
    plt.show()
```

```python
def AditionalInfo(data):

    X,S,E,I,R,infected,worstday,maxinfects,maxexposure,maxexposure_day,death_count,hosp_total,rec_total,firstfoundcase,first_case_date, Hosp,daysTotal = data

    inf = int(infected)
    dt = daysTotal - 1
    day0 = first_case_date
    maxinfects = int(maxinfects)
    maxexposure = int(maxexposure)
    death_count = int(death_count)

    print("Total de infectados: ", inf,"(",(inf/population*100),"% de la población)")
    print("Muertes: ", death_count, "(",(death_count/population*100),"% de la población)")
    print("Pico de infección:", calc_date(day0,worstday),"con un total de", int(I[worstday]), "infectados")
    print("Día de mayor exposición al virus:", calc_date(day0,maxexposure_day),"con un total de", maxexposure, "expuestos")
    print("Pacientes infectados hospitalizados:",int(Hosp[dt]), "(",(Hosp[dt]/I[dt]*100),"% de los infectados actuales)")
    print("Primera hospitalización el:", calc_date(day0,firstfoundcase))
    print("Cantidad de pacientes recuperados:", int(R[dt]))
```

```python
def Init():
    params = GetParams()
    solution = BuildSolution(params)
    data = BuildStadistics(solution)
    PlotSolution(data)
    AditionalInfo(data)
```

```python
Init()
```

```python

```


<!-- dom:TITLE: Demo - Working with Functions -->
# Demo - Working with Functions
<!-- dom:AUTHOR: Mikael Mortensen Email:mikaem@math.uio.no at Department of Mathematics, University of Oslo. -->
<!-- Author: -->
**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **August 7, 2020**

**Summary.** This is a demonstration of how the Python module [shenfun](https://github.com/spectralDNS/shenfun) can be used to work with
global spectral functions in one and several dimensions.

## Construction

A global spectral function $u(x)$ is represented on a one-dimensional
domain (a line) as

$$
u(x) = \sum_{k=0}^{N-1} \hat{u}_k \psi_k(x)
$$

where $\psi_k(x)$ is the $k$'th basis function and $x$ is a
position inside the domain. $\{\hat{u}_k\}_{k=0}^{N-1}$ are the
expansion coefficient for the series, and often referred to as the
degrees of freedom. There is one degree of freedom per basis function.
We can use any number of basis functions,
and the span of the chosen basis is then a function space. Also part of the
function space is the domain, which is
specified when a function space is created. To create a function space
$T=\text{span}\{T_k\}_{k=0}^{N-1}$ for
the first N Chebyshev polynomials of the first kind on the default domain $[-1, 1]$,
do

```python
from shenfun import *
N = 8
T = FunctionSpace(N, 'Chebyshev', domain=(-1, 1))
```

The function $u(x)$ can now be created with all N coefficients
equal to zero as

```python
u = Function(T)
```

When using Chebyshev polynomials the computational domain is always
$[-1, 1]$. However, we can still use a different physical domain,
like

```python
T = FunctionSpace(N, 'Chebyshev', domain=(0, 1))
```

and under the hood shenfun will then map this domain to the reference
domain through

$$
u(x) = \sum_{k=0}^{N-1} \hat{u}_k \psi_k(2(x-0.5))
$$

## Approximating analytical functions

The `u` function above was created with only zero
valued coefficients, which is the default. Alternatively,
a [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) may be initialized using a constant
value

```python
T = FunctionSpace(N, 'Chebyshev', domain=(-1, 1))
u = Function(T, val=1)
```

but that is not very useful. A third method to initialize
a [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) is to interpolate using an analytical
Sympy function.

```python
import sympy as sp
x = sp.Symbol('x', real=True)
u = Function(T, buffer=4*x**3-3*x)
print(u)
```

    [-6.93889390e-17 -5.47405982e-17 -1.82237827e-18  1.00000000e+00
      1.96261557e-17 -3.33066907e-16  1.49457239e-16 -2.93881977e-16]

Here the analytical Sympy function will first be evaluated
on the entire quadrature mesh of the `T` function space,
and then forward transformed to get the coefficients. This
corresponds to a projection to `T`. The projection is

Find $u_h \in T$, such that

$$
(u_h - u, v)_w = 0 \quad \forall v \in T,
$$

where $v \in \{T_j\}_{j=0}^{N-1}$ is a test function,
$u_h=\sum_{k=0}^{N-1} \hat{u}_k T_k$ is a trial function and the
notation $(\cdot, \cdot)_w$ represents a weighted inner product.
In this projection $u_h$ is the [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function), $u$ is the sympy function and we use sympy
to exactly evaluate $u$ on all quadrature points
$\{x_j\}_{j=0}^{N-1}$. With quadrature we then have

$$
(u, v)_w = \sum_{j\in\mathcal{I}^N} u(x_j) v(x_j) w_j \forall v \in T,
$$

where $\mathcal{I}^N = (0, 1, \ldots, N-1)$ and $\{w_j\}_{j\in \mathcal{I}^N}$
are the quadrature weights. The left hand side of the projection is

$$
(u_h, v)_w = \sum_{j\in\mathcal{I}^N} u_h(x_j) v(x_j) w_j \forall v \in T.
$$

A linear system of equations arise when inserting for the
basis functions

$$
\left(u, T_i\right)_w = \tilde{u}_i \forall i \in \mathcal{I}^N,
$$

and

$$
\begin{align*}
\left(u_h, T_i \right)_w &= (\sum_{k\in \mathcal{I}^N} \hat{u}_k T_k , T_i)_w \\
  &= \sum_{k\in \mathcal{I}^N} \left( T_k, T_i\right)_w \hat{u}_k
\end{align*}
$$

with the mass matrix

$$
a_{ik} = \left( T_k, T_i\right)_w   \forall (i, k) \in \mathcal{I}^N \times \mathcal{I}^N,
$$

we can now solve to get the unknown
expansion coefficients. In matrix notation

$$
\hat{u} = A^{-1} \tilde{u},
$$

where $\hat{u}=\{\hat{u}_i\}_{i\in \mathcal{I}^N}$,
$\tilde{u}=\{\tilde{u}_i\}_{i \in \mathcal{I}^N}$ and
$A=\{a_{ki}\}_{(i,k) \in \mathcal{I}^N \times \mathcal{I}^N}$

## Adaptive function size

The number of basis functions can also be left open during creation
of the function space, through

```python
T = FunctionSpace(0, 'Chebyshev', domain=(-1, 1))
```

This is useful if you want to approximate a function and
are uncertain how many basis functions that are required.
For example, you may want to approximate the function $\cos(20 x)$.
You can then find the required [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function) using

```python
u = Function(T, buffer=sp.cos(20*x))
print(len(u))
```

    45

We see that $N=45$ is required to resolve this function. This agrees
well with what is reported also by [Chebfun](https://www.chebfun.org/docs/guide/guide01.html).
Note that in this process a new [FunctionSpace()](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.FunctionSpace) has been
created under the hood. The function space of `u` can be
extracted using

```python
Tu = u.function_space()
print(Tu.N)
```

    45

To further show that shenfun is compatible with Chebfun we can also
approximate the Bessel function

```python
T1 = FunctionSpace(0, 'Chebyshev', domain=(0, 100))
u = Function(T1, buffer=sp.besselj(0, x))
print(len(u))
```

    83

which gives 83 basis functions, in close agreement with Chebfun (89).
The difference lies only in the cut-off criteria. We cut frequencies
with a relative tolerance of 1e-12 by default, but if we make this criteria
a little bit stronger, then we will also arrive at a slightly higher number:

```python
u = Function(T1, buffer=sp.besselj(0, x), reltol=1e-14)
print(len(u))
```

    87

Plotting the function on its quadrature points looks
a bit ragged, though:

```python
%matplotlib inline

import matplotlib.pyplot as plt
Tu = u.function_space()
plt.plot(Tu.mesh(), u.backward())
```

To improve the quality of this plot we can instead evaluate the
function on more points

```python
xj = np.linspace(0, 100, 1000)
plt.plot(xj, u(xj))
```

Alternatively, we can refine the function, which simply
pads zeros to $\hat{u}$

```python
up = u.refine(200)
Tp = up.function_space()
plt.plot(Tp.mesh(), up.backward())
```

The padded expansion coefficients are now given as

```python
print(up)
```

    [ 6.43655435e-02 -1.37262942e-01  1.26593632e-01 -1.31630372e-01
      1.19810309e-01 -1.19499576e-01  1.07387826e-01 -9.95196201e-02
      8.81197795e-02 -7.07299943e-02  6.13974837e-02 -3.40222966e-02
      2.84578300e-02  6.05886310e-03 -6.23468323e-03  4.01956708e-02
     -3.44202241e-02  5.58353154e-02 -4.58848577e-02  4.33923041e-02
     -3.39778387e-02  6.26445963e-03 -3.37831264e-03 -3.27304442e-02
      2.63630973e-02 -4.12313235e-02  3.07429560e-02 -7.73458857e-03
      4.10441868e-03  3.29922472e-02 -2.47812478e-02  2.71654360e-02
     -1.81421444e-02 -2.02112290e-02  1.53660102e-02 -3.02943723e-02
      1.97324599e-02  1.94051084e-02 -1.41460820e-02  2.47321146e-02
     -1.49668436e-02 -3.02763148e-02  1.98156190e-02 -2.08517347e-03
     -6.91643640e-05  3.13799203e-02 -1.81924272e-02 -3.77110141e-02
      2.23051599e-02  2.82766986e-02 -1.67226896e-02 -1.59965877e-02
      9.40673776e-03  7.34392211e-03 -4.28398348e-03 -2.84497604e-03
      1.64428109e-03  9.53197714e-04 -5.45450869e-04 -2.80998822e-04
      1.59136941e-04  7.38240716e-05 -4.13664358e-05 -1.74579288e-05
      9.67744748e-06  3.74610168e-06 -2.05413769e-06 -7.34256601e-07
      3.98257402e-07  1.32202237e-07 -7.09282453e-08 -2.19709845e-08
      1.16601214e-08  3.38459230e-09 -1.77684428e-09 -4.85085285e-10
      2.51925667e-10  6.48958256e-11 -3.33432158e-11 -8.12799381e-12
      4.13195149e-12  9.55498771e-13 -4.80687010e-13 -1.05592419e-13
      5.24752655e-14  1.10454430e-14 -6.02774535e-15  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]

## More features

Since we have used a regular Chebyshev basis above, there
are many more features that could be explored simply by going through
[Numpy's Chebyshev module](https://numpy.org/doc/stable/reference/routines.polynomials.chebyshev.html).
For example, we can create a Chebyshev series like

```python
import numpy.polynomial.chebyshev as cheb
c = cheb.Chebyshev(u, domain=(0, 100))
```

The Chebyshev series in Numpy has a wide range of possibilities,
see [here](https://numpy.org/doc/stable/reference/generated/numpy.polynomial.chebyshev.Chebyshev.html#numpy.polynomial.chebyshev.Chebyshev).
However, we may also work directly with the Chebyshev
coefficients already in `u`. To find the roots of the
polynomial that approximates the Bessel function on
domain $[0, 100]$, we can do

```python
z = Tu.map_true_domain(cheb.chebroots(u))
```

Note that the roots are found on the reference domain $[-1, 1]$
and as such we need to move the result to the physical domain using
`map_true_domain`. The resulting roots `z` are both real and imaginary,
so to extract the real roots we need to filter a little bit

```python
z2 = z[np.where((z.imag == 0)*(z.real > 0)*(z.real < 100))].real
print(z2[:5])
```

    [ 2.40482556  5.52007811  8.65372791 11.79153444 14.93091771]

Here `np.where` returns the indices where the condition is true. The condition
is that the imaginary part is zero, whereas the real part is within the
true domain $[0, 100]$.

**Notice.**

Using directly `cheb.chebroots(c)` does not seem to work (even though the
series has been generated with the non-standard domain) because
Numpy only looks for roots in the reference domain $[-1, 1]$.

We could also use a function space with boundary conditions built
in, like

```python
Td = FunctionSpace(0, 'C', bc=(sp.besselj(0, 0), sp.besselj(0, 100)), domain=(0, 100))
ud = Function(Td, buffer=sp.besselj(0, x))
print(len(ud))
```

    82

As we can see this leads to a function space of dimension
very similar to the orthogonal space.

The major advantages of working with a space with boundary conditions
built in only comes to life when solving differential equations. As
long as we are only interested in approximating functions, we may just
as well stick to the orthogonal spaces. This goes for Legendre as
well as Chebyshev.

## Multidimensional functions

Multidimensional tensor product spaces are created
by taking the tensor products of one-dimensional function spaces.
For example

```python
C0 = FunctionSpace(20, 'C')
C1 = FunctionSpace(20, 'C')
T = TensorProductSpace(comm, (C0, C1))
u = Function(T)
```

Here $\text{T} = \text{C0} \otimes \text{C1}$, the basis function is
$T_i(x) T_j(y)$ and the Function `u` is

$$
u(x, y) = \sum_{i=0}^{N-1} \sum_{j=0}^{N-1} \hat{u}_{ij} T_i(x) T_j(y).
$$

The multidimensional Functions work more or less exactly like for the
1D case. We can here interpolate 2D Sympy functions

```python
y = sp.Symbol('y', real=True)
u = Function(T, buffer=sp.cos(10*x)*sp.cos(10*y))
X = T.local_mesh(True)
plt.contourf(X[0], X[1], u.backward())
```

Like for 1D the coefficients are computed through projection,
where the exact function is evaluated on all quadrature points
in the mesh.

The Cartesian mesh represents the quadrature points of the
two function spaces, and can be visualized as follows

```python
X = T.mesh()
for xj in X[0]:
    for yj in X[1]:
        plt.plot((xj, xj), (X[1][0, 0], X[1][0, -1]), 'k')
        plt.plot((X[0][0], X[0][-1]), (yj, yj), 'k')
```

We may alternatively plot on a uniform mesh

```python
X = T.local_mesh(broadcast=True, uniform=True)
plt.contourf(X[0], X[1], u.backward(kind='uniform'))
```

## Curvilinear coordinates

With shenfun it is possible to use curvilinear coordinates,
and not necessarily with orthogonal basis vectors. With
curvilinear coordinates the computational coordinates are
always straight lines, rectangles and cubes. But the physical
coordinates can be very complex.

Consider the unit disc with polar coordinates. Here
the position vector $\mathbf{r}$ is given by

$$
\mathbf{r} = r\cos \theta \mathbf{i} + r\sin \theta \mathbf{j}
$$

The physical domain is $\Omega = \{(x, y): x^2 + y^2 < 1\}$,
whereas the computational domain is the Cartesian product
$D = \{(r, \theta) \in [0, 1] \times [0, 2 \pi]\}$.

We create this domain in shenfun through

```python
r, theta = psi = sp.symbols('x,y', real=True, positive=True)
rv = (r*sp.cos(theta), r*sp.sin(theta))
B0 = FunctionSpace(20, 'C', domain=(0, 1))
F0 = FunctionSpace(20, 'F')
T = TensorProductSpace(comm, (B0, F0), coordinates=(psi, rv))
```

Note that we are using a Fourier space for the azimuthal
direction, since the solution here needs to be periodic.
We can now create functions on the space using an
analytical function in computational coordinates

```python
u = Function(T, buffer=(1-r)*r*sp.sin(sp.cos(theta)))
```

However, when this is plotted it may not be what you expect

```python
X = T.local_mesh(True)
plt.contourf(X[0], X[1], u.backward(), 100)
```

We see that the function has been plotted in computational coordinates,
and not on the disc, as you probably expected. To plot on
the disc we need the physical mesh, and not the computational

```python
X = T.local_cartesian_mesh()
plt.contourf(X[0], X[1], u.backward(), 100)
```

**Notice.**

The periodic plot does not wrap all around the circle. This is
not wrong, we have simply not used the same point twice, but it
does not look very good. To overcome this problem we can wrap the
grid all the way around and re-plot.

```python
up = u.backward()
xp, yp, up = wrap_periodic([X[0], X[1], up], axes=[1])
plt.contourf(xp, yp, up, 100)
```

## Adaptive functions in multiple dimensions

If you want to find a good resolution for a function in multiple
dimensions, the procedure is exactly like in 1D. First create function
spaces with 0 quadrature points, and then call [Function](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Function)

```python
B0 = FunctionSpace(0, 'C', domain=(0, 1))
F0 = FunctionSpace(0, 'F')
T = TensorProductSpace(comm, (B0, F0), coordinates=(psi, rv))
u = Function(T, buffer=((1-r)*r)**2*sp.sin(sp.cos(theta)))
print(u.shape)
```

    (5, 12)

The algorithm used to find the approximation in multiple dimensions
simply treat the problem one direction at the time. So in this case
we would first find a space in the first direction by using
a function ` ~ ((1-r)*r)**2`, and then along the second using
a function ` ~ sp.sin(sp.cos(theta))`.

<!-- ======= Bibliography ======= -->