# Introduction to Python

Programming is a process that instructs the computer to perform a task. Let's think you have a task to write 1 to 1000. It will take a lot of time if you want to write this using pen and paper. But you can easily do it by writing a few lines of a program. Programming will make your life simple in complex work. There are so many programming languages have been demonstrated so far. Python is one of them, which is widely used and the most popular one. In fact, Python becomes the No. 1 programming language in 2017 based on [IEEE Spectrum](https://spectrum.ieee.org/computing/software/the-2017-top-programming-languages), and is used to build many popular websites like Youtube and Instagram. In this notebook, we will learn this python language to solve simple real-life sequential algorithm-based problems using the input-process-output (IPO) approach. The sequential algorithm performs it's each part of task sequentially, which is easy to implement in Python programming language. Each part of IPO approach will help us to learn the introductory Python programming knowledge. Thus, this notebook will give the user introductory level Python programming knowledge.

```python
from IPython.display import HTML
import io
import base64

hide_me = ''
HTML('''
<form action="javascript:code_toggle()"><input style="opacity:0" type="submit" value="Click here to toggle on/off the raw code."></form>''')
```

## Basic: Python Programming

A tradition in computer science for your first program is to write a program which outputs a simple phrase: `Hello World!`. To complete this tradition, input the following into the cell below:

```python
print("Hello World!")
 ```

```python

```

You can run the above cell with either pressing `Ctrl + Enter` or by selecting the cell and clicking `Run` at the top”, which will give the output: `Hello World!`. Here we have used the `print` statement, which can output a sentence enclosed within quotations to the screen.

### Variables and Input Reading

#### Variables

Let's think, you have a pen and a pencil. Now, you want to find the total of these two items. You can do this thing like `print(1 + 1)`. This line of the program will execute $2$. Let's see this by running the following section:

```python
print(1+1)
```

Instead of using `print(1 + 1)`, you can give them a proper name such as `pen` and `pencil`. These `pen` and `pencil` are known as **variables**. This is Python’s way of “remembering” a value. Variables can be numbers like integers or decimals  (for a computer, decimals are known as “floating point numbers”), or even words and characters, which are known as “strings”. Let's write a program to find the total amount of your pen and pencil using variable concept.

```python
pen =1

pencil = 1

total = pen + pencil

print(total)
```

#### Reading Input from the Keyboard

Almost in all programs, we need to read some values from the user as the **input**, and store them in some variables, so that we can use them later. The input function always returns '*string*' which is a sequence of characters or words. Let's see the following example of taking an input variable as string type:

```python
variable = input("Prompt message")

color = input("Your favourite color:" )
```
From the above expression, `color` input variable can be any string such as `Red`.

Let's try the above input function in the cell below, you can run it with either pressing `Ctrl + Enter` or by selecting the cell and clicking `Run` at the top”.

```python
#NBVAL_SKIP

color = input("your favourite color: ")

wish = input("Wish message of your friend's birthday: ")
```

In the below cell, you can create your own input function based on the above example.

```python

```

We can also take any '*numerical*' input from the **input** function. But we have to initialize it before the input function. There are various types of numerical input. Here we will familiar with `integer` and `float` numbers. Integer is a whole number, and it can be positive, negative and zero. But integer can not be the fractional number. Float defines the number which has decimal point. We can take an integer and float input number by writing '`int`' and '`float`' just before the `input` function respectively. Float number will also work in case of the whole number, but integer number will not take float number. Let's see a couple of example of taking input using integer and float:

```python
price = int(input("Price of your pen: "))
```

From the above expression, `price` input variable can be any whole number such as $4$. But it can not be any decimal point number such as $4.5$.

```python
price = float(input("Price of your pen: "))
```
From the above expression. `price` variable will be decimal point number such as $4.5$. But, it will also work in case of whole number such as $4$.

We have to decide the type of numerical input type based on our goal. Let's run the cell below as we did earlier to check the above examples:

```python
#NBVAL_SKIP

price = int(input("Price of your pen: "))

price = float(input("Price of your pen: "))
```

## Creating Your First Algorithm
An algorithm is a set of operations or steps to be performed to solve a problem or a task. Everything can be written as an algorithm, for example, let's write a simple algorithm of our daily morning life:

1. Wake up from sleep
2. Brush your teeth
3. Take a shower
4. Take the breakfast
5. Get into the car or bus for school

Let's write another simple algorithm to call to your friend:

1. Take your mobile phone
2. Go to your contact list
3. Search your friend's name
4. Call your friend

These two algorithms are the simple examples that we follow in our real life. In computing, programmers write the set of instructions of an algorithm that instructs computer to perform that task.

A sequential algorithm performs the set of operations sequentially from start to finish. Sequential algorithm is easy to understand and implement. We will familiar with the basic level of python programming from this notebook using this sequential algorithm.

The input/process/output (IPO) approach is widely used to solve a computing problem. We will also use this approach to solve the sequential algorithm computing problem using the python programming. This IPO approach will also help us to learn the introductory Python knowledge. Let's see the following points and it will give you the clear idea:

* Input part will teach us how to take input for a problem.
* Process part will teach us to perform any arithmetic operations.
* Output part will help to display the output on the computer screen.

Let's try to understand each part of a IPO approach.

1. **Input** is pre-existing data of the algorithm, which has been provided by an external system. In this notebook, pre-existing data will be variable and external system will be you or reader.

2. **Process** is the most important in this IPO approach. In another way, we can say that it is the heart of the algorithm. It takes the input and performs some logical operations based on the goal of the algorithm to transform it to the next output stage. Here, logical operation will be arithmetic operations based on the output of the algorithm.

3. **Output** is the result of the algorithm which has been obtained by processing the input.

In a sequential algorithm, operations have been performed sequentially. So, processing occurs only when all input required data are available and output occurs only after appropriate processing has occurred. Let's see the following example that will give you a clear idea of input, process, and output (IPO) approach of a sequential algorithm:

### Example 1

***Write an algorithm or program that will take numbers from you/user, and show the average of these three number.***

### Solution

To solve this algorithm or program, we first have to define the input, process and output. Let's see the following points to define them for this computing problem:

* The goal of this algorithm is to compute the average of three numbers. So, we have to take these three numbers, which will act as **input** of the algorithm.

* Now, we have to find the process part of the algorithm. To find that we have to think how you will get the average from these three numbers. We have to perform following arithmetic operation to get the output, which will act as **process** part of the algorithm.

    \begin{equation}
    \text{avg} = \frac{\text{num1} + \text{num2} + \text{num3}}{3}
    \end{equation}

* Finally, we will get the average of these three numbers at the **output** stage from the process part.

From the above explanation we can say that to define the input and process we can ask the following questions:

* *Input*: **What do we need to obtain the output?**

* *Process*: **How will we obtain the output from the input?**

#### Algorithm

Let's see the algorithm and IPO part of this problem by running the following cell:

```python
# This Part is not related to reader. But reader will run this part by select this cell and press "ctrl + enter" to watch the video

hide_me

video = io.open('avg-flowchart.mp4', 'rb').read()
encoded = base64.b64encode(video)
HTML(data=''''''.format(encoded.decode('ascii')))
```

```python
#NBVAL_SKIP

#
# Compute the average
#

# Read the input value from user : 3 numbers
num1 = int(input("Enter your number 1: "))
num2 = int(input("Enter your number 2: "))
num3 = int(input("Enter your number 3: "))

# Compute the avaerage of the numbers : Processing
avg = (num1 + num2 + num3)/3

# Display the result : Output
print("The average of the numbers:", avg)
```

#### Testing

Testing is an important part of programming. Testing is performed to check whether the program is running expectedly or not. Let's test the above program using the following test cases:

1. Case 1: **Input** = (0, 5, 7) -----> **Output** = 4

2. Case 2: **Input** = (0, -2, 5)-----> **Output** = 1

Input = (1.5,  , ) -----> Error will be shown after taking the first number as input. It will be shown as **`ValueError: invalid literal for int() with base 10: '1.5'`**. It will also indicate the line of the program that we are getting this error by showing an arrow in that line. In that line, we take integer as input and which is a whole number. But we are giving input $1.5$ which is a decimal point float number, not a whole number. That's why this value error is showing.

#### Revised Program

We can protect ourselves from this **`"ValueError"`** by building in some protection into the input statements. So, we will take the input as a float number. This will allow us to take both whole, decimal point, and negative numbers. Now let's see the revised program of this algorithm below:

```python
#NBVAL_SKIP

#
# Compute the average
#

# Read the input value from user : 3 number
num1 = float(input("Enter your number 1: "))  #Change occurs
num2 = float(input("Enter your number 2: "))  #Change occurs
num3 = float(input("Enter your number 3: "))  #Change occurs

# Compute the avaerage of the numbers : Processing
avg = (num1 + num2 + num3)/3

# Display the result : Output
print("The average of the numbers:", avg)
```

Let's think another problem as read the three numbers and display these numbers in the output.

**Is the above problem can be solved using the IPO approach?**

Let's use the technique that we set earlier to obtain the input and process part.

*Input*: *What do we need to obtain the output?*

We need three numbers from user to obtain the output. So, these three numbers will act as input.

*Process*: *How will we obtain the output from the input?*

Our goal is to display the three numbers that you get from the input. But we do not have to perform any operation from input to get the output. So, this problem does not have any process part.

The above problem can not be solved using IPO approach as it does not have any process part. But this is a sequential algorithm. So, we have to decide by yourself that it can be solved using the IPO approach or not based on the problem. Let’s take a look at a more practical real-world example of a sequential algorithm that can be solved using the IPO approach.

### Example 2

***Write an algorithm or program that will read seconds and then displays the equivalent hours, minutes remaining seconds.***

### Solution

Let's compute the input and process part using the technique that we used earlier for this problem:

* Input: Number of seconds.

* Process: Arithmetic operations.

* Output: Equivalent hour, minute and remaining seconds.

#### Algorithm

Let's run the below cell to see the input, process and output of this problem:

```python
# This Part is not related to reader. But reader will run this part by select this cell and press "ctrl + enter" to watch the video

hide_me

video = io.open('time-flowchart.mp4', 'rb').read()
encoded = base64.b64encode(video)
HTML(data=''''''.format(encoded.decode('ascii')))
```

```python
#NBVAL_SKIP

#
# Time conversion : Seconds to equivalent hours, minutes and remaining seconds
#

# Read the seconds from user : Input
second = int(input("Enter the time in seconds: "))

# Compute the hours: Processing
hour = second/3600
second = second - (hour * 3600)

# Compute the minutes and remaining seconds: Processing
minute = second/60
second = second - (minute * 60)

# Display the result of Hours, Minutes and Seconds : Output
print("Equivalent Time:", hour , "hours", minute, "minutes", second, "seconds")
```

### Testing

Let's test the above program using the following test cases:

1. Case 1: **Input** = (3600) -----> **Output** = 1 hours 0 minutes 0 seconds

2. case 2: **Input** = (3620) -----> **Output** = 1.005555 hours 0 minutes 0 seconds, But the output should be 1 hours 0 minutes 20 seconds

3. Case 3: **Input** = (3690) -----> **Output** = 1.025 hours 7.57912e-15 minutes 0 seconds, But the output should be 1 hours 1 minutes 30 seconds

Case 1 and 2 are not giving the expected outcomes. But no error is showing on the screen. As the goal of the program is to convert the seconds into hours, minutes and seconds. So, hour and minute cannot be decimal point float number. Let's see the following lines of the program:

```python
hour = second/3600
minute = second/60
```

In these two lines, we have not indicated the type of the hour and minute variables. So, the program will take them as a float variable by default from the division. But hour variable will be a whole number, not a decimal point number as remaining second from hour variable will go to calculate the minute. Same thing for the minute variable. So, these two variables (i.e. hour and minute) will be the integer type. We have to indicate it in the above lines of the program.

#### Revised Program

```python
#NBVAL_SKIP

#
# Time conversion : Seconds to equivalent hours, minutes and remaining seconds
#

# Read the seconds from user : Input
second = int(input("Enter the time in seconds: "))

# Compute the hours: Processing
hour = int(second/3600)         # Change occurs
second = second - (hour * 3600)

# Compute the minutes and remaining seconds: Processing
minute = int(second/60)        # Change occur
second = second - (minute * 60)

# Display the result of Hours, Minutes and Seconds : Output
print("Equivalent Time:", hour , "hours", minute, "minutes", second, "seconds")
```

The above two examples give you the clear idea of solving algorithm using the IPO approach of python programming. Let's try to solve following exercises from the knowledge of above examples. The python programs of the exercises are also given. It will be better if you solve these exercises by hiding the python programs. You can easily hide the program by clicking the first cell of this notebook and press '`ctrl + enter`'. Finally, check the output from the test cases. If you are not getting expected output, you can check the given program by doing the same thing at the first cell that you did earlier. Try to find out program error by comparing the given program.

### Exercise 1
***Write a program that will calculate total cost of a restaurant bill. The program will read initial cost of the bill. Tax rate is $10\%$, and tip is $5\%$ of the initial cost (without tax). The program displays the output as tax, tip and overall cost of the meal.***

Test the program by taking the following inputs:
1. Case 1: **Input** = initial cost: 120 ----> **Output** = Tax amount: $12$, Tip amount: $6$, Overall Cost: $138$
2. Case 2: **Input** = initial cost: 150.5 ----> **Output** = Tax amount: $15.05$, Tip amount: $7.525$, Overall Cost: $173.075$

```python
#NBVAL_SKIP

# Hide this part when you are solving the exercise
# Hide Technique: Click the first cell and press 'ctrl + enter' or Select the first cell and press 'Run' at the top

#
# Restaurant bill : Compute the overall cost, tax and tip amount
#

TAX_RATE = .1
TIP_RATE = .05

# Read the seconds from user : Input
initial_cost = float(input("Enter the initial cost: "))

# Compute the tax, tip and total: Processing
tax = initial_cost * TAX_RATE
tip = initial_cost * TIP_RATE
overall_cost = initial_cost + tax + tip

# Display the result of tax, tip and overall cost : Output
print("Tax amount:", tax , "\nTip amount:", tip, "\nOverall Cost:", overall_cost) # \n will display the output in a new line
```

### Exercise 2
***Write a program that will take a two digit number and display the sum of its two digits.***

Test the program using the following inputs:
1. Case 1: **Input** = number: 10 -----> **Output** = Tens digit: 1, Ones digit: 0, Sum: 1
2. Case 2: **Input** = number: 15 -----> **Output** = Tens digit: 1, Ones digit: 5, Sum: 6

```python
#NBVAL_SKIP

# Hide this part when you are solving the exercise
# Hide Technique: Click the first cell and press 'ctrl + enter' or Select the first cell and press 'Run' at the top

#
# Adding the digits of a two digit number
#

# Read the number from user : Input
num = int(input("Enter the number: "))

# Compute the tens, ones and add them: Processing
tens = int(num/10)
ones = num - (tens * 10)
add = tens + ones

# Display the result of Hours, Minutes and Seconds : Output
print("Tens digit:", tens , "\nOnes digit:", ones, "\nSum: ", add)
```

If you have to see the python program to solve the above two exercises then try to solve the below real-world problem and check the result of your program with given test case result. The python program is not given for the below problem.

### Exercise 3

***Let's think you are filling an application form for a part-time job. In the application form, you have to fill the height in feet and inches. But you know your height in centimeters. Let's write a program to convert your height from centimeters to feet and inches.***

**Hint**: 1 foot = 12 inches and 1 inch = 2.54 centimetres

Test the program using the following inputs:
1. Case 1: **Input** = centimetres: 183 -----> **Output** = Feet: 6, Inches: 0.0472
2. Case 2: **Input** = centimetres: 182 -----> **Output** = Feet: 5, Inches: 11.6535

## Conclusion

This is an introductory level of python notebook. First, we introduced some fundamental knowledge of python programming as printing a statement using `print` command, familiar with various forms (i.e. string, integer, float) of variables, and how to a take various forms of variables as input from the user using `input` function. Then we familiarized with the sequential algorithm and the IPO approach to solve the algorithm. Each part of IPO approach was also discussed. Finally, we introduced two simple and real-life examples of the sequential algorithm using the IPO approach and solved them using the introductory knowledge of python programming that we learned earlier in this notebook. In those examples, we showed the technique to select the input and process unit, test cases to check the Python program, error from program, and revised program to overcome the error. We also introduced three exercises for practice purpose. The Python programs were also provided for the first two exercises for learning purposes. Various test cases are provided for each exercise to test the output. This notebook will help the user learning the introductory level python programming skill by solving sequential algorithms using the IPO approach.

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


# Matrix generation

## Init symbols for *sympy*

```python
from sympy import *
from geom_util import *
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
import sys
sys.path.append("../")

%matplotlib inline

%reload_ext autoreload
%autoreload 2
%aimport geom_util
```

```python
# Any tweaks that normally go in .matplotlibrc, etc., should explicitly go here
%config InlineBackend.figure_format='retina'
plt.rcParams['figure.figsize'] = (12, 12)

plt.rc('text', usetex=True)

plt.rc('font', family='serif')
# SMALL_SIZE = 42
# MEDIUM_SIZE = 42
# BIGGER_SIZE = 42

# plt.rc('font', size=SMALL_SIZE)          # controls default text sizes
# plt.rc('axes', titlesize=SMALL_SIZE)     # fontsize of the axes title
# plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels
# plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
# plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
# plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize
# plt.rc('figure', titlesize=BIGGER_SIZE)  # fontsize of the figure title

init_printing()
```

```python
N = CoordSys3D('N')
alpha1, alpha2, alpha3 = symbols("alpha_1 alpha_2 alpha_3", real = True, positive=True)
```

## Cylindrical coordinates

```python
R, L, ga, gv = symbols("R L g_a g_v", real = True, positive=True)
```

```python
a1 = pi / 2 + (L / 2 - alpha1)/R

a2 = 2 * pi * alpha1 / L

x1 = (R + ga * cos(gv * a1)) * cos(a1)
x2 = alpha2
x3 = (R + ga * cos(gv * a1)) * sin(a1)

r = x1*N.i + x2*N.j + x3*N.k

z = ga/R*gv*sin(gv*a1)
w = 1 + ga/R*cos(gv*a1)

dr1x=(z*cos(a1) + w*sin(a1))
dr1z=(z*sin(a1) - w*cos(a1))

r1 = dr1x*N.i + dr1z*N.k
r2 =N.j

mag=sqrt((w)**2+(z)**2)

nx = -dr1z/mag
nz = dr1x/mag

n = nx*N.i+nz*N.k

dnx=nx.diff(alpha1)
dnz=nz.diff(alpha1)

dn= dnx*N.i+dnz*N.k
```

```python
Ralpha = r+alpha3*n

R1=r1+alpha3*dn
R2=Ralpha.diff(alpha2)
R3=n
```

```python
r1
```

```python
R1a3x=-1/(mag**3)*(w*cos(a1) - z*sin(a1))*(-1/R*w+ga*gv*gv/(R*R)*cos(gv*a1))*z+(1/mag)*(1/R*w*sin(a1)+ga*gv*gv/(R*R)*cos(gv*a1)*sin(a1)+2/R*z*cos(a1))
R1a3x
```

```python

ddr=r1.diff(alpha1)
cp=r1.cross(ddr)

k=cp.magnitude()/(mag**3)
k

```

k=trigsimp(k)
k

```python
k=simplify(k)
k
```

```python
q=(1/R*w+ga*gv*gv/(R*R)*cos(gv*a1))
f=q**2+4/(R*R)*z*z

f=trigsimp(f)
f
```

```python
f=expand(f)
f
```

```python
trigsimp(f)
```

```python
q=(1/R*w+ga*gv*gv/(R*R)*cos(gv*a1))
f1=q*w+2/R*z*z

f1=trigsimp(f1)
f1
```

```python
f1=expand(f1)
f1
```

```python
f1=trigsimp(f1)
f1
```

```python
R1a3x = trigsimp(R1a3x)
R1a3x
```

```python
R1
```

```python
R2
```

```python
R3
```

### Draw

```python
import plot

%aimport plot

x1 = Ralpha.dot(N.i)
x3 = Ralpha.dot(N.k)

alpha1_x = lambdify([R, L, ga, gv, alpha1, alpha3], x1, "numpy")
alpha3_z = lambdify([R, L, ga, gv, alpha1, alpha3], x3, "numpy")

R_num = 1/0.8
L_num = 2
h_num = 0.1
ga_num = h_num/3
gv_num = 20

x1_start = 0
x1_end = L_num
x3_start = -h_num/2
x3_end = h_num/2

def alpha_to_x(a1, a2, a3):
    x=alpha1_x(R_num, L_num, ga_num, gv_num, a1, a3)
    z=alpha3_z(R_num, L_num, ga_num, gv_num, a1, a3)
    return x, 0, z

plot.plot_init_geometry_2(x1_start, x1_end, x3_start, x3_end, alpha_to_x)
```

```python
%aimport plot

R3_1=R3.dot(N.i)
R3_3=R3.dot(N.k)

R3_1_x = lambdify([R, L, ga, gv, alpha1, alpha3], R3_1, "numpy")
R3_3_z = lambdify([R, L, ga, gv, alpha1, alpha3], R3_3, "numpy")

def R3_to_x(a1, a2, a3):
    x=R3_1_x(R_num, L_num, ga_num, gv_num, a1, a3)
    z=R3_3_z(R_num, L_num, ga_num, gv_num, a1, a3)
    return x, 0, z

plot.plot_vectors(x1_start, x1_end, 0, alpha_to_x, R3_to_x)
```

```python
%aimport plot

R1_1=R1.dot(N.i)
R1_3=R1.dot(N.k)

R1_1_x = lambdify([R, L, ga, gv, alpha1, alpha3], R1_1, "numpy")
R1_3_z = lambdify([R, L, ga, gv, alpha1, alpha3], R1_3, "numpy")

def R1_to_x(a1, a2, a3):
    x=R1_1_x(R_num, L_num, ga_num, gv_num, a1, a3)
    z=R1_3_z(R_num, L_num, ga_num, gv_num, a1, a3)
    return x, 0, z

plot.plot_vectors(x1_start, x1_end, h_num/2, alpha_to_x, R1_to_x)
```

### Lame params

```python
H1 = sqrt((alpha3*((-(1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*sin((L/2 - alpha1)/R) - ga*gv*sin(gv*(pi/2 + (L/2 - alpha1)/R))*cos((L/2 - alpha1)/R)/R)*(-ga*gv*(1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*sin(gv*(pi/2 + (L/2 - alpha1)/R))/R**2 + ga**2*gv**3*sin(gv*(pi/2 + (L/2 - alpha1)/R))*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R**3)/((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)**2 + ga**2*gv**2*sin(gv*(pi/2 + (L/2 - alpha1)/R))**2/R**2)**(3/2) + ((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*cos((L/2 - alpha1)/R)/R + ga*gv**2*cos((L/2 - alpha1)/R)*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R**2 - 2*ga*gv*sin((L/2 - alpha1)/R)*sin(gv*(pi/2 + (L/2 - alpha1)/R))/R**2)/sqrt((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)**2 + ga**2*gv**2*sin(gv*(pi/2 + (L/2 - alpha1)/R))**2/R**2)) + (1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*cos((L/2 - alpha1)/R) - ga*gv*sin((L/2 - alpha1)/R)*sin(gv*(pi/2 + (L/2 - alpha1)/R))/R)**2 + (alpha3*(((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*cos((L/2 - alpha1)/R) - ga*gv*sin((L/2 - alpha1)/R)*sin(gv*(pi/2 + (L/2 - alpha1)/R))/R)*(-ga*gv*(1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*sin(gv*(pi/2 + (L/2 - alpha1)/R))/R**2 + ga**2*gv**3*sin(gv*(pi/2 + (L/2 - alpha1)/R))*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R**3)/((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)**2 + ga**2*gv**2*sin(gv*(pi/2 + (L/2 - alpha1)/R))**2/R**2)**(3/2) + ((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*sin((L/2 - alpha1)/R)/R + ga*gv**2*sin((L/2 - alpha1)/R)*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R**2 + 2*ga*gv*sin(gv*(pi/2 + (L/2 - alpha1)/R))*cos((L/2 - alpha1)/R)/R**2)/sqrt((1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)**2 + ga**2*gv**2*sin(gv*(pi/2 + (L/2 - alpha1)/R))**2/R**2)) + (1 + ga*cos(gv*(pi/2 + (L/2 - alpha1)/R))/R)*sin((L/2 - alpha1)/R) + ga*gv*sin(gv*(pi/2 + (L/2 - alpha1)/R))*cos((L/2 - alpha1)/R)/R)**2)
H2=S(1)
H3=S(1)

H=[H1, H2, H3]
DIM=3
dH = zeros(DIM,DIM)
for i in range(DIM):
    dH[i,0]=H[i].diff(alpha1)
    dH[i,1]=H[i].diff(alpha2)
    dH[i,2]=H[i].diff(alpha3)

trigsimp(H1)

```

### Metric tensor

${\displaystyle \hat{G}=\sum_{i,j} g^{ij}\vec{R}_i\vec{R}_j}$

```python
%aimport geom_util
G_up = getMetricTensorUpLame(H1, H2, H3)
```

${\displaystyle \hat{G}=\sum_{i,j} g_{ij}\vec{R}^i\vec{R}^j}$

```python
G_down = getMetricTensorDownLame(H1, H2, H3)
```

### Christoffel symbols

```python
DIM=3

G_down_diff = MutableDenseNDimArray.zeros(DIM, DIM, DIM)
for i in range(DIM):
    for j in range(DIM):
        for k in range(DIM):

            G_down_diff[i,i,k]=2*H[i]*dH[i,k]

GK = getChristoffelSymbols2(G_up, G_down_diff, (alpha1, alpha2, alpha3))
```

### Gradient of vector

$
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)
=
B \cdot
\left(
\begin{array}{c}
u_1 \\
\frac { \partial u_1 } { \partial \alpha_1} \\
\frac { \partial u_1 } { \partial \alpha_2} \\
\frac { \partial u_1 } { \partial \alpha_3} \\
u_2 \\
\frac { \partial u_2 } { \partial \alpha_1} \\
\frac { \partial u_2 } { \partial \alpha_2} \\
\frac { \partial u_2 } { \partial \alpha_3} \\
u_3 \\
\frac { \partial u_3 } { \partial \alpha_1} \\
\frac { \partial u_3 } { \partial \alpha_2} \\
\frac { \partial u_3 } { \partial \alpha_3} \\
\end{array}
\right)
= B \cdot D \cdot
\left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right)
$

```python
def row_index_to_i_j_grad(i_row):
    return i_row // 3, i_row % 3

B = zeros(9, 12)
B[0,1] = S(1)
B[1,2] = S(1)

B[2,3] = S(1)

B[3,5] = S(1)
B[4,6] = S(1)
B[5,7] = S(1)

B[6,9] = S(1)
B[7,10] = S(1)
B[8,11] = S(1)

for row_index in range(9):
    i,j=row_index_to_i_j_grad(row_index)
    B[row_index, 0] = -GK[i,j,0]
    B[row_index, 4] = -GK[i,j,1]
    B[row_index, 8] = -GK[i,j,2]

```

### Strain tensor

$
\left(
\begin{array}{c}
\varepsilon_{11} \\
\varepsilon_{22} \\
\varepsilon_{33} \\
2\varepsilon_{12} \\
2\varepsilon_{13} \\
2\varepsilon_{23} \\
\end{array}
\right)
=
\left(E + E_{NL} \left( \nabla \vec{u} \right) \right) \cdot
\left(
\begin{array}{c}
\nabla_1 u_1 \\ \nabla_2 u_1 \\ \nabla_3 u_1 \\
\nabla_1 u_2 \\ \nabla_2 u_2 \\ \nabla_3 u_2 \\
\nabla_1 u_3 \\ \nabla_2 u_3 \\ \nabla_3 u_3 \\
\end{array}
\right)$

```python
E=zeros(6,9)
E[0,0]=1
E[1,4]=1
E[2,8]=1
E[3,1]=1
E[3,3]=1
E[4,2]=1
E[4,6]=1
E[5,5]=1
E[5,7]=1
E
```

```python
def E_NonLinear(grad_u):
    N = 3

    du = zeros(N, N)

    #    print("===Deformations===")

    for i in range(N):
        for j in range(N):
            index = i*N+j
            du[j,i] = grad_u[index]

    #    print("========")

    I = eye(3)

    a_values = S(1)/S(2) * du * G_up

    E_NL = zeros(6,9)
    E_NL[0,0] = a_values[0,0]
    E_NL[0,3] = a_values[0,1]
    E_NL[0,6] = a_values[0,2]

    E_NL[1,1] = a_values[1,0]
    E_NL[1,4] = a_values[1,1]
    E_NL[1,7] = a_values[1,2]

    E_NL[2,2] = a_values[2,0]
    E_NL[2,5] = a_values[2,1]
    E_NL[2,8] = a_values[2,2]

    E_NL[3,1] = 2*a_values[0,0]
    E_NL[3,4] = 2*a_values[0,1]
    E_NL[3,7] = 2*a_values[0,2]

    E_NL[4,0] = 2*a_values[2,0]
    E_NL[4,3] = 2*a_values[2,1]
    E_NL[4,6] = 2*a_values[2,2]

    E_NL[5,2] = 2*a_values[1,0]
    E_NL[5,5] = 2*a_values[1,1]
    E_NL[5,8] = 2*a_values[1,2]

    return E_NL

%aimport geom_util
u=getUHat3DPlane(alpha1, alpha2, alpha3)

# u=getUHatU3Main(alpha1, alpha2, alpha3)

gradu=B*u

E_NL = E_NonLinear(gradu)*B

```

### Physical coordinates

$u_i=u_{[i]} H_i$

```python
P=zeros(12,12)
P[0,0]=H[0]
P[1,0]=dH[0,0]
P[1,1]=H[0]
P[2,0]=dH[0,1]
P[2,2]=H[0]
P[3,0]=dH[0,2]
P[3,3]=H[0]

P[4,4]=H[1]
P[5,4]=dH[1,0]
P[5,5]=H[1]
P[6,4]=dH[1,1]
P[6,6]=H[1]
P[7,4]=dH[1,2]
P[7,7]=H[1]

P[8,8]=H[2]
P[9,8]=dH[2,0]
P[9,9]=H[2]
P[10,8]=dH[2,1]
P[10,10]=H[2]
P[11,8]=dH[2,2]
P[11,11]=H[2]
P=simplify(P)
P
```

```python
B_P = zeros(9,9)

for i in range(3):
    for j in range(3):

        row_index = i*3+j

        B_P[row_index, row_index] = 1/(H[i]*H[j])

Grad_U_P = simplify(B_P*B*P)
Grad_U_P
```

```python
StrainL=simplify(E*Grad_U_P)
StrainL
```

```python
%aimport geom_util

u=getUHatU3Main(alpha1, alpha2, alpha3)

gradup=Grad_U_P*u

E_NLp = E_NonLinear(gradup)*Grad_U_P

simplify(E_NLp)
```

### Tymoshenko theory

$u_1 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u\left( \alpha_1 \right)+\alpha_3\gamma \left( \alpha_1 \right) $

$u_2 \left( \alpha_1, \alpha_2, \alpha_3 \right)=0 $

$u_3 \left( \alpha_1, \alpha_2, \alpha_3 \right)=w\left( \alpha_1 \right) $

$ \left(
\begin{array}{c}
u_1 \\
\frac { \partial u_1 } { \partial \alpha_1} \\
\frac { \partial u_1 } { \partial \alpha_2} \\
\frac { \partial u_1 } { \partial \alpha_3} \\
u_2 \\
\frac { \partial u_2 } { \partial \alpha_1} \\
\frac { \partial u_2 } { \partial \alpha_2} \\
\frac { \partial u_2 } { \partial \alpha_3} \\
u_3 \\
\frac { \partial u_3 } { \partial \alpha_1} \\
\frac { \partial u_3 } { \partial \alpha_2} \\
\frac { \partial u_3 } { \partial \alpha_3} \\
\end{array}
\right) = T \cdot
\left(
\begin{array}{c}
u \\
\frac { \partial u } { \partial \alpha_1} \\
\gamma \\
\frac { \partial \gamma } { \partial \alpha_1} \\
w \\
\frac { \partial w } { \partial \alpha_1} \\
\end{array}
\right) $

```python
T=zeros(12,6)
T[0,0]=1
T[0,2]=alpha3
T[1,1]=1
T[1,3]=alpha3
T[3,2]=1

T[8,4]=1
T[9,5]=1
T
```

```python
D_p_T = StrainL*T
simplify(D_p_T)
```

```python
u = Function("u")
t = Function("theta")
w = Function("w")

u1=u(alpha1)+alpha3*t(alpha1)
u3=w(alpha1)

gu = zeros(12,1)
gu[0] = u1
gu[1] = u1.diff(alpha1)
gu[3] = u1.diff(alpha3)

gu[8] = u3
gu[9] = u3.diff(alpha1)

gradup=Grad_U_P*gu

# o20=(K*u(alpha1)-w(alpha1).diff(alpha1)+t(alpha1))/2
# o21=K*t(alpha1)
# O=1/2*o20*o20+alpha3*o20*o21-alpha3*K/2*o20*o20
# O=expand(O)
# O=collect(O,alpha3)
# simplify(O)

StrainNL = E_NonLinear(gradup)*gradup
simplify(StrainNL)
```

### Square theory

$u^1 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u_{10}\left( \alpha_1 \right)p_0\left( \alpha_3 \right)+u_{11}\left( \alpha_1 \right)p_1\left( \alpha_3 \right)+u_{12}\left( \alpha_1 \right)p_2\left( \alpha_3 \right) $

$u^2 \left( \alpha_1, \alpha_2, \alpha_3 \right)=0 $

$u^3 \left( \alpha_1, \alpha_2, \alpha_3 \right)=u_{30}\left( \alpha_1 \right)p_0\left( \alpha_3 \right)+u_{31}\left( \alpha_1 \right)p_1\left( \alpha_3 \right)+u_{32}\left( \alpha_1 \right)p_2\left( \alpha_3 \right) $

$ \left(
\begin{array}{c}
u^1 \\
\frac { \partial u^1 } { \partial \alpha_1} \\
\frac { \partial u^1 } { \partial \alpha_2} \\
\frac { \partial u^1 } { \partial \alpha_3} \\
u^2 \\
\frac { \partial u^2 } { \partial \alpha_1} \\
\frac { \partial u^2 } { \partial \alpha_2} \\
\frac { \partial u^2 } { \partial \alpha_3} \\
u^3 \\
\frac { \partial u^3 } { \partial \alpha_1} \\
\frac { \partial u^3 } { \partial \alpha_2} \\
\frac { \partial u^3 } { \partial \alpha_3} \\
\end{array}
\right) = L \cdot
\left(
\begin{array}{c}
u_{10} \\
\frac { \partial u_{10} } { \partial \alpha_1} \\
u_{11} \\
\frac { \partial u_{11} } { \partial \alpha_1} \\
u_{12} \\
\frac { \partial u_{12} } { \partial \alpha_1} \\
u_{30} \\
\frac { \partial u_{30} } { \partial \alpha_1} \\
u_{31} \\
\frac { \partial u_{31} } { \partial \alpha_1} \\
u_{32} \\
\frac { \partial u_{32} } { \partial \alpha_1} \\
\end{array}
\right) $

```python
L=zeros(12,12)
h=Symbol('h')
p0=1/2-alpha3/h
p1=1/2+alpha3/h
p2=1-(2*alpha3/h)**2

L[0,0]=p0
L[0,2]=p1
L[0,4]=p2

L[1,1]=p0
L[1,3]=p1
L[1,5]=p2

L[3,0]=p0.diff(alpha3)
L[3,2]=p1.diff(alpha3)
L[3,4]=p2.diff(alpha3)

L[8,6]=p0
L[8,8]=p1
L[8,10]=p2

L[9,7]=p0
L[9,9]=p1
L[9,11]=p2

L[11,6]=p0.diff(alpha3)
L[11,8]=p1.diff(alpha3)
L[11,10]=p2.diff(alpha3)

L
```

```python
D_p_L = StrainL*L
simplify(D_p_L)
```

```python
h = 0.5
exp=(0.5-alpha3/h)*(1-(2*alpha3/h)**2)#/(1+alpha3*0.8)
p02=integrate(exp, (alpha3, -h/2, h/2))
integral = expand(simplify(p02))
integral
```

## Mass matrix

```python
rho=Symbol('rho')
B_h=zeros(3,12)
B_h[0,0]=1
B_h[1,4]=1
B_h[2,8]=1
M=simplify(rho*P.T*B_h.T*G_up*B_h*P)
M
```

```python
M_p = L.T*M*L*(1+alpha3/R)
mass_matr = simplify(integrate(M_p, (alpha3, -h/2, h/2)))
mass_matr
```


```
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```
#export
from exp.nb_06 import *
```

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=5899)

```
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```
nfs = [8,16,32,64,64]
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```
%time run.fit(2, learn)
```

    train: [1.48223671875, tensor(0.5053)]
    valid: [0.343594189453125, tensor(0.8879)]
    train: [0.232376171875, tensor(0.9287)]
    valid: [0.1355825439453125, tensor(0.9573)]
    CPU times: user 34.9 s, sys: 6.8 s, total: 41.7 s
    Wall time: 16.7 s

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6018)

```
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.ones(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
        if self.training:
            with torch.no_grad(): m,v = self.update_stats(x)
        else: m,v = self.means,self.vars
        x = (x-m) / (v+self.eps).sqrt()
        return x*self.mults + self.adds
```

```
a = torch.randn(3,3)
```

```
class SimpleBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom, self.eps = mom, eps
        self.variance = nn.Parameter(torch.ones(nf, 1, 1))
        self.mean = nn.Parameter(torch.zeros(nf, 1, 1))
        self.register_buffer('vars', torch.ones(1, nf, 1, 1))
        self.register_buffer('means', torch.zeros(1, nf, 1, 1))
        self.iter = 0

    def update_stats(self, x):
        m = x.mean((0, 2, 3), keepdim=True)
        v = x.var ((0, 2, 3), keepdim=True)
        return m ,v

    def forward(self, x):
#         if self.training:
        if self.iter > 2: # Taking only 10% Gradient to avoid explosions
            with torch.no_grad():
                self.mean.data    = self.means.data +     ((self.mean - self.means) * 0.1).data
                self.variance.data= self.vars.data + ((self.variance - self.vars) * 0.1).data
        if self.iter < 3:
            self.iter += 1
            with torch.no_grad():
                m, v = self.update_stats(x)
            self.mean.data = m.data
            self.variance.data = v.data
        x = (x - self.mean) / (self.variance + self.eps).sqrt()
        self.vars = self.variance.clone().detach()
        self.means = self.mean.clone().detach()

        return x
```

```
torch.tan
```

```
class TanSigmoidBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         self.register_buffer('vars',  torch.ones(1,nf,1,1))
#         self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
#         if self.training:
#             with torch.no_grad(): m,v = self.update_stats(x)
#         else: m,v = self.means,self.vars
#         x = (x-m) / (v+self.eps).sqrt()
#         x = torch.exp(-x)
        x = torch.sigmoid(x) * 2 - 1
        return x*self.mults + self.adds
```

```
class SigmoidBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         self.register_buffer('vars',  torch.ones(1,nf,1,1))
#         self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
#         if self.training:
#             with torch.no_grad(): m,v = self.update_stats(x)
#         else: m,v = self.means,self.vars
#         x = (x-m) / (v+self.eps).sqrt()
#         x = torch.exp(-x)
        x = torch.sigmoid(x)
        return x*self.mults + self.adds
```

```
class TanhBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         self.register_buffer('vars',  torch.ones(1,nf,1,1))
#         self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
#         if self.training:
#             with torch.no_grad(): m,v = self.update_stats(x)
#         else: m,v = self.means,self.vars
#         x = (x-m) / (v+self.eps).sqrt()
#         x = torch.exp(-x)
        x = torch.tanh(x)
        return x*self.mults + self.adds
```

```
class GaussianBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         self.register_buffer('vars',  torch.ones(1,nf,1,1))
#         self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
#         if self.training:
#             with torch.no_grad(): m,v = self.update_stats(x)
#         else: m,v = self.means,self.vars
#         x = (x-m) / (v+self.eps).sqrt()
#         x = torch.exp(-x)
        x = torch.exp(-x)
        return x*self.mults + self.adds
```

```
class TanhNOBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         self.register_buffer('vars',  torch.ones(1,nf,1,1))
#         self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
#         if self.training:
#             with torch.no_grad(): m,v = self.update_stats(x)
#         else: m,v = self.means,self.vars
#         x = (x-m) / (v+self.eps).sqrt()
#         x = torch.exp(-x)
        x = torch.exp(-x)
        return x
#         return x*self.mults + self.adds
```

```
a = torch.randn(3, 3, 3, 3)
```

```
torch.max(a)
```

    tensor(2.7635)

```
class MaxBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
#         self.register_buffer('vars',  torch.ones(1,nf,1,1))
#         self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
#         if self.training:
#             with torch.no_grad(): m,v = self.update_stats(x)
#         else: m,v = self.means,self.vars
#         x = (x-m) / (v+self.eps).sqrt()
#         x = torch.exp(-x)
        x = x / x.max()
        return x * self.mults + self.adds
#         return x*self.mults + self.adds
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(SimpleBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(GaussianBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=False, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(GaussianBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(SigmoidBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(TanSigmoidBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(TanhBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(TanhNOBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(MaxBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None: m.bias.data.zero_()
    for l in m.children(): init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

```
with Hooks(learn.model, append_stats) as hooks:
    run.fit(5, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```
learn.model[0][2].mults, learn.model[0][2].adds
```

    (Parameter containing:
     tensor([[[1.2080]],

             [[1.0936]],

             [[1.1243]],

             [[0.7734]],

             [[1.1104]],

             [[0.8719]],

             [[1.1091]],

             [[1.0684]]], requires_grad=True), Parameter containing:
     tensor([[[-0.2667]],

             [[-0.3311]],

             [[-0.2167]],

             [[-0.0838]],

             [[-0.2540]],

             [[-0.1945]],

             [[-0.3655]],

             [[-0.0923]]], requires_grad=True))

```
with Hooks(learn.model, append_stats) as hooks:
    run.fit(5, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```
learn,run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

lets measure the speed difference in taking the mean and variance of a tensor and taking tanh

```
a = torch.randn(64, 3, 28, 28)
```

```
%time
for i in range(100):
    a.mean(), a.var()
```

    CPU times: user 3 µs, sys: 1e+03 ns, total: 4 µs
    Wall time: 5.96 µs

```
%time
for i in range(100):
    a = torch.tanh(a)
```

    CPU times: user 2 µs, sys: 0 ns, total: 2 µs
    Wall time: 6.91 µs

```
%time run.fit(3, learn)
```

    train: [0.27833810546875, tensor(0.9105, device='cuda:0')]
    valid: [0.1912491943359375, tensor(0.9386, device='cuda:0')]
    train: [0.08977265625, tensor(0.9713, device='cuda:0')]
    valid: [0.09156090698242188, tensor(0.9716, device='cuda:0')]
    train: [0.06145498046875, tensor(0.9810, device='cuda:0')]
    valid: [0.09970919799804688, tensor(0.9707, device='cuda:0')]
    CPU times: user 3.71 s, sys: 584 ms, total: 4.29 s
    Wall time: 4.29 s

### Builtin batchnorm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6679)

```
#export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [0.27115255859375, tensor(0.9206, device='cuda:0')]
    valid: [0.1547997314453125, tensor(0.9496, device='cuda:0')]
    train: [0.07861462890625, tensor(0.9755, device='cuda:0')]
    valid: [0.07472044067382813, tensor(0.9776, device='cuda:0')]
    train: [0.0570835498046875, tensor(0.9818, device='cuda:0')]
    valid: [0.0673232666015625, tensor(0.9813, device='cuda:0')]
    CPU times: user 3.37 s, sys: 747 ms, total: 4.12 s
    Wall time: 4.12 s

### With scheduler

Now let's add the usual warm-up/annealing.

```
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched)])
```

```
run.fit(8, learn)
```

    train: [0.2881914453125, tensor(0.9116, device='cuda:0')]
    valid: [0.5269224609375, tensor(0.8394, device='cuda:0')]
    train: [0.153792421875, tensor(0.9551, device='cuda:0')]
    valid: [0.462953662109375, tensor(0.8514, device='cuda:0')]
    train: [0.087637158203125, tensor(0.9736, device='cuda:0')]
    valid: [0.07029829711914062, tensor(0.9788, device='cuda:0')]
    train: [0.049282421875, tensor(0.9853, device='cuda:0')]
    valid: [0.08710025024414063, tensor(0.9740, device='cuda:0')]
    train: [0.0356986328125, tensor(0.9888, device='cuda:0')]
    valid: [0.07853966674804687, tensor(0.9773, device='cuda:0')]
    train: [0.0268300439453125, tensor(0.9918, device='cuda:0')]
    valid: [0.04807376098632812, tensor(0.9870, device='cuda:0')]
    train: [0.0219412109375, tensor(0.9939, device='cuda:0')]
    valid: [0.04363873901367187, tensor(0.9882, device='cuda:0')]
    train: [0.018501048583984374, tensor(0.9951, device='cuda:0')]
    valid: [0.04355916137695313, tensor(0.9877, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6717)

```
class LayerNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    def forward(self, x):
        m = x.mean((1,2,3), keepdim=True)
        v = x.var ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [nan, tensor(0.1321, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 4.56 s, sys: 862 ms, total: 5.42 s
    Wall time: 5.42 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7114)

```
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
        m = x.mean((2,3), keepdim=True)
        v = x.var ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

```
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 4.46 s, sys: 718 ms, total: 5.18 s
    Wall time: 5.18 s

*Question*: why can't this classify anything?

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

### Group norm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7213)

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7304)

```
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```
%time run.fit(1, learn)
```

    train: [2.35619296875, tensor(0.1649, device='cuda:0')]
    valid: [2867.7198, tensor(0.2604, device='cuda:0')]
    CPU times: user 1min 32s, sys: 835 ms, total: 1min 33s
    Wall time: 1min 33s

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7516)

```
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel()/nc)
        mom1 = 1 - (1-self.mom)/math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias*(1-self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step<100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c    = c    / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

```
learn,run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

```
%time run.fit(1, learn)
```

    train: [0.157932021484375, tensor(0.9511, device='cuda:0')]
    valid: [0.0986408935546875, tensor(0.9730, device='cuda:0')]
    CPU times: user 16.5 s, sys: 1.57 s, total: 18.1 s
    Wall time: 18.1 s

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=8068)

```
data = DataBunch(*get_dls(train_ds, valid_ds, 32), c)
```

```
learn,run = get_learn_run(nfs, data, 0.9, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

```
%time run.fit(1, learn)
```

    train: [0.1573110546875, tensor(0.9521, device='cuda:0')]
    valid: [0.09242745971679688, tensor(0.9818, device='cuda:0')]
    CPU times: user 16.6 s, sys: 1.52 s, total: 18.1 s
    Wall time: 18.2 s

## Export

```
nb_auto_export()
```

    <IPython.core.display.Javascript object>

```

```