(c) Juan Gomez 2019. Thanks to Universidad EAFIT for support. This material is part of the course Introduction to Finite Element Analysis

# Introduction to notebooks

A **Notebook (NB)** is a digital document that combines computer code with enriched text elements such as text, equations, figures, videos, etc. Tipically a NB can be easily read allowing to deliver theoretical information together with executable code snippets that can be tested interactively.

In this introductory Finite Element Method (FEM) course we will use notebooks as support material within the flipped class methodology. In that context NBs will allow the instructor to maintain the flow of the discussion treating the theoretical aspects in a condensed form and at the same time vary problem parameters, data and analysis techniques in an interactive fashion while establishing a direct connection between the mathematical respresentation in terms of equations and computer codes. **After completing this notebook you should be able to:**

* Create independent notebooks and modify existing ones.

* Represent simple numerical tasks in terms of computer algorithms.

* Understand basic programming structures and its Python representation.

## Text and code

In a NB it is possible to combine text and code to describe a subject and conduct verifications, for instance, through parametric analyses.

Equations in a NB can be written using [$\LaTeX$](https://katex.org/) like:

$$f(x) = x^3+4x^2-10.$$

It is also possible to insert images which are available in memory **(double click in the image to see the used command)**

## Control structures

In order to show how code snippets are introduced in NBs and used in Python we will describe the basic data flow structures used in computer programming. We will use combinations of text describing the structure and its corresponding Python implementation.

The first block of code in every Python program imports (or activates) modules or **libraries** which are sets of subroutines to perform a family of specific taks. For instance, in the following case we import the modulus **numpy** and **matplotlib**.

**Questions:**

*** Find out what type of computing tasks are performed by the subroutines in the module numpy.**

***Find out what type of computing tasks are performed by the subroutines in the module matplotlib.**

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import Image
```

### Sequential structure

The most simple algorithmic form corresponds to a sequential structure where a series of consecutive commandsexecute one after the other (see figure).

The sequential structure is illustrated by the following code where we compute the average of two numbers.

```python
A = 45
B = 20
sum = A + B
prom = sum/2
print(prom)
```

    32.5

### Simple If

As shown in the figure in this structure a group of sequntial commands is executed only if the condition specified by **condition** is true.

In the following code snippet we ask the user to input the radious of a circle. The code computes the resulting area provided that the value given by the user is in fact greater than 0.0.

```python
R = 1.0
if R > 0.0:
    Area = np.pi*R**2
print(Area)
```

    3.141592653589793

### Double If

The double if includes an alternative path for the flow in case **condition** is false.

This structure is explained in the following code where we check if a number is odd or even using the **%** operator.

**What are the arithmetic operators in Python?**

```python
num = 4
if (num % 2) == 0:
    print('The number is even')
else:
    print('The number is odd')
```

    The number is even

### While loop.

In this structure a sequence of commands is repeteadly executed as long as the condition specified by **condition** is true. The execution stops once the condition changes to false. For the structure to execute at least once the condition specified by **condition** must be true right before the beginning and to avoid the structure executing indefinetly the possibility for the condition changing to false must exist within the structure.

In the following block the integer variable *icount* is initially set to zero and the loop executes as long as *icount* is smaller than 5. After each iterattion *icount* is incremented by 1 until it reaches 5 which sets **condition** to false.

```python
icount = 0
while icount < 5:
    icount = icount + 1
print(icount)
```

    5

### Until loop.

This structure is similar to the **While** stucture but now **condition** is evaluated at the end of the loop.

**Find out what is the form (if any) of the *Until* structure in Python**

### For loop.

This structure executes a sequential list of commands a finite number of times according to a counter which is controlled by a prescribed limit.

**Find out the specific use of the *For* structure in Python**

```python
for i in range(5):
    print('Tha value of the counter is', i)
```

    Tha value of the counter is 0
    Tha value of the counter is 1
    Tha value of the counter is 2
    Tha value of the counter is 3
    Tha value of the counter is 4

## Defining and plotting a function

In the following block of code we show how to define and plot a function. To declare the function we will use the intrinsic Python option **lambda** which allows us to declare the function and leave it ready to perform a mapping between numbers pretty much like in standard calculus. Once the function is defined we will use the **matplotlib** function **plot()**.

```python
def fx(x):
    return x**3+4.0*x**2-10.0

npts = 200
yy = np.zeros((npts))
xx = np.linspace(-1, 1, npts)
yy[:] = fx(xx[:])
plt.figure(0)
plt.plot(xx, yy, 'r--')
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x247256c9a20>]

## Inserting a Video

The Python function **YouTubeVideo** allows insertig youtube videos in Notebooks as shown in the following code snippet.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('ibeIKaSYG2U')
```

### Glossary of terms

**Flow structure:** In programming a flow structure is a logical statement that chooses a code path based on values of prescribed parameters.

**Loop:** A flow structure that executes a sequence of commands a prescribed or variable number of times.

**Counter:** This is the typical name given to an integer variable that is used to count events in a program.

**Flag:** An integer variable that usually takes the value of $0$ or $1$ and is used to evaluate a condition in a block of code.

## Class activity

In the flipped classroom format of this course most of the work is developed during the class where the students perform activities under the orientation of the instructor. Many of these activities involve either completing specific parts of a notebook or creating a new notebook from scratch.

In this activity the students are required to find and plot the first derivative of the function:

$$
f(x) = x^3+4x^2-10
$$

in the interval $[-1 , 1]$ using Python explicit diffrentiation.

**Add comments to explain each one of the used commands.**

```python
from sympy import *
zz = np.zeros((npts))
x = symbols('x')
N = x**3+4.0*x**2-10.0
fdx = diff(N, x)
for i in range(npts):
    zz[i] = fdx.subs([(x, xx[i])])
```

```python
print(fdx)
```

    3*x**2 + 8.0*x

```python
plt.figure(1)
plt.plot(xx, zz, 'r--')
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x2116f5f8080>]

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


```
import scipy as sc
import numpy as np
from sympy import pprint , Matrix
from scipy.optimize import minimize
from numpy import exp ,cos ,sin ,pi ,dot ,abs
from numpy.random import random_sample
```

```
X=np.array([[0,1],[1,0]])
I=np.array([[1,0],[0,1]])

up=np.array([[1,0],[0,0]])
down=np.array([[0,0],[0,1]])

Z=np.array([[1,0],[0,-1]])

```

```
def u1(x):
  return np.array([[1,0],[0,exp(1j * x)]])

def u3(x,y,z):
  return np.array([[cos(x/2),-exp(1j*z)*sin(x/2)],[exp(1j*y)*sin(x/2),exp((1j*z) +(1j*y))*cos(x/2)]])
```

```
q0=np.array([[1],[0]])
q1=q0
```

```
Q=np.kron(q0,q1)
```

```
def count(values, probabilities, size):
    bins = np.add.accumulate(probabilities)
    return values[np.digitize(random_sample(size), bins)]

```

```
def Z1(theta):
  U1=np.kron(u1(theta[0]),I)
  U2=np.kron(u3(theta[1],-pi/2,pi/2),I)
  U3=np.kron(u1(theta[2]),I)
  U4=np.kron(up,I)+np.kron(down,X)
  U5=np.kron(I,u1(theta[3]))
  U6=np.kron(I,u3(theta[4],-pi/2,pi/2))
  U7=np.kron(I,u1(theta[5]))
  U8=np.kron(I,up)+np.kron(X,down)
  U9=np.kron(u1(theta[6]),I)
  U10=np.kron(I,u1(theta[7]))
  U11=np.kron(u3(theta[8],-pi/2,pi/2),I)
  U12=np.kron(I,u3(theta[9],-pi/2,pi/2))
  U13=np.kron(u1(theta[10]),I)
  U14=np.kron(I,u1(theta[11]))
  U15=np.kron(Z,I)

  U=[U1,U2,U3,U4,U5,U6,U7,U8,U9,U10,U11,U12,U13,U14,U15]
  r=Q
  for i in U:
    r=dot(i,r)

  val=np.array([1,2,3,4])
  prob=np.array([abs(r[0,0])**2,abs(r[1,0])**2,abs(r[2,0])**2,abs(r[3,0])**2])

  shot=8192
  counts=count(val,prob,shot)

  p00=0
  p01=0
  p10=0
  p11=0

  for i in counts:
    if i==1:
      p00+=1

    if i==2:
      p01+=1

    if i==3:
      p10+=1

    if i==4:
      p11+=1

  z0=(p00/shot)+(p01/shot)-(p10/shot)-(p11/shot)

  return z0

```

```
theta0=[0,pi/2,0,0,pi/2,0,0,0,pi/2,pi/2,0,0]
```

```
resultz0=[]
for i in range(1,120):
  res=minimize(Z1,theta0,method='cobyla',options={'maxiter' : i})
  resultz0.append(res.x)
```

```
theta1=resultz0[len(resultz0)-1]
```

```
expz0=[]
for i in resultz0:
  expz0.append(Z1(i))
```

```
def Z2(theta):
  U1=np.kron(u1(theta[0]),I)
  U2=np.kron(u3(theta[1],-pi/2,pi/2),I)
  U3=np.kron(u1(theta[2]),I)
  U4=np.kron(up,I)+np.kron(down,X)
  U5=np.kron(I,u1(theta[3]))
  U6=np.kron(I,u3(theta[4],-pi/2,pi/2))
  U7=np.kron(I,u1(theta[5]))
  U8=np.kron(I,up)+np.kron(X,down)
  U9=np.kron(u1(theta[6]),I)
  U10=np.kron(I,u1(theta[7]))
  U11=np.kron(u3(theta[8],-pi/2,pi/2),I)
  U12=np.kron(I,u3(theta[9],-pi/2,pi/2))
  U13=np.kron(u1(theta[10]),I)
  U14=np.kron(I,u1(theta[11]))
  U15=np.kron(I,Z)

  U=[U1,U2,U3,U4,U5,U6,U7,U8,U9,U10,U11,U12,U13,U14,U15]
  r=Q
  for i in U:
    r=dot(i,r)

  val=np.array([1,2,3,4])
  prob=np.array([abs(r[0,0])**2,abs(r[1,0])**2,abs(r[2,0])**2,abs(r[3,0])**2])

  shot=8192
  counts=count(val,prob,shot)

  p00=0
  p01=0
  p10=0
  p11=0

  for i in counts:
    if i==1:
      p00+=1

    if i==2:
      p01+=1

    if i==3:
      p10+=1

    if i==4:
      p11+=1

  z1=(p00/shot)-(p01/shot)+(p10/shot)-(p11/shot)

  return z1
```

```
resultz1=[]
for i in range(1,120):
  res=minimize(Z2,theta1,method='cobyla',options={'maxiter' : i})
  resultz1.append(res.x)
```

```
theta2=resultz1[len(resultz1)-1]
```

```
expz1=[]
for i in resultz1:
  expz1.append(Z2(i))
```

```
def Z3(theta):
  U1=np.kron(u1(theta[0]),I)
  U2=np.kron(u3(theta[1],-pi/2,pi/2),I)
  U3=np.kron(u1(theta[2]),I)
  U4=np.kron(up,I)+np.kron(down,X)
  U5=np.kron(I,u1(theta[3]))
  U6=np.kron(I,u3(theta[4],-pi/2,pi/2))
  U7=np.kron(I,u1(theta[5]))
  U8=np.kron(I,up)+np.kron(X,down)
  U9=np.kron(u1(theta[6]),I)
  U10=np.kron(I,u1(theta[7]))
  U11=np.kron(u3(theta[8],-pi/2,pi/2),I)
  U12=np.kron(I,u3(theta[9],-pi/2,pi/2))
  U13=np.kron(u1(theta[10]),I)
  U14=np.kron(I,u1(theta[11]))
  U15=np.kron(Z,Z)

  U=[U1,U2,U3,U4,U5,U6,U7,U8,U9,U10,U11,U12,U13,U14,U15]
  r=Q
  for i in U:
    r=dot(i,r)

  val=np.array([1,2,3,4])
  prob=np.array([abs(r[0,0])**2,abs(r[1,0])**2,abs(r[2,0])**2,abs(r[3,0])**2])

  shot=8192
  counts=count(val,prob,shot)

  p00=0
  p01=0
  p10=0
  p11=0

  for i in counts:
    if i==1:
      p00+=1

    if i==2:
      p01+=1

    if i==3:
      p10+=1

    if i==4:
      p11+=1

  z2=(p00/shot)-(p01/shot)-(p10/shot)+(p11/shot)

  return z2
```

```
resultz2=[]
for i in range(1,120):
  res=minimize(Z3,theta2,method='cobyla',options={'maxiter' : i})
  resultz2.append(res.x)
```

```
expz2=[]
for i in resultz2:
  expz2.append(Z3(i))
```

```
expH=[]
for i in range(0,119):
  h=((1/4)*(1-expz0[i]))+((1/4)*(1-expz1[i]))+((5/64)*(1-expz0[i]-expz1[i]+expz2[i]))
  expH.append(h)
```

```
y=[]
for i in expH:
  y.append(-i)
```

```
y[len(y)-1]
```

    -1.1553802490234375

```
import matplotlib.pyplot as plt
```

```
x=np.arange(1,120)
```

```
g=[]
for i in range(0,119):
  g.append(-0.52952)

```

```
plt.plot(x,y)
plt.plot(g)
plt.xlabel("Iterations")
plt.ylabel("Energy(hartree)")
```

```

```


# Introduction

Since Newton invented calculus, differentiating a function has been essential to the advancement of humanity. Calculating the derivative of a function is crucial to finding the extrema for a function and determining zeros for a function, two operations that are central to optimization (1). Often, we can find the symbolic/analytical solution to the derivative of a function, however this has become increasingly complex and computationally expensive as our functions/equations have grown in size and complexity. Numerically solving differential equations forms a cornerstone of modern science and engineering and is intimately linked with machine learning; however this method suffers from rounding errors and numerical instability. Many of these issues can be solved using Automatic Differentiation (AD) because AD can calculate the exact derivative up to machine precision (2). The logic and processes behind AD enables it to be implemented using computer code, making it easily accessible for use by scientists and mathematicians. This python package will implement the forward mode of AD.

# How to use the package

### Installation

Ideally, the user should have Anaconda installed (https://www.anaconda.com/download/).

To install the package, a user will download or clone the repository onto their local machine. Then, the user can, but does not need to, create a virtual environment using virtualenv. If the user has Anaconda installed, then the user will already have virtualenv installed, but if they do not, the user can the user can run the following command in the terminal:

    sudo easy_install virtualenv
Then, to create a virtual environment, the user runs the following command in the terminal in the package directory:

    virtualenv env
To activate the virtual environment, the user runs the following command in the terminal in the package directory:

    source env/bin/activate
To deactivate the virtual environment, the user runs the following command in the terminal in the package directory:

    deactivate
Whether or not the user uses a virtual environment, they will need to make sure they have the necessary dependencies. In the package directory includes a requirements.txt file that lists the necessary dependencies. The user can easily install all the dependencies with the following command:

    pip install -r requirements.txt
After installing the necessary dependencies, the user can then start using the package in their code with an import statement such as

    import autodiff as ad
To run the tests for the package, the user can run the following command in terminal in the package folder:

    pytest

### Usage

For example, suppose $f(x, y) = x^2 + 2xy + y^2$, and $x = 1, y = 2$.

$
f(1, 2) = 9\\
\frac{\partial f}{\partial x} = 2x + 2y \implies \frac{\partial f}{\partial x}|_{x = 1, y = 2} = 6 \\
\frac{\partial f}{\partial y} = 2x + 2y \implies \frac{\partial f}{\partial y}|_{x = 1, y = 2} = 6
$

To solve this problem using our package, the user will run the following straightforward code.

```python
% cd ../
#Ignore this. Used to import autodiff in the docs folder.

```

    /Users/blin/Documents/cs207/cs207-FinalProject

```python
import autodiff as ad

# Define x and y scalars with variable name and value. The actual python variable names do not matter.
x = ad.Scalar('x', 1)
y = ad.Scalar('y', 2)
# Express f in terms of x and y. User does not need to define any more scalar objects besides the basic ones x and y above.
f = x ** 2 + 2 * x * y + y ** 2
```

```python
# Get value of f(1, 2)
f.getValue()
```

    9.0

```python
# Get partial derivative with respect to x. Pass in variable names as String in list.
f.getGradient(['x'])
```

    array([6.])

```python
# Get partial derivative with respect to y
f.getGradient(['y'])
```

    array([6.])

```python
# Get both partial derivatives
f.getGradient(['x', 'y'])
```

    array([6., 6.])

```python
f.getDeriv()
```

    {'x': 6.0, 'y': 6.0}

With our package, users will be able to easily define functions that rely on variables and their derivatives. For example, consider Newton's method in finding roots: For a scalar function $f(x)$, we want to find the $x$ such that $f(x) = 0$. Newton's method states that we are able to find the root by updating from an intial `x_0` value with the update equation $x_{n} = x_{n-1} - \frac{f(x_{n-1})}{f^{\prime}(x_{n-1})} $ until we get an $x_n$ such that $f(x_n) \approx 0$. With our package, the method can be written as the following:

```python
def newton_method(fn, init_x = 0, threshold = 1e-10):
    # Initialize scalar with an init value
    x = ad.Scalar('x', init_x)
    y = fn(x)
    # While f(x) is greater than threshold, update x.
    while y.getValue() > threshold:
        x = x - y.getValue() / y.getGradient('x')
        y = fn(x)
    # Return x such that f(x) approximately equals 0.
    return x.getValue()
```

# Background

The following mathematical concepts and background are required for understanding automatic differentiation:

### 1. Differential calculus

Differential calculus is a subfield of calculus concerned with the study of the rates at which quantities change.
For example, given the function:
\begin{align}
 f\left(x\right) &=  {x^{2}}
 \end{align}

 Increment x by h:
 \begin{align}
 f\left(x+h\right) &=  {(x+h)^{2}}
 \end{align}

 Apply the finite difference approximation to calculate the slope:
  \begin{align}
 \frac{f\left(x+h\right) - f\left(x\right) }{h}
 \end{align}

Simplify the equation:
  \begin{align}
 &= \frac{x^{2}+2xh+h^{2}-x^{2} }{h}\\
 &= \frac{2xh+h^{2}}{h}\\
 &=2x+h
 \end{align}

 Set $h\rightarrow 0$:
   \begin{align}
 2x +0 &= 2x
  \end{align}

The derivative is then defined is:
\begin{align}
 \lim_{h\to0} \frac{f\left(x+h\right) - f\left(x\right) }{h}
 \end{align}

### 2. Elementary functions and their derivatives

|       Function $f(x)$                |       Derivative $f^{\prime}(x)$                |
| :-------------------:  | :------------------------------------------------------------------------------:  |
| ${c}$           | $0$         |
| ${x}$           | $1$         |
| ${x^{n}}$           | ${nx^{n-1}}$         |
| $\frac{1}{x}$     | $\frac{-1}{x^{2}}$     |
| $ln{x}$     | $\frac{1}{x}$     |
| $\sin(x)$           |   $\cos(x)$         |
| $\cos(x)$           |   $-\sin(x)$         |
| $\tan(x)$           |   $\dfrac{1}{\cos^2(x)}$         |
| $\exp(x)$           |   $\exp(x)$         |
| ${a^{x}}$           |   ${a^{x}\ln{a}}$         |

### 3. The chain rule$^{(1)}$

For a function $h(u(t))$, the derivative of $h$ with respect to $t$ can be expressed as:
$$\dfrac{\partial h}{\partial t} = \dfrac{\partial h}{\partial u}\dfrac{\partial u}{\partial t}.$$
If the function is expressed as a combination of multiple variables that are expressed in terms of t, i.e. $h(u(t), v(t))$, the the derivative of $h$ with respect to $t$ can be expressed as:
$$\frac{\partial h}{\partial t} = \frac{\partial h}{\partial u}\frac{\partial u}{\partial t} + \frac{\partial h}{\partial v}\frac{\partial v}{\partial t}$$

Note that we are only looking at scalar variables in this case, but this idea can be extended to vector variables as well.

  For any $h\left(y\left(x\right)\right)$ where $y\in\mathbb{R}^{n}$ and $x\in\mathbb{R}^{m}$,

  \begin{align}
    \nabla_{x}h = \sum_{i=1}^{n}{\frac{\partial h}{\partial y_{i}}\nabla y_{i}\left(x\right)}.
  \end{align}

### 4. The graph structure of calculations and forward accumulation

Forward accumulation is computing the derivative using the chain rule starting from the inner most derivative to the outer most derivative, where we assume the most basic variables have seed values. Using a graph helps visualize forward accumulation. For example,

\begin{align}
 f\left(x,y\right) &= \frac{x}{y} +cos(x)sin(y)\\
 x &= y = 1
\end{align}

| Trace | Elementary Function | Current Value | Elementary Function Derivative |     $\nabla_{x}$ Value          |     $\nabla_{y}$ Value
| :---: | :-----------------: | :-----------: | :----------------------------: | :-----------------: | :-----------------: | :-----------------: |
| $w_{1}$ | $1$ | $1$ | $\dot{w_1}$ | $1$ | $0$ |
| $w_{2}$ | $1$ | $1$ | $\dot{w_2}$ | $0$ | $1$ |
| $w_{3}$ | $cos{(w_1})$ | $cos{(1)}$ | $-sin{(w_1)}\dot{w_1}$ | $-sin(1)$ | $0$ |
| $w_{4}$ | $sin{(w_2})$ | $sin{(1)}$ | $cos{(w_2)}\dot{w_2}$ | $0$ | $cos{(1)}$ |
| $w_{5}$ | $w_3\dot w_4$ | $sin{(1)}cos{(1)}$ | $w_4\dot{w_3} + w_3\dot{w_4}$ | $-sin^2{(1)}$ | $cos^2{(1)}$ |
| $w_{6}$ | $w_1 / w_2$ | $1$ | $\dot{w_1}/w_2 - w_1 \dot{w_2}/ w_2^2$ | $1$ | $-1$ |
| $w_{7}$ | $w_5 + w_6$ | $sin{(1)}cos{(1)} + 1$ | $\dot{w_5} + \dot{w_6}$ | $-sin^2{(1)} + 1$ | $cos^2{(1)}-1$ |

# Software Organization

The directory structure would look like that:

 ```
 autodiff\
          autodiff\
                    __init__.py
                    functions.py
                    scalar.py
          tests\
                    test_functions.py
                    test_scalar.py
          docs\
                    milestone1.ipynb
                    milestone2.ipynb
          README.md
          requirements.txt
          setup.cfg
          LICENSE

 ```

**Basic modules and what they do?**

This module aims to compute forward automatic differentiation. The module we implemented can efficiently compute the derivatives of a function of automatic differentiation of a scalar input. The *scalar.py* contains the objects which compute the scalar variables returning the value and the derivative. The dunder methods add, sub, mul, truediv, pow, iadd, isub, imul, idiv, ipow are implemented in this module. The *function.py* contains sine, cosine, power, and exponential functions. Both *scalar.py* and *function.py* return the value and the derivative.

**Where do tests live? How are they run? How are they integrated?**

We are using both `TravisCI` and `Coveralls` to test our module. Each test function exists in the tests folder, and it utilizes the pytest package.

**How can someone install your package?**

We are planning to release our package in `PyPI`, but have not done it yet. Therefore, our auto-differentiating software can be downloaded at https://github.com/cs207FinalProjectGroup/cs207-FinalProject.git.

# Implementation Details

## Current Implementation

We have so far implemented the *Scalar* class, which represents scalar variables. To initialize a *Scalar* class object, the user will pass in a string that represents the variable (i.e. 'x', 'y', 'x1', etc.) and also the value of variable (the seed value). The *Scalar* class holds two attributes: 1) the value of the variable `_val` at the current step and 2) a dictionary `_deriv` containing the derivative or partial derivatives (keys will be the names of the variables (i.e *x* and *y*) and the values will be the derivative value with respect to each variable).

```python
import autodiff as ad

x = ad.Scalar('x',2);
print(x._val);
print(x._deriv);
```

    2.0
    {'x': 1.0}

Storing the partial derivatives with respect to each variable allows us to easily compute additional derivatives with respect to each variable when we are performing mathematical operations because we can update each partial derivative individually. When a *Scalar* object is initialized, by default `_deriv` will just be a dictionary with the only key being the string the user passes in with value 1. A user can access the value of a *Scalar* object using the *getValue()* method and access the derivative (or partial derivatives) for the object through the *getDeriv()* method. The user can also get the derivatives/partial derivatives as a numpy array with the *getGradient()* method, which takes in a list of strings, with each element representing the variable to take the derivative with respect to, as an argument.

The dunder methods __add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __itruediv__, __ipow__ (and the right equivalents for the ones that have one) have been overwritten so that they return a new *Scalar* object with an updated value and derivatives. Thus, the adding or substracting two Scalars or raising a Scalar to the power of another Scalar does not change the values or derivatives of the original Scalar objects. By overwriting these methods, we are implementing forward accumulation, as the orders of operation allows us to traverse the chainrule starting from the inside.

```python
#addition example
x,y = ad.Scalar('x', 2),ad.Scalar('y', 5);
val = x+y;
print(val.getValue()); #should be 7.0
print(val.getDeriv()['x']); #should be 1.0
print(val.getDeriv()['y']); #should be 1.0
print();
print(x.getValue(),',', y.getValue()); #x, y should retain original values
```

    7.0
    1.0
    1.0

    2.0 , 5.0

```python
#pow example
x=ad.Scalar('x', 2);
val = x**2;
print(val.getValue()) #should be 4.0
print(val.getDeriv()['x']) #should be 4.0
```

    4.0
    4.0

```python
#division example
x,y=ad.Scalar('x', 3),ad.Scalar('y', 2);
val = x/y;
print(val.getValue()) # should be 1.5
print(val.getDeriv()['x']); #should be 0.5
print(val.getDeriv()['y']); #should be -0.75
print();
print(x.getValue(),',', y.getValue()); #x, y should retain original values
```

    1.5
    0.5
    -0.75

    3.0 , 2.0

We have also implemented the functions **sin**, **cos**, **tan**, **power**, and **exp**. All of these functions can take in an `int`, `float`, or *Scalar* object. If only an `int`/`float` is provided, then a `float` is returned, but if *Scalar* object is provided, then a new *Scalar* object. No changes are made to the value or derivatives of the original *Scalar* object passed in. Within these functions, the *numpy* functions *sin*, *cos*, *tan* and *exp* are used to calculate the appropriate values.

The trigonometric functions  **sin(x)**, **cos(x)**, and **tan(x)** each take in an `int`, `float`, or *Scalar* object *x* and apply the respective trigonometric function to the *x*. If a *x* is a *Scalar* object, the trigonometric function is applied to the *x._val* attribute and the derivative(s) is updated accordingly.

```python
import autodiff as ad
import numpy as np

# Sine functions
x = ad.Scalar('x', 2.0) # float input
val = ad.sin(x)
print(val.getValue()); #shoud be 0.90929742682
print(val.getDeriv()['x']); #should be -0.41614683654

# Cosine functions
x = ad.Scalar('x', 0) # integer input
val = ad.cos(x)
print(val.getValue()); #should be 1.0
print(val.getDeriv()['x']); #should be 0.0

# tan function
x = ad.Scalar('x', 2.0); # float input
val = ad.tan(x);
print(val.getValue()); #shoudl be -2.185039863261519
print(val.getDeriv()['x']); #should be 5.7743992040419174
```

    0.9092974268256817
    -0.4161468365471424
    1.0
    -0.0
    -2.185039863261519
    5.774399204041917

The **power(x1, x2)** function raises *x1* to the power of *x2*. *x1* and *x2* can be any combination of `ints`, `floats`, or *Scalar* objects. If only ints and floats are provided, then *power* will return a `float` with value *x1* raised to the power of *x2*. If at least one *Scalar* object is provided, then **power** works just like using the **__pow__** operator and returns a new *Scalar* object without changing any values in the original *Scalar* object(s).

```python
#power example
x = 5.0
p = 3.0
print (ad.power(x, p)) # should be 125.0
```

    125.0

The **exp(x)** function raises *e* to the power of *x* , where *x* can be an `int`, `float`, or *Scalar* object. If *x* is an `int` or `float`, then a `float` with value equal to *e* raised to the power of *x* is returned. If *x* is a Scalar object, then a new *Scalar* is returned with an updated value and derivative(s).

```python
#exponential example
x = 1.0
print (ad.exp(x)) # should be 2.718281828459045
```

    2.718281828459045

## Future plans

We still need to implement the Vector class, which will be composed of Scalar objects.

For now, we think that our implementation will be something like this:

*Vector* will take in a list or array of *Scalar* objects. A *Vector* only has one attribute: a numpy array of *Scalar* objects, since each *Scalar* object will track its current value and derivative. The dunder methods __add__, __sub__, __mul__,  __truediv__, __pow__, __iadd__, __isub__, __imul__, __idiv__, __ipow__ (and the right equivalents) will all be overwritten so that they return a new array of *Scalar* objects with updates values and derivatives. Similar to numpy methods, the operations are conducted element-wise, i.e. In an addition operation between two *Vector* objects, the first row is added to the first row, second row is added to the second row, etc. As a result, one vector operation becomes multiple scalar operations. To access the values in the *Vector* object, the user can use the *getValue()* method, which returns a *numpy.array* of values. To access the derivatives in the *Vector* object, the user can use the *getDeriv()* method, which returns a list of dictionaries containing derivatives or partial derivatives for each *Scalar* object in the array. We can also add a function that returns this as a matrix, which is the Jacobian and add an optional argument to this function such that the user can just get the derivatives or partial derivatives with respect to the desired variables only (i.e. with respect to 'x', with respect to 'y', etc.). The user can obtain a copy of the *numpy.array* with *Scalar* objects using the *getVector()* method, which will return a copy of the *numpy.array* to the user.

Likewise, for the functions that we have implemented, we have only implemented them to account for Scalars, but not Vectors, so we need to implement them to take into account Vectors.

We will also implement Newton's method for higher dimensions with usage examples.

# Citations
1. Sondak, David. “Automatic Differentiation: The Basics.” CS207-Lecture9. Cambridge, MA. 2 October 2018.

2. Hoffman, Philipp H.W. “A Hitchhiker’s Guide to Automatic Differentiation.” *Numerical Algorithms*, 72, 24 October 2015, 775-811, *Springer Link*, DOI 10.1007/s11075-015-0067-6.


# Sinais periódicos

Neste notebook avaliaremos os sinais periódicos e quais são as condições necessárias para periodicidade.

Esta propriedade dos sinais está ligada ao ***deslocamento no tempo***, uma transformação da variável independente.

Um sinal periódico, contínuo, é aquele para o qual a seguinte propriedade é válida:

\begin{equation}
x(t) = x(t \pm mT_p),
\end{equation}
ou seja, o valor do sinal no instante $t$ [s] é o mesmo para o instante $t \pm mT_p$ [s]. Dessa forma, o sinal se repete a cada
período $T_p$.

$T_p$ é o chamado período fundamental do sinal periódico. Neste caso, $x(t) = x(t \pm T_p) = x(t \pm 2T_p) = ... = x(t \pm kT_p)$.

Para os sinais discretos a definição é análoga:

\begin{equation}
x[n] = x[n \pm m N_p],
\end{equation}
com $N_p$ sendo um número de amostras inteiro.

Um sinal que não é periódico é chamado de aperiódico.

Vamos ver alguns exemplos de sinais periódicos contínuos e discretos.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
from scipy import signal # some signals
import IPython.display as ipd # to play signals
```

```python
# Configurações gerais
fs = 44100
t = np.linspace(0, 1, fs) # vetor temporal
freq = 2000 # Frequencia fundamental
```

```python
# seno ou cosseno
xt = np.sin(2*np.pi*freq*t)
# Figura
plt.figure()
plt.title('Seno')
plt.plot(t, xt, '-b', linewidth = 2, label = 'seno - 1000 [Hz]')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/1000))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

## Um seno com 2 frequências
Se tivermos um sinal

\begin{equation}
x(t) = x(t) = \mathrm{sin}(2  \pi \ m_1 \ f t) + \mathrm{sin}(2  \pi \ m_2 \ f t),
\end{equation}

ele será um sinal periódico desde que $\frac{m_2}{m_1}$ seja um número racional. Do contrário, o sinal será quase-periódico. Ele parecerá periódico, mas se você olhar os detalhes, vai notar que o sinal nunca se repete.

```python
# seno ou cosseno - 2 frequencias
m = 3 #1.4*np.sqrt(2)
xt = np.sin(2*np.pi*freq*t) +  np.sin(2*np.pi*m*freq*t)
# Figura
plt.figure()
plt.title('Seno')
plt.plot(t, xt, '-b', linewidth = 2, label = 'seno - 2 freq')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/freq))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python
# dente de serra
xt = signal.sawtooth(2 * np.pi * freq * t)
# Figura
plt.figure()
plt.title('Dente de serra')
plt.plot(t, xt, '-b', linewidth = 2, label = 'sawtooth')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/freq))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python
# onda quadrada
xt = signal.square(2 * np.pi * freq * t)
# Figura
plt.figure()
plt.title('Onda quadrada')
plt.plot(t, xt, '-b', linewidth = 2, label = 'square')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/freq))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python
N = 9
n = np.arange(N)
xn = [2, 1, -1, 2, 1, -1, 2, 1, -1]

# Figura
plt.figure()
plt.title('Sinal discreto periódico')
plt.stem(n, xn, '-b', basefmt=" ", use_line_collection=  True)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Amostra [-]')
plt.ylabel('Amplitude [-]')
plt.ylim((-2.2, 2.2))
plt.tight_layout()
plt.show()
```

## Sinais discretos periódicos

A periodicidade em sinais discretos tem um limite prático. Pra pensar nisso, podemos imaginar um sinal contínuo $x(t) = \mathrm{cos}(\omega t)$. À medida que a frequência, $f$, do sinal aumenta, sua taxa de oscilação também aumenta. Mas, o que aconteceria no caso de um sinal do tipo

\begin{equation}
x[n] = \mathrm{cos}(\omega n) \ ?
\end{equation}

```python
N = 50
n = np.arange(N)
w = 0
xn = np.cos(w*n)

# Figura
plt.figure()
plt.title('Cosseno discreto')
plt.stem(n, xn, '-b', label = r'\omega = {:.3} [rad/s]'.format(float(w)), basefmt=" ", use_line_collection=  True)
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Amostra [-]')
plt.ylabel('Amplitude [-]')
plt.ylim((-1.2, 1.2))
plt.tight_layout()
plt.show()
```

Aula 5 - sinais periodicos/sinais periodicos.ipynb


# Using LaTeX to Typeset Equations

[LaTeX](http://en.wikipedia.org/wiki/LaTeX) is a markup and programming language for typesetting mathematical languages. It is used extensively in Physics, Math, Statistics and other technical fields for writing papers and other documents with equations.

You can use LaTeX syntax in the Markdown cells of you Jupyer notebooks. This enables you to create notebooks with have equations in addition to code and narrative text. The Jupyter Notebook renders LaTeX using the excellent [MathJax](http://docs.mathjax.org) library.

## Learning LaTeX

This is $\alpha^{x^{2}}$

If you don't already know LaTeX syntax, a good starting place is the [Short Math Guide to LaTeX](ftp://ftp.ams.org/pub/tex/doc/amsmath/short-math-guide.pdf) by the American Mathematical Society.

The rest of this document show examples of LaTeX in the notebook.

## The Lorenz equations

### Source

```\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}
```

### Display

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

## The Cauchy-Schwarz inequality

### Source

```\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}
```
### Display

\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}

## A cross product formula

### Source

```\begin{equation*}
\mathbf{V}_1 \times \mathbf{V}_2 =  \begin{vmatrix}
\mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial X}{\partial u} &  \frac{\partial Y}{\partial u} & 0 \\
\frac{\partial X}{\partial v} &  \frac{\partial Y}{\partial v} & 0
\end{vmatrix}
\end{equation*}
```

### Display

\begin{equation*}
\mathbf{V}_1 \times \mathbf{V}_2 =  \begin{vmatrix}
\mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial X}{\partial u} &  \frac{\partial Y}{\partial u} & 0 \\
\frac{\partial X}{\partial v} &  \frac{\partial Y}{\partial v} & 0
\end{vmatrix}
\end{equation*}

## The probability of getting \(k\) heads when flipping \(n\) coins is

### Source

```\begin{equation*}
P(E)   = {n \choose k} p^k (1-p)^{ n-k}
\end{equation*}
```

### Display

\begin{equation*}
P(E)   = {n \choose k} p^k (1-p)^{ n-k}
\end{equation*}

## An identity of Ramanujan

### Source

```\begin{equation*}
\frac{1}{\Bigl(\sqrt{\phi \sqrt{5}}-\phi\Bigr) e^{\frac25 \pi}} =
1+\frac{e^{-2\pi}} {1+\frac{e^{-4\pi}} {1+\frac{e^{-6\pi}}
{1+\frac{e^{-8\pi}} {1+\ldots} } } }
\end{equation*}
```

### Display

\begin{equation*}
\frac{1}{\Bigl(\sqrt{\phi \sqrt{5}}-\phi\Bigr) e^{\frac25 \pi}} =
1+\frac{e^{-2\pi}} {1+\frac{e^{-4\pi}} {1+\frac{e^{-6\pi}}
{1+\frac{e^{-8\pi}} {1+\ldots} } } }
\end{equation*}

## A Rogers-Ramanujan identity

### Source

```\begin{equation*}
1 +  \frac{q^2}{(1-q)}+\frac{q^6}{(1-q)(1-q^2)}+\cdots =
\prod_{j=0}^{\infty}\frac{1}{(1-q^{5j+2})(1-q^{5j+3})},
\quad\quad \text{for $|q|<1$}.
\end{equation*}
```

### Display

\begin{equation*}
1 + \frac{q^2}{(1-q)}+\frac{q^6}{(1-q)(1-q^2)}+\cdots =
\prod_{j=0}^{\infty}\frac{1}{(1-q^{5j+2})(1-q^{5j+3})},
\quad\quad \text{for $|q|<1$}.
\end{equation*}

## Maxwell's equations

### Source

```\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
```

### Display

\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\   \nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

## Equation Numbering and References

Equation numbering and referencing will be available in a future version of the Jupyter Notebook.

## Inline Typesetting (Mixing Markdown and TeX)

While display equations look good for a page of samples, the ability to mix math and *formatted* **text** in a paragraph is also important.

## Source

``` This expression $\sqrt{3x-1}+(1+x)^2$ is an example of a TeX inline equation in a **[Markdown-formatted](http://daringfireball.net/projects/markdown/)** sentence.
```

## Display

This expression $\sqrt{3x-1}+(1+x)^2$ is an example of a TeX inline equation in a **[Markdown-formatted](http://daringfireball.net/projects/markdown/)** sentence.

## Other syntax

You will notice in other places on the web that `$$` are needed explicitly to begin and end MathJax typesetting.  This is **not** required if you will be using TeX environments, but the IPython notebook will accept this syntax on legacy notebooks.

### Source

```
$$
\begin{array}{c}
y_1 \\\
y_2 \mathtt{t}_i \\\
z_{3,4}
\end{array}
$$
```

```
$$
\begin{array}{c}
y_1 \cr
y_2 \mathtt{t}_i \cr
y_{3}
\end{array}
$$
```

```
$$\begin{eqnarray}
x' &=& &x \sin\phi &+& z \cos\phi \\
z' &=& - &x \cos\phi &+& z \sin\phi \\
\end{eqnarray}$$
```

```
$$
x=4
$$
```

### Display

$$
\begin{array}{c}
y_1 \\\
y_2 \mathtt{t}_i \\\
z_{3,4}
\end{array}
$$

$$
\begin{array}{c}
y_1 \cr
y_2 \mathtt{t}_i \cr
y_{3}
\end{array}
$$

$$\begin{eqnarray}
x' &=& &x \sin\phi &+& z \cos\phi \\
z' &=& - &x \cos\phi &+& z \sin\phi \\
\end{eqnarray}$$

$$
x=4
$$


# Case Studies

Recall from the introduction that a common statistical analysis is to compare two groups. In a frequentist setting, statistical hypothesis testing is often applied as a default approach. In a Bayesian setting, we will instead use **estimation** to tackle this problem, in the hopes of obtaining more reliable inference.

## Estimation for one group

Before we compare two groups using Bayesian analysis, let's start with an even simpler scenario: statistical inference for one group.

For this we will use Gelman et al.'s (2007) radon dataset. In this dataset the amount of the radioactive gas radon has been measured among different households in all counties of several states. Radon gas is known to be the highest cause of lung cancer in non-smokers. It is believed to be more strongly present in households containing a basement and to differ in amount present among types of soil.

>  the US EPA has set an action level of 4 pCi/L. At or above this level of radon, the EPA recommends you take corrective measures to reduce your exposure to radon gas.

Let's import the dataset:

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
sns.set_context('notebook')

import warnings
warnings.filterwarnings("ignore", category=UserWarning)

RANDOM_SEED = 20090425
```

```python
radon = pd.read_csv('../data/radon.csv', index_col=0)
radon.head()
```

      idnum
      state
      state2
      stfips
      zip
      region
      typebldg
      floor
      room
      basement
      ...
      pcterr
      adjwt
      dupflag
      zipflag
      cntyfips
      county
      fips
      Uppm
      county_code
      log_radon

      0
      5081.0
      MN
      MN
      27.0
      55735
      5.0
      1.0
      1.0
      3.0
      N
      ...
      9.7
      1146.499190
      1.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      1
      5082.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      14.5
      471.366223
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.832909

      2
      5083.0
      MN
      MN
      27.0
      55748
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      9.6
      433.316718
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      1.098612

      3
      5084.0
      MN
      MN
      27.0
      56469
      5.0
      1.0
      0.0
      4.0
      Y
      ...
      24.3
      461.623670
      0.0
      0.0
      1.0
      AITKIN
      27001.0
      0.502054
      0
      0.095310

      4
      5085.0
      MN
      MN
      27.0
      55011
      3.0
      1.0
      0.0
      4.0
      Y
      ...
      13.8
      433.316718
      0.0
      0.0
      3.0
      ANOKA
      27003.0
      0.428565
      1
      1.163151

5 rows × 29 columns

Let's focus on the (log) radon levels measured in a single county (Hennepin).

Suppose we are interested in:

- whether the mean log-radon value is greater than 4 pCi/L in Hennepin county
- the probability that any randomly-chosen household in Hennepin county has a reading of greater than 4

```python
hennepin_radon = radon.query('county=="HENNEPIN"').log_radon
sns.distplot(hennepin_radon)
```

```python
hennepin_radon.shape
```

    (105,)

### The model

Recall that the first step in Bayesian inference is specifying a **full probability model** for the problem.

This consists of:

- a likelihood function(s) for the observations
- priors for all unknown quantities

The measurements look approximately normal, so let's start by assuming a normal distribution as the sampling distribution (likelihood) for the data.

$$y_i \sim N(\mu, \sigma^2)$$

(don't worry, we can evaluate this assumption)

This implies that we have 2 unknowns in the model; the mean and standard deviation of the distribution.

#### Prior choice

How do we choose distributions to use as priors for these parameters?

There are several considerations:

- discrete vs continuous values
- the support of the variable
- the available prior information

While there may likely be prior information about the distribution of radon values, we will assume no prior knowledge, and specify a **diffuse** prior for each parameter.

Since the mean can take any real value (since it is on the log scale), we will use another normal distribution here, and specify a large variance to allow the possibility of very large or very small values:

$$\mu \sim N(0, 10^2)$$

For the standard deviation, we know that the true value must be positive (no negative variances!). I will choose a uniform prior bounded from below at zero and from above at a value that is sure to be higher than any plausible value the true standard deviation (on the log scale) could take.

$$\sigma \sim U(0, 10)$$

We can encode these in a Python model, using the PyMC3 package, as follows:

```python
from pymc3 import Model, Normal, Uniform

with Model() as radon_model:

    μ = Normal('μ', mu=0, sd=10)
    σ = Uniform('σ', 0, 10)
```

> ## Software
> Today there is an array of software choices for Bayesians, including both open source software (*e.g.*, Stan, PyMC, JAGS, emcee) and commercial (*e.g.*, SAS, Stata). These examples can be replicated in any of these environments.

All that remains is to add the likelihood, which takes $\mu$ and $\sigma$ as parameters, and the log-radon values as the set of observations:

```python
with radon_model:

    y = Normal('y', mu=μ, sd=σ, observed=hennepin_radon)
```

Now, we will fit the model using a numerical approach called **variational inference**. This will estimate the posterior distribution using an optimized approximation, and then draw samples from it.

```python
from pymc3 import fit

with radon_model:

    samples = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 136.32: 100%|██████████| 10000/10000 [00:16<00:00, 596.00it/s]
    Finished [100%]: Average Loss = 136.28

```python
from pymc3 import plot_posterior

plot_posterior(samples, varnames=['μ'], ref_val=np.log(4), color='LightSeaGreen');
```

The plot shows the posterior distribution of $\mu$, along with an estimate of the 95% posterior **credible interval**.

The output

    83.1% < 1.38629 < 16.9%%

informs us that the probability of $\mu$ being less than $\log(4)$ is 83.1%% and the corresponding probability of being greater than $\log(4)$ is 16.9%.

> The posterior probability that the mean level of household radon in Henneprin County is greater than 4 pCi/L is 0.17.

### Prediction

What is the probability that a given household has a log-radon measurement larger than one? To answer this, we make use of the **posterior predictive distribution**.

$$p(z |y) = \int_{\theta} p(z |\theta) p(\theta | y) d\theta$$

where here $z$ is the predicted value and y is the data used to fit the model.

We can estimate this from the posterior samples of the parameters in the model.

```python
mus = samples['μ']
sigmas = samples['σ']
```

```python
radon_samples = Normal.dist(mus, sigmas).random()
```

```python
(radon_samples > np.log(4)).mean()
```

    0.453

> The posterior probability that a randomly-selected household in Henneprin County contains radon levels in excess of 4 pCi/L is 0.48.

## Two Groups with Continiuous Outcome

To illustrate how this Bayesian estimation approach works in practice, we will use a fictitious example from Kruschke (2012) concerning the evaluation of a clinical trial for drug evaluation. The trial aims to evaluate the efficacy of a "smart drug" that is supposed to increase intelligence by comparing IQ scores of individuals in a treatment arm (those receiving the drug) to those in a control arm (those recieving a placebo). There are 47 individuals and 42 individuals in the treatment and control arms, respectively.

```python
drug = pd.DataFrame(dict(iq=(101,100,102,104,102,97,105,105,98,101,100,123,105,103,100,95,102,106,
        109,102,82,102,100,102,102,101,102,102,103,103,97,97,103,101,97,104,
        96,103,124,101,101,100,101,101,104,100,101),
                         group='drug'))
placebo = pd.DataFrame(dict(iq=(99,101,100,101,102,100,97,101,104,101,102,102,100,105,88,101,100,
           104,100,100,100,101,102,103,97,101,101,100,101,99,101,100,100,
           101,100,99,101,100,102,99,100,99),
                            group='placebo'))

trial_data = pd.concat([drug, placebo], ignore_index=True)
trial_data.hist('iq', by='group');
```

Since there appear to be extreme ("outlier") values in the data, we will choose a Student-t distribution to describe the distributions of the scores in each group. This sampling distribution adds **robustness** to the analysis, as a T distribution is less sensitive to outlier observations, relative to a normal distribution.

The three-parameter Student-t distribution allows for the specification of a mean $\mu$, a precision (inverse-variance) $\lambda$ and a degrees-of-freedom parameter $\nu$:

$$f(x|\mu,\lambda,\nu) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\pi\nu}\right)^{\frac{1}{2}} \left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\frac{\nu+1}{2}}$$

the degrees-of-freedom parameter essentially specifies the "normality" of the data, since larger values of $\nu$ make the distribution converge to a normal distribution, while small values (close to zero) result in heavier tails.

Thus, the likelihood functions of our model are specified as follows:

$$\begin{align}
y^{(drug)}_i &\sim T(\nu, \mu_1, \sigma_1) \\
y^{(placebo)}_i &\sim T(\nu, \mu_2, \sigma_2)
\end{align}$$

As a simplifying assumption, we will assume that the degree of normality $\nu$ is the same for both groups.

### Exercise

Draw 10000 samples from a Student-T distribution (`StudentT` in PyMC3) with parameter `nu=3` and compare the distribution of these values to a similar number of draws from a Normal distribution with parameters `mu=0` and `sd=1`.

```python
from pymc3 import StudentT

t = StudentT.dist(nu=3).random(size=10000)
n = Normal.dist(0, 1).random(size=10000)
```

```python
sns.distplot(t, label='Student-T')
sns.distplot(n, label='Normal')
plt.legend()
plt.xlim(-10,10);
```

### Prior choice

Since the means are real-valued, we will apply normal priors. Since we know something about the population distribution of IQ values, we will center the priors at 100, and use a standard deviation that is more than wide enough to account for plausible deviations from this population mean:

$$\mu_k \sim N(100, 10^2)$$

```python
with Model() as drug_model:

    μ_0 = Normal('μ_0', 100, sd=10)
    μ_1 = Normal('μ_1', 100, sd=10)
```

Similarly, we will use a uniform prior for the standard deviations, with an upper bound of 20.

```python
with drug_model:
    σ_0 = Uniform('σ_0', lower=0, upper=20)
    σ_1 = Uniform('σ_1', lower=0, upper=20)
```

For the degrees-of-freedom parameter $\nu$, we will use an **exponential** distribution with a mean of 30; this allocates high prior probability over the regions of the parameter that describe the range from normal to heavy-tailed data under the Student-T distribution.

```python
from pymc3 import Exponential

with drug_model:
    ν = Exponential('ν_minus_one', 1/29.) + 1

```

```python
sns.distplot(Exponential.dist(1/29).random(size=10000), kde=False);
```

```python
from pymc3 import StudentT

with drug_model:

    drug_like = StudentT('drug_like', nu=ν, mu=μ_1, lam=σ_1**-2, observed=drug.iq)
    placebo_like = StudentT('placebo_like', nu=ν, mu=μ_0, lam=σ_0**-2, observed=placebo.iq)
```

Now that the model is fully specified, we can turn our attention to tracking the posterior quantities of interest. Namely, we can calculate the difference in means between the drug and placebo groups.

As a joint measure of the groups, we will also estimate the "effect size", which is the difference in means scaled by the pooled estimates of standard deviation. This quantity can be harder to interpret, since it is no longer in the same units as our data, but it is a function of all four estimated parameters.

```python
from pymc3 import Deterministic

with drug_model:

    diff_of_means = Deterministic('difference of means', μ_1 - μ_0)

    effect_size = Deterministic('effect size',
                        diff_of_means / np.sqrt((σ_1**2 + σ_0**2) / 2))

```

```python
with drug_model:

    drug_trace = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 244.72: 100%|██████████| 10000/10000 [00:21<00:00, 470.88it/s]
    Finished [100%]: Average Loss = 244.72

```python
plot_posterior(drug_trace[100:],
                varnames=['μ_0', 'μ_1', 'σ_0', 'σ_1', 'ν_minus_one'],
                color='#87ceeb');
```

```python
plot_posterior(drug_trace[100:],
          varnames=['difference of means', 'effect size'],
          ref_val=0,
          color='#87ceeb');
```

> The posterior probability that the mean IQ of subjects in the treatment group is greater than that of the control group is 0.99.

### Exercise

Load the `nashville_precip.txt` dataset. Build a model to compare rainfall in January and July.

- What's the probability that the expected rainfall in January is larger than in July?
- What's the probability that January rainfall exceeds July rainfall in a given year?

```python
nash_precip = pd.read_table('../data/nashville_precip.txt',
                            delimiter='\s+', na_values='NA', index_col=0)
nash_precip.head()
```

      Jan
      Feb
      Mar
      Apr
      May
      Jun
      Jul
      Aug
      Sep
      Oct
      Nov
      Dec

      Year

      1871
      2.76
      4.58
      5.01
      4.13
      3.30
      2.98
      1.58
      2.36
      0.95
      1.31
      2.13
      1.65

      1872
      2.32
      2.11
      3.14
      5.91
      3.09
      5.17
      6.10
      1.65
      4.50
      1.58
      2.25
      2.38

      1873
      2.96
      7.14
      4.11
      3.59
      6.31
      4.20
      4.63
      2.36
      1.81
      4.28
      4.36
      5.94

      1874
      5.22
      9.23
      5.36
      11.84
      1.49
      2.87
      2.65
      3.52
      3.12
      2.63
      6.12
      4.19

      1875
      6.15
      3.06
      8.14
      4.22
      1.73
      5.63
      8.12
      1.60
      3.79
      1.25
      5.46
      4.30

```python
# Write your answer here
```

## Two Groups with Binary Outcome

Now that we have seen how to generalize normally-distributed data to another distribution, we are equipped to analyze other data types. Binary outcomes are common in clinical research:

- survival/death
- true/false
- presence/absence
- positive/negative

> *Never, ever dichotomize continuous or ordinal variables prior to statistical analysis*

In practice, binary outcomes are encoded as ones (for event occurrences) and zeros (for non-occurrence). A single binary variable is distributed as a **Bernoulli** random variable:

$$f(x \mid p) = p^{x} (1-p)^{1-x}$$

Such events are sometimes reported as sums of individual events, such as the number of individuals in a group who test positive for a condition of interest. Sums of Bernoulli events are distributed as **binomial** random variables.

$$f(x \mid n, p) = \binom{n}{x} p^x (1-p)^{n-x}$$

The parameter in both models is $p$, the probability of the occurrence of an event. In terms of inference, we are typically interested in whether $p$ is larger or smaller in one group relative to another.

To demonstrate the comparison of two groups with binary outcomes using Bayesian inference, we will use a sample pediatric dataset. Data on 671 infants with very low (<1600 grams) birth weight from 1981-87 were collected at Duke University Medical Center. Of interest is the relationship between the outcome intra-ventricular hemorrhage (IVH) and predictor such as birth weight, gestational age, presence of pneumothorax and mode of delivery.

```python
vlbw = pd.read_csv('../data/vlbw.csv', index_col=0).dropna(axis=0, subset=['ivh', 'pneumo'])
vlbw.head()
```

      birth
      exit
      hospstay
      lowph
      pltct
      race
      bwt
      gest
      inout
      twn
      ...
      vent
      pneumo
      pda
      cld
      pvh
      ivh
      ipe
      year
      sex
      dead

      5
      81.593002
      81.598999
      2.0
      6.969997
      54.0
      black
      925.0
      28.0
      born at Duke
      0.0
      ...
      1.0
      1.0
      0.0
      0.0
      definite
      definite
      NaN
      81.594055
      female
      1

      6
      81.601997
      81.771004
      62.0
      7.189999
      NaN
      white
      940.0
      28.0
      born at Duke
      0.0
      ...
      1.0
      0.0
      0.0
      0.0
      absent
      absent
      absent
      81.602295
      female
      0

      13
      81.683998
      81.853996
      62.0
      7.179996
      182.0
      black
      1110.0
      28.0
      born at Duke
      0.0
      ...
      0.0
      1.0
      0.0
      1.0
      absent
      absent
      absent
      81.684448
      male
      0

      14
      81.689003
      81.877998
      69.0
      7.419998
      361.0
      white
      1180.0
      28.0
      born at Duke
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      absent
      absent
      absent
      81.689880
      male
      0

      16
      81.696999
      81.952004
      93.0
      7.239998
      255.0
      black
      770.0
      26.0
      born at Duke
      0.0
      ...
      1.0
      0.0
      0.0
      1.0
      absent
      absent
      absent
      81.698120
      male
      0

5 rows × 26 columns

To demonstrate binary data analysis, we will try to estimate the difference between the probability of an intra-ventricular hemorrhage for infants with a pneumothorax.

```python
pd.crosstab(vlbw.ivh, vlbw.pneumo)
```

      pneumo
      0.0
      1.0

      ivh

      absent
      359
      73

      definite
      45
      30

      possible
      6
      4

We will create a binary outcome by combining `definite` and `possible` into a single outcome.

```python
ivh = vlbw.ivh.isin(['definite', 'possible']).astype(int).values
x = vlbw.pneumo.astype(int).values
```

### Prior choice

What should we choose as a prior distribution for $p$?

We could stick with a normal distribution, but note that the value of $p$ is **constrained** by the laws of probability. Namely, we cannot have values smaller than zero nor larger than one. So, choosing a normal distribution will result in ascribing positive probability to unsupported values of the parameter. In many cases, this will still work in practice, but will be inefficient for calculating the posterior and will not accurately represent the prior information about the parameter.

A common choice in this context is the **beta distribution**, a continuous distribution with 2 parameters and whose support is on the unit interval:

$$ f(x \mid \alpha, \beta) = \frac{x^{\alpha - 1} (1 - x)^{\beta - 1}}{B(\alpha, \beta)}$$

- Support: $x \in (0, 1)$
- Mean: $\dfrac{\alpha}{\alpha + \beta}$
- Variance: $\dfrac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$

```python
from pymc3 import Beta

params = (5, 1), (1, 3), (5, 5), (0.5, 0.5), (1, 1)

fig, axes = plt.subplots(1, len(params), figsize=(14, 4), sharey=True)
for ax, (alpha, beta) in zip(axes, params):
    sns.distplot(Beta.dist(alpha, beta).random(size=10000), ax=ax, kde=False)
    ax.set_xlim(0, 1)
    ax.set_title(r'$\alpha={0}, \beta={1}$'.format(alpha, beta));
```

So let's use a beta distribution to model our prior knowledge of the probabilities for both groups. Setting $\alpha = \beta = 1$ will result in a uniform distribution of prior mass:

```python
with Model() as ivh_model:

    p = Beta('p', 1, 1, shape=2)
```

We can now use `p` as the parameter of our Bernoulli likelihood. Here, `x` is a vector of zeros an ones, which will extract the approproate group probability for each subject:

```python
from pymc3 import Bernoulli

with ivh_model:

    bb_like = Bernoulli('bb_like', p=p[x], observed=ivh)
```

Finally, since we are interested in the difference between the probabilities, we will keep track of this difference:

```python
with ivh_model:

    p_diff = Deterministic('p_diff', p[1] - p[0])
```

```python
with ivh_model:
    ivh_trace = fit(random_seed=RANDOM_SEED).sample(1000)
```

    Average Loss = 226.71: 100%|██████████| 10000/10000 [00:12<00:00, 822.06it/s]
    Finished [100%]: Average Loss = 226.71

```python
plot_posterior(ivh_trace[100:], varnames=['p'], color='#87ceeb');
```

We can see that the probability that `p` is larger for the pneumothorax with probability one.

```python
plot_posterior(ivh_trace[100:], varnames=['p_diff'], ref_val=0, color='#87ceeb');
```

## References and Resources

- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis, Third Edition. CRC Press.
- Kruschke, J.K. *Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan.* 2015. Academic Press / Elsevier.
- O'Shea M, Savitz D.A., Hage M.L., Feinstein K.A.: *Prenatal events and the risk of subependymal / intraventricular haemorrhage in very low birth weight neonates*. **Paediatric and Perinatal Epdiemiology** 1992;6:352-362


Физтех-Школа Прикладной математики и информатики (ФПМИ) МФТИ

---

Нейрон с сигмоидой

---

```python
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
import numpy as np
import pandas as pd
```

Напомним, что сигмоидальная функция (сигмоида) выглядит так:

$$\sigma(x)=\frac{1}{1+e^{-x}}$$

В данном случае мы снова решаем задачу бинарной классификации (2 класса: 1 или 0), но здесь уже будет другая функция активации:

$$MSE\_Loss(\hat{y}, y) = \frac{1}{n}\sum_{i=1}^{n} (\hat{y_i} - y_i)^2 = \frac{1}{n}\sum_{i=1}^{n} (\sigma(w \cdot X_i) - y_i)^2$$

Здесь $w \cdot X_i$ - скалярное произведение, а $\sigma(w \cdot X_i) =\frac{1}{1+e^{-w \cdot X_i}} $ - сигмоида.

**Примечание:** В формуле предполагается, что $b$ - свободный член - является частью вектора весов: $w_0$. Тогда, если к $X$ приписать слева единичный столбец, в скалярном произведении $b$ будет именно как свободный член. При реализации класса $b$ нужно считать отдельно (чтобы было нагляднее).

Можем взять производную лосса по весам и спускаться в пространстве весов в направлении наискорейшего убывания функции лосса. Формула для обновления весов в градиентном спуске такая:

$$w^{j+1} = w^{j} - \alpha \frac{\partial Loss}{\partial w} (w^{j})$$

$w^j$ -- вектор весов на $j$-ой итерации

Распишем дальше:

* Для веса $w_j$:

$$ \frac{\partial Loss}{\partial w_j} =
\frac{2}{n} \sum_{i=1}^n \left(\sigma(w \cdot x_i) - y_i\right)(\sigma(w \cdot x_i))_{w_j}' = \frac{2}{n} \sum_{i=1}^n \left(\sigma(w \cdot x_i) - y_i\right)\sigma(w \cdot x_i)(1 - \sigma(w \cdot x_i))x_{ij}$$

* Градиент $Loss$'а по вектору весов -- это вектор, $j$-ая компонента которого равна $\frac{\partial Loss}{\partial w_j}$ (помним, что весов всего $m$):

$$\begin{align}
    \frac{\partial Loss}{\partial w} &= \begin{bmatrix}
           \frac{2}{n} \sum_{i=1}^n \left(\sigma(w \cdot x_i) - y_i\right)\sigma(w \cdot x_i)(1 - \sigma(w \cdot x_i))x_{i1} \\
           \frac{2}{n} \sum_{i=1}^n \left(\sigma(w \cdot x_i) - y_i\right)\sigma(w \cdot x_i)(1 - \sigma(w \cdot x_i))x_{i2} \\
           \vdots \\
           \frac{2}{n} \sum_{i=1}^n \left(\sigma(w \cdot x_i) - y_i\right)\sigma(w \cdot x_i)(1 - \sigma(w \cdot x_i))x_{im}
         \end{bmatrix}
\end{align}=\frac{1}{n} X^T (\sigma(w \cdot X) - y)\sigma(w \cdot X)(1 - \sigma(w \cdot X))$$

Реализуем сигмоиду и её производную:

```python
def sigmoid(x):
    """Сигмоидальная функция"""
    return 1 / (1 + np.exp(-x))
```

```python
def sigmoid_derivative(x):
    """Производная сигмоиды"""
    return sigmoid(x) * (1 - sigmoid(x))
```

Теперь нужно написать нейрон с сигмоидной функцией активации. Здесь всё очень похоже на перцептрон, но будут по-другому обновляться веса и другая функция активации:

```python
def loss(y_pred, y):
    '''
    Считаем среднеквадратичную ошибку
    '''
    y_pred = y_pred.reshape(-1,1)
    y = np.array(y).reshape(-1,1)
    return 0.5 * np.mean((y_pred - y) ** 2)
```

```python
class Neuron:
    def __init__(self, w=None, b=0):
        """
        :param: w -- вектор весов
        :param: b -- смещение
        """
        # пока что мы не знаем размер матрицы X, а значит не знаем, сколько будет весов
        self.w = w
        self.b = b

    def activate(self, x):
        return sigmoid(x)

    def forward_pass(self, X):
        """
        Рассчитывает ответ нейрон при предъявлении набора объектов
        :param: X -- матрица примеров размера (n, m), каждая строка - отдельный объект
        :return: вектор размера (n, 1) из нулей и единиц с ответами нейрона
        """
        n = X.shape[0]
        y_pred = np.zeros((n, 1))  # y_pred == y_predicted - предсказанные классы
        y_pred = self.activate(X @ self.w.reshape(X.shape[1], 1) + self.b)
        return y_pred.reshape(-1, 1)

    def backward_pass(self, X, y, y_pred, learning_rate=0.1):
        """
        Обновляет значения весов нейрон в соответствии с этим объектом
        :param: X -- матрица входов размера (n, m)
                y -- вектор правильных ответов размера (n, 1)
                learning_rate - "скорость обучения" (символ alpha в формулах выше)
        В этом методе ничего возвращать не нужно, только правильно поменять веса
        с помощью градиентного спуска.
        """
        n = len(y)
        y = np.array(y).reshape(-1, 1)
        sigma = self.activate(X @ self.w + self.b)
        self.w = self.w - learning_rate * (X.T @ ((sigma - y) * sigma * (1 - sigma))) / n
        self.b = self.b - learning_rate * np.mean((sigma - y) * sigma * (1 - sigma))

    def fit(self, X, y, num_epochs=5000):
        """
        Спускаемся в минимум
        :param: X -- матрица объектов размера (n, m)
                y -- вектор правильных ответов размера (n, 1)
                num_epochs -- количество итераций обучения
        :return: loss_values -- вектор значений функции потерь
        """
        self.w = np.zeros((X.shape[1], 1))  # столбец (m, 1)
        self.b = 0  # смещение
        Loss_values = []  # значения функции потерь на различных итерациях обновления весов

        for i in range(num_epochs):
            # предсказания с текущими весами
            y_pred = self.forward_pass(X)
            # считаем функцию потерь с текущими весами
            Loss_values.append(loss(y_pred, y))
            # обновляем веса в соответсвие с тем, где ошиблись раньше
            self.backward_pass(X, y, y_pred)

        return Loss_values
```

**Тестирование нейрона**

Протестировуем новый нейрон **на тех же данных** по аналогии с тем, как это было проделано с перцептроном.

**Проверка forward_pass()**

```python
w = np.array([1., 2.]).reshape(2, 1)
b = 2.
X = np.array([[1., 3.],
              [2., 4.],
              [-1., -3.2]])

neuron = Neuron(w, b)
y_pred = neuron.forward_pass(X)
print ("y_pred = " + str(y_pred))
```

    y_pred = [[0.99987661]
     [0.99999386]
     [0.00449627]]

**Проверка backward_pass()**

```python
y = np.array([1, 0, 1]).reshape(3, 1)
```

```python
neuron.backward_pass(X, y, y_pred)

print ("w = " + str(neuron.w))
print ("b = " + str(neuron.b))
```

    w = [[0.99985106]
     [1.99952388]]
    b = 2.000148326741343

Посмотрим, как меняется функция потерь в течение процесса обучения на реальных данных - датасет "Яблоки и Груши":

```python
data = pd.read_csv("./homework/data/apples_pears.csv")
```

```python
data.head()
```

      yellowness
      symmetry
      target

      0
      0.779427
      0.257305
      1.0

      1
      0.777005
      0.015915
      1.0

      2
      0.977092
      0.304210
      1.0

      3
      0.043032
      0.140899
      0.0

      4
      0.760433
      0.193123
      1.0

```python
plt.figure(figsize=(10, 8))
plt.scatter(data.iloc[:, 0], data.iloc[:, 1], c=data['target'], cmap='rainbow')
plt.title('Яблоки и груши', fontsize=15)
plt.xlabel('симметричность', fontsize=14)
plt.ylabel('желтизна', fontsize=14)
plt.show();
```

Обозначим, что здесь признаки, а что - классы:

```python
X = data.iloc[:,:2].values  # матрица объекты-признаки
y = data['target'].values.reshape((-1, 1))  # классы (столбец из нулей и единиц)
```

**Вывод функции потерь**
Функция потерь должна убывать и в итоге стать близкой к 0

```python
%%time

neuron = Neuron()
Loss_values = neuron.fit(X, y)

plt.figure(figsize=(10, 8))
plt.plot(Loss_values)
plt.title('Функция потерь', fontsize=15)
plt.xlabel('номер итерации', fontsize=14)
plt.ylabel('$Loss(\hat{y}, y)$', fontsize=14)
plt.show()
```

Посмотрим, как нейрон классифицировал объекты из выборки:

```python
plt.figure(figsize=(10, 8))
plt.scatter(data.iloc[:, 0], data.iloc[:, 1], c=np.array(neuron.forward_pass(X) > 0.5).ravel(), cmap='spring')
plt.title('Яблоки и груши', fontsize=15)
plt.xlabel('симметричность', fontsize=14)
plt.ylabel('желтизна', fontsize=14)
plt.show();
```

На самом деле то, что вы здесь наблюдаете (плохое качество разделения) -- последствие **затухающих градиентов (vanishing gradients)**. Мы позже ещё поговорим об этом (то есть о том, почему это происходит в данном случае).

Попробуем увеличить количество итераций градиентного спуска (50k итераций):

```python
%%time

neuron = Neuron()
loss_values = neuron.fit(X, y, num_epochs=50000)

plt.figure(figsize=(10, 8))
plt.scatter(data.iloc[:, 0], data.iloc[:, 1], c=np.array(neuron.forward_pass(X) > 0.5).ravel(), cmap='spring')
plt.title('Яблоки и груши', fontsize=15)
plt.xlabel('симметричность', fontsize=14)
plt.ylabel('желтизна', fontsize=14)
plt.show();
```

Что ж, стало лучше. Градиенты "текут" медленно, но верно.

---

Полезные ссылки

0). Статья от Стэнфорда: http://cs231n.github.io/neural-networks-1/

1). Хорошая статья про функции активации: https://www.jeremyjordan.me/neural-networks-activation-functions/

2). [Видео от Siraj Raval](https://www.youtube.com/watch?v=-7scQpJT7uo)

3). Современная статья про функции активации. Теперь на хайпе активация **$swish(x) = x\sigma (\beta x)$:** https://arxiv.org/pdf/1710.05941.pdf (кстати, при её поиске в некоторой степени использовался neural architecture search)

4). **SeLU** имеет очень интересные, доказанные с помощью теории вероятностей свойства: https://arxiv.org/pdf/1706.02515.pdf (да, в этой статье 102 страницы)


```julia
using Symbolics
using LinearAlgebra
using StaticArrays
```

```julia
n = 3
@variables a[1:n, 1:n] x[1:n]
```

    2-element Vector{Symbolics.Arr{Num, N} where N}:
     a[1:3,1:3]
     x[1:3]

```julia
A = collect(a)
```

\begin{equation}
\left[
\begin{array}{ccc}
a{{_1}}ˏ{_1} & a{{_1}}ˏ{_2} & a{{_1}}ˏ{_3} \\
a{{_2}}ˏ{_1} & a{{_2}}ˏ{_2} & a{{_2}}ˏ{_3} \\
a{{_3}}ˏ{_1} & a{{_3}}ˏ{_2} & a{{_3}}ˏ{_3} \\
\end{array}
\right]
\end{equation}

```julia
as =  unique(vec(A))
```

\begin{equation}
\left[
\begin{array}{c}
a{{_1}}ˏ{_1} \\
a{{_2}}ˏ{_1} \\
a{{_3}}ˏ{_1} \\
a{{_1}}ˏ{_2} \\
a{{_2}}ˏ{_2} \\
a{{_3}}ˏ{_2} \\
a{{_1}}ˏ{_3} \\
a{{_2}}ˏ{_3} \\
a{{_3}}ˏ{_3} \\
\end{array}
\right]
\end{equation}

```julia
X = collect(x)
```

\begin{equation}
\left[
\begin{array}{c}
x{_1} \\
x{_2} \\
x{_3} \\
\end{array}
\right]
\end{equation}

```julia
X'A*X/2
```

\begin{equation}
\frac{1}{2} \left( x{_1} a{{_1}}ˏ{_1} + x{_2} a{{_2}}ˏ{_1} + x{_3} a{{_3}}ˏ{_1} \right) x{_1} + \frac{1}{2} \left( x{_1} a{{_1}}ˏ{_2} + x{_2} a{{_2}}ˏ{_2} + x{_3} a{{_3}}ˏ{_2} \right) x{_2} + \frac{1}{2} \left( x{_1} a{{_1}}ˏ{_3} + x{_2} a{{_2}}ˏ{_3} + x{_3} a{{_3}}ˏ{_3} \right) x{_3}
\end{equation}

```julia
X'A*X/2 |> expand
```

\begin{equation}
\frac{1}{2} x{_1}^{2} a{{_1}}ˏ{_1} + \frac{1}{2} x{_2}^{2} a{{_2}}ˏ{_2} + \frac{1}{2} x{_3}^{2} a{{_3}}ˏ{_3} + \frac{1}{2} x{_1} x{_2} a{{_1}}ˏ{_2} + \frac{1}{2} x{_1} x{_2} a{{_2}}ˏ{_1} + \frac{1}{2} x{_1} x{_3} a{{_1}}ˏ{_3} + \frac{1}{2} x{_1} x{_3} a{{_3}}ˏ{_1} + \frac{1}{2} x{_2} x{_3} a{{_2}}ˏ{_3} + \frac{1}{2} x{_2} x{_3} a{{_3}}ˏ{_2}
\end{equation}

```julia
Symbolics.gradient(X'A*X/2, X)
```

\begin{equation}
\left[
\begin{array}{c}
x{_1} a{{_1}}ˏ{_1} + \frac{1}{2} x{_2} a{{_1}}ˏ{_2} + \frac{1}{2} x{_2} a{{_2}}ˏ{_1} + \frac{1}{2} x{_3} a{{_1}}ˏ{_3} + \frac{1}{2} x{_3} a{{_3}}ˏ{_1} \\
x{_2} a{{_2}}ˏ{_2} + \frac{1}{2} x{_1} a{{_1}}ˏ{_2} + \frac{1}{2} x{_1} a{{_2}}ˏ{_1} + \frac{1}{2} x{_3} a{{_2}}ˏ{_3} + \frac{1}{2} x{_3} a{{_3}}ˏ{_2} \\
x{_3} a{{_3}}ˏ{_3} + \frac{1}{2} x{_1} a{{_1}}ˏ{_3} + \frac{1}{2} x{_1} a{{_3}}ˏ{_1} + \frac{1}{2} x{_2} a{{_2}}ˏ{_3} + \frac{1}{2} x{_2} a{{_3}}ˏ{_2} \\
\end{array}
\right]
\end{equation}

```julia
f_expr = build_function(Symbolics.gradient(X'A*X/2, X), A, X)
```

    (:(function (ˍ₋arg1, ˍ₋arg2)
          #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:282 =#
          #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:283 =#
          let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
              #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:375 =#
              (SymbolicUtils.Code.create_array)(typeof(ˍ₋arg1), nothing, Val{1}(), Val{(3,)}(), (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]"))), (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]"))), (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]"))))
          end
      end), :(function (ˍ₋out, ˍ₋arg1, ˍ₋arg2)
          #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:282 =#
          #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:283 =#
          let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
              #= D:\.julia\packages\Symbolics\fd3w9\src\build_function.jl:373 =#
              #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:329 =# @inbounds begin
                      #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:325 =#
                      ˍ₋out[1] = (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]")))
                      ˍ₋out[2] = (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]")))
                      ˍ₋out[3] = (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]")))
                      #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:327 =#
                      nothing
                  end
          end
      end))

```julia
f_expr[1] |> Base.remove_linenums!
```

    :(function (ˍ₋arg1, ˍ₋arg2)
          let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
              (SymbolicUtils.Code.create_array)(typeof(ˍ₋arg1), nothing, Val{1}(), Val{(3,)}(), (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]"))), (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]"))), (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]"))))
          end
      end)

```julia
f = eval(f_expr[1])
```

    #1 (generic function with 1 method)

```julia
AA = SA[
    2 -1 0
    -1 2 -1
    0 -1 2
]
```

    3×3 SMatrix{3, 3, Int64, 9} with indices SOneTo(3)×SOneTo(3):
      2  -1   0
     -1   2  -1
      0  -1   2

```julia
XX = SVector(X...)
```

\begin{equation}
\left[
\begin{array}{c}
x{_1} \\
x{_2} \\
x{_3} \\
\end{array}
\right]
\end{equation}

```julia
f(AA, XX) |> typeof
```

    SVector{3, Num} (alias for SArray{Tuple{3}, Num, 1, 3})

```julia
string(f_expr[1]) |> print
```

    function (ˍ₋arg1, ˍ₋arg2)
        let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
            (SymbolicUtils.Code.create_array)(typeof(ˍ₋arg1), nothing, Val{1}(), Val{(3,)}(), (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]"))), (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]"))), (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]"))))
        end
    end

```julia
g_expr = build_function(Symbolics.gradient(X'A*X/2, X), A, X; expression=Val(false))
```

    (RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋arg1, :ˍ₋arg2), Symbolics.var"#_RGF_ModTag", Symbolics.var"#_RGF_ModTag", (0x9a8745f3, 0xc6f732c3, 0x31df4e8c, 0x4a5420e1, 0xa9b39910)}(quote
        #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:282 =#
        #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:283 =#
        let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
            #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:375 =#
            (SymbolicUtils.Code.create_array)(typeof(ˍ₋arg1), nothing, Val{1}(), Val{(3,)}(), (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]"))), (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]"))), (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]"))))
        end
    end), RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋out, :ˍ₋arg1, :ˍ₋arg2), Symbolics.var"#_RGF_ModTag", Symbolics.var"#_RGF_ModTag", (0xc5f304e4, 0xcebd6a78, 0x1617d93f, 0x8bbba735, 0x070674d5)}(quote
        #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:282 =#
        #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:283 =#
        let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
            #= D:\.julia\packages\Symbolics\fd3w9\src\build_function.jl:373 =#
            #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:329 =# @inbounds begin
                    #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:325 =#
                    ˍ₋out[1] = (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]")))
                    ˍ₋out[2] = (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]")))
                    ˍ₋out[3] = (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]")))
                    #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:327 =#
                    nothing
                end
        end
    end))

```julia
g_expr[1] |> Base.remove_linenums!
```

    RuntimeGeneratedFunction(#=in Symbolics=#, #=using Symbolics=#, :((ˍ₋arg1, ˍ₋arg2)->begin
              #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:282 =#
              #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:283 =#
              let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
                  #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:375 =#
                  (SymbolicUtils.Code.create_array)(typeof(ˍ₋arg1), nothing, Val{1}(), Val{(3,)}(), (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]"))), (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]"))), (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]"))))
              end
          end))

```julia
typeof(g_expr[1])
```

    RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋arg1, :ˍ₋arg2), Symbolics.var"#_RGF_ModTag", Symbolics.var"#_RGF_ModTag", (0x9a8745f3, 0xc6f732c3, 0x31df4e8c, 0x4a5420e1, 0xa9b39910)}

```julia
g = g_expr[1]
```

    RuntimeGeneratedFunction(#=in Symbolics=#, #=using Symbolics=#, :((ˍ₋arg1, ˍ₋arg2)->begin
              #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:282 =#
              #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:283 =#
              let var"a[1, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[1]), var"a[2, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[2]), var"a[3, 1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[3]), var"a[1, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[4]), var"a[2, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[5]), var"a[3, 2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[6]), var"a[1, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[7]), var"a[2, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[8]), var"a[3, 3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg1[9]), var"x[1]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[1]), var"x[2]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[2]), var"x[3]" = #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:169 =# @inbounds(ˍ₋arg2[3])
                  #= D:\.julia\packages\SymbolicUtils\Hwe4r\src\code.jl:375 =#
                  (SymbolicUtils.Code.create_array)(typeof(ˍ₋arg1), nothing, Val{1}(), Val{(3,)}(), (+)((+)((*)(var"x[1]", var"a[1, 1]"), (*)(1//2, var"x[2]", var"a[1, 2]"), (*)(1//2, var"x[2]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[1, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 1]"))), (+)((+)((*)(var"x[2]", var"a[2, 2]"), (*)(1//2, var"x[1]", var"a[1, 2]"), (*)(1//2, var"x[1]", var"a[2, 1]"), (*)(1//2, var"x[3]", var"a[2, 3]")), (+)((*)(1//2, var"x[3]", var"a[3, 2]"))), (+)((+)((*)(var"x[3]", var"a[3, 3]"), (*)(1//2, var"x[1]", var"a[1, 3]"), (*)(1//2, var"x[1]", var"a[3, 1]"), (*)(1//2, var"x[2]", var"a[2, 3]")), (+)((*)(1//2, var"x[2]", var"a[3, 2]"))))
              end
          end))

```julia
g(AA, XX) |> typeof
```

    SVector{3, Num} (alias for SArray{Tuple{3}, Num, 1, 3})

```julia

```