```python
%%html

<p> Code is hidden for ease of viewing. Click the Show/Hide button to see. </>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>
```

 Code is hidden for ease of viewing. Click the Show/Hide button to see. </>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

```python
from IPython.display import display, Math, Latex, HTML
from helper import *
%matplotlib inline
```

# Graphing two variables

** or **

## Understanding the Relationship Between Two Variables Using Graphs
***

*GIF taken from https://giphy.com/gifs/reaction-BmmfETghGOPrW, June 11th, 2018.*

***

## **Introduction**

You may have heard the old saying, "you get what you give".  It means what you get is worth the same as what you give up for it.  But have you ever noticed that if you *change* what you give, it can *change* what you get?  Like when you eat more food, you get more energy, or when you spend more time on homework, you get better grades.

In these examples, the things you get (or give) are like **variables**—they are amounts that are allowed to change.  Sometimes, changing the value of one variable can affect the value of another.  Think about this example:

> ### Example
> Say you have a job that pays you 10 dollars per hour. The amount of money you earn after working 5 hours depends on this wage (5 hours)x(10 dollars per hour) = 50 dollars. If your wage goes up to 12 dollars per hour, the amount of money you would make in 5 hours would change as well (5 hours)x(12 dollars per hour) = 60 dollars. So, your wage is a *variable* that determines how much money you are going to make.

Variables are a part of our lives every day, but their exact effect can be hard to understand on their own.  A useful tool to help us see how variables affect results is called a **graph**, which is simply a visual way to show how variables are related.

This lesson will focus on how we can use graphs to improve our understanding of how two variables interact with each other. There are many different types of graphs, but we will focus on line graphs in this lesson to show you how to work with **linear functions**. We will go through:
- how to create a graph of a linear function
- how to analyze some important features of a graph

Take a look at the graph shown below. This is a line graph that shows the relationship between a person's wage and how much money they earn after working 5 hours. You can use the toolbar at the top of the graph to play with it.

```python
wages = np.linspace(0,50,11)
money = []
for item in wages:
    y = item*5
    money.append(y)

layout = go.Layout(
    title = "Money Earned After Working 5 Hours",
    yaxis = dict(
        title="Money Earned (Dollars)"
    ),
    xaxis = dict(
        title="Wage (Dollars per hour)",
    ),
)

trace1 = go.Scatter(
    x = wages,
    y = money,
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(205, 12, 24)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)
```

 Figure 1: This graph shows the relationship between a person's wage, and the money they will make after working for 5 hours.

***

## **Background**

### **Algebraic Equations**

In mathematics, the relationship between variables can be written as an **equation**. An equation is like a sentence—it uses an 'equals' sign to connect two expressions, like how a sentence uses a verb to connect a subject and an object.  For example, the relationship between wage and the amount of money made after working for 5 hours, as described above, can be written like this:

\begin{equation}
\label{1}
\rm MONEY\: MADE = 5\: HOURS \times WAGE
\end{equation}

Here is the same equation, only now it is expressed using **algebra** (with letters in place of the words):

\begin{equation}
\label{2}
M = 5W
\end{equation}

This is an example of an **algebraic equation**.  It compresses all the information about the variables into a very small space that's easy to work with. The $W$ corresponds to wage, and the $M$ corresponds to the resulting amount of money made.

When creating an algebraic expression, the letters that you use for variables really doesn't matter. A good choice is to use the first letter of the word (like how we used $M$ for money and $W$ for wage). However, in mathematics, the default letters are $x$ and $y$, especially when a graph will be used.

In most of the equations you'll see in class, the variables may not have any real-world meaning, but that's okay. Take a look at the examples below—each one is an algebraic equation.

>#### **Examples**
>
>\begin{equation}
y = 3x + 9
\end{equation}
>
>\begin{equation}
y = -5x
\end{equation}
>
>\begin{equation}
y = \frac{x}{4} + 3
\end{equation}
>
>\begin{equation}
x = \frac{y + 7}{2}
\end{equation}
***

### **Linear Functions**

If an equation says a variable is equal to something that doesn't involve that variable, that equation is called a **function**. For example, the equation $y=2x-3$ says that $y$ (a variable) is equal to $2x-3$ (which uses $x$, but not $y$). In this case, we say $y$ is a *function* of $x$.  A **linear function** has the form

\begin{equation}
y = mx + b
\end{equation}

where both $x$ and $y$ are variables, while $m$ and $b$ are **constants**. A *constant* is a number in an equation that does not change—in other words, it is not a variable. A constant can be any number—positive, negative, a fraction, etc.  It can even be zero!

Take another look at the equations shown in the section above, these are all linear functions! When a linear function is used to create a graph, the graph will look like a straight line (hence the name *linear* function).

Use the section below to better your understanding of the difference between a *linear* function, and a *non-linear* function.

```python
display("Is this function linear or non-linear?")
display(Math('y = 12x -3'))

interact(lin_or_non1, val = widgets.Dropdown(options=[' ', 'Linear', 'Non-Linear'],value = ' ',description = 'Choose One:',disabled = False));

```

    'Is this function linear or non-linear?'

$\displaystyle y = 12x -3$

    interactive(children=(Dropdown(description='Choose One:', options=(' ', 'Linear', 'Non-Linear'), value=' '), O…

```python
display("Is this function linear or non-linear?")
display(Math('y = 2x^2 + 17'))

interact(lin_or_non2, val = widgets.Dropdown(options=[' ', 'Linear', 'Non-Linear'],value = ' ',description = 'Choose One:',disabled = False));

```

    'Is this function linear or non-linear?'

$\displaystyle y = 2x^2 + 17$

    interactive(children=(Dropdown(description='Choose One:', options=(' ', 'Linear', 'Non-Linear'), value=' '), O…

```python
x = np.linspace(-10,10,21)
y = []
for num in x:
    y_val = num**2
    y.append(y_val)

layout = go.Layout(
    title = "<i>Y</i> as a Function of <i>X</i>",
    yaxis = dict(
        title="<i>Y</i> Values"
    ),
    xaxis = dict(
        title="<i>X</i> Values"
    ),
)

trace1 = go.Scatter(
    x = x,
    y = y,
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(22, 96, 167)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)
```

```python
display("Is the above function linear or non-linear?")

interact(lin_or_non3, val = widgets.Dropdown(options=[' ', 'Linear', 'Non-Linear'],value = ' ',description = 'Choose One:',disabled = False));
```

    'Is the above function linear or non-linear?'

    interactive(children=(Dropdown(description='Choose One:', options=(' ', 'Linear', 'Non-Linear'), value=' '), O…

```python
x = np.linspace(-10,4,10)
y = []
for num in x:
    y_val = -num - 6
    y.append(y_val)

layout = go.Layout(
    title = "<i>Y</i> as a Function of <i>X</i>",
    yaxis = dict(
        title="<i>Y</i> Values"
    ),
    xaxis = dict(
        title="<i>X</i> Values"
    ),
)

trace1 = go.Scatter(
    x = x,
    y = y,
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(100, 100, 100)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)

```

```python
display("Is the above function linear or non-linear?")

interact(lin_or_non4, val = widgets.Dropdown(options=[' ', 'Linear', 'Non-Linear'],value = ' ',description = 'Choose One:',disabled = False));
```

    'Is the above function linear or non-linear?'

    interactive(children=(Dropdown(description='Choose One:', options=(' ', 'Linear', 'Non-Linear'), value=' '), O…

Now that we know how linear functions can look, let's talk about how we can draw them on a graph!

## **Creating a Graph**

### **Independent and Dependent Variables**

When you want to graph a linear function, you need to figure out which variable is **independent** and which one is **dependent**. The *independent* variable is the one that is allowed to change freely; the *dependent* variable is the one that responds to those changes—in other words, it *depends* on the independent variable. The two variables form an "if-then" pair: "*IF* the independent variable is this, *THEN* the dependent variable must be that." The difference between the two types of variables may be difficult to understand at first, so here is an example that may help.
***
>#### **Example**
Let's say you want to measure how many push-ups you can do in a certain amount of time. There are two variables here: the amount of time, and the number of push-ups you do in that time. In this scenario, you are able to freely change how much time you have to do push-ups. You could give yourself 30 seconds, an hour, or a day! Therefore, *time is the independent variable*.
But the number of push-ups you are actually able to do *depends* on the amount of time you have to do them. If you give yourself more time, you can do more push-ups.  Therefore, the *dependent variable is the number of push-ups you do*.
***
When the two variables in an equation are $x$ and $y$, we usually treat $x$ as the independent variable, and $y$ as the dependent variable.
***

```python
display("From the wage example shown in the section above, which variable do you think is the independent variable?")
display(Math('M = 5W'))

interact(wage, val = widgets.Dropdown(options=[' ', 'W (Wage)', 'M (Money Made)'],value = ' ',description = 'Choose One:',disabled = False));

```

    'From the wage example shown in the section above, which variable do you think is the independent variable?'

$\displaystyle M = 5W$

    interactive(children=(Dropdown(description='Choose One:', options=(' ', 'W (Wage)', 'M (Money Made)'), value='…

***
### **The Cartesian Plane**

Once we have established the independent and dependent variables in our linear function, we are ready to set up a graph of their relationship. When graphing two variables, we use what is called the **Cartesian Plane**. You have probably seen a graph on the Cartesian Plane before, but here is what it looks like:

```python
layout = go.Layout(
    title = "The Cartesian Plane",
    yaxis = dict(
        title="<i>Y</i> Axis",
        range = [-10,10],
        dtick = 1.00
    ),
    xaxis = dict(
        title="<i>X</i> Axis",
        range = [-10,10],
        dtick = 1.00
    ),
)

trace1 = go.Scatter(
    x = [5,-5,-5,5],
    y = [5,5,-5,-5],
    mode = "markers+text",
    name = "Plot 1",
    text=['Quadrant 1', 'Quadrant 2', 'Quadrant 3', 'Quadrant 4'],
    textposition='bottom center'
)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)
```

 Figure 2: This is the standard layout for the Cartesian Plane, where you will plot all linear functions. The point where $x = 0$ and $y = 0$ is called the origin.

On the Cartesian Plane, there are two **axes**: the $x$ axis (horizontal), and the $y$ axis (vertical). When we want to graph a certain function, we label the $y$ axis with the dependent variable, and the $x$ axis with the independent variable. When you aren't given a specific name for what the independent and dependent variable represent, it is okay to just label the $x$ axis "$X$" and the $y$ axis "$Y$".

When making a title for the graph, just describe what is being shown. Your title should allow whoever is looking at the graph to understand what is being shown. In most cases, you will describe the dependent variable as a function of the independent variable, or "$Y$ as a Function of $X$".

The Cartesian Plane consists of four **quadrants**, or sections. The top right section, where both $x$ and $y$ are positive (+,+), is quadrant 1. Then going counter-clockwise, quadrant 2 is the section where $x$ is negative and $y$ is positive (-,+), in quadrant 3 both $x$ and $y$ are negative (-,-), and in quadrant 4, $x$ is positive and $y$ is negative (+,-).

Now that we have established how a graph should look, let's talk about how we decide which values we want to plot on the graph!

### **Finding the Data Points**

The data points that are shown on a graph don't just appear out of nowhere; they can come from scientific observations, measurements, or mathematical relationships. When graphing on the Cartesian Plane, each data point has an $x$-value and a $y$-value (sometimes called an $x$-coordinate and a $y$-coordinate), and these values are what determine where the point will appear on the graph.

The best way to describe how we find the $x$-value and $y$-value for each data point is through an example.

***
#### **Example:** Plot a graph of $y = 3x + 2$ with 5 data points, starting from $x = 1$ up to $x = 5$.

To begin, we pick 5 $x$-values between $1$ and $5$.

For simplicity, let's pick 1, 2, 3, 4, and 5.

\begin{array}{| c | c |}
\hline
  X\: Value & Y\: Value  \\
  \hline
  1 & ? \\\hline
  2 & ? \\\hline
  3 & ? \\\hline
  4 & ? \\\hline
  5 & ? \\\hline
\end{array}

Now we need to find the value of $y$ at each one of the chosen values for $x$. We can do this by plugging our chosen $x$-values into the equation that we have been given: $y = 3x + 2$.

Let's take our first $x$-value, which is 1, and plug it into the equation:

\begin{equation}
y = 3x + 2
\end{equation}

\begin{equation}
y = 3(1) + 2
\end{equation}
\begin{equation}
y = 3 + 2
\end{equation}
\begin{equation}
y = 5
\end{equation}

We now have the $y$-value of our first data point! When $x = 1$, $y = 5$. We would write this data point down as (1,5). Let's fill in the table so we can keep track of the data points we find:

\begin{array}{| c | c |}
\hline
  X\: Value  & Y\: Value  \\
  \hline
  1 & 5 \\\hline
  2 & ? \\\hline
  3 & ? \\\hline
  4 & ? \\\hline
  5 & ? \\\hline
\end{array}

In the section below, use the above method to find the remaining $y$ values.

```python
check_input_numbers()
```

    The equation is: y = 3x + 2

    IntText(value=5, description='When x = 1, y = ', style=DescriptionStyle(description_width='initial'))

    IntText(value=0, description='When x = 2, y = ', style=DescriptionStyle(description_width='initial'))

    IntText(value=0, description='When x = 3, y = ', style=DescriptionStyle(description_width='initial'))

    IntText(value=0, description='When x = 4, y = ', style=DescriptionStyle(description_width='initial'))

    IntText(value=0, description='When x = 5, y = ', style=DescriptionStyle(description_width='initial'))

    Button(description='Check results', style=ButtonStyle())

    HTML(value='')

Good job! Now that we have all of the data points, we are ready to plot them on the graph.

### **Plotting the Data Points on the Graph**

To plot our data points on a graph, we will need our Cartesian Plane, with the $x$-axis and $y$-axis labelled as follows:

```python
layout = go.Layout(
    title = "<i>Y</i> as a Function of <i>X</i>",
    yaxis = dict(
        title="<i>Y</i> Values",
        range = [-1,20],
        dtick = 1.00
    ),
    xaxis = dict(
        title="<i>X</i> Values",
        range = [-1,10],
        dtick = 1.00
    )
)

trace1 = go.Scatter(
    x = [],
    y = [],
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(205, 12, 24)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)

```

 Figure 3: The layout of our graph. As all of our data points have positive $x$ and $y$ values, we only need to focus on the first quadrant (top right corner).

Now, using our chart of data points, let's plot our first point. The $x$-value of our first point is 1, and the $y$-value is 5. So let's find the point on the graph where $x=1$ and $y=5$ and mark this point

```python
layout = go.Layout(
    title = "<i>Y</i> as a Function of <i>X</i>",
    yaxis = dict(
        title="<i>Y</i> Values",
        range = [-1,20],
        dtick = 1.00
    ),
    xaxis = dict(
        title="<i>X</i> Values",
        range = [-1,10],
        dtick = 1.00
    )
)

trace1 = go.Scatter(
    x = [1],
    y = [5],
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(205, 12, 24)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)
```

 Figure 4: Here is the location of our first data point.

That is your first data point!

In the section below, use the above methods to create a graph and plot all of the data points.

\begin{array}{| c | c |}
\hline
  X\: Value & Y\: Value \\
  \hline
  1 & 5 \\\hline
  2 & 8 \\\hline
  3 & 11 \\\hline
  4 & 14 \\\hline
  5 & 17 \\\hline
\end{array}

```python
g = User_Graph([-1,10],[-1,20])
g.run_it()
```

    Enter the x-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Enter the y-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Button(description='Generate plot', style=ButtonStyle())

    HTML(value='')

If done correctly, your graph should look something like this:

```python
layout = go.Layout(
    title = "<i>Y</i> as a Function of <i>X</i>",
    yaxis = dict(
        title="<i>Y</i> Values",
        range = [-1,20]
    ),
    xaxis = dict(
        title="<i>X</i> Values",
        range = [-1,10]
    )
)

trace1 = go.Scatter(
    x = [1,2,3,4,5],
    y = [5,8,11,14,17],
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(205, 12, 24)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)
```

If the graph you made looks significantly different, try running the code segment again to re-try!

Now that you know how to create a graph, let's talk about some of the important things that a graph can tell us.

***

**Features of a Linear Graph: Slope**

An important characteristic of a linear function is **slope**. The *slope* of a function can be thought of as the steepness or the angle of the function as it appears on a graph. What it really represents is the **rate of change** as you move along the graph.

That may sound complicated at first, but here is an example that might make it easier to understand what slope is.

***

> ### **Example**

Say you have been given a set of data points describing how far a car has travelled in a given amount of time:

\begin{array}{| c | c |}
\hline
  Time(seconds) & Distance Travelled(meters) \\
  \hline
  0 & 0 \\\hline
  20 & 400 \\\hline
  40 & 800 \\\hline
  60 & 1200 \\\hline
  80 & 1600 \\\hline
  100 & 2000 \\\hline
  120 & 2400 \\\hline
  140 & 2800 \\\hline
  160 & 3200 \\\hline
  180 & 3600 \\\hline
  200 & 4000 \\\hline
\end{array}

If you want to find out how fast the car is travelling, you are really trying to find out how quickly the distance travelled increases, or rather, the **rate** at which it is changing.

Let's set up a graph of this data to take a closer look.

```python
time = np.linspace(0,200,11)
distance = np.linspace(0,4000,11)

layout = go.Layout(
    title = "Distance Travelled as a Function of Time",
    yaxis = dict(
        title="Distance Travelled (Meters)"
    ),
    xaxis = dict(
        title="Time (Seconds)"
    ),
)

trace1 = go.Scatter(
    x = time,
    y = distance,
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(22, 96, 167)'),
        shape = "spline",
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)
```

 Figure 6: This graph shows the distance a car travels after a given amount of time.

We want to find how fast the car is travelling. Knowing that speed is a measurement of distance over time, let's use the total distance travelled and the total amount of time to find the average speed of the car.
From the graph, we can see that the car initially started at position 0 meters, and travelled a total of 4000 meters. We also know that the car started moving at time 0 seconds, and stopped moving after 200 seconds. Using this information, we can set up the following equations to find the speed of the car:

\begin{equation}
Speed = \frac{Total\: Distance}{Total\: Time}
\end{equation}

and we know that:

\begin{equation}
Total\: Distance = Final\: Distance - Initial\: Distance
\end{equation}
\begin{equation}
Total\: Time = Final\: Time - Initial\: Time
\end{equation}

so:

\begin{equation}
Speed = \frac{Final\: Distance - Initial\: Distance}{Final\: Time - Initial\: Time}
\end{equation}

\begin{equation}
Speed = \frac{4000m - 0m}{200s - 0s}
\end{equation}

\begin{equation}
Speed = \frac{4000m}{200s}
\end{equation}

\begin{equation}
 Speed = 20 m/s
\end{equation}

From the equations above, we can see that the car travels 20 meters every second. While it may not be obvious, we just found the slope of the graph! By finding the average speed of the car, we identified the rate at which the dependent variable (Distance Travelled) changes with respect to the independent variable (Time).

***

When looking at a graph, the general equation used to find the slope of a linear function from one point: ($x_1$,$y_1$), to a second point: ($x_2$,$y_2$), is given as follows:

\begin{equation}
Slope = \frac{y_2 - y_1}{x_2 - x_1}
\end{equation}

When looking at a linear function that has the form:
\begin{equation}
y = mx + b
\end{equation}
the value of the slope is the same as the value of $m$.

Use the section below to see the effect that slope can have on a function.

```python
def slider(slope):

    x_list = [0,1,2,3,4,5,6,7,8,9,10]
    y_list = []
    for i in x_list:
        y_val = slope*(i)
        y_list.append(y_val)

    layout = go.Layout(
        showlegend = True,
        title ='y = ' + str(slope) + 'x',
        yaxis = dict(
            title="<i>X</i> Values",
            range = [-1,20]
        ),
        xaxis = dict(
            title="<i>Y</i> Values",
            range = [-1,20]
        ),
    )

    trace1 = go.Scatter(
        x = x_list,
        y = y_list,
        mode = "lines+markers",
        name = str('$y = ' + str(slope) + 'x$'),
        line=dict(
            color = ('rgb(22, 96, 167)'),
            shape = "spline",
            dash = "dot"
        )

    )

    fig = go.Figure(data = [trace1], layout = layout)
    py.offline.iplot(fig, filename = 'show-legend')

interactive(slider, slope=(0,5,0.5))

```

    interactive(children=(FloatSlider(value=2.0, description='slope', max=5.0, step=0.5), Output()), _dom_classes=…

 Figure 7: This graph shows the way in which slope can affect the shape of a function. The larger the slope, the "steeper" the graph is.

When dealing with a single linear function, the slope will be constant across the entire graph. However, if a graph consists of more than one linear function, the slope may change at some point.
Take a look at the following graph to see what a graph with more than one linear function may look like:

```python
x1 = [0,1,2,3,4,5]
y1 = []

for num in x1:
    y1_val = num + 1
    y1.append(y1_val)

x2 = [6,7,8,9,10]
y2 = []

for num in x2:
    y2_val = 3*(num) - 9
    y2.append(y2_val)

x_list = x1 + x2
y_list = y1 + y2

layout = go.Layout(
    title = "<i>Y</i> as a Function of <i>X</i>",
    yaxis = dict(
        title="<i>Y</i> Values"
    ),
    xaxis = dict(
        title="<i>X</i> Values"
    ),
)

trace1 = go.Scatter(
    x = x_list,
    y = y_list,
    mode = "lines+markers",
    name = "Plot 1",
    line=dict(
        color = ('rgb(22, 96, 167)'),
        dash = "dot"
    )

)

fig = go.Figure(data = [trace1], layout = layout)
py.offline.iplot(fig)

```

 Figure 8: On this plot, notice the distinct change in steepness at $x = 5$. The slope increases at this point.

As you can see above, the graph has two distinct sections: one from $x = 0$ to $x = 5$, and the other from $x = 5$ to $x = 10$. As we have seen from the equations of linear functions, we know that the slope of one function has a constant value. Because these sections each have a different slope, they must represent two distinct linear functions.

You may be wondering how one graph can show two different functions at the same time, but consider our example of the car shown above. What if halfway through the trip, the car began to move at a faster speed? If this were the case, the slope of the graph would increase at the halfway point, much like the graph shown above.

## **Questions**

The following section is meant to test your understanding and improve your familiarity with the concepts we have gone over in this lesson.

### **Question 1.**

Use the following table of data points describing the motion of a car driving to a destination and complete the following tasks:
- Plot a graph (be sure to label the axes, and to give the graph a title!)
- Find the slope of the graph
- Create an equation for the linear function

\begin{array}{| c | c |}
\hline
  Time\: (seconds) & Distance\: Away\: from\: Destination\: (meters) \\
  \hline
  0 & 10 \\\hline
  1 & 8 \\\hline
  2 & 6 \\\hline
  3 & 4 \\\hline
  4 & 2 \\\hline
  5 & 0 \\\hline
  6 & -2 \\\hline
  7 & -4 \\\hline
  8 & -6 \\\hline
  9 & -8 \\\hline
\end{array}

Use the section below to create the graph:

```python
q1 = User_Graph([-1,10], [-10,12])
q1.run_it()
```

    Enter the x-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Enter the y-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Button(description='Generate plot', style=ButtonStyle())

    HTML(value='')

Use the following sections to answer some questions about the graph you just made:

```python
ask_slope()
```

    What is the value of the slope of the above graph?

    IntText(value=0)

    Button(description='Check results', style=ButtonStyle())

    HTML(value='')

```python
ask_equation()
```

    What is the equation of the linear function shown on the graph? (Enter your answer as: y = mx + b)

    Text(value='')

    Button(description='Check results', style=ButtonStyle())

    HTML(value='')

```python
ask_question1()
```

    At what point did the person walk past their destination? (Enter your answer as (x,y))

    Text(value='')

    Button(description='Check results', style=ButtonStyle())

    HTML(value='')

###### Question 2.

Quentin and Isabelle both earn their allowance by doing exercise. Quentin's parents pay him $\$8.00$ each time he goes for a run, plus an additional $\$0.50$ for every kilometer he runs.

Isabelle gets paid $\$5.00$ every time she goes for a run, and an additional $\$1.00$ for every kilometer she runs.

At what distance do Quentin and Isabelle make the same amount of money?
Who makes the most money after running for 10 kilometers?

To help answer these questions, here are the linear equations that describe Quentin's and Isabelle's allowance:

Quentin:
\begin{equation}
y = \frac{1}{2}x + 8
\end{equation}

Isabelle:
\begin{equation}
y = x + 5
\end{equation}

Create a graph that shows both of these functions at the points:

\begin{array}{| c | c |}
\hline
  Distance\: (km) & Money\: Made\: (Dollars)\\
  \hline
  0 & ? \\\hline
  1 & ? \\\hline
  2 & ? \\\hline
  3 & ? \\\hline
  4 & ? \\\hline
  5 & ? \\\hline
  6 & ? \\\hline
  7 & ? \\\hline
\end{array}

*Hint*: Create one table for each of the equations and find the corresponding $y$-values, then use the section below to enter the points.

```python
D = Double_Graph([0,8],[0,15])
D.run_it()
```

    Please enter data points for Quentin:
    Enter the x-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Enter the y-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Please enter data points for Isabelle:
    Enter the x-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Enter the y-values as a list of the form [a,b,c,d,...]:

    Text(value='')

    Button(description='Generate plot', style=ButtonStyle())

    HTML(value='')

```python
ask_question2()
```

    At what distance do Quentin and Isabelle make the same amount of money? (Enter your answer as a whole number)

    IntText(value=0)

    Button(description='Check results', style=ButtonStyle())

    HTML(value='')

```python
ask_question3()
```

    Who makes the most money after running 10 kilometers? (Enter: Quentin, or Isabelle)

    Text(value='')

    Button(description='Check results', style=ButtonStyle())

    HTML(value='')

## **Conclusion**

In this lesson, we have highlighted some of the many ways that a graph can help us understand the relationship between two variables in a linear function. Some of the important methods that you should remember include:

- Defining a Linear Function
- Identifying the Independent and Dependent Variables
- Setting up a Graph on the Cartesian Plane
- Creating and Plotting Data Points
- Finding the slope of a graph

Throughout your life, you will be presented with all sorts of statistics, data, charts and graphs. It is important that you have the skills required to analyze what a graph is trying to tell you, not just so you can pass your math class, but to enable you to make informed decisions when you need to consider how different variables can interact and affect results.

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


# Tutorial 2: Effects of Input Correlation
**Week 2, Day 3: Biological Neuron Models**

**By Neuromatch Academy**

__Content creators:__ Qinglong Gu, Songtin Li, John Murray, Richard Naud, Arvind Kumar

__Content reviewers:__ Maryam Vaziri-Pashkam, Ella Batty, Lorenzo Fontolan, Richard Gao, Matthew Krause, Spiros Chavlis, Michael Waskom

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

*Estimated timing of tutorial: 50 minutes*

In this tutorial, we will use the leaky integrate-and-fire (LIF) neuron model (see Tutorial 1) to study how they transform input correlations to output properties (transfer of correlations). In particular, we are going to write a few lines of code to:

- inject correlated GWN in a pair of neurons

- measure correlations between the spiking activity of the two neurons

- study how the transfer of correlation depends on the statistics of the input, i.e. mean and standard deviation.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/8djsm/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
# Imports

import matplotlib.pyplot as plt
import numpy as np
import time
```

```python
# @title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
# use NMA plot style
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
my_layout = widgets.Layout()
```

```python
# @title Plotting Functions

def example_plot_myCC():
  pars = default_pars(T=50000, dt=.1)

  c = np.arange(10) * 0.1
  r12 = np.zeros(10)
  for i in range(10):
    I1gL, I2gL = correlate_input(pars, mu=20.0, sig=7.5, c=c[i])
    r12[i] = my_CC(I1gL, I2gL)

  plt.figure()
  plt.plot(c, r12, 'bo', alpha=0.7, label='Simulation', zorder=2)
  plt.plot([-0.05, 0.95], [-0.05, 0.95], 'k--', label='y=x',
           dashes=(2, 2), zorder=1)
  plt.xlabel('True CC')
  plt.ylabel('Sample CC')
  plt.legend(loc='best')

# the function plot the raster of the Poisson spike train
def my_raster_Poisson(range_t, spike_train, n):
  """
  Generates poisson trains

  Args:
    range_t     : time sequence
    spike_train : binary spike trains, with shape (N, Lt)
    n           : number of Poisson trains plot

  Returns:
    Raster plot of the spike train
  """

  # find the number of all the spike trains
  N = spike_train.shape[0]

  # n should smaller than N:
  if n > N:
    print('The number n exceeds the size of spike trains')
    print('The number n is set to be the size of spike trains')
    n = N

  # plot rater
  i = 0
  while i < n:
    if spike_train[i, :].sum() > 0.:
      t_sp = range_t[spike_train[i, :] > 0.5]  # spike times
      plt.plot(t_sp, i * np.ones(len(t_sp)), 'k|', ms=10, markeredgewidth=2)
    i += 1
  plt.xlim([range_t[0], range_t[-1]])
  plt.ylim([-0.5, n + 0.5])
  plt.xlabel('Time (ms)', fontsize=12)
  plt.ylabel('Neuron ID', fontsize=12)

def plot_c_r_LIF(c, r, mycolor, mylabel):
  z = np.polyfit(c, r, deg=1)
  c_range = np.array([c.min() - 0.05, c.max() + 0.05])
  plt.plot(c, r, 'o', color=mycolor, alpha=0.7, label=mylabel, zorder=2)
  plt.plot(c_range, z[0] * c_range + z[1], color=mycolor, zorder=1)
```

```python
# @title Helper Functions
def default_pars(**kwargs):
  pars = {}

  ### typical neuron parameters###
  pars['V_th'] = -55.     # spike threshold [mV]
  pars['V_reset'] = -75.  # reset potential [mV]
  pars['tau_m'] = 10.     # membrane time constant [ms]
  pars['g_L'] = 10.       # leak conductance [nS]
  pars['V_init'] = -75.   # initial potential [mV]
  pars['V_L'] = -75.      # leak reversal potential [mV]
  pars['tref'] = 2.       # refractory time (ms)

  ### simulation parameters ###
  pars['T'] = 400. # Total duration of simulation [ms]
  pars['dt'] = .1  # Simulation time step [ms]

  ### external parameters if any ###
  for k in kwargs:
    pars[k] = kwargs[k]

  pars['range_t'] = np.arange(0, pars['T'], pars['dt'])  # Vector of discretized
                                                         # time points [ms]
  return pars

def run_LIF(pars, Iinj):
  """
  Simulate the LIF dynamics with external input current

  Args:
    pars       : parameter dictionary
    Iinj       : input current [pA]. The injected current here can be a value or an array

  Returns:
    rec_spikes : spike times
    rec_v      : mebrane potential
  """

  # Set parameters
  V_th, V_reset = pars['V_th'], pars['V_reset']
  tau_m, g_L = pars['tau_m'], pars['g_L']
  V_init, V_L = pars['V_init'], pars['V_L']
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size
  tref = pars['tref']

  # Initialize voltage and current
  v = np.zeros(Lt)
  v[0] = V_init
  Iinj = Iinj * np.ones(Lt)
  tr = 0.

  # simulate the LIF dynamics
  rec_spikes = []   # record spike times
  for it in range(Lt - 1):
    if tr > 0:
      v[it] = V_reset
      tr = tr - 1
    elif v[it] >= V_th:  # reset voltage and record spike event
      rec_spikes.append(it)
      v[it] = V_reset
      tr = tref / dt

    # calculate the increment of the membrane potential
    dv = (-(v[it] - V_L) + Iinj[it] / g_L) * (dt / tau_m)

    # update the membrane potential
    v[it + 1] = v[it] + dv

  rec_spikes = np.array(rec_spikes) * dt

  return v, rec_spikes

def my_GWN(pars, sig, myseed=False):
  """
  Function that calculates Gaussian white noise inputs

  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    myseed     : random seed. int or boolean
                 the same seed will give the same random number sequence

  Returns:
    I          : Gaussian white noise input
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # Set random seed. You can fix the seed of the random number generator so
  # that the results are reliable however, when you want to generate multiple
  # realization make sure that you change the seed for each new realization
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate GWN
  # we divide here by 1000 to convert units to sec.
  I_GWN = sig * np.random.randn(Lt) * np.sqrt(pars['tau_m'] / dt)

  return I_GWN

def LIF_output_cc(pars, mu, sig, c, bin_size, n_trials=20):
  """ Simulates two LIF neurons with correlated input and computes output correlation

  Args:
  pars       : parameter dictionary
  mu         : noise baseline (mean)
  sig        : noise amplitute (standard deviation)
  c          : correlation coefficient ~[0, 1]
  bin_size   : bin size used for time series
  n_trials   : total simulation trials

  Returns:
  r          : output corr. coe.
  sp_rate    : spike rate
  sp1        : spike times of neuron 1 in the last trial
  sp2        : spike times of neuron 2 in the last trial
  """

  r12 = np.zeros(n_trials)
  sp_rate = np.zeros(n_trials)
  for i_trial in range(n_trials):
    I1gL, I2gL = correlate_input(pars, mu, sig, c)
    _, sp1 = run_LIF(pars, pars['g_L'] * I1gL)
    _, sp2 = run_LIF(pars, pars['g_L'] * I2gL)

    my_bin = np.arange(0, pars['T'], bin_size)

    sp1_count, _ = np.histogram(sp1, bins=my_bin)
    sp2_count, _ = np.histogram(sp2, bins=my_bin)

    r12[i_trial] = my_CC(sp1_count[::20], sp2_count[::20])
    sp_rate[i_trial] = len(sp1) / pars['T'] * 1000.

  return r12.mean(), sp_rate.mean(), sp1, sp2
```

The helper function contains the:

- Parameter dictionary: `default_pars( **kwargs)` from Tutorial 1
- LIF simulator: `run_LIF` from Tutorial 1
- Gaussian white noise generator: `my_GWN(pars, sig, myseed=False)` from Tutorial 1
- Poisson type spike train generator: `Poisson_generator(pars, rate, n, myseed=False)`
- Two LIF neurons with correlated inputs simulator: `LIF_output_cc(pars, mu, sig, c, bin_size, n_trials=20)`

---
# Section 1: Correlations (Synchrony)
Correlation or synchrony in neuronal activity can be described for any readout of brain activity. Here, we are concerned with the spiking activity of neurons.

In the simplest way, correlation/synchrony refers to coincident spiking of neurons, i.e., when two neurons spike together, they are firing in **synchrony** or are **correlated**. Neurons can be synchronous in their instantaneous activity, i.e., they spike together with some probability. However, it is also possible that spiking of a neuron at time $t$ is correlated with the spikes of another neuron with a delay (time-delayed synchrony).

## Origin of synchronous neuronal activity:
- Common inputs, i.e., two neurons are receiving input from the same sources. The degree of correlation of the shared inputs is proportional to their output correlation.
- Pooling from the same sources. Neurons do not share the same input neurons but are receiving inputs from neurons which themselves are correlated.
- Neurons are connected to each other (uni- or bi-directionally): This will only give rise to time-delayed synchrony. Neurons could also be connected via gap-junctions.
- Neurons have similar parameters and initial conditions.

## Implications of synchrony
When neurons spike together, they can have a stronger impact on downstream neurons. Synapses in the brain are sensitive to the temporal correlations (i.e., delay) between pre- and postsynaptic activity, and this, in turn, can lead to the formation of functional neuronal networks - the basis of unsupervised learning (we will study some of these concepts in a forthcoming tutorial).

Synchrony implies a reduction in the dimensionality of the system. In addition, correlations, in many cases, can impair the decoding of neuronal activity.

```python
# @title Video 1: Input & output correlations
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1Bh411o7eV", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="nsAYFBcAkes", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

A simple model to study the emergence of correlations is to inject common inputs to a pair of neurons and measure the output correlation as a function of the fraction of common inputs.

Here, we are going to investigate the transfer of correlations by computing the correlation coefficient of spike trains recorded from two unconnected LIF neurons, which received correlated inputs.

The input current to LIF neuron $i$ $(i=1,2)$ is:

\begin{equation}
\frac{I_i}{g_L} =\mu_i + \sigma_i (\sqrt{1-c}\xi_i + \sqrt{c}\xi_c) \quad (1)
\end{equation}

where $\mu_i$ is the temporal average of the current. The Gaussian white noise $\xi_i$ is independent for each neuron, while $\xi_c$ is common to all neurons. The variable $c$ ($0\le c\le1$) controls the fraction of common and independent inputs. $\sigma_i$ shows the variance of the total input.

So, first, we will generate correlated inputs.

```python
# @markdown Execute this cell to get a function `correlate_input` for generating correlated GWN inputs
def correlate_input(pars, mu=20., sig=7.5, c=0.3):
  """
  Args:
    pars       : parameter dictionary
    mu         : noise baseline (mean)
    sig        : noise amplitute (standard deviation)
    c.         : correlation coefficient ~[0, 1]

  Returns:
    I1gL, I2gL : two correlated inputs with corr. coe. c
  """

  # generate Gaussian whute noise xi_1, xi_2, xi_c
  xi_1 = my_GWN(pars, sig)
  xi_2 = my_GWN(pars, sig)
  xi_c = my_GWN(pars, sig)

  # Generate two correlated inputs by Equation. (1)
  I1gL = mu + np.sqrt(1. - c) * xi_1 + np.sqrt(c) * xi_c
  I2gL = mu + np.sqrt(1. - c) * xi_2 + np.sqrt(c) * xi_c

  return I1gL, I2gL

help(correlate_input)
```

    Help on function correlate_input in module __main__:

    correlate_input(pars, mu=20.0, sig=7.5, c=0.3)
        Args:
          pars       : parameter dictionary
          mu         : noise baseline (mean)
          sig        : noise amplitute (standard deviation)
          c.         : correlation coefficient ~[0, 1]

        Returns:
          I1gL, I2gL : two correlated inputs with corr. coe. c

## Coding Exercise 1A: Compute the correlation

The _sample correlation coefficient_ between two input currents $I_i$ and $I_j$ is defined as the sample covariance of $I_i$ and $I_j$ divided by the square root of the sample variance of $I_i$ multiplied with the square root of the sample variance of $I_j$. In equation form:

\begin{align}
r_{ij} &= \frac{cov(I_i, I_j)}{\sqrt{var(I_i)} \sqrt{var(I_j)}}\\
cov(I_i, I_j) &= \sum_{k=1}^L (I_i^k -\bar{I}_i)(I_j^k -\bar{I}_j) \\
var(I_i) &= \sum_{k=1}^L (I_i^k -\bar{I}_i)^2
\end{align}

where $\bar{I}_i$ is the sample mean, k is the time bin, and L is the length of $I$.  This means that $I_i^k$ is current i at time $k\cdot dt$. Note that the equations above are not accurate for sample covariances and variances as they should be additionally divided by L-1 - we have dropped this term because it cancels out in the sample correlation coefficient formula.

The _sample correlation coefficient_ may also be referred to as the _sample Pearson correlation coefficient_. Here, is a beautiful paper that explains multiple ways to calculate and understand correlations [Rodgers and Nicewander 1988](https://www.stat.berkeley.edu/~rabbee/correlation.pdf).

In this exercise, we will create a function, `my_CC` to compute the sample correlation coefficient between two time series. Note that while we introduced this computation here in the context of input currents, the sample correlation coefficient is used to compute the correlation between any two time series - we will use it later on binned spike trains.

We then check our method is accurate by generating currents with a certain correlation (using `correlate_input`), computing the correlation coefficient using `my_CC`, and plotting the true vs sample correlation coefficients.

```python
def my_CC(i, j):
  """
  Args:
    i, j  : two time series with the same length

  Returns:
    rij   : correlation coefficient
  """
  ########################################################################
  ## TODO for students: compute rxy, then remove the NotImplementedError #
  # Tip1: array([a1, a2, a3])*array([b1, b2, b3]) = array([a1*b1, a2*b2, a3*b3])
  # Tip2: np.sum(array([a1, a2, a3])) = a1+a2+a3
  # Tip3: square root, np.sqrt()
  # Fill out function and remove
  #raise NotImplementedError("Student exercise: compute the sample correlation coefficient")
  ########################################################################

  # Calculate the covariance of i and j
  cov = np.sum((i-np.mean(i))*(j-np.mean(j)))

  # Calculate the variance of i
  var_i = np.sum((i-np.mean(i))**2)

  # Calculate the variance of j
  var_j = np.sum((i-np.mean(i))**2)

  # Calculate the correlation coefficient
  rij = cov/(np.sqrt(var_i)*np.sqrt(var_j))

  return rij

example_plot_myCC()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D3_BiologicalNeuronModels/solutions/W2D3_Tutorial2_Solution_313f41e4.py)

*Example output:*

The sample correlation coefficients (computed using `my_CC`) match the ground truth correlation coefficient!

In the next exercise, we will use the Poisson distribution to model spike trains. Remember that you have seen the Poisson distribution used in this way in the [pre-reqs math day on Statistics](https://compneuro.neuromatch.io/tutorials/W0D5_Statistics/student/W0D5_Tutorial1.html#section-2-2-poisson-distribution). Remember that a Poisson spike train has the following properties:
- The ratio of the mean and variance of spike count is 1
- Inter-spike-intervals are exponentially distributed
- Spike times are irregular i.e. 𝐶𝑉ISI=1
- Adjacent spike intervals are independent of each other.

In the following cell, we provide a helper function `Poisson_generator` and then use it to produce a Poisson spike train.

```python
# @markdown Execute this cell to get helper function `Poisson_generator`
def Poisson_generator(pars, rate, n, myseed=False):
  """
  Generates poisson trains

  Args:
    pars       : parameter dictionary
    rate       : noise amplitute [Hz]
    n          : number of Poisson trains
    myseed     : random seed. int or boolean

  Returns:
    pre_spike_train : spike train matrix, ith row represents whether
                      there is a spike in ith spike train over time
                      (1 if spike, 0 otherwise)
  """

  # Retrieve simulation parameters
  dt, range_t = pars['dt'], pars['range_t']
  Lt = range_t.size

  # set random seed
  if myseed:
      np.random.seed(seed=myseed)
  else:
      np.random.seed()

  # generate uniformly distributed random variables
  u_rand = np.random.rand(n, Lt)

  # generate Poisson train
  poisson_train = 1. * (u_rand < rate * (dt / 1000.))

  return poisson_train

help(Poisson_generator)
```

    Help on function Poisson_generator in module __main__:

    Poisson_generator(pars, rate, n, myseed=False)
        Generates poisson trains

        Args:
          pars       : parameter dictionary
          rate       : noise amplitute [Hz]
          n          : number of Poisson trains
          myseed     : random seed. int or boolean

        Returns:
          pre_spike_train : spike train matrix, ith row represents whether
                            there is a spike in ith spike train over time
                            (1 if spike, 0 otherwise)

```python
# @markdown Execute this cell to visualize Poisson spike train

pars = default_pars()
pre_spike_train = Poisson_generator(pars, rate=10, n=100, myseed=2020)
my_raster_Poisson(pars['range_t'], pre_spike_train, 100)
```

## Coding Exercise 1B: Measure the correlation between spike trains

After recording the spike times of the two neurons, how can we estimate their correlation coefficient?

In order to find this, we need to bin the spike times and obtain two time series. Each data point in the time series is the number of spikes in the corresponding time bin. You can use `np.histogram()` to bin the spike times.

Complete the code below to bin the spike times and calculate the correlation coefficient for two Poisson spike trains. Note that `c` here is the ground-truth correlation coefficient that we define.

```python
# @markdown Execute this cell to get a function for generating correlated Poisson inputs (`generate_corr_Poisson`)

def generate_corr_Poisson(pars, poi_rate, c, myseed=False):
  """
  function to generate correlated Poisson type spike trains
  Args:
    pars       : parameter dictionary
    poi_rate   : rate of the Poisson train
    c.         : correlation coefficient ~[0, 1]

  Returns:
    sp1, sp2   : two correlated spike time trains with corr. coe. c
  """

  range_t = pars['range_t']

  mother_rate = poi_rate / c
  mother_spike_train = Poisson_generator(pars, rate=mother_rate,
                                         n=1, myseed=myseed)[0]
  sp_mother = range_t[mother_spike_train > 0]

  L_sp_mother = len(sp_mother)
  sp_mother_id = np.arange(L_sp_mother)
  L_sp_corr = int(L_sp_mother * c)

  np.random.shuffle(sp_mother_id)
  sp1 = np.sort(sp_mother[sp_mother_id[:L_sp_corr]])

  np.random.shuffle(sp_mother_id)
  sp2 = np.sort(sp_mother[sp_mother_id[:L_sp_corr]])

  return sp1, sp2

print(help(generate_corr_Poisson))
```

    Help on function generate_corr_Poisson in module __main__:

    generate_corr_Poisson(pars, poi_rate, c, myseed=False)
        function to generate correlated Poisson type spike trains
        Args:
          pars       : parameter dictionary
          poi_rate   : rate of the Poisson train
          c.         : correlation coefficient ~[0, 1]

        Returns:
          sp1, sp2   : two correlated spike time trains with corr. coe. c

    None

```python
def corr_coeff_pairs(pars, rate, c, trials, bins):
  """
  Calculate the correlation coefficient of two spike trains, for different
  realizations

  Args:
      pars   : parameter dictionary
      rate   : rate of poisson inputs
      c      : correlation coefficient ~ [0, 1]
      trials  : number of realizations
      bins   : vector with bins for time discretization

  Returns:
    r12      : correlation coefficient of a pair of inputs
  """

  r12 = np.zeros(n_trials)

  for i in range(n_trials):
    ##############################################################
    ## TODO for students
    # Note that you can run multiple realizations and compute their r_12(diff_trials)
    # with the defined function above. The average r_12 over trials can get close to c.
    # Note: change seed to generate different input per trial
    # Fill out function and remove
    #raise NotImplementedError("Student exercise: compute the correlation coefficient")
    ##############################################################

    # Generate correlated Poisson inputs
    sp1, sp2 = generate_corr_Poisson(pars, rate, c, myseed=2020+i)

    # Bin the spike times of the first input
    sp1_count, _ = np.histogram(sp1, bins=bins)

    # Bin the spike times of the second input
    sp2_count, _ = np.histogram(sp2, bins=bins)

    # Calculate the correlation coefficient
    r12[i] = my_CC(sp1_count, sp2_count)

  return r12

poi_rate = 20.
c = 0.2  # set true correlation
pars = default_pars(T=10000)

# bin the spike time
bin_size = 20  # [ms]
my_bin = np.arange(0, pars['T'], bin_size)
n_trials = 100  # 100 realizations

r12 = corr_coeff_pairs(pars, rate=poi_rate, c=c, trials=n_trials, bins=my_bin)
print(f'True corr coe = {c:.3f}')
print(f'Simu corr coe = {r12.mean():.3f}')
```

    True corr coe = 0.200
    Simu corr coe = 0.199

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D3_BiologicalNeuronModels/solutions/W2D3_Tutorial2_Solution_e5eaac3e.py)

Sample output

```
True corr coe = 0.200
Simu corr coe = 0.197
```

---
# Section 2: Investigate the effect of input correlation on the output correlation

*Estimated timing to here from start of tutorial: 20 min*

Now let's combine the aforementioned two procedures. We first generate the correlated inputs. Then we inject the correlated inputs $I_1, I_2$ into a pair of neurons and record their output spike times. We continue measuring the correlation between the output and
investigate the relationship between the input correlation and the output correlation.

In the following, you will inject correlated GWN in two neurons. You need to define the mean (`gwn_mean`), standard deviation (`gwn_std`), and input correlations (`c_in`).

We will simulate $10$ trials to get a better estimate of the output correlation. Change the values in the following cell for the above variables (and then run the next cell) to explore how they impact the output correlation.

```python
# Play around with these parameters

pars = default_pars(T=80000, dt=1.)  # get the parameters
c_in = 1.  # set input correlation value
gwn_mean = 10.
gwn_std = 4.
```

```python
# @title

# @markdown Do not forget to execute this cell to simulate the LIF

bin_size = 10.  # ms

starttime = time.perf_counter()  # time clock

r12_ss, sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=gwn_mean, sig=gwn_std, c=c_in,
                                        bin_size=bin_size, n_trials=10)

# just the time counter
endtime = time.perf_counter()
timecost = (endtime - starttime) / 60.
print(f"Simulation time = {timecost:.2f} min")

print(f"Input correlation = {c_in}")
print(f"Output correlation = {r12_ss}")

plt.figure(figsize=(12, 6))
plt.plot(sp1, np.ones(len(sp1)) * 1, '|', ms=20, label='neuron 1')
plt.plot(sp2, np.ones(len(sp2)) * 1.1, '|', ms=20, label='neuron 2')
plt.xlabel('time (ms)')
plt.ylabel('neuron id.')
plt.xlim(1000, 8000)
plt.ylim(0.9, 1.2)
plt.legend()
plt.show()
```

## Think! 2: Input and Output Correlations
- Is the output correlation always smaller than the input correlation? If yes, why?
- Should there be a systematic relationship between input and output correlations?

You will explore these questions in the next figure but try to develop your own intuitions first!

Lets vary `c_in` and plot the relationship between the `c_in` and output correlation. This might take some time depending on the number of trials.

```python
#@title

#@markdown Don't forget to execute this cell!

pars = default_pars(T=80000, dt=1.)  # get the parameters
bin_size = 10.
c_in = np.arange(0, 1.0, 0.1)  # set the range for input CC
r12_ss = np.zeros(len(c_in))  # small mu, small sigma

starttime = time.perf_counter() # time clock
for ic in range(len(c_in)):
  r12_ss[ic], sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=10.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=10)

endtime = time.perf_counter()
timecost = (endtime - starttime) / 60.
print(f"Simulation time = {timecost:.2f} min")

plt.figure(figsize=(7, 6))
plot_c_r_LIF(c_in, r12_ss, mycolor='b', mylabel='Output CC')
plt.plot([c_in.min() - 0.05, c_in.max() + 0.05],
         [c_in.min() - 0.05, c_in.max() + 0.05],
         'k--', dashes=(2, 2), label='y=x')

plt.xlabel('Input CC')
plt.ylabel('Output CC')
plt.legend(loc='best', fontsize=16)
plt.show()
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D3_BiologicalNeuronModels/solutions/W2D3_Tutorial2_Solution_71e76f4d.py)

---
# Section 3: Correlation transfer function

*Estimated timing to here from start of tutorial: 30 min*

The above plot of input correlation vs. output correlation is called the __correlation transfer function__ of the neurons.

## Section 3.1: How do the mean and standard deviation of the Gaussian white noise (GWN) affect the correlation transfer function?

The correlations transfer function appears to be linear. The above can be taken as the input/output transfer function of LIF neurons for correlations, instead of the transfer function for input/output firing rates as we had discussed in the previous tutorial (i.e., F-I curve).

What would you expect to happen to the slope of the correlation transfer function if you vary the mean and/or the standard deviation of the GWN of the inputs ?

```python
# @markdown Execute this cell to visualize correlation transfer functions

pars = default_pars(T=80000, dt=1.) # get the parameters
no_trial = 10
bin_size = 10.
c_in = np.arange(0., 1., 0.2)  # set the range for input CC
r12_ss = np.zeros(len(c_in))   # small mu, small sigma
r12_ls = np.zeros(len(c_in))   # large mu, small sigma
r12_sl = np.zeros(len(c_in))   # small mu, large sigma

starttime = time.perf_counter()  # time clock
for ic in range(len(c_in)):
  r12_ss[ic], sp_ss, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=10.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=no_trial)
  r12_ls[ic], sp_ls, sp1, sp2 = LIF_output_cc(pars, mu=18.0, sig=10.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=no_trial)
  r12_sl[ic], sp_sl, sp1, sp2 = LIF_output_cc(pars, mu=10.0, sig=20.,
                                              c=c_in[ic], bin_size=bin_size,
                                              n_trials=no_trial)
endtime = time.perf_counter()
timecost = (endtime - starttime) / 60.
print(f"Simulation time = {timecost:.2f} min")

plt.figure(figsize=(7, 6))
plot_c_r_LIF(c_in, r12_ss, mycolor='b', mylabel=r'Small $\mu$, small $\sigma$')
plot_c_r_LIF(c_in, r12_ls, mycolor='y', mylabel=r'Large $\mu$, small $\sigma$')
plot_c_r_LIF(c_in, r12_sl, mycolor='r', mylabel=r'Small $\mu$, large $\sigma$')
plt.plot([c_in.min() - 0.05, c_in.max() + 0.05],
         [c_in.min() - 0.05, c_in.max() + 0.05],
         'k--', dashes=(2, 2), label='y=x')
plt.xlabel('Input CC')
plt.ylabel('Output CC')
plt.legend(loc='best', fontsize=14)
plt.show()
```

### Think! 3.1: GWN and the Correlation Transfer Function
Why do both the mean and the standard deviation of the GWN affect the slope of the correlation transfer function?

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D3_BiologicalNeuronModels/solutions/W2D3_Tutorial2_Solution_2deb4ccb.py)

## Section 3.2: What is the rationale behind varying $\mu$ and $\sigma$?
The mean and the variance of the synaptic current depends on the spike rate of a Poisson process. We can use something called [Campbell's theorem](https://en.wikipedia.org/wiki/Campbell%27s_theorem_(probability)) to estimate the mean and the variance of the synaptic current:

\begin{align}
\mu_{\rm syn} = \lambda J \int P(t) \\
\sigma_{\rm syn} = \lambda J \int P(t)^2 dt\\
\end{align}

where $\lambda$ is the firing rate of the Poisson input, $J$ the amplitude of the postsynaptic current and $P(t)$ is the shape of the postsynaptic current as a function of time.

Therefore, when we varied $\mu$ and/or $\sigma$ of the GWN, we mimicked a change in the input firing rate. Note that, if we change the firing rate, both $\mu$ and $\sigma$ will change simultaneously, not independently.

Here, since we observe an effect of $\mu$ and $\sigma$ on correlation transfer, this implies that the input rate has an impact on the correlation transfer function.

# Think!: Correlations and Network Activity

- What are the factors that would make output correlations smaller than input correlations? (Notice that the colored lines are below the black dashed line)
- What does the fact that output correlations are smaller mean for the correlations throughout a network?
- Here we have studied the transfer of correlations by injecting GWN. But in the previous tutorial, we mentioned that GWN is unphysiological. Indeed, neurons receive colored noise (i.e., Shot noise or OU process). How do these results obtained from injection of GWN apply to the case where correlated spiking inputs are injected in the two LIFs? Will the results be the same or different?

Reference
- De La Rocha, Jaime, et al. "Correlation between neural spike trains increases with firing rate." Nature (2007) (https://www.nature.com/articles/nature06028/)

- Bujan AF, Aertsen A, Kumar A. Role of input correlations in shaping the variability and noise correlations of evoked activity in the neocortex. Journal of Neuroscience. 2015 Jun 3;35(22):8611-25. (https://www.jneurosci.org/content/35/22/8611)

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D3_BiologicalNeuronModels/solutions/W2D3_Tutorial2_Solution_be65591c.py)

---
# Summary

*Estimated timing of tutorial: 50 minutes*

In this tutorial, we studied how the input correlation of two LIF neurons is mapped to their output correlation. Specifically, we:

- injected correlated GWN in a pair of neurons,

- measured correlations between the spiking activity of the two neurons, and

- studied how the transfer of correlation depends on the statistics of the input, i.e., mean and standard deviation.

Here, we were concerned with zero time lag correlation. For this reason, we restricted estimation of correlation to instantaneous correlations. If you are interested in time-lagged correlation, then we should estimate the cross-correlogram of the spike trains and find out the dominant peak and area under the peak to get an estimate of output correlations.

We leave this as a future to-do for you if you are interested.

If you have time, check out the bonus video to think about responses of ensembles of neurons to time-varying input.

---
# Bonus

---
## Bonus Section 1: Ensemble Response

Finally, there is a short BONUS lecture video on the firing response of an ensemble of neurons to time-varying input. There are no associated coding exercises - just enjoy.

```python
# @title Video 2: Response of ensemble of neurons to time-varying input
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV18K4y1x7Pt", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="78_dWa4VOIo", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```


```python
%matplotlib inline
```

Adversarial Example Generation
==============================

**Author:** `Nathan Inkawhich <https://github.com/inkawhich>`__

If you are reading this, hopefully you can appreciate how effective some
machine learning models are. Research is constantly pushing ML models to
be faster, more accurate, and more efficient. However, an often
overlooked aspect of designing and training models is security and
robustness, especially in the face of an adversary who wishes to fool
the model.

This tutorial will raise your awareness to the security vulnerabilities
of ML models, and will give insight into the hot topic of adversarial
machine learning. You may be surprised to find that adding imperceptible
perturbations to an image *can* cause drastically different model
performance. Given that this is a tutorial, we will explore the topic
via example on an image classifier. Specifically we will use one of the
first and most popular attack methods, the Fast Gradient Sign Attack
(FGSM), to fool an MNIST classifier.

Threat Model
------------

For context, there are many categories of adversarial attacks, each with
a different goal and assumption of the attacker’s knowledge. However, in
general the overarching goal is to add the least amount of perturbation
to the input data to cause the desired misclassification. There are
several kinds of assumptions of the attacker’s knowledge, two of which
are: **white-box** and **black-box**. A *white-box* attack assumes the
attacker has full knowledge and access to the model, including
architecture, inputs, outputs, and weights. A *black-box* attack assumes
the attacker only has access to the inputs and outputs of the model, and
knows nothing about the underlying architecture or weights. There are
also several types of goals, including **misclassification** and
**source/target misclassification**. A goal of *misclassification* means
the adversary only wants the output classification to be wrong but does
not care what the new classification is. A *source/target
misclassification* means the adversary wants to alter an image that is
originally of a specific source class so that it is classified as a
specific target class.

In this case, the FGSM attack is a *white-box* attack with the goal of
*misclassification*. With this background information, we can now
discuss the attack in detail.

Fast Gradient Sign Attack
-------------------------

One of the first and most popular adversarial attacks to date is
referred to as the *Fast Gradient Sign Attack (FGSM)* and is described
by Goodfellow et. al. in `Explaining and Harnessing Adversarial
Examples <https://arxiv.org/abs/1412.6572>`__. The attack is remarkably
powerful, and yet intuitive. It is designed to attack neural networks by
leveraging the way they learn, *gradients*. The idea is simple, rather
than working to minimize the loss by adjusting the weights based on the
backpropagated gradients, the attack *adjusts the input data to maximize
the loss* based on the same backpropagated gradients. In other words,
the attack uses the gradient of the loss w.r.t the input data, then
adjusts the input data to maximize the loss.

Before we jump into the code, let’s look at the famous
`FGSM <https://arxiv.org/abs/1412.6572>`__ panda example and extract
some notation.

.. figure:: /_static/img/fgsm_panda_image.png
   :alt: fgsm_panda_image

From the figure, $\mathbf{x}$ is the original input image
correctly classified as a “panda”, $y$ is the ground truth label
for $\mathbf{x}$, $\mathbf{\theta}$ represents the model
parameters, and $J(\mathbf{\theta}, \mathbf{x}, y)$ is the loss
that is used to train the network. The attack backpropagates the
gradient back to the input data to calculate
$\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y)$. Then, it adjusts
the input data by a small step ($\epsilon$ or $0.007$ in the
picture) in the direction (i.e.
$sign(\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y))$) that will
maximize the loss. The resulting perturbed image, $x'$, is then
*misclassified* by the target network as a “gibbon” when it is still
clearly a “panda”.

Hopefully now the motivation for this tutorial is clear, so lets jump
into the implementation.

```python
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
import numpy as np
import matplotlib.pyplot as plt
```

Implementation
--------------

In this section, we will discuss the input parameters for the tutorial,
define the model under attack, then code the attack and run some tests.

Inputs
~~~~~~

There are only three inputs for this tutorial, and are defined as
follows:

-  **epsilons** - List of epsilon values to use for the run. It is
   important to keep 0 in the list because it represents the model
   performance on the original test set. Also, intuitively we would
   expect the larger the epsilon, the more noticeable the perturbations
   but the more effective the attack in terms of degrading model
   accuracy. Since the data range here is $[0,1]$, no epsilon
   value should exceed 1.

-  **pretrained_model** - path to the pretrained MNIST model which was
   trained with
   `pytorch/examples/mnist <https://github.com/pytorch/examples/tree/master/mnist>`__.
   For simplicity, download the pretrained model `here <https://drive.google.com/drive/folders/1fn83DF14tWmit0RTKWRhPq5uVXt73e0h?usp=sharing>`__.

-  **use_cuda** - boolean flag to use CUDA if desired and available.
   Note, a GPU with CUDA is not critical for this tutorial as a CPU will
   not take much time.

```python
epsilons = [0, .05, .1, .15, .2, .25, .3]
pretrained_model = "lenet_mnist_model.pth"
use_cuda=True
```

```python
pretrained_model
```

    'lenet_mnist_model.pth'

Model Under Attack
~~~~~~~~~~~~~~~~~~

As mentioned, the model under attack is the same MNIST model from
`pytorch/examples/mnist <https://github.com/pytorch/examples/tree/master/mnist>`__.
You may train and save your own MNIST model or you can download and use
the provided model. The *Net* definition and test dataloader here have
been copied from the MNIST example. The purpose of this section is to
define the model and dataloader, then initialize the model and load the
pretrained weights.

```python
# LeNet Model definition
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

# MNIST Test dataset and dataloader declaration
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('datamnist', train=False, download=True, transform=transforms.Compose([
            transforms.ToTensor(),
            ])),
        batch_size=1, shuffle=True)

# Define what device we are using
print("CUDA Available: ",torch.cuda.is_available())
device = torch.device("cuda" if (use_cuda and torch.cuda.is_available()) else "cpu")

# Initialize the network
model = Net().to(device)

# Load the pretrained model
model.load_state_dict(torch.load(pretrained_model, map_location='cpu'))

# Set the model in evaluation mode. In this case this is for the Dropout layers
model.eval()
```

    CUDA Available:  True

    Net(
      (conv1): Conv2d(1, 10, kernel_size=(5, 5), stride=(1, 1))
      (conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
      (conv2_drop): Dropout2d(p=0.5, inplace=False)
      (fc1): Linear(in_features=320, out_features=50, bias=True)
      (fc2): Linear(in_features=50, out_features=10, bias=True)
    )

FGSM Attack
~~~~~~~~~~~

Now, we can define the function that creates the adversarial examples by
perturbing the original inputs. The ``fgsm_attack`` function takes three
inputs, *image* is the original clean image ($x$), *epsilon* is
the pixel-wise perturbation amount ($\epsilon$), and *data_grad*
is gradient of the loss w.r.t the input image
($\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y)$). The function
then creates perturbed image as

\begin{align}perturbed\_image = image + epsilon*sign(data\_grad) = x + \epsilon * sign(\nabla_{x} J(\mathbf{\theta}, \mathbf{x}, y))\end{align}

Finally, in order to maintain the original range of the data, the
perturbed image is clipped to range $[0,1]$.

```python
# FGSM attack code
def fgsm_attack(image, epsilon, data_grad):
    # Collect the element-wise sign of the data gradient
    sign_data_grad = data_grad.sign()
    # Create the perturbed image by adjusting each pixel of the input image
    perturbed_image = image + epsilon*sign_data_grad
    # Adding clipping to maintain [0,1] range
    perturbed_image = torch.clamp(perturbed_image, 0, 1)
    # Return the perturbed image
    return perturbed_image
```

Testing Function
~~~~~~~~~~~~~~~~

Finally, the central result of this tutorial comes from the ``test``
function. Each call to this test function performs a full test step on
the MNIST test set and reports a final accuracy. However, notice that
this function also takes an *epsilon* input. This is because the
``test`` function reports the accuracy of a model that is under attack
from an adversary with strength $\epsilon$. More specifically, for
each sample in the test set, the function computes the gradient of the
loss w.r.t the input data ($data\_grad$), creates a perturbed
image with ``fgsm_attack`` ($perturbed\_data$), then checks to see
if the perturbed example is adversarial. In addition to testing the
accuracy of the model, the function also saves and returns some
successful adversarial examples to be visualized later.

```python
def test( model, device, test_loader, epsilon ):

    # Accuracy counter
    correct = 0
    adv_examples = []

    # Loop over all examples in test set
    for data, target in test_loader:

        # Send the data and label to the device
        data, target = data.to(device), target.to(device)

        # Set requires_grad attribute of tensor. Important for Attack
        data.requires_grad = True

        # Forward pass the data through the model
        output = model(data)
        init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability

        # If the initial prediction is wrong, dont bother attacking, just move on
        if init_pred.item() != target.item():
            continue

        # Calculate the loss
        loss = F.nll_loss(output, target)

        # Zero all existing gradients
        model.zero_grad()

        # Calculate gradients of model in backward pass
        loss.backward()

        # Collect datagrad
        data_grad = data.grad.data

        # Call FGSM Attack
        perturbed_data = fgsm_attack(data, epsilon, data_grad)

        # Re-classify the perturbed image
        output = model(perturbed_data)

        # Check for success
        final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
        if final_pred.item() == target.item():
            correct += 1
            # Special case for saving 0 epsilon examples
            if (epsilon == 0) and (len(adv_examples) < 5):
                adv_ex = perturbed_data.squeeze().detach().cpu().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )
        else:
            # Save some adv examples for visualization later
            if len(adv_examples) < 5:
                adv_ex = perturbed_data.squeeze().detach().cpu().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )

    # Calculate final accuracy for this epsilon
    final_acc = correct/float(len(test_loader))
    print("Epsilon: {}\tTest Accuracy = {} / {} = {}".format(epsilon, correct, len(test_loader), final_acc))

    # Return the accuracy and an adversarial example
    return final_acc, adv_examples
```

Run Attack
~~~~~~~~~~

The last part of the implementation is to actually run the attack. Here,
we run a full test step for each epsilon value in the *epsilons* input.
For each epsilon we also save the final accuracy and some successful
adversarial examples to be plotted in the coming sections. Notice how
the printed accuracies decrease as the epsilon value increases. Also,
note the $\epsilon=0$ case represents the original test accuracy,
with no attack.

```python
accuracies = []
examples = []

# Run test for each epsilon
for eps in epsilons:
    acc, ex = test(model, device, test_loader, eps)
    accuracies.append(acc)
    examples.append(ex)
```

    Epsilon: 0	Test Accuracy = 9810 / 10000 = 0.981
    Epsilon: 0.05	Test Accuracy = 9423 / 10000 = 0.9423
    Epsilon: 0.1	Test Accuracy = 8514 / 10000 = 0.8514
    Epsilon: 0.15	Test Accuracy = 6839 / 10000 = 0.6839
    Epsilon: 0.2	Test Accuracy = 4338 / 10000 = 0.4338
    Epsilon: 0.25	Test Accuracy = 2102 / 10000 = 0.2102
    Epsilon: 0.3	Test Accuracy = 878 / 10000 = 0.0878

Results
-------

Accuracy vs Epsilon
~~~~~~~~~~~~~~~~~~~

The first result is the accuracy versus epsilon plot. As alluded to
earlier, as epsilon increases we expect the test accuracy to decrease.
This is because larger epsilons mean we take a larger step in the
direction that will maximize the loss. Notice the trend in the curve is
not linear even though the epsilon values are linearly spaced. For
example, the accuracy at $\epsilon=0.05$ is only about 4% lower
than $\epsilon=0$, but the accuracy at $\epsilon=0.2$ is 25%
lower than $\epsilon=0.15$. Also, notice the accuracy of the model
hits random accuracy for a 10-class classifier between
$\epsilon=0.25$ and $\epsilon=0.3$.

```python
plt.figure(figsize=(5,5))
plt.plot(epsilons, accuracies, "*-")
plt.yticks(np.arange(0, 1.1, step=0.1))
plt.xticks(np.arange(0, .35, step=0.05))
plt.title("Accuracy vs Epsilon")
plt.xlabel("Epsilon")
plt.ylabel("Accuracy")
plt.show()
```

Sample Adversarial Examples
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Remember the idea of no free lunch? In this case, as epsilon increases
the test accuracy decreases **BUT** the perturbations become more easily
perceptible. In reality, there is a tradeoff between accuracy
degredation and perceptibility that an attacker must consider. Here, we
show some examples of successful adversarial examples at each epsilon
value. Each row of the plot shows a different epsilon value. The first
row is the $\epsilon=0$ examples which represent the original
“clean” images with no perturbation. The title of each image shows the
“original classification -> adversarial classification.” Notice, the
perturbations start to become evident at $\epsilon=0.15$ and are
quite evident at $\epsilon=0.3$. However, in all cases humans are
still capable of identifying the correct class despite the added noise.

```python
# Plot several examples of adversarial samples at each epsilon
cnt = 0
plt.figure(figsize=(8,10))
for i in range(len(epsilons)):
    for j in range(len(examples[i])):
        cnt += 1
        plt.subplot(len(epsilons),len(examples[0]),cnt)
        plt.xticks([], [])
        plt.yticks([], [])
        if j == 0:
            plt.ylabel("Eps: {}".format(epsilons[i]), fontsize=14)
        orig,adv,ex = examples[i][j]
        plt.title("{} -> {}".format(orig, adv))
        plt.imshow(ex, cmap="gray")
plt.tight_layout()
plt.show()
```

Where to go next?
-----------------

Hopefully this tutorial gives some insight into the topic of adversarial
machine learning. There are many potential directions to go from here.
This attack represents the very beginning of adversarial attack research
and since there have been many subsequent ideas for how to attack and
defend ML models from an adversary. In fact, at NIPS 2017 there was an
adversarial attack and defense competition and many of the methods used
in the competition are described in this paper: `Adversarial Attacks and
Defences Competition <https://arxiv.org/pdf/1804.00097.pdf>`__. The work
on defense also leads into the idea of making machine learning models
more *robust* in general, to both naturally perturbed and adversarially
crafted inputs.

Another direction to go is adversarial attacks and defense in different
domains. Adversarial research is not limited to the image domain, check
out `this <https://arxiv.org/pdf/1801.01944.pdf>`__ attack on
speech-to-text models. But perhaps the best way to learn more about
adversarial machine learning is to get your hands dirty. Try to
implement a different attack from the NIPS 2017 competition, and see how
it differs from FGSM. Then, try to defend the model from your own
attacks.

```python

```


# Exploring molecular interactions using electronic structure methods.

For today's activity we will use the package of programs psi4, so we will need to install it first.
The psi4 python module can be installed using the anaconda python platform. To do this download anaconda in:

https://www.anaconda.com/download/#linux

You can download either the 2.7 or 3.6 verison depending on your preference.

Once downloaded and installed it is recomended to create a psi4 environment form which
the psi4 module can be loaded. Run the comand:

**conda create -n p4env python=2.7 psi4 psi4-rt -c psi4/label/dev -c psi4**

This will install all the psi4 binaries and a python module which can be imported from the notebook.
To finish the installation you need to provide a scratch directory in your .bashrc, for example:

**export PSI_SCRATCH=/home/user_name/scratch/psi4**

Finally source your .bashrc and activate the environment

**source activate p4env**

Now you may open jupyter-notebook and install psi4.

```python
import psi4
import numpy as np
```

# 1. Compute the energy of a diatomic molecules.

As a first example we will compute the scf energy of the diatomic molecule hydrogen fluoride (HF):

```python
# ==> Opciones Básicas Psi4 <==
# Memoria
psi4.set_memory(int(5e8))
numpy_memory = 500

# Output
psi4.core.set_output_file('output.dat', False)

# Geometría
hf_mol = psi4.geometry("""
0 1
H
F 1 0.917
""")

#psi4.optimize('scf/cc-pvtz')
#di_mat = hf_mol.distance_matrix()
#print np.asarray(di_mat)
energy_hf_mol , wfn_hf_mol = psi4.energy('mp2/cc-pvtz', return_wfn=True)
print energy_hf_mol
```

This corresponds to the Born-Oppenheimer scf energy of HF. Now it's your turn. Compute the single point energy
of F$_2$ and N$_2$ using the cc-pVDZ and cc-pVTZ.

# 2. Compute the dipole and quadrupole moment of diatomic molecules

Since we are interested in studying long range  molecular interactions using classical electrodynamics, it is necessary
to compute the dipole and quadrupole moments. Quantum mechanically the dipole can be computed using the one electron dipole operator:

\begin{equation}
\hat{\mu} = \sum_i q_i r_i
\end{equation}

where $q_i$ is the charge of the particle and $r_i$ is the position vector of the particle.  The dipole moment can be computed using the wavefunction through the expectation value of the operator $\mu$.

\begin{equation}
\mu = <\psi|\hat{\mu}|\psi>
\end{equation}

In psi4 we can obtain the dipole moment from the wavefunction object that was defined above

```python
psi4.oeprop(wfn_hf_mol, 'DIPOLE', 'QUADRUPOLE', title='HF SCF')

mux = psi4.core.get_variable('HF SCF DIPOLE X') # in debye
muy = psi4.core.get_variable('HF SCF DIPOLE Y')
muz = psi4.core.get_variable('HF SCF DIPOLE Z')
quad_zz = psi4.core.get_variable('HF SCF QUADRUPOLE ZZ')
```

```python
print muz
print quad_zz
mu = (np.sqrt(mux**2 + muy**2 + muz**2))
print mu
```

# 3.  Compute a potential energy surface of HF dimer.

In order to study the physical interactions between two molecules it is convenient to draw
a potential energy surface along the interaction coordinate. In this section we will
obtain a potential energy profile for the most favorable dipole-dipole interaction, which is the
horizontal orientation with opposing dipole vectors, HF---HF. First we need to define a list containing
the distances between both dimers for which the energy will be obtained.

```python
hf_dimer = psi4.geometry("""
        0 1
        H
        F 1 0.917
        H 2 R 1 180.0
        F 3 0.917 2 180.0 1 0.0
        """)
```

Next, we write a loop and in each step of the loop we compute the energy at the mp4 level of theory.

```python
energy = []
dist   = []

Rval = np.arange(1.5,10.0,0.1)

for d in Rval:
    hf_dimer.R = d
    psi4.set_options({'freeze_core': 'True'})
    en = psi4.energy('scf/cc-pvtz')
    print en
    print d
    energy.append(en)
    dist.append(d)

```

Now we are ready to plot the potential energy profile. We will use the matplotlib python library for this
purpose. The function ref_cero_kcal transforms the energy which is in hartee to kcal/mol and takes the
energy of the dimer with the farthest separation as the reference energy.

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def ref_cero_kcal(en_list):
    energy_kcal = []
    for x in range(len(en_list)):
        energy_kcal.append((en_list[x] - en_list[-1])*627.51)
    return energy_kcal

energy_kcal = ref_cero_kcal(energy)
plt.plot(dist,energy_kcal)

```

**Exercise:** Using the data from the plot, make an estimate of the interaction energy in kcal/mol. Corroborate your results computing the interaction energy using the super-molecule approach. ( Hint: 1 hartree = 627.51 kcal/mol). What percentage of the total energy is the intermolecular interaction energy?

```python
hf_dimer_min = psi4.geometry("""
        0 1
        H
        F 1 0.917
        H 2 1.980 1 180.0
        F 3 0.917 2 180.0 1 0.0
        """)

energy_hf_dimer_min , wfn_hf_dimer_min = psi4.energy('mp2/cc-pvtz', return_wfn=True)
```

```python
print energy_hf_dimer_min

int_energy = (energy_hf_dimer_min - 2*energy_hf_mol)

print int_energy*627.51

print("Percentage: "+str((int_energy/energy_hf_dimer_min)*100))

```

# 4. The effect of electron correlation

Since we know that the dispersion energy arises only when the motion of the electrons on the different molecular fragments are correlated, it is possible to evalute the importance of disperison efects by using
methods that include diferent levels of electron correlation.

**Exercices**: Construct a potential energy curve at the SCF, MP2 and MP4 level of theory and
asses the importance of including electron correlation along the intermolecular coordinate.

```python
energy_scf = []
energy_mp2 = []
energy_mp4 = []
dist = []

for d in Rval:
    hf_dimer.R = d
    psi4.set_options({'freeze_core': 'True'})
    en_scf = psi4.energy('scf/cc-pvtz')
    en_mp2 = psi4.energy('mp2/cc-pvtz')
    en_mp4 = psi4.energy('mp4/cc-pvtz')
    print en_scf
    print d
    energy_scf.append(en_scf)
    energy_mp2.append(en_mp2)
    energy_mp4.append(en_mp4)
    dist.append(d)
```

```python
energy_scf_kcal = ref_cero_kcal(energy_scf)
energy_mp2_kcal = ref_cero_kcal(energy_mp2)
energy_mp4_kcal = ref_cero_kcal(energy_mp4)

plt.plot(dist,energy_scf_kcal)
plt.plot(dist,energy_mp2_kcal)
plt.plot(dist,energy_mp4_kcal)

```

# 5. Energy decomposition with SAPT

So far we have only computed the interaction energy as a whole, however we know that we can
decompose the interaction energy into contributions stemming from different physical phenomena,
in particular, electrostatic, induction, dispersion and exchange interactions. First we will run a
SAPT0 calculation to obtain the total interaction energy and compare it to the results we obtained
above.

```python
hf_dimer_sapt = psi4.geometry('''
     0 1
     H  0.0 0.0 0.0
     F  0.0 0.0 0.917
     --
    0 1
     H  0.0 0.0 2.951
     F  0.0 0.0 3.868
     symmetry c1
''')

psi4.set_options({'basis': 'jun-cc-pVDZ',
                  'e_convergence': 1e-8,
                  'd_convergence': 1e-8})

psi4.energy('sapt0')
```

Check the output file for energy decomposition analysis. Which contribution is dominant, and what do you think
is the physical reason for it?

**Exercise:** Repeat the above analysis with the Argon dimer $Ar_2$. What can you say about the importance
of the correlation energy in this case? Which is the dominant contribution in the case of the Ar dimer?

```python

```