Geospatial Analysis & Visualization w/ Python

# Step 0) Setup
* To get started, we need to import all the packages we'll use.

```python
import numpy as np
import pandas as pd
import geopandas as gpd
import scipy.stats as stats
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from matplotlib.collections import PatchCollection
%matplotlib notebook

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:90% !important; }</style>"))
```

# Step 1) Importing our data
* We'll load the Police Killing Data as a "DataFrame" using pandas

* Then we'll convert it into a "GeoDataFrame" using Geopandas
    * To do this, we must assign the "geometry".  In this case its point data, and the coordinates are in lat/long

* Then we need to assign a Coordiante Reference System (CRS) manually
    * ESPG is a standardized code that is used to represent CRSs.
    * 'espg:4326' is for the refers to the WGS 1984 datum, which our latitude/longitude data is based in.
        * This is a CRS that is widely used by many web-based platforms because like Google Maps and Mapbox
        * The original only had addresses, not coordinates, so we used a webservice (Mapbox) to generate the coordinates of our addresses

* Once we have the data loaded, calling .head() will give us a "preview" of our dataset

```python
# We import the Police Killings file, and set the incident ID as the index
police_Killings_Tabular = pd.read_csv('Data/PoliceKillings.csv',
                                      parse_dates=['date'],
                                      index_col=['id_incident']
                                     )

# We can then convert the pandas dataframe into a geopandas "GeodataFrame"
police_Killings = gpd.GeoDataFrame(police_Killings_Tabular,
    geometry=gpd.points_from_xy(police_Killings_Tabular.longitude,
                                police_Killings_Tabular.latitude
                               )
                                  )

# Now we can assign a CRS
WGS_1984={'init' :'epsg:4326'}
police_Killings.crs = WGS_1984

# Lets sort the incidents by date and then take a quick look.
police_Killings=police_Killings.sort_values(by='date')
police_Killings.head()
```

### Now we'll load some data from the 2016 Census

* We have a tabular dataset of population data.

* We'll load it using pandas

```python
# We'll import the tabualr census data with pandas
Census_Tabular = pd.read_csv('Data/Census.csv',index_col=['PRUID'])
Census_Tabular.head()
```

### We also have a provincial boundary shapefile

* Shapefile are used to store georphric data.  They already have projections and coordiantes associated with them.
    * Geopandas has similar functionality to pandas and we can use it to import shapefiles.  But the read_file() method had less options, so we have to set the index manually.

```python
# We'll import provincial boundaries using geopandas
Provincial_Boundaries = gpd.read_file('Data/Provincial_Boundaries.shp').set_index('PRUID')
Provincial_Boundaries.head()
```

# Step 2) Joining our census data

* This will let us map the disparity by province and do a more detailed analysis

* PRUID is a "unique identifier" that represents the provinces.

    * Since both have the PRUID set as the index, we don't need to specify a join key.

```python
Test_Join = Provincial_Boundaries.join(Census_Tabular)
Test_Join.head()
```

# But our join fails :(

* ### Notice the NaN values.

* NaN represetns missing data values
    * Lets look at the index for both files?  Maybe we have a datatype missmatch?

```python
print(Provincial_Boundaries.index.dtype)
print(Census_Tabular.index.dtype)
```

### Sure enough!  The Provincial_Boundaries index is an "object", not an integer.

* We can fix that easily and then do the join!
    * We just need to change the datatype of the Provincial_Boundaries layer.

### We can assign the datatype using the .astype() function.

### But datatype do we assign?
* Hint The anser is in the cell above!!

```python
dtype =
Provincial_Boundaries.index = Provincial_Boundaries.index.astype(dtype)
Provincial_Data = Provincial_Boundaries.join(Census_Tabular)
Provincial_Data.head()
```

# Step 3) Exploring the data

### First lets make a quick map.

* Our Layers need to be in the same coordinate system to match up properly on a map!

* We can re-project the police_Killings layer using the .to_crs function to set the CRS to that of the Provinces
    * The provinces layer uses the Canada Lambert Conformal Conic projection (LCC).  This is the standard projection used by stats canada and is ideally suited for displaying the whole of country.

* Once both datasets are in the same coordinate system, we can make a map!

* First we must define a plot, using the matplotlib.pyplot package.  We imported this earlier as "plt"
    * We use the plt.subplots() to create a figure, and we can define how big we want it to be

* Geoapandas can then use the .plot() fucntion to create a map using matplotlib.
    * We simply tell it what axis to draw the plot on with ax="axes"
    * Then set a few other parameters:
        * We just want the provinces as a grey background so we can set the color
        * We want to classify killings by race, so we can set race as the column.  THen we can add a legend to aid interpretation of the data

```python
# We can use .to_crs() to create a police killings layer with the same projection as the provinces layer.
police_Killings = police_Killings.to_crs(Provincial_Data.crs)

# Now, we can create a figure using matplotlib (plt), first we define the figure and the size
fig,axes=plt.subplots(
    figsize=(6,6)
)

# Now we can add the provinces using the .plot() function.  We set the plotting axes and give it a grey color
cb = Provincial_Data.plot(
    ax=axes,
    column='Total',
    cmap = 'Greys',
    edgecolor='grey',
    legend=True,
)

# Then we add the police_Killings_LCC.  We'll set the column to 'race', so we can disply by race,
# give the point markers a few more parameters, and add them to a legend
police_Killings.plot(
    ax=axes,
    column='race',
    edgecolor='k',
    markersize=15,
    legend=True,
    legend_kwds={'loc': 'upper right','fontsize':8}
)
```

### And now you've made your first map with python!

* But its an ugly map :(
    * It doesn't look great.  This is just the quick and dirty way to look ata data
    * To make things more presentable, we'll have to be more explicit in setting up our map.  But that's a task for later.

### For now, lets move on and look at the dataset in more detail.

* Pandas & Geopandas have some nice features to quickly summarize our dataset.

* We can use .count() to get the total # incidents.
    * Callling .count() as is, will give us a list of all the columns, and a count for each.  We can see most collumns are "full" but in the "geocoding_Notes" column, we can see that 4 points don't have coordinates associated with their address.  This suggests there was an error in the data entry process.  We don't need to worry about this though.

```python
police_Killings.count()
```

* We can use .mean(), .min(), etc. followed by ['age'] to get some vital statistics on the age of victims.

* We can use .describe() to summarize multiple attributes of the age column.

```python
print('Age Distribution of Victims')
print()
print('Mean:                ',
      police_Killings.mean()['age']
     )
print()
print('Standard Deviation:  ',
      police_Killings.std()['age']
     )
print()
print('Youngest:            ',
      police_Killings.max()['age']
     )
print()
print('Oldest:              ',
      police_Killings.min()['age']
     )

police_Killings.describe()['age']
```

### We can resample our data to look for trends
* The date column is a special type of data that allows us to resample our data by year, month, etc
* The dataset has to be in order by date for this to work (we did this alread).

```python
Resampled = police_Killings.set_index('date').resample('Y').count()

## scipy can be used to calculate a linear regression line and print the results
Regression_Line = stats.linregress(Resampled.index.year,Resampled['id_victim'])
print(Regression_Line)

plt.figure(figsize=(6,6))

## We can make a scatter plot of the annual killings
plt.scatter(
    Resampled.index.year,
    Resampled['id_victim'],
    color='black',
    label='Yearly Total'
)

## Then plot he increasing trendline over it and add the slope and pvalue to the legend.
plt.plot(
    Resampled.index.year,
    Resampled.index.year*Regression_Line[0]+Regression_Line[1],
    label='Trend Line: '+str(np.round(Regression_Line[0],3))+'\np-value: '+str(np.round(Regression_Line[3],3)),
    color='red'
    )

plt.legend()
plt.title('Police Killings per Year in Canda')
```

### We can group our data to look for patterns too.

* the .groupby() function can accept one or multple paramters to group our dataset by.
    * This allows us to create complex queries if we want.
* We can have to follow up with .count(), .mean(), etc.
    * This tells us "how" to aggregate

```python
fig,ax = plt.subplots(figsize=(5,5))

## This is a dictionary to set colors based on specific values.  We're using Hex color codes here.
## They're a common way for specifiying colors
Pie_Colors = {'None':'#FF0000',
              'Knife':'#fecc5c',
              'Firearm':'#ffffb2',
              'Other weapons':'#fd8d3c'}

## Group by the variable we want, summarize by count, then make our pie chart!
Armed = police_Killings.groupby(['armed_type']).count()
ax.pie(
    Armed['id_victim'],
    labels=Armed.index,
    textprops={'fontsize': 8},
    colors=[Pie_Colors[i] for i in Armed.index],
    autopct='%1.1f%%',
    wedgeprops={"edgecolor":"k",'linewidth': 1, 'linestyle': 'dashed'}
)
ax.set_title('Police Killings: Was the Victim Armed?')
plt.tight_layout()
```

### Groupby allows us to create very complex queries if we want

* We can search combinations of two or more variables

* This retruns a record with two indexes: Department and armed type

```python
Group = police_Killings.groupby(['Department','armed_type']).count()['id_victim']#.sort_values(ascending=True)

print(Group)
```

### What police departments kill the most unarmed people?

* We can use the .loc command to do a search for all departments with more than one unarmed killing.

* Then we sort the record, select the top 5, and plot them.

```python
## The unstack command allows us to turn the last index into our colum names
Force=Group.unstack()
Force['Total'] = Force.sum(axis=1)
Force['Unarmed_Frac']=Force['None']/Force['Total']

## .loc is the search function.
## .sort_values() sorts our records in ascending order.
## [-5:] grabs just the last five records
Force = Force.loc[Force['None']>1].sort_values(by='None')[-5:]
print(Force)

Force_Labels={'Toronto Police Service':'Toronto',
              'RCMP':'RCMP',
              'Vancouver Police Department':'Vancouver',
              'Service de police de la Ville de Montreal':'Montreal',
              'Edmonton Police Service':'Edmonton'}

fig,ax=plt.subplots(figsize=(6,5))

ax.barh(Force.index,Force['None'],facecolor='#FF0000',edgecolor='black')
ax.set_yticklabels([Force_Labels[f] for f in Force.index.values])
ax.set_title('Unarmed Victims by Deparment (2000-2015)')
```

### We're intersted in a specific question.  What's the distribution of police killings by race?

```python
police_Killings.groupby(['race']).count()['date'].sort_values()
```

# Step 4) Normalizing our Data

* The racial demographics of Canada aren't evenly split however!

* We need to Normalize our data by population statistics.

* Lets look at our census data again

```python
Provincial_Data[Census_Tabular.columns]
```

### The first row contains the total values for the whole country.  We can use this to calculate a police killing rate.

* But the Canadian Census' racial categories don't match up perfectly with the police violence dataset's racial
* How can we work around this?
    * We have the largest three groups in the police killing set: White, Indigenous, and Black.  So we can work with them as is
    * The other races make up a small portion of total killings.  And we can't be entirely sure how the CBC defined their groupings.  So, lets add a new category: "Other Minorities"

* We'll do this for both the provincial boundaires and the police_Killings
    * For the police killings, we'll leave the unknow records alone

```python
Other_Minorities=['South Asian', 'Chinese', 'Filipino','Latin American',
 'Arab', 'Southeast Asian', 'West Asian', 'Korean',
'Japansese', 'Visible minority, n.i.e', 'Mixed']
Provincial_Data['Other Minorities']=Provincial_Data[Other_Minorities].sum(axis=1)

Other_Minorities=['Latin American', 'Arab', 'Other', 'South Asian', 'Asian']
police_Killings['race'] = police_Killings['race'].replace(to_replace=Other_Minorities,value='Other Minorities')

```

# From here, we can calculate the Police Killing Rate (PKR).

* Dividing the total number of killings by the population gives us ...

```python
Races = ['Indigenous','Black','Caucasian','Other Minorities']
Race_Breakdown = police_Killings.groupby(['race']).count()['id_victim']
Can_Pop = Provincial_Data[Races].sum()

Racial_Rates = Race_Breakdown.T[Races]/Can_Pop
Racial_Rates['CA. Average']=Race_Breakdown.T[Races].sum()/Can_Pop.sum()
print(Racial_Rates)
# police_Killings.groupby(['race']).count()['date'].sort_values()
```

### This number isn't that meaningful though.  It represents the number of killings "per person" over the whole study period.

* Lets convert the rate to a more meaninful unit.  Killings / Million Residents / Year

* The date record is a "date" object.
* It has some added functionality like being able to query the the year, month, day

```python
First_Year = police_Killings['date'].min().year
Last_Year = police_Killings['date'].max().year
print(First_Year,Last_Year)
```

### How might we calculate our police killing rate?

* What should we set as scale and duration to convert units?

```python
Scale =
Duration =
rate_Conversion = Scale / Duration

Racial_Rates=Racial_Rates.sort_values(ascending=True)

fig, ax = plt.subplots(figsize = (5,5))
ax.barh(
    Racial_Rates.index,
    Racial_Rates.values * rate_Conversion,
    facecolor='#FF0000',
    edgecolor='black',
    linewidth=1
)
ax.set_title('Police Killings Rates by Race in Canada')
ax.set_xlabel('Killings per Year per Million People')
plt.tight_layout()
```

# The Police killing rates are 5x higher for Indigenous people and 4x higher for Black people than it is fo White people.

* ## This is an abhorent example of systemic racism in Canadian Policing.

# Lets look at the PKR by province.
Now we  want to normalize by provincial demographics.

* We have a few more steps to go through first.
    * The police killings and census data use different abbreviations.  To do a join our dataset with the census data we'll need to assign an new abbreviaton
    * We'll us a dictionary to do this

* Then we can summarize the killings by province and join it to the Provinces_Join layer

* Now we can summarize the killings by province and join it to the Provinces_Join layer

* Note Prince Edward Island doesn't have any.

```python
race_by_Province = police_Killings.groupby(['prov','race']).count()
race_by_Province = race_by_Province['date'].unstack()
race_by_Province['Total'] = race_by_Province.sum(axis=1)

for col in Races:
    Provincial_Data = Provincial_Data.join(race_by_Province[col],on='prov',rsuffix='_Killings')

for col in ['Unknown','Total']:
    Provincial_Data = Provincial_Data.join(race_by_Province[col],on='prov',rsuffix='_Killings')
Provincial_Data

# Some provines/groups don't have any records.  Those are given NaN values, and need to be repalced with zeros
Provincial_Data[[x+'_Killings' for x in Races]]=Provincial_Data[[x+'_Killings' for x in Races]].fillna(0)
Provincial_Data['Total_Killings']=Provincial_Data['Total_Killings'].fillna(0)
Provincial_Data[['Unknown' for x in Races]].fillna(0)

Provincial_Data.head()
```

# Step 5) Calcualte and map the Police Killing Rate (PKR) on the provincial level
* Nunavut has a huge problem.  Its not a conicidence that the population is 75% Inuit.

```python
Provincial_Data['PKR']=(Provincial_Data['Total_Killings']/Provincial_Data['Total']*rate_Conversion)

print(Provincial_Data[['prov','PKR']].sort_values(by='PKR'))
```

### We can make a chloropleth map of this pattern.

* Lets check out colorbrewer for help picking a good color scheme

https://colorbrewer2.org/

```python
## We can define our own class break values, labels, and colors for the map
bins = [-0.01,0.5,1,1.4,4,9]
labels = ['<0.5','0.5 - 1','1 - 1.3','3.1','7.7']
colors = ['#fee5d9','#fcae91','#fb6a4a','#de2d26','#FF0000']

## We can use the labels and colors to create a color dictionary
PKR_Color = {key:value for key,value in zip(labels,colors)}

## The pd.cut() command will creat a new record for us containg class labels.
Provincial_Data['PKR_Classes']=(pd.cut(Provincial_Data['PKR'],bins=bins,labels=labels)).astype('str')

fig,ax=plt.subplots(figsize=(6,6))

## To make a better map with geopandas, we have to loop through recrods and plot each class seprately
## Then create a custom legend item for it.

Patches=[]
for pkr_class in Provincial_Data['PKR_Classes'].unique():
    ## kwargs allows us to set multiple arguments for the plot and save them
    ## We can then used them in multiple commands
    kwargs = {'facecolor':PKR_Color[pkr_class],
             'edgecolor':'k',
             'label':pkr_class}

    Provincial_Data.loc[Provincial_Data['PKR_Classes']==pkr_class].plot(
        ax=ax,
        **kwargs
             )

    ## mpatches.Patch() allows us to create a custom legend item for each class
    Patches.append(mpatches.Patch(**kwargs))

## We add the legend items to the legend
ax.legend(handles=Patches)

## This turns off the numeric x,y labels
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)

ax.set_title('Police Killings per Year per Million People')
```

# Step 6) Calculate a Police Killings Discrimination Index (PKDI):

* For this, we'll compare the PKR for white people to the combined PKR of black and indigenous people

* We'll use the following equations:

\begin{align}
\ PKR_{W} & = (\frac{White Killings}{White Population}) * 1e6 / 18\\
\end{align}

\begin{align}
\ PKR_{BI} & = (\frac{Black Killings + Indigenous Killings}{Black Population + Indigenous Population}) * 1e6 / 18
\end{align}

\begin{align}
\ PKDI & = PKR_{BI} - PKR_{W}\\
\end{align}

## This will hightlight the disparities in police killings
* We'll classify the data using the following scheme:

        * "Low Bias": -0.5 to 0.5 - This is the rate killings of whites.  Within these ranges, differences might be due to presence or lacktherof of a certain groups
        * "Moderate Bias": 0.5 to 1 - Greater than the white rate, less than the national average
        * "Severe Bias": 1 to 3 - Greater than the national rate, less than the indigenouos rate
        * "Extreme Bias: 3 to 10 - Greater than the national indigenous rate

```python
Provincial_Data['PKR_W']=Provincial_Data['Caucasian_Killings']/Provincial_Data['Caucasian']*rate_Conversion
Provincial_Data['PKR_BI']=(Provincial_Data['Indigenous_Killings']+Provincial_Data['Black_Killings'])/(Provincial_Data['Indigenous']+Provincial_Data['Black'])*rate_Conversion

Provincial_Data['PKDI'] = Provincial_Data['PKR_BI'] - Provincial_Data['PKR_W']

Provincial_Data['PKDI']=Provincial_Data['PKDI'].fillna(0)

bins = [-0.5,0.5,1,2,10.0]
labels = ['Low Bias','Moderate Bias','Severe Bias','Extreme Bias']
Provincial_Data['PKDI_Classes']=(pd.cut(Provincial_Data['PKDI'],bins=bins,labels=labels)).astype('str')

Provincial_Data.round(2)

# print(Provincial_Data[['prov','PKDI','PKDI_Classes']].sort_values(by='PKDI').round(2))
```

### Lets map the patterns

```python
fig,ax1=plt.subplots(figsize=(5,5))
colors = ['#ffffb2','#fecc5c','#fd8d3c','#FF0000']
PKDI_Color = {key:value for key,value in zip(labels,colors)}

Patches = []
for pkdi_class in Provincial_Data['PKDI_Classes'].unique():
    kwargs = {'facecolor':PKDI_Color[pkdi_class],
             'edgecolor':'black',
              'linewidth':.5,
             'label':pkdi_class}
    Provincial_Data.loc[Provincial_Data['PKDI_Classes']==pkdi_class].plot(
        ax=ax1,
        **kwargs
             )
    Patches.append(mpatches.Patch(**kwargs))

ax1.legend(handles=Patches,)
ax1.get_xaxis().set_visible(False)
ax1.get_yaxis().set_visible(False)
ax1.set_title('Police Killing Discrimination Index')
```

# Step 7) Save the data so we can use it in the future
* We're going to save it as a shapefile for use with geopandas or a desktop GIS
* We're also going to save it as a "GeoJSON" file.  This datatype is well suited for webmapping.  Which I cover in a dfferent workshp!

```python
Provincial_Data.to_file('Data/Provincial_Police_Violence.shp')
```

# Step 8) Putting Everything Together: Create an Infographic

* Matplotlib alows us to be very specific in determining our layout with gridspec.

* We can create a large plot and define specifically what we want.

* We'll have two maps, showing the PKR and the PKDI on the left

* Then we'll add some smaller plots on the right showing the annual trend, national PKR by race, and some pie charts

* We can set our default ontsize for consistency

```python
SMALL_SIZE = 8
MEDIUM_SIZE = 10
BIGGER_SIZE = 16

plt.rc('font', size=SMALL_SIZE)          # controls default text sizes
plt.rc('axes', titlesize=MEDIUM_SIZE)     # fontsize of the axes title
plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels
plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize
plt.rc('figure', titlesize=BIGGER_SIZE)  # fontsize of the figure title

fig = plt.figure(figsize=(10,10))
gs = fig.add_gridspec(100,100)

PKR_Map = fig.add_subplot(gs[0:45 , 0:45])
PKDI_Map = fig.add_subplot(gs[0:45, 55:])

Annual_Trend = fig.add_subplot(gs[47:65, 5:45])
PKR_national = fig.add_subplot(gs[47:65, 65:])

Armed_Stats = fig.add_subplot(gs[74:92, 5:45])
Dept_Stats = fig.add_subplot(gs[74:92, 65:])

SourceStatement = fig.add_subplot(gs[93:,:])

plt.subplots_adjust(left=.05, bottom=.05, right=.95, top=.95, wspace=.1, hspace=.1)

fig.patch.set_facecolor([.9,.9,.9])
plt.suptitle('Police Killings in Canada (2000-2017)')

```

### Now we can add things to the figure
* First lets do the maps

```python
## Plot the PKR
Patches=[]
for pkr_class in Provincial_Data['PKR_Classes'].unique():
    kwargs = {'facecolor':PKR_Color[pkr_class],
             'edgecolor':'k',
             'label':pkr_class}
    Provincial_Data.loc[Provincial_Data['PKR_Classes']==pkr_class].plot(
        ax=PKR_Map,
        **kwargs
             )
    Patches.append(mpatches.Patch(**kwargs))
PKR_Map.legend(handles=Patches)
PKR_Map.get_xaxis().set_visible(False)
PKR_Map.get_yaxis().set_visible(False)
PKR_Map.set_title('Police Killings per Year per Million People')

# Plot the PKDI
Patches = []
for pkdi_class in Provincial_Data['PKDI_Classes'].unique():
    kwargs = {'facecolor':PKDI_Color[pkdi_class],
             'edgecolor':'k',
             'label':pkdi_class}
    Provincial_Data.loc[Provincial_Data['PKDI_Classes']==pkdi_class].plot(
        ax=PKDI_Map,
        **kwargs
             )
    Patches.append(mpatches.Patch(**kwargs))
PKDI_Map.legend(handles=Patches,)
PKDI_Map.get_xaxis().set_visible(False)
PKDI_Map.get_yaxis().set_visible(False)
PKDI_Map.set_title('Police Killing Racial Discrimination Index')

## Plot the annual trend
Annual_Trend.plot(
    Resampled.index.year,
    Resampled['id_victim'],
    color='black',
)
Annual_Trend.plot(
    Resampled.index.year,
    Resampled.index.year*Regression_Line[0]+Regression_Line[1],
    label='Trend Line: '+str(np.round(Regression_Line[0],3))+'\np-value: '+str(np.round(Regression_Line[3],3)),
    color='red'
    )
Annual_Trend.legend()
Annual_Trend.set_title('Annual Trend')
Annual_Trend.set_xticks([2000,2005,2010,2015])

## Plot the Average PKR
PKR_national.barh(
    Racial_Rates.index,
    Racial_Rates.values * rate_Conversion,
    facecolor='#FF0000',
    edgecolor='black',
    linewidth=1
)
PKR_national.set_title('Racial Disparity: National Average')
PKR_national.set_xlabel('Killings per Year per million residents')

## plot the weapon type
Armed_Stats.pie(
    Armed['id_victim'],
    labels=Armed.index,
    colors=[Pie_Colors[i] for i in Armed.index],
    textprops={'fontsize': 8},
    autopct='%1.1f%%',
    wedgeprops={"edgecolor":"k",'linewidth': 1, 'linestyle': 'dashed'}
)
Armed_Stats.set_title('Was the Victim Armed?')

## Plot the breakdown by department
Dept_Stats.barh(Force.index,Force['None'],facecolor='#FF0000',edgecolor='black')
Dept_Stats.set_yticklabels([Force_Labels[f] for f in Force.index.values])
Dept_Stats.set_title('Unarmed Victims by Deparment')

## Add a source statement
Descriptor = \
'''Infographic Created by June Skeeter. Data Soruces: The CBC "Deadly Force (2018)" & Stats Canada

The Police Killing Discrimination Index (PKDI) quantifies the disparitiy in police killing rates (PKR) between Black and Idigenous people and White people in Canada.
The PKDI is deined as: PKDI = PKR$_{Black+Indigenous}$-PKR$_{White}$'''

# Descriptor = 'Kitties'
SourceStatement.set_axis_off()
SourceStatement.text(0, 0,
                     Descriptor,
                     horizontalalignment='left',
                     verticalalignment='center',
                    )

plt.savefig('InfoGraphic.png',facecolor=fig.get_facecolor(),edgeolor='k')
```

# Updated Data for BC

2013 - May 2021

```python
# the .read_file() function reads shapefiles
MVan_CT = gpd.read_file('Data/CensusTracts/SimplyAnalytics_Shapefiles_2021-06-01_17_44_45_9e5629a2de473cd5362919f9edc33853.shp')
print(MVan_CT.crs)
MVan_CT = MVan_CT.rename(columns={
'VALUE0': 'Aboriginal identity, 2016',
'VALUE1': 'Population, 2016',
'VALUE2': 'Total visible minority population, 2016'
                    })

MVan_CT['NonWhitePCT'] = MVan_CT[['Aboriginal identity, 2016',
'Total visible minority population, 2016']].sum(axis=1)/MVan_CT['Population, 2016']*100

# .to_crs()changes the coordinate system
# Provincial_Data = Provincial_Data.to_crs('EPSG:4326')
# .to_file() saves our data to the specified format
print('Data Converted')
print(MVan_CT.NonWhitePCT.describe())
MVan_CT.to_file("Data/MVan_CT.json", driver = "GeoJSON")
MVan_CT.head()
```

```python
# We import the Police Killings file, and set the incident ID as the index
BC_Tabular = pd.read_csv('Data/BC_Geocoded.csv',
                                      parse_dates=['date'],
                                      index_col=['id_incident']
                                     )

# We can then convert the pandas dataframe into a geopandas "GeodataFrame"
BC_Data = gpd.GeoDataFrame(BC_Tabular,
    geometry=gpd.points_from_xy(BC_Tabular.longitude,
                                BC_Tabular.latitude
                               )
                                  )

# Now we can assign a CRS
WGS_1984={'init' :'epsg:4326'}
BC_Data.crs = WGS_1984

# Lets sort the incidents by date and then take a quick look.
BC_Data=BC_Data.sort_values(by='date')
BC_Data.head()
```

# Point in Polygon Analysis

```python
MVan_CT['Incidents'] = 0
for i,row in MVan_CT.iterrows():
    pip = BC_Data.within(row['geometry'])
    if pip.sum()>0:
#         print(pip.sum())
        MVan_CT.loc[MVan_CT.index==i,'Incidents']+=pip.sum()
print(MVan_CT['Incidents'].describe())
```

```python

# Now, we can create a figure using matplotlib (plt), first we define the figure and the size
fig,axes=plt.subplots(
    figsize=(8,8)
)

# Now we can add the provinces using the .plot() function.  We set the plotting axes and give it a grey color
cb = MVan_CT.plot(
    ax=axes,
    column='Incidents',
    cmap = 'Greys',
    edgecolor='grey',
    legend=True,
)

# Then we add the police_Killings_LCC.  We'll set the column to 'race', so we can disply by race,
# give the point markers a few more parameters, and add them to a legend
BC_Data.plot(
    ax=axes,
    edgecolor='k',
    markersize=15,
    legend_kwds={'loc': 'upper right','fontsize':8}
)
X_bounds = [MVan_CT.bounds.minx.min(),MVan_CT.bounds.maxx.max()]
Y_bounds = [MVan_CT.bounds.miny.min(),MVan_CT.bounds.maxy.max()]
axes.set_ylim(Y_bounds)
axes.set_xlim(X_bounds)
```

```python
MVan_CT['Rate'] = MVan_CT['Incidents']/MVan_CT['Population, 2016']*1e4/8.5
# Now, we can create a figure using matplotlib (plt), first we define the figure and the size
fig,axes=plt.subplots(
    figsize=(8,8)
)

# Now we can add the provinces using the .plot() function.  We set the plotting axes and give it a grey color
cb = MVan_CT.plot(
    ax=axes,
    column='Rate',
    cmap = 'Greys',
    edgecolor='grey',
    legend=True,
)

# Then we add the police_Killings_LCC.  We'll set the column to 'race', so we can disply by race,
# give the point markers a few more parameters, and add them to a legend
BC_Data.plot(
    ax=axes,
    edgecolor='k',
    markersize=15,
    legend_kwds={'loc': 'upper right','fontsize':8}
)
X_bounds = [MVan_CT.bounds.minx.min(),MVan_CT.bounds.maxx.max()]
Y_bounds = [MVan_CT.bounds.miny.min(),MVan_CT.bounds.maxy.max()]
axes.set_ylim(Y_bounds)
axes.set_xlim(X_bounds)
```

```python

```

_site/Geospatial Analysis & Visualization with Python v2.0.ipynb


# Introduccion a Colab de Google
---

Colaboraty es una plataforma online de google gratuita para la ejecucion de Jupyter Notebooks https://jupyter.org/
Tiene una integracion con drive

Para montar una unidad de drive diferente a la sesion actual iniciada

```python
from google.colab import drive
drive.mount('/gdrive')
%cd /gdrive/My\ Drive/Taller
```

    Mounted at /gdrive
    /gdrive/My Drive/Taller

```python
%cd
%cd ..
drive.flush_and_unmount()
```

    /root

Si se monta la misma unidad, cambiar a una carpeta especifica indicando la ruta

```python
from google.colab import drive
drive.mount('/content/drive')
%cd content/drive/MyDrive/Taller
```

    Mounted at /content/drive

Obtener la ubicacion actual dentro de las carpetas del computador

```python
!pwd
```

    /content/drive/MyDrive/Taller

Obtener los documentos dentro de la carpeta

```python
!ls
```

    test.png

Mostrar la imagen test.png

Otros comandos de la consola https://www.hostinger.co/tutoriales/linux-comandos

```python
from IPython.display import Image
Image('test.png')
```

# Operaciones Basicas

---

### Suma

```python
O_sum = 3 + 11
O_sum += 5
O_sum
```

    19

### Multiplicacion

```python
O_mult = 3 * 10
O_mult *= 3
O_mult
```

    90

### Division

```python
O_div = 7 / 10
O_div
```

    0.7

### Exponencial

```python
O_exp = 2 ** 6
O_exp
```

    64

### Modulo

```python
O_mod = 20 % 3
O_mod
```

    2

### Cociente

```python
O_coci = 20 // 3
O_coci
```

    6

### Operaciones de comparacion

```python
mi_boolean = 2 == 3
mi_boolean
```

    False

```python
mi_boolean = 'hola' != "hola"
mi_boolean
```

    False

```python
mi_boolean = 34 < 10
mi_boolean
```

    False

```python
mi_boolean = 35 >= 35
mi_boolean
```

    True

```python
mi_boolean = 35 == 35 and 2 > 10
mi_boolean
```

    False

```python
mi_boolean = 14 <= 15 or 16 > 20
mi_boolean
```

    True

```python
mi_boolean = not 'hola' != "hola"
mi_boolean
```

    True

# Variables String (alfanumerico)
---

### String

Se puede usar tanto comillas dobles " " como comillas simples ' ' y sera interpretado como tipo string

```python
mensaje = 'Hola mundo'
print(type(mensaje))

mensaje = "Hola mundo"
print(type(mensaje))
```

    <class 'str'>
    <class 'str'>

### Concatenar string

```python
mensaje += '\nBienvenidos'
print(mensaje)
```

    Hola mundo
    Bienvenidos

### Replicar String

```python
mensaje = mensaje + '\n'*3 + 'Hello world '*2
print(mensaje)
```

    Hola mundo
    Bienvenidos

    Hello world Hello world

### Obtener una entrada del usuario

```python
x = input()
print(x)
type(x)
```

    56
    56

    str

### String format

```python
mensaje = 'El nombre de la ciudad es {} del pais {}'.format('Bogota', 'Colombia')
mensaje
```

    'El nombre de la ciudad es Bogota del pais Colombia'

# Tipos de conjuntos
---

### Tuple

Tuple vacia

```python
mi_tuple = ()
mi_tuple
```

Se pueden guardar multiple tipos de archivos en una tupla

```python
mi_tuple = (1, 2, 'hola')
mi_tuple
```

Se usa los [ ] para llamar a los elementos de una tupla, iniciando desde el elemento 0 en *adelante*

```python
numero1 = mi_tuple[0]
numero1
```

Llamar multiples elementos

```python
print(mi_tuple[0:3:2])
```

### List

Lista vacia

```python
mi_lista = []
mi_lista
```

Agregar elementos a una lista

```python
mi_lista.append('Hola')
mi_lista.append('Mundo')
mi_lista
```

Lista con 3 elementos tipo string

```python
mi_lista = ['Andres', 'Andrea', 'Karen']
print(mi_lista)
print(len(mi_lista)) # len(list) devuelve el tamaño de una lista
mi_lista[0]
```

Lista con elemntos tipo float y string

```python
mi_lista = [4.5, 'hola']
print(type(mi_lista[0]))
print(type(mi_lista[1]))
mi_lista
```

### Diccionarios

```python
diccionario = {
    "Andres": [24, 173],
    "Andrea": [25, 175],
    1: 123
}
diccionario['Andres']
```

    [24, 173]

```python
lista = diccionario.get("Andrea")
print(lista, type(lista))
```

    [25, 175] <class 'list'>

```python
diccionario[1]
```

    123

```python
diccionario.pop(1)
```

    123

```python
diccionario['Alex'] = [21, 124]
diccionario
```

    {'Alex': [21, 124], 'Andrea': [25, 175], 'Andres': [24, 173]}

```python
diccionario.clear()
```

```python
diccionario
```

    {}

# Estructuras de Control
---

### Clase booleana

```python
mi_boolean = True
mi_boolean
```

    True

```python
mi_boolean = not(mi_boolean)
mi_boolean
```

    False

```python
booleano = "Andres" in diccionario
booleano
```

    False

### Declaracion If, Else y Elif

```python
a = 3

if a < 10:
  print('Menor que 10')
```

    Menor que 10

```python
if a > 10:
  print('Mayor que 10')
else:
  print('Menor que 10')
```

    Menor que 10

```python
a = float(input())

if a == 10:
  print('Igual que 10')
elif a > 10:
  print('Mayor que 10')
else:
  print('Menor que 10')
```

    1564
    Mayor que 10

### For

Se usa in para iteral en cada uno de los elementos de una lista

```python
lista = [0, 1, 2, 3, 4, 5]
for i in lista:
  print(i)
```

    0
    1
    2
    3
    4
    5

```python
lista = ['Andres', 'Andrea', 'Felipe']
for i in lista:
  print(i)
```

    Andres
    Andrea
    Felipe

Uso de range

```python
for i in range(0, 6, 1):
  print(i)
```

    0
    1
    2
    3
    4
    5

```python
lista1 = [1 ,2, 3, 4, 5]
lista2 = ['a', 'b', 'c', 'd', 'e']
lista3 = [1.73, 1.86, 1.84, 1.62, 1.70]

for i, j, k in zip(lista1, lista2, lista3):
  print(i, j, k)
```

    1 a 1.73
    2 b 1.86
    3 c 1.84
    4 d 1.62
    5 e 1.7

For else, sirve para realizar acciones en caso de no ejecutarse un "break"

```python
lista1 = [1 ,2, 3, 4, 5]
lista2 = ['a', 'b', 'c', 'd', 'e']
lista3 = [1.73, 1.86, 1.84, 1.62, 1.70]
numero = 3

for i, j, k in zip(lista1, lista2, lista3):
  print(i, j, k)
  if numero <= 1:
    break
  numero -= 1
else:
  print('Todos los elementos fueron impresos')
```

    1 a 1.73
    2 b 1.86
    3 c 1.84

### While

```python
print('hola')
```

    hola

```python
print('funciona?')
```

    funciona?

# Debugging en Jupyter Notebook

### Debug despues de un error

```python
a = 14
b = 5
b -= (a + 1)/3

Division = a / b
Division
```

```python
%debug
```

### Debugging y breakpoints

Para ejecutar el codigo paso a paso creamos una funcion Code_debug y usamos la libreria de debug de Ipython

```python
def Code_debug():
  from IPython.core.debugger import set_trace

  set_trace() # Se crea un breakpoint

  a = 14
  b = 5
  b -= (a + 1)/3

  Division = a / b

Code_debug()
```

### Debugging a funciones

```python
from IPython.core.debugger import set_trace

def Funcion1(a=1):
  set_trace()

  b = a ** 10
  c = a / b

  return c
Funcion1()
```

# Bibliotecas Numpy y Sympy

### Funciones

```python
import numpy as np

def f(x):
    return np.sqrt(x + 2)

x = np.array([-2, -1, 0, 2, 4, 6])  # Creando el vector de valores de x
y = f(x)
list(zip(x, y))
```

### Derivadas

```python
from sympy import Derivative, diff, simplify, Symbol

x = Symbol('x') # Creando el simbolo x.
fx = (2*x + 1)*(x**3 + 2)
dx = Derivative(fx, x).doit()
dx
```

```python
# simplificando los resultados
simplify(dx)
```

```python
# Derivada de segundo orden con el 3er argumento.
Derivative(fx, x, 2).doit()
```

```python
# Calculando derivada de (3x +1) / (2x)
fx = (3*x + 1) / (2*x)
dx = Derivative(fx, x).doit()
simplify(dx)
```

```python
# la función diff nos da directamente el resultado
simplify(diff(fx, x))
```

```python
# con el metodo subs sustituimos el valor de x
# para obtener el resultado numérico. Ej x = 1.
diff(fx, x).subs(x, 1)
```

### Integrales

```python
from sympy import Integral, integrate

fx = x**3 - 6*x
dx = Integral(fx, x).doit()
dx
```

```python
# la función integrate nos da el mismo resultado
integrate(fx, x)
```

```python
# Calculando integral definida para [0, 3]
Integral(fx, (x, 0, 3)).doit()
```


# Tutorial 3: Synapses and Networks

## Neuronal connectivity

Neurons are connected at specific sites called **synapses**. Usually, the axon of a **presynaptic** neuron will make contact with the dendrite (or soma) of a **postsynaptic** in what's called a **chemical synapse**: **neurotransmitters** are transferred via the tiny gap between the two neurons (**synaptic cleft**) thanks to biochemical processes. This generates a change in membrane potential at the postsynaptic site, called **postsynaptic potential**. **Electrical synapses**, also known as **gap junctions**, can also be present -- in this case, specialized proteins make a direct electrical connection between neurons. [1]

from [2]

In this tutorial we will connect two or more neurons in NEST to create our first neuronal networks. First, we will learn the basic commands following [3] and [4], and then apply these to a well-known balanced network known as the **Brunel network**. By the end of this tutorial, you should be able to understand that a network's stability and activity behavior is deeply influenced by its parametrization.

### Sources:

>[1] Wulfram Gerstner and Werner M. Kistler, "Spiking Neuron Models". Cambridge University Press, 2002

>[2] Peter Dayan and L. F. Abbot, "Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems". The MIT Press, 2001

>[3] [PyNEST tutorial: Part 2, Populations of Neurons](https://nest-simulator.readthedocs.io/en/latest/tutorials/pynest_tutorial/part_2_populations_of_neurons.html)

>[4] [PyNEST tutorial: Part 3, Connecting Networks with Synapses](https://nest-simulator.readthedocs.io/en/latest/tutorials/pynest_tutorial/part_3_connecting_networks_with_synapses.html)

## Introduction: synapses in NEST

All synapse types included in NEST can be found in the following link:

>[5] [NEST Docs: All synapse models](https://nest-simulator.readthedocs.io/en/latest/models/synapses.html)

Alternatively, you can verify all synapse models present in NEST using the following command:

```python
import nest
nest.Models('synapses')
```

    ('bernoulli_synapse',
     'bernoulli_synapse_lbl',
     'clopath_synapse',
     'clopath_synapse_lbl',
     'cont_delay_synapse',
     'cont_delay_synapse_hpc',
     'cont_delay_synapse_lbl',
     'diffusion_connection',
     'diffusion_connection_lbl',
     'gap_junction',
     'gap_junction_lbl',
     'ht_synapse',
     'ht_synapse_hpc',
     'ht_synapse_lbl',
     'quantal_stp_synapse',
     'quantal_stp_synapse_hpc',
     'quantal_stp_synapse_lbl',
     'rate_connection_delayed',
     'rate_connection_delayed_lbl',
     'rate_connection_instantaneous',
     'rate_connection_instantaneous_lbl',
     'static_synapse',
     'static_synapse_hom_w',
     'static_synapse_hom_w_hpc',
     'static_synapse_hom_w_lbl',
     'static_synapse_hpc',
     'static_synapse_lbl',
     'stdp_dopamine_synapse',
     'stdp_dopamine_synapse_hpc',
     'stdp_dopamine_synapse_lbl',
     'stdp_facetshw_synapse_hom',
     'stdp_facetshw_synapse_hom_hpc',
     'stdp_facetshw_synapse_hom_lbl',
     'stdp_pl_synapse_hom',
     'stdp_pl_synapse_hom_hpc',
     'stdp_pl_synapse_hom_lbl',
     'stdp_synapse',
     'stdp_synapse_hom',
     'stdp_synapse_hom_hpc',
     'stdp_synapse_hom_lbl',
     'stdp_synapse_hpc',
     'stdp_synapse_lbl',
     'stdp_triplet_synapse',
     'stdp_triplet_synapse_hpc',
     'stdp_triplet_synapse_lbl',
     'tsodyks2_synapse',
     'tsodyks2_synapse_hpc',
     'tsodyks2_synapse_lbl',
     'tsodyks_synapse',
     'tsodyks_synapse_hom',
     'tsodyks_synapse_hom_hpc',
     'tsodyks_synapse_hom_lbl',
     'tsodyks_synapse_hpc',
     'tsodyks_synapse_lbl',
     'vogels_sprekeler_synapse',
     'vogels_sprekeler_synapse_hpc',
     'vogels_sprekeler_synapse_lbl')

## Synapse types

The simplest synapse type is the `static_synapse`. In this case, the **synaptic weight**, a measure of how strong is the influence of the presynaptic neuron on the postsynaptic neuron, is static, i.e. does not change over time. The synaptic transmission, however, is not instantaneous, and hence a **synaptic delay** is defined as the time between the presynaptic neuron activation (**action potential**) and the moment the postsynaptic potential is generated.

In NEST, we can check the default values for all parameters using the `GetDefaults` function:

```python
nest.GetDefaults('static_synapse')
```

    {'delay': 1.0,
     'has_delay': True,
     'num_connections': 0,
     'receptor_type': 0,
     'requires_symmetric': False,
     'sizeof': 32,
     'synapse_model': <SLILiteral: static_synapse>,
     'weight': 1.0,
     'weight_recorder': -1}

Biological synapses, however, are constantly being created, destroyed and modified. Many models have been created to describe all of these processes. On synaptic modification, one of the most common models is the **spike-time dependent plasticity**. In this example, the synaptic weight changes according to the temporal order between pre and postsynaptic spike times.

In NEST, the most common STDP mechanisms are implemented in `stdp_synapse`. The change for **normalized** synaptic weights is described by

\begin{equation}
    \Delta w = \begin{cases} - \lambda f_{-}(w) \times K_-(\Delta t) & \text{if $\Delta t \leq 0$,} \\
    \lambda f_{+}(w) \times K_+(\Delta t) & \text{if $\Delta t > 0$,} \end{cases}
\end{equation}

where $\Delta t \equiv t_{post} - t_{pre}$ and the temporal filter is defined as $K_{(+,-)}(\Delta t) = \exp(-|\Delta t| / \tau_{(+,-)})$. The update functions

\begin{equation}
    f_{+}(w) = (1-w)^{\mu_{+}} \text{ and } f_{-}(w) = \alpha w^{\mu_{-}}
\end{equation}

create **synaptic potentiation** (stronger weights) when **causal spiking** is detected ($\Delta t > 0$), otherwise generating **synaptic depression** (weaker weights). This rule is also known as **temporally asymmetric Hebbian plasticity** and has been thoroughly studied under different parametrizations:

|  STDP Type          |  Parametrization                 | Ref. |
|---------------------|----------------------------------|------|
| multiplicative STDP | $\mu_{+}=\mu_{-}=1.0$            | [7]  |
| additive STDP       | $\mu_{+}=\mu_{-}=0.0$            | [8]  |
| Guetig STDP         | $\mu_{+}=\mu_{-}=[0.0, 1.0]$     | [6]  |
| van Rossum STDP     | $\mu_{+}=0.0, \mu_{-} = 1.0$     | [9]  |

### Sources:

> [6] Guetig et al. (2003). Learning input correlations through nonlinear temporally asymmetric hebbian plasticity. Journal of Neuroscience, 23:3697-3714 DOI: https://doi.org/10.1523/JNEUROSCI.23-09-03697.2003

> [7] Rubin J, Lee D, Sompolinsky H (2001). Equilibrium properties of temporally asymmetric Hebbian plasticity. Physical Review Letters, 86:364-367. DOI: https://doi.org/10.1103/PhysRevLett.86.364

> [8] Song S, Miller KD, Abbott LF (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neuroscience 3(9):919-926. DOI: https://doi.org/10.1038/78829

> [9] van Rossum MCW, Bi G-Q, Turrigiano GG (2000). Stable Hebbian learning from spike timing-dependent plasticity. Journal of Neuroscience, 20(23):8812-8821. DOI: https://doi.org/10.1523/JNEUROSCI.20-23-08812.2000

Again, we can check the default values for all parameters of `stdp_synapse` using the `GetDefaults` function:

```python
nest.GetDefaults('stdp_synapse')
```

    {'lambda': 0.01,
     'alpha': 1.0,
     'delay': 1.0,
     'has_delay': True,
     'mu_minus': 1.0,
     'mu_plus': 1.0,
     'num_connections': 0,
     'receptor_type': 0,
     'requires_symmetric': False,
     'sizeof': 96,
     'synapse_model': <SLILiteral: stdp_synapse>,
     'tau_plus': 20.0,
     'weight': 1.0,
     'weight_recorder': -1,
     'Wmax': 100.0}

You may have noticed that `tau_minus` is absent from the above list. For STDP synaptic models, the time constant of the depressing window of STDP is exceptionally a parameter of the **post-synaptic neuron**.

```python
nest.Create("iaf_psc_alpha", params={"tau_minus": 30.0})
```

    (1,)

To change the default value of an **accessible** parameter, we can use the function `SetDefaults`.

Note that *only some parameters listed by `GetDefaults` are changeable*. Please verify the details of the synapse type you want to use beforehand. [[5]](https://nest-simulator.readthedocs.io/en/latest/models/synapses.html)

```python
nest.SetDefaults("stdp_synapse",{"tau_plus": 15.0})
```

Customized variants of a synapse model can be created using `CopyModel()`, and can be used anywhere that a built-in model name can be used.

```python
nest.CopyModel("stdp_synapse","layer1_stdp_synapse",{"Wmax": 90.0})
```

When connecting multiple neurons, connectivity rules can be defined. Besides simple set-ups like `one_to_one` and `all_to_all`, sparse methods like `fixed_indegree`, `fixed_outdegree`, `fixed_total_number` and `pairwise_bernoulli` are also available. Please check [[3]](https://nest-simulator.readthedocs.io/en/latest/tutorials/pynest_tutorial/part_2_populations_of_neurons.html) to learn more details about creating and connecting neuron populations.

```python
epop1 = nest.Create("iaf_psc_delta", 10, params={"tau_m": 30.0})
epop2 = nest.Create("iaf_psc_delta", 10)
K = 5

conn_dict = {"rule": "fixed_indegree", "indegree": K}
syn_dict = {"model": "stdp_synapse", "alpha": 1.0}
nest.Connect(epop1, epop2, conn_dict, syn_dict)
```

Synaptic parameters can also be randomly distributed by assigning a dictionary to the parameter. This should contain the target distribution and its optional parameters, as listed below:

| Distributions | Keys             |
|---------------|------------------|
| `normal`      | `mu`, `sigma`    |
| `lognormal`   | `mu`, `sigma`    |
| `uniform`     | `low`, `high`    |
| `uniform_int` | `low`, `high`    |
| `binomial`    | `n`, `p`         |
| `exponential` | `lambda`         |
| `gamma`       | `order`, `scale` |
| `poisson`     | `lambda`         |

```python
neuron = nest.Create("iaf_psc_alpha")

alpha_min = 0.1
alpha_max = 2.
w_min = 0.5
w_max = 5.

syn_dict = {"model": "stdp_synapse",
            "alpha": {"distribution": "uniform", "low": alpha_min, "high": alpha_max},
            "weight": {"distribution": "uniform", "low": w_min, "high": w_max},
            "delay": 1.0}
nest.Connect(epop1, neuron, "all_to_all", syn_dict)
```

Synapse information can be retrieved from its origin, target and synapse model using `GetConnections()`:

```python
nest.GetConnections(epop1, target=epop2, synapse_model="stdp_synapse")
```

    (array('l', [2, 14, 0, 14, 0]),
     array('l', [2, 14, 0, 14, 1]),
     array('l', [2, 21, 0, 14, 3]),
     array('l', [2, 19, 0, 14, 4]),
     array('l', [2, 18, 0, 14, 5]),
     array('l', [2, 17, 0, 14, 6]),
     array('l', [2, 18, 0, 14, 7]),
     array('l', [3, 20, 0, 14, 8]),
     array('l', [3, 17, 0, 14, 10]),
     array('l', [3, 20, 0, 14, 11]),
     array('l', [3, 12, 0, 14, 12]),
     array('l', [3, 21, 0, 14, 13]),
     array('l', [3, 20, 0, 14, 14]),
     array('l', [4, 13, 0, 14, 15]),
     array('l', [4, 21, 0, 14, 16]),
     array('l', [4, 15, 0, 14, 17]),
     array('l', [4, 13, 0, 14, 18]),
     array('l', [4, 13, 0, 14, 19]),
     array('l', [4, 18, 0, 14, 20]),
     array('l', [4, 17, 0, 14, 21]),
     array('l', [4, 14, 0, 14, 22]),
     array('l', [4, 12, 0, 14, 23]),
     array('l', [4, 19, 0, 14, 24]),
     array('l', [4, 12, 0, 14, 25]),
     array('l', [4, 17, 0, 14, 26]),
     array('l', [4, 13, 0, 14, 27]),
     array('l', [5, 17, 0, 14, 29]),
     array('l', [5, 16, 0, 14, 30]),
     array('l', [5, 20, 0, 14, 31]),
     array('l', [5, 15, 0, 14, 32]),
     array('l', [6, 16, 0, 14, 34]),
     array('l', [6, 16, 0, 14, 36]),
     array('l', [7, 20, 0, 14, 37]),
     array('l', [7, 16, 0, 14, 39]),
     array('l', [7, 14, 0, 14, 40]),
     array('l', [8, 14, 0, 14, 41]),
     array('l', [8, 18, 0, 14, 42]),
     array('l', [8, 15, 0, 14, 44]),
     array('l', [8, 19, 0, 14, 45]),
     array('l', [9, 21, 0, 14, 46]),
     array('l', [9, 15, 0, 14, 48]),
     array('l', [9, 13, 0, 14, 49]),
     array('l', [9, 12, 0, 14, 50]),
     array('l', [9, 16, 0, 14, 51]),
     array('l', [9, 18, 0, 14, 52]),
     array('l', [10, 15, 0, 14, 53]),
     array('l', [10, 12, 0, 14, 54]),
     array('l', [11, 21, 0, 14, 56]),
     array('l', [11, 19, 0, 14, 57]),
     array('l', [11, 19, 0, 14, 58]))

We can then extract the data using `GetStatus()`. Specific information can be retrieved by providing a list of desired parameters.

```python
conns = nest.GetConnections(epop1, synapse_model="stdp_synapse")
conn_vals = nest.GetStatus(conns, ["target","weight"])
```

---

## Example: connecting two neurons

In this example, we will create and connect two neurons, A and B. Neuron A is of type `iaf_psc_delta` and receives external current $I_e = 376.0 pA$. This current is sufficient to elicit a spike every $\approx$ 50ms. Neuron B is solely connected to neuron A and hence can only spike if the input from A is strong enough. Let's observe how B can be influenced by A.

First, verify the activity of neuron A running the block of code below.

```python
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np

# reset kernel for new example

nest.ResetKernel()

neuron_A = nest.Create("iaf_psc_delta", 1, {"I_e": 376.0})

multimeter_A = nest.Create("multimeter", params={"withtime": True, "record_from":["V_m"]})
spikedetector_A = nest.Create("spike_detector", params={"withgid": True, "withtime": True})

nest.Connect(multimeter_A, neuron_A)
nest.Connect(neuron_A, spikedetector_A)

nest.Simulate(300.0)

# plot membrane potential and spiking activity

plt.rcParams['figure.dpi'] = 300
fig, ax = plt.subplots(2, 1, sharex=True, sharey=False)

multimeter_A_readout = nest.GetStatus(multimeter_A)[0]
V_A = multimeter_A_readout["events"]["V_m"]
t_A = multimeter_A_readout["events"]["times"]

spikedetector_A_readout = nest.GetStatus(spikedetector_A, keys="events")[0]
event_A = spikedetector_A_readout["senders"]
te_A = spikedetector_A_readout["times"]

ax[0].set_ylabel('V (mV)')

ax[0].plot(t_A, V_A, color="tab:blue", label="A")
ax[0].legend()

ax[1].set_xlabel('time (ms)')
ax[1].set_ylabel('Spike times')
ax[1].plot(te_A, event_A, ".")

ax[1].set_yticks([1])
ax[1].set_yticklabels(["A"])

plt.show()
```

Now let's create neuron B as `iaf_psc_delta` with no external current:

```python
neuron_B = nest.Create("iaf_psc_delta", 1, {"I_e": 0.0})

multimeter_B = nest.Create("multimeter", params={"withtime": True, "record_from":["V_m"]})
spikedetector_B = nest.Create("spike_detector", params={"withgid": True, "withtime": True})

nest.Connect(multimeter_B, neuron_B)
nest.Connect(neuron_B, spikedetector_B)
```

We'll use the function below to easily plot the activity of the two neurons at the same time:

```python
def plot2neurons(multimeter_A, multimeter_B, spikedetector_A, spikedetector_B):
    multimeter_A_readout = nest.GetStatus(multimeter_A)[0]
    V_A = multimeter_A_readout["events"]["V_m"]
    t_A = multimeter_A_readout["events"]["times"]

    multimeter_B_readout = nest.GetStatus(multimeter_B)[0]
    V_B = multimeter_B_readout["events"]["V_m"]
    t_B = multimeter_B_readout["events"]["times"]

    spikedetector_A_readout = nest.GetStatus(spikedetector_A, keys="events")[0]
    event_A = spikedetector_A_readout["senders"]
    te_A = spikedetector_A_readout["times"]

    spikedetector_B_readout = nest.GetStatus(spikedetector_B, keys="events")[0]
    event_B = spikedetector_B_readout["senders"]
    te_B = spikedetector_B_readout["times"]

    plt.rcParams['figure.dpi'] = 300
    fig, ax = plt.subplots(2, 1, sharex=True, sharey=False)

    ax[0].set_ylabel('V (mV)')

    ax[0].plot(t_A, V_A, color="tab:blue", label="A")
    ax[0].plot(t_B, V_B, color="tab:orange", label="B")
    ax[0].legend()

    ax[1].set_ylim(0,5)
    ax[1].set_yticks([1,4])
    ax[1].set_yticklabels(["A","B"])
    ax[1].set_xlabel('time (ms)')
    ax[1].set_ylabel('Spike times')

    ax[1].plot(te_A, event_A, ".", color="tab:blue")
    ax[1].plot(te_B, event_B, ".", color="tab:orange")

    plt.show()
```

```python
nest.Simulate(300.0)
plot2neurons(multimeter_A, multimeter_B, spikedetector_A, spikedetector_B)
```

Neuron B is still inactive. We need to connect both neurons:

```python
nest.Connect(neuron_A, neuron_B, {"rule": "one_to_one"}, {"model": "static_synapse"})

nest.Simulate(300.0)
plot2neurons(multimeter_A, multimeter_B, spikedetector_A, spikedetector_B)
```

### Suggest at least 3 alterations we can make in this example to elicit spiking activity from B.

Run the code below and check if B spikes using the spike time raster (bottommost graph).

Additionally, check what happens if the neuron model is different: for example, change `iaf_psc_delta` to `hh_psc_alpha`.

Finally, connect B to A reciprocally and check their dynamics. How can you describe their behavior?

```python
nest.ResetKernel()

neuron_A = nest.Create("iaf_psc_delta", 1, {"I_e": 376.0})

multimeter_A = nest.Create("multimeter", params={"withtime": True, "record_from":["V_m"]})
spikedetector_A = nest.Create("spike_detector", params={"withgid": True, "withtime": True})

nest.Connect(multimeter_A, neuron_A)
nest.Connect(neuron_A, spikedetector_A)

neuron_B = nest.Create("iaf_psc_delta", 1, {"I_e": 0.0})

multimeter_B = nest.Create("multimeter", params={"withtime": True, "record_from":["V_m"]})
spikedetector_B = nest.Create("spike_detector", params={"withgid": True, "withtime": True})

nest.Connect(multimeter_B, neuron_B)
nest.Connect(neuron_B, spikedetector_B)

nest.Connect(neuron_A, neuron_B, {"rule": "one_to_one"}, {"model": "static_synapse", "weight": 1.})

nest.Simulate(300.0)
plot2neurons(multimeter_A, multimeter_B, spikedetector_A, spikedetector_B)
```

```python

```

```python

```


# Advanced free energy analyses with SOMD

In this notebook we will explore advanced analysis techniques for alchemical free enregy calculations, using Sire tool `analyse_freenrg mbar`.
Initially, you will learn how to correctly interpret phase space overlap matrix. Following, advanced commands will be shown to enhance the mbar analysis.

The notebook forms part of the CCPBio-Sim workshop **Alchemical Free Energy Simulation Analysis with analyse_freenrg** run on the 11th of April 2018 at the University of Bristol.

*Author: Anotnia Mey & Stefano Bosisio
Email: antonia.mey@ed.ac.uk*

**Reading time of the document: xx mins**

## Overlap matrix

```python
%pylab inline
import glob
import seaborn as sbn
sbn.set_style("ticks")
sbn.set_context("notebook", font_scale = 2)
```

### The overlap matrix

can be used to look at the phase space overlap of neighbouring lambdas.
By adding the flag `--overlap` this matrix will be automatically computed and added to the output file.

So let's look at the overlap matrix for a simulation of a host-guest system, shown in figure, obtained by running 16 $\lambda$ windows of 8 ns length each.
This time we will write an output file called `good_overlap.dat`

```python
%%capture run_info_good
#Let's run the analysis again with the keyword --overlap
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i good_overlap/lambda-*/simfile.dat -o good_overlap.dat --subsampling --overlap
```

```python
#A helper function to read the overlap matrix from file
def get_overlp_matrix(filename, lambda_val ):
    fh = open (filename, 'r')
    lines = fh.readlines()
    fh.close()
    count = 0
    matrix = []
    for line in lines:
        if line.startswith('#Overlap'):
            matrix = lines[(count+1):(count+1+lambda_vals)]
            break
        count = count+1
    for i in range(len(matrix)):
        temp = matrix[i].strip().split(' ')
        float_temp = [float(j) for j in temp]
        matrix[i] = float_temp
    matrix =np.array(matrix)
    return matrix
```

```python
#an exercise could be add the lambda_vals in the function above
good_overlap = get_overlp_matrix('good_overlap.dat',16)

```

### Plotting the overlap matrix
The plotting library has a nice advanced heat map feature that allows you to not only plot a pictorial image of a matrix or heatmap but also add the numercal values making it easier to read the plot

```python
fig = figure(figsize=(12,12))
ax = sbn.heatmap(good_overlap, annot=True, fmt='.2f', linewidths=.5, annot_kws={"size": 12})
ax.set_xlabel(r'$\lambda$ index')
ax.set_ylabel(r'$\lambda$ index')
ax.set_title('Good overlap matrix')
```

### Example of a bad overlap matrix
Below we have the same simulation as before, but reducing the number of lambda windows from 11 to 6. What do you observe in terms of the overlap matrix?

Belowe we have the same simulation as before, but the number of $\lambda$ windows was reduced from 16 to 10. What do you observe in terms of the overlap matrix?
Repeat the procedure above and create a file called `bad_overlap.dat`

```python
%%capture run_info_bad
#Let's run the analysis again with the keyword --overlap

```

```python
bad_overlap = get_overlp_matrix('bad_overlap.dat',10)
```

```python
fig = figure(figsize=(12,12))
ax = sbn.heatmap(good_overlap, annot=True, fmt='.2f', linewidths=.5, annot_kws={"size": 12})
ax.set_xlabel(r'$\lambda$ index')
ax.set_ylabel(r'$\lambda$ index')
ax.set_title('Bad overlap matrix')
```

### Advanced tasks

As an advanced task we are going to deal with mbar command `subsampling` and `discard`. The former option performs a subsamplin operation over all the samles data, written in the `simfile.dat` files. The second choice, `discard`, will discard a number of frames from the beginning of the simulation. This is beneficial for our estimation, as in the very first frames the system is usually equilibrating, giving rise to noise to the final free energy calculation.

We are going to study a host-guest system, similar to the one shown before. To compute the binding free energy, the following thermodynamic cycle is adopted:

Initially, a `discharging` step is performed, where ligand's charges are turned off both in solvated and  bound phases. Following, a `vanishing` step is done, by  switching off ligand's Lennard Jones terms, in order to have a fully decouple molecule.
The final free energy of binding is computed, by summing up the contribution of each leg of the cycle, as:
\begin{equation}
\Delta G_\mathrm{bind} = (\Delta G^\mathrm{solv}_\mathrm{elec} + \Delta G^\mathrm{solv}_\mathrm{vdW}) - (\Delta G^\mathrm{host}_\mathrm{elec} + \Delta G^\mathrm{host}_\mathrm{vdW} )
\end{equation}

Try to perform these steps:
1. Compute the discharging and vanishing free energy for the solvated and bound phase using the standard `analyse_freenrg mbar` command. Thus, retrieve the binding free energy using the equation above
2. Compute the discharging and vanishing free energy by adding the option `--subsampling` and discarding the first 500 frames (`discard 500`). What do you notice? What is happening to TI? and MBAR? What is the final binding free energy?
3. What conclusions can you draw from the previous points?

```python
#1. Compute the discharging and vanishing free energy for the solvated and bound phase:
#bound phase -discharging
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/bound/run001/discharge/output/lambda-*/simfile.dat -o bound_discharge.dat
#bound phase -vanishing
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/bound/run001/vanish/output/lambda-*/simfile.dat -o bound_vanish.dat
#solvated phase -discharging
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/free/run001/discharge/output/lambda-*/simfile.dat -o free_discharge.dat
#solvated phase -vanishing
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/free/run001/vanish/output/lambda-*/simfile.dat -o free_vanish.dat

```

```python
#helper function to extract DG from the freenrg_analysis generated files.
def get_free_energy(ifile):
    reader = open(ifile,"r").readlines()
    mbar = float(reader[-3].split(",")[0])
    ti = float(reader[-1].split()[0])
    return mbar, ti
```

```python
#Extract the free energy changes from the output files
bound_discharge_mbar, bound_discharge_ti = get_free_energy('bound_discharge.dat')
bound_vanish_mbar,bound_vanish_ti = get_free_energy('bound_vanish.dat')

free_discharge_mbar,free_discharge_ti = get_free_energy('free_discharge.dat')
free_vanish_mbar, free_vanish_ti = get_free_energy('free_vanish.dat')

#Compute the free energy change in the bound and water phase for mbar
DG_bound_mbar =
DG_free_mbar =
#Compute the free energy change in the bound and water phase for TI
DG_bound_ti =
DG_free_ti =

#Thus, using the equation above, compute the estimation of binding free energy using mbar and TI
DG_bind_mbar = DG_free_mbar - DG_bound_mbar
DG_bind_ti =

#What is the binding free energy with MBAR? and with TI?

```

```python
#3. Try to re run using the --discard option. What do you notice?
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/bound/run001/discharge/output/lambda-*/simfile.dat -o bound_discharge.dat --subsampling  --discard 500
#bound phase -vanishing
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/bound/run001/vanish/output/lambda-*/simfile.dat -o bound_vanish.dat --subsampling  --discard 500
#solvated phase -discharging
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/free/run001/discharge/output/lambda-*/simfile.dat -o free_discharge.dat --subsampling  --discard 500
#solvated phase -vanishing
!~/sire_2018/sire.app/bin/analyse_freenrg mbar -i subsampling/free/run001/vanish/output/lambda-*/simfile.dat -o free_vanish.dat --subsampling --discard 500

```

```python
#Extract the free enery change from the output files
bound_discharge_mbar, bound_discharge_ti = get_free_energy('bound_discharge.dat')
bound_vanish_mbar,bound_vanish_ti = get_free_energy('bound_vanish.dat')

free_discharge_mbar,free_discharge_ti = get_free_energy('free_discharge.dat')
free_vanish_mbar, free_vanish_ti = get_free_energy('free_vanish.dat')

#Compute the free energy change in the bound and water phase for mbar
DG_bound_mbar =
DG_free_mbar =
#Compute the free energy change in the bound and water phase for TI
DG_bound_ti =
DG_free_ti =

#Thus, using the equation above, compute the estimation of binding free energy using mbar and TI
DG_bind_mbar =
DG_bind_ti =

#What is the binding free energy with MBAR? and with TI?

```

The experimental standard binding free energy $\Delta G^\circ_\mathrm{bind}$ is 7.08 $\pm$ 0.01 kcal$\cdot$mol$^{-1}$.
Although our predictions are quite far apart, the subsampling and discard show an improvement of about  1 kcal$\cdot$mol$^{-1}$ on the final MBAR free energy estimation.  On the other side, TI fails to correctly predict the binding in this case. This is due to the noise present along the simulation, which greatly influences the free energy gradients values, thus the final free energy integration. Such a behaviour does not exist in MBAR, which is always consistent for all the evaluation process.
Finally, it is worth to mention that this is just the first step toward accurate predictions. What w ehave obtained here is just a binding free energy $\Delta G_\mathrm{bind}$ which cannot be compared to the experimental value. Indeed, we are missing the definition of a standard state for the simulation, thus a standard state correction term for the free energy evaluation. Additionally, we should consider also Lennard Jones corrections and electrostatic finite size artefacts correction, which can remarkably improve the final free energy estimation.

Congratulations you have finished this tutorial! Time for a coffee or tea break :-)