# Programming for Data Analysis Project

## The Task
For this project I will be synthesizing a dataset, to do so I will be using various functions of the numpy.random package to create interpretations of real world phenomenon, a lot of the work done for this notebook will be in determining what this data looks like in reality so that the synthetic data is a fairly good reflection of it.

An example to illustrate this (as provided by the lecturer), is to consider students enrolled in a module, based on this example it seems sensible to consider each student has a grade at the end of a module and as we know from the common misconception that lecturers grade to a bell curve, is in fact due to the how phenomenon such as this tends to follow a normal distribution.
Following on from this we can assert that there are other factors that will help to determine their grade, such as their level of education going into this module (it stands to reason that someone with a PhD will in general outperform someone who has a bachelors degree), the amount of hours a student studies is also probably effecting their grade as is perhaps whether the student is full time or part time.

## My Synthetic Data Idea -  A representation of Irish adults prior to retirement Gender, Height, Weight and Age

My idea here is to simulate a dataset for adults based on their Gender, Height, Weight and Age, for this I will be considering only those from the age of 18 to 64.

- Gender is straight forward to start with as the population can be reasonably divided into half female and half male.

- Age is another direct factor to consider as it is wiill be independent of height and weight as neither of these will influence someones age.

- Height will be exclusively influenced by their gender, one of the assumptions made here is that a person stops getting taller after 18, which is not strictly true and likewise as people get past a certain age they also do lose some height.

- Weight will be influenced by all of the other factors, as someones gender and height will factor into their weight, determining weight will be broken up into two steps.
    - First is to determine their BMI category using the proportion of the population by gender that fall under each of the BMI categories.
    - Second is to use that BMI category to determine a value for BMI and using that value with the previously determined height, calculate the weight.

### Exploring the variables in more detail and their relationships

In general with dependent and independent variables a variable can only be one or the other, however for this project as I am generating the variables in different steps I think it is fair to say that they all are dependent varaibles in a certain context, even though they are not directly influenced by the other variables within the dataset.

What I mean by this is that Gender is something that is technically an independent variable for each of the other variables but when generating the different genders it is itself effected by the proportion of the overall population, which I will be using to determine the likelihood of any individual being female or male, therefore in that context Gender is a dependent variable and the independent variable is the female to male ratio of the population in Ireland.

#### Dependent variables and the Independent Variables that effect them
[Gender] <= [Proportion of Females to Males in Population]

[Age] <= [Proportion of Ages in Population], [Gender]

[Height] <= [Gender]

[Weight] <= [Gender], [Height]

***

## Determining Gender and Age using 2016 Census Data

On the face of it gender appears to be an easy 50 50 chance but for the sake of accuracy and to check the veracity of that claim I took a look at the Central Statistics Office (CSO) Census data for Men and Women, handily the CSO page also includes information for their ages as well.
[Link to CSO page on Men and Women](https://www.cso.ie/en/releasesandpublications/ep/p-cp3oy/cp3/assr/)

Based on the most recent census data (from 2016) the ratio of men to women is not quite equal, there is a minor difference between the ratio of men to women between the ages 15-64 of 980 men to 1000 women. More interestingly however the ratio of men to women broadens significantly when looking at the 65+ age group with 871 men to 1000 women.

Looking more indepth into the data we can more accurately determine how age is distributed from the ages of 18-64 as the census data is freely available to download, I took a look at the dataset and have used it to determine the actual ratio of Female's to Male's  across the Irish population (as of 2016 at least), with this dataset I have also been able to determine how age is distributed across the genders also.

As mentioned above, this dataset was taken from the CSO https://data.cso.ie/table/EY007 (you can find the downloaded dataset in csv format in the Data folder of this repository), it contains the Census data for 2016 which gives the numbers of individuals by gender across different ages.

```python
#Setting up imports for the rest of the notebook
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(rc={'figure.figsize':(11.7,8.27)}) #setting seaborn plot size
plt.style.use('ggplot') #setting plot style
```

```python
#Seed a random number generator
rng = np.random.default_rng(1)

cso_df = pd.read_csv('Data\cso population 2011-2016.csv') #reading in csv file for dataset

#filter down to 2016, filter out all rows containing Both Sexes and include only rows showing population
cso_df = cso_df[(cso_df['CensusYear']==2016) & (cso_df['Sex']!='Both sexes') & (cso_df['At Each Year of Age']=='Population') ]

#Convert column containing age to be numeric and assign to new column
cso_df['Age'] = pd.to_numeric(cso_df['C02076V03371'],errors='coerce')

#drop unecessary columns as there is a lot of unecessary columns and confusingly named
cso_df = cso_df.drop(['STATISTIC', 'C02199V02655', 'Age Last Birthday', 'At Each Year of Age', 'UNIT', 'C02076V03371', 'C02786V03355', 'C02786V03355', 'TLIST(A1)', 'Statistic'], axis=1)

#filter down to the age group of interest 18 to 64
cso_df = cso_df[(cso_df['Age'] >=18) & (cso_df['Age'] <=64)].reset_index(drop=True)

#subset the dataframe into a male and female set, this is done to make getting proportions of the population simpler
cso_male_df = cso_df[cso_df['Sex']=='Male'].reset_index(drop=True)
cso_female_df = cso_df[cso_df['Sex']=='Female'].reset_index(drop=True)

# Now that dataframes have been cleaned and subsetted to the relevant data and groupings,
# can get some statistics on the population

#get total female population
total_female_pop = cso_female_df['VALUE'].sum()
#get total male population
total_male_pop = cso_male_df['VALUE'].sum()
#total population as sum of female and male population
total_pop = total_male_pop + total_female_pop
#get percentage of population female
female_percentage = total_female_pop/total_pop
#get percentage of population male
male_percentage = total_male_pop/total_pop
print('Total women between ages 18-64 -', total_female_pop)
print('Total men between ages 18-64 - ', total_male_pop)
print('Total population between ages 18-64 -', total_pop)
print('Females as percentage of population -', female_percentage)
print('Males as percentage of population -', male_percentage)
```

    Total women between ages 18-64 - 1484990
    Total men between ages 18-64 -  1448806
    Total population between ages 18-64 - 2933796
    Females as percentage of population - 0.5061667546073415
    Males as percentage of population - 0.49383324539265855

Based on the above I can now start to synthesize some fake data as I now have some representative figures for what percentage of the population Females and Males make up, so this is my first step.

### Generating Gender data

```python
#Creating a dataframe that starts with single column for gender
df = pd.DataFrame(data=rng.choice(['Female','Male'], size=500, p=[female_percentage, male_percentage]), columns=['Gender'])
#Taking a peak at table so far
df
```

      Gender

      0
      Male

      1
      Male

      2
      Female

      3
      Male

      4
      Female

      ...
      ...

      495
      Male

      496
      Male

      497
      Female

      498
      Male

      499
      Male

500 rows × 1 columns

With the first variable in my dataset generated, I now want to look at adding age, but first I want to use the female and male dataframes from before to determine the probabilities for being a certain age.

To do so I will first create some lists that track the percentage of the population for each age as well as the ages, as these will be added to empty lists during a for loop the index's of the lists will line up.

```python
#Get proportion of male population at different ages
#storing percentage, age, and count in different lists for simplicity and readability as they all get assigned in same loop
#should be no issues with ordering, see commented out dict line's below to see why it was removed
male_age_percentage_list = []
male_age_list = []
male_count_list = []
#male_age_dict = {}
for i in range(len(cso_male_df)):
    #get percentage of males at current age over total male population
    percentage = cso_male_df['VALUE'][i]/cso_male_df['VALUE'].sum()
    #append to list
    male_count_list.append(cso_male_df['VALUE'][i])
    male_age_percentage_list.append(percentage)
    male_age_list.append(int(cso_male_df['Age'][i]))
    #male_age_dict[int(cso_male_df['Age'][i])] = {'Percentage': percentage, 'Number':cso_male_df['VALUE'][i]}
```

```python
#Get proportion of female population at different ages
female_age_percentage_list = []
female_age_list = []
female_count_list = []
#female_age_dict = {}
for i in range(len(cso_male_df)):
    #get percentage of females at current age over total female population
    percentage = cso_female_df['VALUE'][i]/cso_female_df['VALUE'].sum()
    #append to list
    female_count_list.append(cso_female_df['VALUE'][i])
    female_age_percentage_list.append(percentage)
    female_age_list.append(int(cso_female_df['Age'][i]))
    #female_age_dict[int(cso_female_df['Age'][i])] = {'Percentage': percentage, 'Number':cso_female_df['VALUE'][i]}
```

```python
#visualising the distribution of age by gender
sns.scatterplot(x=cso_female_df['Age'],y=cso_female_df['VALUE'], color='b', label='female')
sns.scatterplot(x=cso_male_df['Age'],y=cso_male_df['VALUE'], color='red', label='male')
plt.title('female and male population by age (18-64)')
plt.ylabel('Number of population');
```

The above scatter plot shows the number of females and males across different ages, and shows that age is indeed not normally distributed therefore I believe the only takeaway I can get from this, is the percentage of the overall population that each age makes up and using that to then determine a persons age by using weighted probabilities with numpy's choice function.

Further inspection of the scatter plot shows a few interesting bits, firstly that there seems to be a trend downwards from teenagers into their twenties, but then a linear population increase for any age up till around 38 years, whereafter it then has a linear descent. The descent after 38 makes sense intuitively to me, naturally as people age they have more chance of having died to some accident or to some disease. what makes less sense is that there is a population dip at around 20 years of age, I would presume that this is due to a tendency in first world countries to have declining birth rates as a country becomes more developed, however this would not explain the fact that there appears to be significantly more 18 and 19 year olds than 22 year olds.

### Generating Age Data
Now that we've determined the probabilities for the various ages across males and females, in the next cell I will use this to assign ages based on the individuals gender. This will update the existing dataframe to now include a new column for our variable Age.

```python
#loop through dataframe with iterrows
for index, row in df.iterrows():
    #if current row's gender is female
    if(row['Gender'] == 'Female'):
        #Age is set using numpy's choice function, within potential ages in female_age_list and corresponding probabilities in female_age_percentage_list
        df.at[index,'Age'] = rng.choice(female_age_list, p=female_age_percentage_list)
    elif(row['Gender'] == 'Male'):
        df.at[index,'Age']  = rng.choice(male_age_list, p=male_age_percentage_list)

df['Age'] = df['Age'].astype('int') #converting type to int as it defaults to float and want integer for age
df #check data frame so far
```

      Gender
      Age

      0
      Male
      37

      1
      Male
      19

      2
      Female
      26

      3
      Male
      51

      4
      Female
      22

      ...
      ...
      ...

      495
      Male
      43

      496
      Male
      61

      497
      Female
      18

      498
      Male
      57

      499
      Male
      62

500 rows × 2 columns

***
### Brief aside - getting mean and standard deviation for age
In this next section I just wanted to see how the mean and standard deviations varied for men and women, this has no bearing on the dataset but I thought it was interesting.

```python
female_age_sum = 0
female_age_sum_sq = 0
for i in range(len(female_age_list)):
    female_age_sum += female_age_list[i]*female_count_list[i]
mean_female_age = female_age_sum/total_female_pop

female_age_sum_sq = 0
for i in range(len(female_age_list)):
    female_age_sum_sq += female_count_list[i]*((female_age_list[i]-mean_female_age)**2)

stddev_female_age = np.sqrt(female_age_sum_sq/total_female_pop)

print(mean_female_age)
print(stddev_female_age)
```

    40.318919992727224
    12.717799985196626

```python
male_age_sum = 0
male_age_sum_sq = 0
for i in range(len(male_age_list)):
    male_age_sum += male_age_list[i]*male_count_list[i]
mean_male_age = male_age_sum/total_male_pop

male_age_sum_sq = 0
for i in range(len(male_age_list)):
    male_age_sum_sq += male_count_list[i]*((male_age_list[i]-mean_male_age)**2)

stddev_male_age = np.sqrt(male_age_sum_sq/total_male_pop)

print(mean_male_age)
print(stddev_male_age)
```

    40.343749266637495
    12.814326493317076

Unsurprisingly the mean and standard deviation don't differ much, as we could see from the earlier scatter plot, they tend to track fairly closesly to one another.
***

### Determining and adding Female and Male heights as variables

This was somewhat tricky to find, which was surprising and I could not find any figures specific to Ireland so instead I am using the information available from [OurWorldInData](https://ourworldindata.org/human-height)

The above linked page gives several useful pieces of information for determining height
- Height is normally distributed
- Females have an average height of 164.7cm
    - With a Std. Deviation of 7.07cm
- Males have an average height of 178.4cm
    - With a Std. Deviation of 7.59cm

With the above information I can now start simulating data for adult Female and Male heights, however I would like to point out some of the generalisations made at this point.

The average height and std. deviations listed above are not necessarily the case for Ireland, in fact they probably aren't totally accurate, however in the absence of any truly accurate figures for Ireland these will make a sufficient approximation.
Additionally the data presented from the link above uses a relatively young group of adults, which may further skew how the average height and std. deviation would be in reality as potentially for adults born in the 1970's their average height could be shorter than adults who were born in more recent years.

That aside, using the figures from above and applying them to my dataset using numpy's normal distribution function is done in the next code cell.

### Generating Height Data

```python
#Loop through each row of dataframe so far and check whether Female or Male
for index, row in df.iterrows():
    #depending on gender use normal distribution with different mean and standard deviation
    if(row['Gender'] == 'Female'):
        df.at[index,'Height (cm)'] = np.round(rng.normal(loc=164.7,scale=7.59, size=1),2)
    elif(row['Gender'] == 'Male'):
        df.at[index,'Height (cm)']  = np.round(rng.normal(loc=178.4,scale=7.07, size=1),2)
```

```python
df.describe()
```

      Age
      Height (cm)

      count
      500.000000
      500.000000

      mean
      41.024000
      171.094480

      std
      12.991882
      10.406555

      min
      18.000000
      145.190000

      25%
      31.000000
      163.865000

      50%
      40.000000
      170.525000

      75%
      53.000000
      178.580000

      max
      64.000000
      200.930000

```python
df.head()
```

      Gender
      Age
      Height (cm)

      0
      Male
      37
      173.02

      1
      Male
      19
      171.25

      2
      Female
      26
      171.39

      3
      Male
      51
      180.32

      4
      Female
      22
      173.00

```python
df.tail()
```

      Gender
      Age
      Height (cm)

      495
      Male
      43
      175.40

      496
      Male
      61
      171.25

      497
      Female
      18
      164.98

      498
      Male
      57
      173.80

      499
      Male
      62
      168.15

## Determining Weight using Gender

I will be determining weight using Central Statistics Office Data from 2019 for BMI categories, this is for those 15 years and older which is slightly younger than the dataset aged 18-64 I'm looking at with the synthetic dataset, but I think it is close enough to not present too much of an issue.

https://data.cso.ie/table/IH308 - CSO dataset for BMI category across age groups.

https://data.cso.ie/table/IH307 - CSO dataset for BMI category across gender.

While we can get data on BMI category by gender and across different age groups, we don't have a combined dataset from the CSO which unfortunately means that we can only look at either in isolation so will have to pick whether to base the BMI category on age groupings or gender.

For the purposes of this synthetic dataset I am going to go with gender.

Using the numbers from the BMI category across gender for 2019 gives the following:
- Female
    - Obese 25%
    - Overweight 24%
    - Normal weight 46%
    - Under weight 4%
- Male
    - Obese 26%
    - Overweight 35%
    - Normal weight 35%
    - Under weight 4%

As you may have noticed from the above the total percentages for females adds up to ony 99% total which presents a problem for picking an individuals BMI category, this has most likely happened due to rounding of the data by the CSO which has unfortunately resulted in the missing percentage.
To get around this I will be basing the new percentages off the 99% we have available.

Knowing the percentage of women and men that fall under the different BMI categories I am going to use this information to determine for a given individual what category they fall under.

Once it is determined what BMI category they fall under I need to determine their BMI, for 'Normal weight' and 'Overweight' this is pretty straight forward as we know the upper and lower limits of their BMI's and using a uniform distribution I can determine this. However taking this same approach for 'Underweight' and 'Obese' presents a problem, firstly we need to set an upper and lower limit as there's probably no individuals who have a BMI of 0 or 1, similarly for Obese there are probably some people who have a BMI greater than 50 but that is probably less likely than individuals who have a BMI around 35, so I will need to determine some way to account for both of these situations.

From searching I could find no reasonable distributions to represent BMI across 'Underweight' and 'Obese', so if I were to use Normal Distribution's for each I would have to infer a mean and standard deviation value for each instead I'll set some fairly arbitrary upper and lower limits for each and use a Uniform Distribution. - Rationale for this being, that while the dataset is looking at adults across the different BMI categories from under weight to obese it excludes individuals that fall below a BMI of 13 and those above 45.

Once BMI is figured out, weight can easily be calculated using the following formula for BMI as we now know both the height and BMI:
\begin{align}
\ BMI & = \frac{m}{h^{2}} \\
\end{align}

Where m is mass (or Weight) in kg and h is height in cm.

So now to determine weight I will first introduce a new variable for weight category, for whether an individual is health or over weight, then based off that gives a range of possible valeus for BMI within those categories.

```python
bmi_categories = ['Underweight', 'Normal weight', 'Overweight', 'Obese']
p_female_bmi = [.04/0.99,.46/0.99,.24/0.99,.25/0.99] #Unfortunately the provided figures for female bmi percentages only add up to .99
p_male_bmi = [.04,.35,.35,.26]

for index, row in df.iterrows(): #loop through each row in dataframe
    if(row['Gender'] == 'Female'): #if row is gender then there is a 48.4% chance they are over weight
        df.at[index,'BMI Category'] = rng.choice(bmi_categories, p=p_female_bmi)
        if(df.at[index,'BMI Category'] =='Obese'): #depending on whether this person was over or healthy weight
            bmi_lower = 30
            bmi_upper = 45
        elif(df.at[index,'BMI Category'] =='Overweight'):
            bmi_upper = 30
            bmi_lower = 25
        elif(df.at[index,'BMI Category'] =='Normal weight'):
            bmi_upper = 25
            bmi_lower = 18.5
        elif(df.at[index,'BMI Category'] =='Underweight'):
            bmi_upper = 18.5
            bmi_lower = 13
    elif(row['Gender'] == 'Male'):
        df.at[index,'BMI Category'] = rng.choice(bmi_categories, p=p_male_bmi)
        if(df.at[index,'BMI Category'] =='Obese'): #depending on whether this person was over or healthy weight
            bmi_lower = 30
            bmi_upper = 45
        elif(df.at[index,'BMI Category'] =='Overweight'):
            bmi_upper = 30
            bmi_lower = 25
        elif(df.at[index,'BMI Category'] =='Normal weight'):
            bmi_upper = 25
            bmi_lower = 18.5
        elif(df.at[index,'BMI Category'] =='Underweight'):
            bmi_upper = 18.5
            bmi_lower = 13
    bmi = rng.uniform(bmi_lower,bmi_upper)
    height = df.at[index,'Height (cm)']/100
    weight = bmi*(height**2)
    df.at[index, 'Weight (KG)'] = np.round(weight, 3)
```

```python
df.to_csv('synthetic-adult-data.csv')
```

```python
sns.scatterplot( x='Weight (KG)', y='Height (cm)', data=df, hue='BMI Category', style='Gender');
```

```python
sns.scatterplot( x='Age', y='Weight (KG)', data=df, hue='BMI Category', style='Gender');
```

```python
sns.countplot( x='Age', data=df, hue='Gender' );
```

### Showing final Data Frame

```python
#print entire dataframe using to_string() pandas function
print('DataFrame Preview:\n',df.to_string())
#print stat summary description of dataframe
print('\n\nDescription of DataFrame:\n',df.describe())
```

    DataFrame Preview:
          Gender  Age  Height (cm)   BMI Category  Weight (KG)
    0      Male   37       173.02     Overweight       81.655
    1      Male   19       171.25    Underweight       46.499
    2    Female   26       171.39  Normal weight       54.440
    3      Male   51       180.32          Obese      141.961
    4    Female   22       173.00  Normal weight       68.189
    5    Female   33       171.95  Normal weight       65.887
    6      Male   30       176.69     Overweight       87.207
    7    Female   50       158.36          Obese       98.317
    8      Male   35       171.27  Normal weight       72.052
    9    Female   22       181.37  Normal weight       72.390
    10     Male   35       185.13     Overweight       99.508
    11     Male   33       182.48  Normal weight       69.560
    12   Female   49       162.01  Normal weight       58.750
    13     Male   33       176.89          Obese      108.021
    14   Female   48       161.97  Normal weight       59.993
    15   Female   41       162.13  Normal weight       57.341
    16   Female   58       146.34  Normal weight       48.301
    17   Female   50       164.12     Overweight       76.576
    18   Female   36       161.69  Normal weight       52.336
    19   Female   39       166.79    Underweight       45.369
    20     Male   39       179.78  Normal weight       59.895
    21   Female   57       171.07  Normal weight       59.355
    22   Female   25       164.49     Overweight       71.671
    23     Male   37       183.93  Normal weight       78.505
    24     Male   41       160.19     Overweight       66.214
    25     Male   37       170.51     Overweight       86.245
    26     Male   44       190.50          Obese      155.107
    27   Female   40       155.34  Normal weight       59.692
    28   Female   36       162.47  Normal weight       55.729
    29     Male   48       183.30  Normal weight       74.510
    30     Male   33       172.06     Overweight       84.190
    31   Female   44       170.08    Underweight       53.236
    32     Male   50       168.20  Normal weight       65.972
    33     Male   24       164.77          Obese      113.219
    34     Male   33       168.90     Overweight       79.001
    35     Male   61       166.61  Normal weight       58.248
    36   Female   62       164.46          Obese      108.777
    37     Male   64       176.05     Overweight       88.044
    38   Female   20       167.09  Normal weight       69.438
    39   Female   54       156.73  Normal weight       48.432
    40     Male   61       196.28     Overweight       99.574
    41     Male   59       177.62          Obese      121.882
    42     Male   49       177.98  Normal weight       59.089
    43   Female   47       168.05     Overweight       71.197
    44     Male   49       176.58  Normal weight       66.043
    45     Male   43       173.84     Overweight       88.077
    46     Male   53       178.91  Normal weight       77.283
    47     Male   40       184.80     Overweight       99.943
    48   Female   29       169.65  Normal weight       54.144
    49     Male   22       184.96     Overweight       89.214
    50     Male   59       174.32     Overweight       88.431
    51     Male   51       172.83          Obese       93.500
    52   Female   27       179.81  Normal weight       66.220
    53     Male   54       176.62  Normal weight       67.180
    54   Female   32       168.23  Normal weight       55.987
    55   Female   45       170.21  Normal weight       62.202
    56     Male   34       188.36          Obese      114.588
    57     Male   28       168.76          Obese       98.929
    58     Male   26       174.92          Obese      130.464
    59   Female   21       161.32     Overweight       74.419
    60   Female   21       162.32     Overweight       78.129
    61   Female   21       146.30          Obese       77.704
    62     Male   22       177.64  Normal weight       61.595
    63     Male   55       157.86     Overweight       65.154
    64     Male   41       180.43  Normal weight       78.892
    65   Female   24       168.70     Overweight       78.826
    66   Female   41       169.16          Obese      108.396
    67     Male   42       169.81     Overweight       81.972
    68     Male   40       185.08     Overweight       89.208
    69     Male   28       172.22  Normal weight       65.686
    70   Female   37       155.31  Normal weight       59.000
    71   Female   57       157.19  Normal weight       59.786
    72     Male   35       172.34          Obese      116.513
    73   Female   40       165.94  Normal weight       65.435
    74     Male   60       184.17     Overweight       96.879
    75   Female   29       148.68  Normal weight       44.078
    76     Male   50       174.52          Obese       94.832
    77     Male   39       184.72  Normal weight       80.486
    78     Male   53       179.79          Obese      112.076
    79     Male   35       181.95  Normal weight       76.049
    80     Male   42       167.76     Overweight       79.484
    81   Female   35       171.90  Normal weight       73.495
    82     Male   57       167.31  Normal weight       63.281
    83   Female   59       166.14  Normal weight       57.073
    84     Male   45       188.02  Normal weight       84.361
    85   Female   63       178.96     Overweight       85.364
    86     Male   50       182.02     Overweight       90.469
    87   Female   55       171.39     Overweight       76.925
    88   Female   58       160.10  Normal weight       55.057
    89   Female   31       161.05  Normal weight       48.225
    90     Male   64       186.67     Overweight       90.722
    91   Female   36       164.71          Obese       82.408
    92   Female   40       163.73  Normal weight       60.569
    93   Female   27       153.99  Normal weight       50.353
    94   Female   54       165.84  Normal weight       55.681
    95   Female   34       169.60  Normal weight       71.461
    96   Female   48       158.67  Normal weight       48.505
    97     Male   29       174.81          Obese       98.934
    98   Female   34       176.70     Overweight       88.342
    99     Male   36       179.66     Overweight       93.175
    100    Male   29       176.36  Normal weight       58.294
    101  Female   19       156.07  Normal weight       52.106
    102    Male   52       174.02  Normal weight       71.968
    103    Male   61       176.63  Normal weight       67.805
    104    Male   29       188.08     Overweight       88.601
    105  Female   26       160.07          Obese       90.633
    106    Male   34       191.09     Overweight       94.225
    107  Female   22       157.65  Normal weight       53.290
    108    Male   46       175.87     Overweight       77.874
    109  Female   35       171.10  Normal weight       63.464
    110  Female   42       162.88          Obese      103.296
    111  Female   59       158.40     Overweight       68.249
    112  Female   56       163.70  Normal weight       66.690
    113    Male   60       181.57     Overweight       98.800
    114    Male   60       166.27     Overweight       81.781
    115  Female   43       162.81     Overweight       77.158
    116  Female   40       163.87          Obese       96.097
    117  Female   19       161.64  Normal weight       64.293
    118    Male   23       183.60  Normal weight       79.528
    119  Female   41       160.38  Normal weight       55.763
    120    Male   56       175.82  Normal weight       62.733
    121    Male   37       178.67  Normal weight       79.085
    122    Male   18       183.55     Overweight       92.060
    123  Female   29       163.02     Overweight       77.674
    124  Female   51       176.21     Overweight       79.618
    125  Female   26       157.81  Normal weight       60.501
    126  Female   28       174.46     Overweight       85.077
    127  Female   32       158.19  Normal weight       48.523
    128  Female   44       163.46  Normal weight       55.083
    129  Female   55       167.46     Overweight       75.329
    130  Female   29       180.30     Overweight       88.281
    131    Male   44       184.73     Overweight      101.555
    132    Male   41       175.53  Normal weight       69.754
    133    Male   38       187.58     Overweight       91.134
    134    Male   43       176.89          Obese       99.636
    135    Male   54       172.80          Obese       98.211
    136    Male   29       181.57     Overweight       88.939
    137    Male   40       179.07     Overweight       90.340
    138    Male   22       185.14  Normal weight       81.566
    139  Female   40       160.53     Overweight       76.259
    140  Female   53       167.60  Normal weight       60.079
    141  Female   64       169.52          Obese      113.569
    142  Female   39       166.81     Overweight       76.107
    143  Female   32       164.48          Obese       81.474
    144  Female   43       159.41          Obese       94.183
    145    Male   35       184.79          Obese      127.200
    146  Female   24       165.74          Obese       86.465
    147    Male   41       181.91     Overweight       95.470
    148    Male   53       175.92          Obese      116.666
    149  Female   58       164.80  Normal weight       63.441
    150    Male   63       171.07     Overweight       75.352
    151  Female   35       165.35  Normal weight       59.500
    152  Female   30       170.51  Normal weight       71.106
    153    Male   23       179.06          Obese      132.373
    154  Female   37       171.22          Obese      102.514
    155    Male   54       180.61  Normal weight       75.804
    156  Female   30       175.05          Obese      123.862
    157  Female   56       163.26  Normal weight       49.772
    158  Female   49       152.26     Overweight       59.696
    159    Male   28       181.46  Normal weight       71.113
    160  Female   45       163.45  Normal weight       65.623
    161    Male   54       171.51  Normal weight       65.145
    162    Male   60       183.00          Obese      134.168
    163    Male   26       180.27  Normal weight       76.027
    164    Male   54       181.55  Normal weight       63.204
    165  Female   52       162.18    Underweight       43.854
    166  Female   30       166.20     Overweight       78.323
    167    Male   32       180.20     Overweight       90.642
    168  Female   62       178.17          Obese      117.130
    169    Male   35       173.56  Normal weight       60.623
    170    Male   32       178.33     Overweight       89.240
    171  Female   49       167.46  Normal weight       59.821
    172    Male   24       166.21  Normal weight       64.175
    173  Female   39       152.53          Obese       81.619
    174  Female   34       181.14          Obese      108.135
    175  Female   42       151.29          Obese       80.894
    176  Female   44       169.06     Overweight       80.059
    177  Female   46       158.26  Normal weight       55.873
    178    Male   37       173.00  Normal weight       64.815
    179  Female   58       149.65     Overweight       66.648
    180    Male   55       172.03  Normal weight       55.684
    181  Female   60       170.65     Overweight       77.317
    182  Female   38       154.16     Overweight       62.449
    183    Male   50       182.61     Overweight       85.464
    184  Female   37       167.08     Overweight       80.558
    185  Female   31       180.02          Obese      113.159
    186    Male   46       175.05     Overweight       91.209
    187  Female   61       174.29  Normal weight       72.590
    188    Male   53       184.68     Overweight      101.741
    189    Male   32       176.37     Overweight       84.847
    190    Male   29       166.64     Overweight       74.437
    191    Male   52       189.07  Normal weight       79.345
    192  Female   49       154.34  Normal weight       51.893
    193    Male   56       175.05     Overweight       82.640
    194  Female   25       171.40  Normal weight       70.213
    195  Female   56       173.17          Obese      100.525
    196  Female   37       158.28     Overweight       67.358
    197  Female   31       164.46     Overweight       68.357
    198  Female   34       162.74          Obese      113.403
    199  Female   64       173.67  Normal weight       68.366
    200    Male   62       196.11  Normal weight       76.400
    201  Female   22       150.81          Obese       93.920
    202    Male   33       184.88     Overweight       96.055
    203    Male   49       158.19          Obese       87.887
    204    Male   19       178.50     Overweight       83.310
    205    Male   19       170.56     Overweight       75.213
    206    Male   20       192.48  Normal weight       84.029
    207  Female   57       165.96  Normal weight       62.642
    208    Male   34       173.19          Obese      118.311
    209  Female   33       160.91  Normal weight       54.476
    210  Female   53       160.33  Normal weight       60.358
    211    Male   33       179.73          Obese      108.597
    212    Male   50       177.28     Overweight       88.042
    213    Male   35       183.04          Obese      134.835
    214    Male   32       189.29  Normal weight       69.429
    215    Male   36       183.36  Normal weight       66.266
    216  Female   26       155.39     Overweight       61.426
    217  Female   21       152.38     Overweight       67.167
    218    Male   57       173.06  Normal weight       72.679
    219    Male   57       185.85          Obese      105.832
    220  Female   38       162.16     Overweight       66.104
    221    Male   48       173.68     Overweight       76.635
    222  Female   56       165.75     Overweight       69.742
    223    Male   36       168.39  Normal weight       70.349
    224  Female   49       156.05          Obese       97.625
    225    Male   51       182.01          Obese      110.007
    226  Female   22       162.40  Normal weight       54.313
    227  Female   25       161.26          Obese      107.481
    228  Female   51       167.11     Overweight       75.637
    229  Female   49       160.26     Overweight       73.711
    230  Female   27       159.49     Overweight       66.407
    231    Male   55       175.15     Overweight       91.033
    232  Female   53       169.27          Obese       89.286
    233    Male   34       178.82  Normal weight       60.363
    234    Male   34       169.95     Overweight       85.178
    235    Male   23       176.26     Overweight       82.837
    236    Male   38       182.98          Obese      112.571
    237    Male   23       191.63     Overweight      108.063
    238    Male   42       170.32          Obese       92.778
    239  Female   45       167.27  Normal weight       59.178
    240  Female   43       162.74          Obese      106.732
    241  Female   21       170.97     Overweight       78.585
    242  Female   45       168.11  Normal weight       62.983
    243    Male   51       188.81          Obese      125.732
    244  Female   25       168.64  Normal weight       52.794
    245  Female   44       160.56  Normal weight       49.891
    246  Female   54       160.61  Normal weight       60.730
    247  Female   28       167.78  Normal weight       54.750
    248    Male   59       181.19  Normal weight       81.069
    249    Male   63       195.35     Overweight      101.061
    250    Male   49       190.25          Obese      160.024
    251  Female   57       155.79     Overweight       66.174
    252  Female   31       167.36  Normal weight       58.646
    253    Male   47       168.99          Obese      116.177
    254  Female   60       152.13          Obese       70.579
    255  Female   20       170.34  Normal weight       63.576
    256  Female   54       168.89     Overweight       77.445
    257  Female   30       156.82     Overweight       72.564
    258  Female   53       177.18     Overweight       82.595
    259  Female   46       157.28  Normal weight       50.550
    260  Female   53       155.84     Overweight       67.848
    261    Male   36       183.01     Overweight      100.043
    262  Female   38       166.38  Normal weight       51.889
    263  Female   60       163.48  Normal weight       58.266
    264    Male   21       177.93          Obese      139.032
    265  Female   26       166.40  Normal weight       67.169
    266  Female   63       163.64  Normal weight       59.813
    267    Male   59       177.18    Underweight       55.400
    268    Male   25       165.00     Overweight       71.012
    269  Female   63       154.51          Obese       74.693
    270  Female   31       169.34  Normal weight       61.741
    271  Female   58       159.35  Normal weight       52.880
    272  Female   59       168.84          Obese       93.931
    273    Male   19       187.13     Overweight       91.263
    274  Female   18       162.01     Overweight       68.904
    275  Female   33       166.85  Normal weight       61.034
    276    Male   60       193.08  Normal weight       86.648
    277    Male   53       183.31  Normal weight       80.453
    278  Female   35       178.68     Overweight       81.879
    279  Female   56       176.34     Overweight       78.769
    280  Female   33       170.42  Normal weight       59.636
    281  Female   34       163.85     Overweight       75.416
    282    Male   27       172.45     Overweight       77.059
    283    Male   62       170.54  Normal weight       66.784
    284  Female   53       168.39  Normal weight       58.404
    285    Male   38       181.71  Normal weight       62.072
    286    Male   31       177.17          Obese      113.756
    287    Male   62       184.40          Obese      114.391
    288    Male   36       174.13     Overweight       79.677
    289    Male   28       186.25          Obese      115.709
    290  Female   37       165.00     Overweight       81.218
    291  Female   49       164.42     Overweight       75.080
    292  Female   52       164.70  Normal weight       59.528
    293    Male   23       177.70    Underweight       43.320
    294    Male   61       195.99  Normal weight       72.232
    295    Male   47       167.01  Normal weight       59.235
    296  Female   21       166.05          Obese      120.728
    297    Male   55       187.20    Underweight       54.421
    298  Female   63       170.15     Overweight       83.268
    299  Female   55       162.41  Normal weight       48.922
    300  Female   18       177.91  Normal weight       65.020
    301    Male   39       170.85  Normal weight       58.184
    302    Male   44       169.16     Overweight       74.710
    303  Female   58       169.01  Normal weight       57.873
    304  Female   40       153.66  Normal weight       46.723
    305    Male   21       161.59     Overweight       77.511
    306    Male   60       184.64          Obese      140.699
    307  Female   61       170.48          Obese      127.371
    308    Male   39       183.58     Overweight       95.910
    309  Female   20       161.00  Normal weight       57.024
    310  Female   55       166.09          Obese      103.762
    311    Male   50       171.58     Overweight       83.287
    312    Male   63       190.66     Overweight       92.606
    313  Female   18       170.70  Normal weight       56.479
    314  Female   37       170.33          Obese      119.279
    315    Male   32       164.61     Overweight       68.700
    316    Male   23       184.11  Normal weight       68.490
    317    Male   50       188.96     Overweight       91.160
    318  Female   34       173.93  Normal weight       59.526
    319  Female   19       170.69  Normal weight       56.186
    320  Female   25       164.69  Normal weight       64.835
    321  Female   53       162.11  Normal weight       50.521
    322    Male   44       182.65          Obese      148.174
    323  Female   29       167.61     Overweight       71.258
    324  Female   48       166.49  Normal weight       53.902
    325  Female   24       162.35  Normal weight       56.994
    326    Male   47       190.29  Normal weight       85.699
    327  Female   46       155.84  Normal weight       48.012
    328  Female   18       164.53  Normal weight       60.094
    329  Female   40       160.30          Obese       99.550
    330  Female   50       172.53  Normal weight       70.603
    331    Male   51       188.51     Overweight       99.239
    332    Male   49       187.20          Obese      140.629
    333  Female   58       170.43  Normal weight       66.848
    334  Female   51       154.05  Normal weight       56.089
    335    Male   60       193.11     Overweight      103.310
    336    Male   37       182.05  Normal weight       61.897
    337  Female   19       163.88          Obese       81.793
    338    Male   27       178.55  Normal weight       68.480
    339    Male   32       176.00          Obese      118.085
    340  Female   20       167.92          Obese      114.873
    341    Male   23       180.59          Obese      137.918
    342  Female   40       161.97          Obese      116.831
    343  Female   30       145.19  Normal weight       48.704
    344  Female   32       164.43     Overweight       80.825
    345    Male   50       181.77     Overweight       89.209
    346  Female   56       168.69    Underweight       48.402
    347    Male   56       177.32    Underweight       52.090
    348  Female   35       168.14  Normal weight       64.716
    349    Male   49       185.67          Obese      121.441
    350  Female   56       161.13     Overweight       72.129
    351    Male   25       188.77  Normal weight       83.015
    352    Male   28       164.58     Overweight       78.164
    353  Female   46       155.13  Normal weight       45.580
    354  Female   31       157.83  Normal weight       47.982
    355  Female   28       146.15  Normal weight       39.751
    356  Female   35       157.11  Normal weight       51.451
    357    Male   18       187.23  Normal weight       73.637
    358  Female   37       174.12    Underweight       46.666
    359  Female   20       165.44     Overweight       72.863
    360    Male   30       190.57     Overweight      105.358
    361  Female   29       166.52     Overweight       82.173
    362    Male   33       174.88          Obese      117.318
    363    Male   24       180.11  Normal weight       69.896
    364    Male   39       177.43    Underweight       57.473
    365    Male   48       183.60  Normal weight       66.920
    366  Female   34       153.29  Normal weight       56.625
    367  Female   60       172.29          Obese      121.379
    368  Female   56       161.21          Obese       88.119
    369    Male   33       184.45          Obese      107.227
    370  Female   19       153.00  Normal weight       56.380
    371    Male   52       175.44     Overweight       91.313
    372  Female   30       165.29          Obese       90.613
    373  Female   33       171.54  Normal weight       73.353
    374    Male   43       169.79     Overweight       83.515
    375  Female   64       177.36  Normal weight       77.696
    376  Female   32       168.36     Overweight       73.040
    377    Male   39       174.68     Overweight       90.248
    378    Male   45       187.01     Overweight      104.798
    379    Male   48       179.13          Obese      120.192
    380  Female   29       173.49          Obese      124.940
    381    Male   44       194.82  Normal weight       72.646
    382    Male   42       183.83    Underweight       53.623
    383    Male   47       174.68          Obese       97.524
    384  Female   24       161.47  Normal weight       64.738
    385  Female   24       169.12          Obese       99.674
    386  Female   63       166.89          Obese      103.501
    387    Male   54       172.51  Normal weight       61.729
    388  Female   55       159.63  Normal weight       58.654
    389  Female   60       169.75  Normal weight       64.130
    390  Female   62       182.51     Overweight       91.677
    391    Male   36       159.80     Overweight       68.338
    392  Female   51       170.19  Normal weight       55.361
    393    Male   46       176.91     Overweight       78.492
    394  Female   42       170.62     Overweight       83.753
    395  Female   47       171.29          Obese      116.260
    396    Male   33       178.77          Obese      114.210
    397  Female   56       156.99  Normal weight       50.225
    398  Female   36       154.10    Underweight       36.589
    399  Female   25       150.47  Normal weight       48.713
    400    Male   18       187.04    Underweight       48.558
    401    Male   46       176.92  Normal weight       66.236
    402    Male   29       174.58  Normal weight       63.599
    403  Female   62       162.59  Normal weight       50.768
    404    Male   27       168.93          Obese      111.631
    405  Female   33       167.01          Obese      108.921
    406  Female   63       174.71     Overweight       81.863
    407    Male   41       186.45          Obese      128.412
    408  Female   60       152.82          Obese       86.428
    409  Female   52       175.05     Overweight       86.756
    410    Male   24       173.00     Overweight       76.185
    411  Female   29       150.18  Normal weight       48.184
    412    Male   36       169.71     Overweight       75.099
    413  Female   57       157.36  Normal weight       51.026
    414  Female   34       164.65     Overweight       68.986
    415  Female   38       156.87  Normal weight       56.433
    416    Male   39       167.72          Obese      106.163
    417    Male   33       176.53          Obese      127.869
    418    Male   53       192.00          Obese      118.065
    419  Female   20       159.37     Overweight       63.948
    420    Male   29       181.68     Overweight       87.885
    421  Female   54       165.31          Obese       87.343
    422  Female   45       184.87  Normal weight       72.013
    423    Male   40       164.98  Normal weight       60.084
    424  Female   35       173.01  Normal weight       55.829
    425    Male   24       169.34  Normal weight       55.902
    426  Female   38       160.20  Normal weight       63.935
    427  Female   64       163.97     Overweight       77.683
    428    Male   44       200.93  Normal weight       79.437
    429  Female   43       158.91  Normal weight       58.138
    430  Female   64       150.36  Normal weight       41.975
    431  Female   58       174.55     Overweight       83.963
    432  Female   64       158.33     Overweight       63.021
    433    Male   62       176.68          Obese      115.270
    434    Male   47       178.23  Normal weight       64.734
    435    Male   32       179.09          Obese      108.970
    436    Male   45       177.20  Normal weight       76.266
    437  Female   34       166.73  Normal weight       52.299
    438  Female   54       167.64  Normal weight       65.760
    439  Female   25       171.18  Normal weight       60.010
    440  Female   38       161.46  Normal weight       64.711
    441    Male   55       173.11     Overweight       88.341
    442    Male   38       170.88          Obese      122.103
    443  Female   44       169.20          Obese       99.025
    444    Male   22       178.07          Obese      118.487
    445    Male   32       182.31  Normal weight       66.946
    446  Female   62       168.78     Overweight       78.075
    447    Male   46       190.32          Obese      142.322
    448    Male   35       179.87     Overweight       81.732
    449  Female   40       170.41  Normal weight       59.427
    450  Female   52       155.57  Normal weight       59.391
    451    Male   53       170.76  Normal weight       67.651
    452  Female   55       162.33     Overweight       69.094
    453    Male   32       169.80          Obese      102.968
    454  Female   30       169.93          Obese       98.480
    455  Female   45       154.58     Overweight       69.026
    456    Male   41       178.88          Obese      113.171
    457    Male   43       161.02  Normal weight       58.994
    458  Female   29       156.31     Overweight       61.658
    459  Female   33       162.13     Overweight       78.100
    460    Male   51       176.57  Normal weight       70.389
    461  Female   34       151.10          Obese       81.680
    462  Female   51       165.52          Obese      119.029
    463    Male   20       185.29  Normal weight       70.056
    464  Female   61       169.62  Normal weight       70.219
    465    Male   28       173.74     Overweight       77.328
    466    Male   39       174.25  Normal weight       61.539
    467    Male   60       170.01          Obese      124.872
    468    Male   30       182.85          Obese      105.782
    469    Male   29       188.59  Normal weight       75.996
    470    Male   64       174.68  Normal weight       74.024
    471  Female   59       151.81  Normal weight       54.090
    472    Male   41       179.81     Overweight       87.338
    473    Male   20       183.40  Normal weight       80.949
    474    Male   44       159.15     Overweight       64.244
    475    Male   30       179.38     Overweight       81.167
    476    Male   28       193.70  Normal weight       91.303
    477  Female   60       170.93  Normal weight       60.365
    478  Female   45       168.88  Normal weight       63.328
    479  Female   38       167.75  Normal weight       59.034
    480    Male   21       185.91          Obese      127.963
    481  Female   35       157.56          Obese       89.505
    482  Female   56       158.28  Normal weight       49.217
    483  Female   32       167.79  Normal weight       62.546
    484  Female   33       159.50  Normal weight       49.481
    485    Male   44       180.23     Overweight       92.865
    486  Female   30       150.18     Overweight       56.601
    487  Female   42       170.45  Normal weight       66.434
    488  Female   52       171.13  Normal weight       62.543
    489    Male   32       178.71          Obese      103.211
    490  Female   39       165.95  Normal weight       64.687
    491    Male   43       189.86          Obese      117.022
    492  Female   35       156.48  Normal weight       58.300
    493    Male   29       171.94     Overweight       77.516
    494    Male   23       170.67  Normal weight       64.308
    495    Male   43       175.40  Normal weight       74.919
    496    Male   61       171.25          Obese      120.338
    497  Female   18       164.98          Obese       94.126
    498    Male   57       173.80  Normal weight       68.511
    499    Male   62       168.15  Normal weight       65.089

    Description of DataFrame:
                   Age  Height (cm)  Weight (KG)
    count  500.000000   500.000000   500.000000
    mean    41.024000   171.094480    79.028314
    std     12.991882    10.406555    22.539331
    min     18.000000   145.190000    36.589000
    25%     31.000000   163.865000    61.711250
    50%     40.000000   170.525000    75.340500
    75%     53.000000   178.580000    91.172250
    max     64.000000   200.930000   160.024000

## Conclusion and Assessing the Synthetic Data

At this point I'm pretty happy with the relationship I have shown between Gender, Weight and Height and I think it is mostly a reasonable simulation.

Gender is probably the most reliable of the variables in the dataset as it is directly based on census data with age being very similar.

Height I am fairly confident is a good representation, while not based directly on Irish heights and instead based on worldwide heights due to availability of the information.

Where I am most skeptical is of my approach around getting BMI values, ideally it would have been better to determine weight as a function of Gender, Height and Age, but I wasn't able to find a reasonable dataset or paper that would show this relationship. Instead I relied on the proportion of the population that fall under different BMI categories and working back from this under the assumption that BMI within these categories is uniformly distributed, figured out the BMI and finally using this BMI to determine the individuals weight as we already have height.

With regards to age I am happy with how this turned out as I had a good dataset to base proportion of age by gender, unfortunately as mentioned above I wasn't able to determine how to reasonably have this impact weight, so ultimately age was only generated based off Gender but had no impact on any other variables. Looking at the above scatterplot for age against weight we can see the result of not having age affect weight as there is no relationship that can be seen.

### Areas to Improve On

As mentioned in the above conclusion the main issue with this dataset is regards generating the BMI/Weight data,  with the above I took a backwards approach of first determining BMI and then weight from that. Ideally instead would like to determine weight directly from values in the variables for Gender, Age and Height which would remove the need to generate BMI information and to make the few assumptions that I've included - namely that within BMI categories they are uniformly distributed and the fixed limits for the under weight and obese categories.

I did look at comparing using uniform distribution vs. normal distribution for determining BMI, but I ultimately chose to use uniform distribution for two reasons, first was with the normal distributions I had to make up mean and std. dev. for each BMI category and secondly for the scatterplot of height v. weight it led to stratification of the different categories where very clear lines ran through the dataset as BMI categories overlapped.

For an improved version I would like to look more indepth at what is an appropriate distribution for weight, from some reading I saw that perhaps a log normal or potentially multi peaked distribution would be more appropriate so this would certainly be an area that could have improved the final dataset.
In addition to this I would like to implement a better way for having multiple variables influence a single variable so that I would have some equation for weight being determined by gender + age + height, I know a reasonable approach for this would be to look at using multiple linear regression on some dataset where there is present gender, age, height and weight, however even after reading up on it and playing around with it I don't feel confident enough in an approach to include it as part of this project.


```python
from gpgLabs.DC import *
from IPython.display import display
%matplotlib inline
```

# 1. Understanding currents, fields, charges and potentials

## Cylinder app

 - **survey**: Type of survey
 - **A**: (+) Current electrode  location
 - **B**: (-) Current electrode  location
 - **M**: (+) Potential electrode  location
 - **N**: (-) Potential electrode  location
 - **r**: radius of cylinder
 - **xc**: x location of cylinder center
 - **zc**: z location of cylinder center
 - **$\rho_1$**: Resistivity of the halfspace
 - **$\rho_2$**: Resistivity of the cylinder
 - **Field**: Field to visualize
 - **Type**: which part of the field
 - **Scale**: Linear or Log Scale visualization

```python
app = cylinder_app();
display(app)
```

    MyApp(children=(ToggleButtons(description='survey', options=('Dipole-Dipole', 'Dipole-Pole', 'Pole-Dipole', 'P…

# 2. Potential differences and Apparent Resistivities

Using the widgets contained in this notebook you will develop a better understand of what values are actually measured in a DC resistivity survey and how these measurements can be processed, plotted, inverted, and interpreted.

## Computing Apparent Resistivity

In practice we cannot measure the potentials everywhere, we are limited to those locations where we place electrodes. For each source (current electrode pair) many potential differences are measured between M and N electrode pairs to characterize the overall distribution of potentials. The widget below allows you to visualize the potentials, electric fields, and current densities from a dipole source in a simple model with 2 layers. For different electrode configurations you can measure the potential differences and see the calculated apparent resistivities.

In a uniform halfspace the potential differences can be computed by summing up the potentials at each measurement point from the different current sources based on the following equations:

\begin{align}
    V_M = \frac{\rho I}{2 \pi} \left[ \frac{1}{AM} - \frac{1}{MB} \right] \\
    V_N = \frac{\rho I}{2 \pi} \left[ \frac{1}{AN} - \frac{1}{NB} \right]
\end{align}
where $AM$, $MB$, $AN$, and $NB$ are the distances between the corresponding electrodes.

The potential difference $\Delta V_{MN}$ in a dipole-dipole survey can therefore be expressed as follows,

\begin{equation}
    \Delta V_{MN} = V_M - V_N = \rho I \underbrace{\frac{1}{2 \pi} \left[ \frac{1}{AM} - \frac{1}{MB} - \frac{1}{AN} + \frac{1}{NB} \right]}_{G}
\end{equation}

and the resistivity of the halfspace $\rho$ is equal to,

$$
    \rho = \frac{\Delta V_{MN}}{IG}
$$

In this equation $G$ is often referred to as the geometric factor.

In the case where we are not in a uniform halfspace the above equation is used to compute the apparent resistivity ($\rho_a$) which is the resistivity of the uniform halfspace which best reproduces the measured potential difference.

In the top plot the location of the A electrode is marked by the red +, the B electrode is marked by the blue -, and the M/N potential electrodes are marked by the black dots. The $V_M$ and $V_N$ potentials are printed just above and to the right of the black dots. The calculted apparent resistivity is shown in the grey box to the right. The bottom plot can show the resistivity model, the electric fields (e), potentials, or current densities (j) depending on which toggle button is selected. Some patience may be required for the plots to update after parameters have been changed.

## Two layer app

- **A**: (+) Current electrode  location
- **B**: (-) Current electrode  location
- **M**: (+) Potential electrode  location
- **N**: (-) Potential electrode  location
- **$\rho_1$**: Resistivity of the top layer
- **$\rho_2$**: Resistivity of the bottom layer
- **h**: thickness of the first layer
- **Plot**: Field to visualize
- **Type**: which part of the field

```python
app = plot_layer_potentials_app()
display(app)
```

    MyApp(children=(FloatSlider(value=-30.0, continuous_update=False, description='A', max=40.0, min=-40.0, step=1…

# 3. Building Pseudosections

2D profiles are often plotted as pseudo-sections by extending $45^{\circ}$ lines downwards from the A-B and M-N midpoints and plotting the corresponding $\Delta V_{MN}$, $\rho_a$, or misfit value at the intersection of these lines as shown below. For pole-dipole or dipole-pole surveys the $45^{\circ}$ line is simply extended from the location of the pole. By using this method of plotting, the long offset electrodes plot deeper than those with short offsets. This provides a rough idea of the region sampled by each data point, but the vertical axis of a pseudo-section is not a true depth.

In the widget below the red dot marks the midpoint of the current dipole or the location of the A electrode location in a pole-dipole array while the green dots mark the midpoints of the potential dipoles or M electrode locations in a dipole-pole array. The blue dots then mark the location in the pseudo-section where the lines from Tx and Rx midpoints intersect and the data is plotted. By stepping through the Tx (current electrode pairs) using the slider you can see how the pseudo section is built up.

The figures shown below show how the points in a pseudo-section are plotted for pole-dipole, dipole-pole, and dipole-dipole arrays. The color coding of the dots match those shown in the widget.

Basic skematic for a uniformly spaced pole-dipole array.

Basic skematic for a uniformly spaced dipole-pole array.

Basic skematic for a uniformly spaced dipole-dipole array.

## Pseudo-section app

```python
app = MidpointPseudoSectionWidget();
display(app)
```

    MyApp(children=(IntSlider(value=0, description='i', max=17), Output()), layout=Layout(align_items='stretch', d…

## DC pseudo-section app

 - **$\rho_1$**: Resistivity of the first layer (thickness of the first layer is 5m)
 - **$\rho_2$**: Resistivity of the cylinder
 - resistivity of the second layer is 1000 $\Omega$m
 - **xc**: x location of cylinder center
 - **zc**: z location of cylinder center
 - **r**: radius of cylinder
 - **surveyType**: Type of survey

```python
app = DC2DPseudoWidget()
display(app)
```

    interactive(children=(FloatText(value=1000.0, description='$\\rho_1$'), FloatText(value=1000.0, description='$…

# 4. Parametric Inversion

In this final widget you are able to forward model the apparent resistivity of a cylinder embedded in a two layered earth. Pseudo-sections of the apparent resistivity can be generated using dipole-dipole, pole-dipole, or dipole-pole arrays to see how survey geometry can distort the size, shape, and location of conductive bodies in a pseudo-section.  Due to distortion and artifacts present in pseudo-sections trying to interpret them directly is typically difficult and dangerous due to the risk of misinterpretation. Inverting the data to find a model which fits the observed data and is geologically reasonable should be standard practice.

By systematically varying the model parameters and comparing the plots of observed vs. predicted apparent resistivity a parametric inversion can be preformed by hand to find the "best" fitting model. Normalized data misfits, which provide a numerical measure of the difference between the observed and predicted data, are useful for quantifying how well and inversion model fits the observed data. The manual inversion process can be difficult and time consuming even with small examples sure as the one presented here. Therefore, numerical optimization algorithms are typically utilized to minimized the data misfit and a model objective function, which provides information about the model structure and complexity, in order to find an optimal solution.

## Parametric DC inversion app

Definition of variables:
- **$\rho_1$**: Resistivity of the first layer
- **$\rho_2$**: Resistivity of the cylinder
- **xc**: x location of cylinder center
- **zc**: z location of cylinder center
- **r**: radius of cylinder
- **predmis**: toggle which allows you to switch the bottom pannel from predicted apparent resistivity to normalized data misfit
- **suveyType**: toggle which allows you to switch between survey types.

Knonw information
- resistivity of the second layer is 1000 $\Omega$m
- thickness of the first layer is known: 5m

Unknowns are: $\rho_1$, $\rho_2$, xc, zc, and r

```python
app = DC2DfwdWidget()
display(app)
```

    MyApp(children=(FloatText(value=1000.0, description='$\\rho_1$'), FloatText(value=1000.0, description='$\\rho_…


```python
%load_ext autoreload
%autoreload 2
```

```python
from IPython.core.display import HTML
with open("style/notebook.css", "r") as f:
    s = f"<style>{f.read()}</style>"
HTML(s)
```

```python
from bokeh.plotting import figure, ColumnDataSource
from bokeh.io import show, output_notebook, push_notebook
import numpy as np
import slfractals as slf

output_notebook()
```

Fun with fractals

Exploring fractal sets defined on the complex number plane

*by Andreas Roth*

**2022-01-13**

## What are fractals?

Objects with a fractal nature exhibit the following properties:

* **self-similarity:** a fractal is composed of smaller copies of itself (not always exact)
* **complex structure on all scales:** There are rich, complex and repeating features regardless of scale

Romanesco Brassica oleracea (Author: [Richard Bartz](https://commons.wikimedia.org/wiki/User:Richard_Bartz), License: [CC BY-SA 2.5](https://creativecommons.org/licenses/by-sa/2.5/deed.en))

## The first (published) fractal in mathematics

The *Koch curve* or *Koch snowflake* is an example of a curve that is continous everywhere, and nowhere differentiable.

[Helge von Koch](https://en.wikipedia.org/wiki/Helge_von_Koch), *On a continuous curve without tangents constructible from elementary geometry* (1904)

## The Koch snowflake

```python
koch = slf.KochCurve(appearance="curve")
p = figure(plot_width=1150, plot_height=300, match_aspect=True)
p.line(source=koch.cds, x="x", y="y", line_width=3)
p.axis.visible = False
p.xgrid.grid_line_color = None
p.ygrid.grid_line_color = None
nh = show(p, notebook_handle=True)
```

```python
koch.increment(by=1)
push_notebook(handle=nh)
```

## The Koch snowflake

```python
koch2 = slf.KochCurve(appearance="snowflake")
p = figure(plot_width=500, plot_height=500, match_aspect=True)
p.line(source=koch2.cds, x="x", y="y", line_width=3)
p.axis.visible = False
p.xgrid.grid_line_color = None
p.ygrid.grid_line_color = None
koch2.increment(by=5)
show(p)
```

* self-similar ✔
* complex (?), at least infinitely detailed, repeating structure on all scales ✔

Koch Curve has *infinite length*, however *bounded area is finite*!

## Fractal sets on the complex number plane

📖 [B.Mandelbrot](https://en.wikipedia.org/wiki/Benoit_Mandelbrot), *The Fractal Geometry of Nature*, W. H. Freeman and Co. (1982)

```python
#xlim = (-0.7461760346346757, -0.7461759788974921); ylim = (0.11127920936647617, 0.11127922442548596)
from time import time
start = time()
show(slf.Fractal(xlim = (-0.7461760346346757, -0.7461759788974921), ylim = (0.11127920936647617, 0.11127922442548596), resw=1000, max_iter=2000, nproc=4).plot())
print(f"{time()-start}s")
```

## Complex numbers: an extension of the real numbers

a complex number $z\in\mathbb{C}$ has the form
$$
    z = x + i \cdot y
$$

* $i$ is the imaginary unit with the property $i^2=-1$
* $x\in\mathbb{R}$ is the real part
* $y\in\mathbb{R}$ is the imaginary part
* $|z| = \sqrt{x^2+y^2}$ is the absolute value

## Complex numbers: an extension of the real numbers

Calculations work the same way as for the real numbers, just keeping real and imaginary part separate and applying $i^2 = -1$:

\begin{align}
(1+2i)^2 &= (1+2i) \cdot (1+2i) \\
&= 1\cdot 1 + 1\cdot2i + 2i\cdot 1 + (2i)\cdot(2i) \\
&= 1+4i-4 \\
&= -3 +4i
\end{align}

and we can have all we know from other number sets, like functions of complex numbers, or sequences of them.

## Repeated evaluation of rational functions

Given any $c\in\mathbb{C}$ and $f_c(z) = z^2 + c$, look at sequences of numbers $(z_n)_{n\in\mathbb{N}}$ with

\begin{align}
z_0 &= 0 \\
z_{n+1} &=f_c(z_n)=z_n^2 +c
\end{align}

there are 2 possibilities based on $c$

* $|z_n|$ stays bounded: $\exists C\in\mathbb{R}: |z_n|<C \;\;\forall n$
* $|z_n|$ grows without bound

$f$ can be any other polynomial.

## Repeated evaluation of rational functions

e.g. $c=-1$, sequence $(z_n)_n$ alternates, bounded:

```python
slf.sequence(lambda z: z**2 - 1 , z=0, n=20)
```

e.g. $c=i$, sequence $(z_n)_n$ alternates, bounded:

```python
slf.sequence(lambda z: z**2 + 1j , z=0, n=10)
```

e.g. $c=2+2i$, sequence $(z_n)_n$ is unbounded:

```python
slf.sequence(lambda z: z**2 + 2 + 2j , z=0, n=5)
```

## coloring numbers according to behavior

We look at the *set of complex numbers where the sequence of based on repeated evaulation of $f_c(z) = z^2 + c$ stays bounded*

Those will occupy an area on the complex number plane, right?

Let's color the numbers with bounded sequence black.

```python
show(slf.Fractal(resw=500, poly=slf.mandel, max_iter=70).plot())
```

### for $f_c(z) = z^4 + c$             and $f_c(z) = z^5 + c$

```python
vp = slf.Fractal(xlim=(-1.2, 1.2), ylim=(-1.2, 1.2), resw=550, poly=slf.zp4, max_iter=35)
show(vp.plot())
```

```python
vp.comp_param["max_iter"] = 25
vp.poly = slf.zp5
show(vp.plot())
```

### Varying the exponent from 1 to 4, so $f_c(z) = z^d +c, 1 < d \le 4$

```python
vp = slf.Fractal((-1.5, 0.9), (-1.25, 1.25), resw=400, poly=lambda z, c: z + c, max_iter=50, calc_fun=slf.serial_compute)
nh = show(vp.plot(), notebook_handle=True)
```

```python
for d in np.linspace(1.1, 4, 50):
    vp.poly = lambda z, c: z**d + c
    push_notebook(handle=nh)
```

## What about the fractal properties?

Let's zoom in:

```python
lims = [
    {"xlim": (-1.8, 1.0), "ylim": (-0.8, 0.8), "max_iter":70},
    {"xlim": (-0.8469467739357451, -0.7098901379935), "ylim": (0.17007084393699703, 0.23826260403053853), "max_iter":100},
    {"xlim": (-0.8068760745573608, -0.7949446690544475), "ylim": (0.17972431554355933, 0.18531690862703154), "max_iter":300},
    {"xlim": (-0.8041723480428702, -0.8035509965822837), "ylim": (0.18154653272120028, 0.18183777803006518), "max_iter":600},
    {"xlim": (-0.8039965083142456, -0.8039795368113285), "ylim": (0.18172875474809908, 0.18173670977986836), "max_iter":1200},
    {"xlim": (-0.8039942821850602, -0.8039938347902356), "ylim": (0.18173488871237856, 0.18173509841920543), "max_iter":1500},
    {"xlim": (-0.8039941723910227, -0.8039941711732712), "ylim": (0.1817349844082922, 0.181734984979087), "max_iter":2000}

]
for lim in lims:
    mandelbrot = slf.Fractal(**lim, resw=900, poly=slf.mandel, nproc=4)
    show(mandelbrot.plot())
```

## Explore fractal sets for yourself!

* Run the interactive app!

        docker run -p 5006:5006 scherbertlemon/slfractals

* In your browser, visit http://localhost:5006

Thanks for your attention

Time for your questions!

```python
lim = {"xlim": (0.25434613209578555, 0.25827297672210214), "ylim": (-0.0015196795967191896, -0.0003614331344158294)}
show(slf.Fractal(**lim, poly=slf.mandel, resw=1000, max_iter=500, nproc=4).plot())
```

## Picture generation

Input:
* grid on the number plane
* maximum number of allowed iterations $I_{max}$
* maximum bound $V_{max}$ after which we consider a sequence unbounded

For all grid points $c$:

* compute $z_{n+1} = f(z_n)$ as long as $n<I_{max}$ and $|z_n|<V_{max}$
* store the maximum reached $n$, assign a color based on its value

Then, we render colors in each grid point as colorful pixels!


## Gradiente Descendente na prática

## O que vamos ver hoje:
* O que é machine learning
* Modelagem no contexto de ML
* Exemplo de um modelo
* Introdução - regressão linear
* Criando dados sintéticos
* O problema fundamental
* Como medir o erro
* Gradiente Descendente
* Fim

## Machine Learning

Fonte: https://www.ironhack.com/br/data-analytics/o-que-e-machine-learning

## Regressão e Classificação

Fonte: https://www.javatpoint.com/regression-vs-classification-in-machine-learning

## Modelagem e aprendizado

## Exemplo: modelo de machine learning

```python
import pandas as pd
import numpy as np
from sklearn.datasets import load_boston

dataset = load_boston()
features = dataset['data']
label = pd.Series(dataset['target'],name='MEDV')
feature_names = dataset['feature_names']
description = dataset['DESCR']

df = pd.DataFrame(features, columns=feature_names)
df = pd.concat((df,label), axis=1)
```

```python
# Dataset: Boston house prices dataset
# RM       average number of rooms per dwelling
# MEDV     Median value of owner-occupied homes in $1000's
df.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      MEDV

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## Exemplo: modelo de machine learning

```python
from sklearn.linear_model import SGDRegressor

X = df[['RM']]
y = df['MEDV']

# Fitando/treinando o modelo (regressão linear usnado stochastic gradient descent)
model = SGDRegressor(loss='squared_loss', alpha=0, eta0=0.01, max_iter=1000)
model.fit(X, y)

# Printando os coeficientes
(model.intercept_, model.coef_)
```

    (array([-6.56136165]), array([4.5346576]))

```python
# Novos exemplos
new_samples = np.array([6, 7]).reshape((-1, 1))

# Fazendo uma predição
model.predict(new_samples)
```

    array([20.64658395, 25.18124155])

## Regressão Linear e equação da reta

\begin{equation}
\hat{Y} = b + w X + \epsilon
\end{equation}

## Gradiente descendente na prática

## Contexto de negócio

Uma ONG chamada MaisAlegria, deseja mapear o grau de felicidade de diferentes regiões do Brasil para assim conseguir aplicar diferentes políticas públicas visando maximizar esse indicador. Você é um cientista social contratado para auxiliar a instituição nessa tarefa.

Nesse contexto, foi feta uma pesquisa com 500 pessoas, cuja a renda varia de 1039.00 até 10000.00, nessa pesquia é pedido que a pessoa classifique sua felicidade em uma escala de 1 a 10. A ONG deseja usar os dados dessa pesquisa para extrapolar o grau de felicidades para diferentes regiões do Brasil usando dados de renda das pessoas como proxy.

**objetivo:** Criar um modelo de regressão para prever o grau de felicidade a partir da renda

## Criando dados sintético

```python
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

# Simulando o mundo real (a pesquisa)

# Algumas pessoas podem ter opiniões diferentes sobre o nível de felicidade
error = np.random.normal(loc=0.0, scale=1, size=1000)

# Temos o mesmo número de pessoas para diferentes salários (pesquisa estratificada)
income = np.random.uniform(1039, 10000, size=1000)
income = income/1000.0

# Vamos tomar como dado, que a felicidade segue a seguinte função matemática:
# y = 1 + 0.7*x + error
# e.g when income = 10000 +> y = 1 + 0.7*10 + error = 8 + error
happiness = 1 + income*0.7 + error
```

## Criando o dataframe sintético

```python
# Criando nosso dataframe
df_survey = pd.DataFrame({'happiness' : happiness, 'income' : income})
df_survey = df_survey[(df_survey.happiness >= 1) & (df_survey.happiness <= 10)]
df_survey['income_group'] = pd.cut(df_survey['income'], 8)
df_survey.head()
```

      happiness
      income
      income_group

      0
      6.025820
      7.461963
      (6.646, 7.763]

      1
      7.993260
      9.276143
      (8.88, 9.997]

      2
      8.052671
      8.714977
      (7.763, 8.88]

      3
      5.767968
      2.642460
      (2.177, 3.294]

      4
      3.435323
      2.502463
      (2.177, 3.294]

## Distribuição do salário

```python
g = sns.countplot(df_survey.income_group)
g.set_xticklabels(g.get_xticklabels(), rotation=90)
g
```

## Plotando o error (error)

```python
sns.distplot(error)
```

## Distribuição do grau de felicidade vs salário

```python
sns.scatterplot(x='income',y='happiness', data=df_survey)
```

## Formulando o problema

Sempre que falamos de aprendizado surpervisionado (e em alguns casos de unsurpervised) em Machine Learning, estamos falando de uma função que estamos tentando **otimizar**.

 Quando falamos de otimização, estamos nos referindo a algo que queremos *minimizar* ou *maximizar*.

# Mas qual a reta perfeita?

Decidimos usar uma reta para fazer uma estimativa dos nossos dados, certo? Mas qual seriam os parametos $w$ e $b$ ideais? Lembra do que falamos no começo da aula? No caso $w$ e $b$ são os parâmetos ideais e queremos usá-los para aprender a reta. Mas como que aprendemos?

Ora, e se a gente chutasse quais são esses valores? Que tal começarmos com 0? Mas pode ser qualquer coisa!

```python
x = df_survey['income']
y = df_survey['happiness']
w=0.0
b=0.0
y_hat = w*x + b
```

```python
sns.scatterplot(x, y)
plt.plot(x, y_hat);
```

# A métrica de erro (loss function)

Visualmente, não parece que a reta resume bem os nossos dados.

*Pergunta:* Mas podemos ter alguma métrica de erro para definir isso?

Lembrem-se que estamos falando de um problema de otimização, então podemos dizer que montar uma função de erro (também conhecida como _função de custo_, ou cost function) que deve ser **minimizada**. Em outras palavras, queremos que dados $w$ e $b$, o erro seja o menor possível.

Existem várias formas de fazer isso, mas vamos definir a seguinte:

- Calculamos a diferença entre o valor real e o predito, ou seja ($y - \hat{y}$)

- Como para alguns pontos $y \geq \hat{y}$ e $y \leq \hat{y}$ para outros, os valores podem se anular. Então elevamos tudo ao quadrado, $(y - \hat{y})^2$

- Somamos para todos os pontos e tiramos a média.

Em resumo, nosso erro é definido por:

Erro = $J = \frac{1}{n}\sum{ (y_i - \hat{y_i})^2}$, para cada $i$ do nosso dataset.

Lembrando que queremos encontrar $w$ e $b$ e $\hat{y} = wx + b$, então:

$J = \frac{1}{n}\sum{ (y_i - (wx_i+b))^2}$

\begin{equation}
y_i = valor \quad real \\
(wx_i+b) = valor \quad predito \\
(y_i - (wx_i+b))^2 = Erro \quad ao \quad quadrado \\
\end{equation}

Ou seja, nós elevamos os erros ao quadrado e tiramos a média, daí o nome Mean Squared Error, uma função de custo (cost/loss) muito famosa no campo de ML.

## Erro quadrado em código

```python
def squared_error(true, pred):
    """
    true: array of true values
    pred: array of predicted values

    returns: squared error loss
    """
    return np.sum((true - pred)**2)
```

```python
fig, ax1 = plt.subplots(1,1, figsize = (7,4))

# array of same target value 10000 times
target = np.repeat(7, 15) # y real
pred = np.arange(0,15, 1) # 0, 1, 2, 3, 4, 5 ... 15

squared_loss = [squared_error(target[i], pred[i]) for i in range(len(pred))]

(y - yh)ˆ2

# plot
ax1.plot(pred, squared_loss)
ax1.set_xlabel('Predictions')
ax1.set_ylabel('Loss')
ax1.set_title("Squared Loss vs. Predictions | target=7")
plt.axvline(x=7, color='red', linestyle='--')

fig.tight_layout()
```

Acima vemos o plot de um erro quadrático quando o target de _um ponto_ é 7 e temos diferentes previsões que variam de $-\infty$ a $+\infty$. Podemos ver claramente que essa função tem um **mínimo** para um ponto. O que queremos é que diferença entre a previsão de *todos* os pontos e o valor *real* seja a menor possível :)

## Montando nossa loss (MSE) em código

```python
def predict(w, b, inputs):
    return w*inputs + b

def squared_error(true, pred):
    return np.sum((true - pred)**2)

def mse(w, b, inputs, targets):
    preds = predict(w, b, inputs)
    # squared_error = np.sum((true - pred)**2)
    return np.mean(squared_error(targets,preds))

loss = mse
```

```python
w, b = 0.0, 0.0
X = df_survey['income'].values
y = df_survey['happiness'].values
loss(w, b, inputs=X, targets=y)
```

    27427.545261886287

No caso, a _média_ dos _erros_ com os nossos dados é de $\approx 26541$ com base na reta que a gente escolheu. Como a gente consegue melhorar isso?

# Aprendendo a reta com gradiente descendente
O que queremos agora, então, é desenvolver um algoritmo que aprenda o peso $w$ e o $b$ (os parâmetos do nosso modelo)

\begin{equation}
\hat{y} = b + wx \\
loss = J = (y - (b + wx))^2 \\
\frac{\partial J}{\partial w} = \frac{\partial J}{\partial w} (y - \hat{y})^2 \\
\frac{\partial J}{\partial b} = \frac{\partial J}{\partial b} (y - \hat{y})^2
\end{equation}

##  Relembrando derivada (Intuição)

## Gradiente na função loss

# Então como usar o gradiente ??

$
\begin{align}
\text(gradiente \; w) = \frac{\partial J}{\partial w} = 2\;(y - \hat{y}) * X \\
\end{align}
$

$
\begin{align}
\text(gradiente \; b) = \frac{\partial J}{\partial b} = 2\;(y - \hat{y}) \\
\end{align}
$

$
\begin{align}
w = w - \frac{\partial J}{\partial w}*\text(learning \; rate) \\
\end{align}
$

$
\begin{align}
b = b - \frac{\partial J}{\partial b}*\text(learning \; rate) \\
\end{align}
$

A taxa de aprendizado é um parâmetro de ajuste em um algoritmo de otimização que determina o tamanho do passo em cada iteração enquanto se move em direção ao mínimo de uma loss.

Pra quem quiser se aventurar com um pouco de cáculo: [Aqui](https://towardsdatascience.com/understanding-the-mathematics-behind-gradient-descent-dde5dc9be06e)

## Vamos brincar de otimização
[Link](https://developers.google.com/machine-learning/crash-course/fitter/graph)

## Computando o gradiente (derivadas parciais em w e b)

```python
w, b = 0, 0
y_hat = predict(w, b, inputs=X)
error = y_hat-y
gradients = {
    'w' : 2*(error*X).mean(),
    'b' : 2*(error).mean()
}
gradients
```

    {'w': -63.77207775468602, 'b': -9.85809774486828}

## Atualizando os parâmetros com o gradiente

```python
L = 0.003 # Learning Rate
w_new = w - L*gradients['w'] # subtraimos pq estamos querendo minimizar)
b_new = b - L*gradients['b'] # subtraimos pq estamos querendo minimizar)

w_new = w_new.copy() #retorna um array do numpy
b_new = b_new.copy()
```

```python
# 0.7 + 1
(w_new , b_new)
```

    (0.19131623326405808, 0.02957429323460484)

## Qual é a nova reta?

```python
y_hat_new = (w_new*x + b_new)
sns.scatterplot(x,y)
plt.plot(x, y_hat_new, color='r');
```

```python
loss(w_new, b_new, inputs=X, targets=y)
```

    16675.316358681906

Nessa primeira rodada conseguimos fazer um ajuste da nossa reta e o nosso erro deu uma reduzida. O ideal é que rodemos esse algoritmos por um número  𝑛  de vezes, chamado épocas (ou epochs) e que ao final desse processo, tenha ocorrido uma convergência.

## Por fim, gradiente descendente em código

```python
L = 1e-5 # Vamos repetir varias vezes entao é melhor reduzir a intensidade
0.00001
epochs = 6000  # The number of iterations to perform gradient descent
w, b = 0, 0 # Resetando os parâmetros da minha regressão linear
loss_history = []

for i in range(epochs):
    y_hat = predict(w, b, inputs=X)
    error = y_hat-y
    gradients = {
        'w' : 2*(error*X).mean(),
        'b' : 2*(error).mean()
    }
    w = w - L*gradients['w']
    b = b - L*gradients['b']

    w = w.copy()
    b = b.copy()

    loss_history.append(loss(w, b, inputs=X, targets=y))
```

## Como está a nossa reta depois de 6 mil iterações (epochs)?

```python
plt.plot(loss_history, label='loss')
plt.xlabel('loss')
```

## Como está a nossa reta depois de 6 mil iterações (epochs)?

```python
y_hat = w*X + b
sns.scatterplot(x, y)
plt.plot(x, y_hat, color='r');
```

```python
print((w, b))
loss(w, b, inputs=X, targets=y)
```

    (0.8182309519072178, 0.14361680084615364)

    1091.891028277401

## Computando a regressão o com o Sklearn

```python
from sklearn.linear_model import LinearRegression

# happiness = 1 + income*0.7 + bias
X_reshaped = X.reshape(-1,1)
lr = LinearRegression() # Utilizando o método dos mínimos quadrados
lr.fit(x_reshaped, y)
```

    LinearRegression()

```python
lr.intercept_, lr.coef_
```

    (1.1805734310182316, array([0.6722904]))

```python
b, w
```

    (0.14361680084615364, 0.8182309519072178)

Conseguimos! Teoricamente encontramos o menor erro(aproximado) da nossa função! Essa técnica de otimização é conhecida como Gradiente Descendente (Gradient Descent) e é muito comum utilizá-la para otimizar diferentes modelos ! Principalmente quando temos um volume de dados muito grande :)