# The problem

The product development department at Dream-Inc is suggesting a software upgrade to a cleaning robot, claiming to decrease its failure rate. To quantify this claim, a test was conducted. Software
version (“treatment”), run-id (“id”), and outcome (“failure”) was recorded and provided to you. You are tasked to help senior management with guidance on an informed decision.

# Tasks

Specifically, you are asked to answer the following questions:
1. Is software version B decreasing the error rate. If yes, by how much?
2. Please quantify your confidence in statement (1)?
3. Each failed run costs Dream-Inc $2.32, the software update \$43k. Which software version should be used?

Please communicate your answers in a short presentation. Feel free to prepare back-up-slides for relevant technical details and interesting observations.

# Interpretation

First, lets cast the problem in statistical language.

Whenever a cleaning robot goes cleaning, it may succeed or fail. We can say there is a probability $p \in [0,1]$ for **any** robot succeed. This implies probability $1-p$ of failure. This is called a **Bernoulli trial** and modelled by a **Bernoulli proccess**, which requires just one **parameter**: the probability of success $p$.

Notice that what we call success is arbitrary: a success in a Bernoulli trial can represent the failure rate of the cleaning sessions. For simplicity of thought, we shall associate a cleaning session success to a Bernoulli success.

A Bernoulli proccess is represented by a **Bernoulli distribution**. Its mathematical representation is:

$
\mathcal{P}(x) =
\begin{cases}
p & \text{if $x = 1$ (success)} \\
1-p & \text{if $x = 0$ (failure)},
\end{cases}
$

where $x$ is a random variable instance. Note that $x \in \{0,1\}$, which means the result of every Bernoulli trial is a success ($x = 1$) or failure ($x = 0$). In other words, despite the probability of success being a number in the closed interval $[0,1]$, every clearning session ends with one and only one result: success **or** failure (in statistical lingo, this is called "mutually exclusive events")

The graphical representation of the Bernoulli distribution (for $p = 0.8$) is the following:

```python
# Charts
import matplotlib.pyplot as plt
%matplotlib inline

p = 0.8
plt.bar([0,1], [1-p,p], width=0.02)
plt.xlabel('P')
plt.ylabel('Probability')
```

The expected value for the Bernoulli distribution is simply the probability of success $p$:

$
\mu \equiv \langle x\rangle := \sum_{x} x \mathcal{P}(x)= 1 \cdot p + 0 \cdot (1-p) = p
$

And the variance is:

$
\sigma^2 \equiv \langle\left(x-\mu\right)^2\rangle := \sum_{x} \left(x-\mu\right)^2 \mathcal{P}(x) = (1-p)^2 \cdot p + (0-p)^2 \cdot (1-p) = p(1-p).
$

Now, back to the problem in hand: we have many cleaning session results (a **sample**), from which we can **estimate** $p$. The main idea is the following: the sample of cleaning sessions running software version 1.4 comes from the same Bernoulli proccess as the sample of cleaning sessions running 1.5? If not, we shall see some difference in success proportion for these samples. Note: when talking about samples, "proportion" is a better word than "probability", as it connects to the calculation.

There is a statistical test which matches this problem: the **test for two proportions**. In this notebook we apply this test to the dataset. We start by exploring and cleaning the data.

# Solution

First things first: the libraries we need.

```python
# Manipulation of dataframes
import pandas as pd

# Matrix/vector calculations
import numpy as np

# Normal distribution
import scipy
from scipy.stats import norm

# Charts
import matplotlib.pyplot as plt
%matplotlib inline
```

Then we check versions. This is useful whenever we need to consult documentation.

```python
print(f'Pandas version: {pd.__version__}')
print(f'Numpy version: {np.__version__}')
print(f'Scipy version: {scipy.__version__}')
```

    Pandas version: 1.1.5
    Numpy version: 1.19.5
    Scipy version: 1.4.1

Mount my Google Drive account so I can access the datasets hosted in `DATA_PATH`.

```python
from google.colab import drive
drive.mount('/content/drive')

DATA_PATH = 'drive/MyDrive/data'
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

Load the data. Notice that, because I already know the datasets, I can instruct Pandas to cast the `treatment` feature as strings instead of floats, which is appropriate because this feature is indeed categorical, not numerical.

```python
labels = pd.read_csv(f'{DATA_PATH}/label.csv', sep='|', dtype={'treatment': str})
outcomes = pd.read_csv(f'{DATA_PATH}/outcome.csv', sep='|')
```

Next we have a look at what we have in hand:

```python
outcomes.head()
```

      id
      failure

      0
      5d6c2b66-27b8-4c31-8670-c9e0cc1c9a3d
      0.0

      1
      d628fd57-c4cd-4bfd-855e-59da588a3925
      0.0

      2
      e185f40d-e88d-41bd-a2fd-2cd0fbaf7ce1
      0.0

      3
      0351b627-06b4-451e-8bbc-842e318ec84b
      1.0

      4
      1ab70e83-0947-4ef7-9a36-2367c0c68f5a
      0.0

```python
labels.head()
```

      id
      treatment

      0
      5d6c2b66-27b8-4c31-8670-c9e0cc1c9a3d
      1.4

      1
      d628fd57-c4cd-4bfd-855e-59da588a3925
      1.4

      2
      e185f40d-e88d-41bd-a2fd-2cd0fbaf7ce1
      1.5

      3
      0351b627-06b4-451e-8bbc-842e318ec84b
      1.4

      4
      1ab70e83-0947-4ef7-9a36-2367c0c68f5a
      1.4

# Checking NA (AKA blanks)

NA stands for "Not Available". This is empty data, which might happen for a lot of reasons. As they generally do not add any useful information to the analysis, we must check for their presence.

For this we first implement an utility function to help us spot NA in the datasets:

```python
def check_na(df):
  for (column_name, data) in df.iteritems():
    n_blanks = len(data) - len(data.dropna())
    if n_blanks > 0:
      print(f'Column "{column_name}" has {n_blanks} NA')
```

Then we run it against both datasets:

```python
check_na(outcomes)
check_na(labels)
```

    Column "failure" has 3 NA

Now that we know where are those NA and how often they are, we can safely drop them:

```python
outcomes = outcomes.dropna()
```

## Checking types

Now we need to inspect how computer is representing data. In the following, we see columns `id` and `treatment` in dataframe `labels` are represented as strings (`object`). That is fine.

```python
print(labels.dtypes)
```

    id           object
    treatment    object
    dtype: object

Next, in dataframe `outcomes` we see column `id` is a string, as before, while column `failure` is a float, which is ok.

```python
print(outcomes.dtypes)
```

    id          object
    failure    float64
    dtype: object

## Checking frequencies and spurious data

### Column `id`

Lets first have a look at the column `id` of both dataframes. Because they identify clearning sessions, we expect no repetition:

```python
def check_repetitions_for_id(dataframe, dataframe_name):
  size = len(dataframe.id)
  print(f'Size of dataframe "{dataframe_name}": {size} observations.')
  id_counts = outcomes.id.value_counts()
  repetitions = id_counts[id_counts > 1]

  if len(repetitions) == 0:
    print('No repetitions of ID.')
  else:
    print(id_counts[id_counts > 1])
```

```python
check_repetitions_for_id(outcomes, 'outcomes')
```

    Size of dataframe "outcomes": 2260 observations.
    aa9eed90-8ba9-4d14-aeae-24f23069c739    2
    Name: id, dtype: int64

```python
check_repetitions_for_id(labels, 'labels')
```

    Size of dataframe "labels": 2263 observations.
    aa9eed90-8ba9-4d14-aeae-24f23069c739    2
    Name: id, dtype: int64

There are repetitions! This is strange and, because their frequencies are small compared to the size of each dataframe, we just drop them and resume:

```python
outcomes = outcomes[outcomes.id != 'aa9eed90-8ba9-4d14-aeae-24f23069c739']
labels = labels[labels.id != 'aa9eed90-8ba9-4d14-aeae-24f23069c739']
```

Finally we double check:

```python
check_repetitions_for_id(outcomes, 'outcomes')
check_repetitions_for_id(labels, 'labels')
```

    Size of dataframe "outcomes": 2258 observations.
    No repetitions of ID.
    Size of dataframe "labels": 2261 observations.
    No repetitions of ID.

### Column `failure`

The column `failure` holds the result of each cleaning session, which as we have argued before, must be a success or a failure; no other value. To check this we count distinct values again, but in this case there is no problem on repetitions. The problem would be to have more than 2 possible values.

```python
outcomes.failure.value_counts()
```

     0.0      2197
     1.0        59
     100.0       1
    -1.0         1
    Name: failure, dtype: int64

Due to frequencies of zeros and ones, they seem to be our result of interest. Indeed, we can argue that in software development zeros is commonly used to represent `false` boolean values, which looks consistent with column title. In other words, zeros seem to indicate lack of failures in cleaning sessions; or successes, if you prefer. In the same fashion, ones seem to represent `true`: failures in cleaning sessions. Moreover, this is consistent with a working cleaning robot, as we expect a lot more successes than failures (otherwise this would be a bad product).

The other two values, -1 and 100, are probable spurious data: bugs in the software or problems on the data pipeline or persistence. Anyway, they are very infrequent. Because of this, we can drop them:

```python
from math import isclose

@np.vectorize
def not_noise(failure):
  return isclose(failure, 0) or isclose(failure, 1)

outcomes = outcomes[not_noise(outcomes.failure)]
```

### Column `treatment` (software version)

As we can se below, there are no spurious data in this column.

```python
labels.treatment.value_counts()
```

    1.4    1506
    1.5     755
    Name: treatment, dtype: int64

## Merging dataframes

The meaningful columns for our problem are `treatment` and `failure`. They reside in different dataframes, linked by the cleaning session ID, colum `id`. So we merge them into a single dataframe:

```python
dataset = outcomes.merge(labels, on='id')
dataset.head()
```

      id
      failure
      treatment

      0
      5d6c2b66-27b8-4c31-8670-c9e0cc1c9a3d
      0.0
      1.4

      1
      d628fd57-c4cd-4bfd-855e-59da588a3925
      0.0
      1.4

      2
      e185f40d-e88d-41bd-a2fd-2cd0fbaf7ce1
      0.0
      1.5

      3
      0351b627-06b4-451e-8bbc-842e318ec84b
      1.0
      1.4

      4
      1ab70e83-0947-4ef7-9a36-2367c0c68f5a
      0.0
      1.4

## Convert `failure` to `success` (and boolean)

On speaking about zeros and ones in column `failure` is not nice, because we have to remember what they represent. Moreover, it would be nicier to speak about success on cleaning sessions, just to match to lingo on Bernoulli proccess. Because of this, next we create another dataframe with the following properties:
- It has no ID (they are useless from now on).
- It has columns (now promoted to "features") `success` and `version`.

```python
@np.vectorize
def success(failure):
  return not bool(failure)

df = pd.DataFrame(data={
    'success': success(dataset.failure),
    'version': dataset.treatment
})

df.head()
```

      success
      version

      0
      True
      1.4

      1
      True
      1.4

      2
      True
      1.5

      3
      False
      1.4

      4
      True
      1.4

## Statistical test

The appropriate test for this problem is the two-proportions hypothesis test, which is implemented by the `proportions_ztest` module in the `statsmodels` package.

### Hypotheses

The first step is to state the hypotheses: null hypothesis $H_0$ and the alternative hypothesis $H_a$. For further simplicity, in the following we use "version A" as synonym for "version 1.4", and "version B" as a synonym for "version 1.5".

$
\begin{align}
H_0&: p_B - p_A = 0 \\
H_a&: p_B - p_A > 0
\end{align}
$

In words: we assert that there are no meaningful difference between versions 1.4 and 1.5, and enforce that, if this is not the case, the proportion of success on version 1.5 (version B) must be greater than version 1.4 (version A). This is importante because we are not interested in the case that the proportion of success for version 1.5 is smaller than that for 1.4.

And now we expect our data provide enough evidence to decide between these two hypothesis. If $H_0$ is true, than there would be no advantage to upgrade; if $H_a$ is the case, than we ought to upgrade.

But to perform the test, we need first decide about confidence level. Without any reason to argue in favor of type-I or type-II errors, we **choose** the standard **significance level** $\alpha = 5\%$, which implies **confidence level** of 95%.
$\alpha$ is, easily saying, the probability of making a wrong decision (or rejecting the $H_0$ when it is true).

```python
𝛼 = 0.05
```

Moreover, the **inequality** above implies we must use a single-tail test. In this case the critical statistics $z_\alpha$ is:

```python
z_alpha = norm.ppf(1-𝛼)
print(f'Critical value, z_alpha = {z_alpha}')
```

    Critical value, z_alpha = 1.6448536269514722

### Proportions

Now we need to calculate success proportion for each sample, as well as for both samples together (joint sample). We do this in Pandas by filtering the dataframe based on the feature `version`:

```python
version_A = df[df.version == '1.4']
version_B = df[df.version == '1.5']
```

```python
n_success_A = len(version_A[version_A.success]) # Amount of successes for 1.4
n_A = len(version_A) # Sample size for 1.4
p_A = n_success_A / n_A # Success rate (or proportion) for 1.4

n_success_B = len(version_B[version_B.success])
n_B = len(version_B)
p_B = n_success_B / n_B

# Joint successes, sample size and success proportion
n_success = n_success_A + n_success_B
n = n_A + n_B
p = n_success / n
```

```python
#Successful events for Verison A
print(n_success_A)
#Failures events for Verison A
print(n_A-n_success_A)
#Total events for Verison A
print(n_A)
#Successful events for Verison B
print(n_success_B)
#Failures events for Verison B
print(n_B-n_success_B)
#Total events for Verison B
print(n_B)
#Total successful events for Verison A and B
print(n_success)
#Total failures events for Verison A and B
print(n-n_success)
#Total of events for Verison A and B
print(n)
```

    1461
    43
    1504
    736
    16
    752
    2197
    59
    2256

```python
print(f'p_A = {p_A}')
print(f'p_B = {p_B}')
print(f'p = {p}')

print(f'{1 - p_A}')
print(f'{1 - p_B}')
print(f'{p_B-p_A}')

```

    p_A = 0.9714095744680851
    p_B = 0.9787234042553191
    p = 0.9738475177304965
    0.02859042553191493
    0.021276595744680882
    0.00731382978723405

```python
# This is a quite complex technique used just to simplify the
# calculation for the amount of decimal places to ensure only one significative
# figure in joint error for success proportion (below).
def compose(*functions):
  def composition(x):
    ans = x
    for f in reversed(functions):
      ans = f(ans)
    return ans

  return composition
```

```python
stdev_A = np.sqrt(p_A*(1-p_A))
stdev_B = np.sqrt(p_B*(1-p_B))

standard_error_A = stdev_A / np.sqrt(n_A)
standard_error_B = stdev_B / np.sqrt(n_B)
```

```python
# Joint standard error on joint success proportion
joint_standard_error = np.sqrt(stdev_A**2/n_A + stdev_B**2/n_B)

# Joint error is the standard joint error scaled by the choosed confidence level (two-tail)
joint_error = norm.ppf(0.975) * joint_standard_error

# The next lines truncate the joint error to just one significative figure.
decimal_places = compose(int, np.abs, np.floor, np.log10)
dp = decimal_places(joint_error)
truncated_joint_error = round(joint_error, dp)
print(f'Joint error: {truncated_joint_error}')

bottom_limit = (p_A - p_B) - truncated_joint_error
top_limit = (p_A - p_B) + truncated_joint_error
print(f'{round(bottom_limit, dp)} < p_A - p_B < {round(top_limit, dp)}')
```

    Joint error: 0.01
    -0.02 < p_A - p_B < 0.0

Notice that, according to confidence interval, we might have $\hat p_A > \hat p_B$ as well as $\hat p_A < \hat p_B$.

### Checking assumptions

The assumptions under which the proportion test is valid are:
1. Independent samples.
1. $n_Ap_A > 5$ and $n_A(1-p_A) > 5$
1. $n_Bp_B > 5$ and $n_B(1-p_B) > 5$

Concerning assumption 1, we argue that, as long the same cleaning robot is not present in both samples, and there are no repetitions of the same ID within each sample and in both samples, then the samples are independent, as are each event within each sample. However, it remains to check if there are any correlation among cleaning robots and the area they clean.

Concerning assumptions 2 and 3, we can compute and check explicitly:

```python
print(f'Condition (2) is satisfied: {n_A * p_A > 5 and n_A * (1-p_A) > 5}')
print(f'Condition (3) is satisfied: {n_B * p_B > 5 and n_B * (1-p_B) > 5}')
```

    Condition (2) is satisfied: True
    Condition (3) is satisfied: True

### The test

The function `proportions_ztest` implements the test for proportion. So we need to import it:

```python
from statsmodels.stats.proportion import proportions_ztest
```

```python
z_stat, p_value = proportions_ztest(
  [n_success_B, n_success_A],
  [n_B, n_A],
  alternative='larger'
)

print(f'Sample A (version 1.4): {n_success_A} successes in {n_A} tries.')
print(f'Sample B (version 1.5): {n_success_B} successes in {n_B} tries.')
print(f'Significance level (𝛼): {𝛼}')
print(f'z-statistic: {z_stat}')
print(f'p-value: {p_value}')
```

    Sample A (version 1.4): 1461 successes in 1504 tries.
    Sample B (version 1.5): 736 successes in 752 tries.
    Significance level (𝛼): 0.05
    z-statistic: 1.0261390159329062
    p-value: 0.15241303096999265

### Interpretation

The (absolute value of) $z$ statistics is smaller than $z_\alpha$, which means the result is not too extreme to be meaningful. In statistical language: the data we have does not provide enough evidence to reject the null hypothesis that $p_B - p_A = 0$. This can also be checked with $p$-value, which is greater than the significance level $\alpha = 5\%$. Indeed, the $p$-value means there are 18% chance that we observe what we observed, given the two samples come from the same population or, equivalently, that null hypothesis is true. In other words, the difference $p_A - p_B$ is not statistically meaningful. In practice it means there is no advantage (or disadvantage) to upgrade software from version 1.4 to 1.5, as it does not provide any meaningul difference in the long run.

### Graphical interpretation

First we define the probability density functions for the sampling proportion, to be reused. Note that, according to central limit theorem, the sampling proportion $\bar P \sim N\left(p, \sqrt{p(1-p)/n}\right)$. i.e., it follows gaussian distribution with mean equal to proportion $p$ and standard deviation $\sqrt{p(1-p)/n}$, where $n$ is the sample size.

```python
# The probability density function for sample A (version 1.4)
@np.vectorize
def pdf_A(x):
  return norm.pdf(x, loc=p_A, scale=standard_error_A)

# The probability density function for sample B (version 1.5)
@np.vectorize
def pdf_B(x):
  return norm.pdf(x, loc=p_B, scale=standard_error_B)
```

```python
fig, ax = plt.subplots()
plt.xlabel('P')
plt.ylabel('Probability density')

# Define the range of the chart
xs = np.linspace(0.95, 1, 100)

# Plot probability density functions
ax.plot(xs, pdf_A(xs), label='1.4')
ax.plot(xs, pdf_B(xs), label='1.5')

# Add vertical line and label for p_A
plt.vlines(p_A, 0, pdf_A(p_A), linestyles='dotted')
plt.text(p_A, 50, '$\hat p_A$', ha='right', fontsize=15)

# Add vertical line and label for p_B
plt.vlines(p_B, 0, pdf_B(p_B), linestyles='dotted')
plt.text(p_B, 50, '$\hat p_B$', ha='left', fontsize=15)

# Add legend
ax.legend();
```

As we can see above, the overlap between probability density functions is not negligible. This is a graphical evidence of the hypothesis test result.

The following chart presents a simplified view of the above result, for didatic purposes only:

```python
fig, ax = plt.subplots()
plt.xlabel('P')
plt.ylabel('Probability density')

# Define the range of the chart
xs = np.linspace(0.95, 1, 100)

# Plot probability density functions
ax.plot(xs, pdf_A(xs), label='1.4')

error = norm.ppf(1-𝛼) * standard_error_A
xs = np.linspace(0.95, p_A+error, 100)
ys = pdf_A(xs)
plt.fill_between(xs, ys, alpha=0.2)

# Add vertical line and label for p_A
plt.vlines(p_A, 0, pdf_A(p_A), linestyles='dotted')
plt.text(p_A, 50, '$\hat p_A$', ha='right', fontsize=15)

# Add vertical line and label for p_B
plt.vlines(p_B, 0, pdf_B(p_B), linestyles='dotted')
plt.text(p_B, 50, '$\hat p_B$', ha='left', fontsize=15)

plt.show()
```

In picture above, were $\hat p_B$ a population parameter, its value would be almost beyond the critical $P$ (the end of shaded area). This means obtaining $\hat p_A$ as far from $\hat p_B$ as seen above is just above 5% (significance level): the blank area.

### Hand calculation

```python
p = (n_success_A + n_success_B) / (n_A + n_B)
p
```

    0.9738475177304965

```python
z_statistic = (p_B - p_A) / np.sqrt(p*(1-p)*(1/n_A + 1/n_B))
z_statistic
```

    1.0261390159329062

```python
p_value = norm.sf(z_statistic) # sf = Survival function. Check documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bernoulli.html
p_value
```

    0.15241303096999265

### Resampling

The framework used to test the hypothesis does not impose any restriction to the difference in samples size. In any case we can perform the test with two samples of the same size. For this we only need to undersample sample A, which is greater than B.

```python
undersample_A = pd.DataFrame(data={
    'success': np.random.choice(version_A.success, size=len(version_B))
})
```

```python
n_success_A_undersample = len(undersample_A[undersample_A.success])
n_A_undersample = len(undersample_A)

z_stat, p_value = proportions_ztest(
  [n_success_A_undersample, n_success_B],
  [n_A_undersample, n_B],
  alternative='smaller'
)

print(f'Sample A (version 1.4): {n_success_A_undersample} successes in {n_A_undersample} tries.')
print(f'Sample B (version 1.5): {n_success_B} successes in {n_B} tries.')
print(f'Significance level (𝛼): {𝛼}')
print(f'z-statistic: {z_stat}')
print(f'p-value: {p_value}')
```

    Sample A (version 1.4): 723 successes in 752 tries.
    Sample B (version 1.5): 736 successes in 752 tries.
    Significance level (𝛼): 0.05
    z-statistic: -1.9675843864295657
    p-value: 0.024557938841102606

Conclusion is the same as before: $p$-value is greater than significance level $\alpha$, which means the observed result is quite probable, given both samples came from the same population. In other words, there is no distinction between success rate for samples A and B.

Equivalent analysis can be done with z-statistic: it is smaller (absolute value) than critical value ($\sim 1.6$).

### Statistical power

Cohen's d metric is a measure of effect size. It represents a measure of separation, or better saying, a measure of distance in a scale of standard deviations:

```python
cohen_effect_size = (p_B - p_A) / joint_standard_error
cohen_effect_size
```

    1.0765244677200276

The value ~0.4 for Cohen measure means we are facing a small to medium effect size.

```python
from statsmodels.stats.power import zt_ind_solve_power
```

```python
n_B
```

    752

```python
zt_ind_solve_power(effect_size=cohen_effect_size, alpha=𝛼, nobs1=n_A, ratio=n_B/n_A, alternative='larger')
```

    1.0

The above result is the _statistical power_ for the test. It means the probability of correctly reject the null hypothesis is nearly 100%, supposing the alternative hypothesis is true. That means we can safely state the conclusion (not rejecting the null hypothesis) of the above test is right.

The statistical power is also $1-\beta$, where $\beta$ is the probability of type II errors. In words, there are nearly 0% chance of not reject the null hypothesis, in the case there is an effect (difference between $p_A$ and $p_B$).

# Answers

## Question 1
_Is software version B decreasing the error rate. If yes, by how much?_

**Answer**: no. Software version B does not improve (decrease) error rate in the long run.

## Question 2

_Please quantify your confidence in statement (1)._

**Answer**: the _confidence level_ for the assertion that $p_A = p_B$ is 95%.

## Question 3

_Each failed run costs Dream-Inc $2.32, the software update \$43k. Which software version should be used?_

**Answer**: because there is no statistically significant difference on success (or failure) rates between versions 1.4 and 1.5, the recommendation is to **not** upgrade. The average loss per run is $0.06.

```python
average_loss_per_event = 2.32 * (1 - p) + 0 * p
average_loss_per_event
```

    0.06067375886524813

# Entendendo populações, amostras e o teorema do limite central

```python
# População
mu = 3
sigma = 0.6
```

```python
xs = np.linspace(-5, 10, 100)
ys = norm.pdf(xs, loc=mu, scale=sigma)
```

```python
plt.plot(xs, ys)
```

```python
sample = norm.rvs(loc=mu, scale=sigma, size=50)
```

```python
sample
```

    array([1.80369436, 2.81023381, 3.06755641, 4.33511054, 3.22751535,
           3.48403609, 2.15748917, 3.90233316, 3.43508823, 3.70749698,
           2.16882595, 3.2188533 , 3.88671796, 3.06983018, 3.42401805,
           2.35355908, 3.71932929, 2.8067929 , 2.67114737, 3.00427683,
           3.29059067, 3.67116927, 1.53672201, 4.10026456, 2.53700617,
           2.7325187 , 1.90326645, 4.36670244, 2.3343703 , 3.04541907,
           2.69271514, 2.84025933, 2.83156558, 3.52460146, 2.36602161,
           2.18574413, 2.27443182, 3.09966473, 2.39791761, 2.82202316,
           3.33109799, 3.29165111, 2.49673141, 2.78989081, 1.71422673,
           2.95079866, 3.23496988, 3.3058522 , 2.85601046, 2.55960591])

```python
x_bar = sample.mean()
x_bar
```

    2.9467542868620074

```python
s = sample_stdev = sample.std(ddof=1)
s
```

    0.6549892141005556

$\bar x = \hat\mu$

$s = \hat\sigma$

$\hat p$

```python
from scipy.stats import bernoulli
```

```python
# População
p = 0.7
```

```python
xs = [0, 1]
ys = bernoulli.pmf(xs, p=p)
```

```python
plt.bar(xs, ys)
```

mu = 0 * 0.3 + 1 * 0.7

mu = 0 * (1-p) + 1 * p

mu = p

```python
mu = p
mu
```

    0.7

```python
stdev = np.sqrt(p*(1-p))
stdev
```

    0.45825756949558405

```python
sample = bernoulli.rvs(p, size=30)
sample
```

    array([1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1,
           1, 1, 1, 0, 1, 0, 1, 1])

```python
sample_size_I_choose = 30000

sample_means = []

for n in range(1000):
  sample_mean = bernoulli.rvs(p, size=sample_size_I_choose).mean()
  sample_means.append(sample_mean)
```

```python
df = pd.DataFrame(data={'means': sample_means})
df.head()
```

      means

      0
      0.701800

      1
      0.702200

      2
      0.697367

      3
      0.703700

      4
      0.701367

```python
standard_error = stdev / np.sqrt(sample_size_I_choose)

df.hist(bins=20)
xs = np.linspace(0.68, 0.72, 100)
ys = norm.pdf(xs, loc=p, scale=standard_error) # fator de normalização obtido na "raça"
plt.plot(xs, ys)
```

```python
# standard error is the standard deviation of the mean of means
standard_error = stdev / np.sqrt(sample_size_I_choose)
standard_error
```

    0.0026457513110645908

```python
standard_error*3
```

    0.007937253933193772

```python
print(p - standard_error*3)
print(p + standard_error*3)

```

    0.6920627460668062
    0.7079372539331937

```python
# Teorema do limite central
# Distribuição normal com:
mu = p
sigma = standard_error
```

```python
# 1.5
v_esperado_B = 2.32 * (1 - p_B) + 0 * p_B
v_esperado_B
```

    0.04936170212765964

```python
# 1.4
v_esperado_A = 2.32 * (1 - p_A) + 0 * p_A
v_esperado_A
```

    0.06632978723404263

```python
v = v_esperado_A - v_esperado_B
v
```

    0.016968085106382993

$\langle f \rangle = \sum_x P(x) f(x)$

$\langle\text{prejuízo}\rangle = \sum_x P(x) \text{prejuízo}(x)$

```python
43e3/v
```

    2534169.278996863

$X \sim N(\mu, \sigma) \Rightarrow \bar X \sim N\left(\mu, \frac{\sigma}{\sqrt{n}}\right)$

$P$ is a random variable which follows the Bernoulli distribution with parameter $p$ (probability of success):

$P \sim \text{Bernoulli}(p) \Rightarrow \bar P \sim N\left(p, \frac{\sqrt{p(1-p)}}{\sqrt{n}}\right)$

## Simulation 2

```python
from scipy.stats import bernoulli
```

```python
p1 = 0.971
p2 = 0.977
```

```python
var1 = p1 * (1-p1)
var2 = p2 * (1-p2)
```

```python
n1 = 30000 # Sample size I've choosen
n2 = 20000
```

```python
polled_std = np.sqrt(var1/n1 + var2/n2)
polled_std
```

    0.0014360304082202907

```python
sample_means = []

for n in range(1000):
  sample1 = bernoulli.rvs(p1, size=n1)
  sample2 = bernoulli.rvs(p2, size=n2)

  hat_p1 = sample1.mean()
  hat_p2 = sample2.mean()

  sample_means.append(hat_p2 - hat_p1)

df = pd.DataFrame(data={'means': sample_means})
```

```python
df.hist(bins=20)

xs = np.linspace(0, 0.01, 100)
ys = norm.pdf(xs, loc=df.means.mean(), scale=polled_std)
plt.plot(xs, ys)
```

```python

```