# Customer Segmentation: Estimate Individualized Responses to Incentives

Nowadays, business decision makers rely on estimating the causal effect of interventions to answer what-if questions about shifts in strategy, such as promoting specific product with discount, adding new features to a website or increasing investment from a sales team. However, rather than learning whether to take action for a specific intervention for all users, people are increasingly interested in understanding the different responses from different users to the two alternatives. Identifying the characteristics of users having the strongest response for the intervention could help make rules to segment the future users into different groups. This can help optimize the policy to use the least resources and get the most profit.

In this case study, we will use a personalized pricing example to explain how the [EconML](https://aka.ms/econml) and [DoWhy](https://github.com/microsoft/dowhy) libraries could fit into this problem and provide robust and reliable causal solutions.

### Summary

1. [Background](#background)
2. [Data](#data)
3. [Create Causal Model and Identify Causal Effect with DoWhy](#identify)
4. [Get Causal Effects with EconML](#estimate)
5. [Test Estimate Robustness with DoWhy](#robustness)
    1. [Add Random Common Cause](#random-common-cause)
    2. [Add Unobserved Common Cause](#unobserved-common-cause)
    3. [Replace Treatment with a Random (Placebo) Variable](#placebo-variable)
    4. [Remove a Random Subset of the Data](#subset)
6. [Understand Treatment Effects with EconML](#interpret)
7. [Make Policy Decisions with EconML](#policy)
8. [Conclusions](#conclusion)

# Background

The global online media market is growing fast over the years. Media companies are always interested in attracting more users into the market and encouraging them to buy more songs or become members. In this example, we'll consider a scenario where one experiment a media company is running is to give small discount (10%, 20% or 0) to their current users based on their income level in order to boost the likelihood of their purchase. The goal is to understand the **heterogeneous price elasticity of demand** for people with different income level, learning which users would respond most strongly to a small discount. Furthermore, their end goal is to make sure that despite decreasing the price for some consumers, the demand is raised enough to boost the overall revenue.

The EconML and DoWhy libraries complement each other in implementing this solution. On one hand, the DoWhy library can help [build a causal model, indentify the causal effect](#identify) and [test causal assumptions](#robustness). On the other hand, EconML’s `DML` based estimators can be used to take the discount variation in existing data, along with a rich set of user features, to [estimate heterogeneous price sensitivities](#estimate) that vary with multiple customer features. Then, the `SingleTreeCateInterpreter` provides a [presentation-ready summary](#interpret) of the key features that explain the biggest differences in responsiveness to a discount, and the `SingleTreePolicyInterpreter` recommends a [policy](#policy) on who should receive a discount in order to increase revenue (not only demand), which could help the company to set an optimal price for those users in the future.

```python
# Some imports to get us started
# Utilities
import os
import urllib.request
import numpy as np
import pandas as pd
from networkx.drawing.nx_pydot import to_pydot
from IPython.display import Image, display

# Generic ML imports
from sklearn.preprocessing import PolynomialFeatures
from sklearn.ensemble import GradientBoostingRegressor

# DoWhy imports
import dowhy
from dowhy import CausalModel

# EconML imports
from econml.dml import LinearDML, ForestDML
from econml.cate_interpreter import SingleTreeCateInterpreter, SingleTreePolicyInterpreter

import matplotlib.pyplot as plt

%matplotlib inline
```

# Data

The dataset* has ~10,000 observations and includes 9 continuous and categorical variables that represent user's characteristics and online behaviour history such as age, log income, previous purchase, previous online time per week, etc.

We define the following variables:

Feature Name|Type|Details
:--- |:---|:---
**account_age** |W| user's account age
**age** |W|user's age
**avg_hours** |W| the average hours user was online per week in the past
**days_visited** |W| the average number of days user visited the website per week in the past
**friend_count** |W| number of friends user connected in the account
**has_membership** |W| whether the user had membership
**is_US** |W| whether the user accesses the website from the US
**songs_purchased** |W| the average songs user purchased per week in the past
**income** |X| user's income
**price** |T| the price user was exposed during the discount season (baseline price * small discount)
**demand** |Y| songs user purchased during the discount season

**To protect the privacy of the company, we use the simulated data as an example here. The data is synthetically generated and the feature distributions don't correspond to real distributions. However, the feature names have preserved their names and meaning.*

The treatment and outcome are generated using the following functions:
$$
T =
\begin{cases}
  1 & \text{with } p=0.2,  \\
  0.9 & \text{with }p=0.3, & \text{if income}<1 \\
  0.8 & \text{with }p=0.5, \\
  \\
    1 & \text{with }p=0.7, \\
  0.9 & \text{with }p=0.2, & \text{if income}\ge1 \\
  0.8 & \text{with }p=0.1, \\
\end{cases}
$$

\begin{align}
\gamma(X) & = -3 - 14 \cdot \{\text{income}<1\} \\
\beta(X,W) & = 20 + 0.5 \cdot \text{avg_hours} + 5 \cdot \{\text{days_visited}>4\} \\
Y &= \gamma(X) \cdot T + \beta(X,W)
\end{align}

```python
# Import the sample pricing data
file_url = "https://msalicedatapublic.blob.core.windows.net/datasets/Pricing/pricing_sample.csv"
train_data = pd.read_csv(file_url)
```

```python
# Data sample
train_data.head()
```

      account_age
      age
      avg_hours
      days_visited
      friends_count
      has_membership
      is_US
      songs_purchased
      income
      price
      demand

      0
      3
      53
      1.834234
      2
      8
      1
      1
      4.903237
      0.960863
      1.0
      3.917117

      1
      5
      54
      7.171411
      7
      9
      0
      1
      3.330161
      0.732487
      1.0
      11.585706

      2
      3
      33
      5.351920
      6
      9
      0
      1
      3.036203
      1.130937
      1.0
      24.675960

      3
      2
      34
      6.723551
      0
      8
      0
      1
      7.911926
      0.929197
      1.0
      6.361776

      4
      4
      30
      2.448247
      5
      8
      1
      0
      7.148967
      0.533527
      0.8
      12.624123

```python
# Define estimator inputs
train_data["log_demand"]=np.log(train_data["demand"])
train_data["log_price"]=np.log(train_data["price"])

X = train_data[["income"]]  # features
```

```python
# Get test data
X_test = np.linspace(0, 5, 100).reshape(-1, 1)
X_test_data = pd.DataFrame(X_test, columns=["income"])
```

# Create Causal Model and Identify Causal Effect with DoWhy

We define the causal assumptions with DoWhy. For example, we can include features we believe as confounders and features we think will influence the heterogeneity of the effect. With these assumptions defined, DoWhy can generate a causal graph for us, and use that graph to first identify the causal effect.

```python
common_causes=["account_age","age","avg_hours","days_visited","friends_count","has_membership","is_US","songs_purchased","income"]
```

```python
model = CausalModel(
    data=train_data,
    treatment='log_price',
    outcome='log_demand',
    common_causes=common_causes,
    effect_modifiers='income'
)
```

    WARNING:dowhy.causal_model:Causal Graph not provided. DoWhy will construct a graph based on data inputs.
    INFO:dowhy.causal_graph:If this is observed data (not from a randomized experiment), there might always be missing confounders. Adding a node named "Unobserved Confounders" to reflect this.
    INFO:dowhy.causal_model:Model to find the causal effect of treatment ['log_price'] on outcome ['log_demand']

```python
# Visualize causal graph
try:
    # Try pretty printing the graph. Requires pydot and pygraphviz
    display(
        Image(to_pydot(model._graph._graph).create_png())
    )
except:
    # Fall back on default graph view
    model.view_model()
```

```python
identified_estimand = model.identify_effect(proceed_when_unidentifiable=True)
print(identified_estimand)
```

    INFO:dowhy.causal_identifier:Common causes of treatment and outcome:['avg_hours', 'income', 'is_US', 'friends_count', 'age', 'has_membership', 'account_age', 'days_visited', 'songs_purchased', 'U']
    WARNING:dowhy.causal_identifier:If this is observed data (not from a randomized experiment), there might always be missing confounders. Causal effect cannot be identified perfectly.
    INFO:dowhy.causal_identifier:Continuing by ignoring these unobserved confounders because proceed_when_unidentifiable flag is True.
    INFO:dowhy.causal_identifier:Instrumental variables for treatment and outcome:[]

    Estimand type: nonparametric-ate
    ### Estimand : 1
    Estimand name: backdoor
    Estimand expression:
         d
    ────────────(Expectation(log_demand|avg_hours,income,is_US,friends_count,age,h
    d[log_price]

    as_membership,account_age,days_visited,songs_purchased))

    Estimand assumption 1, Unconfoundedness: If U→{log_price} and U→log_demand then P(log_demand|log_price,avg_hours,income,is_US,friends_count,age,has_membership,account_age,days_visited,songs_purchased,U) = P(log_demand|log_price,avg_hours,income,is_US,friends_count,age,has_membership,account_age,days_visited,songs_purchased)
    ### Estimand : 2
    Estimand name: iv
    No such variable found!

# Get Causal Effects with EconML

Based on the identified causal effect above, we fit the model as follows using EconML:

\begin{align}
log(Y) & = \theta(X) \cdot log(T) + f(X,W) + \epsilon \\
log(T) & = g(X,W) + \eta
\end{align}

where $\epsilon, \eta$ are uncorrelated error terms.

The models we fit here aren't an exact match for the data generation function above, but if they are a good approximation, they will allow us to create a good discount policy.  Although the model is misspecified, we hope to see that our `DML` based estimators can still capture the right trend of $\theta(X)$ and that the recommended policy beats other baseline policies (such as always giving a discount) on revenue.  Because of the mismatch between the data generating process and the model we're fitting, there isn't a single true $\theta(X)$ (the true elasticity varies with not only X but also T and W), but given how we generate the data above, we can still calculate the range of true $\theta(X)$ to compare against.

```python
# Define underlying treatment effect function given DGP
def gamma_fn(X):
    return -3 - 14 * (X["income"] < 1)

def beta_fn(X):
    return 20 + 0.5 * (X["avg_hours"]) + 5 * (X["days_visited"] > 4)

def demand_fn(data, T):
    Y = gamma_fn(data) * T + beta_fn(data)
    return Y

def true_te(x, n, stats):
    if x < 1:
        subdata = train_data[train_data["income"] < 1].sample(n=n, replace=True)
    else:
        subdata = train_data[train_data["income"] >= 1].sample(n=n, replace=True)
    te_array = subdata["price"] * gamma_fn(subdata) / (subdata["demand"])
    if stats == "mean":
        return np.mean(te_array)
    elif stats == "median":
        return np.median(te_array)
    elif isinstance(stats, int):
        return np.percentile(te_array, stats)
```

```python
# Get the estimate and range of true treatment effect
truth_te_estimate = np.apply_along_axis(true_te, 1, X_test, 1000, "mean")  # estimate
truth_te_upper = np.apply_along_axis(true_te, 1, X_test, 1000, 95)  # upper level
truth_te_lower = np.apply_along_axis(true_te, 1, X_test, 1000, 5)  # lower level
```

    INFO:numexpr.utils:NumExpr defaulting to 8 threads.

## Parametric heterogeneity
First of all, we can try to learn a **linear projection of the treatment effect** assuming a polynomial form of $\theta(X)$. We use the `LinearDML` estimator. Since we don't have any priors on these models, we use a generic gradient boosting tree estimators to learn the expected price and demand from the data.

```python
lineardml_estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.econml.dml.LinearDML",
    control_value=0,
    treatment_value=1,
    target_units=X_test_data,
    method_params={
        "init_params": {"model_y": GradientBoostingRegressor(), "model_t": GradientBoostingRegressor(),"featurizer":PolynomialFeatures(degree=2, include_bias=False)},
        "fit_params": {"inference": "statsmodels"},
    },
)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income

```python
est = lineardml_estimate._estimator_object
# Get treatment effect and its confidence interval
te_pred = est.effect(X_test).flatten()
te_pred_interval = est.effect_interval(X_test)
```

```python
# Compare the estimate and the truth
plt.figure(figsize=(10, 6))
plt.plot(X_test.flatten(), te_pred, label="Sales Elasticity Prediction")
plt.plot(X_test.flatten(), truth_te_estimate, "--", label="True Elasticity")
plt.fill_between(
    X_test.flatten(),
    te_pred_interval[0].flatten(),
    te_pred_interval[1].flatten(),
    alpha=0.2,
    label="90% Confidence Interval",
)
plt.fill_between(
    X_test.flatten(),
    truth_te_lower,
    truth_te_upper,
    alpha=0.2,
    label="True Elasticity Range",
)
plt.xlabel("Income")
plt.ylabel("Songs Sales Elasticity")
plt.title("Songs Sales Elasticity vs Income")
plt.legend(loc="lower right")
```

From the plot above, it's clear to see that the true treatment effect is a **nonlinear** function of income, with elasticity around -1.75 when income is smaller than 1 and a small negative value when income is larger than 1. The model fits a quadratic treatment effect, which is not a great fit. But it still captures the overall trend: the elasticity is negative and people are less sensitive to the price change if they have higher income.

```python
# Get the final coefficient and intercept summary
est.summary(feature_names=X.columns)
```

Coefficient Results

      	     point_estimate	 stderr	  zstat	  pvalue	 ci_lower	 ci_upper

  income	        2.333	      0.087	 26.919	    0.0	    2.19	     2.475

  income^2	     -0.401	      0.03	  -13.301	   0.0	   -0.451	   -0.352

CATE Intercept Results

         	        point_estimate	 stderr	  zstat	  pvalue	 ci_lower	 ci_upper

  cate_intercept	     -2.969	      0.052	 -57.267	   0.0	   -3.054	   -2.884

A linear parametric conditional average treatment effect (CATE) model was fitted:
$Y = \Theta(X)\cdot T + g(X, W) + \epsilon$
where for every outcome $i$ and treatment $j$ the CATE $\Theta_{ij}(X)$ has the form:
$\Theta_{ij}(X) = \phi(X)' coef_{ij} + cate\_intercept_{ij}$
where $\phi(X)$ is the output of the `featurizer` or $X$ if `featurizer`=None. Coefficient Results table portrays the $coef_{ij}$ parameter vector for each outcome $i$ and treatment $j$. Intercept Results table portrays the $cate\_intercept_{ij}$ parameter.

`LinearDML` estimator can also return the summary of the coefficients and intercept for the final model, including point estimates, p-values and confidence intervals. From the table above, we notice that $income$ has positive effect and ${income}^2$ has negative effect, and both of them are statistically significant.

## Nonparametric Heterogeneity
Since we already know the true treatment effect function is nonlinear, let us fit another model using `ForestDML`, which assumes a fully **nonparametric estimation of the treatment effect**.

```python
forestdml_estimate = model.estimate_effect(
    identified_estimand,
    method_name="backdoor.econml.dml.ForestDML",
    control_value=0,
    treatment_value=1,
    target_units=X_test_data.values,
    method_params={
        "init_params": {"model_y": GradientBoostingRegressor(), "model_t": GradientBoostingRegressor()},
        "fit_params": {"inference": "blb"},
    },
)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income

```python
est = forestdml_estimate._estimator_object
# Get treatment effect and its confidence interval
te_pred = est.effect(X_test).flatten()
te_pred_interval = est.effect_interval(X_test)
```

```python
# Compare the estimate and the truth
plt.figure(figsize=(10, 6))
plt.plot(X_test.flatten(), te_pred, label="Sales Elasticity Prediction")
plt.plot(X_test.flatten(), truth_te_estimate, "--", label="True Elasticity")
plt.fill_between(
    X_test.flatten(),
    te_pred_interval[0].flatten(),
    te_pred_interval[1].flatten(),
    alpha=0.2,
    label="90% Confidence Interval",
)
plt.fill_between(
    X_test.flatten(),
    truth_te_lower,
    truth_te_upper,
    alpha=0.2,
    label="True Elasticity Range",
)
plt.xlabel("Income")
plt.ylabel("Songs Sales Elasticity")
plt.title("Songs Sales Elasticity vs Income")
plt.legend(loc="lower right")
```

We notice that this model fits much better than the `LinearDML`, the 90% confidence interval correctly covers the true treatment effect estimate and captures the variation when income is around 1. Overall, the model shows that people with low income are much more sensitive to the price changes than higher income people.

# Test Estimate Robustness with DoWhy

### Add Random Common Cause

How robust are our estimates to adding another confounder? We use DoWhy to test this!

```python
res_random = model.refute_estimate(
    identified_estimand, forestdml_estimate, method_name="random_common_cause"
)
print(res_random)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased+w_random | income

    Refute: Add a Random Common Cause
    Estimated effect:-0.9574516356588789
    New effect:-0.9525350058337957

### Add Unobserved Common Cause

How robust are our estimates to unobserved confounders? Since we assume the model is under unconfoundedness, adding an unobserved confounder might bias the estimates. We use DoWhy to test this!

```python
res_unobserved = model.refute_estimate(
    identified_estimand,
    forestdml_estimate,
    method_name="add_unobserved_common_cause",
    confounders_effect_on_treatment="linear",
    confounders_effect_on_outcome="linear",
    effect_strength_on_treatment=0.1,
    effect_strength_on_outcome=0.1,
)
print(res_unobserved)
```

    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income

    Refute: Add an Unobserved Common Cause
    Estimated effect:-0.9574516356588789
    New effect:0.2057419839963558

### Replace Treatment with a Random (Placebo) Variable

What happens our estimates if we replace the treatment variable with noise? Ideally, the average effect would be wildly different than our original estimate. We use DoWhy to investigate!

```python
res_placebo = model.refute_estimate(
    identified_estimand, forestdml_estimate,
    method_name="placebo_treatment_refuter", placebo_type="permute",
    num_simulations=3
)
print(res_placebo)
```

    INFO:dowhy.causal_refuters.placebo_treatment_refuter:Refutation over 3 simulated datasets of permute treatment
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~placebo+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~placebo+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~placebo+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income
    WARNING:dowhy.causal_refuters.placebo_treatment_refuter:We assume a Normal Distribution as the sample has less than 100 examples.
                     Note: The underlying distribution may not be Normal. We assume that it approaches normal with the increase in sample size.

    Refute: Use a Placebo Treatment
    Estimated effect:-0.9574516356588789
    New effect:-0.008993954621420814
    p value:0.20846265515250384

### Remove a Random Subset of the Data

Do we recover similar estimates on subsets of the data? This speaks to the ability of our chosen estimator to generalize well. We use DoWhy to investigate this!

```python
res_subset = model.refute_estimate(
    identified_estimand, forestdml_estimate,
    method_name="data_subset_refuter", subset_fraction=0.8,
    num_simulations=3)
print(res_subset)
```

    INFO:dowhy.causal_refuters.data_subset_refuter:Refutation over 0.8 simulated datasets of size 8000.0 each
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income
    INFO:dowhy.causal_estimator:INFO: Using EconML Estimator
    INFO:dowhy.causal_estimator:b: log_demand~log_price+avg_hours+income+is_US+friends_count+age+has_membership+account_age+days_visited+songs_purchased | income
    WARNING:dowhy.causal_refuters.data_subset_refuter:We assume a Normal Distribution as the sample has less than 100 examples.
                     Note: The underlying distribution may not be Normal. We assume that it approaches normal with the increase in sample size.

    Refute: Use a subset of data
    Estimated effect:-0.9574516356588789
    New effect:-0.9594005906084321
    p value:0.39986509639504575

# Understand Treatment Effects with EconML
EconML includes interpretability tools to better understand treatment effects. Treatment effects can be complex, but oftentimes we are interested in simple rules that can differentiate between users who respond positively, users who remain neutral and users who respond negatively to the proposed changes.

The EconML `SingleTreeCateInterpreter` provides interperetability by training a single decision tree on the treatment effects outputted by the any of the EconML estimators. In the figure below we can see in dark red users respond strongly to the discount and the in white users respond lightly to the discount.

```python
intrp = SingleTreeCateInterpreter(include_model_uncertainty=True, max_depth=2, min_samples_leaf=10)
intrp.interpret(est, X_test)
plt.figure(figsize=(25, 5))
intrp.plot(feature_names=X.columns, fontsize=12)
```

# Make Policy Decision with EconML
We want to make policy decisions to maximum the **revenue** instead of the demand. In this scenario,

\begin{align}
Rev & = Y \cdot T \\
    & = \exp^{log(Y)} \cdot T\\
    & = \exp^{(\theta(X) \cdot log(T) + f(X,W) + \epsilon)} \cdot T \\
    & = \exp^{(f(X,W) + \epsilon)} \cdot T^{(\theta(X)+1)}
\end{align}

With the decrease of price, revenue will increase only if $\theta(X)+1<0$. Thus, we set `sample_treatment_cast=-1` here to learn **what kinds of customers we should give a small discount to maximum the revenue**.

The EconML library includes policy interpretability tools such as `SingleTreePolicyInterpreter` that take in a treatment cost and the treatment effects to learn simple rules about which customers to target profitably. In the figure below we can see the model recommends to give discount for people with income less than $0.985$ and give original price for the others.

```python
intrp = SingleTreePolicyInterpreter(risk_level=0.05, max_depth=2, min_samples_leaf=1, min_impurity_decrease=0.001)
intrp.interpret(est, X_test, sample_treatment_costs=-1, treatment_names=["Discount", "No-Discount"])
plt.figure(figsize=(25, 5))
intrp.plot(feature_names=X.columns, fontsize=12)
```

Now, let us compare our policy with other baseline policies! Our model says which customers to give a small discount to, and for this experiment, we will set a discount level of 10% for those users. Because the model is misspecified we would not expect good results with large discounts. Here, because we know the ground truth, we can evaluate the value of this policy.

```python
# define function to compute revenue
def revenue_fn(data, discount_level1, discount_level2, baseline_T, policy):
    policy_price = baseline_T * (1 - discount_level1) * policy + baseline_T * (1 - discount_level2) * (1 - policy)
    demand = demand_fn(data, policy_price)
    rev = demand * policy_price
    return rev
```

```python
policy_dic = {}
# our policy above
policy = intrp.treat(X)
policy_dic["Our Policy"] = np.mean(revenue_fn(train_data, 0, 0.1, 1, policy))

## previous strategy
policy_dic["Previous Strategy"] = np.mean(train_data["price"] * train_data["demand"])

## give everyone discount
policy_dic["Give Everyone Discount"] = np.mean(revenue_fn(train_data, 0.1, 0, 1, np.ones(len(X))))

## don't give discount
policy_dic["Give No One Discount"] = np.mean(revenue_fn(train_data, 0, 0.1, 1, np.ones(len(X))))

## follow our policy, but give -10% discount for the group doesn't recommend to give discount
policy_dic["Our Policy + Give Negative Discount for No-Discount Group"] = np.mean(revenue_fn(train_data, -0.1, 0.1, 1, policy))

## give everyone -10% discount
policy_dic["Give Everyone Negative Discount"] = np.mean(revenue_fn(train_data, -0.1, 0, 1, np.ones(len(X))))
```

```python
# get policy summary table
res = pd.DataFrame.from_dict(policy_dic, orient="index", columns=["Revenue"])
res["Rank"] = res["Revenue"].rank(ascending=False)
res
```

      Revenue
      Rank

      Our Policy
      14.686241
      2.0

      Previous Strategy
      14.349342
      4.0

      Give Everyone Discount
      13.774469
      6.0

      Give No One Discount
      14.294606
      5.0

      Our Policy + Give Negative Discount for No-Discount Group
      15.564411
      1.0

      Give Everyone Negative Discount
      14.612670
      3.0

**We beat the baseline policies!** Our policy gets the highest revenue except for the one raising the price for the No-Discount group. That means our currently baseline price is low, but the way we segment the user does help increase the revenue!

# Conclusions

In this notebook, we have demonstrated the power of using EconML and DoWhy to:

* Estimate the treatment effect correctly even the model is misspecified
* Test causal assumptions and investigate the robustness of the resulting estimates
* Interpret the resulting individual-level treatment effects
* Make the policy decision beats the previous and baseline policies

To learn more about what EconML can do for you, visit our [website](https://aka.ms/econml), our [GitHub page](https://github.com/microsoft/EconML) or our [docummentation](https://econml.azurewebsites.net/).

To learn more about what DoWhy can do for you, visit the [GitHub page](https://github.com/microsoft/dowhy) or [documentation](https://microsoft.github.io/dowhy/index.html).

notebooks/CustomerScenarios/Case Study - Customer Segmentation at An Online Media Company - EconML + DoWhy.ipynb


## Data Analytics

### Distribution Transformations in Python

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Distribution Transformations

Here's a demonstration of making and general use of distribution transformations in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics at the Cockrell School of Engineering at the University of Texas at Austin.

#### Distribution Transformations

Why do we do this?

* **Inference**: variable has expected shape

* **Data Preparation / Cleaning**: correcting for too few data and outliers

* **Theory**: a specific distribution assumption required for a method

How do we do it?

We apply this to all sample data, $x_{\alpha}$ $\forall$ $\alpha = 1,\ldots,n$.

\begin{equation}
y_{\alpha} = G^{-1}_Y\left(F_X(x_{\alpha})\right)
\end{equation}

were $X$ is the original feature with a $F_X$ original cumulative distribution function and $Y$ is transformed feature with a $G_Y$ transformed cumulative distribution function.

* Mapping from one distribution to another through percentiles

* This may be applied to any parametric or nonparametric distributions

* This is a rank preserving transform, e.g. P50 of 𝑋 is P50 of 𝑌

I have a lecture on distribution transformations available on [YouTube](https://www.youtube.com/watch?v=ZDIpE3OkAIU&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=14).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - [sample_data.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/sample_data.csv).

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # randon numbers
import seaborn as sns                     # matrix scatter plots
from scipy.stats import norm              # Gaussian parametric distribution
from sklearn import preprocessing
import geostatspy.GSLIB as GSLIB
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
```

#### Set the Random Number Seed

Set the random number seed so that we have a repeatable workflow

```python
seed = 73073
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
#os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.  For fun try misspelling the name. You will get an ugly, long error.

```python
#df = pd.read_csv('sample_data.csv')     # load our data table
df = pd.read_csv('https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/sample_data.csv') # load from Dr. Pyrcz's GitHub repository
```

It worked, we loaded our file into our DataFrame called 'df'. But how do you really know that it worked? Visualizing the DataFrame would be useful and we already leard about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n=6)                           # we could also use this command for a table preview
```

      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      100.0
      900.0
      1.0
      0.100187
      1.363890
      5110.699751

      1
      100.0
      800.0
      0.0
      0.107947
      12.576845
      4671.458560

      2
      100.0
      700.0
      0.0
      0.085357
      5.984520
      6127.548006

      3
      100.0
      600.0
      0.0
      0.108460
      2.446678
      5201.637996

      4
      100.0
      500.0
      0.0
      0.102468
      1.952264
      3835.270322

      5
      100.0
      400.0
      0.0
      0.110579
      3.691908
      5295.267191

#### Calculating and Plotting a CDF by Hand

Let's demonstrate the calculation and plotting of a non-parametric CDF by hand

1. make a copy of the feature as a 1D array (ndarray from NumPy)
2. sort the data in ascending order
3. assign cumulative probabilities based on the tail assumptions
4. plot cumuative probability vs. value

```python
por = df['Porosity'].copy(deep = True).values # make a deepcopy of the feature from the DataFrame
print('The ndarray has a shape of ' + str(por.shape) + '.')

por = np.sort(por)                           # sort the data in ascending order
n = por.shape[0]                             # get the number of data samples

cprob = np.zeros(n)
for i in range(0,n):
    index = i + 1
    cprob[i] = index / n                     # known upper tail
    # cprob[i] = (index - 1)/n               # known lower tail
    # cprob[i] = (index - 1)/(n - 1)         # known upper and lower tails
    # cprob[i] = index/(n+1)                 # unknown tails

plt.subplot(111)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 30, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.1, hspace=0.2)
plt.show()
```

#### Transformation to a Parametric Distribution

We can transform our data feature distribution to any parametric distribution with this workflow.

1. Calculate the cumulative probability value of each of our data values, $p_{\alpha} = F_x(x_\alpha)$, $\forall$ $\alpha = 1,\ldots, n$.

2. Apply the inverse of the target parametric cumulative distribution function (CDF) to calculate the transformed values. $y_{\alpha} = G_y^{-1}\left(F_x(x_\alpha)\right)$, $\forall$ $\alpha = 1,\ldots, n$.

```python
y = np.zeros(n)

for i in range(0,n):
    y[i] = norm.ppf(cprob[i],loc=0.0,scale=1.0)

plt.subplot(121)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 30, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplot(122)
plt.plot(y,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(y,cprob,s = 30, alpha = 1.0, c = 'blue', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([-3.0,3.0]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("After Distribution Transformation to Gaussian")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

```

Let's make an interactive version of this plot to visualize the transformation.

```python
# widgets and dashboard
l = widgets.Text(value='                                                  Data Analytics, Distribution Transformation, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

data_index = widgets.IntSlider(min=1, max = n-1, value=1.0, step = 1.0, description = 'Data Index, $\\alpha$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)

ui = widgets.VBox([l,data_index],)

def run_plot(data_index):                       # make data, fit models and plot
    plt.subplot(131)
    plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por,cprob,s = 20, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Original Feature, $x$"); plt.ylabel("Cumulative Probability"); plt.title("CDF Original Feature")
    plt.plot([por[data_index-1],por[data_index-1]],[0.0,cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.plot([por[data_index-1],3.0],[cprob[data_index-1],cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.scatter(por[data_index-1],0,marker='s',s = 100, c = 'red', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points

    plt.annotate('x = ' + str(round(por[data_index-1],2)), xy=(por[data_index-1]+0.01, 0.01))
    plt.annotate('p = ' + str(round(cprob[data_index-1],2)), xy=(0.225, cprob[data_index-1]+0.02))

    plt.subplot(132)
    plt.plot(y,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(y,cprob,s = 20, alpha = 1.0, c = 'blue', edgecolor = 'black') # plot the CDF points
    plt.grid(); plt.xlim([-3.0,3.0]); plt.ylim([0.0,1.0])
    plt.xlabel("Gaussian Transformed Feature, $y$"); plt.ylabel("Cumulative Probability"); plt.title("CDF After Distribution Transformation to Gaussian")
    plt.plot([-3.0,y[data_index-1]],[cprob[data_index-1],cprob[data_index-1]],color = 'blue',linestyle='dashed')
    plt.plot([y[data_index-1],y[data_index-1]],[0.0,cprob[data_index-1]],color = 'blue',linestyle='dashed')
    plt.scatter(y[data_index-1],0,marker='v',s = 100, c = 'blue', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points

    plt.annotate('p = ' + str(round(cprob[data_index-1],2)), xy=(-2.90, cprob[data_index-1]+0.02))
    plt.annotate('y = ' + str(round(y[data_index-1],2)), xy=(y[data_index-1]+0.3, 0.01))

    plt.subplot(133)
    plt.plot(por,y, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([-3.0,3.0])
    plt.xlabel("Original Porosity (fraction)"); plt.ylabel("Gaussian Transformed Porosity (N[fraction])"); plt.title("Q-Q Plot, Distribution Transformation")
    #plt.plot([0.05,0.25],[0.05,0.25],color = 'red',linestyle='dashed', alpha = 0.4)
    plt.plot([por[data_index-1],por[data_index-1]],[-3,y[data_index-1]],color = 'red',linestyle='dashed')
    plt.plot([0.05,por[data_index-1]],[y[data_index-1],y[data_index-1]],color = 'blue',linestyle='dashed')

    plt.scatter(por[data_index-1],y[data_index-1],marker='+',s = 700, c = 'purple', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por[data_index-1],y[data_index-1],marker='x',s = 500, c = 'purple', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por[data_index-1],y[data_index-1],s = 200, c = 'purple', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por,y,s = 20, c = 'purple', edgecolor = 'black', alpha = 0.7, zorder=100) # plot the CDF points
    plt.scatter(por[data_index-1],-3,marker='s',s = 100, c = 'red', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(0.05,y[data_index-1],marker='s',s = 100, c = 'blue', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.1, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(run_plot, {'data_index':data_index})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Data Analytics Distribution Transformation Demonstration

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

Select any data value and observe the distribution transform by mapping through cumulative probability.

### The Inputs

* **data_index** - the data index from 1 to n in the sorted ascending order

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                                                  Data Analytics, Distribution Tran…

    Output()

#### Distribution Transform to a Non-Parametric Distribution

We can apply the mapping through cumulative probabilities to transform from any distribution to any other distribution.

* let's make a new data set by randomly sampling from the previous one and adding error

Then we can demonstrate transforming this dataset to match the original distribution

* this is mimicking the situation where we transform a dataset to match the distribution of a better sampled analog distribution

```python
n_sample = 30
df_sample = df.sample(n_sample,random_state = seed)

df_sample = df_sample.copy(deep = True) # make a deepcopy of the feature from the DataFrame

df_sample['Porosity'] = df_sample['Porosity'].values + np.random.normal(loc = 0.0, scale = 0.01, size = n_sample)

df_sample = df_sample.sort_values(by = 'Porosity')                # sort the DataFrame
por_sample = df_sample['Porosity'].values
print('The sample ndarray has a shape of ' + str(por_sample.shape) + '.')

cprob_sample = np.zeros(n_sample)
for i in range(0,n_sample):
    index = i + 1
    cprob_sample[i] = index / n_sample       # known upper tail
    # cprob[i] = (index - 1)/n               # known lower tail
    # cprob[i] = (index - 1)/(n - 1)         # known upper and lower tails
    # cprob[i] = index/(n+1)                 # unknown tails

plt.subplot(121)
plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Sparse Sample with Noise Cumulative Distribution Function")

plt.subplot(122)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 30, alpha = 1.0, c = 'blue', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's transform the values and show them on the target distribution.

```python
y_sample = np.zeros(n_sample)

for i in range(0,n_sample):
    y_sample[i] = np.percentile(por,cprob_sample[i]*100, interpolation = 'linear') # piecewise linear interpolation of inverse of target CDF

plt.subplot(121)
plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'red', edgecolor = 'black', zorder = 100) # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Sparse Sample with Noise Cumulative Distribution Function")

plt.subplot(122)
plt.plot(por,cprob, alpha = 0.6,c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 20, c = 'red', edgecolor = 'black', alpha = 0.6) # plot the CDF points
plt.scatter(y_sample,cprob_sample,s = 60, c = 'white', alpha = .7, zorder = 99) # plot the CDF points
plt.scatter(y_sample,cprob_sample,s = 30, c = 'blue', edgecolor = 'black', alpha = 1.0, zorder = 100) # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's make an interactive version of this plot to visualize the transformation.

```python
# widgets and dashboard
l_sample = widgets.Text(value='                                                  Data Analytics, Distribution Transformation, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

data_index_sample = widgets.IntSlider(min=1, max = n_sample, value=1.0, step = 1.0, description = 'Data Sample Index, $\\beta$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)

ui_sample = widgets.VBox([l_sample,data_index_sample],)

def run_plot_sample(data_index_sample):                       # make data, fit models and plot
    plt.subplot(131)
    plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'red', edgecolor = 'black',zorder = 100) # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Original Sparse Sample with Noise, Cumulative Distribution Function")
    plt.plot([por_sample[data_index_sample-1],por_sample[data_index_sample-1]],[0.0,cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.plot([por_sample[data_index_sample-1],3.0],[cprob_sample[data_index_sample-1],cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.annotate('x = ' + str(round(por_sample[data_index_sample-1],2)), xy=(por_sample[data_index_sample-1]+0.003, 0.01))
    plt.annotate('p = ' + str(round(cprob_sample[data_index_sample-1],2)), xy=(0.225, cprob_sample[data_index_sample-1]+0.02))

    plt.subplot(132)
    plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por,cprob,s = 30, c = 'blue', edgecolor = 'black', alpha = 1.0) # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Target Porosity Cumulative Distribution Function")
    plt.plot([0.0,y_sample[data_index_sample-1]],[cprob_sample[data_index_sample-1],cprob_sample[data_index_sample-1]],color = 'blue',linestyle='dashed')
    plt.plot([y_sample[data_index_sample-1],y_sample[data_index_sample-1]],[0.0,cprob_sample[data_index_sample-1]],color = 'blue',linestyle='dashed')
    plt.annotate('p = ' + str(round(cprob_sample[data_index_sample-1],2)), xy=(0.053, cprob_sample[data_index_sample-1]+0.02))
    plt.annotate('y = ' + str(round(y_sample[data_index_sample-1],2)), xy=(y_sample[data_index_sample-1]+0.003, 0.01))
    plt.scatter(y_sample[data_index_sample-1],cprob_sample[data_index_sample-1],s = 200, c = 'white', alpha = 1.0, zorder=99) # plot the CDF points
    plt.scatter(y_sample[data_index_sample-1],cprob_sample[data_index_sample-1],s = 70, c = 'blue', edgecolor = 'black', alpha = 1.0, zorder=100) # plot the CDF points

    plt.subplot(133)
    plt.plot(por_sample,y_sample, alpha = 0.4, c = 'black') # plot piecewise linear interpolation
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.05,0.25])
    plt.xlabel("Original Porosity (fraction)"); plt.ylabel("Transformed Porosity (fraction)"); plt.title("Q-Q Plot for a Non-parametric Distribution Transformation")
    plt.plot([0.05,0.25],[0.05,0.25],color = 'black',linestyle='dashed', alpha = 1.0)
    plt.scatter(por_sample[data_index_sample-1],y_sample[data_index_sample-1],s = 120, c = 'white', alpha = 1.0, zorder=190) # plot the CDF points
    plt.scatter(por_sample[data_index_sample-1],y_sample[data_index_sample-1],s = 80, c = 'purple', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por_sample,y_sample,s = 30, c = 'purple', edgecolor = 'black', alpha = 0.8, zorder=100) # plot the CDF points
    plt.scatter(por_sample[data_index_sample-1],y_sample[data_index_sample-1],marker='+',s = 700, c = 'purple', edgecolor = 'black', alpha = 1.0, zorder=199) # plot the CDF points
    plt.scatter(por_sample[data_index_sample-1],y_sample[data_index_sample-1],marker='x',s = 500, c = 'purple', edgecolor = 'black', alpha = 1.0, zorder=199) # plot the CDF points

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot_s = widgets.interactive_output(run_plot_sample, {'data_index_sample':data_index_sample})
#interactive_plot_sample.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Data Analytics Distribution Transformation Demonstration

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

Select any data value and observe the distribution transform by mapping through cumulative probability.

#### The Inputs

* **data_index** - the data index from 1 to n in the sorted ascending order

```python
display(ui_sample, interactive_plot_s)                           # display the interactive plot
```

    VBox(children=(Text(value='                                                  Data Analytics, Distribution Tran…

    Output()

To summarize let's look at a DataFrame with the original noisey sample and the transformed to match the original distribution.

* we're making and showing a table of original values, $x_{\beta}$ $\forall$ $\beta = 1, \ldots, n_{sample}$, and the transformed values, $y_{\beta}$ $\forall$ $\beta = 1, \ldots, n_{sample}$.

```python
df_sample['Transformed_Por'] = y_sample
df_sample.head(n=n_sample)
```

      X
      Y
      Facies
      Porosity
      Perm
      AI
      Transformed_Por

      80
      900.0
      100.0
      0.0
      0.078139
      1.280257
      4573.656072
      0.081044

      207
      201.0
      426.0
      0.0
      0.099933
      0.400658
      5263.542112
      0.085867

      3
      100.0
      600.0
      0.0
      0.102014
      2.446678
      5201.637996
      0.091834

      41
      500.0
      400.0
      0.0
      0.104302
      6.312198
      5515.918646
      0.095628

      218
      251.0
      416.0
      0.0
      0.111108
      1.003374
      5822.467914
      0.099378

      226
      211.0
      396.0
      0.0
      0.112444
      6.368529
      5725.334803
      0.101987

      47
      600.0
      700.0
      0.0
      0.113304
      12.384496
      3595.586977
      0.103970

      210
      231.0
      426.0
      0.0
      0.114971
      5.584040
      4919.074871
      0.106704

      189
      201.0
      456.0
      0.0
      0.115299
      0.546396
      5018.355476
      0.108460

      5
      100.0
      400.0
      0.0
      0.120576
      3.691908
      5295.267191
      0.111491

      72
      900.0
      900.0
      0.0
      0.120786
      12.433996
      6242.704810
      0.113887

      71
      800.0
      100.0
      0.0
      0.139345
      7.739105
      5274.532660
      0.117984

      53
      600.0
      100.0
      1.0
      0.146009
      42.396044
      4204.150893
      0.121592

      165
      955.0
      469.0
      1.0
      0.175355
      26.197239
      2889.196647
      0.127131

      245
      690.0
      529.0
      1.0
      0.179410
      316.905689
      4271.013148
      0.137062

      151
      995.0
      489.0
      1.0
      0.181649
      460.494986
      2792.804322
      0.153759

      84
      955.0
      559.0
      1.0
      0.184025
      74.215058
      3386.182722
      0.176536

      93
      955.0
      549.0
      1.0
      0.189082
      374.298925
      3181.557281
      0.185270

      99
      925.0
      539.0
      1.0
      0.189614
      211.163296
      3442.885245
      0.188136

      136
      935.0
      499.0
      1.0
      0.190983
      523.287810
      2579.032897
      0.191655

      111
      955.0
      529.0
      1.0
      0.193601
      1113.971076
      3177.635737
      0.195998

      174
      955.0
      459.0
      1.0
      0.195106
      45.002088
      3394.563038
      0.198182

      158
      975.0
      479.0
      1.0
      0.197169
      69.336576
      2493.128177
      0.199315

      125
      1005.0
      519.0
      1.0
      0.198837
      54.667195
      2577.714678
      0.201943

      150
      985.0
      489.0
      1.0
      0.203183
      73.133040
      2672.294567
      0.206934

      181
      935.0
      449.0
      1.0
      0.203501
      368.507601
      4249.477923
      0.209051

      113
      975.0
      529.0
      1.0
      0.214610
      1548.094062
      3167.185377
      0.211638

      141
      985.0
      499.0
      1.0
      0.228755
      68.276148
      2547.526113
      0.215686

      149
      975.0
      489.0
      1.0
      0.236303
      21.109085
      2412.875330
      0.224214

      129
      955.0
      509.0
      1.0
      0.241784
      1525.247066
      2512.061434
      0.242298

It would be straitforward to modify the code above to perform distribution transformations:

* to a parametric distribution like Gaussian

* to a non-parametric distribution from actual data (build a CDF and interpolate between the data samples)

#### Comments

This was a basic demonstration of distribution transformations.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


```python

from six.moves import range
```

```python
%matplotlib inline
```

# The heat equation in 2D

$$
\renewcommand{\DdQq}[2]{{\mathrm D}_{#1}{\mathrm Q}_{#2}}
\renewcommand{\drondt}{\partial_t}
\renewcommand{\drondx}{\partial_x}
\renewcommand{\drondtt}{\partial_{tt}}
\renewcommand{\drondxx}{\partial_{xx}}
\renewcommand{\drondyy}{\partial_{yy}}
\renewcommand{\dx}{\Delta x}
\renewcommand{\dt}{\Delta t}
\renewcommand{\grandO}{{\mathcal O}}
\renewcommand{\density}[2]{\,f_{#1}^{#2}}
\renewcommand{\fk}[1]{\density{#1}{\vphantom{\star}}}
\renewcommand{\fks}[1]{\density{#1}{\star}}
\renewcommand{\moment}[2]{\,m_{#1}^{#2}}
\renewcommand{\mk}[1]{\moment{#1}{\vphantom{\star}}}
\renewcommand{\mke}[1]{\moment{#1}{e}}
\renewcommand{\mks}[1]{\moment{#1}{\star}}
$$

In this tutorial, we test a very classical lattice Boltzmann scheme $\DdQq{2}{5}$ on the heat equation.

The problem reads
$$
\begin{gathered} \drondt u = \mu (\drondxx+\drondyy) u, \quad t>0, \quad (x, y)\in(0,1)^2,\\ u(0) = u(1) = 0, \end{gathered}
$$

where $\mu$ is a constant scalar.

## The scheme $\DdQq{2}{5}$

The numerical simulation of this equation by a lattice Boltzmann scheme consists in the approximatation of the solution on discret points of $(0,1)^2$ at discret instants.

To simulate this system of equations, we use the $\DdQq{2}{5}$ scheme given by

* five velocities $v_0=(0,0)$, $v_1=(1,0)$, $v_2=(0,1)$, $v_3=(-1,0)$, and $v_4=(0,-1)$ with associated distribution functions $\fk{i}$, $0\leq i\leq 4$,
* a space step $\dx$ and a time step $\dt$, the ration $\lambda=\dx/\dt$ is called the scheme velocity,
* five moments
  $$ \mk{0}=\sum_{i=0}^{4} \fk{i}, \quad \mk{1}= \sum_{i=0}^{4} v_{ix} \fk{i}, \quad \mk{2}= \sum_{i=0}^{4} v_{iy} \fk{i}, \quad \mk{3}= \frac{1}{2} \sum_{i=0}^{5} (v_{ix}^2+v_{iy}^2) \fk{i}, \quad \mk{4}= \frac{1}{2} \sum_{i=0}^{5} (v_{ix}^2-v_{iy}^2) \fk{i},$$

  and their equilibrium values $\mke{k}$, $0\leq k\leq 4$.
* two relaxation parameters $s_1$ and $s_2$ lying in $[0,2]$ ($s_1$ for the odd moments and $s_2$ for the odd ones).

In order to use the formalism of the package pylbm, we introduce the five polynomials that define the moments: $P_0 = 1$, $P_1=X$, $P_2=Y$, $P_3=(X^2+Y^2)/2$, and $P_4=(X^2-Y^2)/2$, such that
$$
\mk{k} = \sum_{i=0}^4 P_k(v_{ix}, v_{iy}) \fk{i}.
$$

The transformation $(\fk{0}, \fk{1}, \fk{2}, \fk{3}, \fk{4})\mapsto(\mk{0},\mk{1}, \mk{2}, \mk{3}, \mk{4})$ is invertible if, and only if, the polynomials $(P_0,P_1,P_2,P_3,P_4)$ is a free set over the stencil of velocities.

The lattice Boltzmann method consists to compute the distribution functions $\fk{i}$, $0\leq i\leq 4$ in each point of the lattice $x$ and at each time $t^n=n\dt$.
A step of the scheme can be read as a splitting between the relaxation phase and the transport phase:

* relaxation:
$$
    \begin{aligned}\mks{1}(t,x,y)&=(1-s_1)\mk{1}(t,x,y)+s_1\mke{1}(t,x,y),\\ \mks{2}(t,x,y)&=(1-s_1)\mk{2}(t,x,y)+s_1\mke{2}(t,x,y),\\ \mks{3}(t,x,y)&=(1-s_2)\mk{3}(t,x,y)+s_2\mke{3}(t,x,y),\\ \mks{4}(t,x,y)&=(1-s_2)\mk{4}(t,x,y)+s_2\mke{4}(t,x,y).\end{aligned}
$$

* m2f:
$$
    \begin{aligned}\fks{0}(t,x,y)&\;=\mk{0}(t,x,y)-2\mks{3}(t,x,y), \\ \fks{1}(t,x,y)&\;=\tfrac{1}{2}(\phantom{-}\mks{1}(t,x,y)+\mks{3}(t,x,y)+\mks{4}(t,x,y)), \\ \fks{2}(t,x,y)&\;=\tfrac{1}{2}(\phantom{-}\mks{2}(t,x,y)+\mks{3}(t,x,y)-\mks{4}(t,x,y)), \\ \fks{3}(t,x,y)&\;=\tfrac{1}{2}(-\mks{1}(t,x,y)+\mks{3}(t,x,y)+\mks{4}(t,x,y)), \\ \fks{4}(t,x,y)&\;=\tfrac{1}{2}(-\mks{2}(t,x,y)+\mks{3}(t,x,y)-\mks{4}(t,x,y)).\end{aligned}
$$

* transport:
$$
    \begin{aligned} \fk{0}(t+\dt, x,y)&\;=\fks{0}(t,x,y), \\ \fk{1}(t+\dt, x,y)&\;=\fks{1}(t,x-\dx,y), \\ \fk{2}(t+\dt, x,y)&\;=\fks{2}(t,x,y-\dx), \\ \fk{3}(t+\dt, x,y)&\;=\fks{3}(t,x+\dx,y), \\ \fk{4}(t+\dt, x,y)&\;=\fks{4}(t,x,y+\dx). \end{aligned}
$$

* f2m:
$$
    \begin{aligned}\mk{0}(t+\dt,x,y)&\;=\fk{0}(t+\dt,x,y)+\fk{1}(t+\dt,x,y)+\fk{2}(t+\dt,x,y)\\&\;\phantom{=}+\fk{3}(t+\dt,x,y)+\fk{4}(t+\dt,x,y), \\ \mk{1}(t+\dt,x,y)&\;=\fk{1}(t+\dt,x,y)-\fk{3}(t+\dt,x,y), \\ \mk{2}(t+\dt,x,y)&\;=\fk{2}(t+\dt,x,y)-\fk{4}(t+\dt,x,y), \\ \mk{3}(t+\dt,x,y)&\;=\tfrac{1}{2}(\fk{1}(t+\dt,x,y)+\fk{2}(t+\dt,x,y)+\fk{3}(t+\dt,x,y)+\fk{4}(t+\dt,x,y)), \\ \mk{4}(t+\dt,x,y)&\;=\tfrac{1}{2}(\fk{1}(t+\dt,x,y)-\fk{2}(t+\dt,x,y)+\fk{3}(t+\dt,x,y)-\fk{4}(t+\dt,x,y)).\end{aligned}
$$

The moment of order $0$, $\mk{0}$, being conserved during the relaxation phase,
a diffusive scaling $\dt=\dx^2$, yields to the following equivalent equation
$$
\drondt\mk{0} = \bigl(\tfrac{1}{s_1}-\tfrac{1}{2}\bigr) \bigl(\drondxx(\mke{3}+\mke{4})+\drondyy(\mke{3}-\mke{4})\bigr) + \grandO(\dx^2),
$$

if $\mke{1}=0$.
In order to be consistent with the heat equation, the following choice is done:
$$
\mke{3}=\tfrac{1}{2}u, \qquad \mke{4}=0, \qquad s_1 = \frac{2}{1+4\mu}, \qquad s_2=1.
$$

## Using pylbm

pylbm uses Python dictionary to describe the simulation. In the following, we will build this dictionary step by step.

### The geometry

In pylbm, the geometry is defined by a box and a label for the boundaries. We define here a square $(0, 1)^2$.

```python
import pylbm
import numpy as np
import pylab as plt
xmin, xmax, ymin, ymax = 0., 1., 0., 1.
dico_geom = {
    'box': {'x': [xmin, xmax], 'y':[ymin, ymax], 'label':0},
}
geom = pylbm.Geometry(dico_geom)
print(geom)
geom.visualize(viewlabel=True)
```

### The stencil

pylbm provides a class stencil that is used to define the discret velocities of the scheme. In this example, the stencil is composed by the velocities $v_0=(0,0)$, $v_1=(1,0)$, $v_2=(-1,0)$, $v_3=(0,1)$, and $v_4=(0,-1)$ numbered by $[0,1,2,3,4]$.

```python
dico_sten = {
    'dim':2,
    'schemes':[{'velocities':list(range(5))}],
}
sten = pylbm.Stencil(dico_sten)
print(sten)
sten.visualize()
```

### The domain

In order to build the domain of the simulation, the dictionary should contain the space step $\dx$ and the stencils of the velocities (one for each scheme).

We construct a domain with $N=10$ points in space.

```python
N = 10
dx = (xmax-xmin)/N
dico_dom = {
    'box': {'x': [xmin, xmax], 'y':[ymin, ymax], 'label':0},
    'space_step':dx,
    'schemes':[
        {
            'velocities':list(range(5)),
        }
    ],
}
dom = pylbm.Domain(dico_dom)
print(dom)
dom.visualize(view_distance=True)
```

### The scheme

In pylbm, a simulation can be performed by using several coupled schemes. In this example, a single scheme is used and defined through a list of one single dictionary. This dictionary should contain:

* 'velocities': a list of the velocities
* 'conserved_moments': a list of the conserved moments as sympy variables
* 'polynomials': a list of the polynomials that define the moments
* 'equilibrium': a list of the equilibrium value of all the moments
* 'relaxation_parameters': a list of the relaxation parameters ($0$ for the conserved moments)
* 'init': a dictionary to initialize the conserved moments

(see the documentation for more details)

The scheme velocity could be taken to $1/\dx$ and the inital value of $u$ to

$$ u(t=0,x) = \sin(\pi x)\sin(\pi y).$$

```python
import sympy as sp

def solution(x, y, t):
    return np.sin(np.pi*x)*np.sin(np.pi*y)*np.exp(-2*np.pi**2*mu*t)

# parameters
mu = 1.
la = 1./dx
s1 = 2./(1+4*mu)
s2 = 1.
u, X, Y, LA = sp.symbols('u, X, Y, LA')

dico_sch = {
    'dim':2,
    'scheme_velocity':la,
    'schemes':[
        {
            'velocities':list(range(5)),
            'conserved_moments':u,
            'polynomials':[1, X/LA, Y/LA, (X**2+Y**2)/(2*LA**2), (X**2-Y**2)/(2*LA**2)],
            'equilibrium':[u, 0., 0., .5*u, 0.],
            'relaxation_parameters':[0., s1, s1, s2, s2],
            'init':{u:(solution, (0.,))},
        }
    ],
    'parameters':{LA: la},
}

sch = pylbm.Scheme(dico_sch)
print(sch)
```

    [0] WARNING  pylbm.scheme in function __init__ line 194
    The value 'space_step' is not given or wrong.
    The scheme takes default value: dx = 1.

    Scheme informations
    	 spatial dimension: dim=2
    	 number of schemes: nscheme=1
    	 number of velocities:
        Stencil.nv[0]=5
    	 velocities value:
        v[0] = (0: 0, 0), (1: 1, 0), (2: 0, 1), (3: -1, 0), (4: 0, -1),
    	 polynomials:
        P[0] = 1, X/LA, Y/LA, (X**2 + Y**2)/(2*LA**2), (X**2 - Y**2)/(2*LA**2),
    	 equilibria:
        EQ[0] = u, 0.0, 0.0, 0.5*u, 0.0,
    	 relaxation parameters:
        s[0] = 0.0, 0.400000000000000, 0.400000000000000, 1.00000000000000, 1.00000000000000,
    	 moments matrices
    M      = Matrix([[1, 1, 1, 1, 1], [0, 10.0/LA, 0, -10.0/LA, 0], [0, 0, 10.0/LA, 0, -10.0/LA], [0, 50.0/LA**2, 50.0/LA**2, 50.0/LA**2, 50.0/LA**2], [0, 50.0/LA**2, -50.0/LA**2, 50.0/LA**2, -50.0/LA**2]])
    M^(-1) = Matrix([[1.00000000000000, 0, 0, -0.02*LA**2, 0], [0, 0.05*LA, 0, 0.005*LA**2, 0.005*LA**2], [0, 0, 0.05*LA, 0.005*LA**2, -0.005*LA**2], [0, -0.05*LA, 0, 0.005*LA**2, 0.005*LA**2], [0, 0, -0.05*LA, 0.005*LA**2, -0.005*LA**2]])

### The simulation

A simulation is built by defining a correct dictionary.

We combine the previous dictionaries to build a simulation. In order to impose the homogeneous Dirichlet conditions in $x=0$, $x=1$, $y=0$, and $y=1$, the dictionary should contain the key 'boundary_conditions' (we use pylbm.bc.Anti_bounce_back function).

```python
dico = {
    'box':{'x':[xmin, xmax], 'y':[ymin, ymax], 'label':0},
    'space_step':dx,
    'scheme_velocity':la,
    'schemes':[
        {
            'velocities':list(range(5)),
            'conserved_moments':u,
            'polynomials':[1, X/LA, Y/LA, (X**2+Y**2)/(2*LA**2), (X**2-Y**2)/(2*LA**2)],
            'equilibrium':[u, 0., 0., .5*u, 0.],
            'relaxation_parameters':[0., s1, s1, s2, s2],
            'init':{u:(solution,(0.,))},
        }
    ],
    'boundary_conditions':{
        0:{'method':{0:pylbm.bc.AntiBounceBack,}, 'value':None},
    },
    'parameters':{LA: la},
}

sol = pylbm.Simulation(dico)
print(sol)
```

    Simulation informations:
    Domain informations
    	 spatial dimension: 2
    	 space step: dx= 1.000e-01
    Scheme informations
    	 spatial dimension: dim=2
    	 number of schemes: nscheme=1
    	 number of velocities:
        Stencil.nv[0]=5
    	 velocities value:
        v[0] = (0: 0, 0), (1: 1, 0), (2: 0, 1), (3: -1, 0), (4: 0, -1),
    	 polynomials:
        P[0] = 1, X/LA, Y/LA, (X**2 + Y**2)/(2*LA**2), (X**2 - Y**2)/(2*LA**2),
    	 equilibria:
        EQ[0] = u, 0.0, 0.0, 0.5*u, 0.0,
    	 relaxation parameters:
        s[0] = 0.0, 0.400000000000000, 0.400000000000000, 1.00000000000000, 1.00000000000000,
    	 moments matrices
    M      = Matrix([[1, 1, 1, 1, 1], [0, 10.0/LA, 0, -10.0/LA, 0], [0, 0, 10.0/LA, 0, -10.0/LA], [0, 50.0/LA**2, 50.0/LA**2, 50.0/LA**2, 50.0/LA**2], [0, 50.0/LA**2, -50.0/LA**2, 50.0/LA**2, -50.0/LA**2]])
    M^(-1) = Matrix([[1.00000000000000, 0, 0, -0.02*LA**2, 0], [0, 0.05*LA, 0, 0.005*LA**2, 0.005*LA**2], [0, 0, 0.05*LA, 0.005*LA**2, -0.005*LA**2], [0, -0.05*LA, 0, 0.005*LA**2, 0.005*LA**2], [0, 0, -0.05*LA, 0.005*LA**2, -0.005*LA**2]])

### Run a simulation

Once the simulation is initialized, one time step can be performed by using the function one_time_step.

We compute the solution of the heat equation at $t=0.1$. On the same graphic, we plot the initial condition, the exact solution and the numerical solution.

```python
import numpy as np
import sympy as sp
import pylab as plt
%matplotlib inline
from mpl_toolkits.axes_grid1 import make_axes_locatable
import pylbm

u, X, Y = sp.symbols('u, X, Y')

def solution(x, y, t, k, l):
    return np.sin(k*np.pi*x)*np.sin(l*np.pi*y)*np.exp(-(k**2+l**2)*np.pi**2*mu*t)

def plot(i, j, z, title):
    im = axarr[i,j].imshow(z)
    divider = make_axes_locatable(axarr[i, j])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar = plt.colorbar(im, cax=cax, format='%6.0e')
    axarr[i, j].xaxis.set_visible(False)
    axarr[i, j].yaxis.set_visible(False)
    axarr[i, j].set_title(title)

# parameters
xmin, xmax, ymin, ymax = 0., 1., 0., 1.
N = 128
mu = 1.
Tf = .1
dx = (xmax-xmin)/N # spatial step
la = 1./dx
s1 = 2./(1+4*mu)
s2 = 1.
k, l = 1, 1 # number of the wave

dico = {
    'box':{'x':[xmin, xmax], 'y':[ymin, ymax], 'label':0},
    'space_step':dx,
    'scheme_velocity':la,
    'schemes':[
        {
            'velocities':list(range(5)),
            'conserved_moments':u,
            'polynomials':[1, X/LA, Y/LA, (X**2+Y**2)/(2*LA**2), (X**2-Y**2)/(2*LA**2)],
            'equilibrium':[u, 0., 0., .5*u, 0.],
            'relaxation_parameters':[0., s1, s1, s2, s2],
            'init':{u:(solution,(0.,k,l))},
        }
    ],
    'boundary_conditions':{
        0:{'method':{0:pylbm.bc.AntiBounceBack,}, 'value':None},
    },
    'generator': 'cython',
    'parameters':{LA: la},
}

sol = pylbm.Simulation(dico)
x = sol.domain.x
y = sol.domain.y

f, axarr = plt.subplots(2, 2)
f.suptitle('Heat equation', fontsize=20)

plot(0, 0, sol.m[u].copy(), 'initial')

while sol.t < Tf:
    sol.one_time_step()

sol.f2m()
z = sol.m[u]
ze = solution(x[:,np.newaxis], y[np.newaxis,:], sol.t, k, l)
plot(1, 0, z, 'final')
plot(0, 1, ze, 'exact')
plot(1, 1, z-ze, 'error')

plt.show()
```

```python

```


# Using the Lorentz Reciprocity Theorem with TMM

Ekaterina Zossimova | June 2021

### Set up

These scripts have been tested in the following environment:
* Python 3.7.9
* tmm 0.1.8
* numpy 1.19.2
* scipy 1.5.2
* matplotlib 3.3.4

### Import

```python
from __future__ import division, print_function, absolute_import

from tmm import (coh_tmm, position_resolved)

from reciprocity import *

import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import interp1d
from scipy.signal import find_peaks
from scipy.integrate import quad

%matplotlib inline

# "5 * degree" is 5 degrees expressed in radians
# "1.2 / degree" is 1.2 radians expressed in degrees
degree = np.pi/180
```

## Kretschmann configuration

This multilayer structure is known to have enhanced fields at the top interface when the incident light is in resonance with eigenmodes of the multilayer. The enhanced fields appear in the Kretschmann configuration under total internal reflection, enabling efficient dark-field spectroscopy of single particles/molecules.

| Material | Refractive index @ 780nm | References |
| --- | --- |  --- |
| Water | 1.33 |  --- |
| SiO2 | 1.52 |  --- |
| Au	| 0.1194+4.7347j | Au data |
| Cr | 3.1243+3.4422j | Cr data |
| Substrate (LaSFN9) | 1.85 | --- |

Further reading:

* Yu. Akimov and S. Sun, Phys. Chem. Chem. Phys., vol. 19, no. 13, pp. 8706–8714, 2017, doi: 10.1039/c7cp00151g.

* Y. Akimov, M. E. Pam, and S. Sun, Phys. Rev. B, vol. 96, no. 15, Oct. 2017, doi: 10.1103/physrevb.96.155433.

* Y. Akimov, Opt. Lett., vol. 43, no. 6, p. 1195, Mar. 2018, doi: 10.1364/ol.43.001195.

Can visit RefractiveIndex.INFO for more refractive index data.

### Field enhancement

Calculate the angle of incidence that gives the maximum field enhancement at the position of the dipole using tmm code.

The electric field at any given point is a superposition of the forward-moving and
backwards-moving electromagnetic waves:

\begin{equation}
\textbf{E}(\textbf{r}) = \textbf{E}^0_f e^{i\textbf{k}_f \cdot \textbf{r}} + \textbf{E}^0_b e^{i\textbf{k}_b \cdot \textbf{r}}
\end{equation}

Here, $\textbf{k}_f$ and $\textbf{k}_b$ are the angular wavevectors for forward- and backwards-moving
waves; $\textbf{E}^0_f$ and $\textbf{E}^0_b$ are some constant vectors; and $\textbf{E}(\textbf{r})$ is the complex electric field at any given point $\textbf{r}$ within a certain layer. $E_x$ and $E_y$ and $E_z$ are the electric field amplitudes, where z is normal to the interfaces and the light rays are in the x, z plane.

The field enhancement is defined as $|\textbf{E}|/|\textbf{E}_0|$ where $\textbf{E}_0$ is the incident field ($|\textbf{E}_0|$ = 1).

```python
# List of refractive indices
n_list = [1.85, 3.1243+3.4422j, 0.1194+4.7347j, 1.52, 1.33]
# List of layer thicknesses in nm
d_list = [np.inf, 2, 48, 22, np.inf]
# Wavelength of incident beam
lam_vac = 780

# Calculate E-field at the position of the dipole
layer = len(d_list)-1
d_in_layer = 0

theta_list = np.linspace(0*degree, 90*degree, num=900)
field_enhancement=[]
for theta in theta_list:
    coh_tmm_data = coh_tmm('p', n_list, d_list, theta, lam_vac)
    data = position_resolved(layer, d_in_layer, coh_tmm_data)
    Ep = np.array([data['Ex'], data['Ey'], data['Ez']])
    magnitude = np.linalg.norm(Ep)

    field_enhancement.append(magnitude)

print(f'Optimized angle of incidence @ {lam_vac} nm', theta_list[find_peaks(field_enhancement)[0]]/degree)
print(f'Electric field at the position of the dipole {Ep}')
plt.figure()
plt.plot(theta_list/degree, field_enhancement)
plt.xlabel('Angle of incidence (degree)')
plt.ylabel('Field enhancement')
plt.title('Optimization of the field enhancement at the position of the dipole in Kretschmann configuration');
```

### Reciprocity theorem

#### Task 1

Use the Lorentz reciprocity theorem to calculate the far-field radiation pattern due to a particle with polarizability $\alpha = 1$ at the spacer/ water interface.

The relationship between near-fields for s- and p- polarized light at the position of the dipole, $\textbf{r}_1$, and the radiated far-field, $\textbf{E}_{rad}$, is given by,

\begin{equation}
|\textbf{E}_{rad}(\textbf{r}_2)|^2 = |\textbf{E}_{rad}^S(\textbf{r}_2)|^2 + |\textbf{E}_{rad}^P(\textbf{r}_2)|^2 = |\textbf{E}_{near}^S(\textbf{r}_1) \cdot \textbf{p}|^2 + |\textbf{E}_{near}^P(\textbf{r}_1) \cdot \textbf{p}|^2
\end{equation}

The dipole is defined as $\textbf{p} = \alpha \textbf{E}_{max}$, where $\textbf{E}_{max}$ is the electric field at the position of the dipole (calculated for the optimized angle of incidence).

```python
alpha = 1.
n_list = [1.85, 3.1243+3.4422j, 0.1194+4.7347j, 1.52, 1.33]
d_list = [np.inf, 2, 48, 22, np.inf]
theta_opt = 49.8 * degree
lam_vac = 780

ax = plt.subplot(111, projection='polar')

dipole = dipole_moment(alpha, n_list, d_list, theta_opt, lam_vac)
theta_list, poynting = radiation_pattern(dipole, n_list, d_list, lam_vac)

plotted_line = ax.plot(theta_list, poynting)
# By symmetry for negative theta
ax.plot(-theta_list, poynting, color=plotted_line[0].get_color(), label=f"alpha = {alpha}")

#ax.set_rmax(1.5)
#ax.set_rticks([])
ax.grid(True)
ax.set_rlabel_position(22)
ax.set_theta_zero_location("N")  # theta=0 at the top
ax.set_theta_direction(-1)  # theta increasing clockwise

plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.show()
```

#### Task 2

Show that for real, scalar values of $\alpha$, that the far-field radiation is proportional to $\alpha^2$.

```python
alpha_list = np.linspace(1., 10., 10)
n_list = [1.85, 3.1243+3.4422j, 0.1194+4.7347j, 1.52, 1.33]
d_list = [np.inf, 2, 48, 22, np.inf]
theta_opt = 49.8 * degree
lam_vac = 780

plt.figure()

intensity = []
for alpha in alpha_list:

    dipole = dipole_moment(alpha, n_list, d_list, theta_opt, lam_vac)
    theta_list, poynting = radiation_pattern(dipole, n_list, d_list, lam_vac)
    intensity.append(max(poynting))

plt.plot(alpha_list ** 2, intensity)
plt.xlabel('$α^2$')
plt.ylabel('Intensity')

plt.show()
```

#### Task 3

Integrate the radiation pattern to compare the results across different multilayer stacks and particles.

```python
alpha = 1.
n_list = [1.85, 3.1243+3.4422j, 0.1194+4.7347j, 1.52, 1.33]
d_list = [np.inf, 2, 48, 22, np.inf]
theta_opt = 49.8 * degree
lam_vac = 780

dipole = dipole_moment(alpha, n_list, d_list, theta_opt, lam_vac)

res, err = quad(theta_integral, 0, np.pi/2, args=(dipole, n_list, d_list, lam_vac))

print("The numerical result is {:f} (+-{:g})"
        .format(res, err))
```

    The numerical result is 1.146842 (+-1.04417e-10)

```python

```


## Introduction

This tutorial illustrates the spectra computation for standard and pure B modes. We will only use
the `HEALPIX` pixellisation to pass through the different steps of generation.

The `HEALPIX` survey mask is a disk centered on longitude 30° and latitude 50° with a radius of 25
radians. The `nside` value is set to 512 for this tutorial to reduce computation time.

## Preamble

`matplotlib` magic

```python
%matplotlib inline
```

Versions used for this tutorial

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import healpy as hp
import pspy
print("     Numpy :", np.__version__)
print("Matplotlib :", mpl.__version__)
print("    healpy :", hp.__version__)
print("      pspy :", pspy.__version__)
```

         Numpy : 1.18.0
    Matplotlib : 3.1.2
        healpy : 1.13.0
          pspy : 1.2.0+4.gcb26dc1

Get default data dir from `pspy` and set Planck colormap as default

```python
from pixell import colorize
colorize.mpl_setdefault("planck")
```

## Generation of the templates, mask and apodisation type

We start by specifying the `HEALPIX` survey parameters namely longitude, latitude and patch size. The
`nside` value is set to 512.

```python
lon, lat = 30, 50
radius = 25
nside = 512
```

Given the `nside` value, we can set the $\ell$max value

```python
lmax = 3 * nside - 1
```

For this example, we will make use of 3 components : Temperature (spin 0) and polarisation Q and U
(spin 2)

```python
ncomp = 3
```

Given the parameters, we can generate the `HEALPIX` template as follow

```python
from pspy import so_map
template = so_map.healpix_template(ncomp, nside)
```

We also define the binary template for the window function pixels

```python
binary = so_map.healpix_template(ncomp=1, nside=nside)
vec = hp.pixelfunc.ang2vec(lon, lat, lonlat=True)
disc = hp.query_disc(nside, vec, radius=radius*np.pi/180)
binary.data[disc] = 1
```

## Generation of spectra

### Generate window

We then create an apodisation for the survey mask. We use a C1 apodisation with an apodisation size
of 5 degrees

```python
from pspy import so_window
window = so_window.create_apodization(binary, apo_type="C1", apo_radius_degree=5)
hp.mollview(window.data, title=None)
```

We can also have a look to the corresponding spin 1 and spin 2 window functions

```python
niter = 3
w1_plus, w1_minus, w2_plus, w2_minus = so_window.get_spinned_windows(window, lmax=lmax, niter=niter)
plt.figure(figsize=(8, 8))
kwargs = {"rot": (lon, lat, 0), "xsize": 3500, "reso": 1, "title": None}
hp.gnomview(w1_plus.data, **kwargs, sub=(2, 2, 1))
hp.gnomview(w1_minus.data, **kwargs, sub=(2, 2, 2))
hp.gnomview(w2_plus.data, **kwargs, sub=(2, 2, 3))
hp.gnomview(w2_minus.data, **kwargs, sub=(2, 2, 4))
```

### Binning file

We create a binning file with the following format : lmin, lmax, lmean

```python
import os
output_dir = "/tmp/tutorial_purebb"
os.makedirs(output_dir, exist_ok=True)
binning_file = os.path.join(output_dir, "binning.dat")
from pspy import pspy_utils
pspy_utils.create_binning_file(bin_size=50, n_bins=300, file_name=binning_file)
```

### Compute mode coupling matrix

For spin 0 and 2 the window need to be a tuple made of two objects: the window used for spin 0 and the
one used for spin 2

```python
window_tuple = (window, window)
```

The windows (for `spin0` and `spin2`) are going to couple mode together, we compute a mode coupling
matrix in order to undo this effect given the binning file. We do it for both calculations *i.e.*
standard and pure B mode

```python
from pspy import so_mcm
print("computing standard mode coupling matrix")
mbb_inv, Bbl = so_mcm.mcm_and_bbl_spin0and2(window_tuple,
                                            binning_file,
                                            lmax=lmax,
                                            niter=niter,
                                            type="Cl")

print("computing pure mode coupling matrix")
mbb_inv_pure, Bbl_pure = so_mcm.mcm_and_bbl_spin0and2(window_tuple,
                                                      binning_file,
                                                      lmax=lmax,
                                                      niter=niter,
                                                      type="Cl",
                                                      pure=True)
```

    computing standard mode coupling matrix
    computing pure mode coupling matrix

### Generation of ΛCDM power spectra

We first have to compute $C_\ell$ data using a cosmology code such as [CAMB](https://camb.readthedocs.io/en/latest/) and we need to install it
since this is not a prerequisite of `pspy`. We can do it within this notebook by executing the
following command

```python
%pip install -U camb
```

    Requirement already up-to-date: camb in /home/garrido/Workdir/CMB/development/pspy/pyenv/lib/python3.8/site-packages (1.1.0)
    Requirement already satisfied, skipping upgrade: scipy>=1.0 in /home/garrido/Workdir/CMB/development/pspy/pyenv/lib/python3.8/site-packages (from camb) (1.4.1)
    Requirement already satisfied, skipping upgrade: six in /home/garrido/Workdir/CMB/development/pspy/pyenv/lib/python3.8/site-packages (from camb) (1.13.0)
    Requirement already satisfied, skipping upgrade: sympy>=1.0 in /home/garrido/Workdir/CMB/development/pspy/pyenv/lib/python3.8/site-packages (from camb) (1.5)
    Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /home/garrido/Workdir/CMB/development/pspy/pyenv/lib/python3.8/site-packages (from scipy>=1.0->camb) (1.18.0)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in /home/garrido/Workdir/CMB/development/pspy/pyenv/lib/python3.8/site-packages (from sympy>=1.0->camb) (1.1.0)
    Note: you may need to restart the kernel to use updated packages.

To make sure everything goes well, we can import `CAMB` and check its version

```python
import camb
print("CAMB version:", camb.__version__)
```

    CAMB version: 1.1.0

Now that `CAMB` is properly installed, we will produce $C_\ell$ data from $\ell$min=2 to
$\ell$max=104 for the following set of $\Lambda$CDM parameters

```python
ellmin, ellmax = 2, 10**4
ell = np.arange(ellmin, ellmax)
cosmo_params = {
    "H0": 67.5,
    "As": 1e-10*np.exp(3.044),
    "ombh2": 0.02237,
    "omch2": 0.1200,
    "ns": 0.9649,
    "Alens": 1.0,
    "tau": 0.0544
}
pars = camb.set_params(**cosmo_params)
pars.set_for_lmax(ellmax, lens_potential_accuracy=1)
results = camb.get_results(pars)
powers = results.get_cmb_power_spectra(pars, CMB_unit="muK")
```

We finally have to write $C_\ell$ into a file to feed the `so_map.synfast` function for both
pixellisation templates

```python
cl_file = os.path.join(output_dir, "cl_camb.dat")
np.savetxt(cl_file,
           np.hstack([ell[:, np.newaxis], powers["total"][ellmin:ellmax]]))
```

## Running simulations

Given the parameters and data above, we will now simulate `n_sims` simulations to check for mean and
variance of BB spectrum. For illustrative purpose, we will only run 10 simulations (~ few minutes)
but for reasonable comparisons, you should increase this number to few tens of simulations.

We will do it for both calculations (standard and pure) and finally we will graphically compare
results

We first need to specify the order of the spectra to be used by `pspy` although only BB spectrum will
be used

```python
spectra = ["TT", "TE", "TB", "ET", "BT", "EE", "EB", "BE", "BB"]
```

and we define a dictionnary of methods regarding the calculation type for B mode spectrum

```python
from pspy import sph_tools
methods = {
    "standard": {"alm" : sph_tools.get_alms, "mbb": mbb_inv, "bb": []},
    "pure": {"alm": sph_tools.get_pure_alms, "mbb": mbb_inv_pure, "bb": []}
}
```

```python
from pspy import so_spectra
n_sims = 10
for i in range(n_sims):
  cmb = template.synfast(cl_file)
  for k, v in methods.items():
    get_alm = v.get("alm")
    alm = get_alm(cmb, window_tuple, niter, lmax)
    ell, ps = so_spectra.get_spectra(alm, spectra=spectra)
    ellb, ps_dict = so_spectra.bin_spectra(ell,
                                           ps,
                                           binning_file,
                                           lmax,
                                           type="Cl",
                                           mbb_inv=v.get("mbb"),
                                           spectra=spectra)
    v["bb"] += [ps_dict["BB"]]
```

Let's plot the mean results against the theory value for BB spectrum

```python
for k, v in methods.items():
    v["mean"] = np.mean(v.get("bb"), axis=0)
    v["std"] = np.std(v.get("bb"), axis=0)

from pspy import pspy_utils
ell_th, ps_theory = pspy_utils.ps_lensed_theory_to_dict(cl_file, output_type="Cl", lmax=lmax)
ps_theory_b = so_mcm.apply_Bbl(Bbl, ps_theory, spectra=spectra)
ps_theory_b_pure = so_mcm.apply_Bbl(Bbl_pure, ps_theory, spectra=spectra)

fac = ellb * (ellb + 1) / (2 * np.pi)
facth = ell_th * (ell_th + 1) / (2 * np.pi)

plt.figure(figsize=(7, 6))
grid = plt.GridSpec(4, 1, hspace=0, wspace=0)

main = plt.subplot(grid[:3], xticklabels=[], xlim=(0, 2*nside))
main.plot(ell_th[:lmax], ps_theory["BB"][:lmax] * facth[:lmax], color="grey")
main.errorbar(ellb, ps_theory_b["BB"] * fac, color="tab:red", label="binned theory BB")
main.errorbar(ellb, ps_theory_b_pure["BB"] * fac, color="tab:blue", label="binned theory BB pure")
main.errorbar(ellb, methods.get("standard").get("mean") * fac,
              methods.get("standard").get("std") * fac, fmt=".", color="tab:red", label="mean BB")
main.errorbar(ellb, methods.get("pure").get("mean") * fac,
             methods.get("pure").get("std") * fac, fmt=".", color="tab:blue", label="mean BB pure")
main.set(ylim=(-0.07, 0.17), ylabel=r"$D^{BB}_{\ell}$")
plt.legend(title=r"$n_{\rm sims}=%s$" % n_sims)

ratio = plt.subplot(grid[3], xlim=(0, 2*nside))
ratio.plot(ellb, methods.get("pure").get("std") / methods.get("standard").get("std"), ".-k")
ratio.set(ylabel=r"$\sigma^{\rm pure}_\ell/ \sigma_\ell$", xlabel=r"$\ell$");
ratio.axhline(1)
```