# Chapter 4
`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`

______

## The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been using this simple idea in every example thus far.

```python
import json
import matplotlib
s = json.load(open("../styles/bmh_matplotlibrc.json"))
matplotlib.rcParams.update(s)
```

    /home/gjc216/.virtualenvs/edge/lib/python3.7/site-packages/matplotlib/__init__.py:855: MatplotlibDeprecationWarning:
    examples.directory is deprecated; in the future, examples will be found relative to the 'datapath' directory.
      "found relative to the 'datapath' directory.".format(key))

### The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear by examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for almost *any distribution*, minus some important cases we will encounter later.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size = 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to its parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```python
%matplotlib inline
import numpy as np
from IPython.core.pylabtools import figsize
import matplotlib.pyplot as plt

figsize( 12.5, 5 )

sample_size = 100000
expected_value = lambda_ = 4.5
poi = np.random.poisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average),
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend();
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This is simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

The above formulae is interpretable as a distance away from the true value (on average), for some $N$. (We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

By computing the above many, $N_y$, times (remember, it is random), and averaging them:

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \rightarrow E[ Y_k ] = E\;\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \right]$$

Finally, taking the square root:

$$ \sqrt{\frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k} \approx D(N) $$

```python
figsize( 12.5, 4)

N_Y = 250 #use this many to approximate D(N)
N_array = np.arange( 1000, 50000, 2500 ) #use this many samples in the approx. to the variance.
D_N_results = np.zeros( len( N_array ) )

lambda_ = 4.5
expected_value = lambda_ #for X ~ Poi(lambda) , E[ X ] = lambda

def D_N( n ):
    """
    This function approx. D_n, the average variance of using n samples.
    """
    Z = poi( lambda_, (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for i,n in enumerate(N_array):
    D_N_results[i] =  D_N(n)

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 2, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " );
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of convergence to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 5, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 5 ) =  \frac{1}{N}\sum_{i=1}^N \mathbb{1}_{z > 5 }(Z_i) $$

```python
N = 100000
print( np.mean( [ np.random.exponential( 0.6 ) > 5 for i in range(N) ] ) )
```

    0.00022

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give us *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```python
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = np.random.randint
norm = np.random.normal

#generate some artificial population numbers
population = pop_generator(100, 1500, n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height,
                                        population[i] ).mean()

#located the counties with the apparently most extreme average heights.
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population size vs. recorded average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1);
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error results from the calculated average of smaller populations not being a good reflection of the true expected value of the population (which in truth should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 1500, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```python
print("Population sizes of 10 'shortest' counties: ")
print(population[ np.argsort( average_across_county )[:10] ], '\n')
print("Population sizes of 10 'tallest' counties: ")
print(population[ np.argsort( -average_across_county )[:10] ])
```

    Population sizes of 10 'shortest' counties:
    [104 127 117 123 253 107 142 281 396 185]

    Population sizes of 10 'tallest' counties:
    [130 108 176 273 345 127 138 174 134 154]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```python
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);
```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How to order Reddit submissions

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, called submissions, for people to comment on. Redditors can vote up or down on each submission (called upvotes and downvotes). Reddit, by default, will sort submissions to a given subreddit by Hot, that is, the submissions that have the most upvotes recently.

How would you determine which submissions are the best? There are a number of ways to achieve this:

1. *Popularity*: A submission is considered good if it has many upvotes. A problem with this model is that a submission with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the submission is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of submission. Depending on when a submission is posted, the website may be experiencing high or low traffic. The difference method will bias the *Top* submissions to be the those made during high traffic periods, which have accumulated more upvotes than submissions that were not so graced, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the submission. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter-example is, if we use per second, a 1 second old submission with 1 upvote would be better than a 100 second old submission with 99 upvotes. One can avoid this by only considering at least t second old submission. But what is a good t value? Does this mean no submission younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old submissions).
3. *Ratio*: Rank submissions by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new submissions who score well can be considered Top just as likely as older submissions, provided they have many upvotes to total votes. The problem here is that a submission with a single upvote (ratio = 1.0) will beat a submission with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter submission is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former submission, with a single upvote, is in fact a better submission than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former submission might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this submission a upvote, versus a downvote"). So the 999 upvote/1 downvote submission probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the submission with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is to look at the historical distribution of upvote ratios. This can be accomplished by scraping Reddit's submissions and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of submissions have very few votes, hence there will be many submissions with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use submissions with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of submissions available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards submissions of these two subreddits are very different: visitors are likely friendly and affectionate in the former, and would therefore upvote submissions more, compared to the latter, where submissions are likely to be controversial and disagreed upon. Therefore not all submissions are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `top_showerthoughts_submissions.py` will scrape the best posts from the `showerthoughts` community on Reddit. This is a text-only community so the title of each post *is* the post. Below is the top post as well as some other sample posts:

```python
#adding a number to the end of the %run call will get the ith top post.
%run top_showerthoughts_submissions.py 2

print("Post contents: \n")
print(top_post)
```

    6.2.0
    Post contents:

    Those "choose your own flavor" Coke machines are probably just there to gather marketing data; They see what the most popular flavors are so they can release bottled/canned versions.

```python
"""
contents: an array of the text from the last 100 top submissions to a subreddit
votes: a 2d numpy array of upvotes, downvotes for each submission.
"""
n_submissions = len(votes)
submissions = np.random.randint( n_submissions, size=4)
print("Some Submissions (out of %d total) \n-----------"%n_submissions)
for i in submissions:
    print('"' + contents[i] + '"')
    print("upvotes/downvotes: ",votes[i,:], "\n")
```

    Some Submissions (out of 98 total)
    -----------
    "At some point in your life, someone will take a photo of you that will be on display at your funeral."
    upvotes/downvotes:  [16  0]

    "Your dog probably sees things they are terrified of outside but decide to bark anyway to warn you/protect you."
    upvotes/downvotes:  [208   9]

    "All money is pizza money it’s just sometimes you have to use it for other things"
    upvotes/downvotes:  [431  59]

    "If you were born in 1869 and lived to be 100, you were born when the transcontinental railroad was finished and lived to see the moon landing. It's possible that same amount of change could happen in our lifetimes."
    upvotes/downvotes:  [81  5]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular submission's upvote/downvote pair.

```python
import pymc3 as pm

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    """
    This function accepts the number of upvotes and downvotes a particular submission recieved,
    and the number of posterior samples to return to the user. Assumes a uniform prior.
    """
    N = upvotes + downvotes
    with pm.Model() as model:
        upvote_ratio = pm.Uniform("upvote_ratio", 0, 1)
        observations = pm.Binomial( "obs",  N, upvote_ratio, observed=upvotes)

        trace = pm.sample(samples, step=pm.Metropolis())

    burned_trace = trace[int(samples/4):]
    return burned_trace["upvote_ratio"]

```

Below are the resulting posterior distributions.

```python
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(submissions)):
    j = submissions[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 10, density = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 10, density = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different submissions");
```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the submissions from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean is a bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```python
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(submissions)):
    j = submissions[i]
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")
plt.title("Posterior distributions of upvote ratios on different submissions");
order = np.argsort( -np.array( lower_limits ) )
print(order, lower_limits)
```

The best submissions, according to our procedure, are the submissions that are *most-likely* to score a high percentage of upvotes. Visually those are the submissions with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best.  When using the lower-bound of the 95% credible interval, we believe with high certainty that the 'true upvote ratio' is at the very least equal to this value (or greater), thereby ensuring that the best submissions are still on top. Under this ordering, we impose the following very natural properties:

1. given two submissions with the same observed upvote ratio, we will assign the submission with more votes as better (since we are more confident it has a higher ratio).
2. given two submissions with the same number of votes, we still assign the submission with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every submission takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```python
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print("Approximate lower bounds:")
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print(lb)
print("\n")
print("Top 40 Sorted according to approximate lower bounds:")
print("\n")
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print(votes[i,0], votes[i,1], contents[i])
    print("-------------")
```

    Approximate lower bounds:
    [0.96711925 0.94162649 0.83521658 0.89948033 0.93298123 0.92139647
     0.95607591 0.92458598 0.95067603 0.91325068 0.95467916 0.91301601
     0.86800628 0.93300071 0.91387422 0.95452891 0.94318204 0.93385266
     0.93771511 0.90866277 0.8436714  0.84979447 0.96023852 0.91465559
     0.75425942 0.96587192 0.90183539 0.92327676 0.89983924 0.9244614
     0.97070774 0.85373162 0.7816908  0.96400616 0.94246974 0.88897414
     0.88621472 0.93303496 0.95131565 0.91494935 0.9073345  0.93792895
     0.94874614 0.86765877 0.86828537 0.89610301 0.8825551  0.94191707
     0.90991491 0.88773339 0.83703075 0.95082337 0.79811635 0.95763257
     0.83730175 0.82382813 0.8027221  0.90184807 0.94854702 0.94864269
     0.87279132 0.76137738 0.9507226  0.85542404 0.89239065 0.73711754
     0.92346577 0.92561395 0.81207876 0.76310347 0.88796035 0.93173568
     0.81109538 0.85941097 0.80731596 0.83490455 0.80933194 0.73142963
     0.94799705 0.81845702 0.74556767 0.93111579 0.82545948 0.77910901
     0.71674317 0.76867393 0.81109538 0.80933194 0.66500492 0.95264061
     0.93577171 0.81019924 0.78188878 0.72708753 0.86498277 0.91228157
     0.85773646 0.66251028]

    Top 40 Sorted according to approximate lower bounds:

    195 2 Sometimes it’s hard to tell if you hang out with someone because you like them, or because of habit.
    -------------
    9833 304 With Doris Day’s death (RIP), Queen Elizabeth II, Brigette Bardot, Chubby Checker, Bob Dylan, Ringo Starr, and Paul McCartney are the only 6 people alive who are referenced in Billy Joel’s “We Didn’t Start the Fire.”
    -------------
    433 9 People don't celebrate your birthday as much when you get older because the novelty of you existing has worn off.
    -------------
    71 0 We all have different voices in our head when people tExT LiKe ThIs
    -------------
    975 30 If robots ever gain sentience then will they make their own "drugs"? Like viruses that mess with their programming so they feel better
    -------------
    18858 786 Banjos sound like a guitar with a southern accent
    -------------
    6938 289 Telling a person to "breathe" is better than telling them to "calm down"
    -------------
    4045 169 One of the craziest moments growing up as a kid was realising that teachers exist and have a life outside of school.
    -------------
    452 14 Out of all the times I’ve seen a squirrel. I’ve never seen one pooping.
    -------------
    148 3 If we were to have a 3rd World War today, kids from opposing countries could be trash talking each other online.
    -------------
    373 12 Nobody has hated you enough to kill you yet
    -------------
    1435 60 Pornography is the only form of media that you want to know all the spoilers before experiencing it.
    -------------
    142 3 In maybe a couple of years, seeing a car with no driver will be an every day sight, and won't be creepy.
    -------------
    1302 54 A bowl cut is a 360° bang
    -------------
    284 9 Every time snakes move, they get a belly rub.
    -------------
    136 3 Artificial intelligence has a long way to go to become the threat to humanity that Natural stupidity already is.
    -------------
    914 38 At some point, we decided to stop carrying around swords everywhere
    -------------
    209 6 Won’t power is probably stronger than will power
    -------------
    458 19 If you make a diamond hoe in minecraft, you're really good at the game or really bad at the game.
    -------------
    188 6 The next generation will never know the fun we had when dragging an error box around a computer to cover the entire screen
    -------------
    43 0 Some poor bastard died in WW1 thinking “At least this is the war that will end all wars”
    -------------
    1905 100 If girls only had sex with guys with big dicks, guys would eventually evolve to only have big dicks
    -------------
    66 1 With self driving cars, knowing how to drive will eventually become an ancient skill
    -------------
    325 14 Because of the internet a guy will see more naked women than all the men in his lineage and through tinder he will get rejected by more women than all the men in his lineage.
    -------------
    128 4 Those fun zones where kids play games and win tickets to exchange for prizes -- they're just practice casinos.
    -------------
    235 10 If cats could talk, they probably wouldn't.
    -------------
    516 27 We look back 100 years ago and feel sorry that people had to suffer through pain without any medication. But in 100 years time, people will have some amazing method of dealing with pain and feel sorry for us that we had to use medication.
    -------------
    548 29 The Target-to-Walmart ratio is a good indicator of how affluent a neighborhood is.
    -------------
    3165 202 We never appreciate breathing through our noses until we can’t anymore
    -------------
    120 4 Censored bad word seem much more serious than uncensored
    -------------
    208 9 Your dog probably sees things they are terrified of outside but decide to bark anyway to warn you/protect you.
    -------------
    33 0 Tug of war is the only time the fat kid gets picked first in gym class
    -------------
    679 43 Once you turn 18, your 6,570 day free trial has expired.
    -------------
    157 7 The show Veggie Tales’ main character is a fruit.
    -------------
    32 0 Your pet probably thinks you're cute, too.
    -------------
    89 3 Seeing a teacher outside of school is the lowest level of celebrity sighting.
    -------------
    2368 178 The heat from the sun just traveled 149.6 million killmeters just to melt our fucking chocolate bar.
    -------------
    166 9 Finding that Santa ate your cookies as a child when you're home alone is exciting and amazing. Finding Santa ate your cookies as an adult when you're home alone is terrifying and scary
    -------------
    822 62 There are infinite realities where the only difference are the position of a single atom.
    -------------
    294 19 This year for Halloween, there's going to be a bunch of fat, middle age men dressing up as Thor
    -------------

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```python
r_order = order[::-1][-40:]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order], capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30].replace("\n",""), ordered_contents) );
```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Extension to Starred rating systems

The above procedure works well for upvote-downvotes schemes, but what about systems that use star ratings, e.g. 5 star rating systems. Similar problems apply with simply taking the average: an item with two perfect ratings would beat an item with thousands of perfect ratings, but a single sub-perfect rating.

We can consider the upvote-downvote problem above as binary: 0 is a downvote, 1 if an upvote. A $N$-star rating system can be seen as a more continuous version of above, and we can set $n$ stars rewarded is equivalent to rewarding $\frac{n}{N}$. For example, in a 5-star system, a 2 star rating corresponds to 0.4. A perfect rating is a 1. We can use the same formula as before, but with $a,b$ defined differently:

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where

\begin{align}
& a = 1 + S \\\\
& b = 1 + N - S \\\\
\end{align}

where $N$ is the number of users who rated, and $S$ is the sum of all the ratings, under the equivalence scheme mentioned above.

##### Example: Counting Github stars

What is the average number of stars a Github repository has? How would you calculate this? There are over 6 million respositories, so there is more than enough data to invoke the Law of Large numbers. Let's start pulling some data. TODO

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting submissions formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead use a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally accurate?

```python
## Enter code here
import scipy.stats as stats
exp = stats.expon( scale=4 )
N = 1e5
X = exp.rvs( int(N) )
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by their percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

In August 2013, [a popular post](http://bpodgursky.wordpress.com/2013/08/21/average-income-per-programming-language/) on the average income per programmer of different languages was trending. Here's the summary chart: (reproduced without permission, cause when you lie with stats, you gunna get the hammer). What do you notice about the extremes?

------

#### Average household income by programming language

<table >
 <tr><td>Language</td><td>Average Household Income ($)</td><td>Data Points</td></tr>
 <tr><td>Puppet</td><td>87,589.29</td><td>112</td></tr>
 <tr><td>Haskell</td><td>89,973.82</td><td>191</td></tr>
 <tr><td>PHP</td><td>94,031.19</td><td>978</td></tr>
 <tr><td>CoffeeScript</td><td>94,890.80</td><td>435</td></tr>
 <tr><td>VimL</td><td>94,967.11</td><td>532</td></tr>
 <tr><td>Shell</td><td>96,930.54</td><td>979</td></tr>
 <tr><td>Lua</td><td>96,930.69</td><td>101</td></tr>
 <tr><td>Erlang</td><td>97,306.55</td><td>168</td></tr>
 <tr><td>Clojure</td><td>97,500.00</td><td>269</td></tr>
 <tr><td>Python</td><td>97,578.87</td><td>2314</td></tr>
 <tr><td>JavaScript</td><td>97,598.75</td><td>3443</td></tr>
 <tr><td>Emacs Lisp</td><td>97,774.65</td><td>355</td></tr>
 <tr><td>C#</td><td>97,823.31</td><td>665</td></tr>
 <tr><td>Ruby</td><td>98,238.74</td><td>3242</td></tr>
 <tr><td>C++</td><td>99,147.93</td><td>845</td></tr>
 <tr><td>CSS</td><td>99,881.40</td><td>527</td></tr>
 <tr><td>Perl</td><td>100,295.45</td><td>990</td></tr>
 <tr><td>C</td><td>100,766.51</td><td>2120</td></tr>
 <tr><td>Go</td><td>101,158.01</td><td>231</td></tr>
 <tr><td>Scala</td><td>101,460.91</td><td>243</td></tr>
 <tr><td>ColdFusion</td><td>101,536.70</td><td>109</td></tr>
 <tr><td>Objective-C</td><td>101,801.60</td><td>562</td></tr>
 <tr><td>Groovy</td><td>102,650.86</td><td>116</td></tr>
 <tr><td>Java</td><td>103,179.39</td><td>1402</td></tr>
 <tr><td>XSLT</td><td>106,199.19</td><td>123</td></tr>
 <tr><td>ActionScript</td><td>108,119.47</td><td>113</td></tr>
</table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. [Web](http://www.sloansportsconference.com/wp-content/uploads/2013/Going%20for%20Three%20Predicting%20the%20Likelihood%20of%20Field%20Goal%20Success%20with%20Logistic%20Regression.pdf). 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

<style>
    img{
        max-width:800px}
</style>


```python
import sympy as sp
from sympy import *
from sympy.abc import *
```

# Summary

Any propositional formula can be rewritten with De Morgan laws as an equivalent formula in Clausal Normal Form. However this can lead to exponential growhth in size and therefore prevent any real life application.

Tseitin transformation https://en.wikipedia.org/wiki/Tseytin_transformation is a classical work around. It transform a propositional formula A into another formula T(A) which can be in turn be rewritten as a CNF with a linear increase in size.

We illustrated Tseitin transformation using sympy

```python
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11 = symbols('x:12')
```

Straight forward rewritten of (x0 & x1) | (x2 & x3) | (x4 & x5) | (x6 & x7)| (x8 & x9)| (x10 & x11) into a CNF has 64 clauses of 6 literals each

```python
sp.to_cnf((x0 & x1) | (x2 & x3) | (x4 & x5) | (x6 & x7)| (x8 & x9)| (x10 & x11))
```

    (x0 | x10 | x2 | x4 | x6 | x8) & (x0 | x10 | x2 | x4 | x6 | x9) & (x0 | x10 | x2 | x4 | x7 | x8) & (x0 | x10 | x2 | x4 | x7 | x9) & (x0 | x10 | x2 | x5 | x6 | x8) & (x0 | x10 | x2 | x5 | x6 | x9) & (x0 | x10 | x2 | x5 | x7 | x8) & (x0 | x10 | x2 | x5 | x7 | x9) & (x0 | x10 | x3 | x4 | x6 | x8) & (x0 | x10 | x3 | x4 | x6 | x9) & (x0 | x10 | x3 | x4 | x7 | x8) & (x0 | x10 | x3 | x4 | x7 | x9) & (x0 | x10 | x3 | x5 | x6 | x8) & (x0 | x10 | x3 | x5 | x6 | x9) & (x0 | x10 | x3 | x5 | x7 | x8) & (x0 | x10 | x3 | x5 | x7 | x9) & (x0 | x11 | x2 | x4 | x6 | x8) & (x0 | x11 | x2 | x4 | x6 | x9) & (x0 | x11 | x2 | x4 | x7 | x8) & (x0 | x11 | x2 | x4 | x7 | x9) & (x0 | x11 | x2 | x5 | x6 | x8) & (x0 | x11 | x2 | x5 | x6 | x9) & (x0 | x11 | x2 | x5 | x7 | x8) & (x0 | x11 | x2 | x5 | x7 | x9) & (x0 | x11 | x3 | x4 | x6 | x8) & (x0 | x11 | x3 | x4 | x6 | x9) & (x0 | x11 | x3 | x4 | x7 | x8) & (x0 | x11 | x3 | x4 | x7 | x9) & (x0 | x11 | x3 | x5 | x6 | x8) & (x0 | x11 | x3 | x5 | x6 | x9) & (x0 | x11 | x3 | x5 | x7 | x8) & (x0 | x11 | x3 | x5 | x7 | x9) & (x1 | x10 | x2 | x4 | x6 | x8) & (x1 | x10 | x2 | x4 | x6 | x9) & (x1 | x10 | x2 | x4 | x7 | x8) & (x1 | x10 | x2 | x4 | x7 | x9) & (x1 | x10 | x2 | x5 | x6 | x8) & (x1 | x10 | x2 | x5 | x6 | x9) & (x1 | x10 | x2 | x5 | x7 | x8) & (x1 | x10 | x2 | x5 | x7 | x9) & (x1 | x10 | x3 | x4 | x6 | x8) & (x1 | x10 | x3 | x4 | x6 | x9) & (x1 | x10 | x3 | x4 | x7 | x8) & (x1 | x10 | x3 | x4 | x7 | x9) & (x1 | x10 | x3 | x5 | x6 | x8) & (x1 | x10 | x3 | x5 | x6 | x9) & (x1 | x10 | x3 | x5 | x7 | x8) & (x1 | x10 | x3 | x5 | x7 | x9) & (x1 | x11 | x2 | x4 | x6 | x8) & (x1 | x11 | x2 | x4 | x6 | x9) & (x1 | x11 | x2 | x4 | x7 | x8) & (x1 | x11 | x2 | x4 | x7 | x9) & (x1 | x11 | x2 | x5 | x6 | x8) & (x1 | x11 | x2 | x5 | x6 | x9) & (x1 | x11 | x2 | x5 | x7 | x8) & (x1 | x11 | x2 | x5 | x7 | x9) & (x1 | x11 | x3 | x4 | x6 | x8) & (x1 | x11 | x3 | x4 | x6 | x9) & (x1 | x11 | x3 | x4 | x7 | x8) & (x1 | x11 | x3 | x4 | x7 | x9) & (x1 | x11 | x3 | x5 | x6 | x8) & (x1 | x11 | x3 | x5 | x6 | x9) & (x1 | x11 | x3 | x5 | x7 | x8) & (x1 | x11 | x3 | x5 | x7 | x9)

Let's have first a look at Tseitin transformation for formula $$((p \lor q) \land r) \implies \neg s$$
we need to
1. compute all subformulas, atoms excepted
2. sort them by depth
3. Conjunct all substitutions and the substitution
4. then transform all substitution in CNF

Step 1: computing $$((p \lor q) \land r) \implies \neg s$$ subformulas

```python
def subformulas(expr):
# subformula of expr without atoms
# considering operator as binary

    if expr.func == sp.Symbol:
        return []
    elif len(expr.args)==1:
        return [expr] + subformulas(expr.args[0])
    elif len(expr.args)==2:
        return [expr] + subformulas(expr.args[0]) + subformulas(expr.args[1])
    elif len(expr.args)>2:
        return [expr] + subformulas(expr.args[0]) + subformulas(eval(str(expr.func)+str(expr.args[1:])))
    else:
        return "error"
```

```python
subformulas(((p |q) & r) >> (~s))
```

    [Implies(r & (p | q), ~s), r & (p | q), p | q, ~s]

**Note**: we considers the operators as binary while sympy consider them as n-ary. So we choose for subformulas (ignoring atoms) of (A & B & C), the subformulas of A & (B & C): A & B & C,  B & C

```python
subformulas(A & B & C)
```

    [A & B & C, B & C]

We define the depth of a formula in the following way
- depth(A)=0 if A is an atom
- depth(~F) = depth(F) + 1
- depth(F1 op F2) = max(depth(F1), depth(F2)) + 1, for op a boolean binary operator (here we consider conjunction, disjunction and implication)

```python
def depth(expr):
# still considering operator as binary
    if expr.func == Not:
        return 1 + depth(expr.args[0])
    elif (len(expr.args)==2):
        return 1 + max([depth(x) for x in expr.args])
    elif ( len(expr.args)>2):
        #return 1 + max(depth(expr.args[0]), depth(eval(str(expr.func)+str(expr.args[1:]))))
        return len(expr.args)-1 + max([depth(x) for x in expr.args])
    elif expr.func == Symbol:
        return 0
    else:
        return"error"
```

```python
depth(((p |q) & r) >> (~s))
```

    3

Again we consider the boolen operator to be binary, so depth(A & B & C) = depth(A & (B & C))=2

```python
depth(A& B & C)
```

    2

Sorting subformulas (not taking into account atoms) of
$$((p \lor q) \land r) \implies \neg s$$
per depth will give us
$$\begin{array}{l}
p \lor q\\
\neg s \\
r \land (p \lor q)\\
r \land (p \lor q) \implies \neg s\\
\end{array}$$

```python
F = ((p |q) & r) >> (~s)
sorted(subformulas(F), key=depth)
```

    [p | q, ~s, r & (p | q), Implies(r & (p | q), ~s)]

We can now  write the following formulas

$\begin{array}{lll}
x_0  & \Leftrightarrow & p \lor q \\
x_1  & \Leftrightarrow & \neg s \\
x_2  & \Leftrightarrow & r \land x_0\\
x_3  & \Leftrightarrow & (x_2 \implies x_1)
\end{array}$

and conjunct them to get

$T(((p \lor q) \land r) \implies \neg s) := x_3 \land (x_3  \Leftrightarrow (x2 \implies x1)) \land (x_2  \Leftrightarrow r \land x0) \land (x_1  \Leftrightarrow \neg s) \land (x_0  \Leftrightarrow p \lor q)$

First We need  a way to get new symbol. This is done using closure:

```python
def symbol_init(x, count=0):
    def new_symbol ():
        nonlocal count
        count += 1
        return var(str(x) + str(count)) #var add the new symbol in the name space
    return new_symbol
```

```python
si = symbol_init('t')
```

```python
si() | si()
```

    t1 | t2

As sympy boolean operator are n aries, sympy subs will not work well for us need. For instance

```python
(x1 & Not(x2) &x3).subs(Not(x2) & x3, A)
```

    x1 & x3 & ~x2

So we need to define a custom substiturion handling boolean operator as binary

```python
def binSubs(x, y, z):
    #replace y by z in x
    #print("subs", x, y, z)
    if (len(y.args) >0) and (len(x.args)>2) and (y!=x) and (y.func==x.func):

        if all([y in x.args for y in y.args]):
            l = list(x.args)
            for i in y.args:
                l.remove(i)
            if (z.func== Symbol):
                t=tuple(l + [z])
            else:
                t=tuple(l+list(z.args))
            return eval(str(x.func) + str(t))

    return x.subs(y,z)

```

```python
binSubs((x1 & ~x2 &x3), x1 & ~x2, A)
```

    A & x3

We can now put together tseitin transformation

```python
def tseitin1(expr):
    # for binary operation

    s=symbol_init("p")
    if expr.func == Symbol:
        return true
    else:
        return tseitin_aux(sorted(subformulas(expr), key=depth), s)

def tseitin_aux(L, s):
    #nonlocal newvar
    newvar = s()
    if (L==[]):
        return true
    if depth(L[0]) > 0:
        if len(L[1:])>0 :
            #newList = [x.subs(L[0], newvar) for x in L[1:]]

            newList = [binSubs(x, L[0], newvar) for x in L[1:]]

            return Equivalent(newvar, L[0]) & tseitin_aux(newList, s)
        else:
            return Equivalent(newvar, L[0]) & newvar
    else:
        print("error")
```

```python
F = ((p |q) & r) >> (~s)
tseitin1(F)
```

    p4 & (Equivalent(p2, ~s)) & (Equivalent(p1, p | q)) & (Equivalent(p3, p1 & r)) & (Equivalent(p4, Implies(p3, p2)))

```python
F2 = (~(p |~q) & r) >> (~s)
tseitin1(F2)
```

    p6 & (Equivalent(p1, ~q)) & (Equivalent(p2, ~s)) & (Equivalent(p4, ~p3)) & (Equivalent(p3, p | p1)) & (Equivalent(p5, p4 & r)) & (Equivalent(p6, Implies(p5, p2)))

From there, we leverage makes the final transformation to CNF leveraging the following equivalences

$\begin{array}{lll}
x  \Leftrightarrow (y \land z)  & \equiv  & (y \lor \neg x) \land (z \lor \neg x) \land (x \lor \neg y \lor \neg z)\\
x  \Leftrightarrow (y \lor z) & \equiv & (x \lor \neg y) \land (x \lor \neg z) \land (y \lor z \lor \neg x)\\
x  \Leftrightarrow (y \implies z) & \equiv & (x \lor y) \land (x \lor \neg z) \land (z \lor \neg x \lor \neg y)\\
x  \Leftrightarrow (\neg y) & \equiv & (x \lor y) \land (\neg x \lor \neg y)\\
\end{array}$

```python
print(sp.to_cnf(Equivalent(x, y & z)))
print(sp.to_cnf(Equivalent(x, y | z)))
print(sp.to_cnf(Equivalent(x, y >> z)))
print(sp.to_cnf(Equivalent(x, ~y)))
```

    (y | ~x) & (z | ~x) & (x | ~y | ~z)
    (x | ~y) & (x | ~z) & (y | z | ~x)
    (x | y) & (x | ~z) & (z | ~x | ~y)
    (x | y) & (~x | ~y)

```python
def tseitin2(Expr):
    #take as input the output of tseitin1: a conjuction of equivalence
    # replaces the equivalences by conjunction of disjunction of litterals
    result = Expr.args[0] #first arg is an atom by construction
    for i in Expr.args[1:]:
        op = i.args[1].func
        x = i.args[0]
        y =  i.args[1].args[0]
        if (op == Not):
            result = result & (x | y) & (~x | ~y)
        else:
            z = i.args[1].args[1]
            if (op == And):
                result = result & (y | ~x) & (z | ~x) & (x | ~y | ~z)
            elif (op == Or):
                result = result & (x | ~y) & (x | ~z) & (y | z | ~x)
            elif (op == Implies):
                result = result & (x | y) & (x | ~z) & (z | ~x | ~y)
            else:
                print("error: unsupported operator")
    return result
```

```python
def tseitin(Expr):
    return tseitin2(tseitin1(Expr))
```

```python
F = ((p |q) & r) >> (~s)
print("original formula:", F)
T = tseitin1(F)
print("step 1: ", T)
c = tseitin2(T)
print("step 2: ", c)

```

    original formula: Implies(r & (p | q), ~s)
    step 1:  p4 & (Equivalent(p2, ~s)) & (Equivalent(p1, p | q)) & (Equivalent(p3, p1 & r)) & (Equivalent(p4, Implies(p3, p2)))
    step 2:  p4 & (p2 | s) & (p3 | p4) & (p1 | ~p) & (p1 | ~p3) & (p1 | ~q) & (p4 | ~p2) & (r | ~p3) & (~p2 | ~s) & (p | q | ~p1) & (p2 | ~p3 | ~p4) & (p3 | ~p1 | ~r)

```python

```

```python
tseitin(Not((A & B ) | C))
```

    p3 & (p2 | p3) & (A | ~p1) & (B | ~p1) & (p2 | ~C) & (p2 | ~p1) & (~p2 | ~p3) & (C | p1 | ~p2) & (p1 | ~A | ~B)

```python
tseitin((x1 & x2 & x3))
```

    p2 & (p1 | ~p2) & (x1 | ~p2) & (x2 | ~p1) & (x3 | ~p1) & (p1 | ~x2 | ~x3) & (p2 | ~p1 | ~x1)

```python
tseitin((x1 & x2) | (x3 & x4) | (x5 & x6))
```

    p5 & (p4 | ~p2) & (p4 | ~p3) & (p5 | ~p1) & (p5 | ~p4) & (x1 | ~p1) & (x2 | ~p1) & (x3 | ~p2) & (x4 | ~p2) & (x5 | ~p3) & (x6 | ~p3) & (p1 | p4 | ~p5) & (p2 | p3 | ~p4) & (p1 | ~x1 | ~x2) & (p2 | ~x3 | ~x4) & (p3 | ~x5 | ~x6)

```python
tseitin((x1 & x2) | (x3 & x4) | (x5 & x6)| (x7 & x8))
```

    p7 & (p5 | ~p3) & (p5 | ~p4) & (p6 | ~p2) & (p6 | ~p5) & (p7 | ~p1) & (p7 | ~p6) & (x1 | ~p1) & (x2 | ~p1) & (x3 | ~p2) & (x4 | ~p2) & (x5 | ~p3) & (x6 | ~p3) & (x7 | ~p4) & (x8 | ~p4) & (p1 | p6 | ~p7) & (p2 | p5 | ~p6) & (p3 | p4 | ~p5) & (p1 | ~x1 | ~x2) & (p2 | ~x3 | ~x4) & (p3 | ~x5 | ~x6) & (p4 | ~x7 | ~x8)

 Tseitin transformation of formula  (x0 & x1) | (x2 & x3) | (x4 & x5) | (x6 & x7)| (x8 & x9)| (x10 & x11) has 34 clauses of one, two or three litteral each, to be compared with the 64 clauses of 6 literals each of the classical CNF transformation

```python
tseitin((x0 & x1) | (x2 & x3) | (x4 & x5) | (x6 & x7)| (x8 & x9)| (x10 & x11))
```

    p11 & (p10 | ~p2) & (p10 | ~p9) & (p11 | ~p1) & (p11 | ~p10) & (p7 | ~p5) & (p7 | ~p6) & (p8 | ~p4) & (p8 | ~p7) & (p9 | ~p3) & (p9 | ~p8) & (x0 | ~p1) & (x1 | ~p1) & (x10 | ~p2) & (x11 | ~p2) & (x2 | ~p3) & (x3 | ~p3) & (x4 | ~p4) & (x5 | ~p4) & (x6 | ~p5) & (x7 | ~p5) & (x8 | ~p6) & (x9 | ~p6) & (p1 | p10 | ~p11) & (p2 | p9 | ~p10) & (p3 | p8 | ~p9) & (p4 | p7 | ~p8) & (p5 | p6 | ~p7) & (p1 | ~x0 | ~x1) & (p2 | ~x10 | ~x11) & (p3 | ~x2 | ~x3) & (p4 | ~x4 | ~x5) & (p5 | ~x6 | ~x7) & (p6 | ~x8 | ~x9)

```python
sp.to_cnf((x0 & x1) | (x2 & x3) | (x4 & x5) | (x6 & x7)| (x8 & x9)| (x10 & x11))
```

    (x0 | x10 | x2 | x4 | x6 | x8) & (x0 | x10 | x2 | x4 | x6 | x9) & (x0 | x10 | x2 | x4 | x7 | x8) & (x0 | x10 | x2 | x4 | x7 | x9) & (x0 | x10 | x2 | x5 | x6 | x8) & (x0 | x10 | x2 | x5 | x6 | x9) & (x0 | x10 | x2 | x5 | x7 | x8) & (x0 | x10 | x2 | x5 | x7 | x9) & (x0 | x10 | x3 | x4 | x6 | x8) & (x0 | x10 | x3 | x4 | x6 | x9) & (x0 | x10 | x3 | x4 | x7 | x8) & (x0 | x10 | x3 | x4 | x7 | x9) & (x0 | x10 | x3 | x5 | x6 | x8) & (x0 | x10 | x3 | x5 | x6 | x9) & (x0 | x10 | x3 | x5 | x7 | x8) & (x0 | x10 | x3 | x5 | x7 | x9) & (x0 | x11 | x2 | x4 | x6 | x8) & (x0 | x11 | x2 | x4 | x6 | x9) & (x0 | x11 | x2 | x4 | x7 | x8) & (x0 | x11 | x2 | x4 | x7 | x9) & (x0 | x11 | x2 | x5 | x6 | x8) & (x0 | x11 | x2 | x5 | x6 | x9) & (x0 | x11 | x2 | x5 | x7 | x8) & (x0 | x11 | x2 | x5 | x7 | x9) & (x0 | x11 | x3 | x4 | x6 | x8) & (x0 | x11 | x3 | x4 | x6 | x9) & (x0 | x11 | x3 | x4 | x7 | x8) & (x0 | x11 | x3 | x4 | x7 | x9) & (x0 | x11 | x3 | x5 | x6 | x8) & (x0 | x11 | x3 | x5 | x6 | x9) & (x0 | x11 | x3 | x5 | x7 | x8) & (x0 | x11 | x3 | x5 | x7 | x9) & (x1 | x10 | x2 | x4 | x6 | x8) & (x1 | x10 | x2 | x4 | x6 | x9) & (x1 | x10 | x2 | x4 | x7 | x8) & (x1 | x10 | x2 | x4 | x7 | x9) & (x1 | x10 | x2 | x5 | x6 | x8) & (x1 | x10 | x2 | x5 | x6 | x9) & (x1 | x10 | x2 | x5 | x7 | x8) & (x1 | x10 | x2 | x5 | x7 | x9) & (x1 | x10 | x3 | x4 | x6 | x8) & (x1 | x10 | x3 | x4 | x6 | x9) & (x1 | x10 | x3 | x4 | x7 | x8) & (x1 | x10 | x3 | x4 | x7 | x9) & (x1 | x10 | x3 | x5 | x6 | x8) & (x1 | x10 | x3 | x5 | x6 | x9) & (x1 | x10 | x3 | x5 | x7 | x8) & (x1 | x10 | x3 | x5 | x7 | x9) & (x1 | x11 | x2 | x4 | x6 | x8) & (x1 | x11 | x2 | x4 | x6 | x9) & (x1 | x11 | x2 | x4 | x7 | x8) & (x1 | x11 | x2 | x4 | x7 | x9) & (x1 | x11 | x2 | x5 | x6 | x8) & (x1 | x11 | x2 | x5 | x6 | x9) & (x1 | x11 | x2 | x5 | x7 | x8) & (x1 | x11 | x2 | x5 | x7 | x9) & (x1 | x11 | x3 | x4 | x6 | x8) & (x1 | x11 | x3 | x4 | x6 | x9) & (x1 | x11 | x3 | x4 | x7 | x8) & (x1 | x11 | x3 | x4 | x7 | x9) & (x1 | x11 | x3 | x5 | x6 | x8) & (x1 | x11 | x3 | x5 | x6 | x9) & (x1 | x11 | x3 | x5 | x7 | x8) & (x1 | x11 | x3 | x5 | x7 | x9)

```python

```


### Prof. Dr. -Ing. Gerald Schuller
 Jupyter Notebook: Renato Profeta

# Allpass Filters

```python
%%html

```

So far we specified the magnitude of our frequency response and didn't care much about the phase. For allpass filters, it is basically the other way around.

In the beginning of filter design, we saw that we can write a transfer function as:

$$\large
H(e^{j\Omega})=e^{j\phi\Omega}\cdot A(e^{j \Omega})$$

Here we specify, or rather, alter the phase, and keep the the magnitude of our frequency response at constant 1, meaning

$$A(e^{j\Omega})=1 $$

Hence we would like to have a filter with transfer function H of magnitude constant 1,

$$\mid H(e^{j\Omega})\mid=1$$

This also means: The magnitude of our z-trnasform is **1 on the unit circle in complex z-plane!**

```python
%%html

```

## Allpass Filter as Fractional Delay

```python
%%html

```

We can use such a filter to implement a “**fractional” delay** $d$, where d is a real number, for instance $d=0.5$ if we want to delay a signal by half a sample. In this case we cannot simply use a memory element as in the case of integer delays. In lecture 8 we saw that in this case, our ideal frequency response in the Discrete Time Fourier Transform (DTFT) domain is

$$\large
H(e^{j\Omega})=e^{-j\Omega \cdot d}
$$

To obtain its **ideal impulse response** for a filter design, we apply the inverse DTFT, also as seen in lecture 8,

$$\large
h(n)= \frac{1 } {2 \pi} \cdot \int_{\Omega=-\pi} ^ \pi H(e^{j\Omega}) e^{j \Omega n} d \Omega =
$$

$$\large
h(n)= \frac{1} {2 \pi} \cdot \int _ {\Omega=-\pi} ^ \pi e^{-j \Omega d} e^{j \Omega n} d \Omega =
$$

$$\large
= \frac{1} {2 \pi} \cdot \int _ {\Omega=-\pi} ^ \pi e^{j \Omega(-d+n)}  d \Omega =
$$

$$\large
 = \frac{1} {2 \pi} \cdot \left( \frac{1} {j \cdot (-d+n)} e^{j \pi (-d+n)} - \frac{1} {j \cdot (-d+n)} e^{-j \pi (-d+n)}   \right) =
$$

using $e^{j \cdot x}-e^{-j \cdot x}=2 \cdot j \cdot sin(x)$ we obtain:

$$\large
=\frac{1} { 2 \cdot \pi \cdot j \cdot (-d+n) } \cdot 2 \cdot j \cdot sin( \pi \cdot (-d+n) )=
$$

$$\large
=\frac{1}  { \pi  \cdot (-d+n) }  \cdot sin( \pi \cdot (-d+n) )=
$$

$$\large
sinc( -d+n )$$

This means our ideal impulse response is simply a **sinc function**, shifted by d! For the case of an integer delay d this degenerates to a 1 at the d’th position of our impulse response, and zeros elsewhere. To make this an FIR filter, we need to apply a “**window function**”, as described in the lecture “Multirate Signal Processing”.

**Example:** The window function is a sine window:

$w(n)=sin(\frac{\pi} { L} \cdot ( n+0.5))$, with n=0..L-1, and L: length of the window. This avoids having sudden ends of the impulse response. We also want to keep the most significant parts of our sinc function, the keep the resulting error small. The samples we drop should be small. To include some of the significant values of the sinc function at negative n, we need to shift it to positive indices (by $n_0$) to make it causal. In this way, we obtain our impulse response,

$$\large
h(n)=sinc(-d+n-n_0) \cdot w(n)$$

**Python example:**

Take $L=10, n_0=4$

```python
%%html

```

```python
%matplotlib inline
from sympy import init_printing
init_printing()
import numpy as np
import matplotlib.pyplot as plt

#time index range:
L=10
n=np.arange(L)
print('n:',n)

w=np.sin(np.pi/L*(n+0.5))
plt.figure()
plt.plot(w)
plt.plot(np.sinc(-0.5+n-4))

h=w*np.sinc(-0.5+n-4)

plt.plot(h)
plt.legend(('Sine Window', 'Sinc Function', 'Windowed SincFunction'),bbox_to_anchor=(1.04,1), loc="upper left")
plt.title('The Shifted Windowed Sinc Function')
plt.grid()

plt.figure()
x=np.hstack((np.arange(4),np.zeros(8)))
plt.plot(x)

import scipy.signal as sp
y=sp.lfilter(h,1,x)
plt.plot(y)
plt.legend(('Original Signal','Shifted signal, by 4.5 samples'),bbox_to_anchor=(1.04,1), loc="upper left")
plt.title('The Test Signal, Shifted by our Filter by 4.5 Samples')
plt.grid()

#Test with longer filter, to reduce errors:
plt.figure()
L=20
n=np.arange(L)
w=np.sin(np.pi/L*(n+0.5))
plt.plot(np.sinc(-0.5+n-9))
h=w*np.sinc(-0.5+n-9)
plt.plot(h)
plt.legend(('The Shifted Sinc Function','The Shifted Windowed SincFunction'),bbox_to_anchor=(1.04,1), loc="upper left")
plt.title('The Shifted Sinc Function')
plt.grid()

plt.figure()
x=np.hstack((np.arange(4),np.zeros(12)))
y=sp.lfilter(h,1,x)
plt.plot(x)
plt.plot(y)
plt.legend(('Shifted signal, by 9.5 samples', 'Original Signal'),bbox_to_anchor=(1.04,1), loc="upper left")
plt.title('The Test Signal, Shifted by our Filter by 9.5 Samples')
plt.grid()
```

**Observe:** The longer we make the filter, keeping the main
lobe of the sinc function in the center, the smaller the
resulting error for the shifted signal x becomes, but the
hight of the ripples remains similar.

```python
%%html

```

```python
from scipy.signal import freqz
import numpy as np
from matplotlib.ticker import FuncFormatter, MultipleLocator

w, h_response  = freqz(h)

fig=plt.figure(figsize=(10,8))
fig.suptitle('Frequency Response', fontsize=16)
plt.subplot(2,1,1)
plt.plot(w, 20 * np.log10(abs(h_response)), 'b')
plt.ylabel('Amplitude (dB)')
plt.xlabel('Normalized Frequency')
plt.grid()
'''ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.2))'''
plt.subplot(2,1,2)
angles = np.angle(h_response)
plt.plot(w, angles, 'g')
'''plt.ylabel('Angle (radians)')
plt.xlabel('Normalized Frequency')
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.2))'''
plt.grid()

```

**Observe** the phase plot at normalized frequency.

Remember: Angle= -d Omega, for a delay d,
hence d= -Angle/Omega, the negative slope.

Also observe that the magnitude deviates from 0 dB at
high frequencies at about above normalized frequency 2.8.

To estimate the obtained delay (for verification), we use
the mouse, got to y=-3.14, which is the first wrap around
point, and read out position x=0.32. From that we obtain
the delay d= -y/x=3.14/0.32=9.8, which is indeed roughly
the desired 9.5 samples delay. Observe that our calculation
from the plot is only a rough estimation.

This gives us a **tool for fractional delays!**
But we have this attenuation at high high frequencies.
IIR filters can improve our filters performance.

```python
# Delay:
peaks, _ = sp.find_peaks(-angles, height=0)
-angles[peaks[0]]/w[peaks[0]]

```

## IIR Fractional Delay Filter Design

In “I. Senesnick, “Low-pass filters realizable as allpass
sums: design via a new flat delay filter,” in
IEEE Transactions on Circuits and Systems II:
Analog and Digital Signal Processing, vol. 46,
1999” we found a way to design fractional delay IIR allpass
filters. https://ieeexplore.ieee.org/document/749080

```python
%%html

```

```python
%%html

```

As a Python function it is,

```python
def allp_delayfilt(tau):

    '''
    produces a Fractional-delay All-pass Filter
    Arguments: tau: fractional delay in samples (float).
    :returns:
    a: Denominator of the transfer function
    b: Numerator of the transfer function
    '''

    L = int(tau)+1
    n = np.arange(0,L)
    a_0 = np.array([1.0])
    a = np.array(np.cumprod( np.divide(np.multiply((L -
    n), (L-n-tau)) , (np.multiply((n+1), (n+1+tau))))))
    a = np.append(a_0, a) # Denominator of the transfer function
    b = np.flipud(a) # Numerator of the transfer function
    return a, b
```

```python
#testing the fractional delay allpass filter
import matplotlib.pyplot as plt
import scipy.signal as sp
#fractional delay of 5.5 samples:
a,b=allp_delayfilt(5.5)
x=np.hstack((np.arange(4),np.zeros(8)))
y=sp.lfilter(b,a,x) #applying the allpass filter

plt.figure(figsize=(10,8))
plt.title('The IIR Fractional Delay Filter Result')
plt.xlabel('Samples')
plt.plot(x, label='Original Signal')
plt.plot(y, label='Delayed Signal')
plt.grid()
plt.legend(bbox_to_anchor=(1.04,1), loc="upper left");
```

**Observe** that the extrapolated zero crossing of the orange
delayed signal would indeed be at around 5.5 samples.
Also observe that particularly before our signal we now
have **fewer ripples**.

```python
# Frequency Response

from scipy.signal import freqz
import numpy as np
from matplotlib.ticker import FuncFormatter, MultipleLocator

w, h_response  = freqz(b,a)

fig=plt.figure(figsize=(10,8))
fig.suptitle('Frequency Response', fontsize=16)
plt.subplot(2,1,1)
plt.plot(w/np.pi, 20 * np.log10(abs(h_response)), 'b')
#plt.ylim((-0.001,0.001))
plt.ylabel('Amplitude (dB)')
plt.xlabel('Normalized Frequency')
plt.grid()
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.2))
plt.subplot(2,1,2)
angles = np.angle(h_response)
plt.plot(w/np.pi, angles, 'g')
plt.ylabel('Angle (radians)')
plt.xlabel('Normalized Frequency')
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.2))
plt.grid()

```

 Now we obtain a nice impulse response or set of coefficients hmin and its frequencies response is:

**Observe** that the magnitude is practically precisely at 0 dB,
except for rounding errors, meaning a gain factor of 1.

**Observe** the phase plot at normalized frequency, put the
mouse over Omega=0.5. The angle is -2.8 radiants.
Remember: Angle= -d Omega, for a delay d,
hence d= -Angle/Omega = 2.8/0.5=5.6.

This fits nicely to our desired delay of 5.5 samples!

```python
# Delays
peaks, _ = sp.find_peaks(-angles, height=0)
-angles[peaks[0]]/w[peaks[0]]

```

Application example: Stereo source separation, with 2
sources and 2 microphones. We want to cancel one of the
2 sources, and for that we need to model the precise delay
from one microphone to the other to be able to remove
the signal by subtraction. See e.g.:

    - Oleg Golokolenko and Gerald Schuller:
    "FAST TIME DOMAIN STEREO AUDIO SOURCE SEPARATION USING FRACTIONAL DELAY FILTERS", 147th AES Convention,
    October, 2019, 16 – 19, New York, NY, USA
    - Oleg Golokolenko, Gerald Schuller: "A FAST STEREO AUDIO SOURCE SEPARATION FOR MOVING SOURCES",
    Asilomar Conference on Signals, Systems, and Computers, Nov 3-6, 2019, Asilomar, CA, USA

## Simple IIR Allpass Filters

The simplest allpass filter has one pole and one zero in the z-domain for the transfer function,

$$\large
H_{ap}(z)=\frac{z^{-1} -\bar{ a} }  {1 -a z^{-1}}=\frac{-\bar{ a} (1- \frac{z^{-1} } {\bar {a}}) }  {1 -a z^{-1}}
$$

where $a$ is a complex number, and $\bar{a}$ specifies the conjugate complex number.

Observe that here we have a zero at $z=\frac{1}{a}$ and a pole at $z=a$! **The pole and the zero are at conjugate reverse locations!**

```python
%%html

```

**Example:** If a=0.5, we obtain the pole/zero plot with Python,

```python
def zplane(B,A):
    # Plot Poles and Zeros
    from  matplotlib import patches
    plt.figure(figsize=(10,8))
    ax = plt.subplot(111)
    r = 2.5; plt.axis('scaled'); plt.axis([-r, r, -r, r])
    #ticks = [-1, 1]; plt.xticks(ticks); plt.yticks(ticks)
    # Unit Circle
    uc = patches.Circle((0,0), radius=1, fill=False, color='black', ls='dashed')
    ax.add_patch(uc)
    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)
    plt.xlabel('Re', horizontalalignment='right', x=1.0)
    plt.ylabel('Im',  y=1.0)
    plt.title('Complex z-Plane', loc='right')
    plt.grid()
    plt.plot(np.real(A),np.imag(A),'rx')
    plt.plot(np.real(B),np.imag(B),'bo')
```

```python
a=0.5; #Pole location
B=[1/a.conjugate()]; #the zero location
A=[a];
zplane(B,A); #plot the pole/zero diagram with axis limits

```

In this plot, the cross at 0.5 is the pole, and the circle at 2 is the zero.

How can we see that the magnitude of the frequency response $H(e^{j \cdot \Omega})$ is 1? We can re-write it as

$$\large
H_{ap}(e^{j \Omega})=\frac{e^{-j \Omega} -\bar{ a}} {1 -a e^{-j \Omega}}=e^{-j \Omega}\frac{1 -\bar {a} e^{j \Omega}}  {1 -a e^{-j \Omega}}
$$

Here you can see that the expression in the numerator is the **conjugate complex** of the denominator, hence their **magnitude** cancels to **one**. The exponential before the fraction also has magnitude 1, hence the entire expression has magnitude 1,

$$\large
\mid{H_{ap}(e^{j}\Omega)}\mid=1
$$

Here we can see, using just 1 pole and one zero, we can obtain a magnitude of constant 1.

More interesting now is the resulting phase. The phase function can be found in the book Oppenheim/Schafer, “Discrete Time Signal Processing”:

$$\large
\begin{equation}
\phi(\Omega)=- \Omega - 2 arctan \left(\frac{r sin(\Omega-\theta)} {1 - r cos(\Omega-\theta)} \right)
\end{equation}
$$

where r is the magnitude of a and $\theta$ is the phase angle of a (hence $a=r\cdot e^{j \theta}$).

Observe that so far we assumed the phase to be linearly dependent on the frequency ($\phi(\Omega=-\Omega \cdot d)$), and here we see it to be quite non-linear, with the trigonometric functions!

We can now plot the resulting phase over the normalized frequency, and compare it with the phase of a delay of 1 sample (of $z^{-1}$), where we get $\phi(\Omega)=-\Omega$. This can be seen in the following plot, for $a=0.5$ and $a=-0.5$:

```python
%%html

```

```python
# For Google Colab
import sys
if 'google.colab' in sys.modules:
    !pip install matplotlib==3.1.3
```

    Requirement already satisfied: matplotlib==3.1.3 in /usr/local/lib/python3.7/dist-packages (3.1.3)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.1.3) (0.11.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.1.3) (1.3.2)
    Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.1.3) (1.19.5)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.1.3) (2.8.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib==3.1.3) (3.0.6)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib==3.1.3) (1.15.0)

```python
from sympy import symbols, I , E, conjugate, Abs
from sympy.functions import arg
from sympy.plotting import plot

a = symbols('a', complex=True)
omega, r  = symbols('\Omega r', real=True)

Hap=(E**(-I*omega)-conjugate(a))/(1-a*E**(-I*omega))
print('Hap=')
display(Hap)
print('Magnitude=')
display(Abs(Hap))
print('For a = 0.5, |Hap| =')
display(Abs(Hap.subs(a,-0.5)))
print('Phase:')
phase=arg(Hap.subs(a,r*E**(I*omega)))

p = plot(arg(Hap.subs(a,-0.5)),arg(Hap.subs(a,0)),arg(Hap.subs(a,0.5)), (omega,0,np.pi), show = False, legend = True)
p[0].line_color = 'green'
p[0].label = 'a=- 0.5'
p[1].line_color = 'red'
p[1].label = 'a=0'
p[2].line_color = 'blue'
p[2].label = 'a=0.5'
p.show()

```

Here, the blue line is the allpass phase for a=0.5, the green line for a=-0.5, and the red line is for a=0, the phase of a pure 1 sample delay $z^{-1}$. Here it can be seen that the beginning and end of the curves are identical (at frequencies 0 and pi), and only in between the allpass phase deviates from the 1 sample delay! For a=0 the allpass indeed becomes identical to $z^{-1}$, a delay of 1 sample. So we can see that it behaves very **similar to a delay**.

Such an IIR allpass filter can also be used to implement **fractional delays**. In the above plot, observe the different **slopes** or **derivatives** of the phase function with respect to frequency, which shows different **group delays** for different coefficients a. For instance the green curve for a=-0.5 at frequencies below about half Nyquist frequency has a slope which is about a third of the slope of the red curve for a delay of 1 sample, and hence represents a fractional delay.

This application is described more detailed e.g. in:
T. I. Laakso, V. Välimäki, M. Karjalainen, and U. K. Laine.
Splitting the unit delay. IEEE Signal
Processing Magazine, 13(1):30–60, January 1996.

```python
%%html

```

## Frequency Warping

```python
%%html

```

For a=0.5 (the blue curve) we observe a **“streching”** of the low frequency range, where a small part of the low frequencies strech over a wider phase range than the other curves.

For the frequency warping we will interpret the negative phase as a normalized frequency.

The plot can also be produced with a simple Python function for the phase function:

```python
def warpingphase(w, a):
    #produces (outputs) phase wy for an allpass filter
    #w: input vector of normlized frequencies (0..pi)
    #a: allpass coefficient
    #phase of allpass zero/pole :
    theta = np.angle(a);
    #magnitude of allpass zero/pole :
    r = np.abs(a);
    wy = -w-2*np.arctan((r*np.sin(w-theta))/(1-r*np.cos(w-theta)));

    return wy
```

```python
w=np.arange(0,3.14,0.01)
plt.figure()
plt.plot(w,warpingphase(w,0.5))
plt.plot(w,warpingphase(w,0))
plt.plot(w,warpingphase(w,-0.5))
plt.grid()
plt.xlabel('Normalized Frequency')
plt.ylabel('Phase Angle');
```

The **phase** at the output of our phase function can also be **interpreted as a normalized frequency**. This means its **output** can be the **input** of another warpingphase function.

An interesting observation is, that the warpingphase function with coefficient $-\bar{a}$ is the inverse of the warpingphase function with coefficient $a$!

We can try this in Python:

```python
import matplotlib.pyplot as plt
#from warpingphase import *
#frequency range:
w = np.arange(0,np.pi, 0.01)
a = 0.5 * (1+1j)
wyy = (warpingphase(warpingphase(w,a), -a.conjugate()))
plt.figure()
plt.plot(w,wyy)
plt.xlabel('Normalized Frequency')
plt.ylabel('Phase Angle')
plt.show()
```

Here we see that it is indeed the **identity** function. This shows that interpreting the allpass as a normalized frequency “warper”, the allpass with coefficient $a$ is inverse to the allpass with $-\bar{a}$.

What is the frequency response of an example allpass filter? For $a=0.5$, we can use ```freqz```.

There we expect to see the magnitude to be contant 1, and the phase angle like we saw from our phase formula for Oppenheim, Schafer, (eq. 1). This also allows “cross-checking” the results!

Looking at the z-transform:
    $$\large
    H_{ap}(z)=\frac{z^{-1} -\bar{ a} } {1 -a z^{-1}}
    $$

we get our coefficient vectors to

`a=0.5;
B=[-a.conjugate(), 1];
A=[1, -a];`

(observe that for freqz the higher exponents of $z^{-1}$ appear to the right)

Now we plot the frequency response and impulse response:

```python
%%html

```

```python
a=0.5;
B=[-a.conjugate(), 1];
A=[1, -a];

# Frequency Response

from scipy.signal import freqz
import numpy as np
from matplotlib.ticker import FuncFormatter, MultipleLocator

w, h_response  = freqz(B,A)

fig=plt.figure(figsize=(10,8))
fig.suptitle('Frequency Response', fontsize=16)
plt.subplot(2,1,1)
plt.plot(w/np.pi, 20 * np.log10(abs(h_response)), 'b')
#plt.ylim((-0.001,0.001))
plt.ylabel('Amplitude (dB)')
plt.xlabel('Normalized Frequency')
plt.grid()
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.2))
plt.subplot(2,1,2)
angles = np.angle(h_response)
plt.plot(w/np.pi, angles, 'g')
plt.ylabel('Angle (radians)')
plt.xlabel('Normalized Frequency')
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.2))
plt.grid()

```

 Here we can see in the above plot of the magnitude, that we indeed obtain a constant 1 (which is 0 dB, 2e-15 comes from the finite accuracy and rounding errors), and that we have the **non-linear** phase in the lower plot, as in the phase plots before.

To obtain the impulse response, we can use the function “lfilter”, and input a unit impulse into it.

```python
from scipy import signal as sp
Imp = np.zeros(21)
Imp[0] = 1
h = sp.lfilter(B, A, Imp)
plt.figure(figsize=(8,6))
plt.plot(h)
plt.title('Impulse Response of our IIR Allpass with a=0.5')
plt.xlabel('Sample')
plt.ylabel('Value')
plt.grid()
```

Here we can see that we have the first, non-delayed, sample not at zero, but at -0.5. This can also be seen by plotting the first 4 elements of our impulse response:

`
print h[0:4]
ans =
[-0.5     0.75    0.375   0.1875]`

The second element corresponds to the delay of 1 sample, our $z^{-1}$, with a factor of 0.75. But then there are more samples, going back into the past, exponentially decaying.

This means, not only the past samples goes into our filtering calculation, but also more past samples, and even the **non-delayed** sample, with a factor of -0.5. This is actually a problem for the so-called frequency warping (next section), if we want to use frequency warping in IIR filters, because here we would get delay-less loops, which are difficult to implement! (With **FIR filters** this is **no problem** though)

# Frequency Warping

These properties of the allpass can now be used to “warp” the frequency scale of a filter (by effectively replacing $e^{j\Omega} \leftarrow e^{j\phi (\Omega)}$ in our frequency response), for instance to map it according to the so-called **Bark scale**, used in psycho-acoustics.

A common approximation of the Bark scale is

$$\large
Bark=13 \cdot arctan(0.0076 \cdot f) + 3.5 \cdot arctan\left(\left(\dfrac{f}{7500}\right)^2\right)
$$

(From Wikipedia, Bark scale, the approximation goes back to Zwicker and Terhard), where f is the frequency in Hz. The Bark scale can be seen as an approximation of the changing frequency resolution over frequency of the inner ear filters of the human ears cochlea.

Because of the structure of our chochlea, the ear has different sensitivites for different frequencies and different signals. The signal dependent threshold of audibility of the ear is called the **Masking Threshold**. It has more spectral detail at lower than at higher frequencies, according to the Bark scale.

We can plot the Bark formula using the python:

```python
%%html

```

```python
#Frequency array between 0 and 20000 Hz in 1000 steps:
f=np.linspace(0,20000,1000)
#Computation of Zwickers Bark approximation formula:
z=13*np.arctan(0.00076*f)+3.5*np.arctan((f/7500.0)**2)
#plot Bark over Hertz:
plt.figure(figsize=(8,6))
plt.plot(f,z)
plt.xlabel('Frequency in Hertz')
plt.ylabel('Frequency in Bark')
plt.title('Zwicker&Terhard Approximation')
plt.grid()
```

Here we can see, that 1 bark at lower frequency has a much lower bandwidth than at higher frequencies. This means the ear can be seen as having a higher frequency resolution at lower frequencies than at higher frequencies. Imagine, we want to **design a filter** or system for **hearing purposes**, for instance, we would like to model the masking threshold of the ear for any given signal by some linear filter (FIR or IIR).

Then it would be useful, to give this filter a **higher frequency resolution at lower frequencies**, such that it matches the smaller details of the **masking threshold** at lower frequencies.  But if we look at the **usual design methods**, they distribute the filter **details independent of the frequency range** (for instance what we saw with the remez method, where we have equally distributed ripples).

Here we can now use frequency warping, such that we **enlarge the low frequency range** and shrink the high frequency range accordingly, such that our filter now works on the **warped frequency**, and **“sees”** the **lower frequencies in more detail**, the lower frequencies are more spread out in comparison to the higher frequencies.

**How do we do this?** For some frequency response $H(e^{j\Omega})$ we would like to warp the frequency $\Omega$ with some function $\phi(\Omega)$ according to our desired frequency scale, such that we get

$$\large
H(e^{j\cdot \phi (\Omega)})
$$

But this is exactly the principle of an **allpass filter**, which has the frequency response

$$\large
H_{ap}(e^{j \Omega})=e^{j \cdot \phi_{ap}(\Omega)}
$$

Usually we would like to map positive frequencies to again positive frequencies, and we saw that $\phi_{ap}(\Omega)$ becomes negative, hence we take the approach to **replace z** in the argument of our transfer function with the reverse of our **allpass** transfer function:

$$\large
z^{-1}\leftarrow H_{ap}(a,z)$$

This is replacing all delays of our filter to be warped by our allpass filter.

In this way we replace our linear function on the unit circle in z with the non-linear, warped function on the unit circle $H_{ap}$.

Hence we get the warped transfer function as:

$$\large
H_{warped}(z)=H(H_{ap}( a,z)^{-1})$$

and the resulting frequency response becomes:

$$\large
H_{warped}(e^{j \Omega})=H(e^{-j \cdot \phi_{ap}(\Omega)})
$$

Here we can now see that we obtained the **desired frequency warping.**

What does this mean for the filter implementation? We know that our FIR  filters always consist of many delay elements $z^{-1}$.

```python
%%html

```

**Example:** Take an FIR filter,

$$\large
H(z)= {\sum _{m=0} ^ L b(m)  \cdot z^{-m}} $$

its warped version is:

$$\large
H(H_{ap}(a,z)^{-1})= {\sum_ {m=0} ^ L b(m)  \cdot H^{m}_{ap}(a,z)} $$

To obtain a desired filter, we now first have to **warp our desired filter**, and then **design** our filter in the **warped domain**.

Observe that the warping turns an **FIR filter into an IIR filter**, because the allpass has poles outside of zero.

An example of this kind of design can be seen in the following picture.

(From [1])

Here we can see that the 12th order filter successfully approximated the more detailed curve at low frequencies, using the warping approach.

    - [1] Julius O. Smith and Jonathan S. Abel, “Bark and ERB Bilinear Transforms,”
    IEEE Transactions on Speech and Audio Processing, vol. 7,
    no. 6, pp. 697 – 708, November 1999.
    - [2] S. Wabnik, G. Schuller, U. Kraemer, J. Hirschfeld:
    "Frequency Warping in Low Delay Audio Coding",
    IEEE International Conference on Acoustics, Speech, and Signal Processing,
    Philadelphia, PA, March 18–23, 2005

```python

```


```
from sympy import *
from ga import Ga
from printer import Format, Fmt
Format()
```

```
xyz_coords = (x, y, z) = symbols('x y z', real=True)
(o3d, ex, ey, ez) = Ga.build('e', g=[1, 1, 1], coords=xyz_coords, norm=True)
```

```
f = o3d.mv('f', 'scalar', f=True)
F = o3d.mv('F', 'vector', f=True)
lap = o3d.grad*o3d.grad
```

```
lap.Fmt(1,r'\nabla^{2}')
```

\begin{equation*} \nabla^{2} = \frac{\partial^{2}}{\partial x^{2}} + \frac{\partial^{2}}{\partial y^{2}} + \frac{\partial^{2}}{\partial z^{2}} \end{equation*}

```
(lap*f).Fmt(1,r'\nabla^{2}f')
```

\begin{equation*} \nabla^{2}f = \partial^{2}_{x} f  + \partial^{2}_{y} f  + \partial^{2}_{z} f  \end{equation*}

```
o3d.grad | (o3d.grad * f)
```

\begin{equation*} \partial^{2}_{x} f  + \partial^{2}_{y} f  + \partial^{2}_{z} f  \end{equation*}

```
o3d.grad|F
```

\begin{equation*} \partial_{x} F^{x}  + \partial_{y} F^{y}  + \partial_{z} F^{z}  \end{equation*}

```
o3d.grad * F
```

\begin{equation*} \left ( \partial_{x} F^{x}  + \partial_{y} F^{y}  + \partial_{z} F^{z} \right )  + \left ( - \partial_{y} F^{x}  + \partial_{x} F^{y} \right ) e_{x}\wedge e_{y} + \left ( - \partial_{z} F^{x}  + \partial_{x} F^{z} \right ) e_{x}\wedge e_{z} + \left ( - \partial_{z} F^{y}  + \partial_{y} F^{z} \right ) e_{y}\wedge e_{z} \end{equation*}

```
sph_coords = (r, th, phi) = symbols('r theta phi', real=True)
(sp3d, er, eth, ephi) = Ga.build('e', g=[1, r**2, r**2 * sin(th)**2], coords=sph_coords, norm=True)
```

```
f = sp3d.mv('f', 'scalar', f=True)
F = sp3d.mv('F', 'vector', f=True)
B = sp3d.mv('B', 'bivector', f=True)
lap = sp3d.grad*sp3d.grad
lap.Fmt(1,r'\nabla^{2}')
```

\begin{equation*} \nabla^{2} = \frac{2}{r} \frac{\partial}{\partial r} + \frac{\cos{\left (\theta  \right )}}{r^{2} \sin{\left (\theta  \right )}} \frac{\partial}{\partial \theta } + \frac{\partial^{2}}{\partial r^{2}} + r^{-2} \frac{\partial^{2}}{\partial \theta ^{2}} + \frac{1}{r^{2} \sin^{2}{\left (\theta  \right )}} \frac{\partial^{2}}{\partial \phi ^{2}} \end{equation*}

```
lap*f
```

\begin{equation*} \frac{1}{r^{2}} \left(r^{2} \partial^{2}_{r} f  + 2 r \partial_{r} f  + \partial^{2}_{\theta } f  + \frac{\partial_{\theta } f }{\tan{\left (\theta  \right )}} + \frac{\partial^{2}_{\phi } f }{\sin^{2}{\left (\theta  \right )}}\right) \end{equation*}

```
sp3d.grad | (sp3d.grad * f)
```

\begin{equation*} \frac{1}{r^{2}} \left(r^{2} \partial^{2}_{r} f  + 2 r \partial_{r} f  + \partial^{2}_{\theta } f  + \frac{\partial_{\theta } f }{\tan{\left (\theta  \right )}} + \frac{\partial^{2}_{\phi } f }{\sin^{2}{\left (\theta  \right )}}\right) \end{equation*}

```
sp3d.grad | F
```

\begin{equation*} \frac{1}{r} \left(r \partial_{r} F^{r}  + 2 F^{r}  + \frac{F^{\theta } }{\tan{\left (\theta  \right )}} + \partial_{\theta } F^{\theta }  + \frac{\partial_{\phi } F^{\phi } }{\sin{\left (\theta  \right )}}\right) \end{equation*}

```
sp3d.grad ^ F
```

\begin{equation*} \frac{1}{r} \left(r \partial_{r} F^{\theta }  + F^{\theta }  - \partial_{\theta } F^{r} \right) e_{r}\wedge e_{\theta } + \frac{1}{r} \left(r \partial_{r} F^{\phi }  + F^{\phi }  - \frac{\partial_{\phi } F^{r} }{\sin{\left (\theta  \right )}}\right) e_{r}\wedge e_{\phi } + \frac{1}{r} \left(\frac{F^{\phi } }{\tan{\left (\theta  \right )}} + \partial_{\theta } F^{\phi }  - \frac{\partial_{\phi } F^{\theta } }{\sin{\left (\theta  \right )}}\right) e_{\theta }\wedge e_{\phi } \end{equation*}

```
(sp3d.grad | B).Fmt(3)
```

 \begin{align*}  & - \frac{1}{r} \left(\frac{B^{r\theta } }{\tan{\left (\theta  \right )}} + \partial_{\theta } B^{r\theta }  + \frac{\partial_{\phi } B^{r\phi } }{\sin{\left (\theta  \right )}}\right) e_{r} \\  &  + \frac{1}{r} \left(r \partial_{r} B^{r\theta }  + B^{r\theta }  - \frac{\partial_{\phi } B^{\phi \phi } }{\sin{\left (\theta  \right )}}\right) e_{\theta } \\  &  + \frac{1}{r} \left(r \partial_{r} B^{r\phi }  + B^{r\phi }  + \partial_{\theta } B^{\phi \phi } \right) e_{\phi }  \end{align*}

```
Fmt([o3d.grad,o3d.grad],1)
```

\begin{equation*}  \left [ \begin{array}{cc} e_{x} \frac{\partial}{\partial x} + e_{y} \frac{\partial}{\partial y} + e_{z} \frac{\partial}{\partial z}, & e_{x} \frac{\partial}{\partial x} + e_{y} \frac{\partial}{\partial y} + e_{z} \frac{\partial}{\partial z}\\ \end{array} \right ]
\end{equation*}

```
F.Fmt(1)
```

\begin{equation*} F = F^{r}  e_{r} + F^{\theta }  e_{\theta } + F^{\phi }  e_{\phi } \end{equation*}

```
Fmt((F,F))
```

\begin{equation*} \begin{array}{c} \left ( F^{r}  e_{r} + F^{\theta }  e_{\theta } + F^{\phi }  e_{\phi }, \right. \\  \left. F^{r}  e_{r} + F^{\theta }  e_{\theta } + F^{\phi }  e_{\phi }\right ) \\ \end{array}\end{equation*}

```

```


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""

# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
a, F, EA = var("a, F, EA")
(l1, l2, l3) = (a, sqrt(2)*a, a)

S1 = - F
S2 = sqrt(2)*F
S3 = - F

(dl1, dl2, dl3) = ( l1/EA*S1, l2/EA*S2, l3/EA*S3 )

s22 = sqrt(2)/2
e1 = Matrix([1, 0])
e2 = Matrix([-s22, s22])
e3 = Matrix([0, 1])

u2x, u2y, u3y = var("u2x, u2y, u3y")
u1 = Matrix([0,0])
u2 = Matrix([u2x,u2y])
u3 = Matrix([0,u3y])

eq1 = Eq(dl1, e1.dot(u2-u1))
eq2 = Eq(dl2, e2.dot(u3-u2))
eq3 = Eq(dl3, e3.dot(u3-u1))

sol = solve([eq1, eq2, eq3],[u2x, u2y, u3y])

(u2x, u2y, u3y) = (sol[u2x], sol[u2y], sol[u3y])

pprint("\nClassical Solution:")
pprint("u2x, u2y, u3y / ( Fa / EA ):")
for u in (u2x, u2y, u3y):
    u*=EA/F/a
    pprint(u)

# FEM-Solution:
EA, a = var("EA, a")
# psi:
p = sqrt(2)/4

u2x, u2y, u3y = var("u2x, u2y, u3y")

K = Matrix((
    [p+1, -p,  p ],
    [ -p,  p, -p ],
    [  p, -p, p+1]
    ))
K*=EA/a
u = Matrix([u2x, u2y, u3y])
f = Matrix([0,-F,0])
eq = Eq( K*u, f )
sol = solve(eq,[u2x, u2y, u3y])
(u2x, u2y, u3y) = (sol[u2x], sol[u2y], sol[u3y])

pprint("\nFEM Solution:")
pprint("u2x, u2y, u3y / (Fa/EA):")
for u in (u2x, u2y, u3y):
    u*=EA/F/a
    pprint(u)

F1x, F1y, F3x = var("F1x, F1y, F3x")

f = EA/a
eq1 = Eq( f * (-u2x), F1x)
eq2 = Eq( f * (-u3y), F1y)
eq3 = Eq( f * p * ( -u2x + u2y - u3y ), F3x)

sol = solve([eq1, eq2, eq3], [F1x, F1y, F3x])

pprint("\n")
pprint(sol)

pprint("\nS1, S2, S3:")
f = sqrt(2)/2
dl1 = u2x
dl2 = f * (u2x + u3y - u2y)
dl3 = u3y
S1 = EA *dl1/l1
S2 = EA *dl2/l2
S3 = EA *dl3/l3
for S in [S1, S2, S3]:
    S = S.simplify()
    pprint(S)

# Classical Solution:
# u2x, u2y, u3y / ( Fa / EA ):
# -1
# -2⋅√2 - 2
# -1
#
# FEM Solution:
# u2x, u2y, u3y / (Fa/EA):
# -1
# -2⋅√2 - 2
# -1
#
# {F1x: F, F1y: F, F3x: -F}
#
# S1, S2, S3:
# -F
# √2⋅F
# -F

```