# メモ

1. python の sympy という数式処理のシステムを勉強する。
1. sympy で狙っているのは、数式レベルでシンボルを使った計算だが、数学の世界は広く、アプローチもさまざまなのでなんでもできるわけではない。
1. 一方、latex はどんな数式でも書ける。 latex を併用する。
1. 本家 SymPy Tutorial  (https://docs.sympy.org/latest/tutorial/index.html#tutorial)
1. 入力例で学ぶPython(SymPy)の使い方(入門) (https://pianofisica.hatenablog.com/entry/2019/04/04/183712)
1. Doing Math With Python PDF ( http://index-of.es/Varios-2/Doing%20Math%20with%20Python.pdf)
1. Doing Math With Python site (http://www.nostarch.com/doingmathwithpython/ )

# はじめに

数学と言っても広い話で、調べながらいろんな話題に進んでしまうとは思うものの、とりあえず思いついたことを書いておくと、興味の対象がわかっていいと思うので書いておく。

1. 数式。数学において数式を数学の世界で普通なように書くことは大事だと思う。$x^2$を `x**2` と書くとき、何が違うかというとフォントが違う、記号が違う。いまは$ \LaTeX$ で書いた。
1. 数式演算。$x$ に数を代入することはプログラミングでできて、手続き的にいろいろ複雑な計算もできるが、数式を数式自体で操作して、概念を抽象化して考えることが数学の本質の一部である。
1. sympy と別に python には numpy モジュールがあって計算に便利に使われる。 必要に応じて使う。
1. 図は matplotlib を使う。

# 数列の合計 シグマ $\Sigma$

```python
# python で数の合計は次のように書く
sum ([12,34])
```

    46

```python
# sympy に数の合計の記号シグマがある
from sympy import *
from sympy.abc import *
Sum(k, (k, 1, m))
```

$\displaystyle \sum_{k=1}^{m} k$

```python
Sum(k, (k, 1, m)).doit()
```

$\displaystyle \frac{m^{2}}{2} + \frac{m}{2}$

```python
# 上記の出力を因数分解して次のようにしたい
# cf. factoring, expand, simplify
%%latex
\displaystyle
\frac{m(m+1)}{2}
```

\displaystyle

\frac{m(m+1)}{2}

```python
# 因数分解は factor を使う
factor(Sum(k, (k, 1, m)).doit())
```

```python
# 完成品
Eq(Sum(k, (k, 1, m)), factor(Sum(k, (k, 1, m)).doit()))
```

# SymPy の歩き方

例えば、数列の合計シグマ $\Sigma$ を sympy でどう書くか、そもそも書けるのか、名前はなんというのか, Sigma なのか Sum なのか、関数として使えるのかただの記号なのか、というとき。

`from sympy import *` して、 dir() を出力する。
出力をエディターにコピペして、それらしき命令を探す。
Sum というのがあった。
コードセルで Sum を実行すると `sympy.concrete.summations.Sum` と出力される。 これを google で検索するとドキュメントが見られる。
help(Sum) で help が見られる。

python の関数やメソッドは小文字 lower caracter で始まるのが原則なので、大文字で始まるのは sympy の出力のからんだ命令である。

あとは実験学習

```python
from sympy import *
Sum
```

    sympy.concrete.summations.Sum

# Abs を例に実験

```python
# Abs について実験
# sympy でシンボルに値が入っている場合、Abs は関数として働き、引数がシンボルの場合、数式処理システムとして働く
from sympy import *
from sympy.abc import *
display(Abs(x))
x = -3
display(Abs(x))
x = symbols('x')
display(Abs(x))
display(latex(Abs(x)))
```

```latex
%%latex
\left|{x}\right|
```

\left|{x}\right|

```python
# help(abs) によると、Abs がある場合、通常の組み込み関数 abs は Abs になる、とのことなので実験
x = symbols('x')
abs(x) # => \left|{x}\right|
```

$\displaystyle \left|{x}\right|$

# 積分記号 Integral と微分記号 Derivative

積分するは integrate 微分するは diff

```python
from sympy import *
init_printing()

x = symbols ('x')
a = Integral (cos(x) *  exp(x), x)
display(a)
display(a.doit())
display(Eq (a, (factor(a.doit()))))
```

今までの例からわかったこと。

1. `from sympy import *` でこれ以降なにもことわらずに四則演算をふくめ数学記号が sympy の演算子になっている。
1. `x = symbols('x')` は、python と共存するので、なんらかの宣言が必要なのはわかる。`x = 3`と入れたらそれ以降は `x` は 3 になって、また `x = symbols('x')` としたら `x` に戻った。 `from sympy.abc import *` という便利な道具もある。
3. 式で項の順序は維持されない。` Integral (cos(x) *  exp(x), x)` の `cos(x)`と`exp(x)`は数式表現になったときに順序が逆になっている。
4. `doit()`は数式レベルの計算というか、評価をするが、いま `((x + 2) * (x - 2)).doit()` としたら展開しなかった。メソッド `factor()`、`expand()` などを使う。

```python
from sympy import *
init_printing()

x = symbols ('x')
((x + 2) * (x - 2)).expand()
```

```python
((x + 2) * (x - 2))
```

```python
expand((x + 2) * (x - 2))
```

```python
from sympy import *
init_printing()
x = symbols ('x')
factor(x**2 - 4)
```

```python
integrate (cos(x) *  exp(x), x)
```

# sympy の出力は必ずしも理想的ではないので、latex で整形する

```python
from sympy import *
init_printing()
x = symbols ('x')
a = Integral (cos(x) *  exp(x), x)
display(Eq (a, a.doit()))
latex(Eq (a, factor(a.doit())))
```

```latex
%%latex
\displaystyle
\int e^{x} \cos{\left(x \right)}\, dx = \frac{e^{x} \sin{\left(x \right)}}{2} + \frac{e^{x} \cos{\left(x \right)}}{2}  \\
\int{e^x \cos (x)}dx = \frac{e^x}{2} \sin(x) +  \frac{e^x}{2} \cos(x) \\
\int e^{x} \cos{\left(x \right)}\, dx = \frac{\left(\sin{\left(x \right)} + \cos{\left(x \right)}\right) e^{x}}{2} \\
\int e^{x} \cos{\left(x \right)}\, dx = \frac{e^{x}}{2}\left(\sin{\left(x \right)} + \cos{\left(x \right)}\right)
```

\displaystyle
\int e^{x} \cos{\left(x \right)}\, dx = \frac{e^{x} \sin{\left(x \right)}}{2} + \frac{e^{x} \cos{\left(x \right)}}{2}  \\
\int{e^x \cos (x)}dx = \frac{e^x}{2} \sin(x) +  \frac{e^x}{2} \cos(x) \\
\int e^{x} \cos{\left(x \right)}\, dx = \frac{\left(\sin{\left(x \right)} + \cos{\left(x \right)}\right) e^{x}}{2} \\
\int e^{x} \cos{\left(x \right)}\, dx = \frac{e^{x}}{2}\left(\sin{\left(x \right)} + \cos{\left(x \right)}\right)

# いまここ

`doit()`のヘルプを読んで見る。

```python
# `doit()`のヘルプを読んで見る。
from sympy import *
x = symbols('x')
help(x.doit)
```

```python
# doit() の使い方。 doit(a) ではダメ。doit(deep=False) とかに使う。
from sympy import *
init_printing()
x = symbols ('x')
a = 2 * Integral (x, x)
display(a)
print()
display(a.doit())
print()
display(Eq(a, a.doit()))
```

`init_printing()`とすると、アウトプットが数式表示になる。

# 微分

微分の実験

```python
# Derivative 微分の実験
from sympy import *
from sympy.abc import *
init_printing()
expr = x**2
display(Eq(Derivative(expr), expr.diff()))
print()
display(Eq(Derivative(expr), diff(expr)))
print()
display(Eq(Derivative(expr), (Derivative(expr)).doit()))
```

```python
Eq(Derivative(exp(x)), diff(exp(x)))
```

# 因数分解と式の展開

`factor()` と `expand()` を使ってみる。

```python
from sympy import *
from sympy.abc import *
init_printing()
a = (x + 2) * (x - 2)
b = (x**2 - 4)
c = (x**2 - 4) / (x - 2)
display(a, b, c)
```

```python
factor((x**2 - 4))
```

```python
display(factor (b))
display(factor (c))
```

```python
display(expand (a))
display(expand (b))
display(expand (c))
```

```python
d = (x - 2)
b / d
```

式の因数分解や展開が全部自動化できるわけではない。

あたりまえだね。

```python
# 多項式はそれなりに並べ替えたり整理はしてくれる
display(x**2 + y**3 + x + x**3 + x**2 + y**4)
display(x * (x+1) ** 2 + y**3 *(y + 1))
display(expand(x * (x+1) ** 2 + y**3 *(y + 1)))

```

Rational() とか

```python
# Rational() とか
from sympy import *
Rational (3, 2)
```

```python
Rational(8, 4)
```

```python
sqrt(8)
```

数式のシンボルを事前に宣言する場合。

```python
# シンボルについて考察、実験
from sympy import *
x, y, z = symbols("x y z")
k, m, n = symbols("k m n", integer=True)
f, g, h = symbols('f g h', cls=Function)
```

意味は多分、`x, y, z` は任意の変数、`f, m, n` は整数、`f, g, h` は関数ということだと思うが、`i, j`がない。

```python
from sympy import *
from sympy.abc import *
init_printing()
display(alpha, beta, chi, delta, epsilon, eta, gamma, iota, kappa, lamda, mu, nu, omega, omicron, phi, pi, psi, rho, sigma, tau, theta, upsilon, xi, zeta)
```

Rational()の中で変数は使えなかった

```python
# Rational(x, y) # => TypeError
display(x / y)
```

数式の割り算とか

```python
(x**2 - 2 * x + 1) / (x - 1)
```

```python
factor((x**2 - 2 * x + 1) / (x - 1))
```

```python
factor ((x**2 - 2*x + 1)) / (x - 1)
```

```python
display(pi, i, e)
```

```python
pi.evalf()
```

```python
exp(1).evalf()
```

# ヘルプの求め方

`evalf()`の例では`evalf?`とか`evalf??`とかが簡単でよい。

`help(evalf)`もある。`help(x.evalf)`もある。

```python
evalf?
# Adaptive numerical evaluation of SymPy expressions, using mpmath
# for mathematical functions.
```

```python
evalf??
```

```python
exp(1).evalf()
```

```python
-oo.evalf()
```

```python
-oo
```

# 式の簡素化

`simplification`というのか。自動でやられるのはごく単純な場合だけ。

```python
from sympy import *
init_printing()
x, y = symbols('x y')
expr = x + 2 * y
display((x * expr))
display(expand (x * expr))
display(factor(expand (x * expr)))
```

```python
from sympy import *
init_printing()
x, t, z, nu = symbols ('x t z nu')

expr = sin (x) * exp(x)
display(Eq(Derivative(expr), diff(expr)))
```

```python
expr2=exp(x)*sin(x) + exp(x)*cos(x)
display(Eq(Integral(expr2), integrate (expr2)))
```

```python
display(integrate(sin(x**2), (x, -oo, oo)))
display(Integral(sin(x**2), (x, -oo, oo)))
# 次の式は実行すると終わらない
# display(Eq(Integral(sin(x**2), (x, -oo, oo)), integrate(sin(x**2), (x, -oo, oo))))
```

```python
# 極限 limit
limit (sin(x)/x, x, 0)
```

# いまここ

```python
# \sqrt 2 $ を100桁まで計算する
sqrt(2).evalf(100)
```

```python
pi.evalf(100)
```

```python
2**(1/2)
```

```python
val = 2**(1/2)
print("{0:.100f}".format(val))
```

    1.4142135623730951454746218587388284504413604736328125000000000000000000000000000000000000000000000000

```python
# これはエラーになる
# val = sqrt(2)
# print("{0:.100f}".format(val))
```

```python
val = 2**(1/2)
```

```python
# 2**(1/2)はfloatなのでevalf()が使えない。exp表示にする必要がある
exp(ln(2) / 2).evalf(100)
```

```python
###### 1/2+1/3を有理数として計算する
Rational(1,2)+Rational(1,3)
```

```python
x + x + y + x
```

```python
expand((x+y)**5)
```

###### 変数入りの数式はベキが大きいのから小さいのに並べられるのはエラい

```python
3*x*y**2 + 3*y*x**2 + x**3 + y**3
```

```python
expand(x + y, complex = True)
```

###### 上のは複素数表記でドイツ文字の$ R $と$ I $を使っている
フラクトゥールというらしい
$$
\mathfrak R (x) + \mathfrak R (y) + i \mathfrak I x + \mathfrak I y
$$
だいたい合っているかな。

```python
%%script false
'''
Unit converter: Miles and Kilometers
'''
def print_menu():
  print('1. Kilometers to Miles')
  print('2. Miles to Kilometers')

def km_miles():
  km = float(input('Enter distance in kilometers: '))
  miles = km / 1.609

  print('Distance in miles: {0}'.format(miles))

def miles_km():
  miles = float(input('Enter distance in miles: '))
  km = miles * 1.609
  print('Distance in kilometers: {0}'.format(km))

if __name__ == '__main__':
  print_menu()
  choice = input('Which conversion would you like to do?: ')
if choice == '1':
  km_miles()
if choice == '2':
  miles_km()
```

### if __name__ == '__main__':
はノートブックにそぐわないみたい。
取ってしまった方がいいみたい。

そんなことを言えば、input()もノートブックにそぐわないみたい。
input()があると評価がそのセルで止まってしまう。
なるほど。

```python
'''
Quadratic equation root calculator
'''
def roots(a, b, c):
  D = (b*b - 4*a*c)**0.5
  x_1 = (-b + D)/(2*a)
  x_2 = (-b - D)/(2*a)
  print('x1: {0}'.format(x_1))
  print('x2: {0}'.format(x_2))

a, b, c = 1, 2, 1
roots(float(a), float(b), float(c))
```

    x1: -1.0
    x2: -1.0

###### (x+y)6 の展開形を計算する。

```python
from sympy import *
init_printing()
x, y = symbols("x,y")

expand((x + y)**6)
```

```python
simplify(x**6 + 6*x**5*y)
```

###### python表記に戻す init_printing()にはどうしたらよいか
アイデアは
init_printing(use_pythonformat: True)
x**2
とか

init_printing(pretty_print=False)

でした。

```python
init_printing(pretty_print=False)
x**2
```

    x**2

```python
expand((x + y)**6)
```

    x**6 + 6*x**5*y + 15*x**4*y**2 + 20*x**3*y**3 + 15*x**2*y**4 + 6*x*y**5 + y**6

```python
# sin(x)/cos(x)を簡単化する
from sympy import *
init_printing()
x = symbols('x')
simplify(sin(x)/cos(x))
```

### simplifyの代替

簡単化とはいくぶん曖昧な用語のため、より目的を明確にした simplify の代替が存在する: powsimp (指数の簡単化), trigsimp (三角関数を含む数式), logcombine, radsimp, togeter.

```python
factor(x**2 + 2*x + 1)
```

```python
simplify(x**2 + 2*x + 2)
```

```python
trigsimp(sin(x)**4 - 2*cos(x)**2*sin(x)**2 + cos(x)**4)
```

```python
expand_trig(sin(x+y))
```

```python
trigsimp(sin(x)*cos(y)+sin(y)*cos(x))
```

```python
expand_trig(tan(2*x))
```

```python
sqrt(x*y)
```

```python
ln(x)
```

```python
x, y = symbols('x y')
expand_log(log(x*y))
```

```python
x, y = symbols('x y', positive=True)
expand_log(log(x*y))
```

```python
x, y = symbols('x y')
expand_log(log(x*y), force=True)
```

```python
logcombine(log(x) + log(y), force=True)
```

```python
n=symbols('n')
logcombine(n*ln(x),force=True)
```

```python
x,y,z=symbols('x y z')
k, m, n = symbols('k m n')
factorial(n)
```

```python
binomial(n,k)
```

```python
binomial(5,3)
```

```python
gamma(z)
```

```python
gamma(10)
```

```python
factorial(9)
```

これは`LaTeX`

$$
\Gamma(z) = \int_0^\infty t^{z - 1}e^{-t}\,dt
$$

```latex
%%latex
\displaystyle

\Gamma(z) = \int_0^\infty t^{z - 1}e^{-t},dt
```

\displaystyle

\Gamma(z) = \int_0^\infty t^{z - 1}e^{-t},dt

```python
# hyper([a_1, ..., a_p], [b_1, ..., b_q], z)
# hypergeometric function
hyper([1,2],[3],z)
```

$\displaystyle {{}_{2}F_{1}\left(\begin{matrix} 1, 2 \\ 3 \end{matrix}\middle| {z} \right)}$

```python
# rewrite
tan(x).rewrite(sin)
```

```python
factorial(x).rewrite(gamma)
# For some tips on applying more targeted rewriting, see the Advanced Expression Manipulation section
```

```python
expand_func(gamma(x + 3))
```

```python
hyperexpand(hyper([1,1],[2],z))
```

```python
expr = meijerg([[1],[1]], [[1],[]],-z)
expr
```

$\displaystyle {G_{2, 1}^{1, 1}\left(\begin{matrix} 1 & 1 \\1 &  \end{matrix} \middle| {- z} \right)}$

```python
hyperexpand(expr)
```

```python
n , k = symbols('n k', integer=True)
combsimp(factorial(n)/factorial(n - 3))
```

```python
combsimp(binomial(n+1, k+1)/binomial(n, k))
```

```python
gammasimp(gamma(x)*gamma(1 - x))
```

### sympyのtutorial

```python
from sympy import *
init_printing()
x = symbols('x')
a = Integral(cos(x)*exp(x),x)
Eq(a, a.doit())
```

```python
a.doit()
```

```python
b = Integral(x**3)
Eq(b, b.doit())
```

### Sympy Tutorialに移りました

```python
from sympy import *
init_printing()
x, t, z, nu, y = symbols('x t z nu y')

diff(sin('x')*exp(x),x)
```

```python
diff(x**2)
```

```python
diff(sin('x')*exp(x))
```

```python
exp(x)
```

```python
exp(1)
```

```python
diff(exp(x))
```

```python
exp(1).evalf()
```

```python
expand_trig(sin(x + y))
```

```python
expand_trig(sin(2*x))
```

```python
expand_trig(cos(x + y))
```

```python
expand_trig(cos(2*x))
```

```python
expand_trig(sin(3*x))
```

```python
integrate(exp(x)*sin(x) + exp(x)*cos(x))
```

```python
integrate(sin(x**2),(x,-oo, oo))
```

```python
limit(sin(x)/x, x, 0)
```

```python
from sympy import *
init_printing()
x = symbols('x')
display(factor(x**2 - 4))
display(factor(x**2 - 2))
```

```python
solve(x**2 - 2, x)
```

```python
# solve the differential equation y'' - y = e^t
from sympy import *
init_printing()
x, y ,z, t = symbols('x y z t')
y = Function('y')
dsolve(Eq(y(t).diff(t, t) - y(t), exp(t)), y(t))
```

```python
y(t).diff(t,t)
```

### eigenvals()

```python
Matrix([[1,2],[2,2]]).eigenvals()
```

```python
from sympy import *
init_printing()
x, y ,z, nu, t = symbols('x y z nu t')

besselj(nu,z).rewrite(jn)
```

```python
besselj(nu, z)
```

```python
Integral(cos(x)**2,(x,0,pi))
```

```python
integrate(cos(x)**2,(x,0,pi))
```

```python
from sympy import*
init_printing()
x = symbols('x')
expr = x + 1
expr.subs(x,2)
```

```python
x
```

```python
Eq((x + 1)**2, x**2 + 2*x + 1)
```

```python
solve(Eq((x + 1)**2, 2*x))
```

```python
(x + 1)**2 == (x**2 + 2*x + 1)
```

    False

```python
expand((x + 1)**2) == (x**2 + 2*x + 1)
```

    True

```python
a = (x + 1)**2
b = x**2 + 2*x + 1
a
```

```python
a - b
```

```python
simplify(a - b)
```

```python
if simplify(a - b) == 0:
    print ("right")
else:
    print ("no")
```

    right

```python
a.equals(b)
```

    True

```python
a = cos(x)**2 - sin(x)**2
b = cos(2*x)
a.equals(b)
```

    True

```python
x^y
```

```python
type(Integer(1) + 1)
```

    sympy.core.numbers.Integer

```python
Integer(3)
```

```python
type(1 +  1)
```

    int

```python
Integer(3) / Integer(2)
```

```python
3/2
```

```python
# from __future__ import division
1/2
```

```python
Rational(1,2)
```

```python
x  + 1/2
```

### 代入

```python
from sympy import *
x, y, z = symbols('x y z')
expr = cos(x) + 1
expr.subs(x, y)
```

```python
expr.subs(x, 0)
```

```python
expr = x ** y
expr
```

```python
expr = expr.subs(y, x**y)
expr

```

```python
expr = sin(2*x) + cos(2*x)
expand_trig(expr)
```

```python
expr.subs(sin(2*x), 2*sin(x)*cos(x))
```

### 多重代入

```python
from sympy import*
init_printing()
x,y,z,t = symbols('x y z t')
expr = x**3 + 4*x*y - z
expr.subs([(x,2),(y,4),(z,0)])
```

```python
expr = x**4 - 4*x**3 + 4*x**2 - 2*x + 3
expr
```

```python
replacements = [(x**i, y**i) for i in range(5) if i % 2 ==0]
print (replacements)
display(expr.subs(replacements))
```

### sympify

```python
str_expr = "x**2+ 3*x - 1/2"
expr = sympify(str_expr)
expr
```

```python
expr.subs(x,2)
```

### evalf

```python
sqrt(8).evalf()
```

```python
E.evalf()
```

```python
pi.evalf(100)
```

```python
expr = cos(2*x)
expr.evalf(subs={x: 2.4})
```

```python
expr.subs(x,2.4).evalf()
```

```python
one = cos(1)**2 + sin(1)**2
(one - 1).evalf(chop=True)
```

```python
one = cos(1)**2 + sin(1)**2
(one - 1).evalf()
```

###  lambdify
`lambdify` どんなときに役にたつのだろうか。

まあいいか。進もう。

```python
import numpy
a = numpy.arange(10)
a
```

    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

```python
range (10)
```

    range(0, 10)

```python
range(0, 10)[3]
```

```python
a[3]
```

    3

```python
list(a)
```

    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

```python
list(range(0, 10))
```

```python
expr = sin(x)
f = lambdify(x, expr, "numpy")
f(a)
```

    array([ 0.        ,  0.84147098,  0.90929743,  0.14112001, -0.7568025 ,
           -0.95892427, -0.2794155 ,  0.6569866 ,  0.98935825,  0.41211849])

```python
f = lambdify(x, expr, "math")
f(0.1)
```

```python
lambdify?
```

### sympy_name:numerical_function pairs

```python
def mysin(x):
    """
    My sine. Note that this is only accurate for small x.
    """
    return x
f = lambdify(x, expr, {"sin":mysin})
f(0.1)
```

### printing

str, srepr, ASCII pretty printer, Unicode pretty printer, LaTeX, MathML, Dot

init_session() works
1. frm sympy import *
2. init_printing()
3. common symbols

```python
from sympy import *
init_session()
```

    IPython console for SymPy 1.7.1 (Python 3.7.10-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.7.1/

### continued fractions

```python
from sympy import *
init_printing()
x,y,z = symbols('x y z')

def list_to_frac(l):
    expr = Integer(0)
    for i in reversed(l[1:]):
        expr += i
        expr = 1 / expr
    return l[0] + expr
frac = list_to_frac([x,y,z])
frac
```

```python
list_to_frac([1,2,3,4])
```

```python
syms = symbols('a0:5')
syms
```

```python
a0, a1, a2, a3, a4 = syms

frac = list_to_frac(syms)
frac
```

```python
# 実験
from sympy import *
init_printing()
a11 = symbols('a11')
display (a11)
display(symbols('a_11'))
```

```python
# 実験
from sympy import *
init_printing()
a_seq = [-1, 3, 23, 8]
n, r = symbols('n, r')
a_n = Function('a')(n)
terms = 4
short_expr = Sum(a_n * r**n, (n, 0, terms - 1))
display(short_expr)
# coeffed_short_expr = short_expr.doit().subs(
#     (a_n.subs(n, i), a_seq[i]) for i in range(terms)) # 8*r**3 + 23*r**2 + 3*r - 1
# func_short_expr = lambdify(r, coeffed_short_expr, 'numpy')
```

```python
# 実験
from sympy import *
init_printing()
a = symbols('a', shape=(3,3))
display(a)
```

### cancel

```python
frac=cancel(frac)
frac
```

    frac

```python
%%script false
l=[]
frac = apart(frac, a0)
frac
```

```python
%%script false
l.append(a0)
frac = 1/(frac - a0)
frac
```

```python
%%script false
frac = apart(frac, a1)
frac
```

```python
%%script false
l.append(a1)
frac = 1/(frac - a1)
frac = apart(frac, a2)
frac
```

```python
%%script false
l.append(a2)
frac = 1/(frac - a2)
frac = apart(frac, a3)
frac
```

```python
%%script false
l.append(a3)
frac = 1/(frac - a3)
frac = apart(frac, a4)
frac
```

```python
%%script false
l.append(a4)
list_to_frac(l)
```

```python
%%script false
l
```

###  random

```python
import random
l = list(symbols('a0:5'))
random.shuffle(l)
orig_frac = frac = cancel(list_to_frac(l))
del l
```

```python
frac
```

```python
l=[]
frac = apart(frac, a1)
frac
```

```python
l.append(a1)
frac = 1/(frac - a1)
frac
```

```python
frac = apart(frac, a3)
frac
```

```python
l.append(a3)
frac = 1/(frac - a3)
frac
```

```python
frac = apart(frac, a0)
frac
```

```python
l.append(a0)
frac = 1/(frac - a0)
frac
```

```python
frac = apart(frac, a2)
frac
```

```python
l.append(a2)
frac = 1/(frac - a2)
frac
```

```python
frac = apart(frac, a4)
frac
```

```python
l.append(a4)
list_to_frac(l)
```

```python
l
```

```python
orig_frac
```

```python
cancel(list_to_frac(l))
```


# Columns and nullspaces in a matrix

+ This notebook is part of lecture 6 *Columnspace and nullspace* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
#import numpy as np
from sympy import init_printing, Matrix, symbols
#import matplotlib.pyplot as plt
#import seaborn as sns
#from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax')
%matplotlib inline
filterwarnings('ignore')
```

# Columnspace and nullspace of a matrix

## Columnspaces of matrices

* We saw in the previous lecture that columns of a matrix can form vectors
* Consider now the LU-decomposition of *A*
$$  PA = PLU $$

* The union P∪L (all vectors in P or L or both) is NOT a subspace
* The intersection P∩L (or vectors in P and L) is a subspace (because their intersection is only the zero vector)

* The intersection of any two subspaces is a subspace

* Consider the following example matrix

```python
A = Matrix([[1, 1, 2], [2, 1, 3], [3, 1, 4], [4, 1, 5]])
A
```

$$\left[\begin{matrix}1 & 1 & 2\\2 & 1 & 3\\3 & 1 & 4\\4 & 1 & 5\end{matrix}\right]$$

* Each of the column spaces are vectors (column space) in ℝ4

* The linear combinations of all the column vectors form a subspace
* Is it the whole *V* = ℝ4, though?

* The reason why we ask is because we want to bring it back to a system of linear equations and ask the question: Is there (always) a solution to the following:
$$ {A} \overline {x}= \overline {b} $$

* Thus, which right-hand sides *b* are allowed?
* In our example above we are in ℝ4 and we ask if linear combination of all of them fill ℝ4

* From our example above some right-hand sides will be allowed (they form a subspace)
* Let's look at an example for **b**

```python
x1, x2, x3 = symbols('x1, x2, x3')
vec_x = Matrix([x1, x2, x3])
b = Matrix([1, 2, 3, 4])
A, vec_x, b
```

$$\left ( \left[\begin{matrix}1 & 1 & 2\\2 & 1 & 3\\3 & 1 & 4\\4 & 1 & 5\end{matrix}\right], \quad \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\end{matrix}\right], \quad \left[\begin{matrix}1\\2\\3\\4\end{matrix}\right]\right )$$

```python
A * vec_x
```

$$\left[\begin{matrix}x_{1} + x_{2} + 2 x_{3}\\2 x_{1} + x_{2} + 3 x_{3}\\3 x_{1} + x_{2} + 4 x_{3}\\4 x_{1} + x_{2} + 5 x_{3}\end{matrix}\right]$$

* You can do the row multiplication, but it's easy to see from above we are asking about linear combinations of the columns, i.e. how many (*x*1) of column 1 plus how many (*x*2) of column 2 plus how many (*x*3) of column 3 equals **b**?
* Well, since **b** is the same as the first column, **x** would be
$$ \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$

* So we can solve for all values of **b** if **b** is in the column space

### Linear independence

* We really need to know if the columns above are linearly independent
* We note that column three above is a linear combination of the first two, so adds nothing new
* Actually, we could also throw away the first one because it is column 3 plus -1 times column 2
* Same for column 2
* We thus have two columns left and we say that the column space is of dimension 2 (a 2-dimensional subspace of ℝ4)

## The nullspace

* It contains all solutions **x** for A**x**=0
* This solution(s) is in ℝ3

```python
zero_b = Matrix([0, 0, 0, 0])
A, vec_x, zero_b
```

$$\left ( \left[\begin{matrix}1 & 1 & 2\\2 & 1 & 3\\3 & 1 & 4\\4 & 1 & 5\end{matrix}\right], \quad \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\end{matrix}\right], \quad \left[\begin{matrix}0\\0\\0\\0\end{matrix}\right]\right )$$

* Some solutions would be
$$ \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix} $$
$$ \begin{bmatrix} 1 \\ 1 \\ -1 \end{bmatrix} $$
$$ \begin{bmatrix} 2 \\ 2 \\ -2 \end{bmatrix} $$
* In fact, we have:
$$ {c} \begin{bmatrix} 1 \\ 1 \\ -1 \end{bmatrix} $$
* It is thus a line
* The nullspace is a line in ℝ3

* **PLEASE** remember, for any space the rules of addition and scalar multiplication must hold for vectors to remain in that space

```python

```


#### Procesamiento Digital de Señales

# Trabajo Práctico Nº0
#### Gisela Farace

# Introducción
Jupyter Notebook es una herramienta para la confección de reportes técnicos, dado que permite la interacción en el mismo ambiente de:
1. un procesador de texto elemental (formato Markdown) que permite resaltar texto, en forma de *itálica* o **negrita** de manera muy legible (haciendo doble click en este texto podrás ver el código fuente estilo Markdown). Cuenta con estilos predefinidos:

# Título 1
## Título 2
### Título 3

y también la capacidad de incluir enlaces a otras páginas, como por ejemplo [esta página](https://medium.com/ibm-data-science-experience/markdown-for-jupyter-notebooks-cheatsheet-386c05aeebed) donde encontrarás más funcionalidades del lenguaje **Markdown**

2. capacidad para incluir lenguaje matemático estilo LaTex, tanto de forma presentada

\begin{equation}
T(z) = \frac{Y(z)}{X(z)} = \frac{ b_2 \, z^{-2} + b_1 \,  z^{-1} + b_0 }
{a_2 \, z^{-2} + a_1 \,  z^{-1} + a_0}
\end{equation}

como *inline* en el propio párrafo $y[k] = \frac{1}{a_0} \left( \sum_{m=0}^{M} b_m \; x[k-m] - \sum_{n=1}^{N} a_n \; y[k-n] \right) $

3. La posibilidad de incluir scripts en Python, como los que usaremos para las simulaciones en los TPs de la materia. En este caso usaremos el *testbench0.py* como ejemplo. Una vez que lo probamos y estamos seguros que funciona de forma esperada en *Spyder*, podemos incluir los resultados de la simulación de manera casi transparente. Solo tenemos que agregar una celda de código donde incluimos el código, y los resultados directamente quedan incluidos en este documento.

```python
# Módulos para Jupyter
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import matplotlib as mpl
#%%  Inicialización de librerías
# Setup inline graphics: Esto lo hacemos para que el tamaño de la salida,
# sea un poco más adecuada al tamaño del documento
mpl.rcParams['figure.figsize'] = (50,20)

import matplotlib.pyplot as plt
import pdsmodulos as pds

#%% Esto tiene que ver con cuestiones de presentación de los gráficos,
# NO ES IMPORTANTE
fig_sz_x = 14
fig_sz_y = 13
fig_dpi = 80 # dpi

fig_font_family = 'Ubuntu'
fig_font_size = 16

plt.rcParams.update({'font.size':fig_font_size})
plt.rcParams.update({'font.family':fig_font_family})

##############################################
#%% A partir de aquí comienza lo IMPORTANTE #
#############################################

def my_testbench( sig_type ):

    # Datos generales de la simulación
    fs = 1000.0 # frecuencia de muestreo (Hz)
    N = 1000   # cantidad de muestras

    ts = 1/fs # tiempo de muestreo
    df = fs/N # resolución espectral

    # grilla de sampleo temporal
    tt = np.linspace(0, (N-1)*ts, N).flatten()

    # grilla de sampleo frecuencial
    ff = np.linspace(0, (N-1)*df, N).flatten()

    # Concatenación de matrices:
    # guardaremos las señales creadas al ir poblando la siguiente matriz vacía
    x = np.array([], dtype=np.float).reshape(N,0)
    ii = 0

    # estructuras de control de flujo
    if sig_type['tipo'] == 'senoidal':

        # calculo cada senoidal de acuerdo a sus parámetros
        for this_freq in sig_type['frecuencia']:
            # prestar atención que las tuplas dentro de los diccionarios también pueden direccionarse mediante "ii"
            aux = sig_type['amplitud'][ii] * np.sin( 2*np.pi*this_freq*tt + sig_type['fase'][ii] )
            # para concatenar horizontalmente es necesario cuidar que tengan iguales FILAS
            x = np.hstack([x, aux.reshape(N,1)] )
            ii += 1

    elif sig_type['tipo'] == 'ruido':

        # calculo cada señal de ruido incorrelado (blanco), Gausiano de acuerdo a sus parámetros
        # de varianza
        for this_var in sig_type['varianza']:
            aux = np.sqrt(this_var) * np.random.randn(N,1)
            # para concatenar horizontalmente es necesario cuidar que tengan iguales FILAS
            x = np.hstack([x, aux] )

        # Podemos agregar algún dato extra a la descripción de forma programática
        # {0:.3f} significa 0: primer argunmento de format
        # .3f formato flotante, con 3 decimales
        # $ ... $ indicamos que incluiremos sintaxis LaTex: $\hat{{\sigma}}^2$
        sig_props['descripcion'] = [ sig_props['descripcion'][ii] + ' - $\hat{{\sigma}}^2$ :{0:.3f}'.format( np.var(x[:,ii]))  for ii in range(0,len(sig_props['descripcion'])) ]

    else:

        print("Tipo de señal no implementado.")
        return

    #%% Presentación gráfica de los resultados

    plt.figure(1)
    line_hdls = plt.plot(tt, x)
    plt.title('Señal: ' + sig_type['tipo'] )
    plt.xlabel('tiempo [segundos]')
    plt.ylabel('Amplitud [V]')
    #    plt.grid(which='both', axis='both')

    # presentar una leyenda para cada tipo de señal
    axes_hdl = plt.gca()

    # este tipo de sintaxis es *MUY* de Python
    axes_hdl.legend(line_hdls, sig_type['descripcion'], loc='upper right'  )

    plt.show()

```

Dado que nuestro *testbench* ha sido desarrollado de manera funcional, llamando a la función *my_testbench()* con diferentes parámetros, podemos lograr funcionalidades diferentes, como mostramos a continuación primero con una senoidal:

```python
f0 = 500
sig_props = { 'tipo': 'senoidal',
              'frecuencia': (f0, f0, f0), # Uso de tuplas para las frecuencias
              'amplitud':   (1, 1,  1),
              'fase':       (0, 0,  0)
             }
# Como también puedo agregar un campo descripción de manera programática
# este tipo de sintaxis es *MUY* de Python
sig_props['descripcion'] = [ str(a_freq) + ' Hz' for a_freq in sig_props['frecuencia'] ]

# Invocamos a nuestro testbench exclusivamente:
my_testbench( sig_props )
```

Y ahora con una señal aleatoria, en este caso ruido blanco Gaussiano incorrelado de varianza $\sigma^2$:

```python
# Usar CTRL+1 para comentar o descomentar el bloque de abajo.
sig_props = { 'tipo': 'ruido',
              'varianza': (0.5, 1, 1) # Uso de tuplas para las frecuencias
             }
sig_props['descripcion'] = [ '$\sigma^2$ = ' + str(a_var) for a_var in sig_props['varianza'] ]

# Invocamos a nuestro testbench exclusivamente:
my_testbench( sig_props )

```

Como puede verse en la figura anterior, al samplear una distribución estadística de media nula y varianza $\sigma^2=1$, obtenemos realizaciones cuyo parámetro $\sigma^2$ estimado, es decir $\hat\sigma^2$, tienen una desviación respecto al verdadero valor (sesgo). Nos ocuparemos de estudiar el sesgo y la varianza de algunos estimadores cuando veamos **Estimación Espectral**.

# Una vez terminado ...
Una vez que hayas termiando con la confección del documento, podemos utilizar una ventaja muy importante de este tipo de documentos que es la posibilidad de compartirlos *online* mediante la [página de nbviewer](http://nbviewer.jupyter.org/). Para ello es necesario que tu notebook y todos los recursos asociados estén alojados en un repositorio de [Github](https://github.com/). Como ejemplo, podés ver este mismo documento disponible [online](http://nbviewer.jupyter.org/github/marianux/pdstestbench/blob/master/notebook0.ipynb).

# Un gatito

# Modificaciones
- Cambie la frecuencia y amplitud del 1er gráfico
- Cambie la varianza del 2do gráfico
- Incluí la imagen de un gatito


# CAM Methods Benchmark

**Goal:** to compare CAM methods using regular explaining metrics.
**Author:** lucas.david@ic.unicamp.br

Use GPUs if you are running *Score-CAM* or *Quantitative Results* sections.

```python
#@title

from google.colab import drive
drive.mount('/content/drive')
```

```python
import tensorflow as tf

# base_dir = '/content/drive/MyDrive/'
base_dir = '/home/ldavid/Workspace'
# data_dir = '/root/tensorflow_datasets'
data_dir = '/home/ldavid/Workspace/datasets/'

class Config:
  seed = 218402

  class data:
    path = '/root/tensorflow_datasets/amazon-from-space'
    size = (256, 256)
    shape = (*size, 3)
    batch_size = 32
    shuffle_buffer_size = 8 * batch_size
    prefetch_buffer_size = tf.data.experimental.AUTOTUNE
    train_shuffle_seed = 120391
    shuffle = False

  class model:
    backbone = tf.keras.applications.ResNet101V2
    last_spatial_layer = 'post_relu'
    # backbone = tf.keras.applications.EfficientNetB6
    # last_spatial_layer = 'eb6'
    # backbone = tf.keras.applications.VGG16
    # last_spatial_layer = 'block5_pool'

    gap_layer_name = 'avg_pool'
    include_top = False
    classifier_activation = None

    custom = True
    fine_tune_layers = 0.6
    freeze_batch_norm = False

    weights = f'{base_dir}/logs/amazon-from-space/resnet101-sw-ce-fine-tune/weights.h5'

  class training:
    valid_size = 0.3

  class explaining:
    noise = tf.constant(.2)
    repetitions = tf.constant(8)

    score_cam_activations = 'all'

    λ_pos = tf.constant(1.)
    λ_neg = tf.constant(1.)
    λ_bg  = tf.constant(1.)

  report = f'{base_dir}/logs/amazon-from-space/resnet101-sw-ce-fine-tune/cam-score.txt'
```

```python
preprocess = tf.keras.applications.resnet_v2.preprocess_input
deprocess = lambda x: (x + 1) * 127.5

# preprocess = tf.keras.applications.res.preprocess_input
# deprocess = lambda x: x

# preprocess = tf.keras.applications.vgg16.preprocess_input
# deprocess = lambda x: x[..., ::-1] + [103.939, 116.779, 123.68]

to_image = lambda x: tf.cast(tf.clip_by_value(deprocess(x), 0, 255), tf.uint8)
masked = lambda x, maps: x * tf.image.resize(maps, Config.data.size)
```

## Setup

```python
! pip -qq install tensorflow_addons
```

```python
import os
import shutil
from time import time
from math import ceil

import numpy as np
import pandas as pd
import tensorflow_addons as tfa
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt
import seaborn as sns

from tensorflow.keras import callbacks
```

```python
for d in tf.config.list_physical_devices('GPU'):
  print(d)
  print(f'Setting device {d} to memory-growth mode.')
  try:
    tf.config.experimental.set_memory_growth(d, True)
  except Exception as e:
    print(e)
```

    PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
    Setting device PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU') to memory-growth mode.
    Physical devices cannot be modified after being initialized

```python
R = tf.random.Generator.from_seed(Config.seed, alg='philox')
C = np.asarray(sns.color_palette("Set1", 21))
CMAP = sns.color_palette("Set1", 21, as_cmap=True)

sns.set_style("whitegrid", {'axes.grid' : False})
```

```python
np.set_printoptions(linewidth=120)
```

```python
def normalize(x, reduce_min=True, reduce_max=True):
  if reduce_min: x -= tf.reduce_min(x, axis=(-3, -2), keepdims=True)
  if reduce_max: x = tf.math.divide_no_nan(x, tf.reduce_max(x, axis=(-3, -2), keepdims=True))

  return x

def visualize(
    image,
    title=None,
    rows=2,
    cols=None,
    i0=0,
    figsize=(9, 4),
    cmap=None,
    full=True
):
  if image is not None:
    if isinstance(image, (list, tuple)) or len(image.shape) > 3:  # many images
      if full: plt.figure(figsize=figsize)
      cols = cols or ceil(len(image) / rows)
      for ix in range(len(image)):
        plt.subplot(rows, cols, i0+ix+1)
        visualize(image[ix],
                 cmap=cmap,
                 title=title[ix] if title is not None and len(title) > ix else None)
      if full: plt.tight_layout()
      return

    if isinstance(image, tf.Tensor): image = image.numpy()
    if image.shape[-1] == 1: image = image[..., 0]
    plt.imshow(image, cmap=cmap)

  if title is not None: plt.title(title)
  plt.axis('off')
```

```python
def observe_labels(probs, labels, ix):
  p = probs[ix]
  l = labels[ix]
  s = tf.argsort(p, direction='DESCENDING')

  d = pd.DataFrame({
    'idx': s,
    'label': tf.gather(CLASSES, s).numpy().astype(str),
    'predicted': tf.gather(p, s).numpy().round(2),
    'ground-truth': tf.gather(l, s).numpy()
  })

  return d[(d['ground-truth']==1) | (d['predicted'] > 0.05)]

def plot_heatmap(i, m):
    plt.imshow(i)
    plt.imshow(m, cmap='jet', alpha=0.5)
    plt.axis('off')
```

## Related Work

### Summary

```python
#@title

d = [
  ['1512.04150', 'CAM',      'GoogleNet', 'ILSVRC-15 val',   56.4,  43.00, None, None, None, None],
  ['1512.04150', 'CAM',      'VGG-16',    'ILSVRC-15 val',   57.2,  45.14, None, None, None, None],
  ['1512.04150', 'Backprop', 'GoogleNet', 'ILSVRC-15 val',   61.31, 50.55, None, None, None, None],
  ['1512.04150', 'Backprop', 'GoogleNet', 'ILSVRC-15 test',  None,  37.10, None, None, None, None],

  ['1610.02391', 'CAM',      'VGG-16',    'ILSVRC-15 val',   57.2,  45.14, None, None, None, None, None],
  ['1610.02391', 'Grad-CAM', 'VGG-16',    'ILSVRC-15 val',   56.51, 46.41, None, None, None, None, None],
  ['1610.02391', 'CAM',      'GoogleNet', 'ILSVRC-15 val',   60.09, 49.34, None, None, None, None, None],
  ['1610.02391', 'Grad-CAM', 'GoogleNet', 'ILSVRC-15 val',   60.09, 49.34, None, None, None, None, None],

  ['1710.11063', 'Grad-CAM',   'VGG-16',    'ILSVRC-2012 val', None, None, 46.56, 13.42, 29.28, None, None],
  ['1710.11063', 'Grad-CAM++', 'VGG-16',    'ILSVRC-2012 val', None, None, 36.84, 17.05, 70.72, None, None],
  ['1710.11063', 'Grad-CAM',   'AlexNet',   'ILSVRC-2012 val', None, None, 82.86, 3.16,  13.44, None, None],
  ['1710.11063', 'Grad-CAM++', 'AlexNet',   'ILSVRC-2012 val', None, None, 62.75, 8.24,  86.56, None, None],
  ['1710.11063', 'Grad-CAM',   'VGG-16',    'Pascal 2007 val', None, None, 28.54, 21.43, 39.44, None, None],
  ['1710.11063', 'Grad-CAM++', 'VGG-16',    'Pascal 2007 val', None, None, 19.53, 18.96, 61.47, None, None],
  ['1710.11063', 'Grad-CAM',   'AlexNet',   'Pascal 2007 val', None, None, 45.82, 14.38, 27.21, None, None],
  ['1710.11063', 'Grad-CAM++', 'AlexNet',   'Pascal 2007 val', None, None, 29.16, 19.76, 72.79, None, None],
  ['1710.11063', 'Grad-CAM',   'ResNet-50', 'ILSVRC-2012 val', None, None, 30.36, 22.11, 39.49, None, None],
  ['1710.11063', 'Grad-CAM++', 'ResNet-50', 'ILSVRC-2012 val', None, None, 28.90, 22.16, 60.51, None, None],
  ['1710.11063', 'Grad-CAM',   'ResNet-50', 'Pascal 2007 val', None, None, 20.86, 21.99, 41.39, None, None],
  ['1710.11063', 'Grad-CAM++', 'ResNet-50', 'Pascal 2007 val', None, None, 16.19, 19.52, 58.61, None, None],

  ['1710.11063', 'Grad-CAM',   'VGG-16',  'Pascal 2012 val', None, None, None, None, None, 0.33, None],
  ['1710.11063', 'Grad-CAM++', 'VGG-16',  'Pascal 2012 val', None, None, None, None, None, 0.34, None],

  ['1910.01279', 'Backprop Vanilla',   'VGG-16', 'ILSVRC-2012 val', None, None, None, None, None, None, 41.3],
  ['1910.01279', 'Backprop Smooth',     'VGG-16', 'ILSVRC-2012 val', None, None, None, None, None, None, 42.4],
  ['1910.01279', 'Backprop Integraded', 'VGG-16', 'ILSVRC-2012 val', None, None, None, None, None, None, 44.7],

  ['1910.01279', 'Grad-CAM',   'VGG-16', 'ILSVRC-2012 val', None, None, 47.80, 19.60, None, None, 48.1],
  ['1910.01279', 'Grad-CAM++', 'VGG-16', 'ILSVRC-2012 val', None, None, 45.50, 18.90, None, None, 49.3],
  ['1910.01279', 'Score-CAM',  'VGG-16', 'ILSVRC-2012 val', None, None, 31.50, 30.60, None, None, 63.7],
]

d = pd.DataFrame(
  d,
  columns=[
    'Source', 'Method', 'Arch', 'Dataset',
    'Loc_Error_T-1', 'Loc_Error_T-5',
    'Avg_Drop', 'Incr_in_confidence', 'Win_%', 'mLoc_I^c(s=0)',
    'E-Pointing_Game'
  ]
)
```

```python
#@title

(d.groupby(['Dataset', 'Method'], as_index=False)
  .mean()
  .replace(np.nan, '', regex=True))
```

      Dataset
      Method
      Loc_Error_T-1
      Loc_Error_T-5
      Avg_Drop
      Incr_in_confidence
      Win_%
      mLoc_I^c(s=0)
      E-Pointing_Game

      0
      ILSVRC-15 test
      Backprop

      37.1

      1
      ILSVRC-15 val
      Backprop
      61.31
      50.55

      2
      ILSVRC-15 val
      CAM
      57.7225
      45.655

      3
      ILSVRC-15 val
      Grad-CAM
      58.3
      47.875

      4
      ILSVRC-2012 val
      Backprop Integraded

      44.7

      5
      ILSVRC-2012 val
      Backprop Smooth

      42.4

      6
      ILSVRC-2012 val
      Backprop Vanilla

      41.3

      7
      ILSVRC-2012 val
      Grad-CAM

      51.895
      14.5725
      27.403333

      48.1

      8
      ILSVRC-2012 val
      Grad-CAM++

      43.4975
      16.5875
      72.596667

      49.3

      9
      ILSVRC-2012 val
      Score-CAM

      31.5
      30.6

      63.7

      10
      Pascal 2007 val
      Grad-CAM

      31.74
      19.266667
      36.013333

      11
      Pascal 2007 val
      Grad-CAM++

      21.626667
      19.413333
      64.29

      12
      Pascal 2012 val
      Grad-CAM

      0.33

      13
      Pascal 2012 val
      Grad-CAM++

      0.34

```python
#@title Full Report

d.replace(np.nan, '', regex=True)
```

      Source
      Method
      Arch
      Dataset
      Loc_Error_T-1
      Loc_Error_T-5
      Avg_Drop
      Incr_in_confidence
      Win_%
      mLoc_I^c(s=0)
      E-Pointing_Game

      0
      1512.04150
      CAM
      GoogleNet
      ILSVRC-15 val
      56.4
      43.0

      1
      1512.04150
      CAM
      VGG-16
      ILSVRC-15 val
      57.2
      45.14

      2
      1512.04150
      Backprop
      GoogleNet
      ILSVRC-15 val
      61.31
      50.55

      3
      1512.04150
      Backprop
      GoogleNet
      ILSVRC-15 test

      37.1

      4
      1610.02391
      CAM
      VGG-16
      ILSVRC-15 val
      57.2
      45.14

      5
      1610.02391
      Grad-CAM
      VGG-16
      ILSVRC-15 val
      56.51
      46.41

      6
      1610.02391
      CAM
      GoogleNet
      ILSVRC-15 val
      60.09
      49.34

      7
      1610.02391
      Grad-CAM
      GoogleNet
      ILSVRC-15 val
      60.09
      49.34

      8
      1710.11063
      Grad-CAM
      VGG-16
      ILSVRC-2012 val

      46.56
      13.42
      29.28

      9
      1710.11063
      Grad-CAM++
      VGG-16
      ILSVRC-2012 val

      36.84
      17.05
      70.72

      10
      1710.11063
      Grad-CAM
      AlexNet
      ILSVRC-2012 val

      82.86
      3.16
      13.44

      11
      1710.11063
      Grad-CAM++
      AlexNet
      ILSVRC-2012 val

      62.75
      8.24
      86.56

      12
      1710.11063
      Grad-CAM
      VGG-16
      Pascal 2007 val

      28.54
      21.43
      39.44

      13
      1710.11063
      Grad-CAM++
      VGG-16
      Pascal 2007 val

      19.53
      18.96
      61.47

      14
      1710.11063
      Grad-CAM
      AlexNet
      Pascal 2007 val

      45.82
      14.38
      27.21

      15
      1710.11063
      Grad-CAM++
      AlexNet
      Pascal 2007 val

      29.16
      19.76
      72.79

      16
      1710.11063
      Grad-CAM
      ResNet-50
      ILSVRC-2012 val

      30.36
      22.11
      39.49

      17
      1710.11063
      Grad-CAM++
      ResNet-50
      ILSVRC-2012 val

      28.9
      22.16
      60.51

      18
      1710.11063
      Grad-CAM
      ResNet-50
      Pascal 2007 val

      20.86
      21.99
      41.39

      19
      1710.11063
      Grad-CAM++
      ResNet-50
      Pascal 2007 val

      16.19
      19.52
      58.61

      20
      1710.11063
      Grad-CAM
      VGG-16
      Pascal 2012 val

      0.33

      21
      1710.11063
      Grad-CAM++
      VGG-16
      Pascal 2012 val

      0.34

      22
      1910.01279
      Backprop Vanilla
      VGG-16
      ILSVRC-2012 val

      41.3

      23
      1910.01279
      Backprop Smooth
      VGG-16
      ILSVRC-2012 val

      42.4

      24
      1910.01279
      Backprop Integraded
      VGG-16
      ILSVRC-2012 val

      44.7

      25
      1910.01279
      Grad-CAM
      VGG-16
      ILSVRC-2012 val

      47.8
      19.6

      48.1

      26
      1910.01279
      Grad-CAM++
      VGG-16
      ILSVRC-2012 val

      45.5
      18.9

      49.3

      27
      1910.01279
      Score-CAM
      VGG-16
      ILSVRC-2012 val

      31.5
      30.6

      63.7

**Localization Error**

As defined in http://image-net.org/challenges/LSVRC/2015/index#maincomp:

Let $d(c_i,C_k)=0$ if $c_i=C_k$ and 1 otherwise. Let $f(b_i,B_k)=0$ if $b_i$ and $B_k$ have more than 50% overlap, and 1 otherwise. The error of the algorithm on an individual image will be computed using:

$$e=\frac{1}{n} \cdot \sum_k min_{i} min_{m}  max \{d(c_i,C_k), f(b_i,B_{km}) \}$$

**Pixel Perturbation** (Full-Gradient)

First form: remove $k$ most salient pixels from the image and measure impact on output confidence (high impact expected for good saliency methods, similar to Avg. Drop %). This might add artifacts (edges).

Second form: remove $k$ least salient pixels from the image and measure output confidence (low impact expected for good saliency methods).

**Average Drop %** (Grad-CAM++, Score-CAM)

The avg. percentual drop in the confidence of a model for a particular image $x_i$ and class $c$, when only the highlighted region is provided ($M_i\circ x_i$):

$$∑_i^N \frac{max(0, Y_i^c − O_i^c)}{Y_i^c} 100$$

* $Y_i^c = f(x_i)^c$
* $O_i^c = f(M_i\circ x_i)^c$

**Increase in confidence %** (Grad-CAM++, Score-CAM)

Measures scenarios where removing background noise must improve classification confidence.

$$∑^N_i \frac{Y^c_i < O^c_i}{N}100$$

### CAM

$M(f, x)^u_{ij} = \text{relu}(\sum_k w^u_k A_{ij}^k)$

```python
@tf.function
def sigmoid_cam(x, y):
  print(f'CAM tracing x:{x.shape} y:{y.shape}')

  l, a = nn_s(x, training=False)
  y = tf.einsum('bhwk,ku->buhw', a, sW)

  return l, y[..., tf.newaxis]
```

### Grad-CAM

$M(f, x)^u_{ij} = \text{relu}(\sum_k \sum_{lm}\frac{\partial S_u}{\partial A_{lm}^k} A_{ij}^k)$

```python
@tf.function
def sigmoid_gradcam(x, y):
  print(f'Grad-CAM tracing x:{x.shape} y:{y.shape}')

  with tf.GradientTape(watch_accessed_variables=False) as t:
    t.watch(x)
    l, a = nn_s(x, training=False)

  dlda = t.batch_jacobian(l, a)

  weights = tf.reduce_sum(dlda, axis=(-3, -2))  # bc(hw)k -> bck
  maps = tf.einsum('bhwc,buc->buhw', a, weights)

  return l, maps[..., tf.newaxis]
```

Note that for fully-convolutional network, with a single densely-connected softmax classifier at its end:

$S_u = \sum_k w^k_u [\frac{1}{hw}\sum_{lm}^{hw} A^k_{lm}]$

Then:

\begin{align}
\frac{\partial S_u}{\partial A_{lm}^k} &= \frac{\partial}{\partial A_{lm}^k} \sum_k w^k_u [\frac{1}{hw}\sum_{lm}^{hw} A^k_{lm}] \\
  &= w^k_u [\frac{1}{hw}\frac{\partial A^k_{lm}}{\partial A^k_{lm}}] \\
  &= \frac{w^k_u}{hw}
\end{align}

All constants (including $\frac{1}{hw}$) are erased when we apply normalization.
Therefore, in these conditions, this this method is equivalent to `CAM`.

### Grad-CAM++

$M(f, x)^u_{ij} = \sum_k \sum_{lm} \alpha_{lm}^{ku} \text{relu}(\frac{\partial S_u}{\partial A_{lm}^k}) A_{ij}^k$

Where

$\alpha_{lm}^{ku} = \frac{(\frac{\partial S_u}{\partial A_{lm}^k})^2}{2(\frac{\partial S_u}{\partial A_{lm}^k})^2 + \sum_{ab} A_{ab}^k(\frac{\partial S_u}{\partial A_{lm}^k})^3}$

```python
@tf.function
def sigmoid_gradcampp(x, y):
  print(f'Grad-CAM++ tracing x:{x.shape} y:{y.shape}')

  with tf.GradientTape(watch_accessed_variables=False) as tape:
    tape.watch(x)
    s, a = nn_s(x, training=False)

  dsda = tape.batch_jacobian(s, a)

  dyda = tf.einsum('bu,buhwk->buhwk', tf.exp(s), dsda)
  d2 = dsda**2
  d3 = dsda**3
  aab = tf.reduce_sum(a, axis=(1, 2))              # (BK)
  akc = tf.math.divide_no_nan(
    d2,
    2.*d2 + tf.einsum('bk,buhwk->buhwk', aab, d3))  # (2*(BUHWK) + (BK)*BUHWK)

  weights = tf.einsum('buhwk,buhwk->buk', akc, tf.nn.relu(dyda))  # w: buk
  maps = tf.einsum('buk,bhwk->buhw', weights, a)                  # a:bhwk, m: buhw

  return s, maps[..., tf.newaxis]
```

### Score CAM

$M(f, x)^u_{ij} = \text{relu}(∑_k \text{softmax}(C(A_l^k)_u) A_l^k)$

Where

$
C(A_l^k)_u = f(X_b \circ \psi(\text{up}(A_l^k)))_u - f(X_b)_u
$

This algorithm has gone through updated. I followed the most recent implementation in [haofanwang/Score-CAM](https://github.com/haofanwang/Score-CAM/blob/master/cam/scorecam.py#L47).

```python
def sigmoid_scorecam(x, y):
  acts_used = Config.explaining.score_cam_activations

  l, a = nn_s(x, training=False)

  if acts_used == 'all' or acts_used is None:
    acts_used = a.shape[-1]

  # Sorting kernels from highest to lowest variance.
  std = tf.math.reduce_std(a, axis=(1, 2))
  a_high_std = tf.argsort(std, axis=-1, direction='DESCENDING')[:, :acts_used]
  a = tf.gather(a, a_high_std, axis=3, batch_dims=1)

  s = tf.Variable(tf.zeros((x.shape[0], y.shape[1], *Config.data.size)), name='sc_maps')

  for ix in range(acts_used):
    a_ix = a[..., ix:ix+1]

    if tf.reduce_min(a_ix) == tf.reduce_max(a_ix):
      break

    s.assign_add(_scorecam_feed(x, a_ix))

  return l, s[..., tf.newaxis]

@tf.function
def _scorecam_feed(x, a_ix):
  print('Score-CAM feed tracing')
  a_ix = tf.image.resize(a_ix, Config.data.size)

  b = normalize(a_ix)
  fm = nn(x * b, training=False)
  fm = tf.nn.sigmoid(fm)
  fm = tf.einsum('bc,bhw->bchw', fm, a_ix[..., 0])

  return fm
```

Vectorized implementation:

The number of activating kernels used was defined in `Config.explaining.score_cam_activations`.
For each batch of 16 samples (`100 MB = 16 × 512×512×3 × 8÷1000÷1000`):

1. 16 samples are feed-forwarded, generating the logits `(16, 20)` and activations `(16, 16, 16, score_cam_activations)`
2. The masked input `(16 × score_cam_activations, 512, 512, 3)` is created (`score_cam_activations × 100 MB`).
3. The masked input is feed-forwarded (batching used to prevent the GPU from blowing up).

```python
def sigmoid_scorecam(x, y):
  acts_used = Config.explaining.score_cam_activations

  l, a = nn_s(x, training=False)

  std = tf.math.reduce_std(a, axis=(1, 2))
  a_high_std = tf.argsort(std, axis=-1, direction='DESCENDING')[:, :acts_used]
  a = tf.gather(a, a_high_std, axis=-1, batch_dims=-1)

  a = tf.image.resize(a, Config.data.size)
  b = normalize(a)

  b = tf.einsum('bhwc,bhwk->bkhwc', x, b)      # outer product over 2 ranks
  b = tf.reshape(b, (-1, *Config.data.shape))  # batchify (B*A, H, W, C)

  fm = nn.predict(b, batch_size=Config.data.batch_size)
  fm = tf.nn.sigmoid(fm)
  fm = tf.reshape(fm, (x.shape[0], acts_used, fm.shape[1]))  # unbatchify

  s = tf.einsum('bhwk,bkc->bchw', a, fm)
  s = tf.nn.relu(s)
  s = s[..., tf.newaxis]

  return l, s
```

## Dataset

### Augmentation Policy

```python
def default_policy_fn(image):
  # image = tf.image.resize_with_crop_or_pad(image, *Config.data.size)
  # mask = tf.image.resize_with_crop_or_pad(mask, *Config.data.size)

  return image
```

### Preparing and Performance Settings

```python
data_path = Config.data.path
```

```bash
%%bash

if [ ! -d /root/tensorflow_datasets/amazon-from-space/ ]; then
  mkdir -p /root/tensorflow_datasets/amazon-from-space/

  # gdown --id 12wCmah0FFPIjI78YJ2g_YWFy97gaA5S9 --output /root/tensorflow_datasets/amazon-from-space/train-jpg.tfrecords

  cp /content/drive/MyDrive/datasets/amazon-from-space/train-jpg.tfrecords \
     /root/tensorflow_datasets/amazon-from-space/
else
  echo "Dir $data_path found. Skipping."
fi
```

```python
class AmazonFromSpace:
    num_train_samples = 40479
    num_test_samples  = 61191

    classes_ = np.asarray(
        ['agriculture', 'artisinal_mine', 'bare_ground', 'blooming', 'blow_down',
         'clear', 'cloudy', 'conventional_mine', 'cultivation', 'habitation', 'haze',
         'partly_cloudy', 'primary', 'road', 'selective_logging', 'slash_burn', 'water'])

    @classmethod
    def int2str(cls, indices):
        return cls.classes_[indices]

    @staticmethod
    def _bytes_feature(value):
        return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value.tobytes()]))

    @staticmethod
    def _int64_feature(value):
        return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

    @staticmethod
    def decode_fn(record_bytes):
        r = tf.io.parse_single_example(record_bytes, {
          'filename': tf.io.FixedLenFeature([], tf.string),
          'image': tf.io.FixedLenFeature([], tf.string),
          'height': tf.io.FixedLenFeature([], tf.int64, default_value=[256]),
          'width': tf.io.FixedLenFeature([], tf.int64, default_value=[256]),
          'channels': tf.io.FixedLenFeature([], tf.int64, default_value=[3]),
          'label': tf.io.VarLenFeature(tf.int64),
        })

        r['image'] = tf.reshape(tf.io.decode_raw(r['image'], tf.uint8),
                                (r['height'], r['width'], r['channels']))
        r['label'] = tf.sparse.to_dense(r['label'])

        return r

    @classmethod
    def load(cls, tfrecords_path):
        return tf.data.TFRecordDataset(tfrecords_path).map(cls.decode_fn, num_parallel_calls=tf.data.AUTOTUNE)
```

```python
CLASSES = AmazonFromSpace.classes_
int2str = AmazonFromSpace.int2str
num_samples = AmazonFromSpace.num_train_samples
num_train_samples = int((1-Config.training.valid_size)*num_samples)
num_valid_samples = int(Config.training.valid_size*num_samples)
```

```python
from functools import partial

@tf.function
def load_fn(d, augment=False):
  image = d['image']
  labels = d['label']

  image = tf.cast(image, tf.float32)
  image = tf.ensure_shape(image, Config.data.shape)

  # image, _ = adjust_resolution(image)
  image = (augment_policy_fn(image)
           if augment
           else default_policy_fn(image))

  image = preprocess(image)

  return image, labels_to_one_hot(labels)

def adjust_resolution(image):
  es = tf.constant(Config.data.size, tf.float32)
  xs = tf.cast(tf.shape(image)[:2], tf.float32)

  ratio = tf.reduce_min(es / xs)
  xsn = tf.cast(tf.math.ceil(ratio * xs), tf.int32)

  image = tf.image.resize(image, xsn, preserve_aspect_ratio=True, method='nearest')

  return image, ratio

def labels_to_one_hot(labels):
  return tf.reduce_max(
      tf.one_hot(labels, depth=CLASSES.shape[0]),
    axis=0)

def prepare(ds, batch_size, cache=False, shuffle=False, augment=False):
  if cache: ds = ds.cache()
  if shuffle: ds = ds.shuffle(Config.data.shuffle_buffer_size, reshuffle_each_iteration=True, seed=Config.data.train_shuffle_seed)

  return (ds.map(partial(load_fn, augment=augment), num_parallel_calls=tf.data.AUTOTUNE)
            .batch(batch_size, drop_remainder=True)
            .prefetch(Config.data.prefetch_buffer_size))
```

```python
train_dataset = AmazonFromSpace.load(f'{data_dir}/amazon-from-space/train-jpg.tfrecords')
valid_dataset = train_dataset.take(num_valid_samples)
train_dataset = train_dataset.skip(num_valid_samples)

# train = prepare(train_dataset, Config.data.batch_size)
valid = prepare(valid_dataset, Config.data.batch_size)
```

### Examples in The Dataset

```python
#@title

for stage, batches, samples in zip(('validation',),
                                   (valid,),
                                   (num_valid_samples,)):
  print(stage)
  print(f'  {batches}')
  print(f'  samples: {samples}')
  print(f'  steps  : {samples // Config.data.batch_size}')
  print()
```

    validation
      <PrefetchDataset shapes: ((32, 256, 256, 3), (32, 17)), types: (tf.float32, tf.float32)>
      samples: 12143
      steps  : 379

```python
#@title

for images, labels in valid.take(1):
  gt = [' '.join((e[:3] for e in CLASSES[l].astype(str))) for l in labels.numpy().astype(bool)]
  visualize(to_image(images[:16]), gt, rows=4, figsize=(12, 10));
```

## Network

```python
print(f'Loading {Config.model.backbone.__name__}')

backbone = Config.model.backbone(
  input_shape=Config.data.shape,
  include_top=Config.model.include_top,
  classifier_activation=Config.model.classifier_activation,
)
```

    Loading ResNet101V2

```python
from tensorflow.keras.layers import Conv2D, Dense, Dropout, GlobalAveragePooling2D

class DenseKur(Dense):
  """Dense with Softmax Weights.
  """
  def call(self, inputs):
    kernel = self.kernel
    ag = kernel  # tf.abs(kernel)
    ag = ag - tf.reduce_max(ag, axis=-1, keepdims=True)
    ag = tf.nn.softmax(ag)

    outputs = inputs @ (ag*kernel)

    if self.use_bias:
      outputs = tf.nn.bias_add(outputs, self.bias)

    if self.activation is not None:
      outputs = self.activation(outputs)

    return outputs
```

```python
def build_specific_classifier(
    backbone,
    classes,
    dropout_rate=0.5,
    name=None,
    gpl='avg_pool',
):
  x = tf.keras.Input(Config.data.shape, name='images')
  y = backbone(x)
  y = GlobalAveragePooling2D(name='avg_pool')(y)
  # y = Dense(classes, name='predictions')(y)
  y = DenseKur(classes, name='predictions')(y)

  return tf.keras.Model(x, y, name=name)

backbone.trainable = False
nn = build_specific_classifier(backbone, len(CLASSES), name='resnet101_afs')
```

```python
if Config.model.fine_tune_layers:
  print(f'Unfreezing {Config.model.fine_tune_layers:.0%} layers.')

  backbone.trainable = True
  frozen_layer_ix = int((1-Config.model.fine_tune_layers) * len(backbone.layers))

  for ix, l in enumerate(backbone.layers):
    l.trainable = (ix > frozen_layer_ix and
                  (not isinstance(l, tf.keras.layers.BatchNormalization) or
                    not Config.model.freeze_batch_norm))

print(f'Loading weights from {Config.model.weights}')
nn.load_weights(Config.model.weights)
```

    Unfreezing 60% layers.
    Loading weights from /home/ldavid/Workspace/logs/amazon-from-space/resnet101-sw-ce-fine-tune/weights.h5

```python
backbone.trainable = False
```

```python
nn_s = tf.keras.Model(
  inputs=nn.inputs,
  outputs=[nn.output, nn.get_layer(Config.model.gap_layer_name).input],
  name='nn_spatial')
```

```python
sW, sb = nn_s.get_layer('predictions').weights
sW = sW * tf.nn.softmax(sW - tf.reduce_max(sW, axis=-1, keepdims=True))
```

## Saliency Methods

```python
DT = 0.5

λ_pos = Config.explaining.λ_pos
λ_neg = Config.explaining.λ_neg
λ_bg = Config.explaining.λ_bg
```

### Min-Max CAM (ours, test ongoing)

$M(f, x)^u_{ij} = \sum_k [w^u_k - \frac{1}{|N_x|} \sum_{n\in N_x} w_k^n] A_{i,j}^k$

Where

$N_x = C_x\setminus \{u\}$

```python
@tf.function
def min_max_sigmoid_cam(x, y):
  print(f'Min-Max CAM (tracing x:{x.shape} p:{y.shape})')

  l, a = nn_s(x, training=False)

  c = len(CLASSES)
  s_n = tf.reduce_sum(sW, axis=-1, keepdims=True)
  s_n = s_n - sW

  w = λ_pos*sW - λ_neg*s_n/(c-1)

  maps = tf.einsum('bhwk,ku->buhw', a, w)
  return l, maps[..., tf.newaxis]
```

```python
@tf.function
def contextual_min_max_sigmoid_cam(x, y):
  print(f'Contextual Min-Max CAM (tracing x:{x.shape} p:{y.shape})')

  l, a = nn_s(x, training=False)
  p = tf.nn.sigmoid(l)

  d = tf.cast(p > DT, tf.float32)
  c = tf.reduce_sum(d, axis=-1)
  c = tf.reshape(c, (-1, 1, 1))     # detections (b, 1)

  w = d[:, tf.newaxis, :] * sW[tf.newaxis, ...]  # expand kernels in d and batches in sW
  w_n = tf.reduce_sum(w, axis=-1, keepdims=True)
  w_n = w_n - w

  w = λ_pos*sW - λ_neg*w_n / tf.maximum(c-1, 1)

  maps = tf.einsum('bhwk,bku->buhw', a, w)
  return l, maps[..., tf.newaxis]
```

#### Contextual ReLU MinMax

$M(f, x)^u_{ij} = \sum_k [w^{u+}_k - \frac{1}{|N_x|} \sum_{n\in N_x} w^{n+}_k +\frac{1}{|C_i|}\sum_{n\in C_i} w^{n-}_k] A_{i,j}^k$

Where

$N_x = C_x\setminus \{u\}$

```python
@tf.function
def contextual_relu_min_max_sigmoid_cam(x, y):
  print(f'Contextual ReLU Min-Max CAM (tracing x:{x.shape} p:{y.shape})')

  l, a = nn_s(x, training=False)
  p = tf.nn.sigmoid(l)

  d = tf.cast(p > DT, tf.float32)
  c = tf.reshape(tf.reduce_sum(d, axis=-1), (-1, 1, 1))  # select only detected

  w = d[:, tf.newaxis, :] * sW[tf.newaxis, ...]  # expand kernels in d and batches in sW
  wa = tf.reduce_sum(w, axis=-1, keepdims=True)
  wn = wa - w

  w = (  λ_pos * tf.nn.relu(sW)
       - λ_neg * tf.nn.relu(wn) / tf.maximum(c-1, 1)
       + λ_bg  * tf.minimum(0., wa) / tf.maximum(c, 1))

  maps = tf.einsum('bhwk,bku->buhw', a, w)
  return l, maps[..., tf.newaxis]
```

```python
@tf.function
def contextual_relu_min_max_sigmoid_cam_2(x, y):
  l, a = nn_s(x, training=False)
  p = tf.nn.sigmoid(l)

  aw = tf.einsum('bhwk,ku->buhw', a, sW)

  d = p > DT
  c = tf.reduce_sum(tf.cast(d, tf.float32), axis=-1)
  c = tf.reshape(c, (-1, 1, 1, 1))

  e = tf.repeat(d[:, tf.newaxis, ...], d.shape[1], axis=1)
  e = tf.linalg.set_diag(e, tf.fill(e.shape[:-1], False))

  z = tf.fill(aw.shape, -np.inf)

  an = tf.where(e[..., tf.newaxis, tf.newaxis], aw[:, tf.newaxis, ...], z[:, tf.newaxis, ...])
  ab = tf.where(d[..., tf.newaxis, tf.newaxis], aw, z)

  an = tf.reduce_max(an, axis=2)
  ab = tf.reduce_max(ab, axis=2, keepdims=True)

  maps = (
      tf.maximum(0., aw)
    - tf.maximum(0., an)
    + tf.minimum(0., ab)
  )

  return l, maps[..., tf.newaxis]
```

### Min-Max Grad-CAM (ours, test ongoing)

$M(f, x)^u_{ij} = \text{relu}(\sum_k \sum_{l,m}\frac{\partial J_u}{\partial A_{l,m}^k} A_{i,j}^k)$

Where

$J_u = S_u - \frac{1}{|N|} \sum_{n\in N_x} S_n$
$N_x = C_x\setminus \{u\}$

```python
def min_max_activation_gain(y, s):
  c = len(CLASSES)
                                                  # shape(s)   == (b, c)
  s_n = tf.reduce_sum(s, axis=-1, keepdims=True)  # shape(s_n) == (b)
  s_n = s_n-s                                     # shape(s_n) == (b, c)

  return λ_pos*s - λ_neg*s_n / (c-1)

@tf.function
def min_max_sigmoid_gradcam(x, y):
  print(f'Min-Max Grad-CAM (tracing x:{x.shape} p:{y.shape})')

  with tf.GradientTape(watch_accessed_variables=False) as t:
    t.watch(x)
    l, a = nn_s(x, training=False)
    loss = min_max_activation_gain(y, l)

  dlda = t.batch_jacobian(loss, a)

  weights = tf.reduce_sum(dlda, axis=(-3, -2))
  maps = tf.einsum('bhwc,buc->buhw', a, weights)

  return l, maps[..., tf.newaxis]
```

```python
def contextual_min_max_activation_gain(y, s, p):
  d = tf.cast(p > DT, tf.float32)
  c = tf.reduce_sum(d, axis=-1, keepdims=True)  # only detections

  sd = s*d
  s_n = tf.reduce_sum(sd, axis=-1, keepdims=True)  # sum logits detected (b, 1)

  return λ_pos*s - λ_neg*(s_n - sd)/tf.maximum(c-1, 1)

@tf.function
def contextual_min_max_sigmoid_gradcam(x, y):
  print(f'Contextual Min-Max Grad-CAM (tracing x:{x.shape} p:{y.shape})')

  with tf.GradientTape(watch_accessed_variables=False) as t:
    t.watch(x)
    l, a = nn_s(x, training=False)
    p = tf.nn.sigmoid(l)
    loss = contextual_min_max_activation_gain(y, l, p)

  dlda = t.batch_jacobian(loss, a)

  weights = tf.reduce_sum(dlda, axis=(-3, -2))
  maps = tf.einsum('bhwc,buc->buhw', a, weights)  # a*weights

  return l, maps[..., tf.newaxis]
```

```python
def contextual_relu_min_max_activation_gain(y, s):
  p = tf.nn.sigmoid(s)
  d = tf.cast(p > DT, tf.float32)
  c = tf.reduce_sum(d, axis=-1, keepdims=True)

  sd = s*d                                        # only detections
  sa = tf.reduce_sum(sd, axis=-1, keepdims=True)  # sum logits detected (b, 1)
  sn = sa - sd

  return tf.stack((
    λ_pos * s,
    λ_neg * sn / tf.maximum(c-1, 1),
    λ_bg  * (sn+sd) / tf.maximum(c, 1)
  ), axis=1)

@tf.function
def contextual_relu_min_max_sigmoid_gradcam(x, y):
  print(f'Contextual ReLU Min-Max Grad-CAM (tracing x:{x.shape} y:{y.shape})')

  with tf.GradientTape(watch_accessed_variables=False) as t:
    t.watch(x)

    l, a = nn_s(x, training=False)
    loss = contextual_relu_min_max_activation_gain(y, l)

  dlda = t.batch_jacobian(loss, a)

  w, wn, wa = dlda[:, 0], dlda[:, 1], dlda[:, 2]

  w = (  tf.nn.relu(w)
       - tf.nn.relu(wn)
       + tf.minimum(0., wa))

  weights = tf.reduce_sum(w, axis=(-3, -2))
  maps = tf.einsum('bhwc,buc->buhw', a, weights)

  return l, maps[..., tf.newaxis]
```

## Qualitative Analysis

```python
#@title

def visualize_explaining_many(
  x,
  y,
  p,
  maps,
  N=None,
  max_detections=3
):
  N = N or len(x)
  plt.figure(figsize=(16, 2*N))
  rows = N
  cols = 1+2*max_detections

  actual = [','.join(CLASSES[_y]) for _y in y.numpy().astype(bool)]
  for ix in range(N):
    detections = p[ix] > DT

    visualize_explaining(
      x[ix],
      actual[ix],
      detections,
      p[ix],
      maps[ix],
      i0=cols*ix,
      rows=rows,
      cols=cols,
      full=False,
      max_detections=max_detections
    )

  plt.tight_layout()

def visualize_explaining(image,
                         labels,
                         detections,
                         probs,
                         cams,
                         full=True,
                         i0=0,
                         rows=2,
                         cols=None,
                         max_detections=3):
  detections = detections.numpy()

  im = to_image(image)
  _maps = tf.boolean_mask(cams, detections)
  _maps = tf.image.resize(_maps, Config.data.size)
  _masked = to_image(masked(image, _maps))

  plots = [im, *_maps[:max_detections]]
  title = [labels] + [f'{d} {p:.0%}' for d, p in zip(CLASSES[detections], probs.numpy()[detections])]

  visualize(plots, title, full=full, rows=rows, cols=cols, i0=i0)

  for ix, s in enumerate(_maps[:max_detections]):
    plt.subplot(rows, cols, i0+len(plots)+ix+1)
    plot_heatmap(im, s[..., 0])
```

```python
cams = {}

for x, y in valid.take(1):
  l = tf.convert_to_tensor(nn.predict(x))
  p = tf.nn.sigmoid(l)

  # Only samples with two or more objects:
  s = tf.reduce_sum(tf.cast(p > DT, tf.int32), axis=1) > 1
  x, y, l, p = x[s], y[s], l[s], p[s]
```

#### CAM

```python
_, maps = sigmoid_cam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)

cams['cam'] = maps
```

```python
visualize_explaining_many(x, y, p, maps)
```

#### Grad-CAM

```python
_, maps = sigmoid_gradcam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
# visualize_explaining_many(x, y, p, maps)

cams['gradcam'] = maps
```

#### Grad-CAM++

```python
_, maps = sigmoid_gradcampp(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
# visualize_explaining_many(x, y, p, maps)

cams['gradcampp'] = maps
```

### Score-CAM

```python
_, maps = sigmoid_scorecam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
visualize_explaining_many(x, y, p, maps)

cams['scorecam'] = maps
```

### Min-Max CAM

#### Vanilla

```python
%%time

_, maps = min_max_sigmoid_cam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
# visualize_explaining_many(x, y, p, maps)

cams['minmax_cam'] = maps
```

#### Contextual

```python
%%time

_, maps = contextual_min_max_sigmoid_cam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
# visualize_explaining_many(x, y, p, maps)

cams['contextual_minmax_cam'] = maps
```

#### Contextual ReLU

```python
%%time

_, maps = contextual_relu_min_max_sigmoid_cam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
# visualize_explaining_many(x, y, p, maps)

cams['contextual_relu_minmax_cam'] = maps
```

```python
%%time

_, maps = contextual_relu_min_max_sigmoid_cam_2(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
# visualize_explaining_many(x, y, p, maps)

cams['contextual_relu_minmax_cam_2'] = maps
```

### Min-Max Grad-CAM

#### Vanilla

```python
_, maps = min_max_sigmoid_gradcam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
visualize_explaining_many(x, y, p, maps)

cams['minmax_gradcam'] = maps
```

#### Contextual

```python
_, maps = contextual_min_max_sigmoid_gradcam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
visualize_explaining_many(x, y, p, maps)

cams['contextual_minmax_gradcam'] = maps
```

#### Contextual ReLU

```python
_, maps = contextual_relu_min_max_sigmoid_gradcam(x, y)

maps = tf.nn.relu(maps)
maps = normalize(maps)
visualize_explaining_many(x, y, p, maps)

cams['contextual_relu_minmax_gradcam'] = maps
```

### Summary

```python
observing = 'cam minmax_cam contextual_minmax_cam contextual_relu_minmax_cam'.split()
titles = 'Input CAM MM C-MM CG-MM'.split()

print('Results selected for vis:', *observing, sep='\n  ')
```

```python
#@title

detections = p > DT
indices = tf.where(detections)
sample_ix, label_ix = indices[:, 0], indices[:, 1]

visualize(
  sum(zip(to_image(tf.gather(x, sample_ix)[:48]).numpy(),
          *(tf.image.resize(cams[c][detections][:48], Config.data.size).numpy()
            for c in observing)),
      ()),
  title=titles,
  rows=sample_ix[:48].shape[0],
  figsize=(12, 80)
);
```

```python
#@title

plt.figure(figsize=(12, 80))

selected_images = to_image(tf.gather(x, sample_ix)[:48]).numpy()

rows = len(selected_images)
cols = len(observing) + 1

for ix, im in enumerate(selected_images):
  plt.subplot(rows, cols, ix*cols + 1)
  plt.imshow(im)
  plt.axis('off')

  for j, method in enumerate(observing):
    map = cams[method][detections][ix]
    map = tf.image.resize(map, Config.data.size)

    plt.subplot(rows, cols, ix*cols + j + 2)
    plot_heatmap(im, map[..., 0])

    if ix == 0:
      plt.title(titles[j+1])

plt.tight_layout()
```

## Quantitative Analysis

#### Metrics

##### **Increase in Confidence %**
> *Removing background noise should improve confidence  (higher=better)*

$\frac{1}{∑ |C_i|} ∑^N_i∑^{C_i}_c [Y^c_i < O_{ic}^c] 100$

Thought: this probably works better for *softmax* classifiers.

```python
def increase_in_confidence(
    p,  # f(x)             (batch, classes)
    y,  # f(x)[f(x) > 0.5] (detections, 1)
    o,  # f(x*mask(x, m))  (detections, classes)
    samples_ix,
    units_ix
):
  oc = tf.gather(o, units_ix, axis=1, batch_dims=1)  # (detections, 1)

  incr = np.zeros(p.shape, np.uint32)
  incr[samples_ix, units_ix] = tf.cast(y < oc, tf.uint32).numpy()

  return incr.sum(axis=0)
```

##### **Average Drop %**
> *Masking with an accurate mask should not decrease confidence (lower=better)*

$\frac{1}{∑ |C_i|} ∑_i^N ∑_c^{C_i} \frac{max(0, Y_i^c − O_{ic}^c)}{Y_i^c} 100$

Measures if your mask is correctly positioned on top of the important regions that determine the class of interest.

```python
def average_drop(p, y, o, samples_ix, units_ix):
  oc = tf.gather(o, units_ix, axis=1, batch_dims=1)

  drop = np.zeros(p.shape)
  drop[samples_ix, units_ix] = (tf.nn.relu(y - oc) / y).numpy()

  return drop.sum(axis=0)
```

##### **Average Retention % (ours, testing ongoing)**
> *Masking the input with an accurate complement mask for class $c$ should decrease confidence in class $c$ (higher=better)*

$\frac{1}{∑ |C_i|} ∑_i^N ∑_c^{C_i} \frac{max(0, Y_i^c − \bar{O}_{ic}^c)}{Y_i^c} 100$

Where $\bar{O}_{ic}^c = f(x_i \circ (1-\psi(M(f, x_i)_{hw}^c))^c$

Masking the input $x_i$ for all classes except $c$ should cause the model's confidence in $c$ to drop.

###### **Average Drop & Retention % (ours)**

\begin{align}
\frac{\text{drop} + (1-\text{retention})}{2}
&= \frac{1}{2 ∑ |C_i|} ∑_i^N ∑_c^{C_i} [\frac{max(0, Y_i^c − O_{ic}^c)}{Y_i^c} + (1-\frac{max(0, Y_i^c − \bar{O}_{ic}^c)}{Y_i^c})] 100 \\
&= \frac{1}{2 ∑ |C_i|} ∑_i^N ∑_c^{C_i} [1 + \frac{Y_i^c − O_{ic}^c - (Y_i^c − \bar{O}_{ic}^c)}{Y_i^c}] 100 \\
&= \frac{1}{2 ∑ |C_i|} ∑_i^N ∑_c^{C_i} [1 + \frac{\bar{O}_{ic}^c − O_{ic}^c}{Y_i^c}] 100
\end{align}

Where

* $O_{ic}^c = f(x_i \circ \psi(M(f, x_i)_{hw}^c)^c$
* $\bar{O}_{ic}^c = f(x_i \circ (1-\psi(M(f, x_i)_{hw}^c))^c$

##### **Average Drop of Others % (ours, testing ongoing)**

> *An ideal mask for class $c$ is not retaining any objects of other classes (higher=better)*

$\frac{1}{∑ |C_i|} ∑_i^N ∑_c^{C_i} \frac{1}{|D_i|} ∑_d^{D_i} \frac{max(0, Y_i^d − O_{ic}^d)}{Y_i^d} 100$

Masking the input $x_i$ for a given class $c$ should cause the confidence in other classes to drop.

I.e., $f(x_i\circ \psi(M(f, x_i)^c_{hw}))^d \sim 0, \forall d\in D_i = C_i\setminus \{c\}$.

For single-label problems, $D_i = \emptyset$ and Average Drop of Others is not defined. How to solve this?

```python
def average_drop_of_others(p, s, y, o, samples_ix, units_ix):
  # Drop of all units, for all detections
  d = tf.gather(p, samples_ix)
  d = tf.nn.relu(d - o) / d

  # Remove drop of class `c` and non-detected classes
  detected = tf.cast(tf.gather(s, samples_ix), tf.float32)
  d = d*detected
  d = tf.reduce_sum(d, axis=-1) - tf.gather(d, units_ix, axis=1, batch_dims=1)
  c = tf.reduce_sum(detected, axis=-1)

  # Normalize by the number of peer labels for detection `c`
  d = d / tf.maximum(1., c -1)

  drop = np.zeros(p.shape)
  drop[samples_ix, units_ix] = d.numpy()

  return drop.sum(axis=0)
```

##### **Average Retention of Others % (ours, testing ongoing)**

> *An ideal mask complement for class $c$ should cover all objects of the other classes (lower=better)*

$\frac{1}{∑ |D_i|} ∑_i^N ∑_c^{C_i} \frac{1}{|D_i|} ∑_d^{D_i} \frac{max(0, Y_i^d − \bar{O}_{ic}^d)}{Y_i^d} 100$

Masking the input $x_i$ for all classes except $c$ should cause the confidence in other classes to stay the same or increase.

I.e., $f(x_i\circ (1-\psi(M(f, x_i)^c_{ij})))^d \approx f(x_i)^d, \forall d\in D_i = C_i\setminus \{c\}$.

#### Experiments

```python
#@title Testing Loop

def experiment_with(dataset, cam_method, cam_modifier):
  print(f'Testing {cam_method.__name__}')
  t = time()
  r = cam_evaluation(nn, dataset, cam_method=cam_method, cam_modifier=cam_modifier)
  print(f'elapsed: {(time() - t)/60:.1f} minutes', end='\n\n')

  return r.assign(method=cam_method.__name__)

def cam_evaluation(nn, dataset, cam_method, cam_modifier):
  metric_names = ('increase %', 'avg drop %', 'avg retention %',
                  'avg drop of others %', 'avg retention of others %',
                  'detections')
  metrics = (np.zeros(len(CLASSES), np.uint16),
             np.zeros(len(CLASSES)),
             np.zeros(len(CLASSES)),
             np.zeros(len(CLASSES)),
             np.zeros(len(CLASSES)),
             np.zeros(len(CLASSES), np.uint16))

  try:
    for step, (x, y) in enumerate(dataset):
      p, maps = cam_method(x, y)
      p = tf.nn.sigmoid(p)
      maps = cam_modifier(maps)

      for e, f in zip(metrics, cam_evaluation_step(nn, x, p, maps)):
        e += f

      print('.', end='' if (step+1) % 80 else '\n')
    print()

  except KeyboardInterrupt:
    print('interrupted')

  metrics, detections = metrics[:-1], metrics[-1]

  results = {n: 100*m/detections for n, m in zip(metric_names, metrics)}
  results['label'] = CLASSES
  results['detections'] = detections
  results = pd.DataFrame(results)

  print(f'Average Drop     %: {results["avg drop %"].mean():.4}%')
  print(f'Average Increase %: {results["increase %"].mean():.4}%')

  return results

def cam_evaluation_step(nn, x, p, m):
  s = p > DT
  w = tf.where(s)
  samples_ix, units_ix = w[:, 0], w[:, 1]

  md = tf.image.resize(m[s], Config.data.size)

  detections = tf.reduce_sum(tf.cast(s, tf.uint32), axis=0)

  y = p[s]                       # (batch, c)           --> (detections)
  xs = tf.gather(x, samples_ix)  # (batch, 300, 300, 3) --> (detections, 300, 300, 3)

  o = nn.predict(masked(xs, md), batch_size=Config.data.batch_size)
  o = tf.nn.sigmoid(o)

  co = nn.predict(masked(xs, 1 -md), batch_size=Config.data.batch_size)
  co = tf.nn.sigmoid(co)

  samples_ix, units_ix = samples_ix.numpy(), units_ix.numpy()

  incr = increase_in_confidence(p, y, o, samples_ix, units_ix)
  drop = average_drop(p, y, o, samples_ix, units_ix)
  rete = average_drop(p, y, co, samples_ix, units_ix)

  drop_of_others = average_drop_of_others(p, s, y, o, samples_ix, units_ix)
  rete_of_others = average_drop_of_others(p, s, y, co, samples_ix, units_ix)

  return incr, drop, rete, drop_of_others, rete_of_others, detections.numpy()
```

```python
methods_being_tested = (
  # Baseline
  # sigmoid_cam,
  # sigmoid_gradcam,

  # Best solutions
  # sigmoid_gradcampp,
  sigmoid_scorecam,

  # Ours
  # min_max_sigmoid_cam,
  # contextual_min_max_sigmoid_cam,
  # contextual_relu_min_max_sigmoid_cam,
  # contextual_relu_min_max_sigmoid_cam_2,
)

relu_and_normalize = lambda c: normalize(tf.nn.relu(c))
```

```python
results = pd.concat(
  [
    experiment_with(valid, m, relu_and_normalize)
    for m in methods_being_tested
  ]
)
```

    Testing sigmoid_scorecam
    Score-CAM feed tracing
    ................................................................................
    ................................................................................
    ................................................................................
    ................................................................................
    ...........................................................
    Average Drop     %: 39.5%
    Average Increase %: 14.13%
    elapsed: 1745.5 minutes

    <ipython-input-42-4d72ab601d27>:40: RuntimeWarning: invalid value encountered in true_divide
      results = {n: 100*m/detections for n, m in zip(metric_names, metrics)}

```python
# if os.path.exists(Config.report):
#   raise FileExistsError('You are asking me to override a report file.')

results.to_csv(Config.report, index=False)
```

#### Report

```python
results = pd.read_csv(Config.report)
```

```python
methods = (
  'sigmoid_cam',
  'sigmoid_gradcampp',
  'sigmoid_scorecam',
  'min_max_sigmoid_cam',
  'contextual_min_max_sigmoid_cam',
  'contextual_relu_min_max_sigmoid_cam',
)

methods_detailed = (
  'sigmoid_cam',
  'sigmoid_scorecam',
  'contextual_min_max_sigmoid_cam',
  'contextual_relu_min_max_sigmoid_cam'
)

metric_names = (
  'increase %',
  'avg drop %',
  'avg retention %',
  'avg drop of others %',
  'avg retention of others %',
  'f1 score',
  'f1 score negatives'
)

minimizing_metrics = {'avg drop %', 'avg retention of others %', 'f1 score negatives'}
```

```python
def fb_score(a, b, beta=1):
  beta2 = beta**2
  denom = (beta2 * b + a)
  denom[denom == 0.] = 1
  return (1+beta2) * a * b / denom

results['f1/2 score'] = fb_score(results['avg retention %'], results['avg drop of others %'], beta=1/2)
results['f1 score']   = fb_score(results['avg retention %'], results['avg drop of others %'], beta=1)
results['f2 score']   = fb_score(results['avg retention %'], results['avg drop of others %'], beta=2)

results['f1/2 score negatives']   = fb_score(results['avg drop %'], results['avg retention of others %'], beta=1/2)
results['f1 score negatives']   = fb_score(results['avg drop %'], results['avg retention of others %'], beta=1)
results['f2 score negatives']   = fb_score(results['avg drop %'], results['avg retention of others %'], beta=2)
```

```python
(results
 .drop('detections', axis=1)
 .groupby('method')
 .mean()
 .round(4)/100)
```

      increase %
      avg drop %
      avg retention %
      avg drop of others %
      avg retention of others %
      f1/2 score
      f1 score
      f2 score
      f1/2 score negatives
      f1 score negatives
      f2 score negatives

      method

      sigmoid_scorecam
      0.141259
      0.395045
      0.436119
      0.352102
      0.267954
      0.064794
      0.172319
      0.309953
      0.050752
      0.133086
      0.247012

```python
#@title Macro Average (Class-Balanced)

macro_avg = (
  results
  .groupby('method')
  .mean()
  .reindex(methods)[list(metric_names)]
) / 100

macro_avg_hm = macro_avg.copy()
for m in minimizing_metrics:
  macro_avg_hm[m] = 1-macro_avg_hm[m]
macro_avg_hm -= macro_avg_hm.min(axis=0)
macro_avg_hm /= macro_avg_hm.max(axis=0) + 1e-07

plt.figure(figsize=(6, 6))
sns.heatmap(
  macro_avg_hm,
  fmt='.2%',
  annot=macro_avg,
  cmap='RdPu',
  cbar=False,
  xticklabels=[c.replace('_', ' ') for c in macro_avg.columns],
  yticklabels=[i.replace('_', ' ') for i in macro_avg.index],
);
```

```python
#@title Weighted Average (Class Frequency Weighted)

total_detections = (
  results
  .groupby('method')
  .agg({'detections': 'sum'})
  .rename(columns={'detections': 'total_detections'})
)

w_avg = results.merge(total_detections, how='left', left_on='method', right_index=True)

metric_results = {
  m: w_avg[m] * w_avg.detections / w_avg.total_detections
  for m in metric_names
}
metric_results['method'] = w_avg.method
metric_results['label'] = w_avg.label

w_avg = (
  pd.DataFrame(metric_results)
  .groupby('method')
  .sum()
  .reindex(methods) / 100
)

hm = w_avg.copy()
for m in minimizing_metrics:
  hm[m] = 1-hm[m]
hm -= hm.min(axis=0)
hm /= hm.max(axis=0) + 1e-07

plt.figure(figsize=(6, 6))
sns.heatmap(
  hm,
  fmt='.2%',
  annot=w_avg,
  cmap='RdPu',
  cbar=False,
  xticklabels=[c.replace('_', ' ') for c in w_avg.columns],
  yticklabels=[i.replace('_', ' ') for i in w_avg.index]
);
```

```python
#@title Detailed Results per Class

plt.figure(figsize=(16, 6))
sns.boxplot(
  data=results[results.method.isin(methods_detailed)]
       .melt(('method', 'label'), metric_names, 'metric'),
  hue='method',
  x='metric',
  y='value'
);
```

```python
#@title F1 Score by Label and CAM Method

plt.figure(figsize=(16, 6))
sns.barplot(
  data=results.sort_values('f1 score', ascending=False),
  hue='method',
  y='f1 score',
  x='label'
)
plt.xticks(rotation=-45);
```