# 2021/12/26/SUN

```python
import numpy as np
import pandas as pd
from openpyxl import load_workbook
```

```python
data = {'Name' : ['S1','S2','S3'],
       'Age' : [25,28,22],
       'Score' : np.array([95,80,75])}
```

```python
data
```

    {'Name': ['S1', 'S2', 'S3'], 'Age': [25, 28, 22], 'Score': array([95, 80, 75])}

```python
type(data)
```

    dict

```python
data['Name']
```

    ['S1', 'S2', 'S3']

```python
# dict을 df로 활용할 수 있다
df=pd.DataFrame(data,index=['row1','row2','row3'])
df
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

- 리스트도 df로 변경 가능

```python
data2=[['S1',25,95],['S2',28,80],['S3',22,75]]
```

```python
data2
```

    [['S1', 25, 95], ['S2', 28, 80], ['S3', 22, 75]]

```python
df2=pd.DataFrame(data2,index=['row1','row2','row3'],columns=['Name','Age','Score'])
```

```python
df2
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

- 결론 : list와 dict은 df로 반환시킬 수 있다.

---

- Subset Observation 부분 관찰해보자

```python
df
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

```python
df[['Name']]
```

      Name

      row1
      S1

      row2
      S2

      row3
      S3

```python
df['Name']
```

    row1    S1
    row2    S2
    row3    S3
    Name: Name, dtype: object

```python
df[['Name','Score']]
# 이렇게 list안에 넣어줘야 함
```

      Name
      Score

      row1
      S1
      95

      row2
      S2
      80

      row3
      S3
      75

```python
# df['row1']
# 행은 이렇게 추출할 수 없음
```

그렇다면 행 추출은?

```python
df.loc[['row1']]
```

      Name
      Age
      Score

      row1
      S1
      25
      95

```python
df.loc['row1']
```

    Name     S1
    Age      25
    Score    95
    Name: row1, dtype: object

```python
# 인덱스를 리스트 안에 넣어서 뽑아주자
df.loc[['row1','row2']]
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

```python
# 행은 row1이고 열은 Name에서 뽑아보자
df.loc['row1','Name']
```

    'S1'

```python
# df.loc['Name'] 이건 안 됨. 행만 가능
df['Name']
```

    row1    S1
    row2    S2
    row3    S3
    Name: Name, dtype: object

- 사용하고 싶다면 이렇게 사용

```python
df.loc[:,'Name']
```

    row1    S1
    row2    S2
    row3    S3
    Name: Name, dtype: object

```python
df.loc[:,['Name']]
```

      Name

      row1
      S1

      row2
      S2

      row3
      S3

```python
df.loc[:,['Score','Name']]
```

      Score
      Name

      row1
      95
      S1

      row2
      80
      S2

      row3
      75
      S3

```python
# : 사용할 땐 중괄호 X
df.loc[:,'Name':'Age']
```

      Name
      Age

      row1
      S1
      25

      row2
      S2
      28

      row3
      S3
      22

```python
df.iloc[0,0]
```

    'S1'

```python
df.iloc[2,0]
```

    'S3'

```python
df.iloc[:,[0,2]]
```

      Name
      Score

      row1
      S1
      95

      row2
      S2
      80

      row3
      S3
      75

```python
# : 사용할 땐 중괄호 사용 X
df.iloc[:,0:2]
```

      Name
      Age

      row1
      S1
      25

      row2
      S2
      28

      row3
      S3
      22

```python
df.iloc[::2,[0,2]]
```

      Name
      Score

      row1
      S1
      95

      row3
      S3
      75

```python
df.iloc[[-1],:]
```

      Name
      Age
      Score

      row3
      S3
      22
      75

```python
df.iloc[-1,:]
```

    Name     S3
    Age      22
    Score    75
    Name: row3, dtype: object

```python
df
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

```python
df.iloc[-1::-1,:]
```

      Name
      Age
      Score

      row3
      S3
      22
      75

      row2
      S2
      28
      80

      row1
      S1
      25
      95

```python
df.head(2)
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

```python
df.tail(1)
```

      Name
      Age
      Score

      row3
      S3
      22
      75

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Index: 3 entries, row1 to row3
    Data columns (total 3 columns):
     #   Column  Non-Null Count  Dtype
    ---  ------  --------------  -----
     0   Name    3 non-null      object
     1   Age     3 non-null      int64
     2   Score   3 non-null      int32
    dtypes: int32(1), int64(1), object(1)
    memory usage: 192.0+ bytes

```python
df.describe()
# 통계적 수치들을 알 수 있음
```

      Age
      Score

      count
      3.0
      3.000000

      mean
      25.0
      83.333333

      std
      3.0
      10.408330

      min
      22.0
      75.000000

      25%
      23.5
      77.500000

      50%
      25.0
      80.000000

      75%
      26.5
      87.500000

      max
      28.0
      95.000000

- deep copy $\to$ copy
- shallow copy $\to$ view

```python
df2=df.copy()
```

```python
df2
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

```python
df2.loc['row2','Score']=np.NaN
```

```python
df2
```

      Name
      Age
      Score

      row1
      S1
      25
      95.0

      row2
      S2
      28
      NaN

      row3
      S3
      22
      75.0

```python
df
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

deep copy

```python
# nunique = n(umber)unique
df2.nunique()
```

    Name     3
    Age      3
    Score    2
    dtype: int64

```python
df2['Score'].nunique()
```

    2

```python
df2
```

      Name
      Age
      Score

      row1
      S1
      25
      95.0

      row2
      S2
      28
      NaN

      row3
      S3
      22
      75.0

```python
df2['Score'].value_counts()
# 각 값들의 갯수를 알 수 있음
```

    95.0    1
    75.0    1
    Name: Score, dtype: int64

```python
df3=df2.copy()
df3.loc['row3','Score']=df2.loc['row1','Score']
```

```python
df3['Score'].value_counts()
```

    95.0    2
    Name: Score, dtype: int64

```python
df3['Score'].count()
```

    2

- value_counts() = 각 값 별 몇 개?
- count = 그냥 총 값이 몇 개 들어가 있는지?

```python
df3['Age'].count()
```

    3

```python
df['Score'].sum()
```

    250

```python
df.max()
```

    Name     S3
    Age      28
    Score    95
    dtype: object

```python
df['Score'].std()
```

    10.408329997330664

```python
df.describe()
```

      Age
      Score

      count
      3.0
      3.000000

      mean
      25.0
      83.333333

      std
      3.0
      10.408330

      min
      22.0
      75.000000

      25%
      23.5
      77.500000

      50%
      25.0
      80.000000

      75%
      26.5
      87.500000

      max
      28.0
      95.000000

---

```python
df4=df.copy()
```

```python
df4
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

```python
df4=df4.iloc[:,[0,2,1]]
```

```python
df4
```

      Name
      Age
      Score

      row1
      S1
      25
      95

      row2
      S2
      28
      80

      row3
      S3
      22
      75

df의 열을 변경해서 df4에 저장

```python
data={
    'class' : ['A','B','C','A','B','C','C'],
    'name' : ['S1','S2','S3','S4','S5','S6','S7'],
    'age' : [20,19,21,22,24,25,26],
    'score' : [90,95,75,80,70,85,90]}
```

```python
df=pd.DataFrame(data)
df
```

      class
      name
      age
      score

      0
      A
      S1
      20
      90

      1
      B
      S2
      19
      95

      2
      C
      S3
      21
      75

      3
      A
      S4
      22
      80

      4
      B
      S5
      24
      70

      5
      C
      S6
      25
      85

      6
      C
      S7
      26
      90

- 이처럼 dict을 df에 활용할 수 있다.

```python
df['score']>=80
```

    0     True
    1     True
    2    False
    3     True
    4    False
    5     True
    6     True
    Name: score, dtype: bool

```python
df.loc[df['score']>=80]
```

      class
      name
      age
      score

      0
      A
      S1
      20
      90

      1
      B
      S2
      19
      95

      3
      A
      S4
      22
      80

      5
      C
      S6
      25
      85

      6
      C
      S7
      26
      90

```python
# 행은 score가 80이상, 열은 name,age
df.loc[df['score']>=80,['name','age']]
```

      name
      age

      0
      S1
      20

      1
      S2
      19

      3
      S4
      22

      5
      S6
      25

      6
      S7
      26

```python
df[df['score']>=80]
```

      class
      name
      age
      score

      0
      A
      S1
      20
      90

      1
      B
      S2
      19
      95

      3
      A
      S4
      22
      80

      5
      C
      S6
      25
      85

      6
      C
      S7
      26
      90

```python
df['result']='NONE'
```

```python
df
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      90
      NONE

      1
      B
      S2
      19
      95
      NONE

      2
      C
      S3
      21
      75
      NONE

      3
      A
      S4
      22
      80
      NONE

      4
      B
      S5
      24
      70
      NONE

      5
      C
      S6
      25
      85
      NONE

      6
      C
      S7
      26
      90
      NONE

```python
df.loc[df['score']>=80,'result']='PASS'
```

```python
df.loc[df['score']<80,'result']='FAIL'
```

```python
df
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      90
      PASS

      1
      B
      S2
      19
      95
      PASS

      2
      C
      S3
      21
      75
      FAIL

      3
      A
      S4
      22
      80
      PASS

      4
      B
      S5
      24
      70
      FAIL

      5
      C
      S6
      25
      85
      PASS

      6
      C
      S7
      26
      90
      PASS

```python
idx = df['result'] == 'PASS'
```

```python
df.loc[idx]
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      90
      PASS

      1
      B
      S2
      19
      95
      PASS

      3
      A
      S4
      22
      80
      PASS

      5
      C
      S6
      25
      85
      PASS

      6
      C
      S7
      26
      90
      PASS

```python
df_sorted=df.loc[idx].sort_values('score',ascending=False)
```

```python
df_sorted
```

      class
      name
      age
      score
      result

      1
      B
      S2
      19
      95
      PASS

      0
      A
      S1
      20
      90
      PASS

      6
      C
      S7
      26
      90
      PASS

      5
      C
      S6
      25
      85
      PASS

      3
      A
      S4
      22
      80
      PASS

- excel 파일 만들고 불러오기

```python
df_sorted.to_excel('data_sorted.xlsx',index=False)
```

```python
df_import=pd.read_excel('data_sorted.xlsx')
```

```python
df_import
```

      class
      name
      age
      score
      result

      0
      B
      S2
      19
      95
      PASS

      1
      A
      S1
      20
      90
      PASS

      2
      C
      S7
      26
      90
      PASS

      3
      C
      S6
      25
      85
      PASS

      4
      A
      S4
      22
      80
      PASS

```python
df.groupby(by='class').mean()
# mean처리 할 수 있는 열에 대해서만 mean 처리 함
```

      age
      score

      class

      A
      21.0
      85.000000

      B
      21.5
      82.500000

      C
      24.0
      83.333333

```python
df.groupby(by='class').std()
```

      age
      score

      class

      A
      1.414214
      7.071068

      B
      3.535534
      17.677670

      C
      2.645751
      7.637626

plotting

```python
df
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      90
      PASS

      1
      B
      S2
      19
      95
      PASS

      2
      C
      S3
      21
      75
      FAIL

      3
      A
      S4
      22
      80
      PASS

      4
      B
      S5
      24
      70
      FAIL

      5
      C
      S6
      25
      85
      PASS

      6
      C
      S7
      26
      90
      PASS

```python
df.plot.bar('name','score')
```

```python
df.loc[[0,2],'score']=np.NaN
```

```python
df
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      NaN
      PASS

      1
      B
      S2
      19
      95.0
      PASS

      2
      C
      S3
      21
      NaN
      FAIL

      3
      A
      S4
      22
      80.0
      PASS

      4
      B
      S5
      24
      70.0
      FAIL

      5
      C
      S6
      25
      85.0
      PASS

      6
      C
      S7
      26
      90.0
      PASS

```python
# NaN이 들어간 곳을 알려줌
df.isnull()
```

      class
      name
      age
      score
      result

      0
      False
      False
      False
      True
      False

      1
      False
      False
      False
      False
      False

      2
      False
      False
      False
      True
      False

      3
      False
      False
      False
      False
      False

      4
      False
      False
      False
      False
      False

      5
      False
      False
      False
      False
      False

      6
      False
      False
      False
      False
      False

```python
df.dropna()
# 데이터 없는 행은 다 날림
```

      class
      name
      age
      score
      result

      1
      B
      S2
      19
      95.0
      PASS

      3
      A
      S4
      22
      80.0
      PASS

      4
      B
      S5
      24
      70.0
      FAIL

      5
      C
      S6
      25
      85.0
      PASS

      6
      C
      S7
      26
      90.0
      PASS

```python
value=0
df.fillna(value) ## NaN값만 value로 채워줌
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      0.0
      PASS

      1
      B
      S2
      19
      95.0
      PASS

      2
      C
      S3
      21
      0.0
      FAIL

      3
      A
      S4
      22
      80.0
      PASS

      4
      B
      S5
      24
      70.0
      FAIL

      5
      C
      S6
      25
      85.0
      PASS

      6
      C
      S7
      26
      90.0
      PASS

```python
# 이것도 가능
df.replace(np.nan,54)
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      54.0
      PASS

      1
      B
      S2
      19
      95.0
      PASS

      2
      C
      S3
      21
      54.0
      FAIL

      3
      A
      S4
      22
      80.0
      PASS

      4
      B
      S5
      24
      70.0
      FAIL

      5
      C
      S6
      25
      85.0
      PASS

      6
      C
      S7
      26
      90.0
      PASS

```python
# 주위 data를 이용하여 채워줄 수 있음
df.interpolate()
```

      class
      name
      age
      score
      result

      0
      A
      S1
      20
      NaN
      PASS

      1
      B
      S2
      19
      95.0
      PASS

      2
      C
      S3
      21
      87.5
      FAIL

      3
      A
      S4
      22
      80.0
      PASS

      4
      B
      S5
      24
      70.0
      FAIL

      5
      C
      S6
      25
      85.0
      PASS

      6
      C
      S7
      26
      90.0
      PASS

- 이렇게 위 아래 값의 평균으로 채워주기도 하는데 위아래 둘다 있는 경우에만 사용가능하다

```python
# function
def add_one(x):
    return x+1
```

```python
add_one(1001)
```

    1002

- apply 활용하기

```python
df['age']=df['age'].apply(add_one)
```

```python
df
```

      class
      name
      age
      score
      result

      0
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS

이렇게 df의 age를 변경해줄 수 있음

```python
df['score'].apply(np.square)
```

    0       NaN
    1    9025.0
    2       NaN
    3    6400.0
    4    4900.0
    5    7225.0
    6    8100.0
    Name: score, dtype: float64

---

```python
df
```

      class
      name
      age
      score
      result

      0
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS

```python
df.filter(regex='[rn]')
# n 또는 r이 들어간 columns 추출
```

      name
      score
      result

      0
      S1
      NaN
      PASS

      1
      S2
      95.0
      PASS

      2
      S3
      NaN
      FAIL

      3
      S4
      80.0
      PASS

      4
      S5
      70.0
      FAIL

      5
      S6
      85.0
      PASS

      6
      S7
      90.0
      PASS

```python
df_vertical=pd.concat([df,df])
df_vertical
```

      class
      name
      age
      score
      result

      0
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS

      0
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS

```python
df_vertical=pd.concat([df,df],ignore_index=True)
df_vertical
```

      class
      name
      age
      score
      result

      0
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS

      7
      A
      S1
      22
      NaN
      PASS

      8
      B
      S2
      21
      95.0
      PASS

      9
      C
      S3
      23
      NaN
      FAIL

      10
      A
      S4
      24
      80.0
      PASS

      11
      B
      S5
      26
      70.0
      FAIL

      12
      C
      S6
      27
      85.0
      PASS

      13
      C
      S7
      28
      90.0
      PASS

```python
df_horizontal=pd.concat([df,df],axis=1)
df_horizontal
```

      class
      name
      age
      score
      result
      class
      name
      age
      score
      result

      0
      A
      S1
      22
      NaN
      PASS
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS
      C
      S7
      28
      90.0
      PASS

---

```python
df.to_csv('data_text.txt',sep='\t',index=False)
# txt파일로 변환할 때 어떻게 구분해서 타이핑해넣을 것인가
# sep=\t,여기선 지금 탭으로 구분지었음
```

```python
pd.read_csv('data_text.txt',delimiter='\t')
# delimiter로 txt파일이 어떻게 이루어져 있나 알려줘야함
```

      class
      name
      age
      score
      result

      0
      A
      S1
      22
      NaN
      PASS

      1
      B
      S2
      21
      95.0
      PASS

      2
      C
      S3
      23
      NaN
      FAIL

      3
      A
      S4
      24
      80.0
      PASS

      4
      B
      S5
      26
      70.0
      FAIL

      5
      C
      S6
      27
      85.0
      PASS

      6
      C
      S7
      28
      90.0
      PASS

----

```python
from sympy import symbols
```

```python
x=symbols('x')
```

```python
type(x)
```

    sympy.core.symbol.Symbol

```python
2*x
```

$\displaystyle 2 x$

즉 x자체가 symbol로 들어갔음

```python
expr=2*x
```

```python
expr.subs(x,3)
```

$\displaystyle 6$

미분해보자

```python
f=x**3
```

```python
from sympy import diff
```

```python
df1=diff(f,x)
```

```python
df1
```

$\displaystyle 3 x^{2}$

```python
df2=diff(df1,x)
```

```python
df2
```

$\displaystyle 6 x$

---

```python
from sympy import sin
```

```python
f=sin(x)
```

```python
df1=diff(f,x)
```

```python
df1
```

$\displaystyle \cos{\left(x \right)}$

----

```python
from sympy import integrate
```

```python
integrate(f,(x,0,2*3.14))
```

$\displaystyle 5.07308662478501 \cdot 10^{-6}$

0에 가깝게 나옴

```python
integrate(f,(x,0,3.14))
```

$\displaystyle 1.99999873172754$

---

```python
from sympy import limit
```

```python
limit(sin(x)/x,x,0)
```

$\displaystyle 1$

----

```python
import matplotlib.pyplot as plt
from scipy import interpolate
```

```python
x=np.array([1,2,3,4,5])
y=np.array([1,0.8,0.4,0.3,0.2])
```

```python
plt.plot(x,y,'*')
```

```python
f_lin=interpolate.interp1d(x,y)
```

```python
x_new = np.arange(1,5,0.1)
```

```python
y_new = f_lin(x_new)
```

```python
fig,ax=plt.subplots()
ax.plot(x,y,'o',label='Data')
ax.plot(x_new,y_new,label='linear')
ax.legend()
```

선형 보간됨을 알 수 있다.

----

```python
tck=interpolate.splrep(x,y,s=0)
y_spl=interpolate.splev(x_new,tck,der=0)
fig,ax=plt.subplots()
ax.plot(x,y,'o',label='Data')
ax.plot(x_new,y_new,label='linear')
ax.plot(x_new,y_spl,label='spline')
ax.legend()
```

원형 보간 추가

------

> #### `BGR 플랏`

```python
import cv2 as cv
im=cv.imread('KakaoTalk_20211210_090822964.png')
plt.figure()
plt.imshow(im)
plt.title('Original')
```

- BGR로 들어오기 때문에 RGB로 바꿔 줄 필요가 있음

```python
rgb=cv.cvtColor(im,cv.COLOR_BGR2RGB)
plt.figure()
plt.imshow(rgb)
plt.title('RGB')
```

```python
GRAY=cv.cvtColor(im,cv.COLOR_BGR2GRAY)
plt.figure()
plt.imshow(GRAY,cmap='gray')
plt.title('GRAY')
```

```python
blur=cv.blur(im,(100,100))
blur=cv.cvtColor(blur,cv.COLOR_BGR2RGB)
plt.subplot(121) # 가로줄 한개, 세로줄 두개, 첫번째에 놓겠다
plt.imshow(rgb)
plt.title('RGB')
plt.subplot(122) # 가로줄 한개, 세로줄 두개, 두번째에 놓겠다
plt.imshow(blur)
plt.title('blur')
```

---

```python
# 윤곽 찾기
edges=cv.Canny(GRAY,0,100)
plt.subplot(121) # 가로줄 한개, 세로줄 두개, 첫번째에 놓겠다
plt.imshow(GRAY,cmap='gray')
plt.title('GRAY')
plt.subplot(122) # 가로줄 한개, 세로줄 두개, 두번째에 놓겠다
plt.imshow(edges)
plt.title('edge detection')
```

머신러닝과 연관하여 비디오 영상의 움직이는 사물을 찾거나 번호판 또는 숫자 인식 여러 분야에서 활용이 가능하다


```python
%matplotlib inline
```

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
```

```python
from plotnine import *
```

```python
class theme_fs(theme_light):
    """
    A theme similar to :class:`theme_linedraw` but with light grey
    lines and axes to direct more attention towards the data.
    Parameters
    ----------
    base_size : int, optional
        Base font size. All text sizes are a scaled versions of
        the base font size. Default is 11.
    base_family : str, optional
        Base font family.
    """

    def __init__(self, base_size=11, base_family='DejaVu Sans'):
        theme_light.__init__(self, base_size, base_family)
        self.add_theme(theme(
            axis_ticks=element_line(color='#DDDDDD', size=0.5),
            panel_border=element_rect(fill='None', color='#838383',
                                      size=1),
            strip_background=element_rect(
                fill='#DDDDDD', color='#838383', size=1),
            strip_text_x=element_text(color='black'),
            strip_text_y=element_text(color='black', angle=-90),
            legend_key=element_blank(),
        ), inplace=True)
```

```python
df = pd.read_csv('./data/settles.acl16.learning_traces.13m.csv.gz')
df['delta_days'] = df['delta'].apply(lambda d: d / (60 * 60 * 24))
```

```python
# df = df.head(3000)
```

```python
(
    ggplot(df)
    + geom_bar(
        aes(x='p_recall'),
        stat=stat_bin(bins=100),
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
)
```

```python
(
    ggplot(df)
    + geom_bar(
        aes(x='delta_days'),
        stat=stat_bin(bins=100),
        fill='blue',
        alpha=0.5
    )
    + scale_y_log10()
    + theme_fs()
)
```

```python
(
    ggplot(df.loc[df.delta_days < 50])
    + geom_bar(
        aes(x='delta_days'),
        stat=stat_bin(bins=100),
        fill='blue',
        alpha=0.5
    )
    + scale_y_log10()
    + theme_fs()
)
```

```python
(
    ggplot(df)
    + geom_bar(
        aes(x='history_seen'),
        stat=stat_bin(bins=100),
        fill='blue',
        alpha=0.5
    )
    + scale_y_log10()
    + theme_fs()
)
```

```python
(
    ggplot(df.loc[df.history_seen < 50])
    + geom_bar(
        aes(x='history_seen'),
        stat=stat_bin(bins=50),
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
)
```

```python
(
    ggplot(df)
    + geom_bar(
        aes(x='history_correct'),
        stat=stat_bin(bins=100),
        fill='blue',
        alpha=0.5
    )
    + scale_y_log10()
    + theme_fs()
)
```

```python
(
    ggplot(df.loc[df.history_correct < 50])
    + geom_bar(
        aes(x='history_correct'),
        stat=stat_bin(bins=50),
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
)
```

```python
(
    ggplot(df)
    + geom_bar(
        aes(x='session_seen'),
        stat=stat_bin(bins=15),
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
)
```

```python
(
    ggplot(df)
    + geom_bar(
        aes(x='session_correct'),
        stat=stat_bin(bins=15),
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
)
```

```python
def _bin_p_recall(group):
    return pd.DataFrame([{'p_recall': group.p_recall.mean()}])
(
    ggplot(
        df.groupby(
            pd.qcut(df.delta_days, 20)
        ).apply(_bin_p_recall).reset_index()
    )
    + geom_bar(
        aes(x='delta_days', y='p_recall', ymin=0.75, ymax=1.0),
        stat='identity',
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
    + theme(
        axis_text_x=element_text(rotation=90)
    )
)
```

## reliability diagram, expected calibration error

In multi-class classification setting, the general idea of calibration is that confidence should match accuracy, i.e. when the model is 60% confidence, the probability of it being correct should be 60%.

reliability diagram: bin validation examples by predicted probability, then calculate the average accuracy within each bin, plot average accuracy against confidence. ideal calibration should be a diagonal line.

expected calibration error (ECE):  the difference in expectation between confidence and accuracy is
$$\mathbb{E}_{\hat{P}}[|\mathbb{P}(\hat{Y}=Y|\hat{P}=p)-p)|$$
This can be approximated by a weighted average of bins' accuracy - confidence difference (the gap showns as red bars in reliability diagrams)
$$\text{ECE}=\sum_{i}\frac{|B_i|}{n}|\text{acc}(B_m)-\text{conf}(B_m)|$$

```python
results = pd.read_csv('./results/hlr.settles.acl16.learning_traces.13m.preds', delimiter='\t')
```

```python
def _bin_prediction(group):
    return pd.DataFrame([{'prediction':  group.pp.mean()}])

(
    ggplot(
        results.groupby(
            pd.cut(results.p, 20)
        ).apply(_bin_prediction).reset_index()
    )
    + geom_bar(
        aes(x='p', y='prediction'),
        stat='identity',
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
    + theme(
        axis_text_x=element_text(rotation=90)
    )
)
```

```python
def _bin_rmse(group):
    return pd.DataFrame([{
        'rmse':  ((group.pp - group.p) ** 2).mean() ** (1/2)
    }])

(
    ggplot(
        results.groupby(
            pd.cut(results.pp, 20)
        ).apply(_bin_rmse).reset_index()
    )
    + geom_bar(
        aes(x='pp', y='rmse'),
        stat='identity',
        fill='blue',
        alpha=0.5
    )
    + theme_fs()
    + theme(
        axis_text_x=element_text(rotation=90)
    )
)
```

When our model directly predicts the probability, instead of using ECE, we can directly measure the miscalibration
$$\text{ECE}=\sum_i\frac{|B_i|}{n}|\text{precition}(B_i) - \text{ground_truth}(B_i)|$$

```python
def _bin_miscalibration(group):
    return pd.DataFrame([{
        'miscalibration': (group.pp - group.p).abs().mean(),
        'prediction': group.pp.mean(),
        'ground_truth': group.p.mean()
    }])

miscalibration = results.groupby(pd.cut(results.p, 16)).apply(_bin_miscalibration).reset_index()

print('expected calibration error', miscalibration.miscalibration.mean())

(
    ggplot(miscalibration)
    + geom_bar(
        aes(x='p', y='ground_truth'),
        stat='identity',
        fill='blue',
        alpha=1.0
    )
    + geom_bar(
        aes(x='p', y='prediction'),
        stat='identity',
        fill='red',
        color='red',
        alpha=0.3
    )
    + theme_fs()
    + theme(
        axis_text_x=element_text(rotation=90)
    )
)
```

## Half-life regression (HLR)

short-hand for each record \begin{align}<\cdot>&=<\Delta,x,P[\text{recall}]\in[0,1]>\\&=<\Delta,x,y\in\{0,1\}>\end{align}

Regression against recall probability $$l_\text{recall}(<\cdot>;\theta)=(p-f_\theta(x,\Delta))^2$$

Regression against back-solved half-life $$l_\text{half-life}(<\cdot>;\theta)=(\frac{-\Delta}{\log_2{p}}-f_\theta(x,\Delta))^2$$

Binary recall classification $$l_\text{binary}(<\cdot>;\theta)=\text{xent}(f_\theta(x,\Delta),y)$$

Assume that half-life increases exponentially with each repeated exposure, with a linear approximator, you get $f_\theta(x,\Delta)=2^{\theta\cdot x}$. Use this parameterization with regression against both recall probability and back-solved half-life, you get Settles' formulation:
$$l(<\cdot>; \theta)=(p-2^{\frac{\Delta}{2^{\theta\cdot x}}})^2+\alpha(\frac{\Delta}{\log_2(p)}-2^{\theta\cdot{x}})^2+\lambda|\theta|_2^2$$

Note that this formulation incorporates two heuristics
1. the memory strength follows an exponential forgetting curve, hence the half-life
2. half-life increases exponentially with number of repetitions

But in their code the `history_seen` and `history_seen_correct` feature are squre-rooted, so essentially throwing away the second heuristic.

```python
splitpoint = int(0.9 * len(df))
train_df, test_df = df.iloc[:splitpoint], df.iloc[splitpoint:]
```

```python
df1 = pd.DataFrame({
    'pp': results.pp.tolist(),
    'hh': results.hh.tolist(),
    'p': results.p.tolist(),
    'h': results.h.tolist(),
    'history_seen': test_df.history_seen.tolist(),
    'history_correct': test_df.history_correct.tolist(),
    'session_seen': test_df.session_seen.tolist(),
    'session_correct': test_df.session_correct.tolist(),
    'delta_days': test_df.delta_days.tolist(),
})
```

```python
def _bin_delta(group):
    return pd.DataFrame([{
        'prediction': group.pp.mean(),
        'ground_truth': group.p.mean(),
        'delta': group.delta_days.mean(),
    }])

(
    ggplot(
        df1.groupby(
            pd.cut(df1.delta_days, 16)
        ).apply(_bin_delta).reset_index()
    )
    + geom_bar(
        aes(x='delta', y='ground_truth'),
        stat='identity',
        fill='blue',
        alpha=0.3
    )
    + geom_bar(
        aes(x='delta', y='prediction'),
        stat='identity',
        fill='red',
        color='red',
        alpha=0.3
    )
    + theme_fs()
    + theme(
        axis_text_x=element_text(rotation=90)
    )
)
```

```python
def _bin_history_seen(group):
    return pd.DataFrame([{
        'prediction': group.pp.mean(),
        'ground_truth': group.p.mean(),
        'seen': group.history_seen.mean(),
    }])

_df1 = df1.loc[df1.history_seen < 100]

(
    ggplot(
        _df1.groupby(
            pd.cut(_df1.history_seen, 30)
        ).apply(_bin_history_seen).reset_index()
    )
    + geom_bar(
        aes(x='seen', y='ground_truth'),
        stat='identity',
        fill='blue',
        alpha=0.3
    )
    + geom_bar(
        aes(x='seen', y='prediction'),
        stat='identity',
        fill='red',
        color='red',
        alpha=0.3
    )
    + theme_fs()
    + theme(
        axis_text_x=element_text(rotation=90)
    )
)
```

There are several knobs to be tweaked in the general formulation:
- labels: derived $p$, binary $y$
- linear, nn
- on/off: exponentially increased half-life
- on/off: loss wrt back-solved half-life (from derived $p$)

```python

```