#web

##can you see cookie?

Cookieを調べてみる

```python
!curl -i http://104.196.236.136:31909
```

    HTTP/1.1 200 OK
    Server: nginx/1.21.4
    Date: Fri, 24 Dec 2021 15:14:39 GMT
    Content-Type: text/html
    Content-Length: 3043
    Last-Modified: Mon, 13 Dec 2021 03:04:20 GMT
    Connection: keep-alive
    ETag: "61b6b834-be3"
    Accept-Ranges: bytes

    <!doctype html><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Web site created using create-react-app"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>can you see 🍪?</title>You need to enable JavaScript to run this app.

ない

よくみると・・・

You need to enable JavaScript to run this app.

と書いてある

さらに

script src="/static/js/2.789fecdc.chunk.js"
script src="/static/js/main.40252036.chunk.js"

などと記載があるので調べたところ

```python
import requests

r = requests.get('http://104.196.236.136:31909/static/js/main.40252036.chunk.js')
r.text
```

    '(this["webpackJsonpcan-you-see-cookie"]=this["webpackJsonpcan-you-see-cookie"]||[]).push([[0],{9:function(e,n,t){"use strict";t.r(n);var c=t(1),i=t.n(c),o=t(3),s=t.n(o),l=t.p+"static/media/cookies.09ddddb3.jpg",a=t(0);var r=function(){return document.cookie=decodeURIComponent(escape(atob("ZmxhZz1jblZ1WDNKMWJsOXlkVzVmWVhOZlptRnpkRjk1YjNWZlkyRnVJUT09"))),Object(a.jsxs)("div",{className:"App",style:{textAlign:"center",display:"flex",flexDirection:"column"},children:[Object(a.jsx)("header",{className:"App-header",style:{backgroundColor:"#282c34",minHeight:"10vh",display:"flex",flexDirection:"column",alignItems:"center",justifyContent:"center",fontSize:"calc(10px + 2vmin)",color:"white"},children:"can you see \\ud83c\\udf6a?"}),Object(a.jsx)("main",{style:{flexGrow:1},children:Object(a.jsx)("img",{src:l,style:{width:"80%",margin:"5% 10%",borderRadius:"0.2pc"},alt:""})}),Object(a.jsx)("footer",{style:{backgroundColor:"#282c34",minHeight:"10vh",display:"flex",flexDirection:"column",alignItems:"center",justifyContent:"center",fontSize:"calc(10px + 2vmin)",color:"white"},children:"Iron Maiden CTF"})]})},d=function(e){e&&e instanceof Function&&t.e(3).then(t.bind(null,10)).then((function(n){var t=n.getCLS,c=n.getFID,i=n.getFCP,o=n.getLCP,s=n.getTTFB;t(e),c(e),i(e),o(e),s(e)}))};s.a.render(Object(a.jsx)(i.a.StrictMode,{children:Object(a.jsx)(r,{style:{margin:0,padding:0,fontFamily:"-apple-system, BlinkMacSystemFont, \'Segoe UI\', \'Roboto\', \'Oxygen\', \'Ubuntu\', \'Cantarell\', \'Fira Sans\', \'Droid Sans\', \'Helvetica Neue\', sans-serif"}})}),document.getElementById("root")),d()}},[[9,1,2]]]);\n//# sourceMappingURL=main.40252036.chunk.js.map'

document.cookie=decodeURIComponent(escape(atob("ZmxhZz1jblZ1WDNKMWJsOXlkVzVmWVhOZlptRnpkRjk1YjNWZlkyRnVJUT09")

Cookieを設定している様子

```python
import base64

cookie = "ZmxhZz1jblZ1WDNKMWJsOXlkVzVmWVhOZlptRnpkRjk1YjNWZlkyRnVJUT09"

base64.b64decode(cookie)

```

    b'flag=cnVuX3J1bl9ydW5fYXNfZmFzdF95b3VfY2FuIQ=='

```python
flag = "cnVuX3J1bl9ydW5fYXNfZmFzdF95b3VfY2FuIQ=="

base64.b64decode(flag)
```

    b'run_run_run_as_fast_you_can!'

JavaScriptが動かせればもっと簡単？

#crypto

##oreore

```python
ciphertext = "6157316a64475a374d484a6c4d484a6c58324e79655842304d4639706331397a6458426c636c397a5a574e31"
```

```python
bytes.fromhex(ciphertext).decode()
```

    'aW1jdGZ7MHJlMHJlX2NyeXB0MF9pc19zdXBlcl9zZWN1'

```python
import base64

base64.b64decode(bytes.fromhex(ciphertext).decode())
```

    b'imctf{0re0re_crypt0_is_super_secu'

##love letter

換字式暗号化なので、次のサイトで一撃。

https://quipqiup.com/

##common

```python
n = 21141157755370440356159016558867591002795131037765353220781763950242197787546988623583333748613811615876039600044067795582387314625044873018261912055648316947575307397145571269092683237196318163742534102252564452361648000741215145961263035358948910750394941304616916013217678630294578579032413375393774307079767343057241805957456859330160131811246961956716626217712879502367810458962663138839207104998925545962808329297100387478763021607429645752111666523372296094568288414788741918520918522347389517685977356020153749161423831061236785869435119648010372663034476232600939002450611487244980761240697255092515294619063
e1 = 65536
c1 = 3791139181938569737398685036032381737646987513051659017777895787043616313081106838760662536132081707503545049439606032069727376790572333643094529595081085118571506461893089039469054703087386815141490544138726142314337827447061522144729907408053083672097701520947030283539764963562701091579919487153984526052282424975277999492006498441654336183574233720547915500793830555227254993335348447191157770264921144000194606354966874727918714437428556969904356868934618577114911104025392029907939301331113924863412657322707967483351467004854177190395076310650546542866389721479249039989850401261446124771193559166329619972412
e2 = 65538
c2 = 9479987054301003593696498729807102589604512322454236806662495038269939932383233766107579478386705533651017857595249113906203900757985576318399557582785561924440535008414598465159052407108648112038858687677552218404377633749009339690058222162908839308847763582094729669124528370642665466262258176588951705959922822288462616323273663488143476234490690202535922260303842658128571121608034815328952318543895796544411651444423903851422043107669054499710424643274879906277104856140792871842468587164924689276336292808339852752054246714774764674969279059667783378523996744567909634744703888058392004064096159069736198607229
```

>$c1 = m^{e1} mod(n)$

>$c2 = m^{e2} mod(n)$

式変形すると・・・

>$c2-c1 = m^{e1}\times\left( m^{e2-e1} - 1\right) mod(n)$

>$c2-c1 = m^{e1}\times\left( m^{2} - 1\right) mod(n)$

>$m^{2} = \left\{ \left( c2-c1\right)\times inv\left( m^{e1}\right) +1\right\} mod(n)$

$m^{2}$なら計算できそう

```python
c3 = c2-c1
```

```python
import sympy

d = sympy.invert(c1, n)
```

```python
m2 = c3 * d % n
```

```python
sympy.sqrt(m2+1)
```

$\displaystyle 11102790441908549537282775747379306669309330780040183964070327776381$

```python
!pip install pycryptodome
```

    Collecting pycryptodome
      Downloading pycryptodome-3.12.0-cp35-abi3-manylinux2010_x86_64.whl (2.0 MB)
         |████████████████████████████████| 2.0 MB 5.3 MB/s
    Installing collected packages: pycryptodome
    Successfully installed pycryptodome-3.12.0

```python
from Crypto.Util.number import *

long_to_bytes(11102790441908549537282775747379306669309330780040183964070327776381)
```

    b'imctf{y0u_are_an_rsa_exp0rt}'

##pxkemxn

```python
M = 54354118932293738974277735009690054672438569183669264435593776508452144737651
X = []
X.append(27562969846242726464849700066934036402432300755063353950289314180737804494366)
X.append(53881790667921922669270635160364649792695336386197364717009941696103524473701)
X.append(18353049655762832365359178635985811169156727614439565352194778213695599906765)
X
```

    [27562969846242726464849700066934036402432300755063353950289314180737804494366,
     53881790667921922669270635160364649792695336386197364717009941696103524473701,
     18353049655762832365359178635985811169156727614439565352194778213695599906765]

A、Bは初期化時点で固定されるので、X[0]〜X[2]からAが求まる。

    X[1] = A*X[0] + B mod M
    X[2] = A*X[1] + B mod M
    X[2] - X[1] = A*(X[1] - X[0]) mod M
    A = (X[2] - X[1]) * inv(X[1] - X[0]) mod M

```python
import sympy

D = sympy.invert(X[1]-X[0], M)
D
```

$\displaystyle 41930284717522653140224594896152154410316861871128518105790131924618364636888$

```python
A = (X[2]-X[1]) * D % M
A
```

$\displaystyle 3888188529568654032289763479153589114108256385402749337961355305249441503481$

 Aが求まればBも求まる。

```python
B = (X[1] - A * X[0]) % M
B
```

$\displaystyle 39846490493920068598077178114888801382526803639516069440643562985086349730290$

あとはX[100]まで順番に計算すればOK

```python
x = X[0]
for i in range(1,101):
    x = (A*x + B) % M
    if i in [1, 2, 100]:
        print(i, '=>', x)
```

    1 => 53881790667921922669270635160364649792695336386197364717009941696103524473701
    2 => 18353049655762832365359178635985811169156727614439565352194778213695599906765
    100 => 41406001562898874001864598429475444818799641840147213206874340730032400227277

#rev

##E・Mo・I・XL

hintoに従ってoletoolsをインストールしてみる

```python
!pip install -U oletools
```

    Requirement already satisfied: oletools in /usr/local/lib/python3.7/dist-packages (0.60)
    Requirement already satisfied: easygui in /usr/local/lib/python3.7/dist-packages (from oletools) (0.98.2)
    Requirement already satisfied: olefile>=0.46 in /usr/local/lib/python3.7/dist-packages (from oletools) (0.46)
    Requirement already satisfied: colorclass in /usr/local/lib/python3.7/dist-packages (from oletools) (2.2.2)
    Requirement already satisfied: msoffcrypto-tool in /usr/local/lib/python3.7/dist-packages (from oletools) (4.12.0)
    Requirement already satisfied: pcodedmp>=1.2.5 in /usr/local/lib/python3.7/dist-packages (from oletools) (1.2.6)
    Requirement already satisfied: pyparsing<3,>=2.1.0 in /usr/local/lib/python3.7/dist-packages (from oletools) (2.4.7)
    Requirement already satisfied: cryptography>=2.3 in /usr/local/lib/python3.7/dist-packages (from msoffcrypto-tool->oletools) (36.0.1)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=2.3->msoffcrypto-tool->oletools) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=2.3->msoffcrypto-tool->oletools) (2.21)

```python
!olevba --reveal /content/drive/MyDrive/imctf2021/rev/EMoIXL/EmoXL.xlsm
```

    olevba 0.60 on Python 3.7.12 - http://decalage.info/python/oletools
    ===============================================================================
    FILE: /content/drive/MyDrive/imctf2021/rev/EMoIXL/EmoXL.xlsm
    Type: OpenXML
    WARNING  For now, VBA stomping cannot be detected for files in memory
    -------------------------------------------------------------------------------
    VBA MACRO ThisWorkbook.cls
    in file: xl/vbaProject.bin - OLE stream: 'VBA/ThisWorkbook'
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Private Sub Workbook_Open()
        Module1.GetPayload
    End Sub
    -------------------------------------------------------------------------------
    VBA MACRO Sheet1.cls
    in file: xl/vbaProject.bin - OLE stream: 'VBA/Sheet1'
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    (empty macro)
    -------------------------------------------------------------------------------
    VBA MACRO Module1.bas
    in file: xl/vbaProject.bin - OLE stream: 'VBA/Module1'
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Sub GetPayload()
        Dim payload As String: payload = vbNullString
        Dim ws As Worksheet
        Set ws = ThisWorkbook.Worksheets(1)
        With ThisWorkbook
            With .BuiltinDocumentProperties
                payload = payload + .Item(ws.Cells(1048573, 16384).Value)
                payload = payload + .Item(ws.Cells(1048574, 16384).Value)
                payload = payload + .Item(ws.Cells(1048572, 16384).Value)
                payload = payload + .Item(ws.Cells(1048575, 16384).Value)
                payload = payload + .Item(ws.Cells(1048576, 16384).Value)
            End With
        End With
        payload = _
        "p" & "w" & "s" & "h" & " " & "-" & "n" & "o" & _
        "p" & " " & "-" & "e" & _
        "p" & " " & "B" & "y" & _
        "p" & "a" & "s" & "s" & " " & "-" & "e" & " " + _
        payload
        CreateObject("WScript.Shell").Run payload
    End Sub

    +----------+--------------------+---------------------------------------------+
    |Type      |Keyword             |Description                                  |
    +----------+--------------------+---------------------------------------------+
    |AutoExec  |Workbook_Open       |Runs when the Excel Workbook is opened       |
    |Suspicious|Shell               |May run an executable file or a system       |
    |          |                    |command                                      |
    |Suspicious|WScript.Shell       |May run an executable file or a system       |
    |          |                    |command                                      |
    |Suspicious|Run                 |May run an executable file or a system       |
    |          |                    |command                                      |
    |Suspicious|CreateObject        |May create an OLE object                     |
    |Suspicious|Hex Strings         |Hex-encoded strings were detected, may be    |
    |          |                    |used to obfuscate strings (option --decode to|
    |          |                    |see all)                                     |
    |Suspicious|VBA obfuscated      |VBA string expressions were detected, may be |
    |          |Strings             |used to obfuscate strings (option --decode to|
    |          |                    |see all)                                     |
    |VBA string|pwsh -nop -ep Bypass|"p" & "w" & "s" & "h" & " " & "-" & "n" & "o"|
    |          |-e                  |&     "p" & " " & "-" & "e" &     "p" & " " &|
    |          |                    |"B" & "y" &     "p" & "a" & "s" & "s" & " " &|
    |          |                    |"-" & "e" & " "                              |
    +----------+--------------------+---------------------------------------------+
    MACRO SOURCE CODE WITH DEOBFUSCATED VBA STRINGS (EXPERIMENTAL):

    Private Sub Workbook_Open()
        Module1.GetPayload
    End Sub

    Attribute VB_Name = "Sheet1"
    Attribute VB_Base = "0{00020820-0000-0000-C000-000000000046}"
    Attribute VB_GlobalNameSpace = False
    Attribute VB_Creatable = False
    Attribute VB_PredeclaredId = True
    Attribute VB_Exposed = True
    Attribute VB_TemplateDerived = False
    Attribute VB_Customizable = True

    Attribute VB_Name = "Module1"
    Sub GetPayload()
        Dim payload As String: payload = vbNullString
        Dim ws As Worksheet
        Set ws = ThisWorkbook.Worksheets(1)
        With ThisWorkbook
            With .BuiltinDocumentProperties
                payload = payload + .Item(ws.Cells(1048573, 16384).Value)
                payload = payload + .Item(ws.Cells(1048574, 16384).Value)
                payload = payload + .Item(ws.Cells(1048572, 16384).Value)
                payload = payload + .Item(ws.Cells(1048575, 16384).Value)
                payload = payload + .Item(ws.Cells(1048576, 16384).Value)
            End With
        End With
        payload =     "pwsh -nop -ep Bypass -e " +     payload
        CreateObject("WScript.Shell").Run payload
    End Sub

Worksheets(1)のws.Cells(1048572, 16384)〜ws.Cells(1048576, 16384)の値を使っているので調べてみる。

```python
import openpyxl

wb = openpyxl.load_workbook("/content/drive/MyDrive/imctf2021/rev/EMoIXL/EmoXL.xlsm", keep_vba=True)
sheet = wb["Sheet1"]

print("{}, {}, {}, {}, {}".format(
      sheet.cell(1048573, 16384).value,
      sheet.cell(1048574, 16384).value,
      sheet.cell(1048572, 16384).value,
      sheet.cell(1048575, 16384).value,
      sheet.cell(1048576, 16384).value))

```

    title, subject, keywords, category, comments

調べたところ、BuiltinDocumentPropertiesはExcelファイルのメタ情報らいし・・・

次のようにすれば取り出せる

```python
print("{}{}{}{}{}".format(
      wb.properties.title,
      wb.properties.subject,
      wb.properties.keywords,
      wb.properties.category,
      wb.properties.description))
```

    JABwAGEAcwBzAHcAbwByAGQAIAA9ACAAKABSAGUAYQBkAC0ASABvAHMAdAAgACIAUABhAHMAcwB3AG8AcgBkACIAKQAKACQAdAB4AHQAIAA9ACAAWwBTAHkAcwB0AGUAbQAuAFQAZQB4AHQALgBFAG4AYwBvAGQAaQBuAGcAXQA6ADoARABlAGYAYQB1AGwAdAAuAEcAZQB0AFMAdAByAGkAbgBnACgACgAgACAAIAAgAFsAUwB5AHMAdABlAG0ALgBDAG8AbgB2AGUAcgB0AF0AOgA6AEYAcgBvAG0AQgBhAHMAZQA2ADQAUwB0AHIAaQBuAGcAKAAKACAAIAAgACAAIAAgACAAIAAoAAoAIAAgACAAIAAgACAAIAAgACAAIAAgACAAQAAoACIANQBhACIALAAgACIANQA2ACIALAAgACIAMwA5ACIALAAgACIANwA0ACIALAAgACIANgAyACIALAAgACIAMwAxACIALAAgACIAMwA5ACIALAAgACIAMwAwACIALAAgACIANgAxACIALAAgACIANQA3ACIALAAgACIAMwA5ACIALAAgACIANwA1ACIALAAgACIANQA4ACIALAAgACIAMwAyACIALAAgACIANABhACIALAAgACIAMwA1ACIALAAgACIANQA4ACIALAAgACIAMwAyACIALAAgACIANABlACIALAAgACIANgA4ACIALAAgACIANgAzACIALAAgACIANgBkACIALAAgACIANwA4ACIALAAgACIAMwA1ACIALAAgACIANQA4ACIALAAgACIAMwAzACIALAAgACIANABhACIALAAgACIANgA4ACIALAAgACIANQBhACIALAAgACIANQA2ACIALAAgACIAMwA5ACIALAAgACIANwAxACIALAAgACIANQBhACIALAAgACIANQA4ACIALAAgACIANAAyACIALAAgACIANwBhACIALAAgACIANQBhACIALAAgACIANQA3ACIALAAgACIAMwA0ACIALAAgACIAMwBkACIAKQAgAHwACgAgACAAIAAgACAAIAAgACAAIAAgACAAIABGAG8AcgBFAGEAYwBoAC0ATwBiAGoAZQBjAHQAIAB7ACAAWwBjAGgAYQByAF0AWwBiAHkAdABlAF0AIgAwAHgAJABfACIAIAB9AAoAIAAgACAAIAAgACAAIAAgACkAIAAtAGoAbwBpAG4AIAAiACIACgAgACAAIAAgACkACgApAAoAJAB0AHgAdAAyACAAPQAgAFsAUwB5AHMAdABlAG0ALgBUAGUAeAB0AC4ARQBuAGMAbwBkAGkAbgBnAF0AOgA6AEQAZQBmAGEAdQBsAHQALgBHAGUAdABTAHQAcgBpAG4AZwAoAAoAIAAgACAAIABbAFMAeQBzAHQAZQBtAC4AQwBvAG4AdgBlAHIAdABdADoAOgBGAHIAbwBtAEIAYQBzAGUANgA0AFMAdAByAGkAbgBnACgACgAgACAAIAAgACAAIAAgACAAKAAKACAAIAAgACAAIAAgACAAIAAgACAAIAAgAEAAKAAiADUAYQAiACwAIAAiADYAZAAiACwAIAAiADcAOAAiACwAIAAiADYAOAAiACwAIAAiADUAYQAiACwAIAAiADcAOQAiACwAIAAiADQAMQAiACwAIAAiADMANgAiACwAIAAiADQAOQAiACwAIAAiADQANwAiACwAIAAiADYAYwAiACwAIAAiADcANAAiACwAIAAiADUAOQAiACwAIAAiADMAMwAiACwAIAAiADUAMgAiACwAIAAiADYAZAAiACwAIAAiADYANQAiACwAIAAiADMAMAAiACwAIAAiADYAYwAiACwAIAAiADYANgAiACwAIAAiADYAMwAiACwAIAAiADYAZAAiACwAIAAiADUANgAiACwAIAAiADYAOAAiACwAIAAiADYAMgAiACwAIAAiADQANwAiACwAIAAiADcAOAAiACwAIAAiADMANQAiACwAIAAiADUAOAAiACwAIAAiADMAMwAiACwAIAAiADQAYQAiACwAIAAiADYAYwAiACwAIAAiADUAOQAiACwAIAAiADUANwAiACwAIAAiADcAOAAiACwAIAAiADcAMwAiACwAIAAiADYANQAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACwAIAAiADcAOQAiACwAIAAiADUAYQAiACwAIAAiADUANwAiACwAIAAiADQANgAiACwAIAAiADcAMwAiACwAIAAiADYAMgAiACwAIAAiADQAOAAiACwAIAAiADYAYwAiACwAIAAiADYANgAiACwAIAAiADYAMwAiACwAIAAiADYAZAAiACwAIAAiADUANgAiACwAIAAiADYAOAAiACwAIAAiADYAMgAiACwAIAAiADQANwAiACwAIAAiADcAOAAiACwAIAAiADMANQAiACwAIAAiADUAOAAiACwAIAAiADMAMwAiACwAIAAiADQAYQAiACwAIAAiADYAYwAiACwAIAAiADUAOQAiACwAIAAiADUANwAiACwAIAAiADcAOAAiACwAIAAiADcAMwAiACwAIAAiADYANQAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACwAIAAiADcAOQAiACwAIAAiADUAYQAiACwAIAAiADUANwAiACwAIAAiADQANgAiACwAIAAiADcAMwAiACwAIAAiADYAMgAiACwAIAAiADQAOAAiACwAIAAiADYAYwAiACwAIAAiADYANgAiACwAIAAiADYAMgAiACwAIAAiADQANwAiACwAIAAiADYAYwAiACwAIAAiADcAMgAiACwAIAAiADUAYQAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACwAIAAiADMANQAiACwAIAAiADYAMgAiACwAIAAiADMAMwAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACkAIAB8AAoAIAAgACAAIAAgACAAIAAgACAAIAAgACAARgBvAHIARQBhAGMAaAAtAE8AYgBqAGUAYwB0ACAAewAgAFsAYwBoAGEAcgBdAFsAYgB5AHQAZQBdACIAMAB4ACQAXwAiACAAfQAKACAAIAAgACAAIAAgACAAIAApACAALQBqAG8AaQBuACAAIgAiAAoAIAAgACAAIAApAAoAKQAKACQAaQBzAFAAYQBzAHMAZQBkACAAPQAgACgAJABwAGEAcwBzAHcAbwByAGQAIAAtAGUAcQAgACQAdAB4AHQAKQAKAFcAcgBpAHQAZQAtAEgAbwBzAHQAIAAoACQAaQBzAFAAYQBzAHMAZQBkACAAPwAgACIAZwBvAG8AZAAgAHkAbwB1ACAAYQByAGUAIABwAGEAcwBzAGUAZAAiACAAOgAgACIAaQBuAGMAbwByAHIAZQBjAHQAIABwAGEAcwBzAHcAbwByAGQAIgApACAALQBGAG8AcgBlAGcAcgBvAHUAbgBkAEMAbwBsAG8AcgAgACgAJABpAHMAUABhAHMAcwBlAGQAIAA/ACAAIgBHAHIAZQBlAG4AIgAgADoAIAAiAFIAZQBkACIAKQAKAGkAZgAgACgAJABpAHMAUABhAHMAcwBlAGQAKQAgAHsACgAgACAAIAAgAFcAcgBpAHQAZQAtAEgAbwBzAHQAIAAkAHQAeAB0ADIAIAAtAEYAbwByAGUAZwByAG8AdQBuAGQAQwBvAGwAbwByACAAQgBsAHUAZQAKAH0ACgAkAGgAbwBzAHQALgBVAEkALgBSAGEAdwBVAEkALgBSAGUAYQBkAEsAZQB5ACgAKQAgAHwAIABPAHUAdAAtAE4AdQBsAGwACgA=

明らかにbase64なのでデコードしてみる

```python
import base64

str_b64 = "JABwAGEAcwBzAHcAbwByAGQAIAA9ACAAKABSAGUAYQBkAC0ASABvAHMAdAAgACIAUABhAHMAcwB3AG8AcgBkACIAKQAKACQAdAB4AHQAIAA9ACAAWwBTAHkAcwB0AGUAbQAuAFQAZQB4AHQALgBFAG4AYwBvAGQAaQBuAGcAXQA6ADoARABlAGYAYQB1AGwAdAAuAEcAZQB0AFMAdAByAGkAbgBnACgACgAgACAAIAAgAFsAUwB5AHMAdABlAG0ALgBDAG8AbgB2AGUAcgB0AF0AOgA6AEYAcgBvAG0AQgBhAHMAZQA2ADQAUwB0AHIAaQBuAGcAKAAKACAAIAAgACAAIAAgACAAIAAoAAoAIAAgACAAIAAgACAAIAAgACAAIAAgACAAQAAoACIANQBhACIALAAgACIANQA2ACIALAAgACIAMwA5ACIALAAgACIANwA0ACIALAAgACIANgAyACIALAAgACIAMwAxACIALAAgACIAMwA5ACIALAAgACIAMwAwACIALAAgACIANgAxACIALAAgACIANQA3ACIALAAgACIAMwA5ACIALAAgACIANwA1ACIALAAgACIANQA4ACIALAAgACIAMwAyACIALAAgACIANABhACIALAAgACIAMwA1ACIALAAgACIANQA4ACIALAAgACIAMwAyACIALAAgACIANABlACIALAAgACIANgA4ACIALAAgACIANgAzACIALAAgACIANgBkACIALAAgACIANwA4ACIALAAgACIAMwA1ACIALAAgACIANQA4ACIALAAgACIAMwAzACIALAAgACIANABhACIALAAgACIANgA4ACIALAAgACIANQBhACIALAAgACIANQA2ACIALAAgACIAMwA5ACIALAAgACIANwAxACIALAAgACIANQBhACIALAAgACIANQA4ACIALAAgACIANAAyACIALAAgACIANwBhACIALAAgACIANQBhACIALAAgACIANQA3ACIALAAgACIAMwA0ACIALAAgACIAMwBkACIAKQAgAHwACgAgACAAIAAgACAAIAAgACAAIAAgACAAIABGAG8AcgBFAGEAYwBoAC0ATwBiAGoAZQBjAHQAIAB7ACAAWwBjAGgAYQByAF0AWwBiAHkAdABlAF0AIgAwAHgAJABfACIAIAB9AAoAIAAgACAAIAAgACAAIAAgACkAIAAtAGoAbwBpAG4AIAAiACIACgAgACAAIAAgACkACgApAAoAJAB0AHgAdAAyACAAPQAgAFsAUwB5AHMAdABlAG0ALgBUAGUAeAB0AC4ARQBuAGMAbwBkAGkAbgBnAF0AOgA6AEQAZQBmAGEAdQBsAHQALgBHAGUAdABTAHQAcgBpAG4AZwAoAAoAIAAgACAAIABbAFMAeQBzAHQAZQBtAC4AQwBvAG4AdgBlAHIAdABdADoAOgBGAHIAbwBtAEIAYQBzAGUANgA0AFMAdAByAGkAbgBnACgACgAgACAAIAAgACAAIAAgACAAKAAKACAAIAAgACAAIAAgACAAIAAgACAAIAAgAEAAKAAiADUAYQAiACwAIAAiADYAZAAiACwAIAAiADcAOAAiACwAIAAiADYAOAAiACwAIAAiADUAYQAiACwAIAAiADcAOQAiACwAIAAiADQAMQAiACwAIAAiADMANgAiACwAIAAiADQAOQAiACwAIAAiADQANwAiACwAIAAiADYAYwAiACwAIAAiADcANAAiACwAIAAiADUAOQAiACwAIAAiADMAMwAiACwAIAAiADUAMgAiACwAIAAiADYAZAAiACwAIAAiADYANQAiACwAIAAiADMAMAAiACwAIAAiADYAYwAiACwAIAAiADYANgAiACwAIAAiADYAMwAiACwAIAAiADYAZAAiACwAIAAiADUANgAiACwAIAAiADYAOAAiACwAIAAiADYAMgAiACwAIAAiADQANwAiACwAIAAiADcAOAAiACwAIAAiADMANQAiACwAIAAiADUAOAAiACwAIAAiADMAMwAiACwAIAAiADQAYQAiACwAIAAiADYAYwAiACwAIAAiADUAOQAiACwAIAAiADUANwAiACwAIAAiADcAOAAiACwAIAAiADcAMwAiACwAIAAiADYANQAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACwAIAAiADcAOQAiACwAIAAiADUAYQAiACwAIAAiADUANwAiACwAIAAiADQANgAiACwAIAAiADcAMwAiACwAIAAiADYAMgAiACwAIAAiADQAOAAiACwAIAAiADYAYwAiACwAIAAiADYANgAiACwAIAAiADYAMwAiACwAIAAiADYAZAAiACwAIAAiADUANgAiACwAIAAiADYAOAAiACwAIAAiADYAMgAiACwAIAAiADQANwAiACwAIAAiADcAOAAiACwAIAAiADMANQAiACwAIAAiADUAOAAiACwAIAAiADMAMwAiACwAIAAiADQAYQAiACwAIAAiADYAYwAiACwAIAAiADUAOQAiACwAIAAiADUANwAiACwAIAAiADcAOAAiACwAIAAiADcAMwAiACwAIAAiADYANQAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACwAIAAiADcAOQAiACwAIAAiADUAYQAiACwAIAAiADUANwAiACwAIAAiADQANgAiACwAIAAiADcAMwAiACwAIAAiADYAMgAiACwAIAAiADQAOAAiACwAIAAiADYAYwAiACwAIAAiADYANgAiACwAIAAiADYAMgAiACwAIAAiADQANwAiACwAIAAiADYAYwAiACwAIAAiADcAMgAiACwAIAAiADUAYQAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACwAIAAiADMANQAiACwAIAAiADYAMgAiACwAIAAiADMAMwAiACwAIAAiADUANgAiACwAIAAiADMAOQAiACkAIAB8AAoAIAAgACAAIAAgACAAIAAgACAAIAAgACAARgBvAHIARQBhAGMAaAAtAE8AYgBqAGUAYwB0ACAAewAgAFsAYwBoAGEAcgBdAFsAYgB5AHQAZQBdACIAMAB4ACQAXwAiACAAfQAKACAAIAAgACAAIAAgACAAIAApACAALQBqAG8AaQBuACAAIgAiAAoAIAAgACAAIAApAAoAKQAKACQAaQBzAFAAYQBzAHMAZQBkACAAPQAgACgAJABwAGEAcwBzAHcAbwByAGQAIAAtAGUAcQAgACQAdAB4AHQAKQAKAFcAcgBpAHQAZQAtAEgAbwBzAHQAIAAoACQAaQBzAFAAYQBzAHMAZQBkACAAPwAgACIAZwBvAG8AZAAgAHkAbwB1ACAAYQByAGUAIABwAGEAcwBzAGUAZAAiACAAOgAgACIAaQBuAGMAbwByAHIAZQBjAHQAIABwAGEAcwBzAHcAbwByAGQAIgApACAALQBGAG8AcgBlAGcAcgBvAHUAbgBkAEMAbwBsAG8AcgAgACgAJABpAHMAUABhAHMAcwBlAGQAIAA/ACAAIgBHAHIAZQBlAG4AIgAgADoAIAAiAFIAZQBkACIAKQAKAGkAZgAgACgAJABpAHMAUABhAHMAcwBlAGQAKQAgAHsACgAgACAAIAAgAFcAcgBpAHQAZQAtAEgAbwBzAHQAIAAkAHQAeAB0ADIAIAAtAEYAbwByAGUAZwByAG8AdQBuAGQAQwBvAGwAbwByACAAQgBsAHUAZQAKAH0ACgAkAGgAbwBzAHQALgBVAEkALgBSAGEAdwBVAEkALgBSAGUAYQBkAEsAZQB5ACgAKQAgAHwAIABPAHUAdAAtAE4AdQBsAGwACgA="

base64.b64decode(str_b64).decode('utf-16')

```

    '$password = (Read-Host "Password")\n$txt = [System.Text.Encoding]::Default.GetString(\n    [System.Convert]::FromBase64String(\n        (\n            @("5a", "56", "39", "74", "62", "31", "39", "30", "61", "57", "39", "75", "58", "32", "4a", "35", "58", "32", "4e", "68", "63", "6d", "78", "35", "58", "33", "4a", "68", "5a", "56", "39", "71", "5a", "58", "42", "7a", "5a", "57", "34", "3d") |\n            ForEach-Object { [char][byte]"0x$_" }\n        ) -join ""\n    )\n)\n$txt2 = [System.Text.Encoding]::Default.GetString(\n    [System.Convert]::FromBase64String(\n        (\n            @("5a", "6d", "78", "68", "5a", "79", "41", "36", "49", "47", "6c", "74", "59", "33", "52", "6d", "65", "30", "6c", "66", "63", "6d", "56", "68", "62", "47", "78", "35", "58", "33", "4a", "6c", "59", "57", "78", "73", "65", "56", "39", "79", "5a", "57", "46", "73", "62", "48", "6c", "66", "63", "6d", "56", "68", "62", "47", "78", "35", "58", "33", "4a", "6c", "59", "57", "78", "73", "65", "56", "39", "79", "5a", "57", "46", "73", "62", "48", "6c", "66", "62", "47", "6c", "72", "5a", "56", "39", "35", "62", "33", "56", "39") |\n            ForEach-Object { [char][byte]"0x$_" }\n        ) -join ""\n    )\n)\n$isPassed = ($password -eq $txt)\nWrite-Host ($isPassed ? "good you are passed" : "incorrect password") -ForegroundColor ($isPassed ? "Green" : "Red")\nif ($isPassed) {\n    Write-Host $txt2 -ForegroundColor Blue\n}\n$host.UI.RawUI.ReadKey() | Out-Null\n'

またbase64の変換が２個含まれているので、何も考えずデコードしてみる

```python
import base64

txt = ("5a", "56", "39", "74", "62", "31", "39", "30", "61", "57", "39", "75", "58", "32", "4a", "35", "58", "32", "4e", "68", "63", "6d", "78", "35", "58", "33", "4a", "68", "5a", "56", "39", "71", "5a", "58", "42", "7a", "5a", "57", "34", "3d")
txt2 = ("5a", "6d", "78", "68", "5a", "79", "41", "36", "49", "47", "6c", "74", "59", "33", "52", "6d", "65", "30", "6c", "66", "63", "6d", "56", "68", "62", "47", "78", "35", "58", "33", "4a", "6c", "59", "57", "78", "73", "65", "56", "39", "79", "5a", "57", "46", "73", "62", "48", "6c", "66", "63", "6d", "56", "68", "62", "47", "78", "35", "58", "33", "4a", "6c", "59", "57", "78", "73", "65", "56", "39", "79", "5a", "57", "46", "73", "62", "48", "6c", "66", "62", "47", "6c", "72", "5a", "56", "39", "35", "62", "33", "56", "39")

print(base64.b64decode(bytearray([ int(x,16) for x in txt]).decode()))
print(base64.b64decode(bytearray([ int(x,16) for x in txt2]).decode()))
```

    b'e_mo_tion_by_carly_rae_jepsen'
    b'flag : imctf{I_really_really_really_really_really_really_like_you}'

flagゲット


```python
# just some boring imports
from typing import Set, Tuple, Iterable, Callable, List, Dict
from types import MethodType
from functools import partial
import re
import timeit

import marisa_trie
import numpy as np
import matplotlib.pyplot as plt
from pympler.asizeof import asizeof
from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all"

%load_ext memory_profiler
%matplotlib inline

# small decorator to split class definitions over several slides
def add_to_class(cls):
    def decorator(fn):
        setattr(cls, fn.__name__, fn)
        return fn
    return decorator
```

    The memory_profiler extension is already loaded. To reload it, use:
      %reload_ext memory_profiler

# Datastructures and Algorithms for spell checking and correction

By Robin Hundt

Git: https://gitlab.gwdg.de/robinwilliam.hundt/programming-pearls-spelling-checker

Steve Johnsons first version of spell, 1975.

``` bash
prepare filename |                 # remove formatting commands
    translit A-Z a-z |             # map upper to lower case
        translit ^a-z @n |         # remove punctuation
            sort |                 # put words in alphabetical order
                unique |           # remove duplicate words
                    common -2 dict # report words not in dictionary
```
(from "Programming Pearls, a spelling checker. Jon Bentley. Communications of the ACM, May 1985 Volume 28 Number 5")

- 1975 Steve Johnson wrote first version of spell
- spent one afternoon combining 6 unix programs into a new useful one
- later became standard unix spell checker for english

# Overview of the talk

##  Spell checking

- Primitive implementation
- Trie data structure

## Spell correcting

- String metrics
- Corrector based on SymSpell algorithm

## Specification of spell checker

```python
def spell(filename: str, word_dict: str) -> Set[str]:
    w = set(word_dict.lower().split('\n'))
    with open(filename) as f:
        return set(re.sub(r"[^a-z]", "\n", f.read().lower()).split('\n')) - w
```

`spell` like spelling checker in just 4 lines of python (excl. imports) by utilizing the standard libraries.
Obvious shortcomings like converting `isn't` to `isn` and `t` because punctuation is stripped.

### Our test file (thanks to [Peter Norvig](http://norvig.com/))

```python
!echo word count: $(wc -w big.txt) '\n'
!head -n 5000 big.txt | tail -n 1
```

    word count: 1095695 big.txt

    "You can easily imagine, Mr. Holmes, how curious I became as to what the meaning of this extraordinary performance could possibly be. They were always very careful, I observed, to turn my face away from the window, so that I became consumed with the desire to see what was going on behind my back. At first it seemed to be impossible, but I soon devised a means. My hand-mirror had been broken, so a happy thought seized me, and I concealed a piece of the glass in my handkerchief. On the next occasion, in the midst of my laughter, I put my handkerchief up to my eyes, and was able with a little management to see all that there was behind me. I confess that I was disappointed. There was nothing. At least that was my first impression. At the second glance, however, I perceived that there was a man standing in the Southampton Road, a small bearded man in a grey suit, who seemed to be looking in my direction. The road is an important highway, and there are usually people there. This man, however, was leaning against the railings which bordered our field and was looking earnestly up. I lowered my handkerchief and glanced at Mrs. Rucastle to find her eyes fixed upon me with a most searching gaze. She said nothing, but I am convinced that she had divined that I had a mirror in my hand and had seen what was behind me. She rose at once.

### Standard unix english dictionary file

```python
!echo word count: $(wc -w /usr/share/dict/words) '\n'
!head -n 5000 /usr/share/dict/words | tail
```

    word count: 99171 /usr/share/dict/words

    Eula
    Eula's
    Euler
    Eumenides
    Eumenides's
    Eunice
    Eunice's
    Euphrates
    Euphrates's
    Eurasia

```python
with open('/usr/share/dict/words') as w:
    words_file = w.read()
```

### Using an american dict. for british texts will result in a lot of false positives...

```python
errors = spell('big.txt', words_file)
len(errors)
list(errors)[:10]
```

    5065

    ['resecting',
     'distal',
     'tubulo',
     'bons',
     'depositaries',
     'disinfection',
     'mitrofanych',
     'dolokhov',
     'emboli',
     'footpace']

### Rough estimate of memory consumption

```python
%memit spell('big.txt', words_file)
```

    peak memory: 189.10 MiB, increment: 27.04 MiB

### Benchmark of execution time

```python
time_set = %timeit -q -o -n 3 -r 10 spell('big.txt', words_file)
f"Best: {time_set.best} s"
```

    'Best: 0.6085993873421103 s'

### That is quite fast for a 1 million words file!

The speed is actually quite good, compared to the already optimized version of `spell` by Doug McIlroy (written in 1978). That version managed to check a 5,000 word document in **under 30 seconds** (on a VAX-11/750 with 3.125 MHz). The dramatic decrease in spell checking time (30 seconds for 5,000 words compared to ~1 seconds for nearly 1,000,000 words) can be largely attributed to tremendous increase in computing power.

### But can we do better?

## Using a trie (prefix tree) to store strings

    Trie.svg created by nd (CC BY-SA 3.0)

We can use a **Trie** or "prefix-tree", which is an ordered tree data structure, to efficiently store all the words from our dictionary.
The nodes themselves don't carry information about the key they're storing, instead the key is solely encoded in the position of the node and the information associated with the edges inside it.

Simple Node object that solely stores information about neighbouring nodes and which letter the the connecting edges represent.

```python
class IllegalTrieEdge(Exception):
    pass

class Node:
    def __init__(self):
        self.edges = {}
        self.end_of_word = False  # note that not only leaves can be valid words, but also intermediate nodes

    def add_edge(self, ch: str):
        if len(ch) != 1:
            raise IllegalTrieEdge('Edges can only consist of one character')
        if ch not in self.edges:
            self.edges[ch] = Node()
        return self.edges[ch]

    def get_edge(self, ch: str):
        return self.edges.get(ch)

    def __contains__(self, edge: str):
        return edge in self.edges
```

Very basic implementation of a prefix tree.

```python
class Trie:
    def __init__(self, words=[]):
        self.root = Node()
        self._build_trie(words)

    def _find_last_prefix_node(self, word: str) -> Tuple[str, Node]:
        v = self.root
        prefix = ''
        while len(word) > 0 and word[0] in v:
            prefix += word[0]
            v = v.get_edge(word[0])
            word = word[1:]
        return prefix, v
```

```python
@add_to_class(Trie)
def _add_word(self, word: str) -> bool:
    prefix, node = self._find_last_prefix_node(word)
    if prefix == word:
        node.end_of_word = True
        return False
    word = word[len(prefix):]
    while len(word) > 0:
        node = node.add_edge(word[0])
        word = word[1:]
    node.end_of_word = True
    return True
```

```python
@add_to_class(Trie)
def _build_trie(self, words: Set[str]):
    for word in words:
        self._add_word(word)
```

```python
@add_to_class(Trie)
def __contains__(self, word: str) -> bool:
    prefix, node = self._find_last_prefix_node(word)
    return prefix == word and node.end_of_word
```

## So lets test it, shall we?
As the dictionary to store inside the trie we'll again use the standard english words file on unix systems, splitted at newlines.

```python
with open('/usr/share/dict/words') as w:
    words_file = w.read()
    words = [word for word in words_file.split('\n') if len(word) > 0]
```

### Quick test of our implementation

```python
trie = Trie(words)
```

```python
[word for word in words if word not in trie]
'adasedadwa' not in trie
'Hallo' not in trie
```

    []

    True

    True

### Memory decrease by using a prefix tree (trie)....

```python
size_words_file = asizeof(words_file)
size_words = asizeof(words)
size_trie = asizeof(trie)
f"Size read file: {size_words_file} B; size word list: {size_words} B; size of trie: {size_trie} B"
f"Increase in size by factor {size_trie/size_words_file}"
```

    'Size read file: 938664 B; size word list: 6884128 B; size of trie: 93362984 B'

    'Increase in size by factor 99.46368881729778'

### An increase by a factor of ~ 100 ?!

Python is a highly dynamic language where **everything** is an object, even simple things like an integer or character. This adds an immense overhead, which results in this dramatic increase in size.
So saving space by using a prefix tree implemented in native Python is not possible, due to the limitations of the language.

### But maybe it is at least fast?

We'll adapt the `spell` function from earlier to calculate $A \backslash B$ where $A = \{w | w$ in file$\}$ and
$B = \{w | w$ in dictionary$\}$ and $B$ is a Python object implementing the `__contains__` method.

```python
def spell_G(filename: str, word_dict) -> Set[str]:
    with open(filename) as f:
        return {
            word for word in
            # read the file to check -> convert to lower -> split at newlines
            # -> remove punctuation -> remove double occurences (conv. to set)
            set(re.sub(r"[^a-z']", "\n", f.read().lower()).split('\n'))
            if word not in word_dict
        }
```

```python
time_trie = %timeit -q -o -n 3 -r 10 spell_G('big.txt', trie)
f"Best: {time_trie.best} s"
```

    'Best: 0.8601375749955574 s'

### Quite underwhelming, but what about a C++ Trie implementation?
(We'll use the Python library [marisa-trie](https://github.com/pytries/marisa-trie), a wrapper for the C++ implementation)

```python
trie_efficient = marisa_trie.Trie(words)
```

```python
time_marisa_trie = %timeit -q -o -n 3 -r 10 spell_G('big.txt', trie_efficient)
f"Best: {time_marisa_trie.best} s"
```

    'Best: 0.6063987583232423 s'

# Spelling checker: Conclusion

```python
plt.figure()
plt.bar(['spell', 'spell_G(Own Trie)', 'spell_G(marisa_trie)'],
        [time_set.best, time_trie.best, time_marisa_trie.best])
plt.show();
```

# Spelling correction

## *Distance* of strings

### Levenshtein distance (edit distance)

Minimal number of single-character edits (insertion, deletion, substitution) needed to change word $a$ into word $b$

\begin{equation}
\qquad\operatorname{lev}_{a,b}(i,j) = \begin{cases}
  \max(i,j) & \text{ if} \min(i,j)=0, \\
  \min \begin{cases}
          \operatorname{lev}_{a,b}(i-1,j) + 1 \\
          \operatorname{lev}_{a,b}(i,j-1) + 1 \\
          \operatorname{lev}_{a,b}(i-1,j-1) + 1_{(a_i \neq b_j)}
       \end{cases} & \text{ otherwise.}
\end{cases}
\end{equation}

where:

- $1_{(a_i \neq b_j)}$ is an indicator function equal to 0 when $a_i = b_j$ and equal to 1 otherwise
- $\operatorname{lev}_{a,b}(i,j)$ is the distance of the first $i$ characters of $a$ and the first $j$ characters of $b$

#### Example:
$\qquad\operatorname{lev}_{\text{kitten},\text{sitting}}(6,7) = 3$
$\qquad$**k**itten $\rightarrow$ *s*itt**e**n $\rightarrow$ sitt*i*n $\rightarrow$ sittin*g*

### Recursive implementation with exponential runtime

```python
def levenshtein_distance(s: str, t: str):
    if min(len(s), len(t)) == 0:
        return max(len(s),len(t))

    cost = 0 if s[-1] == t[-1] else 1

    return min(levenshtein_distance(s[:-1], t) +1,
               levenshtein_distance(s, t[:-1]) +1,
               levenshtein_distance(s[:-1], t[:-1]) + cost
              )
```

```python
def close_strings(s: str, words: Iterable[str],
                  distace_func: Callable[[str, str], int],
                  max_distance: int = 2):
    return [word for word in words if 0 <= distace_func(s, word) <= max_distance]
```

```python
close_strings('hal', words, levenshtein_distance, 2)[:10]
```

    ['Ahab', 'Al', 'Aral', 'Ba', 'Baal', 'Bali', 'Ball', 'Ca', 'Cal', 'Cali']

```python
time_lev = %timeit -q -o -r 3 -n 1 close_strings('hal', words, levenshtein_distance, 2)
f"Best: {time_lev.best} s"
```

    'Best: 177.78292495990172 s'

### Utilize dynamic programming

Brings down runtime to $O(nm)$ instead of exponential to the power of 3

```python
def levenshtein_dp(s: str, t: str):
    n = len(s) + 1
    m = len(t) + 1
    d = np.zeros((n, m))
    d[:, 0] = range(n)
    d[0, :] = range(m)

    for j in range(1, m):
        for i in range(1, n):
            cost = 0 if s[i-1] == t[j-1] else 1
            d[i, j] = min(d[i-1, j] + 1,
                          d[i, j-1] + 1,
                          d[i-1, j-1] + cost
                         )
    return d[n-1,m-1]
```

```python
time_lev_dp = %timeit -q -o -r 3 -n 1 close_strings('hallo', words, levenshtein_dp, 2)
f"Best: {time_lev_dp.best} s"
```

    'Best: 9.601151441922411 s'

### Damerau-Levenshtein distance

\begin{equation}
d_{a,b}(i,j) = \begin{cases}
  \max(i,j) & \text{ if} \min(i,j)=0, \\
\min \begin{cases}
          d_{a,b}(i-1,j) + 1 \\
          d_{a,b}(i,j-1) + 1 \\
          d_{a,b}(i-1,j-1) + 1_{(a_i \neq b_j)} \\
          d_{a,b}(i-2,j-2) + 1
       \end{cases} & \text{ if } i,j > 1 \text{ and } a_i = b_{j-1} \text{ and } a_{i-1} = b_j \\
  \min \begin{cases}
          d_{a,b}(i-1,j) + 1 \\
          d_{a,b}(i,j-1) + 1 \\
          d_{a,b}(i-1,j-1) + 1_{(a_i \neq b_j)}
       \end{cases} & \text{ otherwise.}
\end{cases}
\end{equation}

Extend `levenshtein_dp()` algorithm to include transpositions.

```python
def damerau_levenshtein(s: str, t: str):
    n = len(s) + 1
    m = len(t) + 1
    d = np.zeros((n, m))
    d[:, 0] = range(n)
    d[0, :] = range(m)

    for j in range(1, m):
        for i in range(1, n):
            cost = 0 if s[i-1] == t[j-1] else 1
            d[i, j] = min(d[i-1, j] + 1,
                          d[i, j-1] + 1,
                          d[i-1, j-1] + cost
                         )
            if i > 1 and j > 1 and s[i-1] == t[j-2] and s[i-2] == t[j-1]:
                d[i,j] = min(d[i,j], d[i-2, j-2] + cost)
    return d[n-1,m-1]
```

# Spelling correction

## Frequency of strings and *fast* corrector

Use word frequency file by wolfgarbe: [GitHub SymSpell](https://github.com/wolfgarbe/SymSpell)

```python
!echo word count: $(wc -l frequency_dictionary_en_82_765.txt) '\n'
!head -n 5000 frequency_dictionary_en_82_765.txt | tail
```

    word count: 82765 frequency_dictionary_en_82_765.txt

    mrs 12206596
    restoration 12202668
    convenience 12201623
    returning 12197026
    ralph 12190683
    opposition 12183104
    container 12181942
    defendant 12169689
    warner 12165874
    confirmation 12157725

```python
with open('frequency_dictionary_en_82_765.txt') as freq:
    freq = {word: int(cnt) for word, cnt in [line.split() for line in freq.read().split('\n') if line]}
```

```python
sorted_freq_tuples = sorted([(w, c) for w,c in freq.items()], key=lambda t: -t[1])

words, counts = zip(*sorted_freq_tuples)

plt.figure(figsize=(8,4), dpi=100)
plt.subplot(121)
plt.bar(words[:10], counts[:10])
plt.xticks(rotation='vertical')
plt.ylabel('word frequency')
plt.subplot(122)
plt.bar(words[-10:], counts[-10:])
plt.xticks(rotation='vertical')
plt.ylabel('word frequency')
plt.tight_layout()
plt.savefig('word-freqs.svg');
```

### Distribution of ten most and least common words

## Symmetric delete spelling correction ([SymSpell](https://towardsdatascience.com/symspell-vs-bk-tree-100x-faster-fuzzy-string-search-spell-checking-c4f10d80a078))

### Idea: Reduce candidates to calc. edit distance for
- pre-calculate possible string by deleting at most n chars
- store deletes in dictionary with candidate correction terms
- calc. deletes for search term and look them up in dictionary
- only calc. edit distance for terms stored as candidate corrections
- sort by edit dist. and frequency

```python
def splits(s: str) -> List[str]:
    return [(s[:i], s[i:]) for i in range(len(s)+1)]

def possible_deletions(s: str, max_edit_dist: int = 1) -> Set[str]:
    deletions = {s}
    for i in range(max_edit_dist):
        i_deletions = set()
        for word in deletions:
            split_words = splits(word)
            i_deletions |= {a+b[1:] for a,b in split_words}
        deletions |= i_deletions
    return deletions

    return {a+b[1:] for a,b in splits if b}
```

```python
class SpellingCorrector:
    def __init__(self, filename, max_edit_dist=2):
        self._dictionary = {}
        self.max_edit_dist = max_edit_dist
        self._init_dict_with_freq_file(filename)

    def _create_dict_entry(self, word, cnt):
        self._dictionary[word] = ([], int(cnt))
        deletes = possible_deletions(word, self.max_edit_dist)
        for part_word in deletes:
            if part_word in self._dictionary:
                self._dictionary[part_word][0].append(word)
            else:
                self._dictionary[part_word] = ([word], 0)

```

```python
@add_to_class(SpellingCorrector)
def _init_dict_with_freq_file(self, filename):
    with open(filename) as f:
        word_cnt_tuples = [line.split() for line in f.read().split('\n') if line]
        for word, cnt in word_cnt_tuples:
            self._create_dict_entry(word, cnt)
```

```python
class InvalidMaxEditDist(Exception):
    pass

@add_to_class(SpellingCorrector)
def _suggestion_candidates(self, word, max_edit_dist=2,
                           edit_dist_func=damerau_levenshtein,
                           verbose=False
                          ):
    if not (0 < max_edit_dist <= self.max_edit_dist):
        raise InvalidMaxEditDist(f"max_edit_dist must be in range (0,{self.max_edit_dist}]")
    deletes = possible_deletions(word, max_edit_dist)
    suggestion_dict = {}
    tried_corrections = set()
    dist_func_counter = 0
    for part_word in deletes:
        corrections = self._dictionary.get(part_word, ([], 0))[0]
        for correction in corrections:
            if correction not in tried_corrections:
                tried_corrections.add(correction)
                dist = edit_dist_func(correction, word)
                dist_func_counter += 1
                if  dist <= max_edit_dist:
                    suggestion_dict[correction] = (dist, self._dictionary[correction][1])
    if verbose:
        print(f"Called {edit_dist_func.__name__}() {dist_func_counter} many times.")
    return suggestion_dict

```

```python
@add_to_class(SpellingCorrector)
def suggestions(self, word: str, verbose=False, **kwargs):
    word = word.lower()
    candidates = self._suggestion_candidates(word, verbose=verbose, **kwargs)
    sorted_candidates = sorted(candidates.items(), key=lambda tup: (tup[1][0], -tup[1][1]))
    if verbose:
        return sorted_candidates
    return [word for word, _ in sorted_candidates]
```

```python
corr = SpellingCorrector('frequency_dictionary_en_82_765.txt', max_edit_dist=2)
```

### Small slice of the underlying dictionary structure

```python
[(k, (data[0][:3]+['...'], data[1])) for k, data in list(corr._dictionary.items())[:5]]
```

    [('the', (['the', 'other', 'they', '...'], 23135851162)),
     ('he', (['he', 'when', 'here', '...'], 842847219)),
     ('e', (['the', 'be', 'are', '...'], 0)),
     ('th', (['the', 'that', 'this', '...'], 0)),
     ('t', (['the', 'to', 'it', '...'], 0))]

## So lets benchmark it!

```python
time_corr = %timeit -q -o corr.suggestions("Hallo", max_edit_dist=2)
f"Best: {time_corr.best} s"
corr.suggestions("Hallo", verbose=True)[:5]
```

    'Best: 0.013340204500127583 s'

    Called damerau_levenshtein() 191 many times.

    [('hall', (1.0, 61265768)),
     ('hello', (1.0, 32960381)),
     ('halo', (1.0, 6194202)),
     ('halls', (1.0, 4607236)),
     ('halle', (1.0, 1568027))]

## That's a speed increase of *several* orders of magnitude!

## But is the corrector actually any good?

### Misspell data

We're using spelling error data formatted by [Roger Mitton](http://www.dcs.bbk.ac.uk/~ROGER/) to test the corrector.
`aspell.dat`
> The aspell file contains 531 misspellings of 450 words. It is derived from one assembled by Atkinson [(click here)](http://aspell.net/test/batch0.tab) for testing the [GNU Aspell spellchecker](http://aspell.net/).

`wikipedia.dat`
> The wikipedia file contains 2,455 misspellings of 1,922 words. It is a list of misspellings made by Wikipedia editors [(click here)](http://en.wikipedia.org/wiki/Wikipedia:List_of_common_misspellings).

```python
def parse_misspell_data(filename):
    with open(filename) as f:
        lines = [line for line in f.read().split('\n') if line]
        misspell_data = {}
        misspelled_words_cnt = 0
        cursor = ''
        for line in lines:
            if line.startswith('$'):
                cursor = line[1:]
                misspell_data[cursor] = []
                continue
            misspell_data[cursor].append(line)
            misspelled_words_cnt += 1
        return misspell_data, misspelled_words_cnt
```

```python
def score(misspell_data: Dict[str, List[str]] ,
          corrector_func: Callable[[str], Iterable[str]] ,
          consider_corrections_count=5):
    score = 0
    for word, misspells in misspell_data.items():
        for ms_word in misspells:
            try:
                corrections = corrector_func(ms_word)
                score += 1 if word in corrections[:consider_corrections_count] else 0
            except IndexError:
                pass
    return score
```

```python
def test_with_spell_data_and_create_graph(filename, corrector, max_edit_dist=2, max_lookup=5):
    misspell_data, misspelled_words_cnt = parse_misspell_data(filename)
    plot_cols = max_edit_dist
    plt.figure(figsize=(4*plot_cols, 4), dpi=100)
    for edit_dist in range(1, max_edit_dist+1):
        corrector_fixed_dist = partial(corrector, max_edit_dist=edit_dist)
        score_data = [score(misspell_data, corrector_fixed_dist, lookup)/misspelled_words_cnt
                      for lookup in range(1, max_lookup+1)]
        print(score_data)
        plt.subplot(1, plot_cols, edit_dist)
        plt.gca().set_ylim([0, 1.0])
        plt.bar(range(1, max_lookup+1), score_data)
        plt.title(f"Max edit dist: {edit_dist}")
        plt.ylabel('score in %')
        plt.xlabel('max lookup')
    plt.tight_layout()
    plt.savefig(f'{filename}-score-graph.svg');
```

```python
test_with_spell_data_and_create_graph('aspell.dat', corr.suggestions, max_lookup=10)
test_with_spell_data_and_create_graph('wikipedia.dat', corr.suggestions, max_lookup=10)
```

# Spelling corrector: Conclusion

(aspell.dat)

(wikipedia.dat)

# Thank you for your attention

All slides, the notebooks, the word files, etc. are available on the [GWDG Gitlab](https://gitlab.gwdg.de/) under [programming-pearls-spelling-checker](https://gitlab.gwdg.de/robinwilliam.hundt/programming-pearls-spelling-checker)

# References

- Programming Pearls, a spelling checker. Jon Bentley. Communications of the ACM, May 1985 Volume 28 Number 5
- 1000x Faster Spelling Correction algorithm, Wolf, [blog.faroo.com](http://blog.faroo.com/2012/06/07/improved-edit-distance-based-spelling-correction/) (published June 7, 2012)
- How to Write a Spelling Corrector, Peter Norvig, [norvig.com](http://norvig.com/spell-correct.html) (published Feb 2007 to August 2016)

## Data sources
- `big.txt`, Peter Norvig, [norvig.com](http://norvig.com/big.txt)
- `frequency_dictionary_en_82_765.txt`, Wolf Garbe, [github.com](https://github.com/wolfgarbe/SymSpell/blob/master/SymSpell/frequency_dictionary_en_82_765.txt)
- `aspell.dat` and `wikipedia.dat`, Roger Mitton, [dcs.bbk.ac.uk](http://www.dcs.bbk.ac.uk/~ROGER/corpora.html)

## Libraries
- [marisa_trie](https://github.com/pytries/marisa-trie) (MIT license)
- [numpy](https://github.com/numpy/numpy) (BSD-new license)
- [matplotlib](https://github.com/matplotlib/matplotlib)
- [pympler](https://github.com/pympler/pympler) (Apache-2.0)


# iCAT Export
---

#### Overview
Export data in pyramidal png stacks for CATMAID.

```python
from pathlib import Path
from tqdm.notebook import tqdm

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

#### Settings

```python
# pandas display settings
pd.set_option('display.max_rows', 20)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', 15)

# Indirectly enable autocomplete
%config Completer.use_jedi = False
```

## Set up `render-ws` environment
---

```python
import renderapi
from icatapi.render_pandas import create_stack_DataFrame, create_stacks_DataFrame
```

```python
# `render` project parameters
# ---------------------------
owner = 'rlane'
project = 'pancreas'
project_dir = Path(f'/long_term_storage/{owner}/SECOM/projects/{project}')
export_dir = Path(f'/long_term_storage/{owner}/CATMAID/projects/{project}')

# Create a renderapi.connect.Render object
# ----------------------------------------
render_connect_params = {
    'host': 'sonic.tnw.tudelft.nl',
    'port': 8080,
    'owner': owner,
    'project': project,
    'client_scripts': '/home/catmaid/render/render-ws-java-client/src/main/scripts',
    'memGB': '2G'
}
render = renderapi.connect(**render_connect_params)
render.make_kwargs()
```

    {'host': 'http://sonic.tnw.tudelft.nl',
     'port': 8080,
     'owner': 'rlane',
     'project': 'pancreas',
     'client_scripts': '/home/catmaid/render/render-ws-java-client/src/main/scripts',
     'client_script': '/home/catmaid/render/render-ws-java-client/src/main/scripts/run_ws_client.sh',
     'memGB': '2G'}

```python
# Infer stack and section info
# ----------------------------
stacks = renderapi.render.get_stacks_by_owner_project(render=render)
stacks_EM = [stack for stack in stacks if 'EM' in stack]
stacks_FM = [stack for stack in stacks if 'EM' not in stack]
stacks_2_export = [
    'EM_himag_stitched',
    'EM_lomag_correlated',
    'Hoechst_correlated',
    'AF594_correlated'
]

# Output
# ------
out = f"""\
project directory... {project_dir} | Exists: {project_dir.exists()}
export directory.... {export_dir} | Exists: {export_dir.exists()}
all stacks.......... {stacks}
EM stacks........... {stacks_EM}
FM stacks........... {stacks_FM}
stacks to export.... {stacks_2_export}
...
"""
print(out)

# Create project DataFrame
# ------------------------
df_project = create_stacks_DataFrame(stacks=stacks_2_export,
                                     render=render)
df_project.groupby('stack')\
          .apply(lambda x: x.head(5))
```

    project directory... /long_term_storage/rlane/SECOM/projects/pancreas | Exists: False
    export directory.... /long_term_storage/rlane/CATMAID/projects/pancreas | Exists: False
    all stacks.......... ['Hoechst_correlated', 'EM_lomag_correlated', 'AF594_correlated', 'EM_minimontages', 'EM_lomag_world', 'EM_himag_world', 'EM_lomag_overlaid', 'AF594_overlaid', 'Hoechst_overlaid', 'EM_himag_stitched', 'EM_himag_montaged', 'Hoechst', 'EM_lomag', 'EM_himag', 'AF594']
    EM stacks........... ['EM_lomag_correlated', 'EM_minimontages', 'EM_lomag_world', 'EM_himag_world', 'EM_lomag_overlaid', 'EM_himag_stitched', 'EM_himag_montaged', 'EM_lomag', 'EM_himag']
    FM stacks........... ['Hoechst_correlated', 'AF594_correlated', 'AF594_overlaid', 'Hoechst_overlaid', 'Hoechst', 'AF594']
    stacks to export.... ['EM_himag_stitched', 'EM_lomag_correlated', 'Hoechst_correlated', 'AF594_correlated']
    ...

      tileId
      z
      width
      height
      minint
      maxint
      imageUrl
      tforms
      stack
      sectionId
      imageRow
      imageCol
      stageX
      stageY

      stack

      AF594_correlated
      108
      aaa_insulin...
      4.0
      2048.0
      2048.0
      441.0
      882.0
      https://son...
      [M=[[1.0000...
      AF594_corre...
      S004
      1
      1
      1025.598744
      6371.521975

      109
      aaa_insulin...
      5.0
      2048.0
      2048.0
      439.0
      878.0
      https://son...
      [M=[[1.0000...
      AF594_corre...
      S005
      1
      1
      1622.072408
      6381.077600

      110
      aaa_insulin...
      6.0
      2048.0
      2048.0
      445.0
      890.0
      https://son...
      [M=[[1.0000...
      AF594_corre...
      S006
      1
      1
      2227.808609
      6381.567861

      111
      aaa_insulin...
      7.0
      2048.0
      2048.0
      439.0
      878.0
      https://son...
      [M=[[1.0000...
      AF594_corre...
      S007
      1
      1
      2832.009499
      6381.316683

      EM_himag_stitched
      0
      aaa_lil_EM-...
      4.0
      4096.0
      4096.0
      31039.0
      32893.0
      https://son...
      [M=[[0.2309...
      EM_himag_st...
      S004
      6
      6
      1078.477000
      6335.559000

      1
      aab_lil_EM-...
      4.0
      4096.0
      4096.0
      31039.0
      32893.0
      https://son...
      [M=[[0.2309...
      EM_himag_st...
      S004
      6
      5
      1060.996000
      6335.559000

      2
      aac_lil_EM-...
      4.0
      4096.0
      4096.0
      31039.0
      32893.0
      https://son...
      [M=[[0.2308...
      EM_himag_st...
      S004
      6
      4
      1043.523000
      6335.571000

      3
      aad_lil_EM-...
      4.0
      4096.0
      4096.0
      31039.0
      32893.0
      https://son...
      [M=[[0.2309...
      EM_himag_st...
      S004
      6
      3
      1025.992000
      6335.580000

      4
      aae_lil_EM-...
      4.0
      4096.0
      4096.0
      31039.0
      32893.0
      https://son...
      [M=[[0.2309...
      EM_himag_st...
      S004
      6
      2
      1008.514000
      6335.597000

      EM_lomag_correlated
      100
      aaa_big_EM-...
      4.0
      4096.0
      4096.0
      31678.0
      34295.0
      https://son...
      [M=[[1.5358...
      EM_lomag_co...
      S004
      1
      1
      1021.110000
      6371.003000

      101
      aaa_big_EM-...
      5.0
      4096.0
      4096.0
      31586.0
      34319.0
      https://son...
      [M=[[1.5332...
      EM_lomag_co...
      S005
      1
      1
      1621.125000
      6381.079000

      102
      aaa_big_EM-...
      6.0
      4096.0
      4096.0
      31639.0
      34260.0
      https://son...
      [M=[[1.5290...
      EM_lomag_co...
      S006
      1
      1
      2226.245000
      6381.133000

      103
      aaa_big_EM-...
      7.0
      4096.0
      4096.0
      31681.0
      34238.0
      https://son...
      [M=[[1.5343...
      EM_lomag_co...
      S007
      1
      1
      2831.405000
      6381.305000

      Hoechst_correlated
      104
      aaa_hoechst...
      4.0
      2048.0
      2048.0
      889.0
      1778.0
      https://son...
      [M=[[1.0000...
      Hoechst_cor...
      S004
      1
      1
      1025.598744
      6371.521975

      105
      aaa_hoechst...
      5.0
      2048.0
      2048.0
      870.0
      1740.0
      https://son...
      [M=[[1.0000...
      Hoechst_cor...
      S005
      1
      1
      1622.072408
      6381.077600

      106
      aaa_hoechst...
      6.0
      2048.0
      2048.0
      884.0
      1768.0
      https://son...
      [M=[[1.0000...
      Hoechst_cor...
      S006
      1
      1
      2227.808609
      6381.567861

      107
      aaa_hoechst...
      7.0
      2048.0
      2048.0
      864.0
      1728.0
      https://son...
      [M=[[1.0000...
      Hoechst_cor...
      S007
      1
      1
      2832.009499
      6381.316683

## Export `render-ws` stacks to CATMAID
---
### Set up CATMAID export parameters

```python
from random import sample
from renderapi.client import ArgumentParameters
```

```python
class CatmaidBoxesParameters(ArgumentParameters):
    """Subclass of `ArgumentParameters` for facilitating CATMAID export client script"""
    def __init__(self, stack, root_directory,
                 height=1024, width=1024, fmt='png', max_level=0,
                 host=None, port=None, baseurl=None,
                 owner=None, project=None, render=None, **kwargs):

        super(CatmaidBoxesParameters, self).__init__(**kwargs)

        self.stack = stack
        self.rootDirectory = root_directory
        self.height = height
        self.width = width
        self.format = fmt
        self.maxLevel = max_level

        render_kwargs = render.make_kwargs()
        host = render_kwargs.get('host')
        port = render_kwargs.get('port')
        self.baseDataUrl = renderapi.render.format_baseurl(host, port)
        self.owner = render_kwargs.get('owner') if owner is None else owner
        self.project = render_kwargs.get('project') if project is None else project
```

#### Logic for maximum zoom level

Ideally `max_level` is set such that

\begin{equation}
\left( \frac{w_s}{w_t \,\, 2^m} \right) < 1
\end{equation}

where $m$ is `max_level`, $w_s$ is the width of the stack and $w_t$ is the width of each tile. Then

\begin{equation}
m = \textrm{ceil} \left( \log{\frac{w_s}{w_t}} \times \frac{1}{\log{2}} \right)
\end{equation}

```python
# Initialize collection for export parameters
export_data = {}
# Update max level
maxest_level = 0
# Set format
fmt = 'png'

# Iterate through stacks
for stack, df_stack in df_project.groupby('stack'):

    # Determine `max_level` such that the full section is in view when fully zoomed out
    w_tile = 1024
    h_tile = 1024
    stack_bounds = renderapi.stack.get_stack_bounds(stack=stack,
                                                    render=render)
    w_stack = max(stack_bounds['maxX'] - stack_bounds['minX'],
                  stack_bounds['maxY'] - stack_bounds['minY'])
    max_level = int(np.ceil(np.log(w_stack / w_tile) * 1/np.log(2)))
    # Export each stack to highest level in the project
    maxest_level = max(max_level, maxest_level)

    # Set parameters for export to CATMAID
    export_params = CatmaidBoxesParameters(stack=stack,
                                           root_directory=export_dir.parent.as_posix(),
                                           width=w_tile,
                                           height=h_tile,
                                           max_level=maxest_level,
                                           fmt=fmt,
                                           project=project,
                                           render=render)

    # Add CATMAID export parameters to collection
    export_data[stack] = export_params

# Preview
stack = sample(export_data.keys(), 1)[0]
list(export_data[stack].to_java_args())
```

    ['--stack',
     'Hoechst_correlated',
     '--rootDirectory',
     '/long_term_storage/rlane/CATMAID/projects',
     '--height',
     '1024',
     '--width',
     '1024',
     '--format',
     'png',
     '--maxLevel',
     '6',
     '--baseDataUrl',
     'http://sonic.tnw.tudelft.nl:8080/render-ws/v1',
     '--owner',
     'rlane',
     '--project',
     'pancreas']

### Call render script
`render_catmaid_boxes.sh`
```sh
Usage: java -cp <render-module>-standalone.jar
      org.janelia.render.client.BoxClient [options] Z values for layers to
      render
  Options:
  * --baseDataUrl
      Base web service URL for data (e.g. http://host[:port]/render-ws/v1)
    --binaryMask
      use binary mask (e.g. for DMG data)
      Default: false
    --createIGrid
      create an IGrid file
      Default: false
    --doFilter
      Use ad hoc filter to support alignment
      Default: false
    --filterListName
      Apply this filter list to all rendering (overrides doFilter option)
    --forceGeneration
      Regenerate boxes even if they already exist
      Default: false
    --format
      Format for rendered boxes
      Default: png
  * --height
      Height of each box
    --help
      Display this note
    --label
      Generate single color tile labels instead of actual tile images
      Default: false
    --maxLevel
      Maximum mipmap level to generate
      Default: 0
    --maxOverviewWidthAndHeight
      Max width and height of layer overview image (omit or set to zero to
      disable overview generation)
    --numberOfRenderGroups
      Total number of parallel jobs being used to render this layer (omit if
      only one job is being used)
  * --owner
      Stack owner
  * --project
      Stack project
    --renderGroup
      Index (1-n) that identifies portion of layer to render (omit if only one
      job is being used)
  * --rootDirectory
      Root directory for rendered tiles (e.g.
      /tier2/flyTEM/nobackup/rendered_boxes)
    --skipInterpolation
      skip interpolation (e.g. for DMG data)
      Default: false
  * --stack
      Stack name
  * --width
      Width of each box
```

#### Wrapper for `render_catmaid_boxes` script for multiprocessing
Multiprocessing is done across sections, so a process is created for each section.

```python
def run_render_catmaid_boxes(z, client_script, java_args):
    """Wrapper for `render_catmaid_boxes` script to enable multiprocessing"""
    p = subprocess.run([client_script.as_posix(), f'{z:.0f}'] + java_args)
```

#### \*\****COMPUTATIONALLY EXPENSIVE*** \**

##### Run `render_catmaid_boxes` on `N_cores`

`renderapi.client.WithPool` ends prematurely (after exporting one section).
Weirdly only happens with `lil_EM_montaged` stack...

```python
import subprocess
from functools import partial
from multiprocessing import Pool
```

```python
# Path to `render_catmaid_boxes` shell script
fp_client = Path(render_connect_params['client_scripts']) / 'render_catmaid_boxes.sh'
# Set number of cores for multiprocessing
N_cores = min(15, df_project['z'].unique().size)
# Get z values
z_values = np.unique([renderapi.stack.get_z_values_for_stack(stack, render=render)\
                      for stack in stacks_2_export])

# Iterate through stacks to export
# for stack in tqdm(stacks_2_export):
for stack in tqdm(stacks_2_export):

    # Create java arguments from export parameters
    java_args = list(export_data[stack].to_java_args())

    # Set up `render_catmaid_boxes` client script
    render_catmaid_boxes_partial = partial(run_render_catmaid_boxes,
                                           client_script=fp_client,
                                           java_args=java_args)

    # Run `render_catmaid_boxes` across `N_cores`
    with Pool(N_cores) as pool:
        pool.map(render_catmaid_boxes_partial, z_values)
```

      0%|          | 0/4 [00:00<?, ?it/s]

## Set up tiles for import to CATMAID
---
### Resort CATMAID tiles
By (unchangeable) default, `render_catmaid_boxes` exports tiles as

`root directory` / `project` / `stack` / `width x height` / `zoomlevel` / `z` / `row` / `col.fmt`

This is ok, but preferred format for importing to CATMAID is [tile source convention 1](https://catmaid.readthedocs.io/en/stable/tile_sources.html#tile-source-types) --- "[File-based image stack](https://catmaid.readthedocs.io/en/stable/tile_sources.html#file-based-image-stack)"

`root directory` / `project` / `stack` / `z` / `row_col_zoomlevel.fmt`

One other tidbit is that CATMAID annoyingly assumes that sections are 0-indexed so $z_{min}$ is subtracted.

#### \*\****CHANGES LOTS & LOTS OF FILEPATHS ON DISK*** \**

```python
from shutil import rmtree
from skimage import io, transform, img_as_ubyte
```

```python
# Iterate through stacks to export
for stack in tqdm(stacks_2_export):

    # Loop through all the exported tiles per stack
    fps = (export_dir / stack).glob(f"1024x1024/**/[0-9]*.{fmt}")
    for fp in fps:

        # Extract tile info from filepath
        zoom_level = int(fp.parents[2].name)
        z = int(fp.parents[1].name) - int(z_values.min())  # 0-index
        row = int(fp.parents[0].name)
        col = int(fp.stem)

        # Reformat tile
        tile_format_1 = export_dir / stack / f"{z}/{row}_{col}_{zoom_level}.{fmt}"
        tile_format_1.parent.mkdir(parents=True, exist_ok=True)
        fp.rename(tile_format_1)

    # Clean up (now presumably empty) directory tree
    rmtree((export_dir / stack / '1024x1024').as_posix())
```

      0%|          | 0/4 [00:00<?, ?it/s]

#### Make thumbnails

```python
from icatapi.plotting import colorize, T_HOECHST, T_AF594
```

```python
# Colorize settings
d_colorize = {
    'Hoechst_correlated': T_HOECHST,
    'AF594_correlated': T_AF594,
}

# Loop through stacks to export
for stack in tqdm(stacks_2_export):

    # Loop through each section
    for z in (z_values - z_values.min()):

        # Load most zoomed out image (0, 0, `maxest_level`)
        image = io.imread(export_dir / f"{stack}/{z:.0f}/0_0_{maxest_level}.{fmt}")
        # Resize
        output_shape = (128, 128)
        thumb = transform.resize(image, output_shape=output_shape)
        # Colorize
        if stack in stacks_FM:
            thumb = colorize(thumb, d_colorize[stack])
        # Save
        fp_thumb = export_dir / f"{stack}/{z:.0f}/small.{fmt}"
        io.imsave(fp_thumb, img_as_ubyte(thumb))
```

      0%|          | 0/4 [00:00<?, ?it/s]

#### Create `project.yaml` file

```python
import sys
from random import sample
from ruamel.yaml import YAML
from tifffile import TiffFile
from bs4 import BeautifulSoup as Soup
import json
```

```python
# Set project yaml file
project_yaml = export_dir / 'project.yaml'

# Collect stack data
stack_data = []
for stack in tqdm(stacks_2_export):

    # Get dimension data
    bounds = renderapi.stack.get_stack_bounds(stack=stack,
                                              render=render)
    dimensions = (int((bounds['maxX'] - bounds['minX']) * 1.1),
                  int((bounds['maxY'] - bounds['minY']) * 1.1),
                  int(bounds['maxZ'] - bounds['minZ'] + 1))

    # Get resolution data (base it off OG EM himag resolution data)
    stack_metadata = renderapi.stack.get_full_stack_metadata(stack='EM_himag',
                                                             render=render)
    resolution = (np.round(stack_metadata['currentVersion']['stackResolutionX'], 5),
                  np.round(stack_metadata['currentVersion']['stackResolutionY'], 5),
                  np.round(stack_metadata['currentVersion']['stackResolutionZ'], 5))

    # Get metadata
    ts = sample(renderapi.tilespec.get_tile_specs_from_stack(stack=stack,
                                                             render=render), 1)[0]
    fp = ts.ip[0]['imageUrl'].split('.nl')[1]
    tif = TiffFile(fp)
    metadata = tif.pages[0].description

    # Project data for output to project yaml file
    stack_datum = {
        "title": f"{stack}",
        "dimension": f"{dimensions}",
        "resolution": f"{resolution}",
        "zoomlevels": f"{(maxest_level + 1):.0f}",
        "metadata": metadata,
        "mirrors": [{
            "title": f"{project}_{stack.split('_')[0]}",
            "tile_width": 1024,
            "tile_height": 1024,
            "tile_source_type": 1,
            "fileextension": f"{fmt}",
            "url": f"http://sonic.tnw.tudelft.nl{(export_dir/stack).as_posix()}"
        }]
    }
    stack_data.append(stack_datum)

# Create dict for input into project yaml file
project_data = {
    "project": {
        "title": f"{project}",
        "stacks": stack_data
    }
}
```

      0%|          | 0/4 [00:00<?, ?it/s]

```python
out = f"""\
{project_yaml}
--------\
"""
print(out)

yaml = YAML()
yaml.indent(mapping=2, offset=0)
yaml.dump(project_data, project_yaml)
yaml.dump(project_data, sys.stdout)
```

    /long_term_storage/rlane/CATMAID/projects/pancreas/project.yaml
    --------
    project:
      title: pancreas
      stacks:
      - title: EM_himag_stitched
        dimension: (20544, 20870, 4)
        resolution: (4.85951, 4.85951, 100.0)
        zoomlevels: '7'
        metadata: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><OME xmlns=\"http://www.openmicroscopy.org/Schemas/OME/2012-06\"\
          \ xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"\
          http://www.openmicroscopy.org/Schemas/OME/2012-06 http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd\"\
          >\n    \n    <Instrument\
          \ ID=\"Instrument:0\">\n        <Microscope Manufacturer=\"Delmic\" />\n   \
          \     <Detector ID=\"Detector:0\" Model=\"pcie-6251\" />\n        <Objective\
          \ CalibratedMagnification=\"20950.000000000000000\" ID=\"Objective:0\" />\n\
          \    </Instrument>\n    <Image ID=\"Image:0\" Name=\"Secondary electrons\">\n\
          \        <AcquisitionDate>2019-11-04T23:48:50</AcquisitionDate>\n        <ObjectiveSettings\
          \ ID=\"Objective:0\" />\n        <Pixels DimensionOrder=\"XYZTC\" ID=\"Pixels:0\"\
          \ PhysicalSizeX=\"0.004859505519093\" PhysicalSizeY=\"0.004859505519093\" SizeC=\"\
          1\" SizeT=\"1\" SizeX=\"4096\" SizeY=\"4096\" SizeZ=\"1\" Type=\"uint16\">\n\
          \            <Channel ID=\"Channel:0:0\" Name=\"Secondary electrons\" />\n \
          \           <TiffData FirstC=\"0\" FirstT=\"0\" FirstZ=\"0\" IFD=\"0\" PlaneCount=\"\
          1\" />\n            <Plane ExposureTime=\"0.000004000000000\" IntegrationCount=\"\
          5\" PositionX=\"0.002804099000000\" PositionY=\"0.006330358000000\" TheC=\"\
          0\" TheT=\"0\" TheZ=\"0\" />\n        </Pixels>\n    </Image>\n</OME>"
        mirrors:
        - title: pancreas_EM
          tile_width: 1024
          tile_height: 1024
          tile_source_type: 1
          fileextension: png
          url: http://sonic.tnw.tudelft.nl/long_term_storage/rlane/CATMAID/projects/pancreas/EM_himag_stitched
      - title: EM_lomag_correlated
        dimension: (37808, 37706, 4)
        resolution: (4.85951, 4.85951, 100.0)
        zoomlevels: '7'
        metadata: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ome xmlns=\"\
          http://www.openmicroscopy.org/Schemas/OME/2012-06\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\
          \ xsi:schemalocation=\"http://www.openmicroscopy.org/Schemas/OME/2012-06 http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd\"\
          ><instrument\
          \ id=\"Instrument:0\"><microscope manufacturer=\"Delmic\" model=\"SECOM\"></microscope><detector\
          \ id=\"Detector:0\" model=\"pcie-6251\"></detector><objective calibratedmagnification=\"\
          3033.000000000000000\" id=\"Objective:2\"></objective></instrument><image id=\"\
          Image:2\" name=\"Secondary electrons\">\n<acquisitiondate>2019-11-02T16:53:46</acquisitiondate>\n\
          <objectivesettings id=\"Objective:2\"></objectivesettings>\n<transform a00=\"\
          1.000000000000000\" a01=\"0.000000000000000\" a02=\"0.000000000000000\" a10=\"\
          0.006820329878077\" a11=\"1.000000000000000\" a12=\"0.000000000000000\"></transform>\n\
          <pixels dimensionorder=\"XYZTC\" id=\"Pixels:2\" physicalsizex=\"0.033927219205267\"\
          \ physicalsizey=\"0.033205415562290\" sizec=\"1\" sizet=\"1\" sizex=\"4096\"\
          \ sizey=\"4096\" sizez=\"1\" type=\"uint16\">\n<channel id=\"Channel:2:0\" name=\"\
          Secondary electrons\"></channel>\n<tiffdata firstc=\"0\" firstt=\"0\" firstz=\"\
          0\" ifd=\"2\" planecount=\"1\"></tiffdata>\n<plane exposuretime=\"0.000005100000000\"\
          \ integrationcount=\"6\" positionx=\"0.002226245000000\" positiony=\"0.006381133000000\"\
          \ thec=\"0\" thet=\"0\" thez=\"0\"></plane>\n</pixels>\n</image></ome>"
        mirrors:
        - title: pancreas_EM
          tile_width: 1024
          tile_height: 1024
          tile_source_type: 1
          fileextension: png
          url: http://sonic.tnw.tudelft.nl/long_term_storage/rlane/CATMAID/projects/pancreas/EM_lomag_correlated
      - title: Hoechst_correlated
        dimension: (54294, 55411, 4)
        resolution: (4.85951, 4.85951, 100.0)
        zoomlevels: '7'
        metadata: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ome xmlns=\"\
          http://www.openmicroscopy.org/Schemas/OME/2012-06\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\
          \ xsi:schemalocation=\"http://www.openmicroscopy.org/Schemas/OME/2012-06 http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd\"\
          ><instrument\
          \ id=\"Instrument:0\"><microscope manufacturer=\"Delmic\" model=\"SECOM\"></microscope><detector\
          \ id=\"Detector:0\" model=\"Andor ZYLA-4.2P-USB3- (s/n: VSC-06794)\"></detector><lightsource\
          \ id=\"LightSource:0\" power=\"757.575757575757621\"></lightsource><objective\
          \ calibratedmagnification=\"60.000000000000000\" id=\"Objective:0\" model=\"\
          Optical Objective\"></objective></instrument><image id=\"Image:0\" name=\"Filtered\
          \ colour 1\">\n<acquisitiondate>2019-11-02T15:48:30</acquisitiondate>\n<objectivesettings\
          \ id=\"Objective:0\"></objectivesettings>\n<detectorsettings id=\"Detector:0\"\
          \ offset=\"100.000000000000000\"></detectorsettings>\n<transform a00=\"0.992909187954275\"\
          \ a01=\"-0.118875331654563\" a02=\"0.000000000000000\" a10=\"0.118875331654563\"\
          \ a11=\"0.992909187954275\" a12=\"0.000000000000000\"></transform>\n<pixels\
          \ dimensionorder=\"XYZTC\" id=\"Pixels:0\" physicalsizex=\"0.106416630143553\"\
          \ physicalsizey=\"0.106416630143553\" significantbits=\"16\" sizec=\"1\" sizet=\"\
          1\" sizex=\"2048\" sizey=\"2048\" sizez=\"1\" type=\"uint16\">\n<channel acquisitionmode=\"\
          WideField\" color=\"#0000ffff\" contrastmethod=\"Fluorescence\" emissionwavelength=\"\
          440\" excitationwavelength=\"405\" id=\"Channel:0:0\" illuminationtype=\"Epifluorescence\"\
          \ name=\"Filtered colour 1\">\n<detectorsettings binning=\"1x1\" gain=\"1.100000000000000\"\
          \ id=\"Detector:0\" readoutrate=\"100.000000000000000\"></detectorsettings>\n\
          <lightsourcesettings id=\"LightSource:0\"></lightsourcesettings>\n</channel>\n\
          <tiffdata firstc=\"0\" firstt=\"0\" firstz=\"0\" ifd=\"0\" planecount=\"1\"\
          ></tiffdata>\n<plane exposuretime=\"3.000000000000000\" positionx=\"0.001025598743565\"\
          \ positiony=\"0.006371521974858\" thec=\"0\" thet=\"0\" thez=\"0\"></plane>\n\
          </pixels>\n</image></ome>"
        mirrors:
        - title: pancreas_Hoechst
          tile_width: 1024
          tile_height: 1024
          tile_source_type: 1
          fileextension: png
          url: http://sonic.tnw.tudelft.nl/long_term_storage/rlane/CATMAID/projects/pancreas/Hoechst_correlated
      - title: AF594_correlated
        dimension: (54294, 55411, 4)
        resolution: (4.85951, 4.85951, 100.0)
        zoomlevels: '7'
        metadata: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ome xmlns=\"\
          http://www.openmicroscopy.org/Schemas/OME/2012-06\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\
          \ xsi:schemalocation=\"http://www.openmicroscopy.org/Schemas/OME/2012-06 http://www.openmicroscopy.org/Schemas/OME/2012-06/ome.xsd\"\
          ><instrument\
          \ id=\"Instrument:0\"><microscope manufacturer=\"Delmic\" model=\"SECOM\"></microscope><detector\
          \ id=\"Detector:1\" model=\"Andor ZYLA-4.2P-USB3- (s/n: VSC-06794)\"></detector><lightsource\
          \ id=\"LightSource:1\" power=\"1000.000000000000000\"></lightsource><objective\
          \ calibratedmagnification=\"60.000000000000000\" id=\"Objective:1\" model=\"\
          Optical Objective\"></objective></instrument><image id=\"Image:1\" name=\"Filtered\
          \ colour 2\">\n<acquisitiondate>2019-11-02T15:48:34</acquisitiondate>\n<objectivesettings\
          \ id=\"Objective:1\"></objectivesettings>\n<detectorsettings id=\"Detector:1\"\
          \ offset=\"100.000000000000000\"></detectorsettings>\n<transform a00=\"0.992909187954275\"\
          \ a01=\"-0.118875331654563\" a02=\"0.000000000000000\" a10=\"0.118875331654563\"\
          \ a11=\"0.992909187954275\" a12=\"0.000000000000000\"></transform>\n<pixels\
          \ dimensionorder=\"XYZTC\" id=\"Pixels:1\" physicalsizex=\"0.106416630143553\"\
          \ physicalsizey=\"0.106416630143553\" significantbits=\"16\" sizec=\"1\" sizet=\"\
          1\" sizex=\"2048\" sizey=\"2048\" sizez=\"1\" type=\"uint16\">\n<channel acquisitionmode=\"\
          WideField\" color=\"#ff9500ff\" contrastmethod=\"Fluorescence\" emissionwavelength=\"\
          607\" excitationwavelength=\"555\" id=\"Channel:1:0\" illuminationtype=\"Epifluorescence\"\
          \ name=\"Filtered colour 2\">\n<detectorsettings binning=\"1x1\" gain=\"1.100000000000000\"\
          \ id=\"Detector:1\" readoutrate=\"100.000000000000000\"></detectorsettings>\n\
          <lightsourcesettings id=\"LightSource:1\"></lightsourcesettings>\n</channel>\n\
          <tiffdata firstc=\"0\" firstt=\"0\" firstz=\"0\" ifd=\"1\" planecount=\"1\"\
          ></tiffdata>\n<plane exposuretime=\"3.000000000000000\" positionx=\"0.001025598743565\"\
          \ positiony=\"0.006371521974858\" thec=\"0\" thet=\"0\" thez=\"0\"></plane>\n\
          </pixels>\n</image></ome>"
        mirrors:
        - title: pancreas_AF594
          tile_width: 1024
          tile_height: 1024
          tile_source_type: 1
          fileextension: png
          url: http://sonic.tnw.tudelft.nl/long_term_storage/rlane/CATMAID/projects/pancreas/AF594_correlated


# Lesson 5: High level plotting

```python
import numpy as np
import pandas as pd

import bokeh_catplot

import bokeh.plotting
import bokeh.io

bokeh.io.output_notebook()
```

    Loading BokehJS ...

In this lesson, do some plotting with a high-level package [Bokeh-catplot](https://github.com/justinbois/bokeh-catplot). If you haven't already, you should install it.

    pip install --upgrade bokeh-catplot

For this lesson, we will use [the frog tongue data set from Kleinteich and Gorb](https://doi.org/10.1038/srep05225) that we used in [lesson 21](l21_practice_with_pandas.html). Let's get the data frame loaded in so we can be on our way.

```python
df = pd.read_csv('Dataset/frog_tongue_adhesion.csv', comment='#')

# Have a look so we remember
df.head()
```

      date
      ID
      trial number
      impact force (mN)
      impact time (ms)
      impact force / body weight
      adhesive force (mN)
      time frog pulls on target (ms)
      adhesive force / body weight
      adhesive impulse (N-s)
      total contact area (mm2)
      contact area without mucus (mm2)
      contact area with mucus / contact area without mucus
      contact pressure (Pa)
      adhesive strength (Pa)

      0
      2013_02_26
      I
      3
      1205
      46
      1.95
      -785
      884
      1.27
      -0.290
      387
      70
      0.82
      3117
      -2030

      1
      2013_02_26
      I
      4
      2527
      44
      4.08
      -983
      248
      1.59
      -0.181
      101
      94
      0.07
      24923
      -9695

      2
      2013_03_01
      I
      1
      1745
      34
      2.82
      -850
      211
      1.37
      -0.157
      83
      79
      0.05
      21020
      -10239

      3
      2013_03_01
      I
      2
      1556
      41
      2.51
      -455
      1025
      0.74
      -0.170
      330
      158
      0.52
      4718
      -1381

      4
      2013_03_01
      I
      3
      493
      36
      0.80
      -974
      499
      1.57
      -0.423
      245
      216
      0.12
      2012
      -3975

## High level plotting and rendering with Bokeh

[HoloViews](https://holoviews.org/) is an excellent high-level plotting package that can use Bokeh to render plots. In my view, it is one of the best high-level plotting languaes in the Python plotting landscape. We will work with HoloViews in a future lesson, but in this lesson, we will do our high-level plotting using a package I wrote called [Bokeh-catplot](https://github.com/justinbois/bokeh-catplot).

Why are we using this package and not HoloViews? In next year's bootcamp, I am almost certain we will use HoloViews exclusively for high-level plotting because I suspect that most of the functionality in Bokeh-catplot will be incorporated into HoloViews. Bokeh-catplot exists because HoloViews lacks some important functionality. (More on these very important plot types in a moment; don't worry if you don't know what they are just yet.)

- It does not natively make ECDFs.
- Its definition of a box plot is non-canonical ([my fault](https://github.com/pyviz/holoviews/pull/1926), but will soon be [fixed](https://github.com/pyviz/holoviews/pull/3755)).
- It does not (easily, or as far as I can tell) enable an axis with more than one categorical variable for strip plots (though it does for box plots).
- It has a [bug when making horizontal strip plots](https://github.com/pyviz/holoviews/issues/3748).

There are all relatively minor fixes for HoloViews, which will likely have this functionality in the near future.

Nonetheless, ECDFs and horizontal strip plots are important visualizations and I advocate using them often, and Bokeh-catplot provides a high-level way to render them using Bokeh.

## Plots with categorical variables

Let us first consider the different kinds of data we may encounter as we think about constructing a plot.

- **Quantitative** data may have continuously varying (and therefore ordered) values.
- **Categorical** data has discrete, unordered values that a variable can take.
- **Ordinal** data has discrete, ordered values. Integers are a classic example.
- **Temporal** data refers to time, which can be represented as dates.

In practice, ordinal data can be cast as quantitative or treated as categorical with an ordering enforced on the categories (e.g., categorical data `[1, 2, 3]` becomes `['1', '2', '3']`.). Temporal data can also be cast as quantitative, (e.g., second from the start time). We will therefore focus out attention on quantitative and categorical data.

When we made scatter plots in the previous lesson, both types of data were quantitative. We did actually incorporate categorical information in the form of colors of the glyph (insomniacs and normal sleepers being colored differently) and in tooltips.

But what if we wanted a single type of measurement, say impact force, for each frog. Here, we have the quantitative impact force data and the categorical frog ID data. One of our axes is now categorical.

## Bar graph

To demonstrate how to set up a categorical axis with Bokeh, I will make a bar graph of the mean impact force for each of the four frogs. But before I even begin this, I will give you the following piece of advice: *Don't make bar graphs.* More on that in a moment.

Before we do that, we need to compute the means from the inputted data frame.

```python
df_mean = df.groupby('ID')['impact force (mN)'].mean().reset_index()

# Take a look
df_mean
```

      ID
      impact force (mN)

      0
      I
      1530.20

      1
      II
      707.35

      2
      III
      550.10

      3
      IV
      419.10

To set up a categorical axis, you need to specify the `x_range` (or `y_range` if you want the y-axis to be categorical) as a list with the categories you want on the axis when you instantiate the figure. I will make a horizontal bar graph, so I will specify `y_range`. Also, when I instantiate this figure, because it is not very tall and I do not want the reset tool cut off, I will also explicitly set the tools I want in the toolbar.

```python
p = bokeh.plotting.figure(
    height=200,
    width=400,
    x_axis_label='impact force (mN)',
    y_range=df_mean['ID'].unique()[::-1],
    tools='pan,wheel_zoom,save,reset'
)
```

Now that we have the figure, we can put the bars on. The `p.hbar()` method populates the figure with horizontal bar glyphs. The `right` kwarg says what column of the data source dictates how far to the right to show the bar, while the `height` kwarg says how think the bars are.

I will also ensure the quantitative axis starts at zero and turn off the grid lines on the categorical axis, which is commonly done.

```python
p.hbar(
    source=df_mean,
    y='ID',
    right='impact force (mN)',
    height=0.6
)

# Turn off gridlines on categorical axis
p.ygrid.grid_line_color = None

# Start axes at origin on quantitative axis
p.x_range.start = 0

bokeh.io.show(p)
```

We similarly make vertical bar graphs specifying `x_range` and using `p.vbar()`.

```python
p = bokeh.plotting.figure(
    height=250,
    width=250,
    y_axis_label='impact force (mN)',
    x_range=df_mean['ID'].unique(),
)

p.vbar(
    source=df_mean,
    x='ID',
    top='impact force (mN)',
    width=0.6
)

p.xgrid.grid_line_color = None
p.y_range.start = 0

bokeh.io.show(p)
```

## Bokeh-catplot

Generating the bar graphs was not too painful, even tough we used Bokeh, a low-level plotting library. We would like to make plots more directly. We do not want to have to explicitly pre-process the data, set up the categorical axis, etc. We would like to just provide a data set, say which column(s) is/are categorical and which is quantitative, and then just get our plot. As I mentioned before, Holoviews is excellent for this purpose, but we will use Bokeh-catplot as our high-level package for making plots.

Bokeh-catplot generates plots from tidy data frames where one or more columns contains categorical data and the column of interest in the plot is quantitative.

There are four types of plots that Bokeh-catplot can generate. As you will see, all four of these modes of plotting are meant to give a picture about how the quantitative measurements are *distributed* for each category.

- **Plots with a categorical axis**
    + Box plots: `bokeh_catplot.box()`
    + Strip plots: `bokeh_catplot.strip()`

- **Plots without a categorical axis**
    + Histograms: `bokeh_catplot.histogram()`
    + [ECDFs](https://en.wikipedia.org/wiki/Empirical_distribution_function): `bokeh_catplot.ecdf()`

The first three arguments of each of these functions are necessary to build the plot. They are:

- `data`: A tidy data frame
- `cats`: A list of columns in the data frame that are to be considered as categorical variables in the plot. If `None`, a single box, strip, histogram, or ECDF is plotted.
- `val`: The column of the data frame to be treated as the quantitative variable.

With this in mind, we will put Bokeh-catplot to use on the frog data set.

## Box plots with Bokeh-catplot

As I [discuss below](#Don't-make-bar-graphs), bar graphs are almost never a good choice for visualization. You distill all of the information in the data set down to one or two summary statistics, and then use giant glyphs to show them. As a start for improvement, you could distill the data set down to five or so summary statistics and show those graphically, as opposed to just one or two.

Box plots provide such a summary. I will first make one using `bokeh_catplot.box()` and then describe how a box plot is interpreted.

```python
p = bokeh_catplot.box(
    data=df,
    cats='ID',
    val='impact force (mN)'
)

bokeh.io.show(p)
```

The line in the middle of each box is the median and the top and bottom of the box at the 75th and 25th percentiles, respectively. The distance between the 25th and 75th percentiles is called the **interquartile range**, or IQR. The whiskers of the box plot extend to the most extreme data point within 1.5 times the interquartile range. If any data points are more extreme than the end of the whisker, they are shown individually, and are often referred to as _outliers_.

A box plot can use a useful visualization if you have many data points and it is difficult to plot them all. I rarely find that there are situations where all data cannot be plotted, either with strip plots of ECDFs, which we will cover in a moment, so I generally do not use box plots. Nonetheless, I do not find them too objectionable, as they effectively display important nonparametric summary statistics of your data set.

## Plot all your data

Box plots summarize a data set with summary statistics, but what not plot all your data? You work hard to acquire them. You should show them all. This is a mantra to live by.

Plot all your data.

Let's do that now.

## Strip plots

A **strip plot** is like a scatter plot; it puts a glyph for every measured data point. The only different is that one of the axes is categorical. In this case, you are plotting all of your data.

```python
p = bokeh_catplot.strip(
    data=df,
    cats='ID',
    val='impact force (mN)'
)

bokeh.io.show(p)
```

This is a good plot to make since you are plotting all of your data, but it does have the problem that you cannot tell if multiple ticks overlap. Let's look at some alternatives.

## Histograms

In plotting all of our data in a strip plot, we can roughly see how the data are distributed. There are more measurements where there are more glyphs.

```python
p = bokeh_catplot.histogram(
    data=df,
    cats='ID',
    val='impact force (mN)'
)

bokeh.io.show(p)
```

## ECDFs

Histograms are typically used to display how data are distributed. As an example I will generate Normally distributed data and plot the histogram. (We will learn how to generate data like this when we study random number generation with NumPy in a future lesson. For not, this is for purposes of discussing plotting options.)

```python
# Generate normally distributed data
np.random.seed(353926)
df_norm = pd.DataFrame(data={'x': np.random.normal(size=500)})

# Plot the histogram
p = bokeh_catplot.histogram(
    data=df_norm,
    cats=None,
    val='x'
)

bokeh.io.show(p)
```

This looks similar to the standard Normal curve we are used to seeing and is a useful comparison to a **probability density function** (PDF). However, Histograms suffer from **binning bias**. By binning the data, you are not plotting all of them. In general, if you can **plot all of your data**, you should. For that reason, I prefer not to use histograms for studying how data are distributed, but rather prefer to use ECDFs, which enable plotting of all data.

The ECDF evaluated at x for a set of measurements is defined as

\begin{align}
\text{ECDF}(x) = \text{fraction of measurements } \le x.
\end{align}

While the histogram is an attempt to visualize a probability density function (PDF) of a distribution, the ECDF visualizes the **cumulative density function** (CDF). The CDF, $F(x)$, and PDF, $f(x)$, both completely define a univariate distribution and are related by

\begin{align}
f(x) = \frac{\mathrm{d}F}{\mathrm{d}x}.
\end{align}

The definition of the ECDF is all that you need for interpretation. Once you get used to looking at CDFs, they will become as familiar as PDFs. A peak in a PDF corresponds to an inflection point in a CDF.

To make this more clear, let us look at plot of a PDF and ECDF for familiar distributions, the Gaussian and Binomial.

Now that we know how to interpret ECDFs, lets plot the ECDF for our dummy Normally-distributed data.

```python
p = bokeh_catplot.ecdf(
    data=df_norm,
    cats=None,
    val='x'
)

bokeh.io.show(p)
```

Now that we understand what an ECDF is and how to plot it, let's make a set of ECDFs for our frog data.

```python
p = bokeh_catplot.ecdf(
    data=df,
    cats='ID',
    val='impact force (mN)'
)

bokeh.io.show(p)
```

Each dot in the ECDF is a single data point that we measured. Given the above definition of the ECDF, it is defined for all real $x$. So, formally, the ECDF is a continuous function (with discontinuous derivatives at each data point). So, it should be plotted like a staircase according to the formal definition. We can plot it that way using the `formal` keyword argument.

```python
p = bokeh_catplot.ecdf(
    data=df,
    cats='ID',
    val='impact force (mN)',
    formal=True
)

bokeh.io.show(p)
```

This is still plotting all of your data. The concave corners of the staircase correspond to the measured data. This can be seen by overlaying the "dot" version of the ECDFs.

```python
p = bokeh_catplot.ecdf(
    data=df,
    cats='ID',
    val='impact force (mN)',
    p=p
)

bokeh.io.show(p)
```

## Customization with Bokeh-catplot

You may have noticed in the discussion of ECDFs that I introduced some new keyword arguments, `formal` and `p`. In fact, each of the four plotting functions also has the following additional optional keyword arguments.

- `palette`: A list of hex colors to use for coloring the markers for each category. By default, it uses the default color scheme of [Vega-Lite](https://vega.github.io/vega-lite/).
- `order`: If specified, the ordering of the categories to use on the categorical axis and legend (if applicable). Otherwise, the order of the inputted data frame is used.
- `p`: If specified, the `bokeh.plotting.Figure` object to use for the plot. If not specified, a new figure is created.

The respective plotting functions also have kwargs that are specific to each (such as `formal` for `bokeh_catplot.ecdf()`. Examples highlighting some, but not all, customizations follow. You can find out what kwargs are available for each function by reading their doc strings, e.g., with

```python
bokeh_catplot.box?
```

Any kwargs not in the function call signature are passed to `bokeh.plotting.figure()` when the figure is instantiated.

#### Customizing box plots

We can also have horizontal box plots.

```python
p = bokeh_catplot.box(
    data=df,
    cats='ID',
    val='impact force (mN)',
    horizontal=True
)

bokeh.io.show(p)
```

We can independently specify properties of the marks using `box_kwargs`, `whisker_kwargs`, `median_kwargs`, and `outlier_kwargs`. For example, say we wanted our colors to be [Betancourt red](https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html#step_four:_build_a_generative_model19), and that we wanted the outliers to also be that color and use diamond glyphs.

```python
p = bokeh_catplot.box(
    data=df,
    cats='ID',
    val='impact force (mN)',
    whisker_caps=True,
    outlier_marker='diamond',
    box_kwargs=dict(fill_color='#7C0000'),
    whisker_kwargs=dict(line_color='#7C0000', line_width=2),
)

bokeh.io.show(p)
```

#### Custominzing strip plots

To help alleviate the overlap problem, we can make a strip plot with dash markers and add some transparency.

```python
p = bokeh_catplot.strip(
    data=df,
    cats='ID',
    val='impact force (mN)',
    marker='dash',
    marker_kwargs=dict(alpha=0.5)
)

bokeh.io.show(p)
```

The problem with strip plots is that they can have trouble with overlapping data points. A common approach to deal with this is to "jitter," or place the glyphs with small random displacements along the categorical axis. I do that here, allowing for hover tools that give more information about the respective data points.

```python
p = bokeh_catplot.strip(
    data=df,
    cats='ID',
    val='impact force (mN)',
    jitter=True,
    tooltips=[
        ('trial', '@{trial number}'),
        ('adh force', '@{adhesive force (mN)}')
    ]
)

bokeh.io.show(p)
```

With any of the plots, you can have more than one categorical column, and the categorical axes are nicely spaced and formatted. Here, we'll categorize by frog ID and by trial number.

```python
p = bokeh_catplot.strip(
    data=df,
    cats=['ID', 'trial number'],
    val='impact force (mN)',
    color_column='trial number',
    width=550
)

bokeh.io.show(p)
```

#### Strip-box plots

Even while plotting all of the data, we sometimes want to graphically display summary statistics, in which case overlaying a box plot and a jitter plot is useful. To populate an existing Bokeh figure with new glyphs from another catplot, pass in the `p` kwarg. You should be careful, though, because you need to make sure the `cats`, `val`, and `horizontal` arguments exactly match.

```python
p = bokeh_catplot.strip(
    data=df,
    cats='ID',
    val='impact force (mN)',
    horizontal=True,
    jitter=True,
    height=250
)

p = bokeh_catplot.box(
    data=df,
    cats='ID',
    val='impact force (mN)',
    horizontal=True,
    whisker_caps=True,
    display_points=False,
    box_kwargs=dict(fill_color=None, line_color='gray'),
    median_kwargs=dict(line_color='gray'),
    whisker_kwargs=dict(line_color='gray'),
    p=p,
)

bokeh.io.show(p)
```

#### Customizing histograms

We could plot normalized histograms using the density kwarg, and we'll make the plot a little wider to support the legend.

```python
# Plot the histogram
p = bokeh_catplot.histogram(
    data=df_norm,
    cats=None,
    val='x',
    density=True,
)

bokeh.io.show(p)
```

#### Customizing ECDFs

Instead of plotting a separate ECDF for each category, we can put all of the categories together on one ECDF and color the points by the categorical variable by using the `kind='colored'` kwarg. Note that if we do this, we can only have the "dot" style ECDF, not the formal staircase.

```python
p = bokeh_catplot.ecdf(
    data=df,
    cats='ID',
    val='impact force (mN)',
    kind='colored',
)

bokeh.io.show(p)
```

## Don't make bar graphs

Bar graphs, especially with error bars (in which case they are called dynamite plots), are typically awful. They are pervasive in biology papers. I have yet to find a single example where a bar graph is the best choice. Strip plots (with jitter) or even box plots, are more informative and almost always preferred. In fact, ECDFs are often better even than these. Here is a simple message:

Don't make bar graphs.

What should I do instead you ask? The answer is simple: **plot all of your data** when you can. If you can't, box plots are always better than bar graphs.

## Computing environment

```python
%load_ext watermark
%watermark -v -p numpy,pandas,bokeh,bokeh_catplot,jupyterlab
```

    CPython 3.6.8
    IPython 7.1.1

    numpy 1.16.4
    pandas 0.24.2
    bokeh 1.2.0
    bokeh_catplot 0.1.3
    jupyterlab 1.0.2

```python

```