| --- |
| license: cc-by-sa-4.0 |
| datasets: |
| - jerteh/cc100-sr-jerteh |
| - jerteh/SrpWiki |
| - jerteh/SrpELTeC |
| - srwac |
| - procesaur/znanje |
| language: |
| - sr |
| tags: |
| - srpski |
| - Serbian |
| - RoBERTa |
| - BERT |
| - MaskedLM |
| widget: |
| - text: Kada bi čovek znao gde će pasti on bi<mask>. |
| --- |
| |
|
|
|
|
| <h4><i class="highlight-container"><b class="highlight">jerteh-355</b></i> — |
| Najveći BERT model specijalno obučen za srpski jezik.</h4> |
|
|
| <img src="cover.png" class="cover"> |
|
|
| <!--div id="zastava"> |
|
|
| <div class="grb"> |
| <img src="https://www.ai.gov.rs/img/logo_60x120-2.png" style="position:relative; left:30px; z-index:10; height:85px"> |
| </div> |
| <table width=100% style="border:0px"> |
| <tr style= |
| "background-color:#C6363C;width:100%;border:0px;height:30px"><td></td></tr |
| > |
| <tr style="background-color:#0C4076;width:100%;border:0px;height:30px"><td></td></tr> |
| <tr style="background-color:#ffffff;width:100%;border:0px;height:30px"><td></td></tr> |
| </table> |
| </div--> |
| |
|
|
| <ul style="font-weight:bold"> |
| <li>Vektorizuje reči, ili dopunjava nedostajuće reči u tekstu</li> |
| <li>Zasnovan na RoBERTa-large arhitekturi, 355 miliona parametara</li> |
| <li>Obučavan na korpusu srpskog jezika veličine 4 milijarde tokena</li> |
| <li>Najbolji rezultati u modelovanju maskiranog jezika za srpski!</li> |
| <li>Jednaka podrška unosa i na ćirilici i na latinici!</li> |
| </ul> |
| </div> |
| |
| Pored skupova navedenih u metapodacima, model je obučavan i na ostalim korpusima [Društva za jezičke resurse i tehnologije](https://jerteh.rs), |
| uključujući korpuse savremenog srpskog jezika: SrpKor2013 i SrpKor2021, |
| kao i korpus [PDRS 1.0](https://www.clarin.si/repository/xmlui/handle/11356/1752) razvijen od strane Instituta za Srpski jezik SANU. |
|
|
|
|
| ## Upotreba |
|
|
| ```python |
| >>> from transformers import pipeline |
| >>> unmasker = pipeline('fill-mask', model='jerteh/jerteh-355') |
| >>> unmasker("Kada bi čovek znao gde će pasti on bi<mask>.") |
| >>> |
| ``` |
|
|
| ``` |
| [{'score': 0.2131326049566269, 'token': 11379, 'token_str': ' pao', 'sequence': 'Kada bi čovek znao gde će pasti on bi pao.'}, |
| {'score': 0.18836458027362823, 'token': 20536, 'token_str': ' pobegao', 'sequence': 'Kada bi čovek znao gde će pasti on bi pobegao.'}, |
| {'score': 0.07937008887529373, 'token': 10799, 'token_str': ' umro', 'sequence': 'Kada bi čovek znao gde će pasti on bi umro.'}, |
| {'score': 0.04340635612607002, 'token': 7797, 'token_str': ' otišao', 'sequence': 'Kada bi čovek znao gde će pasti on bi otišao.'}, |
| {'score': 0.038474686443805695, 'token': 25984, 'token_str': ' odustao', 'sequence': 'Kada bi čovek znao gde će pasti on bi odustao.'}] |
| ``` |
|
|
| ```python |
| >>> from transformers import AutoTokenizer, AutoModelForMaskedLM |
| >>> from torch import LongTensor, no_grad |
| >>> from scipy import spatial |
| >>> tokenizer = AutoTokenizer.from_pretrained('jerteh/jerteh-355') |
| >>> model = AutoModelForMaskedLM.from_pretrained('jerteh/jerteh-355', output_hidden_states=True) |
| >>> x = " pas" |
| >>> y = " mačka" |
| >>> z = " svemir" |
| >>> tensor_x = LongTensor(tokenizer.encode(x, add_special_tokens=False)).unsqueeze(0) |
| >>> tensor_y = LongTensor(tokenizer.encode(y, add_special_tokens=False)).unsqueeze(0) |
| >>> tensor_z = LongTensor(tokenizer.encode(z, add_special_tokens=False)).unsqueeze(0) |
| >>> model.eval() |
| >>> with no_grad(): |
| >>> vektor_x = model(input_ids=tensor_x).hidden_states[-1].squeeze() |
| >>> vektor_y = model(input_ids=tensor_y).hidden_states[-1].squeeze() |
| >>> vektor_z = model(input_ids=tensor_z).hidden_states[-1].squeeze() |
| >>> print(spatial.distance.cosine(vektor_x, vektor_y)) |
| >>> print(spatial.distance.cosine(vektor_x, vektor_z)) |
| >>> |
| ``` |
|
|
| ``` |
| 0.029090166091918945 |
| 0.0369451642036438 |
| ``` |
|
|
| <h4>U slučaju potrebe za bržim modelom, pogledajte <a href="https://huggingface.co/jerteh/jerteh-81" class="highlight-container"> |
| <b class="highlight">jerteh-81</b></a> — mali BERT model za srpski jezik.</h4> |
|
|
| <h4>U slučaju potrebe za generativnim modelom, pogledajte <a href="https://huggingface.co/jerteh/gpt2-orao" class="highlight-container"> |
| <b class="highlight">gpt2-orao</b></a> i <a href="https://huggingface.co/jerteh/gpt2-vrabac" class="highlight-container"> |
| <b class="highlight">gpt2-vrabac</b></a></h4> |
|
|
|
|
|
|
| <div class="inline-flex flex-col" style="line-height: 1.5;padding-right:40px"> |
| <div style="text-align: center; margin-top: 3px; font-size: 16px; font-weight: 800">Autor</div> |
| <a href="https://huggingface.co/procesaur"> |
| <div class="flex"> |
| <div |
| style="display:DISPLAY_1; margin-left: auto; margin-right: auto; width: 92px; height:92px; border-radius: 50%; |
| background-size: cover; background-image: url('https://huggingface.co/proxy/cdn-uploads.huggingface.co/production/uploads/1673534533167-63bc254fb8c61b8aa496a39b.jpeg?w=200&h=200&f=face')"> |
| </div> |
| </div> |
| </a> |
| <div style="text-align: center; font-size: 16px; font-weight: 800">Mihailo Škorić</div> |
| <div> |
| <a href="https://huggingface.co/procesaur"> |
| <div style="text-align: center; font-size: 14px;">@procesaur</div> |
| </a> |
| </div> |
| </div> |
| </div> |
| |
| <div class="inline-flex flex-col" style="line-height: 1.5;padding-right:40px"> |
| <div style="text-align: center; margin-top: 3px; font-size: 16px; font-weight: 800">Computation</div> |
| <a href="https://rgf.bg.ac.rs/"> |
| <div class="flex"> |
| <div |
| style="display:DISPLAY_1; margin-left: auto; margin-right: auto; width: 92px; height:92px; border-radius: 50%; |
| background-size: contain; background-image: url(https://rgf.bg.ac.rs/slike/Rgf-logo.jpg);background-repeat: no-repeat; |
| background-position: center;"> |
| </div> |
| </div> |
| </a> |
| <div style="text-align: center; font-size: 16px; font-weight: 800" title="Rudarsko-Geološki fakultet Univerziteta u Beogradu">Rudarsko-Geološki fakultet</div> |
| <div> |
| <a href="https://www.ai.gov.rs/"> |
| <div style="text-align: center; font-size: 14px;">rgf.bg.ac.rs</div> |
| </a> |
| </div> |
| </div> |
| </div> |
| |
| <div class="inline-flex flex-col" style="line-height: 1.5;padding-right:40px"> |
| <div style="text-align: center; margin-top: 3px; font-size: 16px; font-weight: 800">Data</div> |
| <a href="https://jerteh.rs/"> |
| <div class="flex"> |
| <div |
| style="display:DISPLAY_1; margin-left: auto; margin-right: auto; width: 92px; height:92px; border-radius: 50%; |
| background-size: contain; background-image: url(https://huggingface.co/proxy/cdn-avatars.huggingface.co/v1/production/uploads/1673601491672-63bc254fb8c61b8aa496a39b.png?w=200&h=200&f=face);background-repeat: no-repeat; |
| background-position: center;"> |
| </div> |
| </div> |
| </a> |
| <div style="text-align: center; font-size: 16px; font-weight: 800" title="Društvo za jezičke resurse i tehnologije">JeRTeh</div> |
| <div> |
| <a href="https://huggingface.co/jerteh"> |
| <div style="text-align: center; font-size: 14px;">@jerteh</div> |
| </a> |
| </div> |
| </div> |
| </div> |
| |
|
|
|
|
| ## Citiranje |
|
|
| ```bibtex |
| @article{skoric24modeli, |
| author = {Mihailo \vSkori\'c}, |
| title = {Novi jezi\vcki modeli za srpski jezik}, |
| journal = {Infoteka}, |
| volume = {24}, |
| issue = {1}, |
| year = {2024}, |
| publisher = {Zajednica biblioteka univerziteta u Srbiji, Beograd}, |
| url = {https://arxiv.org/abs/2402.14379} |
| } |
| ``` |
|
|
| <style> |
| .ffeat: { |
| color:red |
| } |
| |
| .cover { |
| width: 100%; |
| margin-bottom: 5pt |
| } |
| |
| .highlight-container, .highlight { |
| position: relative; |
| text-decoration:none |
| } |
| |
| .highlight-container { |
| display: inline-block; |
| |
| } |
|
|
| .highlight{ |
| color:white; |
| text-transform:uppercase; |
| font-size: 16pt; |
| } |
|
|
| .highlight-container{ |
| padding:5px 10px |
| } |
| |
| .highlight-container:before { |
| content: " "; |
| display: block; |
| height: 100%; |
| width: 100%; |
| margin-left: 0px; |
| margin-right: 0px; |
| position: absolute; |
| background: #e80909; |
| transform: rotate(2deg); |
| top: -1px; |
| left: -1px; |
| border-radius: 20% 25% 20% 24%; |
| padding: 10px 18px 18px 10px; |
| } |
|
|
| div.grb, #zastava>table { |
| position:absolute; |
| top:0px; |
| left: 0px; |
| margin:0px |
| } |
|
|
| div.grb>img, #zastava>table{ |
| margin:0px |
| } |
| |
| #zastava { |
| position: relative; |
| margin-bottom:120px |
| } |
| </style> |
