Natural language processing

Natural language processing (NLP) is a field of computer science that studies how computers and humans interact. In the 1950s, Alan Turing published an article that proposed a measure of intelligence, now called the Turing test. More modern techniques, such as deep learning, have produced results in the fields of language modeling, parsing, and natural-language tasks.

@anmolsjoshi

🚀 Add missing tokenizer test files

Several tokenizers currently have no associated tests. I think that adding the test file for one of these tokenizers could be a very good way to make a first contribution to transformers.

Tokenizers concerned

not yet claimed

claimed

Flaubert @anmolsjoshi

In gensim/models/fasttext.py:

    model = FastText(
        vector_size=m.dim,
        vector_size=m.dim,
        window=m.ws,
        window=m.ws,
        epochs=m.epoch,
        epochs=m.epoch,
        negative=m.neg,
        negative=m.neg,
        # FIXME: these next 2 lines read in unsupported FB FT modes (loss=3 softmax or loss=4 onevsall,
        # or model=3 supervi

Describe the bug

Streaming Datasets can't be pickled, so any interaction between them and multiprocessing results in a crash.

Steps to reproduce the bug

import transformers
from transformers import Trainer, AutoModelForCausalLM, TrainingArguments
import datasets

ds = datasets.load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True).with_format("

Is your feature request related to a problem? Please describe.
I typically used compressed datasets (e.g. gzipped) to save disk space. This works fine with AllenNLP during training because I can write my dataset reader to load the compressed data. However, the predict command opens the file and reads lines for the Predictor. This fails when it tries to load data from my compressed files.

Rather than simply caching nltk_data until the cache expires and it's forced to re-download the entire nltk_data, we should perform a check on the index.xml which refreshes the cache if it differs from some previous cache.

I would advise doing this in the same way that it's done for requirements.txt:
https://github.com/nltk/nltk/blob/59aa3fb88c04d6151f2409b31dcfe0f332b0c9ca/.github/wor

Mar	APR	May
	15
2021	2022	2023

Natural language processing

Here are 18,386 public repositories matching this topic...

huggingface / transformers

🚀 Add missing tokenizer test files

Tokenizers concerned

not yet claimed

claimed

apachecn / ailearning

google-research / bert

hankcs / HanLP

explosion / spaCy

oxford-cs-deepnlp-2017 / lectures

RasaHQ / rasa

virgili0 / Virgilio

keon / awesome-nlp

RaRe-Technologies / gensim

huggingface / datasets

Describe the bug

Steps to reproduce the bug

bharathgs / Awesome-pytorch-list

NLP-LOVE / ML-NLP

flairNLP / flair

allenai / allennlp

nltk / nltk

graykode / nlp-tutorial

spencermountain / compromise

chiphuyen / stanford-tensorflow-tutorials

botpress / botpress

ashishpatel26 / 500-AI-Machine-learning-Deep-learning-Computer-vision-NLP-Projects-with-code

stanfordnlp / CoreNLP

sloria / TextBlob

PaddlePaddle / PaddleHub

brightmart / text_classification

brightmart / nlp_chinese_corpus

crownpku / Awesome-Chinese-NLP

ymcui / Chinese-BERT-wwm

ml-tooling / best-of-ml-python

jadore801120 / attention-is-all-you-need-pytorch