🚀 Add missing tokenizer test files

Several tokenizers currently have no associated tests. I think that adding the test file for one of these tokenizers could be a very good way to make a first contribution to transformers.

Tokenizers concerned

not yet claimed

none

claimed

• LED @nnlnr
• Flaubert @anmolsjoshi
• Longformer @tgadeliya
• Electra @rajathb

https://github.com/open-mmlab/mmdetection/blob/7a9bc498d5cc972171ec4f7332afcd70bb50e60e/tools/analysis_tools/coco_error_analysis.py#L43
This I believe is for coco format, but I couldn't find any files for plotting precision or precision vs recall chart for pascal voc format.

🐛 Bug

tuner.scale_batch_size finds the suitable batch size and update the batch size of the model AND datamodule.
For the model, tuner.scale_batch_size updates the batch size in the model regardless of model.batch_size and model.hparams.batch_size.

However, for the datamodule, tuner.scale_batch_size updates datamodule.batch_size only, and keep datamodule.hparams.batch_size

Change tensor.data to tensor.detach() due to
pytorch/pytorch#6990 (comment)
tensor.detach() is more robust than tensor.data.

🐛 Describe the bug

When running T.ToUndirect()(data) on a heterogeneous graph you will still get false for running is_undirect()

from torch_geometric.datasets import OGB_MAG
import torch_geometric.transforms as T

dataset = OGB_MAG(root='data', preprocess='metapath2vec')
data = dataset[0]
data .is_undirected()
data = T.ToUndirected()(data)
data .is_undirected()

Found

Although the results look nice and ideal in all TensorFlow plots and are consistent across all frameworks, there is a small difference (more of a consistency issue). The result training loss/accuracy plots look like they are sampling on a lesser number of points. It looks more straight and smooth and less wiggly as compared to PyTorch or MXNet.

It can be clearly seen in chapter 6([CNN Lenet](ht

Describe the bug

Streaming Datasets can't be pickled, so any interaction between them and multiprocessing results in a crash.

Steps to reproduce the bug

import transformers
from transformers import Trainer, AutoModelForCausalLM, TrainingArguments
import datasets

ds = datasets.load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True).with_format("

New Operator

Describe the operator

Why is this operator necessary? What does it accomplish?

This is a frequently used operator in tensorflow/keras

Can this operator be constructed using existing onnx operators?

If so, why not add it as a function?

I don't know.

Is this operator used by any model currently? Which one?

Are you willing to contribute it?

Describe the issue:
During computing Channel Dependencies reshape_break_channel_dependency does following code to ensure that the number of input channels equals the number of output channels:

in_shape = op_node.auxiliary['in_shape']
out_shape = op_node.auxiliary['out_shape']
in_channel = in_shape[1]
out_channel = out_shape[1]
return in_channel != out_channel

This is correct

Is your feature request related to a problem? Please describe.
I typically used compressed datasets (e.g. gzipped) to save disk space. This works fine with AllenNLP during training because I can write my dataset reader to load the compressed data. However, the predict command opens the file and reads lines for the Predictor. This fails when it tries to load data from my compressed files.