Boost Engine for Token Economy

snapdragon is an extremely pluggable, powerful and easy-to-use parser-renderer factory.

Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.

NLP Functions for amplifying negations, managing elisions, creating ngrams, stems, phonetic codes to tokens and more.

Extract JavaScript code comments from a string or glob of files.

Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work?

Uses babel to extract JavaScript code comments from a string. Returns an array of comment objects, with line, column, index, comment type and comment string.

Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.

Implemented transformer NN block for Machine translation, text classfication, Natural language inference as well as Machine reading comprehension model.

A token based HTML Document parser and minifier written in PHP. Extract attribute values and text using CSS selectors.

Transforms tokens into original source code (while preserving whitespace)

More detailed documentation for the Python tokenize module

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Korean text data preprocess toolkit for NLP

Snapdragon utility for creating a stack.

Splits a JSON string into an annotated list of tokens

Adds a location object to snapdragon token or AST node.

simple regex for correcting punctuations

String ngram splitter.

Create a snapdragon token. Used by the snapdragon lexer, but can also be used by plugins.

A Python toolkit to generate a tokenized dump of Wikipedia for NLP

Python3 module to tokenize english sentences.

A PHP Library to extract n-grams from a text. Simple preprocessing tools (cleaning, tokenizing) included.

Token(ize) Sort, also support weight sort

Basic text to numbers tokenizer for machine learning

Sentiment analysis for amazon product reviews using NLTK, Scikit-Learn, and Keras. Using hyperparameter search and LSTM, our best model achieves ~96% accuracy.

Basic Tokenizer - Creates tokens - enabling for creation of personal syntax; removal of unwanted characters etc

A Read–Eval–Print Loop (REPL) and Better Bash Lang & Compiler written in C++