A small library for converting tokenized PHP source code into XML (and potentially other formats)

Parser Building Toolkit for JavaScript

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

Solves basic Russian NLP tasks, API for lower level Natasha projects

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

Self-contained Japanese Morphological Analyzer written in pure Go

Open Korean Text Processor - An Open-source Korean Text Processor

minigo🐥is a small Go compiler made from scratch. It can compile itself. The goal is to be the most easy-to-understand Go compiler.

🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

The fast scanner generator for Java™ with full Unicode support

Lex machinary for go.

A multilingual command line sentence tokenizer in Golang

High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

Python port of Moses tokenizer, truecaser and normalizer

Juman++ (a Morphological Analyzer Toolkit)

Query Translator is a search query translator with AST representation

Fast, Consistent Tokenization of Natural Language Text

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

Bitextor generates translation memories from multilingual websites.

Tiny JavaScript tokenizer.

C++14 fast and efficient tokenizer

Collection of developer toolkits

Source code tokenizer

aim to use JapaneseTokenizer as easy as possible

一个微型&算法全面的中文分词引擎 | A micro tokenizer for Chinese

Text tokenization and sentence segmentation (segtok v2)

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer