The Wayback Machine - http://web.archive.org/web/20220430073227/https://github.com/topics/tokenization
Skip to content
#

tokenization

Star

Here are 417 public repositories matching this topic...

Language: All
Filter by language
All 417 Jupyter Notebook 123 Python 98 JavaScript 30 Java 23 Go 19 HTML 14 Rust 12 C++ 10 C 9 PHP 9
Sort: Best match
Sort options
Best match Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

lunasec-io / lunasec

Sponsor
Star 1k

LunaSec - Open Source AppSec platform that automatically notifies you the next time vulnerabilities like Log4Shell or node-ipc happen. Track your dependencies and builds in a centralized service. Get started in one-click via our GitHub App or host it yourself. https://github.com/apps/lunatrace-by-lunasec/

security dependency-analysis cybersecurity pci-dss web-security compliance hardening scanning cve-scanning tokenization gdpr security-tools devsecops zero-trust soc2 privacy-by-design sbom scanning-tool sbom-generator log4shell
  • Updated Apr 30, 2022
  • TypeScript

RavenProject / Ravencoin

Star 980
Open

Make 'transaction details' more uniform

soullesscomputerboy
soullesscomputerboy commented Jun 1, 2021

Currently different transaction types show different details. Asset transactions don't show "To" and "From". On RVN Transactions the "From" is almost always listed as "unknown". Maybe change this to list a sending address and if there is a label on one of the sending addresses, put the label after it in parenthesis like it has handled in the "To" section. See inconsistencies below:
![assettx](htt

Read more
enhancement gui good first issue
Find more good first issues

cbaziotis / ekphrasis

Star 550

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
  • Updated Feb 8, 2021
  • Python

milaan9 / Python_Natural_Language_Processing

Star 125

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.

nlp ipython-notebook named-entity-recognition bag-of-words tf-idf stopwords tokenization stemming lemmatization sentence-segmentation termfrequency partofspeech-tagger vocabulary-matching python4everybody python4datascience tutor-milaan9 inversedocumentfrequency
  • Updated Nov 9, 2021
  • Jupyter Notebook

GlitchedPolygons / l8w8jwt

Star 62
Open

Suggested extension / improvement

1
firthmj
firthmj commented Jan 21, 2021

How easy would it be to change the library to have versions of the encode and decode functions where the payload JSON was provided / returned just as the JSON text?

There are other good JSON generation / parsing libraries available, and some people may wish to use them to generate or process the payload, rather than the built in claim processing.

Read more
enhancement good first issue

Improve this page

Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."

Learn more