token

Tokenizer classes for several NLP tasks: splitting a text on white space, using a REGEX expression, ... This package is based on the tokenspan package, see https://framagit.org/nlp/tokenspan

Extract part of a string in a versatile way, and without destroying information from the parent string. Allows discontinuous part of a string to be collected as an ExtractionString. Allows several strategies of string-splitting at the same time, for a given string.

Deprecated from sept. 2022. See https://framagit.org/nlp/extractionstring for improved tools to extract any sub-string from a parent one without losing information from the parent string.