nlp

Bag of Words tools to represent natural language processing, and associate a few graph representation of a document. The main interest of this module is to be agnostic of the preprocessing and/or normalizing and or clean and/or tokenization protocols

Modification of strings without loss of information. Useful for cleaning, normalizing, de-noising, filtering, ... any work when insertion and deletion from and to a string are in use.

Tools and thoughts to construct learnable tokenizer, that is, the module that cuts a string (say, some text) into sub-strings (say some words). The goal is to work on (possibly really) noisy environment, as for instance at the output of optical character recognition task, or for specific applications / expert-domain speech recognition.

Tokenizer classes for several NLP tasks: splitting a text on white space, using a REGEX expression, ... This package is based on the tokenspan package, see https://framagit.org/nlp/tokenspan

Deprecated from sept. 2022. See https://framagit.org/nlp/extractionstring for improved tools to extract any sub-string from a parent one without losing information from the parent string.