nlp-analyst

Analyze text data: distributions, vocabulary, language detection, quality assessment.

PublishedJun 16, 2026

Loading actions...

5 minBeginnerprompt4 files

Skill content

Main instructions and any bundled files for this skill.

markdown

Additional Files (3)

NLP Analyst

Relevance Gate (when running at a hook point)

When invoked at after-eda in a core workflow:

Check for NLP/text indicators in the project:
- Text corpus files (.txt, .csv with text columns, .jsonl with text fields)
- Python files importing nltk, spacy, transformers, gensim, sklearn.feature_extraction.text
- Requirements listing NLP packages (nltk, spacy, textblob, gensim)
- Directories named corpus/, text_data/, documents/

If NO NLP indicators found -- write skip report and exit:

from ml_utils import save_agent_report
save_agent_report("nlp-analyst", {
    "status": "skipped",
    "reason": "No NLP/text artifacts found in project"
})

If NLP indicators found: proceed with text analysis

Capabilities

Vocabulary Analysis

Token count, unique token count, type-token ratio
Vocabulary growth curve (Heaps' law fit)
Hapax legomena and rare word counts
Vocabulary overlap between corpus splits

Distribution Analysis

Document length distribution (words, characters, sentences)
Token frequency distribution (Zipf's law fit)
N-gram frequency analysis (unigrams, bigrams, trigrams)
Part-of-speech tag distributions

Language Detection

Per-document language identification
Multilingual corpus detection and language distribution
Encoding issues and non-text content detection

Text Quality Assessment

Missing/empty text field detection
Duplicate document identification (exact and near-duplicate via MinHash)
Noise detection (HTML tags, URLs, email addresses, special characters)
Text readability scores (Flesch-Kincaid, Coleman-Liau)

Corpus Statistics

Class/label distribution for labeled datasets
Train/test vocabulary overlap
Out-of-vocabulary rate estimation
Corpus-level summary statistics

Report Bus

Write report using save_agent_report("nlp-analyst", {...}) with:

vocabulary stats (total tokens, unique tokens, TTR, hapax count)
document length distribution (mean, median, min, max, std)
language distribution
quality issues (duplicates, empty docs, noise)
n-gram frequency tables (top 20)
recommendations for preprocessing

Contents

View Original Source

Related Skills

General

PromptBeginner5 minmarkdown

Untitled Skill

193

Jan 12, 2026

General

PromptBeginner5 minmarkdown

Frontend Typescript Linting.mdc

TypeScript and ESLint rules that MUST be followed when creating, modifying, or reviewing any file under apps/frontend/, including .ts, .tsx, .js, and .jsx files. Also apply when discussing frontend li...

160

Feb 15, 2026

General

PromptBeginner5 minmarkdown

2. Apply Deepthink Protocol (reason about dependencies

risks

127

Jan 15, 2026