melanietosikmelanietosik
13

Viterbi Pos Tagger

screenshot of Viterbi Pos Tagger

Viterbi part-of-speech tagger, trained on Wall Street Journal (WSJ) data

Github

Overview

The Viterbi Part-of-Speech (POS) Tagger is an innovative project designed to accurately assign POS tags to words in a sentence, based on the hidden Markov model (HMM) approach outlined in "Speech and Language Processing" by Jurafsky and Martin. By leveraging the Viterbi algorithm, this tagger estimates transition and emission probabilities from a training dataset and efficiently decodes POS tags for unseen sentences. This makes it an invaluable tool for natural language processing tasks, enhancing text analysis and improving machine understanding of language nuances.

Implementing a bigram distribution system, the tagger effectively handles vocabulary and unknown tokens through a well-structured training phase. The transition and emission counts are meticulously derived, leading to accurate probability distributions that contribute to the robustness of the tagging process. For anyone interested in linguistic data processing and algorithmic approaches to language, the Viterbi POS Tagger stands out as a powerful resource.

Features

  • Hidden Markov Model Implementation: Utilizes HMM to estimate probabilities effectively, enhancing POS tagging accuracy.

  • Bigram Distribution Training: Trains on bigram distributions to capture relationships between adjacent tokens, resulting in improved tagging precision.

  • Handling Unknown Tokens: Assigns a special unknown word token for rare occurrences, ensuring that the model maintains its integrity even with limited data.

  • Additive Smoothing Techniques: Applies smoothing to address unseen transitions and emissions, enhancing the model's adaptability.

  • Efficient Training and Decoding: Requires only about 60 seconds for model training and decoding of the development splits, making it quick to deploy.

  • Customizable Settings: Provides adjustable parameters in a dedicated settings script, allowing users to fine-tune paths and other configurations easily.

  • Comprehensive Evaluation Reports: Includes an evaluation script that generates detailed classification metrics and confusion matrices, facilitating robust performance assessment.

  • User-Friendly Execution: Simplifies the running and rerunning processes for tagging through straightforward scripts, accommodating both development and test sets.