mmdt-tokenizer
"mmdt-tokenizer" is an open-source Python toolkit for Myanmar-language text segmentation. It uses a transparent, grammar-informed pipeline to protect URLs, emails, dates, phone numbers, and numeric expressions before segmenting text into syllable- and word-level units. Curated linguistic lexicons and structural rules recognize postpositions, conjunctions, classifiers, verbal particles, and sentence-final particles, while merging related predicate constructions into meaningful tokens. Designed for low-resource settings, the system is lightweight, auditable, and extensible without model retraining, providing essential language infrastructure for Myanmar NLP research, corpus development, search, information extraction, and other language technologies.