Saeed Rahmani - Department of Computer and IT Engineering, Shiraz University, Shiraz, Iran
Seyyed Mostafa Fakhrahmad - Department of Computer and IT Engineering, Shiraz University, Shiraz, Iran
Mohammad Hadi Sadredini - Department of Computer and IT Engineering, Shiraz University, Shiraz, Iran

By the advent of new information sources and the expansion of text data, natural language processing (NLP) has become one of the key parts of all the systems dealing with human written texts, and part of speech (POS) tagging is an inseparable part of all NLP tasks. As a result, it is of the paramount importance to enhance the accuracy of POS tagging. In this paper, applying language model and statistical information, we introduce a new approach to tokenize sentences and prepare them to be labeled by POS taggers. An evaluation shows that the proposed method yields a precision of 98 percent for tokenizing, and

Tokenizer, Part of Speech Tagging, Probabilistic Model, Compound Tokens