论文部分内容阅读
大量的新词伴随着微博的快速发展而产生,这些新词具有传播速度快及与其他词组合方式灵活的特点,而且在进行分词处理时容易被切分为不同的字符串。提出了一种融合词频特性及邻接变化数的微博新词识别方法。该方法首先对大规模的微博语料进行分词,然后将在两停用词间的相邻字串两两组合,根据组合后的字串频率统计取得新词候选串,再通过组合成词规则进行筛选获得候选新词,最后通过词的邻接域变化特性去除垃圾串获得新词。利用该方法在COAE 2014评测任务上进行了新词的发现实验,准确率达到36.5%,取得了较好的成绩。
A large number of new words accompanied with the rapid development of microblogging generated these new words with fast propagation and flexible combination with other features of the word, and in the word segmentation is easy to be cut into different strings. This paper proposes a new method of word recognition based on the combination of word frequency characteristics and adjacent changes. Firstly, the method divides the large-scale Weibo corpus, and then combines the adjacent strings in the two stop-phrases with each other to obtain new candidate strings according to the combined string frequency statistics. Then, Screening new candidate words, and finally by neighbors changing the characteristics of the word to remove the rubbish string to obtain new words. The new word discovery experiment was carried out on the COAE 2014 evaluation task using this method, with an accuracy rate of 36.5% and a good result.