中文分词算法解析-豆柴文库

您所在位置：网站首页 / 中文分词算法解析.docx / 文档详情

在线预览结束，喜欢就下载吧，查找使用更方便

5 金币

下载文档

/ 2

下载提示

如果您无法下载资料，请参考说明：

1、部分资料下载需要金币，请确保您的账户上有足够的金币

2、已购买过的文档，再次下载不重复扣费

3、资料包下载后请先用软件解压，在使用对应软件打开

文本预览

中文分词算法解析
Introduction
Whenprocessingnaturallanguagetext,oneofthefundamentaltasksistosegmentthetextintomeaningfulunits,suchaswordsorphrases.Thisprocessisknownastextsegmentationortokenization.InChinese,textsegmentation,alsocalledChinesewordsegmentation,isparticularlyimportantbecausetherearenospacesbetweenChinesewords.Therefore,segmentingaChinesesentenceintoitsconstituentwordsisessentialformanynaturallanguageprocessingtasks,suchasmachinetranslation,informationretrieval,andtextclassification,etc.ThispaperwilldiscussvariousChinesewordsegmentationalgorithms.
BasicPrinciplesofChineseWordSegmentation
ThemainprincipleofChinesewordsegmentationistoidentifytheboundariesbetweenwordsinasentence.SincetherearenospacesbetweenChinesewords,thisprocessismorechallengingthaninotherlanguages.ThetwomainapproachestoChinesewordsegmentationarerule-basedmethodsandstatisticalmethods.
Rule-basedmethodsrelyondictionariescomposedofwordsandtheircorrespondingPOStagstosegmentasentence.Themainideabehindrule-basedmethodsistomatchthewordsinthesentencewiththoseinthedictionary.Theapproachidentifiesthelongestmatchingsequencewhiletakingintoaccountthewordfrequencyandthecontext.Whenthesegmenterdoesnotfindafullmatchingword,itcanuserulestosplitthecharactersintosmallerunits.
Statisticalmethods,ontheotherhand,usemachinelearningalgorithmstosegmentChinesetext.Themethodreliesontrainingastatisticalmodelonalargecorpusoftext,labeledwithwordboundaries.Themodelcanthenapplylearnedrulestosegmentnewandunseentext.Statisticalmethodsaregenerallymoreaccurateandrequirelessmanualannotationthanrule-basedmethods.
ChineseWordSegmentationAlgorithms
1.MaximumMatchAlgorithm(MMA)
Amongtherule-basedmethods,themaximummatchalgorithmisamongthesimplestandmostwidelyused.Itstartsbysettingthelargestwordlengthtobesegmentedinagivensentence.Thealgorithmthenlooksupthecorrespondingwordswiththatlengthinthedictionaryandremovesthewordsfoundfromthesentence.Thisprocessisrepeatedfortheremainingtextuntilallwordsaresegmented.
Forinstance,considerthesentence