基于中文维基的大规模命名实体识别语料自动生成方法(英文).docx

基于中文维基的大规模命名实体识别语料自动生成方法(英文)
Introduction
NamedEntityRecognition(NER)isanimportantnaturallanguageprocessingtaskthatinvolvesidentifyingandclassifyingentitiesinatextintopredefinedcategoriessuchaspeople,locations,organizations,etc.NERiswidelyusedinvariousnaturallanguageprocessingapplicationssuchasinformationextraction,textclassification,entitylinking,questionanswering,etc.
OneofthebiggestchallengesinNERistheavailabilityofhigh-qualitylabeledtrainingdata.However,manuallylabelinglarge-scaledataistime-consuming,expensive,andoftendifficulttoacquire.Therefore,therehasbeenagrowinginterestindevelopingmethodsforgeneratinglabeleddataautomatically.Inthispaper,wepresentamethodforgeneratingNERtrainingdatabasedontheChineseWikipedia.
Background
ChineseWikipediaisoneofthelargestonlineencyclopediasintheworld,withover1.1millionarticles.Itcoversvarioustopicsrangingfromhistory,culture,science,technology,etc.Thearticlesarestructuredandwell-organized,makingthemanidealsourcefortrainingdata.
However,theChineseWikipediadoesnotprovideannotationsfornamedentities,whichmakesitchallengingtouseitasalabeleddatasetforNER.Therefore,weneedtodevelopamethodtoautomaticallygeneratelabeleddatafromtheChineseWikipedia.
Methodology
Ourmethodconsistsofthreemainsteps:datacollection,entityrecognition,anddatalabeling.
DataCollection:WedownloadedasubsetoftheChineseWikipediaconsistingof500,000articles.WechosearticlesthatarerelevantforNER,suchashistory,geography,culture,science,andtechnology.
EntityRecognition:Weusedapre-trainedChineseNERmodeltotagentitiesinthetext.Themodelisbasedonabidirectionallongshort-termmemorynetwork(BiLSTM)withaconditionalrandomfield(CRF)lossfunction.Ithasbeentrainedonalarge-scaledatasetconsistingofnewsarticles,onlinereviews,andsocialmediadata.
DataLabeling:Aftertaggingtheentitiesinthetext,weassignedalabeltoeachentitybasedonitstype(person,location,organization,etc.).Weusedasetofpredefinedrulestoassignthelabels.Forexample,iftheentityhasageographicnameorisrelatedtoaplace,welabeleditasalocation.
R