第二章数据预处理2013年9月18日第二章数据预处理1.WhyDataPreprocessing2.DataIntegration3.MissingValue4.NoisyData5.DataReduction2.1WhyDataPreprocessing?•高质量决策来自于高质量数据（Qualitydecisionscomefromqualitydata）.•数据质量的含义•正确性（Correctness）•一致性（Consistency）•完整性（Completeness）•可靠性（Reliability）2.1WhyDataPreprocessing?•数据错误的不可避免性•数据输入和获得过程数据错误的不可避免性•数据集成所表现出来的错误•数据传输过程所引入的错误•据统计有错误的数据占总数据的5%左右•其他类型的数据质量问题:•Dataneedstobeintegratedfromdifferentsources•Missingvalues•Noisyandinconsistentvalues•Dataisnotattherightlevelofaggregation数据质量问题的分类数据质量问题单数据源问题多数据源问题模式相关实例相关粗劣的模式设计）——拼写错误和模式设计）的数据）——唯一值——冗余/重复——命名冲突——不一致的聚集层次——参考完整性——矛盾的数据——结构冲突——不一致的时间点…………模式相关（缺乏完整性约束，（数据输入错误）（不同的数据模型实例相关（矛盾的或不一致包含数据清理过程的三个主要领域•数据仓库（datawarehousing）•数据库中的知识发现(kdd)•总体数据质量管理(totaldataqualitymanagement,TDQM)2.2DataIntegration•Integratedatafrommultiplesourcesintoacommonformatfordatamining.•Note:Agooddatawarehousehasalreadytakencareofthisstep.DataWarehouseServerOLTPDBMSsOtherDataSourcesExtract,clean,transform,aggregate,load,updateDataMartsDataIntegration(Contd.)Problem:Heterogeneousschemaintegration•Differentattributenames•Differentunits:Salesin$,salesinYen,salesinDMcidnamebyear1Jones19602Smith19743Smith1950Customer-IDstate1NY2CA3NYDataIntegration(Contd.)Problem:Heterogeneousschemaintegration•Differentscales:Salesindollarsversussalesinpennies•Derivedattributes:AnnualsalaryversusmonthlysalarycidmonthlySalary150002240033000cidSalary650,0007100,000840,000DataIntegration(Contd.)Problem:Inconsistencyduetoredundancy•Customerwithcustomer-id150hasthreechildreninrelation1andfourchildreninrelation2•Computationofannualsalaryfrommonthlysalaryinrelation1doesnotmatch“annual-salary”attributeinrelation2cidnumChildren13cidnumChildren14cidmonthlySalary1500026000cidSalary160,000280,0002.3MissingValues常常会有一些记录的某些属性值不知道的情况出现.出现缺失值的原因:•Attributedoesnotapply(e.g.,maiden娘家姓name)•Inconsistencywithotherrecordeddata•Equipmentmalfunction•Humanerrors•Attributeintroducedrecently(e.g.,emailaddress)MissingValues:Approaches•Ignoretherecord•Completethemissingvalue:•Manualcompletion:Tediousandlikelytobeinfeasible•Fillinaglobalconstant,e.g.,“NULL”,“unknown”•Usetheattributemean•Constructadataminingmodelthatpredictsthemissingvalue2.3NoisyData•Examples:•Faultydatacollectioninstruments•Dataentryproblems,misspellings•Datatransmissionproblems•Technologylimitation•Inconsistencyinnamingconventions（命名习惯）•DuplicaterecordswithdifferentvaluesforacommonfieldNoisyData:RemoveOutliersNoisyData:SmoothingxX1yY1Y1’NoisyData:Normalization•Scaledatatofallwithinasmall,specifiedrange•Leaveoutextremeorderstatistics•Min-maxnormalization•Z-scorenormalization•Normalizationbydecimalscaling2.4DataReductionProblem:•Datamightnotbeattherightscaleforanalysis.Example:Individualphonecallsversusmonthlyphonecallusage•Complexdataminingtasksmightrunaverylongtime.Example:Multi-terabytedatawarehousesOnediskdrive:About20MB/sDataReduction:AttributeSelection•Selectthe“relevant”attributes...
