曲朝阳,张艺竞,王永文,赵莹.基于spark框架的能源互联网电力能源大数据清洗模型[J].电测与仪表,2018,55(2):39-44. quzhaoyang,张艺竞,wangyongwen,zhaoying.Energy data cleaning model for Energy Internet based on Spark framework[J].Electrical Measurement & Instrumentation,2018,55(2):39-44.
基于spark框架的能源互联网电力能源大数据清洗模型
Energy data cleaning model for Energy Internet based on Spark framework
Big energy data cleaning can improve the quality of energy large data accuracy, completeness, consistency, and reliability. Big Data for energy extraction cleaning process difficult unified anomaly detection mode, continuous and abnormal data correction accuracy is low and other issues, we proposed a framework based Spark Energy clean energy large data model. First, based on improved CURE clustering algorithm to obtain normal cluster; secondly, to achieve a normal cluster boundary sample acquisition method, and designed based anomaly recognition algorithm boundary samples; finally weighted moving average realized the abnormal data corrected by the index. By a wind farm wind power generation monitoring data analysis of experimental data cleansing, verification of the cleaning efficiency of the model accuracy.
