王林童,赵 腾,张 焰,苏 运.基于Hadoop的风力发电监测大数据存储优化及并行查询方法[J].电测与仪表,2018,55(11):01-06. Wang Lintong,Zhao Teng,Zhang Yan,Su Yun.Storage optimization and parallel query method for big data of wind power monitoring based on Hadoop[J].Electrical Measurement & Instrumentation,2018,55(11):01-06.
基于Hadoop的风力发电监测大数据存储优化及并行查询方法
Storage optimization and parallel query method for big data of wind power monitoring based on Hadoop
With the extensive development of wind power generation and the generalized application of intelligent monitoring technology, wind power monitoring data shows the big data characteristics of large volume, multi types and fast growth. In order to solve the two major problems with big data efficient storage and quick query, in this paper, the optimization method of big data storage is studied based on Hadoop platform. A Hash bucket algorithm considering wind power monitoring data correlation is proposed. The algorithm realizes the centralized storage of related data, so as to enhance the efficiency of data query and processing. On the basis of data storage optimization, the parallel association query for multi-source big data of wind power monitoring based on MapReduce is realized. Tests on a Hadoop platform show that, after optimization of hash bucket storage, the time of the multi-source data parallel association query is significantly shortened than traditional Hadoop method.
