The reliability evaluation of embedded software of intelligent electricity meters is very important for power enterprises to realize supplier management. In order to realize the comparative analysis of embedded software reliability of suppliers, a consistency analysis method for reliability test results of embedded software from different suppliers based on attribute consistency theory is proposed. Firstly, the embedded software reliability test system and test cases were built based on the black box test method, and then Cohen’s Kappa was designed to calculate and evaluate the consistency of reliability test results. The reliability test results of 24 test items of 6 suppliers'' electricity meter software and 4 types of electricity meter software from the same supplier were analyzed, the highest Kappa value was obtained between the 3rd supplier and the 6th supplier, and the Cohen’s Kappa values was 0.625. In the 3rd Supplier''s list of 4 categories, the highest Kappa value was obtained between the 2nd category and the 3rd category, and the Cohen’s Kappa values was 0.833. The results show that this method can reduce the influence of random factors on the consistency evaluation of reliability test results. The method can also help power enterprises to realize the reliability management of suppliers and realize the identification of suppliers with abnormal reliability or alternative suppliers with good reliability consistency with quality suppliers.