In order to improve the performance of DC motor drive system, a passivity-based control (PBC) method for DC motor drive system based on load observer is proposed. The load observer reconstructs the load torque by detecting the armature current and speed of DC motor. Based on the port-controlled Hamiltonian system with dissipation (PCHD) model of DC motor drive system and the observed load, use the interconnection and damping assignment passivity-based control (IDA-PBC) methodology with parameters conversion to track the given speed quickly and stably. The simulation results show the DC motor drive system using the proposed PBC method has good dynamic and static performances and good robustness.