In this paper, the operation principles and basic properties of the axial permanent magnet hysteresis damper (APHD) are introduced. First, the double-side APHD (DSAPHD) is taken as an example to analyze the magnetic flux distribution in the stators and the rotor. According to the magnetic flux distribution in the rotor, a finite element (FE) model is established to calculate the minimum and maximum torque of DSAPHD, which has a faster calculation speed. A hybrid method of particle swarm optimization (PSO) and Hook-Jeeves direct search algorithm is used to design and optimize the DSAPHD. This reduces the influence of design experience, resulting in optimal design parameters. The optimized result has a smaller minimum torque while meeting the required maximum torque. Compared with the calculation results of the vector hysteresis model, the simplified FE model has higher calculation accuracy, which can be used in the design of APHD.