In order to study the interactive effects between the semi-rigid frame and the infill plates, three-story single bay specimens of semi-rigid composite steel frame with steel plate shear wall were tested under low cyclic loading. Structure seismic performance and failure mode are obtained. The paper explored the framework of beams, columns, semi-rigid nodes in the local mechanical properties, analyzed the load transfer mechanism and the components failure sequence. The results showed that the specimen have great plastic deformation capacity and good energy dissipation capacity. The frame and the steel plate shear wall work well, the specimen have high safety margin. The requirement of ductility was lowered by arranging infill plates. More than 95% of the overturning moment were undertake by frame. Infill plates as the main component to resist lateral force can bear about 75% of the shear. The whole specimen of the test in-plane bending, but the out-plane bending deformation of the steel columns are large. Through finite element analysis, the influence of connection stiffness on load carrying capacity is related to the stiffness of column and thickness of the infill plate. This influence is enlarged when increasing the stiffness of the column and reducing the thickness of the infill plate.For the specimens with a flexibility coefficient of column of 2.5, the load carrying capacity was enhanced by about 10% when changing pinned connections to rigid connections.