Chondritic meteorites primarily contain millimeter-sized spherical objects, chondrules; however, the co-accretion process of chondrules and matrix grains is not yet understood. In this study, we investigate the ejection process of chondrules via collisions of fluffy aggregates composed of chondrules and matrices. We reveal that fluffy aggregates cannot grow into planetesimals without losing chondrules if we assume that the chondrite parent bodies are formed via direct aggregation of similar-sized aggregates. Therefore, an examination of other growth pathways is necessary to explain the formation of rocky planetesimals in our solar system.