Multi-party voice-over-IP (MVoIP) services allow a group of people to freely communicate with each other via Internet, which have many important applications such as on-line gaming and tele-conferencing. In this paper, we present a peer-to-peer MVoIP system called peerTalk. Compared to traditional approaches such as server-based mixing, peerTalk achieves better scalability and failure resilience by dynamically distributing stream processing workload among different peers. Particularly, peerTalk decouples the MVoIP service delivery into two phases: mixing phase and distribution phase. The decoupled model allows us to explore the asymmetric property of MVoIP services (e.g., distinct speaking/listening activities, unequal in-bound/out-bound bandwidths) so that the system can better adapt to distinct stream mixing and distribution requirements. To overcome arbitrary peer departures/failures, peerTalk provides light-weight backup schemes to achieve fast failure recovery. We have implemented a prototype of the peerTalk system and evaluated its performance using both large-scale simulation testbed and real Internet environment. Our initial implementation demonstrates the feasibility of our approach and shows promising results: peerTalk can outperform existing approaches such as P2P overlay multicast and coupled distributed processing for providing MVoIP services.