Process-to-process copy in Linux

More exciting news on the Linux kernel front (thanks for the heads-up, Brice!): our friends at Big Blue have contributed a patch and started good conversation on the LKML mailing list about process-to-process copying. We still don’t have a good solution for being notified when registered memory is freed (my last post on this topic mentioned that the ummunotify patch had hit the -mm tree, but that eventually didn’t make it up to Linus’ tree), but hey — this is progress, too (albeit in a slightly different direction), so I’ll take it!

“Why do I care?” you say.

I’m glad you asked. Let me explain…

Most MPI implementations use some flavor of shared memory to delivery messages between two processes on the same node. That is, the source process copies the message into a shared memory block (that was previously setup, perhaps during MPI_INIT) and the destination process copies the message out of the shared memory block.

Most schemes like this are pipeline the source and destination process copies so that the receiver doesn’t have to wait for the entire message to arrive in shared memory before it can be copied out. The performance is actually fairly reasonable.

That being said, a direct process-to-process copy has two benefits:

The performance can be greater (see Chris Yeoh’s post to the LKML, linked at the beginning of this entry)

The fact that only 1 memcpy is occurring means less traffic on the internal networks/buses, which is significantly friendlier to other processes running on the same system (vs. doing 2 copies)

I’ll be following this thread; I hope to see Chris’ work make it into the main line kernel!

Some of the individuals posting to this site, including the moderators, work for Cisco Systems. Opinions expressed here and in any corresponding comments are the personal opinions of the original authors, not of Cisco. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Cisco or any other party. This site is available to the public. No information you consider confidential should be posted to this site. By posting you agree to be solely responsible for the content of all information you contribute, link to, or otherwise upload to the Website and release Cisco from any liability related to your use of the Website. You also grant to Cisco a worldwide, perpetual, irrevocable, royalty-free and fully-paid, transferable (including rights to sublicense) right to exercise all copyright, publicity, and moral rights with respect to any original content you provide. The comments are moderated. Comments will appear as soon as they are approved by the moderator.