5 Answers
5

Applications communicate over some sort of private API to the WindowServer process, the WindowServer process is the one that actually gets hardware events (mouse, keyboard) and dispatches those to the client applications. (this is still an open question: what protocol do they use if any, do they use Mach ports and MIG, or some Socket-based API, not sure).

The WindowServer is the Quartz Compositor. Typically applications use the Quartz2D API which exposed in the CoreGraphics API (CGXXX funtions). Applications create CoreGraphics "Contexts" (CGContext) and draw there. Whether the context is pushed when it is done as big bitmap, or if the operations are sent to the server like they are on X11 is still an open question.

There is a limited API exposed to control certain aspects of the WindowServer process, the sort of configuration settings that are typically done from the Settings application, but there is no documentation on how apps actually communicate graphic requests or pump messages from the server, other than the Carbon/Cocoa APIs exposed.

XQuartz is the open-source project of the X.org X Window System that runs on Mac OS X. Their developer wiki and mailing list should help you with your questions, seems like you need someone deeply involved in the project to answer those.

Right, but that only covers XQuartz, which is just a program that talks to their real windowing system. The question was about how do apps like XQuarts or Finder talk to this windowing system
–
miguel.de.icazaAug 17 '10 at 2:34

(sorry, can't post notes yet): Quartz or Core Graphics is the drawing and windowing system in OS X (replacing QuickDraw). Core Graphics is one part of Cocoa (Cocoa as a whole being more comparable to the entire Win32 API, rather than GDI, Direct2D or X)
–
KellyClowersAug 21 '10 at 21:57