GEM started life at DRI as a more general purpose graphics library known as GSX (Graphics System Extension), written by a team led by Don Heiskell. Lee Lorenzen (at Graphic Software Systems, Inc.) who had recently left Xerox PARC (birthplace of the GUI) wrote much of the code. GSX was essentially a DRI-specific implementation of the GKS graphics standard proposed in the late 1970s. GSX was intended to allow DRI to write graphics programs (charting, etc.) for any of the platforms CP/M-80, CP/M-86 and MS-DOS (NEC APC-III) would run on, a task that would otherwise require considerable effort to port due to the large differences in graphics hardware (and concepts) between the various systems of that era.

GSX consisted of two parts: a selection of routines for common drawing operations, and the device drivers that are responsible for handling the actual output. The former was known as GDOS and the latter as GIOS, a play on the division of CP/M into machine-independent BDOS and machine-specific BIOS. GDOS was a selection of routines that handled the GKS drawing, while GIOS actually used the underlying hardware to produce the output.

GSX evolved into one part of what would later be known as GEM, which was an effort to build a fully GUI system using the earlier GSX work as its basis. Originally known as Crystal as a play on an IBM project called Glass, the name was later changed to GEM.

Under GEM, GSX became GEM VDI (Virtual Device Interface), responsible for basic graphics and drawing. VDI also added the ability to work with multiple fonts and added a selection of raster drawing commands to the formerly vector-only GKS-based drawing commands. VDI also added multiple viewports, a key addition for use with windows.

A new module, GEM AES (Application Environment Services), provided the window management and UI elements, and GEM Desktop used both libraries in combination to provide a GUI. The 8086 version of the entire system was first demoed at the 1984 COMDEX,[4] and shipped as GEM/1 on 28 February 1985.[5]

DRI originally designed GEM for DOS so that it would check for and only run on IBM computers, and not PC compatibles like those from Compaq, as the company hoped to receive license fees from compatible makers. Developers reacted with what BYTE described as "a small explosion"; it reported that at a DRI-hosted seminar in February 1985, more than half of the attendees agreed that GEM's incompatibility with Compaq was a serious limitation. Later that month the company removed the restriction.[9] Applications that supported GEM included Lifetree Software's GEM Write.[10]

DRI responded with the "lawsuit friendly" GEM Desktop 2.0, released in March 1986, with support for VGA displays.[12][unreliable source] It allowed the display of only two fixed windows on the "desktop" (other programs could do what they wished however), changed the trash can icon, and removed the animations for things like opening and closing windows. It was otherwise similar to GEM/1, but also included a number of bug fixes and cosmetic improvements.

In 1988 Stewart Alsop II said that GEM was among several GUIs that "have already been knocked out" of the market by Apple, IBM/Microsoft, and others.[13]

The last commercial release was GEM/3 Desktop, released in November 1988,[14][unreliable source] which had speed improvements and shipped with a number of basic applications. Commercial sales of GEM ended with GEM/3; the source code was subsequently made available to a number of DRI's leading customers.

Another version of GEM called GEM/5, released in 1988,[16][unreliable source] was produced by GST for Timeworks Publisher 2.1. It contained an updated look with 3D buttons. Font scaling on the fly was included. It came complete with all the standard 3.1 tools. This version was produced from GEM 3.13 with only the Bézier handling taken from GEM/4.

GEM Desktop itself was spun off in 1990 as a product known as ViewMAX which was used solely as a file management shell under DR-DOS. In this form the system could not run other GEM programs. This led to a situation where a number of applications (including ViewMAX) could exist all with their own statically linked copy of the GEM system. This scenario was actually rare, as few native GEM programs were published. In 1991 ViewMAX/2 was is released.

In these forms GEM survived until DRI was purchased by Novell and all GEM development was cancelled.

Throughout this time DRI had also been working on making the GEM system capable of multitasking. This started with X/GEM based on GEM/1, but this required use of one of the multitasking CP/M based operating systems. GEM/XM was an updated version of GEM/2 which allowed multitasking and the ability to run DOS programs in shell windows (as Windows does today). None of these saw the light of day, but the GEM/XM source code is now freely available under the terms of GNU General Public License.

Lee Lorenzen left soon after the release of GEM/1, when it became clear that DRI had no strong interest in applications development. He then joined with two other former DRI employees, Don Heiskell and John Meyer, to start Ventura Software. They developed Ventura Publisher (which was later marketed by Xerox and eventually by Corel), which would go on to be a very popular desktop publishing program for some time.

Development of the production 68000 version of GEM began in September 1984, when Atari sent a team called "The Monterey Group" to Digital Research to begin work on porting GEM. Originally the plan was to run GEM on top of CP/M-68K, both ostensibly ported to Motorola 68000 by DRI prior to the ST design being created. In fact, these ports were unusable and would require considerable development. Atari eventually decided to give up on the existing code and port GEMDOS, a DOS-like operating system that DRI had experimented with instead, referring to it as TOS.[17]

As Atari had provided most of the development of the 68k version, they were given full rights to continued developments without needing to reverse-license it back to DRI, who had apparently lost interest in the 68000 platform. As a result, the Apple-DRI lawsuit did not apply to the Atari versions of GEM, and they were allowed to keep a more Mac-like UI.

Over the next seven years, from 1985 to 1992, new versions of TOS were released with each new generation of the ST line. Updates included support for more colors and higher resolutions in the raster-side of the system, but remained generally similar to the original in terms of GKS support. In 1992 Atari released TOS 4, or MultiTOS, along with their final computer system, the Falcon030. In combination with MiNT, TOS 4 allowed full multitasking support in GEM.

On the Atari ST platform, the original DRI sources were ported again to be used in the free and open source TOS clone EmuTOS. New implementations of the AES portions of GEM have been implemented from scratch in the form of XaAES, and MyAES, both of which are fully re-entrant and support multitasking on top of the FreeMiNT multitasking extensions to TOS.

GEM VDI was the core graphics system of the overall GEM engine. It was responsible for "low level" drawing in the form of "draw line from here to here". VDI included a resolution and coordinate independent set of vector drawing instructions which were called from applications through a fairly simple interface. TVDI also included environment information (state, or context), current color, line thickness, output device, etc.

These commands were then examined by GDOS, whose task it was to send the commands to the proper driver for actual rendering. For instance, if a particular GEM VDI environment was connected to the screen, the VDI instructions were then routed to the screen driver for drawing. Simply changing the environment to point to the printer was all that was needed (in theory) to print, dramatically reducing the developer workload (they formerly had to do printing "by hand" in all applications). GDOS was also responsible for loading up the drivers and any requested fonts when GEM was first loaded.

One major advantage VDI provided over the Macintosh was the way multiple devices and contexts were handled. In the Mac such information was stored in memory inside the application. This resulted in serious problems when attempting to make the Mac handle pre-emptive multitasking, as the drawing layer (QuickDraw) needed to have direct memory access into all programs. In GEM VDI however, such information was stored in the device itself, with GDOS creating "virtual devices" for every context – each window for instance.

GEM AES provided the window system, window manager, UI style and other GUI elements (widgets). For performance reasons, many of the GUI widgets were actually drawn using character graphics. Compared to the Macintosh, AES provided a rather spartan look and the system shipped with a single monospaced font.

AES performs its operations by calling the VDI, but in a more general sense the two parts of GEM were often completely separated in applications. Applications typically called AES commands to set up a new window, with the rest of the application using VDI calls to actually draw into that window.

GEM Desktop was an application program that used AES to provide a file manager and launcher, the traditional "desktop" environment that users had come to expect from the Macintosh. Unlike the Macintosh, the GEM Desktop was based on top of DOS (MS-DOS, DOS Plus or DR DOS on the PC, GEMDOS on the Atari), and as a result the actual display was cluttered with computer-like items including path names and wildcards. In general GEM was much more "geeky" than the Mac, but simply running a usable shell on DOS was a huge achievement on its own. Otherwise, GEM has its own advantages over Mac OS such as proportional sliders.

^Jemmett, Ben A. L. (April 1999). "Caldera releases GEM under the GPL". Deltasoft - GEM News. Archived from the original on 2016-11-07. Retrieved 2016-11-07. Caldera Thin Clients, Inc. released the source code for GEM and ViewMAX under the GNU Public License in mid April, following years of speculation over GEM's future. Caldera bought the GEM sources from Novell along with the DR-DOS in 1996, at the time noting that they may develop GEM into a platform for mobile computers and thin clients. However, these plans were dropped, and GEM was instead released into the open-source community.