Pawel's article is a first look at the subject of undocumented NetWare. The second edition of Undocumented DOS, which I am just finishing now (and which should be in bookstores in November or December 1993), will have a new section on NetWare, including details on the NETX shell and the F2 interface. Even better, Tim Farley is actively working on a book, Undocumented NetWare, to be published in early 1994 as part of a series I edit for Addison-Wesley. Tim contributed major additions to Pavel's article. Interestingly, the idea for this book first came from an engineer at Microsoft. With Microsoft's accusations against Novell, and Novell's accusations against Microsoft, the safest thing is probably to assume that they're both telling the truth. Our friend at Microsoft says of NCP, "who knows how many inventions would be enabled by publishing this."

To really use any of the information Pawel provides here, you need to download his NCPTEST source code (see "Availability," page 3).

Right now it looks like future "Undocumented Corners" will look at low-level aspects of Windows, such as undocumented DPMI, the Virtual Machine Control Block structure, Instance Data lists, and so on. Another possibility is the fascinating topic of undocumented NT, though I find it difficult to be enthusiastic about an operating system that requires 16 Mbytes of memory. Please send your comments and suggestions to me on CompuServe at 76320,302 (that's 76320.302@compuserve.com from the Internet).

The NetWare Core Protocol (NCP) is a packet format that client workstations on the network use to communicate with a NetWare server. NCP provides services such as file manipulation, message sending, transaction processing, printing, and the like.

In a DOS-based workstation, the NetWare shell (NETX.EXE, NET3.EXE, VLM. EXE, and so on) is responsible for establishing and maintaining an NCP session with a file server. The NetWare shell also provides a DOS-like interface for applications to use; higher-level APIs, like the NetWare C Interface, use this low-level interface.

Requests originating from applications, regardless of the API used, are eventually converted by the NetWare shell to NCP packets and sent to the server. Responses from the server are converted back to the format of whatever API the application is using. Some of the functions specified in the different interfaces may end up sending several NCP packets. There are also several calls that access local data within the workstation shell and consequently never generate any requests to the server. But, in general, almost everything in NetWare rests upon NCP.

NCP isn't just for communication between users and file servers. Print servers also rely on NCP, as can any NetWare object type. Regardless of the physical implementation of the print server (VAP, NLM, EXE, or a standalone device), at some point it has to use NCP to access the NetWare queue management system; that is, check for data present, read queue data, and so on. It can then distribute the print data to the printers (in case they are not directly connected to the print server itself) using any protocol it likes. Novell's own PSERVER is a good example: It uses NCP to log in to the server and access queues, and uses a separate RPRINTER/

NPRINTER protocol to communicate with remote printers.

In the OSI model, NCP covers the transport and session layers, and is conceptually similar to the SMB protocol in Microsoft's LAN Manager. You can think of NCP as an operating-system interface or collection of remote procedure calls. The difference, compared to a traditional operating-system interface, is that here, the OS itself resides on a different physical machine--the NetWare server. NCP is a client/server or distributed interface.

NCP uses Novell's Internetwork Packet Exchange (IPX) as the underlying network-level protocol. IPX is a connectionless (datagram) transport service that does not guarantee packet delivery. The session control and packet error checking are performed by NCP.

Now, you may ask: Why should I care about the undocumented NetWare Core Protocol? Isn't programming using published and documented Novell APIs, like the NetWare C Interface, the recommended way to develop software in this environment?

It certainly is, as this makes your programs portable to different client platforms and frees you from having to write your own NCP requester. Nevertheless, there are several reasons why NetWare developers have long wanted information on NCP.

Understanding what happens between the NetWare shell and a file server may help you to more quickly locate problems, especially in complex network environments involving several pieces of software. Once you know what NCP traffic is produced by the standard NetWare API calls, you may find other and more efficient ways to accomplish the same tasks. Also, some software, such as programs that boot the workstation from the file server, can't be implemented without knowing how NCP works.

If you want to understand NCP, you'll need some kind of network traffic analyzer to see what happens on the wire. LAN analyzers such as The Snooper, The Sniffer, or NetWare's LANalyzer are essential tools for developing software in networked environments.

This article explains the basic principles behind NCP. Recent additions to NCP like packet signing, Burst Mode, and Large Internet Packets aren't covered. NCPTEST.C, a sample program that shows how to access services of a NetWare file server is available electronically (see "Availability," page 3). NCPTEST requires that IPX be loaded, but does not require Novell's NetWare shell. The NetWare shell typically occupies 30--50 Kbytes, so in some specialized applications NCP programming might even be a good low-memory alternative to using the shell. If you compile NCPTEST using Borland C++ 3.1, use command line: bcc -ml -a -K ncptest.c ipxint.c ncpint.c. (The appropriate memory model is very important.) Since I use Borland's screen I/O library, NCPTEST isn't directly generic and you'll have to make changes before compiling with another compiler. NCPTEST requires a NetWare 2.1x or higher server.

NCPTEST finds the nearest NetWare server, and sets up three separate connections to it, all under the same user ID and password. This is normally impossible under the regular DOS client software for NetWare, so it's a good example of something you can do by going directly down to NCP and ignoring the shell. The program also displays the state as it builds up the connection, then sits in an "idle" state until you press any key to kill each connection.

Before you kill the program, go to another station that's logged into the same server, and execute the Novell command USERLIST /A. This shows all the users logged in, and their addresses. You'll see three separate connections being taken up by the user specified on the NCPTEST command line, but all of them have the same net address. This is normally impossible in NetWare--you can log in more than once, but not from the same machine.

NCP is a half-duplex, request/re-sponse protocol. The client (generally NETX.EXE or some other NetWare workstation shell) is the active side: It initiates the communication process by sending a request. The server (generally the NetWare file server) processes the request and responds with a reply. The client then processes the reply and possibly issues a new request. For example, the client may issue a request to open a file; the server would then send a response indicating whether the file could be opened. The client can have only one request (packet) outstanding at any time. The client must wait for a response before sending the next request (this is not true for Burst Mode NCP).

Since NCP uses an unreliable network protocol for delivery, it must ensure that the packets are delivered in sequence and without duplication. The client accomplishes this by assigning a sequence number to each request. The server is expected to respond with a packet containing the same sequence number. If it does, the client increments the sequence number by one and transmits the next request. Responses having incorrect sequence numbers are discarded by the client. It is legal for a client to resend a request to which the server already replied, that is to resubmit the latest request. In this case, the server assumes that the reply got lost on its way to the client and reexecutes the request.

If the server does not respond at all to a request, the client resubmits the same request a number of times before assuming that the server is unreachable. This is necessary because of the unreliable underlying transport protocol and media. Because of a busy network, packets may never get sent. They may get dropped by a router, malformed on their way to the destination, and so on. It's up to the client to choose the number of, and time between, retries. For instance, the DOS client, NETX, has a number of parameters in its INI file (NET.CFG) which control the client's NCP timeout behavior, including NCP SET TIMEOUT= and NCP TIMEOUT MULTIPLIER=. Then, if the server can't immediately respond because it is busy, it will send a special packet (called "Positive Acknowledge") to the client. This indicates that the request is being processed; the client should continue waiting for the reply. Such a packet has the same sequence number as the request that caused it. The server will first send when it receives a duplicate request (because the client times out) and the server still processes that request. (Novell calls this "Positive Acknowledge" packet "Request Being Processed," at least in LANalyzer traces.)

The server also needs to know if a client is still alive. If the client stops sending requests without properly terminating the connection, the server must know whether the person operating the workstation temporarily stopped using it or if the station has been turned off. Since each workstation occupies at least one user connection at the server, there's a possibility that a number of workstations could unnecessarily take up connections that could be made available for others to use. After a period of workstation inactivity, the server sends a watchdog packet to the client, asking if the client is still active. If the client doesn't respond, the server will send a number of watchdog packets, at regular intervals. Should none of the watchdog packets produce a reply from the client, the server will break the corresponding connection and make it available for other workstations.

Except for the information exchanged on the watchdog socket, NCP uses no special system packets devoted to keeping the session alive, confirming successful arrival of data packets, and the like. This makes NCP an efficient protocol. However, the request/response nature of the protocol makes it slow on WAN links where the time between a request and a response can be considerable. To remedy this, Novell has introduced Burst Mode NCP which makes NCP a windowed protocol, able to send several packets in one direction before requiring a reply. Burst Mode is currently used only for reading and writing files.

Regardless of its type, an NCP packet occupies the data portion of an IPX packet. All NCP packets, except for the Burst Mode NCP, have a structure like that in Tables 1 and 2.

A request packet starts with a 16-bit Packet Type field, which indicates the request type. Request types are listed in Table 3. Of the six request types shown, the three most important are type 0x1111 (Allocate Slot Request), used when creating a connection; type 0x2222 (Request), used for ordinary NCP requests; and type 0x5555 (Deallocate Slot Request), used when terminating a connection. (Novell LAN alyzer's terminology for 0x1111 and 0x5555 is "Create Service Connection" and "Destroy Service Connection."

The client sets and increases the Packet Sequence Number field , as explained earlier.

The Client Task Number in Table 1 identifies the client task that issued the request. Each connection may have several tasks (programs) executing on the same connection. The server tracks resources like semaphores, file locks, and so on, for each of the client's tasks. When the client issues an End-of-Job command, the server releases resources belonging to that task.

Together, the two Server Connection Number fields constitute a 16-bit connection number, assigned by the server when creating a connection, and later exchanged between the client and the server in all requests and replies. Until recently, the connection number high byte was always 0 in all commercially-available versions of NetWare, none of which allow more than 250 users to be on the server at once and therefore never need a connection number over 250. However, Novell has been for years selling to special customers a 1000-user version of NetWare 3.11. Unfortunately, Novell didn't publicly document how the 1000-user API differs from the older APIs which use a single byte for a connection number. This is being resolved this year, with the retail version of NetWare 4.0, which supports 500 and 1000 users.

The Function Code in Table 1 indicates which NCP function is to be executed by this request (see Table 4, page 128). This field may be followed either directly by the Request Data field, or by the subfunction length and code.

The Subfunction Length, if present, indicates the length of the following data, excluding this field. If this field is present, the Subfunction Code field is also present. Some commands, such as the Transaction Tracking System (TTS) and semaphore handling functions, lack the length field. In this case, the subfunction code directly follows the function code.

The Request Data field in Table 1 carries zero or more bytes of function/subfunction dependent data.

A reply packet also starts with a 16-bit Packet Type field (see Table 2). The allowable values are 0x1111 (Reply), used for the normal NCP replies, and 0x9999 (Positive Acknowledge), used to indicate that a server is busy processing a request.

The reply Sequence Number carries the sequence number of the corresponding request.

The reply Completion Code field (Table 2) indicates whether or not the request executed successfully. A 0 indicates success; any other value usually indicates an error. Novell's NetWare System Calls for DOS lists error codes. The same completion code can take on different meanings, depending on which function was executed.

The Connection Status field indicates the status of the connection itself. The lower four bits should be interpreted as a value between 0--15: 0 means that the connection is alive; any other value indicates that a connection is in error and is no longer valid. Bit 4 set indicates that the server is down. Bit 6 set means the workstation has a message waiting at the server.

The NCP client is responsible for creating a connection with the server. A workstation may have several simultaneous NCP connections open to the same server or to different servers. Each NCP connection is equivalent to a user connection at the file server. A full connection address consists of the 4-byte network number, 6-byte physical node number, and 2-byte IPX socket number. A client needs to open three IPX sockets for every NCP connection it wishes to maintain. New additions to NCP, like the Burst Mode and Large Internet Packets, require additional sockets.

The client's socket numbers must be consecutive: If the first socket is 0x4003, the second and third should be 0x4004 and 0x4005. Each socket can be considered a communication channel, independent of other sockets, on which conversation with the server takes place.

The first socket is used to exchange NCP requests and replies. The destination socket at the file server is always 0x0451. NCP's IPX packet type is 0x11. The second socket is a watchdog socket, used by the file server to determine whether the client is alive. The third socket is a message socket that receives message notifications from the server.

Before the client can create a connection to a server, it must know the server address; that is, its network, node and socket numbers. Given a server name, one can use the Service Advertising Protocol (SAP) to obtain the server address. For an example, see the SAPGetNearestServer() function in NCPINT.C (available electronically).

Once the client knows a server address, it uses an NCP Allocate Slot Request packet to establish the connection. The client must set the packet type to 0x1111. Also, it should fill the connection bytes with 0xFF to indicate that the client does not yet have a connection. The sequence number and function number should be set to 0. The task number may be assigned any value.

The server will respond with a packet type 0x3333. The client should examine the completion code and connection status for possible errors before proceeding. For example, the server may be out of user connections. Provided that the request is successful, the connection bytes in the reply packet will contain the file server connection number the client has been assigned. This number, which does not change during a lifetime of a connection, must be supplied in all subsequent requests to the server until the connection is terminated. The function NCPAttachToFileServerWithAddress() in NCPINT.C demonstrates the connection process.

After creating the connection, you should also negotiate the maximum NCP packet size that can be used. The two sides inform each other of their respective maximum receive buffer sizes. The smaller of the two will be used by both the client and the server. The value returned by a NetWare server is 1024 if there are no routers between the server and the client, otherwise it is 512.

The procedure described above is basically what happens behind the scenes whenever a NetWare shell loads, or a program issues the NetWare AttachToFileServer call. At this point, if everything worked as expected, the client has created a connection at the server. The client still needs to log in, however, because by itself having a connection provides only a very limited access to the server resources. You can use the NetWare FCONSOLE utility to verify that a connection has been created. On a 3.x server you'll see a string "NOT-LOGGED-IN" as the user name; this indicates that a connection exists, but no one has yet logged in. No indication is given on 2.x servers.

Any NetWare object can log in to a file server. The most common object is the user object (0x0001), but other types of objects (such as job and database servers) also frequently log in.

There are two ways to log in to a server: using unencrypted or encrypted passwords. Unencrypted passwords are visible on the network and present a security threat. Encrypted passwords, supported by NetWare 2.15c and higher, are encrypted at the workstation and not visible as plain text on the network.

Briefly, encryption of the passwords works as follows: The client requests an encryption key from the server (NCP request 0x17 0x17; see Table 4). It then asks the user for the plain password and encrypts it using the key and the object's 4-byte ID number obtained using the GetBinderyObjectID request (0x17 0x35). The encrypted password is then sent to the file server for verification. For each login attempt, the client needs to request a new (different) encryption key from the server.

NCPTEST.C and NCPINT.C (available electronically, see page 3) use encrypted passwords. That part of the source code is a direct translation of a Pascal program written by Barry Nance. (See "Automatic NetWare Log-ins: How to Log into NetWare from Your Programs without User Interaction," Byte, March 1993. The program is available as ELOGON.ZIP in the BPASCAL forum on CompuServe, and through other sources. According to Nance's article, he got the algorithm from Terje Mathesen's NETWARE.PAS code on BIX.)

Once logged in to a server, the client has access to its resources as specified by the client's privilege level. To use these resources, you need to know the format of the NCP functions you plan to use. Table 4 lists over 300 NCP function and subfunction codes. It isn't a complete reference, but rather a bare listing of the most used calls. To use any of these calls, you'll need to know the call's specific packet format. Some sample formats are provided in NCPINT.C (available electronically).

Often, the exact format of a request and reply can be inferred from Novell's NetWare System Calls for DOS. The request buffers described in that document are to be appended directly after the NCP request header, starting either at the Subfunction Length or at the Request Data field, depending on the type of function. The reply buffers, without the reply length field, normally start at the Reply Data field in the NCP reply packet.

Also, a careful study of the source code and header files provided with the NetWare C Interface helps to understand NCP packet formats. Some NCP functions, however, are not directly accessible through standard APIs and consequently are not described in easily available documents. One way to learn about the format of such packets it to use a protocol analyzer capable of decoding NCP. Another, if you can afford it, is to license NCP documentation directly from Novell.

When the client no longer needs a connection, it should free it, thus making its corresponding server connection available for others to use. To accomplish that, the client sends a NCP Deallocate Slot Request to the server. (According to LANalyzer, "NCP Deallocate Slot" is "Destroy Service Connection.") Such a request looks like an ordinary NCP request, except for the Packet Type field, which should be set to 0x5555 (see Table 3). The Function Code field should be set to 0; no subfunction code is present. Upon successful execution of this request, the connection is invalid.

The format of the watchdog request and reply, mentioned earlier, differs from that of normal NCP packets. The file server socket from which these packets originate is dynamic, often 0x4001. The IPX packet type varies, although it usually is 0x11 or 0x00. The NetWare 3.1x file servers have three setable parameters for watchdog packet configuration: "Delay Before First Watchdog Packet," "Number Of Watchdog Packets," and "Delay Between Watchdog Packets."

When a server has a message for a client, it informs the client of the message's presence by sending a packet to client's message notification socket. The client is expected to issue a NCP Get Broadcast Message request (0x15 0x01) to read the message itself, which stops the server from sending more notifications. The client does not have to read the message immediately; the server will continue sending notification packets approximately once a second. Also, all replies originating from the server will set bit 6 in the Connection Status until the client reads the message. The file server socket from which these packets originate is dynamic, often 0x4001. The format of the watchdog and message packets can be found in NCPTEST source code.

Using NCP could enable a whole range of innovative NetWare programming. Some possibilities include building your own simple NetWare protocol analyzer, writing an asynchronous version of the NetWare C interface, connecting client hardware to NetWare, writing replacement workstation shells, or perhaps (like Ornetix Technologies' SerView) even writing your own NCP server, only implementing a subset of the NCP functionality.

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Video

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!