1 Introduction

frpc

This is an implementation of the ONC-RPC ("SunRPC") protocol in Common Lisp. It provides both a generalized
eXtensible Data Representation (XDR) serializer and a flexible Remote Procedure Call (RPC) framework to build robust, secure networked
services. It supports the most commonly used authentication flavours (see below) including RPCSEC_GSS (i.e. Kerberos).

See the related project nefarious, which uses frpc to implement an NFSv3 client and server.

An XDR protocol compiler is also included, which provides similar functionality to rpcgen typically used with the C programming language (see section 9).

1. Defining RPC interfaces

RPC interfaces are given a unique integer called a program number, see
IANA
for a detailed list. Each program may have multiple versions of its interface, with each version
having a different set of functions/arguments. Each procedure in the interface is also given a
unique integer. Together these 3 integers define the procedure identifier.

In frpc, both clients and servers must define the interface. This supplies argument and result types.
Servers must additionally implement handlers for each procedure they wish to support.

The :HANDLER option specifies a function which accepts a single argument, which will be the value decoded according
to the rule defined for the arg-type by the DEFRPC form. The handler function should return a single value which will be
passed to the result-type serializer defined by the DEFRPC form. If the handler signals an error,
the RPC server will be silent (i.e. not send a reply). Typically, you should indicate to the caller that an error occured
by returning a status code as a part of your return value. Most RPC interfaces define return value structures in this way.

The types provided to DEFRPC can be a generalized type specifier, as described below in section 4.5.

2. Client

The DEFRPC macro defines a wrapper around the underlying CALL-RPC function, with default values provided
for the argument writer, result reader, program, version and proc arguments.

Thus, with the example above, the client will be able to call a remote RPC server using, e.g.,

(call-hello "hello" :host "10.1.1.1" :port 8000)

The default client function accepts a single mandatory argument, which must match the corresponding XDR type specifier.
However, typically the programmer would like to provide a better interface to hide the underlying implementation.
For instance, consider a procedure which accepts and returns a structure of two strings:

The :ARG-TRANSFORMER option specifies how to transform the arguments to the function into arguments for
the RPC call. You may use this to allow some of the arguments be passed in as keyword parameters.
The :TRANSFORMER option specifies how to transform the result of the RPC call back to Lisp.
Documentation strings can be provided with the :DOCUMENTATION option.

2.1 CALL-RPC

The low-level client functionality is provided by CALL-RPC. This function is used by a client to
send an RPC request to a remote server and blocks until a reply is received.

2.2 TCP connections

You can provide a connection to the functions defined by DEFRPC. This makes it more efficient to send multiple
messages to a single server, without having to reconnect for each request.

Use RPC-CONNECT and RPC-CLOSE to establish and close a connection. The macro WITH-RPC-CONNECTION can be used to ensure the connection is closed on exit.

;; normal way to do it. establishes a connection and closes it at the end
(frpc.bind:call-dump "192.168.0.8" :protocol :tcp)
;; reuses a connection to the server
(frpc:with-rpc-connection (c "192.168.0.8" 111 :tcp)
(list (frpc.bind:call-dump :connection c)
(frpc.bind:call-dump :connection c)))

2.3 UDP

Specifying :UDP as the protocol will send the message using the UDP transport instead of TCP (UDP is the default). If you care about the result, then specify a timeout in seconds to wait for the reply. If no reply is received an RPC-TIMEOUT-ERROR will be signalled, otherwise the function returns immediately with result nil.

(frpc.bind:call-null :host "192.168.0.1" :protocol :udp)

Users may also supply a connection argument for UDP so that they don't need to keep making new UDP sockets for each RPC.

2.4 UDP broadcast

You may send messages using UDP broadcast to find services on your local network.

(frpc.bind:call-null :host "255.255.255.255" :protocol :broadcast)

Broadcast messages return a list of the responses received within the timeout -- no timeout error
is raised if no replies are received. Each element in the list is a list of 2
items (host result), where host is where the response came from
and result is the result of decoding the message that was received.

Note: not all implementations support UDP broadcast. Check with usocket to find out whether your implementation is supported.

3. RPC Server

An RPC server runs from within a single thread and listens on a set of TCP and UDP ports.
It may serve a subset of available RPC programs, by default serving all programs.

;; make a server instance
(defvar *server* (make-rpc-server :tcp-ports '(8000) :udp-ports '(8000)))
;; start the server in a new thread, it will listen for requests on TCP and UDP ports 8000
(start-rpc-server *server*)
;; stop the server thread
(stop-rpc-server *server*)

When the server accepts a TCP connection, it is added to a list of currently open connections.
The server will select a connection to process using USOCKET:WAIT-FOR-INPUT. This allows the server to
keep open multiple TCP connections without blocking other traffic. Note that the socket IO is
still synchronous. Connections which are idle for TIMEOUT seconds (default 60 seconds) are closed by the RPC server.

3.1 Server handlers

The handler function, which is invoked to process an RPC request, should return an object which matches the type specified in the associated DEFRPC
form. If the handler signals an RPC-AUTH-ERROR, the request will be rejected with the auth-stat provided (or AUTH-TOOWEAK otherwise).

It is the handler's responsibility to ensure both that the caller has authenticated to a suffiently secure level and that
the caller is authorized to call the function.

If any other error is signalled, then the RPC server will be silent, i.e. not return any response to the caller.
Some APIs require this behaviour, this is the way server handlers should support it.

Please note that because the RPC server is singly threaded, your handler function must not block execution because that will prevent
the server from processing other requests. If your handler needs to do work which takes an extended period of time
(lots of disk IO, making other RPCs etc.) then you should design your API in such a way that the initial call returns
immediately with the work taking place in another thread; the client can poll for progress or
be notified on completion (e.g. via a callback RPC).

3.2 Restricting programs

By default, an RPC server will serve all available programs. However, it might be that you want
only a subset of defined programs to be served from a particular RPC server. For instance, you might want an NFS server
to run from one thread and your portmap to run from another. To do this, supply a list of program identifiers (integers, symbols or strings)
naming the programs you wish to run in that server.

3.3 Listening on wildcard ports

If you don't supply any ports to MAKE-RPC-SERVER then a wildcard port will be selected for TCP and UDP (port number 0), this
allows for a randomly allocated high-numberd port to be used. You may enforce such befhaviour yourself by supplying 0 as a port number
if you wish.

3.4 Advanced usage

If you want more fine-grained control over the RPC server processing, you can make use of the following functions:

5. Authentication

The authentication system that was used for the request is bound to *RPC-REMOTE-AUTH* special
variable in the context of an rpc handler function. This allows handlers to implemente authorization,
i.e. determining whether the client is permitted to perform the action requested.

Supported flavours:

[x] AUTH-NULL: i.e. no authentication

[x] AUTH-UNIX and AUTH-SHORT: uid/gid and machine name. Not really authentication as such,
but a simple tagging mechanism. Provided directly by frpc.

[x] AUTH-DES: public-key exchange verified by encrypted timestamps. This requires both the client
and server have access to the public keys for each other. frpc implements its own system using a shared database
for local access and an RPC interface for remote access.

5.3 AUTH-DES

This is provided by the FRPC-DES system and should be loaded using, e.g.

CL-USER> (ql:quickload "frpc-des")

5.3.1 Overview

AUTH-DES authentication is based on a Diffie-Hellman key exchange. Each party (client and server) have a secret
key, from which public keys are derived. The public keys are exchanged beforehand by some unspecified mechanism.
Traditionally this was implemented using an RPC service (defined by key_prot.x, see programs/keyserv.lisp) but this is somewhat award to use.
Instead frpc implements its own shared database of public keys, which is exported using RPC so that remote machines
can access its entries. Local processes can simply read from the database, remote processes use the RPC interface.

The local API is

FIND-PUBLIC-KEY name ::= lookup the public key for this name

ADD-PUBLIC-KEY name public ::= store the public key for this name

REMOVE-PUBLIC-KEY name ::= delete the entry for this name

LIST-PUBLIC-KEYS ::= list all entries in the local database.

The RPC API is:

CALL-GET name ::= lookup the public key for this name

CALL-SET name public ::= set the public key for this name

CALL-UNSET name ::= delete the entry for this name

CALL-LIST ::= enumerate all entries

CALL-SET and CALL-UNSET may only be called by the named user and must have been authenticated using AUTH-DES. Note that
this means the database entry can only be created using the local API. However, they can be modified/deleted remotely.

5.3.2 Usage

Note that the client must know the name of the principal the service is running under. That has to be
agreed in advance.

On the server:

;; open the database and ensure the database has an entry with the service name and secret key
CL-USER> (frpc-des:des-init "service-user" 123123123)

5.4 AUTH-GSS (Kerberos)

This is provided by the FRPC-GSS system, load using e.g.

CL-USER> (ql:quickload "frpc-gss")

RPCSEC_GSS provides both integrity (checksumming) and privacy (encryption) of the call arguments/results. Set
the :SERVICE level to :INTEGRITY for checksumming and :PRIVACY for encryption and checksumming of the call
arguments/results. The default is :NONE which sends the args/results as normal.

;; you must first logon before you can request credentials for the application server
CL-USER> (cerberus:logon-user "myusername@myrealm" "mypassword" :kdc-address "10.1.2.3")
CL-USER> (defvar *cred* (glass:acquire-credentials :kerberos "service/hostname.com@myrealm"))
;; make the instance of the gss client
CL-USER> (defvar *client* (make-instance 'frpc:gss-client :credentials *cred* :service :privacy))
;; attempt to call the function, this will first negotiate the authentication before calling the proc
CL-USER> (frpc.bind:call-null :client *client*)
;; the server should initialize itself with a credentials handle
CL-USER> (cerberus:logon-service "service/hostname.com@myrealm" "password")
CL-USER> (frpc:gss-init)

5.5 Reauthentication

RPC servers are free to flush their tables of allocated nicknames/handles. When this happens you will
receive an RPC-AUTH-ERROR (AUTH-REJECTED) error. You should set your client back to its initial state and retry,
this should reallocate a new nickname/context handle.

CL-USER> (reinitialize-instance *client*)

5.6 Authorization

When server handlers are executed, the special variable *RPC-REMOTE-AUTH* is bound to the authenticator
that was used in the request. This allows the server to decide whether to honour the request or
to signal an RPC-AUTH-ERROR instead.

You may call RPC-AUTH-PRINCIPAL to get a string representing the authenticated caller. This can be used to aid authorization.

6. Portmap/rpcbind

Typically each RPC service listens on a randomly allocated high-numbered port. In order to find out
the port number to contact the service on you must first query a service which listens on a well-known port,
this service is called portmap or rpcbind.

See the pounds documentation for more information on the logging system.

9. XDR parser/generator

Typically RPC interfaces are described by an "x-file" which is used as input into the program rpcgen which generates code for the C programming language.
The system frpcgen (file gen/gen.lisp) provides a function to parse xfiles and generate a Lisp file with contents suitable for use with frpc.

This makes it easy to freely interoperate between Lisp and C services using RPC, because they will both be derived from the same definition.

Usage:

(frpcgen:gen "test.x")

This generates a file called "test.lisp" which contains Lisp code suitable for use with frpc. Some hand modifications will be probably be required
to make the generated code more usable, but it should at least provide a reasonable starting point.

10. Notes

At the moment, reading from TCP streams requires buffering the input to cope with reading multiple fragments. This is REALLY bad if
large payloads are sent. A fragmented-stream type could be defined to wrap the underlying socket stream so that we can avoid the buffering on reads.
You still need to buffer writes because you need to know how much you intend to write before you've written it.

Could make it easier to add more transports, e.g. SSL/TLS stream, writing to shared memory etc. Probably not much call for this though.

UDP multicast?

Currently no way to define transient programs, i.e. programs which get assigned a random program number at runtime.

Need to support batching in some sort of useful way. So that a client can setup a connection and use wait-for-input to process replies.
Basically need a pair of functions to send requests and receive replies from the connection.

The XDR serializer is probably not as efficient as it could be, but who really cares so long as it works.

Developed using SBCL on Windows. Also tested with Clozure CL on Windows, LispWorks on Windows and SBCL on Linux.

Declare an RPC interface and define a client function that invokes CALL-RPC. This must be defined before a partner DEFHANDLER form.

NAME should be a symbol naming the client function to be defined.

PROC should be an integer or constant form.

ARG-TYPE and RESULT-TYPE should be XDR type specification forms.

By default the generated function will accept a single argument which must match the type specifed by the ARG-TYPE.

OPTIONS allow customization of the generated client function:

(:program program version) PROGRAM and VERSION name the program name/number and the version number.

(:arg-transformer lambda-list &body body) makes it possible to augment the default function parameters before passing them to CALL-RPC. The body should return a value matching the type specified by ARG-TYPE.

(:transformer (var) &body body) runs after CALL-RPC has returned with VAR bound to the result. This makes it possible to destructure the result object.

(:documentation doc-string) specifies the docu-string for the client function.

(:handler function-designator) specifies a server handler for the rpc. It should designate a function of a single parameter.

Execute an RPC via the remote port mapper proxy. Returns (list PORT RES) where RES is an opaque array
of the packed result. The result needs to be extracted using FRPC:UNPACK. The result type is
recommended to be a well-defined type, i.e. represented by a symbol, so that it has an easy reader
function available.

Establish a connection and execute an RPC to a remote machine. Returns the value decoded by the RESULT-TYPE.
This function will block until a reply is received or the request times out.

ARG-TYPE should be either a writer function or a symbol naming a valid XTYPE.

ARG should be a value which can be passed to the ARG-TYPE function.

RESULT-TYPE should be either a reader function or a symbol naming a valid XTYPE.

HOST and PORT name the server to send the message to.

PROGRAM, VERSION and PROC define the RPC procedure to call.

If provided, AUTH and VERF should be OPAQUE-AUTH structures, as returned from MAKE-OPAQUE-AUTH. These are used to authenticate
the request. Note: do not use these unless you understand what you are doing.
The easy way to authenticate requests is to provide a CLIENT parameter (see below).

If provided, REQUEST-ID should be an integer specifying the message ID to use. If not provided an incrementing seqno will be used.
Generally there is no reason for users to provide this parameter.

If TIMEOUT is specified, it will be set as the RECEIVE-TIMEOUT (is using TCP) or to time to wait for UDP responses.

PROTOCOL should be :TCP, :UDP or :BROADCAST. :UDP is the default, and will block until TIMEOUT seconds for a reply
and will raise an RPC-TIMEOUT-ERROR if it doesn’t receive one. Specify TIMEOUT to NIL to return immediately and not await
a response.
:TCP will block until a response it received.
:BROADCAST should be used for UDP broadcasts. The client will wait for up to TIMEOUT seconds and collect all the repsonses
received in that time. Note that it will return a list of (host port result) instead of just the result.

CONNECTION should be a TCP or UDP connection, as returned by RPC-CONNECT.

CLIENT should be an instance of RPC-CLIENT or its subclasses. This is the ONLY way to authenticate calls.

Lookup the program identifier. The type of ID changes behaviour:
Integer: the program with a this program number is returned.
String: the first program with a case-insensitive program name is returned.
Symbol: the program with a matching symbol name is returned.

Generate a program identifier in the user-defined range, as specified by the RFC.
If TRANSIENT is true, a runtime program number is generated. These should be used by programs which
need to generate program numbers at runtime.

PROGRAMS should be a list of program numbers to be accepted by the server,
all RPC requests for programs not in this list will be rejected.
If not supplied all program requests are accepted.

UDP-PORTS and TCP-PORTS should each be a list of integers, specifying the ports
to listen on. If USOCKET:*WILDCARD-PORT* is supplied, a random unused port will be selected.
These lists will be replaced with the ports actually used once the server starts.

TIMEOUT specifies the duration (in seconds) that a TCP connection should remain open.

Initialize the UNIX authentication context table. MAX-CONTEXTS is the maximum number of valid contexts that will be granted, slots in the table will be cleared and reused once the table has been filled.

Authenticate the transaction.
FLAVOUR is the authentication flavour, data is the authentication data. VERF is the opaque-auth verifier.
Returns a response verifier to be sent back to the client or nil in the case of failure.