Reowolf: Application Programming Interface

Author: Hans-Dieter Hiep (Centrum Wiskunde & Informatica)

Logo

Date: January 22th, 2019

DRAFT VERSION

Reowolf is accessible to application programmers through an application programming interface (API). This document intends to describe the API and provides concrete examples of using the API in the C and Java programming languages.

Reowolf is a practical application of Reo. For readers unfamiliar with Reo, we refer to the introduction to Reo which walks through the central design concepts. Here, we assume basic familiarity with those concepts and focus on the technical aspect how application programming is done using Reowolf. Basic familiarity with network programming and operating environments such as POSIX is also assumed.

1. Introduction

Most network application programmers are familiar with BSD sockets. Networked applications are implemented relying on the BSD socket API, and this API is supported by most operating systems. The API allows different transport layer protocol implementations to provide streaming capabilities and messaging primitives. Some example protocols are Transmission Control Protocol (TCP), Stream Control Transmission Protocol (SCTP), and User Datagram Protocol (UDP). Sockets are used for communication between typically two endpoints, although some protocols (e.g. SCTP) support multihoming to increase reliability by redundant communication paths.

With BSD sockets, application programmers typically perform the following actions: they first open a socket, then connect or bind that socket, followed by numerous send and receive operations, finally closing down the socket when it is not used for further communication. Additionally, socket operations report back error conditions in case exceptional network conditions arise.

Although network programming using sockets is succesfully applied on Internet scale, there are a number of drawbacks:

Lack of multi-party communication: for $N$ peers, there is a need for $N*(N-1)/2$ bi-directional connections at the application layer, each maintaining a separate state. Each peer has to work with $N-1$ separate sockets.
Peer-to-peer session establishment and maintainance: many peers are behind a NAT or firewall, limiting the ability to establish direct peer-to-peer connections. Techniques such as Hole Punching and Port Number Prediction still allow peer-to-peer connections to be established, and some clients can configure routers from the local side. Such techniques are typically implemented at the application layer, or are non-standard.
Transparancy of application-defined protocols: Internet infrastructure is typically agnostic to the protocol state of applications. This lack of opacity could restrict infrastructure to find better paths or improve quality of service. Infrastructure providers currently could resort to deep packet inspection and apply non-standard techniques to predict behavioral patterns, potentially leading to concerns for privacy.
Non-conforming peers: abuse of connections can only be detected at the application layer, thus limiting Internet infrastructure to preemptively detect malformed or malicious peers. This severely limits the infrastructure to mitigate security issues, such as denial of service. Application programmers are responsible for checking conformance of peers, and may lead to security issues if done incorrectly.
Lacking communication primitives: many communication patterns typical for distributed algorithms are implemented at the application layer. Sockets provide no guarantees of simultaneous delivery on different end-points, have no native way of establishing consensus among multiple peers, do not support non-deterministic choice among destinations, nor provide means for stateful routing.
Low-level concurrency abstraction: the correctness of programs using multiple sockets (streaming and message passing) are difficult to verify compositionally, lacking separation of concerns of local computation and non-local communication.

These drawbacks all amount to more complex to develop and maintain applications, that results in more development efforts, and lower rates of quality improvement. In Reowolf, we overcome the above drawbacks by introducing a new application programming interface. This API allows programmers to:

Establish multi-party connectors, that represent the connections among multiple peers, possibly behind NATs (alleviating drawbacks 1 and 2).
Declaratively specify application-defined protocols, which is amenable to efficient verification of the conformance of peers, and supports the diagnosis and detection of deviation from the protocol by malformed or malicious peers, also by Internet infrastructure (alleviating drawbacks 3 and 4).
Simplify the development and verification effort of networked applications (alleviating drawbacks 5 and 6).

All the while, implementors of the API are given more room to apply optimization techniques in the choice of algorithms and routing policies. Advances in optimization techniques benefit all applications built on top of Reowolf. As a concrete example, recent research has shown that inter-process communication within one host can be optimized aggressively by applying compilation techniques, that is comparable to, and in some cases even exceeds, the run-time performance of hand-crafted synchronization code. Other optimization techniques, spanning distributed networks, are also foreseeable.

In section 2, we give the details of the API and give concrete interface definitions in the C and Java programming languages. Section 3 provides numerous examples of use cases, and compares the Reowolf implementation to an implementation using sockets to demonstrate the benefits and simplicity of the new API.

2. Programming Interface

A summary of the interface procedures is given below:

Connector creates a connector structure and performs a set-up of the process environment.
Configure takes a protocol description text as defined by the connector specification language, that identifies the local peer and the remote peers by name or address, the port names associated to each peer, and a protocol to be established between the associated ports of all identified peers.
Connect attempts to establish the connector by performing a handshake with all other involved peers and reaching an agreement on the shared protocol.
Close shuts a connector down, cleaning up the process environment.

A typical program would first create a connector, configure it with a valid protocol description, and connect it. If all these steps succeed, the program enters its main loop. The operations performed in the main loop depend on the selected connector class.

The API supports three communication classes. A peer prepares data for outbound and inbound communication, and synchronizes with other peers in rounds.

Real-time communication: a round has a fixed duration, measured by wall-clock time. The programmer is required to synchronize in a regular interval. If the local peer or other peers exceed the deadline, an error is detected.
- Put prepares outbound communication by supplying a port with data. The operation has effect when the connector is synchronized.
- Get prepares inbound communication. The communication must take place. The operation has effect when the connector is synchronized.
- Optional intends inbound communication, without the requirement for actual communication to take place. The operation has effect when the connector is synchronized.
- Synchronize attempts to synchronize with other peers, by the exchange of data on ports in accordance to the configured protocol. Even if no data is exchanged, synchronization is mandatory.
Logical communication: a logical round not tied to a real-time round. The programmer is not required to synchronize in a regular interval: synchronization happens automatically. The local peer stutters if other peers cause a delay in synchronization.
- Offer intends outbound communication, but allows an alternative value to be returned instead. The operation takes effect when the connector is synchronized.
- Poll intends inbound communication, by signaling other peers. The operation takes effect when the connector is synchronized.
- Await attempts to synchronize with other peers, but allows the local peer to stutter. Waits until the moment actual data was exchanged.
- Decide retrieves the decided value of a port after the connector has synchronized. The decided value can be different than what was locally offered.
Dynamic communication: this communication class extends the previous class of logical communication, with the addition of dynamic port allocation.
- Fresh intends outbound communication, similar to offer. The data to be exchanged is not application-defined, but a freshly allocated local port that is not used by any other connector.

The classes are compatbile: a peer that uses real-time communication can communicate with a peer that uses logical communication. Only the decided values are observable to the real-time peer. Vice versa, a peer that uses logical communication can communicate with a peer that uses real-time communication by hiding the effect of polling signals.

Exceptional conditions could arrise creation, connection, and synchronization operations. A reason is provided when any of these operations fail, allowing applications to recover from erroneous conditions. For example: lack of local system resources, invalid protocol description, failed to reach remote peers, unable to reach agreement on the protocol, unable to synchronize with a remote peer, or protocol renegotiation. Under certain conditions, applications can recover communication by reconfiguring the connector.

Operations are typically blocking the calling process, and in particular during connection of the connector and synchronization with other peers. Only a single thread or process performs operations on a connector. However, connectors are also suitable as mechanism for inter-process communication (IPC) within a local host, and can be used as a synchronization mechanism for threads and processes.

Non-blocking operations are out of scope for the current version of the API.

2.1. Connector

int T_REALTIME;
int T_LOGICAL;
int T_DYNAMIC;

int connector(int class);

The creation procedure takes a parameter that indicates the class of connector. The constant T_REALTIME indicates real-time communication, T_LOGICAL indicates logical communication, T_DYNAMIC indicates dynamic communication. Later versions of the API can extend the API by providing additional operations with different semantics.

Creation of a connector allocates system resources and prepares the process environment for operating on the connector. It returns an integer identifying the connector, or returns -1 and sets errno to indicate the error.

The returned integer identifying the connector must be supplied as first argument to all other operations.

public abstract class Connector {
    Connector();
    ...
}
public class RealtimeConnector extends Connector {
    public RealtimeConnector();
    ...
}
public class LogicalConnector extends Connector {
    public LogicalConnector();
    ...
}
public class DynamicConnector extends LogicalConnector {
    public DynamicConnector();
    ...
}

The abstract base class Connector provides the interface, of which RealtimeConnector and LogicalConnector are concrete subtypes. Other subtypes might be provided in the future.

2.2. Configure

int configure(int conn, char *desc_sz);
size_t checkdesc(char *desc_sz, char *out, size_t len);

Configure takes a zero-terminated string containing a protocol description. The protocol description is checked on well-formedness as defined by the connector specification language. The protocol description determines which of the peers is associated to this connector object. The peer associated to the connector from the perspective of the calling process environment is called the local peer; local is a relative notion, with a different meaning for each peer.

Returns 0 on success, or returns -1 and sets errno to indicate the error. In case an error is returned, the connector does not change state.

If the error is due to an invalid argument, the checkdesc procedure gives access to a human-readable error message explaining the error. Checking validity of a protocol description is deterministic, hence does not require a connector identifier. The out parameter is a client-supplied buffer of the supplied length; the return value is the actual length of the error message. Use of the verify procedure is not mandatory, and only provided for troubleshooting.

public abstract class Connector {
    ...
    public void configure(String desc) throws IOException;
    public static void checkdesc(String desc)
          throws InvalidDescriptionException;
}
public class ConnectorException extends IOException {}
public class InvalidDescriptionException extends ConnectorException {}

2.3. Connect

int connect(int conn);

Attempts to establish the connector, returning after when the connector is established. If the connector is not configured, the call immediately fails with an error. Otherwise, the connector will perform a handshake with all other involved peers as specified by the configured protocol. The configured protocol is negotiated, and an agreement on a shared protocol is reached if all peers accept the configured protocol. If a peer is unreachable or an agreement on the protocol cannot be made, an error is returned.

Returns 0 on success, or returns -1 and sets errno to indicate the error.

public abstract class Connector {
    ...
    public void connect() throws IOException;
}
public class NegotiationException extends ConnectorException {}

2.4. Close

int close(int conn, char* port_sz);

Attempts to close the supplied port. A closed port will for the remaining duration of the protocol not communicate, and remain silent. If all ports of a connector are closed, the connector is effectively closed too.

public abstract class Connector {
    ...
    public void close(String port) throws IOException;
}

2.5. Put

int put(int conn, char *port_sz, char *buf, size_t len);

Prepare outbound communication by supplying a port with data. The operation takes effect when the connector is synchronized. No communication takes place during the put operation. The port is given as a zero-terminated string. The data pointed to by the buffer is copied to an internal storage before the operation completes; this allows applications to reuse the same buffer when performing multiple put operations. The data in the buffer is left unmodified by this operation.

Put guarantees that after the next time synchronization completes, all peers could have the knowledge that the local port has fired with the supplied information.

Returns 0 on success, or returns -1 and sets errno to indicate the error. This operation is only applicable to connectors of the T_REALTIME class.

Multiple calls to put on the same port before synchronization results in the last supplied data being used. It is valid to perform the put operation with the buffer pointing to NULL, thereby erasing the previous put operation, if any. Not calling put for a port associated to the self-peer has the same effect as calling put for that port with a NULL buffer.

If a local deviation from the protocol is detected, an error is raised: the connector is not configured or not connected, or if the port name is not declared in the protocol description, or if the port name is not associated to the self-peer of this connector, or if the buffer size is not compatible with the port’s type. These errors are programming errors that under normal circumstance should not happen. If an error arises, the state the connector is not affected. Violations that depend on the protocol state, possibly out of control of the programmer, are deferred until the moment of synchronization.

public class RealtimeConnector extends Connector {
    ...
    public void put(String port, byte[] buffer);
}

2.6. Get

int get(int conn, char *port_sz, char *buf, size_t len);

Prepare inbound communication by supplying a port with a target buffer. The operation takes effect when the connector is synchronized. No communication takes place during the get operation. The port is given as a zero-terminated string. The buffer is left unmodified by this operation. The buffer will be modified the next time the connector synchronizes succesfully. The undefined behavior results if the target buffer is overlapping with inaccessible memory at the time of synchronization.

Get guarantees that after the next time synchronization completes, all peers could have the knowledge that the local port has fired with the information stored in the target buffer.

Returns 0 on success, or returns -1 and sets errno to indicate the error. This operation is only applicable to connectors of the T_REALTIME class.

Multiple calls to get or opt on the same port before synchronization results in the last buffer being used for receiving data. It is valid to perform the get operation with the buffer pointing to NULL, thereby erasing the previous operation, if any. Not calling get or opt for a port associated to the self-peer has the same effect as calling get or opt for that port with a NULL buffer.

A program is allowed to perform a get operation on different ports with the same buffer, but it is advisable not to do so. In such a case, it is underspecified which inbound value takes precedence. After successful synchronization, the value of only one port appears to be written to the buffer.

public class RealtimeConnector extends Connector {
    ...
    public void get(String port, byte[] buffer);
}

2.7. Optional

int opt(int conn, char *port_sz, char *buf, size_t len);

Intend inbound communication by supplying a port with a target buffer. The operation takes effect when the connector is synchronized. No communication takes place during the opt operation. The port is given as a zero-terminated string. The buffer is left unmodified by this operation. The buffer may be modified the next time the connector synchronizes succesfully. The undefined behavior results if the target buffer is overlapping with inaccessible memory at the time of synchronization.

Opt differs from get: the application must check whether actual information was transmitted. The buffer is left unmodified if the port did not fire after the last succesful synchronization.

Returns 0 on success, or returns -1 and sets errno to indicate the error. This operation is only applicable to connectors of the T_REALTIME class.

A program is allowed to perform an opt operation on different ports with the same buffer, but it is advisable not to do so. In such a case, it is underspecified which inbound value takes precedence. After successful synchronization, the value of at most one port appears to be written to the buffer.

public class RealtimeConnector extends Connector {
    ...
    public void opt(String port, byte[] buffer);
}

2.8. Synchronize

int sync(int conn);

Attempt to synchronize with other peers, by the exchange of data on ports in accordance to the configured protocol. The outbound communication, as prepared by put operations, is attempted. Any inbound communication, as prepared by get operations, is anticipated. The operation blocks until the connector has succesfully synchronized, or until an error is detected.

Returns a non-negative integer on success, or returns -1 and sets errno to indicate the error. This operation is only applicable to connectors of the T_REALTIME class.

If the operation completed successfully, the program has the knowledge common with every other peer connected by the connector, that every peer succesfully completed synchronization. After completion, all values prepared by put are erased, all buffers prepared by get are written to, and the pointers to buffers forgotten.

The operation may raise an error, depending on the protocol state and the communication with other peers. If another peer did not synchronize in a timely manner, e.g. by becoming unreachable, the synchronization attempt fails by a timeout. If a put or get operation is inconsistent with the application-defined protocol, no behavior can satisfy and synchronization fails. If one or more of the remote peers does not conform to the application-defined protocol, a deviation is detected and synchronization fails. Synchronization also fails if renegotiation of the protocol does not lead to an agreement.

The operation returns with 1 to indicate that the protocol has expired and may be renegotiated. A subsequent call to configure proposes a new protocol, that takes effect the next time the connector synchronizes. The last protocol and the new protocol are independent. If a call to configure is not made before the next time the connector synchronizes, the last protocol is renegotiated implicitly. If the protocol has not expired, 0 is returned.

public class RealtimeConnector extends Connector {
    ...
    public boolean sync() throws IOException;
}
public class ConnectorTimeoutException extends ConnectorException {}
public class UnsatisfiableBehaviorException extends ConnectorException {}
public class DeviatedBehaviorException extends ConnectorException {}

2.9. Offer

int offer(int conn, char *port_sz, char *buf, size_t len);

public class LogicalConnector extends Connector {
    ...
    public void offer(String port, byte[] buffer);
}

2.10. Poll

int poll(int conn, char *port_sz, char *buf, size_t len);

public class LogicalConnector extends Connector {
    ...
    public void poll(String port, byte[] buffer);
}

2.11. Await

int await(int conn);

Attempt to synchronize with other peers. The operation suspends the calling process and keeps synchronizing until an actual communication happend.

Returns a non-negative integer on success, or returns -1 and sets errno to indicate the error.

After completion, all values prepared by offer are erased, some buffers prepared by poll are written to, and the pointers to buffers forgotten.

The operation may raise an error, depending on the protocol state and the communication with other peers. If another peer did not synchronize in a timely manner, e.g. by becoming unreachable, the synchronization attempt fails by a timeout. If an offer or poll operation is inconsistent with the application-defined protocol, no behavior can satisfy and synchronization fails. If one or more of the remote peers does not conform to the application-defined protocol, a deviation is detected and synchronization fails. Synchronization also fails if renegotiation of the protocol does not lead to an agreement.

The operation returns with 1 to indicate that the protocol has expired and may be renegotiated. A subsequent call to configure proposes a new protocol, that takes effect the next time the connector synchronizes. The last protocol and the new protocol are independent. If a call to configure is not made before the connector synchronizes, the last protocol is renegotiated implicitly. If the protocol has not expired, 0 is returned.

public class LogicalConnector extends Connector {
    ...
    public boolean await() throws IOException;
}

2.12. Decide

int decide(int conn, char *port_sz, char *buf, size_t len);

Returns 0 if the supplied port did not fire the last round, and leaves the buffer unmodified. Returns 1 if the supplied port did fire, and modifies the buffer to reflect the data that was decided upon the previous synchronization round.

public class LogicalConnector extends Connector {
    ...
    public boolean decide(String port, byte[] buffer);
}

2.13. Fresh

int fresh(int conn, char *port_sz);

Offers a freshly allocated local port name as data to the supplied port. The actual value decided upon can be retrieved after synchronization using decide.

public class DynamicConnector extends LogicalConnector {
    ...
    public void fresh(String port);
}

3. Examples

TODO