Reo is a channel-based exogenous coordination language.
Author: Hans-Dieter Hiep (Centrum Wiskunde & Informatica)
Date: January 22th, 2019
DRAFT VERSION
Reowolf is accessible to application programmers through an application programming interface (API). This document intends to describe the API and provides concrete examples of using the API in the C and Java programming languages.
Reowolf is a practical application of Reo. For readers unfamiliar with Reo, we refer to the introduction to Reo which walks through the central design concepts. Here, we assume basic familiarity with those concepts and focus on the technical aspect how application programming is done using Reowolf. Basic familiarity with network programming and operating environments such as POSIX is also assumed.
Most network application programmers are familiar with BSD sockets. Networked applications are implemented relying on the BSD socket API, and this API is supported by most operating systems. The API allows different transport layer protocol implementations to provide streaming capabilities and messaging primitives. Some example protocols are Transmission Control Protocol (TCP), Stream Control Transmission Protocol (SCTP), and User Datagram Protocol (UDP). Sockets are used for communication between typically two endpoints, although some protocols (e.g. SCTP) support multihoming to increase reliability by redundant communication paths.
With BSD sockets, application programmers typically perform the following actions: they first open a socket, then connect or bind that socket, followed by numerous send and receive operations, finally closing down the socket when it is not used for further communication. Additionally, socket operations report back error conditions in case exceptional network conditions arise.
Although network programming using sockets is succesfully applied on Internet scale, there are a number of drawbacks:
These drawbacks all amount to more complex to develop and maintain applications, that results in more development efforts, and lower rates of quality improvement. In Reowolf, we overcome the above drawbacks by introducing a new application programming interface. This API allows programmers to:
All the while, implementors of the API are given more room to apply optimization techniques in the choice of algorithms and routing policies. Advances in optimization techniques benefit all applications built on top of Reowolf. As a concrete example, recent research has shown that inter-process communication within one host can be optimized aggressively by applying compilation techniques, that is comparable to, and in some cases even exceeds, the run-time performance of hand-crafted synchronization code. Other optimization techniques, spanning distributed networks, are also foreseeable.
In section 2, we give the details of the API and give concrete interface definitions in the C and Java programming languages. Section 3 provides numerous examples of use cases, and compares the Reowolf implementation to an implementation using sockets to demonstrate the benefits and simplicity of the new API.
A summary of the interface procedures is given below:
A typical program would first create a connector, configure it with a valid protocol description, and connect it. If all these steps succeed, the program enters its main loop. The operations performed in the main loop depend on the selected connector class.
The API supports three communication classes. A peer prepares data for outbound and inbound communication, and synchronizes with other peers in rounds.
The classes are compatbile: a peer that uses real-time communication can communicate with a peer that uses logical communication. Only the decided values are observable to the real-time peer. Vice versa, a peer that uses logical communication can communicate with a peer that uses real-time communication by hiding the effect of polling signals.
Exceptional conditions could arrise creation, connection, and synchronization operations. A reason is provided when any of these operations fail, allowing applications to recover from erroneous conditions. For example: lack of local system resources, invalid protocol description, failed to reach remote peers, unable to reach agreement on the protocol, unable to synchronize with a remote peer, or protocol renegotiation. Under certain conditions, applications can recover communication by reconfiguring the connector.
Operations are typically blocking the calling process, and in particular during connection of the connector and synchronization with other peers. Only a single thread or process performs operations on a connector. However, connectors are also suitable as mechanism for inter-process communication (IPC) within a local host, and can be used as a synchronization mechanism for threads and processes.
Non-blocking operations are out of scope for the current version of the API.
int T_REALTIME;
int T_LOGICAL;
int T_DYNAMIC;
int connector(int class);
The creation procedure takes a parameter that indicates the class of
connector. The constant T_REALTIME indicates real-time communication,
T_LOGICAL indicates logical communication, T_DYNAMIC indicates
dynamic communication.
Later versions of the API can extend the API by providing
additional operations with different semantics.
Creation of a connector allocates system resources and prepares the
process environment for operating on the connector.
It returns an integer identifying the connector,
or returns -1 and sets errno to indicate the error.
The returned integer identifying the connector must be supplied as first argument to all other operations.
public abstract class Connector {
Connector();
...
}
public class RealtimeConnector extends Connector {
public RealtimeConnector();
...
}
public class LogicalConnector extends Connector {
public LogicalConnector();
...
}
public class DynamicConnector extends LogicalConnector {
public DynamicConnector();
...
}
The abstract base class Connector provides the interface, of which
RealtimeConnector and LogicalConnector are concrete subtypes.
Other subtypes might be provided in the future.
int configure(int conn, char *desc_sz);
size_t checkdesc(char *desc_sz, char *out, size_t len);
Configure takes a zero-terminated string containing a protocol description. The protocol description is checked on well-formedness as defined by the connector specification language. The protocol description determines which of the peers is associated to this connector object. The peer associated to the connector from the perspective of the calling process environment is called the local peer; local is a relative notion, with a different meaning for each peer.
Returns 0 on success, or returns -1 and sets errno to indicate
the error. In case an error is returned, the connector does not change
state.
If the error is due to an invalid argument, the checkdesc procedure
gives access to a human-readable error message explaining the error.
Checking validity of a protocol description is deterministic, hence
does not require a connector identifier. The out parameter is a
client-supplied buffer of the supplied length;
the return value is the actual length of the error message.
Use of the verify procedure is not mandatory,
and only provided for troubleshooting.
public abstract class Connector {
...
public void configure(String desc) throws IOException;
public static void checkdesc(String desc)
throws InvalidDescriptionException;
}
public class ConnectorException extends IOException {}
public class InvalidDescriptionException extends ConnectorException {}
int connect(int conn);
Attempts to establish the connector, returning after when the connector is established. If the connector is not configured, the call immediately fails with an error. Otherwise, the connector will perform a handshake with all other involved peers as specified by the configured protocol. The configured protocol is negotiated, and an agreement on a shared protocol is reached if all peers accept the configured protocol. If a peer is unreachable or an agreement on the protocol cannot be made, an error is returned.
Returns 0 on success,
or returns -1 and sets errno to indicate the error.
public abstract class Connector {
...
public void connect() throws IOException;
}
public class NegotiationException extends ConnectorException {}
int close(int conn, char* port_sz);
Attempts to close the supplied port. A closed port will for the remaining duration of the protocol not communicate, and remain silent. If all ports of a connector are closed, the connector is effectively closed too.
public abstract class Connector {
...
public void close(String port) throws IOException;
}
int put(int conn, char *port_sz, char *buf, size_t len);
Prepare outbound communication by supplying a port with data.
The operation takes effect when the connector is synchronized.
No communication takes place during the put operation.
The port is given as a zero-terminated string.
The data pointed to by the buffer is copied to an internal storage
before the operation completes; this allows applications to reuse the
same buffer when performing multiple put operations.
The data in the buffer is left unmodified by this operation.
Put guarantees that after the next time synchronization completes, all peers could have the knowledge that the local port has fired with the supplied information.
Returns 0 on success,
or returns -1 and sets errno to indicate the error.
This operation is only applicable to connectors of the T_REALTIME class.
Multiple calls to put on the same port before synchronization
results in the last supplied data being used.
It is valid to perform the put operation with the buffer pointing to
NULL, thereby erasing the previous put operation, if any.
Not calling put for a port associated to the self-peer has
the same effect as calling put for that port with a NULL buffer.
If a local deviation from the protocol is detected, an error is raised: the connector is not configured or not connected, or if the port name is not declared in the protocol description, or if the port name is not associated to the self-peer of this connector, or if the buffer size is not compatible with the port’s type. These errors are programming errors that under normal circumstance should not happen. If an error arises, the state the connector is not affected. Violations that depend on the protocol state, possibly out of control of the programmer, are deferred until the moment of synchronization.
public class RealtimeConnector extends Connector {
...
public void put(String port, byte[] buffer);
}
int get(int conn, char *port_sz, char *buf, size_t len);
Prepare inbound communication by supplying a port with a target buffer.
The operation takes effect when the connector is synchronized.
No communication takes place during the get operation.
The port is given as a zero-terminated string.
The buffer is left unmodified by this operation.
The buffer will be modified the next time the connector synchronizes
succesfully.
The undefined behavior results if the target buffer is overlapping with
inaccessible memory at the time of synchronization.
Get guarantees that after the next time synchronization completes, all peers could have the knowledge that the local port has fired with the information stored in the target buffer.
Returns 0 on success,
or returns -1 and sets errno to indicate the error.
This operation is only applicable to connectors of the T_REALTIME class.
Multiple calls to get or opt on the same port before synchronization
results in the last buffer being used for receiving data.
It is valid to perform the get operation with the buffer pointing to
NULL, thereby erasing the previous operation, if any.
Not calling get or opt for a port associated to the self-peer has
the same effect as calling get or opt for that port with a NULL buffer.
A program is allowed to perform a get operation on different ports with the same buffer, but it is advisable not to do so. In such a case, it is underspecified which inbound value takes precedence. After successful synchronization, the value of only one port appears to be written to the buffer.
If a local deviation from the protocol is detected, an error is raised: the connector is not configured or not connected, or if the port name is not declared in the protocol description, or if the port name is not associated to the self-peer of this connector, or if the buffer size is not compatible with the port’s type. These errors are programming errors that under normal circumstance should not happen. If an error arises, the state the connector is not affected. Violations that depend on the protocol state, possibly out of control of the programmer, are deferred until the moment of synchronization.
public class RealtimeConnector extends Connector {
...
public void get(String port, byte[] buffer);
}
int opt(int conn, char *port_sz, char *buf, size_t len);
Intend inbound communication by supplying a port with a target buffer.
The operation takes effect when the connector is synchronized.
No communication takes place during the opt operation.
The port is given as a zero-terminated string.
The buffer is left unmodified by this operation.
The buffer may be modified the next time the connector synchronizes
succesfully.
The undefined behavior results if the target buffer is overlapping with
inaccessible memory at the time of synchronization.
Opt differs from get: the application must check whether actual information was transmitted. The buffer is left unmodified if the port did not fire after the last succesful synchronization.
Returns 0 on success,
or returns -1 and sets errno to indicate the error.
This operation is only applicable to connectors of the T_REALTIME class.
Multiple calls to get or opt on the same port before synchronization
results in the last buffer being used for receiving data.
It is valid to perform the get operation with the buffer pointing to
NULL, thereby erasing the previous operation, if any.
Not calling get or opt for a port associated to the self-peer has
the same effect as calling get or opt for that port with a NULL buffer.
A program is allowed to perform an opt operation on different ports with the same buffer, but it is advisable not to do so. In such a case, it is underspecified which inbound value takes precedence. After successful synchronization, the value of at most one port appears to be written to the buffer.
If a local deviation from the protocol is detected, an error is raised: the connector is not configured or not connected, or if the port name is not declared in the protocol description, or if the port name is not associated to the self-peer of this connector, or if the buffer size is not compatible with the port’s type. These errors are programming errors that under normal circumstance should not happen. If an error arises, the state the connector is not affected. Violations that depend on the protocol state, possibly out of control of the programmer, are deferred until the moment of synchronization.
public class RealtimeConnector extends Connector {
...
public void opt(String port, byte[] buffer);
}
int sync(int conn);
Attempt to synchronize with other peers, by the exchange of data on ports in accordance to the configured protocol. The outbound communication, as prepared by put operations, is attempted. Any inbound communication, as prepared by get operations, is anticipated. The operation blocks until the connector has succesfully synchronized, or until an error is detected.
Returns a non-negative integer on success,
or returns -1 and sets errno to indicate the error.
This operation is only applicable to connectors of the T_REALTIME class.
If the operation completed successfully, the program has the knowledge
common with every other peer connected by the connector, that every peer
succesfully completed synchronization.
After completion,
all values prepared by put are erased,
all buffers prepared by get are written to,
and the pointers to buffers forgotten.
The operation may raise an error, depending on the protocol state and the communication with other peers. If another peer did not synchronize in a timely manner, e.g. by becoming unreachable, the synchronization attempt fails by a timeout. If a put or get operation is inconsistent with the application-defined protocol, no behavior can satisfy and synchronization fails. If one or more of the remote peers does not conform to the application-defined protocol, a deviation is detected and synchronization fails. Synchronization also fails if renegotiation of the protocol does not lead to an agreement.
The operation returns with 1 to indicate that
the protocol has expired and may be renegotiated.
A subsequent call to configure proposes a new protocol,
that takes effect the next time the connector synchronizes.
The last protocol and the new protocol are independent.
If a call to configure is not made before the next time the connector synchronizes,
the last protocol is renegotiated implicitly.
If the protocol has not expired, 0 is returned.
public class RealtimeConnector extends Connector {
...
public boolean sync() throws IOException;
}
public class ConnectorTimeoutException extends ConnectorException {}
public class UnsatisfiableBehaviorException extends ConnectorException {}
public class DeviatedBehaviorException extends ConnectorException {}
int offer(int conn, char *port_sz, char *buf, size_t len);
public class LogicalConnector extends Connector {
...
public void offer(String port, byte[] buffer);
}
int poll(int conn, char *port_sz, char *buf, size_t len);
public class LogicalConnector extends Connector {
...
public void poll(String port, byte[] buffer);
}
int await(int conn);
Attempt to synchronize with other peers. The operation suspends the calling process and keeps synchronizing until an actual communication happend.
Returns a non-negative integer on success,
or returns -1 and sets errno to indicate the error.
After completion,
all values prepared by offer are erased,
some buffers prepared by poll are written to,
and the pointers to buffers forgotten.
The operation may raise an error, depending on the protocol state and the communication with other peers. If another peer did not synchronize in a timely manner, e.g. by becoming unreachable, the synchronization attempt fails by a timeout. If an offer or poll operation is inconsistent with the application-defined protocol, no behavior can satisfy and synchronization fails. If one or more of the remote peers does not conform to the application-defined protocol, a deviation is detected and synchronization fails. Synchronization also fails if renegotiation of the protocol does not lead to an agreement.
The operation returns with 1 to indicate that
the protocol has expired and may be renegotiated.
A subsequent call to configure proposes a new protocol,
that takes effect the next time the connector synchronizes.
The last protocol and the new protocol are independent.
If a call to configure is not made before the connector synchronizes,
the last protocol is renegotiated implicitly.
If the protocol has not expired, 0 is returned.
public class LogicalConnector extends Connector {
...
public boolean await() throws IOException;
}
int decide(int conn, char *port_sz, char *buf, size_t len);
Returns 0 if the supplied port did not fire the last round, and
leaves the buffer unmodified.
Returns 1 if the supplied port did fire, and modifies the buffer
to reflect the data that was decided upon the previous synchronization round.
public class LogicalConnector extends Connector {
...
public boolean decide(String port, byte[] buffer);
}
int fresh(int conn, char *port_sz);
Offers a freshly allocated local port name as data to the supplied port.
The actual value decided upon can be retrieved after synchronization using decide.
public class DynamicConnector extends LogicalConnector {
...
public void fresh(String port);
}
TODO