\" To run off, use: pic file | tbl | troff -ms | printer .nr PS 12 .nr VS 14 .de P[ .nr Pf \\n(.f .nr Ps \\n(.s .ft 5 .ps 10p .FI 0 .ta 36n .. .de P] .FI .ps \\n(Ps .ft \\n(Pf .. .de AM Amoeba\\$1 .. .ND .TL Wide-Area Communication under .AM .AU Robbert van Renesse* Hans van Staveren .AI Vrije Universiteit Amsterdam .AB This document describes the earliest Amoeba wide-area gateway service that makes it possible to execute .AM transactions over existing long-haul networks. Furthermore, the specification of the interface between network drivers and the gateway is given. Finally, we will propose how to use the gateway service. .AE .FS * Part of the work was done at the University of Troms\o'o/', Norway. .FE .NH INTRODUCTION .PP When given the task of implementing a gateway, we face some non-trivial problems. First, we want to have transparent communication between .AM systems in different countries all over Europe as part of a COST-11 project. Second, the .AM systems are running on heterogeneous computers, local-area networks, and network layers. For example, there are .AM operating systems running on several different processors, and there are .AM .I layers running under .UX using Berkeley sockets. Byte ordering is also a problem. Third, the wide-area links are far from optimal. Many sites do not have the authority to address the network interfaces directly (they are maintained by the institution, not by the department), the interfaces are different on all sites, and sometimes the only connection between a pair of sites is over several different networks. Fourth, .AM ports do not give a clue about the location of servers waiting for requests on that port, making server lookup close to impossible. .PP To minimize development costs we decided to use whatever software was available and not to change any existing network software. Because all the sites in the initial group of participants had .UX we decided to implement the initial service by .UX processes. .NH MODEL .PP We divide the .AM wide-area system into three layers, as shown in Fig. 1. .sp .F1 .TS allbox, center; l l. Amoeba layer Amoeba clients and servers Gateway layer Gateway software Network layer Existing wide-area services .TE .F2 .ce \s-2\fBFig. 1.\fR Layers in the Amoeba wide-area system.\s0 .F3 .PP The highest layer, the .AM layer, contains the clients and servers. Clients and servers on the same .AM site communicate with each other directly, without using lower layers. .PP The middle layer, called the .I gateway layer, provides the means for making transparent wide-area transactions possible. This layer contains the new software that had to be written. .PP The lowest layer, the .I network layer, consists of whatever wide-area communication services are available. It usually provides virtual circuits. .NH 2 The .AM Layer .PP The .AM layer can differ from site to site, as long as it supports the .AM transaction calls. We have made no changes to this layer whatsoever. Each site may have its own local-area network protocol and port location mechanism. .PP At the time that this document is being written we have two different .AM systems. First, we have an .AM operating system with links into several kinds of .UX systems. Second, and most frequently used, there is an .AM library that uses .UX BSD 4.2 sockets to carry out transactions. The gateway runs on both these systems, making communication between them possible. .NH 2 The Network Layer .PP This layer consists of several heterogeneous networks, different in size, quality of service, and interface. Currently we only use X.25 for inter-country communication, usually via X.29 PADs. To reach the PADs, however, we often have to cross several intra- and inter-institute networks and port selectors. .PP To get from one .UX machine to another, a program is needed that connects to the ``outside.'' Then a series of login procedures and network interface commands are needed to log on to the remote .UX node. Although some, or maybe even all, networks on the path may provide reliable communication, the total line is usually not reliable. For example, PAD to PAD communication is reliable, but the V.24 communication between PAD and computer is not. A simple network protocol running on both nodes is sufficient for obtaining end-to-end reliability. .NH 2 The Gateway Layer .PP The gateway layer functions as the glue between the .AM layer and the network layer. On the borders of the gateway layer we have abstracted away from the communication problems in the adjoining layers, so additional network protocols are not needed to achieve reliability. .PP The function of both adjoining layers is to provide reliable communication between processes. The interface to each layer differs, however. The .AM layer provides request/reply exchanges, while the network layer provides virtual circuits. Naming is also radically different. One task of the gateway layer is to convert between the two types of communication. .PP Another task of the gateway is to extract those requests and replies from the .AM layer that have to be sent to remote sites, and, conversely, to insert request and replies received from remote sites. To simplify matters, we use the term .I message as being either a request or a reply. The gateway's task then is to transfer messages to and from .AM systems by sending them over virtual circuits between the .AM sites. .NH DESIGN .PP This section only discusses the design of the gateway layer, since the other two layers were designed and implemented a long time ago, and we use them as they are. .PP The gateway layer consists of a gateway server at each .AM site. Since it has to communicate with .AM layer processes, it runs as an ordinary .AM server. Since it has to communicate with remote gateway servers over virtual circuits, it needs to be able to set up and close down connections, and to send and receive data over the connections. .PP A gateway server consists of four parts, as shown in Fig. 2. .sp .F1 .PS boxwid=2i boxht=0.5i A: box "Network agent" B: box with .sw at A.sw + (0, 1i) "Gateway switch" C: box with .se at B.nw + (0.25i, 0.5i) "Client agent" D: box with .sw at B.ne + (-0.25i, 0.5i) "Server agent" arrow from D.s to B.ne + (-0.75i, 0) arrow from B.nw + (0.75i, 0) to C.s line <-> from A.n to B.s .PE .F2 .ce \s-2\fBFig. 2.\fR Modules in the gateway server.\s0 .F3 .sp .PP The server agent accepts requests from local clients and sends them to the gateway switch. Ultimately the gateway switch will send a reply back to the server agent, which will subsequently send it to the client. The server agent plays the role of the remote server to the client. .PP The client agent does the reverse. It waits for a request message from the gateway switch, does a local transaction to send the request to the appropriate server, and sends the reply back to the gateway switch. The client agent acts like the remote client to the server. .PP The network agent transfers messages between the gateway switch and the wide-area network. .PP The gateway switch decides where messages should go. When it gets a request message, it looks up the destination port in a table internal to the gateway server. If an entry for this port exists, it uses the address information therein to send the message to the appropriate remote site. If there is no entry for the port, it sends the request to the client agent, which will try to find a local server for the request. The request messages are tagged with the source address, to allow the replies to be routed to the originator of the request. .PP Reply messages are routed using their address tag, and can be sent by the gateway switch to either the network agent, or the server agent. The gateway switch can also generate reply messages when a wide-area connections fails. In that case it sends an error reply message back for every outstanding request on that (former) connection. .NH IMPLEMENTATION .PP Since the gateway server needs to be able to handle multiple clients and servers, drive multiple wide-area networks, and run the gateway switch, the server needs multiple .AM tasks. As mentioned earlier, the initial gateway servers run on .UX systems supporting the .AM transaction interface. These .UX interfaces do not allow tasks or shared memory. The real gateways, running on .AM will have these features. .PP Our implementation therefore simulates multiple tasks by having a .UX process for each .AM task, and shared memory using .UX pipes. To minimize conflicts (like waiting for more than one event), all messages to the gateway switch are sent using transactions, whereas messages .I from the gateway switch are sent over pipes. This reduces the conflicts to only one: sometimes a process in the network agent is interested in messages from both the gateway and a virtual circuit at the same time. This conflict is resolved by having the process regularly poll the pipe from the gateway switch. .PP The .UX process structure is as follows. There is one process for each .AM task. The gateway switch is the overall parent process, and forks off the processes in the different agents of the gateway server. These child processes are called .I stubs . There is one pipe from the gateway switch process to each stub. .sp .F1 .PS boxwid=1.5i boxht=0.5i A: box "Network stub" B: box with .se at A.se + (0, 1i) "Switch" C: box with .se at B.se + (0, 1i) "Client stub" D: box with .se at C.se + (0, 1i) "Client" E: box with .sw at A.se + (2i, 0) "Network stub" F: box with .se at E.se + (0, 1i) "Switch" G: box with .se at F.se + (0, 1i) "Server stub" H: box with .se at G.se + (0, 1i) "Server" arrow from D.s to C.n arrow from C.s to B.n arrow from B.s to A.n line from A.s down 1i line from A.s + (0, -1i) to E.s + (0, -1i) arrow from E.s + (0, -1i) to E.s arrow from E.n to F.s arrow from F.n to G.s arrow from G.n to H.s line dotted from D.s + (-1i, -0.25i) to H.s + (1i, -0.25i) line dotted from A.s + (-1i, -0.25i) to E.s + (1i, -0.25i) boxwid=2i box invis with .sw at D.se "Amoeba layer" box invis with .sw at B.se "Gateway layer" box invis with .sw at A.se + (0, -0.75i) "Network layer" .PE .F2 .ce \s-2\fBFig. 3.\fR How request messages travel through the system.\s0 .F3 .sp .PP Messages are made by concatenating the .AM header and buffer, and prepending a small header containing the message type (request or reply), and an address tag used for routing replies (see Fig. 4(a)). A request message travels from client to server as shown in Fig. 3. .ne 16 .F1 .PS boxwid=1.5i boxht=0.25i A: box ht 1i "Amoeba buffer" "(Up to 2K bytes)" B: box ht 0.5i with .sw at A.nw "Amoeba header" "(32 bytes)" C: box with .sw at B.nw "Reserved" D: box with .sw at C.nw "Reserved" E: box with .sw at D.nw "Address tag" F: box with .sw at E.nw "Message type" G: box ht 1.5i with .sw at A.se + (2i, 0) "Message" H: box with .sw at G.nw "Low byte of size" I: box with .sw at H.nw "High byte of size" J: box with .sw at I.nw "Reserved" K: box with .sw at J.nw "Buffer type" "(a)" at A.s + (0, -0.1i) below "(b)" at G.s + (0, -0.1i) below .PE .F2 .ce \s-2\fBFig. 4.\fR (a) Amoeba request or reply message. (b) Buffer structure.\s0 .F3 .NH 2 Communication Over Pipes .PP Communication over a pipe is as follows. First the gateway switch creates a capability for using the switch, and writes it onto the pipe. The capability identifies the stub to the switch when the stub later sends requests using this capability. All further communication is via .I buffers . The buffer structure is depicted in Fig. \*(Fp (b). .PP A buffer consists of a header and a message. The header contains the type and the size of the message. There are three types: .IP OPEN: 9 open a virtual circuit. This message can only be sent to a network stub, requesting it to open a connection. The address is in the message. .IP CLOSE: close a virtual circuit. This requests a network stub to close the current connection. .IP DATA: the message is either a request or a reply. A server stub sends it to its client, a client stub sends it to some server, and a network stub sends it over its current connection. .LP The stub is required to send an acknowledgement for each of the buffers it receives, to prevent the switch from being blocked on a full pipe. .NH 2 The Gateway Switch .PP The gateway switch is a true server process waiting for and executing requests. Its main task is to send messages, received in the requests, to the right stubs over the pipes. Furthermore, it manages the connections with remote gateways by requesting network stubs to set up or close virtual circuits. An additional feature is an error recovery scheme for broken connections. .NH 3 Requests to the Switch .PP An .AM request consists of a request header and a request buffer. The contents of these for the different types of requests to the gateway switch are discussed in this section. .IP SEND: 14 The request buffer contains either a request or a reply message from some stub. The switch has to route the message to the correct destination stub. .IP DONE: This is an acknowledgement for a buffer that was sent over a pipe. The switch can now send another buffer over the pipe without danger of blocking on a full pipe. .IP FAILED: Something went wrong at the stub that sent this request. In the current implementation this can only be a broken virtual circuit. .IP REMCONN: A remote gateway seeks contact with the local gateway. The buffer contains information about what type of network stub to use for communication; there is a different network stub for each type of network service. The switch is supposed to send an OPEN request to this stub. .IP INSTALL: Install a remote port in the table of ports. The address of the port is in the request buffer. The switch has to fork off a server stub to collect the local requests on that port. .IP CONNECT: Add a new network stub to the gateway server. The type of wide-area network is in the request buffer. The switch has to fork off a process for this stub. .IP SWEEP: Virtual circuits are closed by the gateway switch when they have not been used for a while. The sweep request is sent periodically to the gateway switch, which then scans its table of network stubs to see whether the connections there are being used. .IP SHUTDOWN: This requests the switch to ``gracefully'' stop all the gateway server activities. The switch sends a signal to all the stubs and exit itself. .LP The gateway switch acknowledges the requests to the sender by sending a reply. This reply may contain an error status if the switch was not able to execute the request. .NH 3 Routing Messages .PP When a request message arrives at the gateway switch, it is routed to the proper stub by comparing the port in the header of the request to each port in the internal port table. If the port is there, the message is transferred to a network stub. The address in the table determines what type of network stub is to be used, and the network address on the corresponding network. If no entry for the port is found, the request message is routed to the client stub, which will try to find a server on the local network. .PP Associated with each stub is a message queue. When a virtual circuit is opened or a DONE message for the stub arrives, the switch scans this queue to see if there are messages to be transmitted to the stub. Each entry in the queue is uniquely identified by a small integer, and contains the address of the sender of the associated message. When a request message is sent over the virtual circuit, it is tagged with this identifier. The associated reply message will contain the same tag. .PP When the reply message arrives at the switch, it locates the queue entry, and use the address therein to route the message to the proper recipient. At the same time, the queue entry is removed. Queue entries for reply messages are deleted as soon as the message has been transferred to the stub. .PP The address information in the queue entries consists of three fields: the stub where the message arrived from, the queue identifier in that message, and an .I "incarnation number" , that is used to detect and recover from broken virtual circuits. .PP Virtual circuits are reliable, in the sense that they transfer data without damaging the contents or the order in which the data was sent. However, virtual circuits can break for reasons like network congestion, or a crash of the remote host. The gateway can then reuse the stub to set up a new connection, possibly to another destination. To differentiate between the connections, there is an .I "incarnation number" associated with each stub, which is incremented with every new connection. The incarnation number in the queue entries can be used to prevent replies being routed to wrong destinations; if the incarnation numbers do not match, the message is simply discarded. .PP The queue can be used to generate error reply messages to the clients when a virtual circuit breaks. When a connection breaks, the stub sends a FAILED request to the gateway switch, which then generates replies for every request entry in the queue of the stub. .PP Saving the queue identifiers of requests in the queue for the destination stub permits requests and replies to be routed through several gateways. The port table in the gateway contains the address of the remote host that is only one hop away; the gateway there decides how to route request messages further. For reply messages there is enough information in the queue to route them back to the clients. Thus arbitrary network topologies are supported by this scheme. .NH 3 Making connections .PP When a request message arrives at the gateway for a remote site for which no virtual circuit exists, a connection has to be set up. The gateway switch takes a free network stub and sends an OPEN request to this stub, which then makes the connection. Meanwhile, messages that arrive for this stub are queued. .PP The site that receives the call sends a REMCONN request to the gateway switch at that site, to tell it that somebody is trying to make contact. The switch then sends a special OPEN request to a free network stub to tell it to make the connection. The address used in the OPEN request is obtained from the REMCONN request to the switch. .PP When connections have not been used for a while, the switch will break the virtual circuit by sending a CLOSE request to the network stub. A connection is defined to be .I idle when there are no messages left to be sent over it, .I and there are no outstanding transactions, either to the remote site or from the remote site. There is an outstanding transaction to a remote site when the reply from that site has not yet been received. There is an outstanding transaction from a remote site when a reply for the remote request has not yet been sent. .PP The switch can test whether a connection is idle by checking whether the queue is empty, and whether the number of incoming requests minus outgoing replies is zero. At regular intervals the switch increments a counter variable associated with the connection when the idle test succeeds. When this variable exceeds some critical value, the switch will break the connection. The variable is reset whenever a message is sent over the connection. .PP The times at which the switch tests whether a connection is idle are determined by the arrival of sweep requests. These are sent at regular intervals by a special process in the gateway, called the .I sweep stub. .NH 2 The Server Stub .PP A server stub is a server process that accepts requests for a remote server. There is at least one stub per remote server, because the .AM primitives do not allow specifying more than one port. Soon this may change however (see the Saper called "Connecting Amoebas Using Gateways"). More that one stub may exist for one remote server to allow for multiple clients using the same remote structure. .PP Server stubs are created by the gateway switch when the switch receives an INSTALL request. A capability for the gateway switch is sent over the pipe between the stub and the switch to enable the stub to send requests to the switch. Now the server stub goes through a loop as depicted in Fig. 5, getting requests, processing them, and sending back replies. .sp .F1 .P[ .nf from_switch(&cap); for ever do getreq(&request); to_big_endian(&request); to_switch(&cap, SEND, &request); from_switch(&reply); to_switch(&cap, DONE, NIL); from_big_endian(&reply); putrep(&reply); done .fi .P] .F2 .ce \s-2\fBFig. 5.\fR Operation of a Server Stub.\s0 .F3 .PP Before a request message is sent to the gateway switch, the stub converts the .AM header in the message to a defined word order, namely .I "big endian" (lowest numbered byte is most significant). Similarly, when it gets a reply message from the switch, it converts the header to the native byte order before it is sent to the client. This way the wide-area network byte order is defined, and there are no byte order problems in the .AM headers, like it is in the local case. Of course, this does not solve the byte order problem in the data buffer of the messages, but fortunately these are usually ascii strings. .NH 2 The Client Stub .PP The client stub gets request messages over the pipe from the gateway switch, and does transactions using these requests, as shown in Fig. 6. Like the server stub it converts messages to and from big endian to solve the byte order problem for .AM headers. Furthermore, it automatically generates an error reply message when a transaction fails to notify the remote client. .F1 .P[ .nf from_switch(&cap); for ever do from_switch(&request); to_switch(&cap, DONE, NIL); if (fork() == CHILD) then from_big_endian(&request); if (trans(&request, &reply) == FAIL) then reply.status = ERROR; fi to_big_endian(&reply); to_switch(&cap, SEND, &reply) fi done .fi .P] .F2 .ce \s-2\fBFig. 6.\fR Operation of a Client Stub.\s0 .F3 .PP Although, as far as the gateway switch is concerned, there is only one client stub, this stub consists of several processes enabling multiple simultaneous transactions. For this purpose, the client stub forks off a process for every transaction it has to perform. .NH 2 The Network Stub .PP This stub manages the connection over wide-area networks. The gateway allows several types of network stubs, one per type of network service. A difficulty in implementing these stubs is that messages come from two sides, the gateway switch and the network, and the stub might want to wait for both sides at the same time. .F1 .P[ .nf from_switch(&cap); for ever do from_anywhere(&request); switch (request.type) case OPEN: open_connection(request); case CLOSE: close_connection(); case DATA: if (request.source == NETWORK) to_switch(&cap, SEND, &request); else /* source == SWITCH */ to_switch(&cap, DONE, NIL); to_network(&request); fi end_switch done .fi .P] .F2 .ce \s-2\fBFig. 7.\fR Operation of a Network Stub.\s0 .F3 .PP The task of the network stub is mainly to pass messages between gateway switch and wide-area network, as shown in Fig. 7. Additional tasks are setting up and closing down connections. The different types of stubs originate from the different network driver libraries used by the stubs. .NH USAGE .PP Now we have designed and implemented a gateway service for wide-area transactions, we will describe how the service can be used to build a wide-area distributed system. This section consists of two parts, one suggesting how to do the management of the gateway servers, the other giving advice on the implementation of global services. .NH 2 Management .PP The gateway service as described offers a raw, low level service, that higher level services can use in building a user-friendly basis for making a wide-area distributed system. The high-level services we identify in this section are port distribution, monitoring, and accounting. .NH 3 Port Distribution .PP On local .AM systems, clients locate services by broadcasting locate requests over the local network. The node that is running the service will respond with its node address. Clearly, this scheme is not possible on the scale of international networks, for they do not provide an efficient, cheap, broadcasting service. Even if it would, the enormous amounts of global locate requests would increase the load on all the machines on the network. .PP Therefore a different port location method has to be used on wide-area networks. Our gateway service allows the registration of remote ports on local gateways. Thus if a client is locating a known remote port, the gateway answers on behalf of the remote server. We would like to have a service for making ports ``known'' at the sites where the service is to be offered. Such a collection of sites is called a .I domain , and making a port known is called .I publishing . .PP This service could be implemented by having a .I "port publishing" server running at each .AM site, each one with its own unique port that is known at every other site. Each port publishing server accepts PUBLISH and INSTALL requests: .IP PUBLISH: 12 A local .AM server wants to have its port installed at all the gateways in the specified domain. .IP INSTALL: A remote port publishing server wants to install a port at the local gateway. .LP When an .AM server wants to make itself known in a domain, it send a PUBLISH request to the local port publishing server, which will send INSTALL requests to each port publishing server in the specified domain. .PP A disadvantage of this method is that every port publishing server in the known domains has to be registered at the gateway service. Furthermore, creating, changing, and deleting domains is difficult, because the domains are large and the management is distributed over the complete wide-area network. .PP This can be resolved by having separate .I "domain managers" . The ports of the domain managers are known at every site. PUBLISH requests are sent to the appropriate domain manager, which will send install requests to all the sites in the domain it manages. This scheme can easily be made hierarchical, to allow large domains being split up into smaller ones, making the management of domains more efficient and flexible. .NH 3 Monitoring .PP An important feature of the gateway server is the generation of logs of its operation. The amount of logging can be changed by setting the .I "log level" of the server: the higher the log level, the more events are logged. The gateway logs arrived and sent messages, connection set up, errors, and, if the log level is high enough, the low level network protocol. .PP An interesting service for the management of a wide-area distributed system is the automatic collection of the log files from the different gateways, filtering out the information that is actually required. Such information could be used to find bottlenecks in the network, determine the availability and efficiency of wide-area .AM services, or find bugs in the system. .PP There are several approaches to setting up such a service. For example, it could consist of monitoring servers at each site, accepting requests to give the logging information according to some conditions, like: ``Give me the number of messages sent to server X during the last week.'' Finding what conditions should be supported in these queries will make an interesting research project. .NH 3 Accounting .PP A wide-area connection is expensive. So are processing cycles and resources. Many managers of .AM sites would like to know which local users use the network, and which remote sites are using local resources. Furthermore, they would like to know which local users are responsible for the bills the .AM site gets from remote sites, for using their resources. What is needed is an accounting service that keeps track of who used which remote resources, and the amount of network traffic they used for accessing the resources. .PP There are two principal ways of setting up such a service: it could keep records of who has done what, and send the bills afterwards, or it could charge the users before doing the requested operations. Then there is the problem of whom is to be charged for a transaction: the client or the server. Again, this has to be divided into communication and resource costs. .PP However the users are charged, the service has to authenticate the users first. This can be done using the digital signature field in the .AM header; the capability for objects give no clue about the identity of clients. Since it is impossible to make each user known at each site, gateways should act as one client to the remote gateways. The local gateway could then keep track of the local users that are responsible for the communication and service costs. .sp 1 .NH 2 Wide-Area Services .PP The gateway should be a non-service, completely transparent to .AM users. However, we were not able to do this for several reasons: .IP \(bu automatic port location is inefficient in a wide-area network; .IP \(bu To keep the total buffer size in our system down, and because of limitations in .UX pipes, we had to limit the .AM message buffers to 2048 bytes; .IP \(bu wide-area communication is expensive and should be accounted for; .IP \(bu the usage of remote resources should be accounted for. .sp .LP In this section we propose how to use the gateway service to obtain optimal performance. First, however, we identify some wide-area applications: .IP \(bu mail service; .IP \(bu file transfer; .IP \(bu remote execution; .IP \(bu distributed database; .IP \(bu banking system; .IP \(bu airline reservation system; .IP \(bu weather forecast; .IP \(bu distributed processing. .LP This last point may require some extra attention. .PP Distributed processing is a feasible application, as long as the communication overhead is low compared to the processing. In wide-area networks this means that it is only worthwhile to execute a process at a remote site when it will run there for a while without exchanging messages with the local site. For example, this might happen if there are several independent jobs that require considerable computation. .PP In .AM systems it is not unusual for applications to generate new ports on the fly. On local .AM systems this does not present any difficulties, because ports can be found by broadcasting locate requests. Using our gateway service, however, it is more difficult. A newly generated port will have to be published in the domain where it is used, generating considerable overhead and consuming many resources (a server stub at each site). Furthermore, the lifetime of such a port is usually short, and it will have to be removed when it is not used any more. .PP The number of published ports should be kept down to a minimum for the same reason. This means that generally the wide-area services have to be more powerful than in the local case, offering more operations on a larger variety of objects. For example, if the mail service, file transfer service, and the remote execution service are merged into one larger service, it consume only one server stub at each site, instead of three. .PP Since the gateway server keeps a virtual circuit open as long as there are outstanding transactions over the connection, it is undesirable to have long transactions. First, because one has to pay for connections even if they are not used, and second, because they consume network resources that can be used for other transactions. Service time should therefore be kept low, that is, less than a few minutes, which is the usual time for unused connections to be closed. .NH CONCLUSIONS .PP Wide-area transactions are feasible, and need not be difficult to implement. We did so using existing local and wide area network services, by implementing a small message switching service. We suggested how the service can be made into a high-level service by adding port distribution mechanisms. Furthermore, by adding monitoring and accounting the wide-area .AM service can be made into a manageable system. .NH ACKNOWLEDGEMENTS .PP We would like to thank Sape Mullender for suggesting a high-level gateway service, and Jennifer Steiner for the careful reading of our document.