Version 3.0
+Sky DeBaun, Jet Propulsion Laboratory, California Institute of +Technology
+Document Change Log +| Ver No. | Date | Affected | Description| Comments | +| ---- | --- | --- | --- |--- | +| 2.2 | Sept 2010 | | | | +| 3.0 | June 2023 | All | Updates and Corrections | |
+Table of Content +- AMS Programmer's Guide +- The AMS Protocol + - Purpose and Scope + - Definitions + - Overview + - General + - Architectural Elements + - Overview of Interactions +- The JPL Implementation +- Primary Transport Services +- Installation + - Automake + - Make + - Support for the Expat XML Parsing Library +- The AMS Daemon +- "C" Application Programming Interface + - Type and Macro Definitions + - Module Management functions + - Message Subscription and Invitation + - Configuration Lookup + - Message Issuance + - Event (Including Message) Reception + - User Event Posting +- Remote AMS + - Library + - ramsgate +- Management Information Base + - MIB file syntax + - A Sample MIB +- Application Development + - Overview + - Compiling Custom AMS Applications + - Running the Application +- A Sample Application +- Acknowledgment
+++The Consultative Committee for Space Data Systems\' (CCSDS) +Asynchronous Message Service (AMS) is a communication architecture for +data systems. It is designed to allow mission system modules to +operate as if they were isolated, each producing and consuming mission +information without explicit knowledge of the other active modules. +This self-configuring communication relationship minimizes complexity +in the development and operation of modular data systems.
+AMS is the foundation of a system that can be described as a +\'society\' of largely autonomous, interoperating modules. These +modules can adapt over time in response to changing mission +objectives, functional upgrades of modules, and recovery from +individual module failures. The primary objective of AMS is to reduce +mission cost and risk by providing a standard, reusable infrastructure +for information exchange among data system modules. This +infrastructure is designed to be user-friendly, highly automated, +flexible, robust, scalable, and efficient.
+Notably, AMS provides a publication and subscription service for both +terrestrial and extraterrestrial communications, utilizing the +Interplanetary Overlay Network (ION). This service ensures a seamless +and efficient communication system that can adapt dynamically to +various missions.
+
++Within the context of this document the following definitions apply:
+A continuum is a closed set of entities that utilize AMS for +purposes of communication among themselves. Each continuum is +identified by a continuum name and corresponding non-negative +continuum number. The continuum name that is the character string of +length zero indicates "all known continua" or "any known continuum", +whichever is less restrictive in the context in which this continuum +name is used; the reserved continuum number zero corresponds to this +continuum name.
+An application is a data system implementation, typically taking the +form of a set of source code text files, that relies on AMS procedures +to accomplish its purposes. Each application is identified by an +application name.
+An authority is an administrative entity or persona that may have +responsibility for the configuration and operation of an application. +Each authority is identified by an authority name.
+A venture is an instance of an application, i.e., a functioning +projection of the application -- for which some authority is +responsible -- onto a set of one or more running computers.
+A message is an octet array of known size which, when copied from +the memory of one module of a venture to that of another +(exchanged), conveys information that can further the purposes of +that venture.
+The content of a message is the array of zero or more octets +embedded in the message containing the specific information that the +message conveys.
+A role is some part of the functionality of an application. Each +role is identified by a role name and corresponding non-negative +role number. The role name that is the character string of length zero +indicates 'all roles' or 'any role', whichever is less restrictive in +the context in which the role name is used; the reserved role number +zero corresponds to this role name. The role name "RAMS \'\' +identifies Remote AMS (RAMS) gateway functionality as discussed below; +the reserved role number 1 corresponds to this role name.
+A module (of some mission data system) is a communicating entity +that implements some part of the functionality of some AMS venture -- +that is, performs some application role -- by, among other activities, +exchanging messages with other modules. Associated with each module is +the name of the role it performs within the application. [Note that +multiple modules may perform the same role in an application, so the +role name of a module need not uniquely identify the module within its +message space.] In order to accomplish AMS message exchange a module +generates AMS service requests and consumes AMS service indications; +the module that is the origin of a given AMS service request or the +destination of a given AMS service indication is termed the operative +module.
+A message space is the set of all of the modules of one AMS venture +that are members of a single AMS continuum; that is, a message space +is the intersection of a venture and a continuum. Each message space +is uniquely identified within that continuum by the combination of the +name of the application and the name of the authority that is +responsible for the venture, and by a corresponding venture number +greater than zero. [Note that unique naming of continua enables +multiple message spaces that are in different continua but are +identified by the same application and authority names to be +concatenated via Remote AMS (discussed below) into a single venture.]
+A unit (i.e., a unit of organization) is an identified subset of the +organizational hierarchy of the modules of one AMS venture, declared +during venture configuration as specified by the responsible authority +for that venture. Each unit is uniquely identified within the venture +by unit name and corresponding non-negative unit number. The root +unit of a venture is the unit that is coterminous with the venture +itself; its unit name is the character string that is of length zero, +and the reserved unit number zero corresponds to this unit name. A +unit whose name is identical to the first N bytes -- where N is +greater than or equal to zero -- of the name of another unit of the +same message space is said to contain that other unit. The +membership of a unit that is contained by another unit is a subset of +the membership of the containing unit.
+
A cell is the set of all modules that are members of one unit of a +given venture and are also members of a given continuum; that is, it is +the intersection of a unit and a continuum. Since each unit is a subset +of a venture, each cell is necessarily a subset of the message space for +that venture in that continuum. Each cell is uniquely identified within +its message space by its unit's name and number. The root cell of a +message space is coterminous with the message space itself. A cell +contains some other cell only if its unit contains that other cell's +unit. A cell may be an empty set; that is, in a given continuum there +may be no modules that are members of the cell's unit. The registered +membership of a cell is the set of all modules in the cell that are not +members of any other cell which does not contain that cell^1^. [Note +that the root cell contains every other cell in the message space, and +every module in the message space is therefore a member -- though not +necessarily a registered member -- of the root cell.]
+++The domain of an AMS service request is the set of modules to which +the request pertains. It comprises all of the modules that are members +of the venture in which the operative module is itself a member, with +the following exceptions:
+
++The subject number (or subject) of a message is an integer +embedded in the message that indicates the general nature of the +information the message conveys, in the context of the AMS venture +within which the message is exchanged. A subject name is a text +string that serves as the symbolic representation of some subject +number.
+To send a message is to cause it to be copied to the memory of a +specified module. To publish a message on a specified subject is to +cause it to be sent to one or more implicitly specified modules, +namely, all those that have requested copies of all messages on the +specified subject. To announce a message is to cause it to be sent +to one or more implicitly specified modules, namely, all those modules +that are located within a specified continuum (or all continua), are +members of a specified unit (possibly the root unit) and that perform +a specified role in the application (possibly "any role").
+A subscription is a statement requesting that one copy of every +message published on some specified subject by any module in the +subscription's domain be sent to the subscribing module; the domain +of a subscription is the domain of the AMS service request that +established the subscription.
+^1^ For example, if cell A contains cells B and C, and cell C contains +cells D and E, any nodes in C that are not in either D or E are in the +registered membership of cell C. Those nodes are also members of cell +A, but because they are in cell C -- which does not contain cell A -- +they are not in cell A's registered membership.
+An invitation is a statement of the manner in which messages on some +specified subject may be sent to the inviting module by modules in the +domain of the invitation; the invitation's domain is the domain of +the AMS service request that established the invitation.
+
++A data system based on AMS has the following characteristics:
+
a. Any module may be introduced into the system at any time. That is, + the order in which system modules commence operation is immaterial; + a module never needs to establish an explicit a priori + communication "connection" or "channel" to any other module in order + to pass messages to it or receive messages from it.
+b. Any module may be removed from the system at any time without + inhibiting the ability of any other module to continue sending and + receiving messages. That is, the termination of any module, whether + planned or unplanned, only causes the termination of other modules + that have been specifically designed to terminate in this event.
+c. When a module must be upgraded to an improved version, it may be + terminated and its replacement may be started at any time; there is + no need to interrupt operations of the system as a whole.
+d. When the system as a whole must terminate, the order in which the + system's modules cease operation is immaterial.
+++AMS-based systems are highly robust, lacking any innate single point +of failure and tolerant of unplanned module termination. At the same +time, communication within an AMS-based system can be rapid and +efficient:
+
e. Messages are exchanged directly between modules rather than through + any central message dispatching nexus.
+f. Messages are automatically conveyed using the "best" (typically -- + though not necessarily -- the fastest) underlying transport service + to which the sending and receiving modules both have access. For + example, messages between two ground system modules running in + different computers on a common LAN would likely be conveyed via + TCP/IP, while messages between modules running on two flight + processors connected to a common bus memory board might be conveyed + via a shared-memory message queue.
+g. Finally, AMS is designed to be highly scalable: partitioning message + spaces into units enables a venture to comprise hundreds or + thousands of cooperating modules without significant impact on + application performance.
+ +++AMS message exchange is fundamentally asynchronous, akin to a +\"postal\" system. An AMS module, after sending a message, can +continue its functions without waiting for a reply.
+While message exchange is asynchronous, AMS provides a mechanism for +linking reply messages to their original context. This is achieved by +including a context number in the original message. The reply message +automatically echoes this context number, allowing the original sender +to link the reply to the application activity that triggered the +initial message. This creates a pseudo-synchronous communication flow. +The specific mechanism for establishing this link is +implementation-dependent.
+In some cases, true message synchrony may be necessary, requiring a +module to suspend operations until a reply is received. AMS supports +this communication model when required.
+The majority of message exchange in an AMS-based system follows a +\"publish-subscribe\" model. A module announces its subscription to a +specific subject using AMS procedures. From that point, any published +message on that subject is automatically delivered to all subscribing +modules. This model simplifies application development and +integration, allowing each module to plug into a data \"grid\" and +exchange data without detailed knowledge of other modules.
+However, there may be instances where a module needs to send a message +privately to a specific module, such as in reply to a published +message. AMS also supports this communication model when necessary.
+
++The architectural elements involved in the asynchronous message +service protocol are depicted in Figure 1 and +described below.
+[]{#_heading=h.3dy6vkm .anchor}**
+Figure 1: Architectural Elements of AMS
+**
+
++All AMS communication is conducted among three types of communicating +entities: modules (defined earlier), registrars, and configuration +servers.
+A registrar is a communicating entity that catalogs information +regarding the registered membership of a single unit of a message +space. It responds to queries for this information, and it updates +this information as changes are announced.
+A configuration server is a communicating entity that catalogs +information regarding the message spaces established within some AMS +continuum, specifically the locations of the registrars of all units +of all message spaces. It responds to queries for this information, +and it updates this information as changes are announced.
+
++AMS, best characterized as a messaging \"middleware\" protocol, +operates between the Transport and Application layers of the OSI +protocol stack model. It relies on underlying Transport-layer +protocols for actual message copying from sender to receiver and for +transmitting meta-AMS (or MAMS) messages for dynamic +self-configuration of AMS message spaces.
+In any AMS continuum, a common transport service, termed the Primary +Transport Service (PTS), is used for MAMS traffic by all entities +involved in the operations of all message spaces. The PTS, being +universally available, can also be used for application message +exchange among all modules in a continuum. However, in some cases, +performance can be improved by using Supplementary Transport Services +(STSs), especially when modules share access to a convenient +communication medium like a shared-memory message queue.
+Supplementary Transport Services (STSs) are performance-optimizing +transport services used in the Asynchronous Message Service (AMS) for +message exchange between modules that share access to a particularly +convenient communication medium, such as a shared-memory message +queue. While the Primary Transport Service (PTS) is universally +available for message exchange, STSs can be employed to enhance +application performance in certain scenarios (see CCSDS Blue Book +Recommended Standard 735.1-B-1 \"Asynchronous Message Service\" for +additional information).
+A module\'s network location for receiving messages via a given +transport service is its delivery point for that service. A module may +have multiple delivery points, each characterized by the same service +mode. For a given service mode, the list of all delivery points +providing that mode to a module, ranked in descending order of +preference (typically network performance), is termed the delivery +vector for that service mode, for that module.
+See "Primary Transport Services" below for additional information.
+
++Every message space in AMS always includes at least one unit, the root +unit, and each module is registered within a unit. In the simplest +case, all modules reside in the root unit. Each unit is served by a +single registrar, which monitors the health of all registered modules +and propagates six types of message space configuration changes.
+Registrars themselves register with the configuration server for the +continuum containing the message space. A list of all potential +network locations for the configuration server, ranked in descending +order of preference, must be well-known and included in the AMS +management information bases (MIBs) accessible to all registrars. Each +continuum must always have an operational configuration server at one +of these locations to enable registration of registrars and modules.
+All registrars and modules of the same message space must register +through the same configuration server.
+
++Each module has a single meta-AMS delivery point (MAPD) for receiving +MAMS messages. A new module joins a message space by registering +within a unit, announcing its role name and MAPD to the unit\'s +registrar. However, the module cannot have hard-coded information +about the registrar\'s communication details, as these can change.
+Therefore, the first step in registering a new module is contacting +the configuration server at one of its known network locations. These +locations, listed in descending order of preference, are included in +the AMS Management Information Bases (MIBs) accessible to all +application modules. The configuration server then provides the new +module with the contact details for its registrar.
+The module obtains a unique module number from the registrar and +completes registration. The registrar ensures that all other modules +in the message space learn the new module\'s role name, module number, +and MAPD. These modules, in turn, announce their own details to the +new module.
+
++Maintaining accurate knowledge of a message space configuration is +crucial for application purposes and resource efficiency. Each +registrar must promptly detect the termination of modules in its +unit\'s registered membership. While a module under application +control notifies its registrar upon termination, a module that crashes +or is powered off does not. To address this, each module sends a +\"heartbeat\" message to its registrar every few seconds (see comment
+3 at top of amscommon.h for additional details). The registrar
+interprets three consecutive missing heartbeats as a module +termination.
+Upon detecting a module\'s termination, either overt or imputed from +heartbeat failure, the registrar informs all other modules in the +unit\'s registered membership and, through other registrars, all +modules in the message space.
+When termination is imputed from a heartbeat failure, the registrar +attempts to notify the presumed terminated module. If the module is +still running, it terminates immediately upon receiving this message, +minimizing system confusion due to other application behavior +triggered by the imputed termination.
+
++Each registrar not only monitors the heartbeats of all modules in its +unit\'s registered membership but also issues its own heartbeats to +those modules. If a module detects three consecutive missing registrar +heartbeats, it assumes the registrar has crashed. The module then +re-queries the configuration server to determine the new network +location of the registrar and resumes exchanging heartbeats.
+This assumption is reasonable because the configuration server also +monitors registrar heartbeats on a slightly shorter cycle. If the +configuration server detects three consecutive missing registrar +heartbeats, it takes action to restart the registrar, possibly on a +different host. Therefore, by the time the registrar\'s modules detect +its crash, it should already be running again.
+Since the module heartbeat interval is two seconds (see N4 in +amscommon.h), the registrar will receive heartbeat messages from every +running module in the unit\'s registered membership within the first +six seconds after restart. This allows the registrar to accurately +know the unit\'s configuration. This accurate configuration +information must be delivered to new modules at startup, enabling them +to orient a newly-restarted registrar if it crashes. Therefore, during +the first six seconds after the registrar starts, it only accepts MAMS +messages from modules already registered in the unit. This prevents +the risk of delivering incorrect information to a new module.
+
++A configuration server, like any other component, can also fail or be +rebooted. Each registrar interprets three consecutive missing +configuration server heartbeats as an indication of a crash. Upon +detecting such a crash, the registrar cycles through all the known +network locations for the continuum's configuration server, attempting +to re-establish communication after the server's restart, possibly at +an alternate network location. New modules attempting to register will +also cycle through network locations seeking a restarted configuration +server and will be unable to contact their registrars, and therefore +unable to register, until they find one. However, application message +exchange and subscription management activity among existing modules +and registrars are not affected by this infrastructure failure.
+Upon the configuration server\'s restart at one of its known network +locations, all registrars will eventually find it and re-announce +themselves, enabling newly registering application modules to +successfully register.
+In certain failure scenarios, multiple configuration servers may +operate concurrently for a brief period, such as when a perceived +failure is caused by a transient network connectivity issue rather +than an actual server crash. To resolve this, each running +configuration server periodically sends an "I am running" MAMS message +to every lower-ranking configuration server network location in the +known list of configuration server locations. If a configuration +server receives such a message, it immediately terminates. All +registrars and modules communicating with it will detect its +disappearance and search again for the highest-ranking reachable +configuration server, eventually restoring orderly operations in the +continuum.
+
++Finally, every registrar can optionally be configured to re-advertise +to the entire message space the detailed configuration of its unit's +registered membership (all active modules, all subscriptions and +invitations) at some user-specified frequency, e.g., once per minute. +This capability is referred to as configuration resync. +Configuration resync of course generates additional message traffic, +and it may be unnecessary in extremely simple or extremely stable +operating environments. But it does ensure that every change in +application message space configuration will eventually be propagated +to every module in the message space, even if some MAMS messages are +lost and even if an arbitrary number of registrars had crashed at the +time the change occurred.
+Taken together, these measures make AMS applications relatively fault +tolerant:
+
a. When a module crashes, its registrar detects the loss of heartbeat + within three heartbeat intervals and notifies the rest of the + message space. Application message transmission everywhere is + unaffected.When a registrar crashes, its configuration server + detects the loss of heartbeat within three heartbeat intervals and + takes action to restart the registrar. During the time that the unit + has no registrar, transmission of application messages among modules + of the message space is unaffected, but the heartbeat failures of + crashed modules are not detected and reconfiguration messages + originating in the unit's registered membership (registrations, + terminations, subscription and invitation assertions, and + subscription and invitation cancellations) are not propagated to any + modules. However, after the registrar is restarted it will + eventually detect the losses of heartbeat from all crashed modules + and will issue obituaries to the message space, and if configuration + resync is enabled it will eventually re- propagate the lost + reconfiguration messages.
+b. When a configuration server crashes, all new registration activity + will come to a standstill. But no application modules fail (at + least, not because of communication failure), and on restart of the + configuration server the registration of new modules eventually + resumes.
+ +++AMS can be configured to confine service access to application modules +that can prove they are authorized to participate. For this purpose, +asymmetric MAMS encryption may be used as follows:
+
a. The AMS MIB exposed to the configuration server contains a list of + all applications for which registration service may be offered, + identified by application name. Associated with each application + name is the AMS public encryption key for that application.
+b. The AMS MIB exposed to every registrar in each message space + contains a list of all functional role names defined for the message + space's application; this list limits the role names under which + modules may register in that message space. Associated with each + role name is the AMS public encryption key for the application + module(s) that may register in that role.
+c. The AMS MIBs exposed to all registrars and application modules in + the message space contain the AMS public encryption key of the + configuration server.
+d. The AMS MIBs exposed to the configuration server and to all + registrars and application modules in the message space contain the + private encryption keys that are relevant to those entities.
+++As described later, this information is used to authenticate registrar +registration and exclude spurious registrars from the message space, +to authenticate module registration attempts and deny registration to +unauthorized application modules, and to assure the authenticity, +confidentiality, and integrity of MAMS traffic exchanged between +modules and their registrars.
+In addition, the confidentiality and integrity of AMS message exchange +may be protected at subject granularity. The AMS MIB exposed to each +module of a given message space may contain, for any subset of the +message subjects (identified by name and number) used in the message +space's application:
+
e. a list of the role names of all modules that are authorized senders + of messages on this subject;
+f. a list of the role names of all modules that are authorized + receivers of messages on this subject;
+g. encryption parameters, including a symmetric encryption key, + enabling encryption of messages on this subject.
+++This information may be used to support secure transmission of +messages on selected subjects.
+Note*, though, that the JPL implementation of AMS does not +implement* any of the cryptographic algorithms that are required to +support these security features.
+
++The structure of the content of messages on a given subject is +application-specific; message content structure is not defined by the +AMS protocol. However, the AMS MIB exposed to all modules of a given +message space will contain, for each message subject (identified by +name and number) used in the message space:
+
a. a description of this message subject, discussing the semantics of + this type of message;
+b. a detailed specification of the structure of the content of messages + on this subject;
+c. optionally, a specification of the manner in which a correctly + assembled message is marshaled for network transmission in a + platform-neutral manner and, on reception, un-marshaled into a + format that is suitable for processing by the application.
+++When AMS is requested to send a message on a given subject, the +message content that is presented for transmission is always in a +format that is suitable for processing by the application. In the +event that this format is not suitable for network transmission in a +platform-neutral manner, as indicated by the presence in the MIB of a +marshaling specification for this subject, AMS will marshal the +message content as required before transmitting the message.
+When AMS receives a message on a subject for which a marshaling +specification is present in the MIB, AMS will un-marshal the message +content into a format that is suitable for processing by the +application before delivering the message.
+Message subjects, as noted earlier, are integers with +application-defined semantics. This minimizes the cost of including +subject information (in effect, message type) in every message, and it +makes processing simpler and faster: subscription and invitation +information are recorded in arrays that are indexed by subject number.
+This implementation choice, however, requires that message management +control arrays be large enough to accommodate the largest subject +numbers used in the application. The use of extremely large subject +numbers would therefore cause these arrays to consume significant +amounts of memory. In general, it is best for an AMS application to +use the smallest subject numbers possible, starting with 1.
+
++AMS' asynchronous message issuance model allows for a high degree of +concurrency in the operations of data system modules. This means that +a module can issue a message without suspending its operation until a +response is received. This feature also largely insulates applications +from variations in signal propagation time across the AMS continuum.
+However, some critical MAMS (Multicast AMS) communication is +unavoidably synchronous. For instance, a newly registering module must +wait for responses from the configuration server and the registrar +before it can proceed with application activity. Therefore, the core +AMS protocol is best suited for operational contexts with generally +uninterrupted network connectivity and relatively small and +predictable signal propagation times, such as the Internet or a +stand-alone local area network. It is typically advantageous for all +entities within a single AMS continuum to operate within such a +\"low-latency\" environment
+AMS application messages can be exchanged between modules in different +AMS continua using the Remote AMS (RAMS) procedures. These procedures +are executed by special-purpose application modules known as RAMS +gateways. Each RAMS gateway interfaces with two communication +environments: the AMS message space it serves and the RAMS network, +which is a mesh or tree of mutually aware RAMS gateways. This network +enables AMS messages produced in one message space to be forwarded to +other message spaces within the same venture.. RAMS gateways operate +as follows:
+
a. RAMS gateways operate by opening private RAMS network communication + channels to the RAMS gateways of other message spaces within the + same venture. These interconnected gateways use these communication + channels to forward message petition assertions and cancellations + among themselves.
+b. Each RAMS gateway subscribes locally to all subjects that are of + interest in any of the linked message spaces.
+c. When a RAMS gateway receives a message on any of these subjects, it + uses the RAMS network to forward the message to every other linked + RAMS gateway whose message space contains at least one module that + has subscribed to messages on that subject.
+d. On receiving a message the RAMS gateway module forwards the message + to any subscribers in its own message space.
+++The RAMS protocol allows for the free flow of published application +messages across deep space links while ensuring efficient utilization +of those links. Only a single copy of any message is ever transmitted +on any RAMS grid communication channel, regardless of how many +subscribers will receive copies when the message reaches its +destination.
+RAMS operations generalize the AMS architecture as shown in Figure 2 +below.
+
{width="6.23578302712161e-3in"
+height="1.9860017497812773e-3in"}
++Figure 2 General AMS application structure
+This extension of the publish/subscribe model to inter-continuum +communications is invisible to application modules. Application +functionality is unaffected by these details of network configuration, +and the only effects on behavior being those intrinsic to variability +in message propagation latency.
+It\'s important to note that the nature of the RAMS network +communication channels depends on the implementation of the RAMS +network. To communicate over the RAMS network for a given venture, +each RAMS gateway must know the RAMS network location, expressed as an +endpoint in the protocol used to implement the RAMS network.
+Also, only AMS application messages are propagated across continuum +boundaries by RAMS. Modules are never notified of registrations, +subscriptions, and invitations that occur in remote continua. The +purpose of RAMS is to limit traffic on the scarce link resources +supporting inter-continuum communication to the minimum necessary for +successful operation of the venture. MAMS message traffic within a +message space is required to enable the operation of the message +space, but venture-wide application message exchange can readily be +provided without propagating MAMS messages to remote continua.
+
++JPL's implementation of AMS has the following components:
+
++The codebase, written in C, relies on a shared library, ICI. This +library supports other JPL implementations, like CFDP and the DTN +Bundle Protocol. ICI includes a \"platform\" portability layer, easing +code compilation and execution in environments like Linux, vxWorks, +and Interix.
+ICI also includes its own dynamic memory management system, called +"PSM", which provides dynamic management of a privately allocated +block of memory. This may be useful in environments such as spacecraft +flight software where the dynamic management of system memory +(malloc, free) cannot be tolerated. Use of PSM by AMS is optional.
+An AMS application program, linked with libams, uses the +ams_register function to instantiate an AMS module registered within +a specified unit of a specified message space. Once registration is +accomplished, the application may commence inviting, subscribing to, +publishing, announcing, sending, and replying to messages.
+This AMS implementation is multi-threaded. The process of registration +starts a pair of POSIX threads, or pthreads, which manage timing and +MAMS events in the background. Additionally, another pthread is +started to receive MAMS messages via the primary transport service and +add them to the MAMS event queue. This queue also includes MAMS +message transmission requests. For each transport service that the +module can receive AMS messages from, one more pthread is started. +These threads receive AMS messages and add them to the AMS event +queue, combining them with \"discovery\" events added by the MAMS +event handling thread.
+The general structure of an AMS module, then, is as shown in Figure +3 below.
+
{width="6.444444444444445in"
+height="3.4027777777777777in"}
++[]{#_heading=h.2s8eyo1 .anchor}Figure 3 AMS module structure
+The application program has the option to start another thread to +manage AMS events. This thread automatically calls event-type-specific +callback functions, leaving the main application thread free to +respond to non-AMS events, such as mouse events or keyboard input. The +application code can also add application-specific events to the AMS +event queue, potentially with higher priority than any queued AMS +messages. However, to prevent certain types of unusual application +behavior, the main application thread is not allowed to receive and +handle any AMS events while the background AMS event handling thread +is running.
+
++As shipped, AMS currently includes support for two underlying +transport services: TCP, and DGR (Datagram Retransmission, a UDP-based +system that includes congestion control and retransmission-based +reliability). Although TCP is faster than DGR, its connection-based +architecture makes it unsuitable as a primary transport service: all +MAMS message traffic is conveyed via connectionless DGR.
+
AMS source is provided in the ION distribution (a gzipped tarfile +containing AMS and all supporting ION packages: ici, dgr, ltp, and bp, +etc.).
+The following two installation methods are provided.
+This method automatically compiles and links all required executables, +installs them, and copies ION library headers to the relevant system +path(s) on your system.Use the following command sequence in the unzipped ION source directory +(Linux).
+Note: if support for the expat XML parsing library is required see 4.3 +"Support for the Expat XML Parsing Library" below
+++This alternate installation method installs all ION packages (if run +from ION root directory), or installs individual ION packages as +follows
+
++Note that for both install methods (e.g. on Linux) the default +configuration used in the ION makefiles is as follows:
+
++If you want a different configuration, you'll need to modify the +makefiles accordingly (e.g. see the OPT variable in 2b above).
+
++The expat open-source XML parsing library is required by AMS only +if MIBs use the XML format (see man amsxml and man amsrc for +additional information).
+Note that Linux environments typically have expat built in, but +for VxWorks installations it is necessary to download and install +expat prior to installing AMS.
+To build ION with support for expat use the following flag during the +.configure step of installation (see 4.1 "Automake" above):
+./configure --with-expat
+
++The AMS daemon program amsd can function as the configuration +server for a continuum, as the registrar for one cell of a specified +message space, or both. To run it, enter a command of the following +form at a terminal window prompt:
+amsd mib_source_name eid_spec
+or
+amsd mib_source_name eid_spec application_name authority_name +unit_name
+The former form of the command starts amsd as a configuration +server only.
+mib_source_name is as discussed in the documentation of ams_register +below; it enables amsd to run.
+eid_spec is a string that specifies the IP address and port that +amsd must establish in order to receive MAMS messages in its +capacity as a configuration server. See man amsd for more +information.
+When the latter form of the amsd command is used, the daemon is +configured to function as the registrar for the indicated message +space unit. If the value "." (period character) is supplied for +eid_spec, then the daemon will [only]{.underline} function as a +registrar. Otherwise the daemon will function as both configuration +server and registrar; this option can be useful when operating a +simple, stand-alone message space, such as a demo.
+
++The AMS application programming interface is defined by the header +file ams.h, which must be #included at the beginning of any +AMS application program source file.
+See section 9 'Application Development Guide' for compilation and +linkage instructions.
+
++define THIS_CONTINUUM (-1)
+define ALL_CONTINUA (0)
+define ANY_CONTINUUM (0)
+define ALL_SUBJECTS (0)
+define ANY_SUBJECT (0)
+define ALL_ROLES (0)
+define ANY_ROLE (0)
+typedef enum
+{
+AmsArrivalOrder = 0, AmsTransmissionOrder
+} AmsSequence;
+typedef enum
+{
+AmsBestEffort = 0, AmsAssured
+} AmsDiligence;
+typedef enum
+{
+AmsMsgUnary = 0, AmsMsgQuery, AmsMsgReply, AmsMsgNone
+} AmsMsgType;
+typedef struct amssapst *AmsModule; typedef struct amsevtst +*AmsEvent;
+
+---------------+-----------------------------------------+------------+ +| > /* | > AMS event types. | > */ | ++===============+=========================================+============+ +| > #define | > AMS_MSG_EVT | > 1 | ++---------------+-----------------------------------------+------------+ +| > #define | > TIMEOUT_EVT | > 2 | ++---------------+-----------------------------------------+------------+ +| > #define | > NOTICE_EVT | > 3 | ++---------------+-----------------------------------------+------------+ +| > #define | > USER_DEFINED_EVT | > 4 | ++---------------+-----------------------------------------+------------+
+++typedef enum
+{
+AmsRegistrationState, AmsInvitationState, AmsSubscriptionState
+} AmsStateType;
+typedef enum
+{
+AmsStateBegins = 1, AmsStateEnds
+} AmsChangeType;
+typedef void (*AmsMsgHandler)(AmsModule module, void *userData, +AmsEvent *eventRef, int continuumNbr, int unitNbr,
+int moduleNbr, int subjectNbr,
+int contentLength, char *content,
+int context, AmsMsgType msgType, int priority);
+typedef void (*AmsRegistrationHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int unitNbr,
+int moduleNbr, int roleNbr);
+typedef void (*AmsUnregistrationHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int unitNbr,
+int moduleNbr);
+typedef void (*AmsInvitationHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int unitNbr,
+int moduleNbr,
+int domainRoleNbr,
+int domainContinuumNbr, int domainUnitNbr,
+int subjectNbr, int priority,
+unsigned char flowLabel, AmsSequence sequence, AmsDiligence +diligence);
+typedef void (*AmsDisinvitationHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int unitNbr,
+int moduleNbr,
+int domainRoleNbr,
+int domainContinuumNbr, int domainUnitNbr,
+int subjectNbr);
+typedef void (*AmsSubscriptionHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int unitNbr,
+int moduleNbr,
+int domainRoleNbr,
+int domainContinuumNbr, int domainUnitNbr,
+int subjectNbr, int priority,
+unsigned char flowLabel, AmsSequence sequence, AmsDiligence +diligence);
+typedef void (*AmsUnsubscriptionHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int unitNbr,
+int moduleNbr,
+int domainRoleNbr,
+int domainContinuumNbr, int domainUnitNbr,
+int subjectNbr);
+typedef void (*AmsUserEventHandler)(AmsModule module,
+void *userData, AmsEvent *eventRef, int code,int dataLength, char +*data);
+typedef void (*AmsMgtErrHandler)(void *userData, AmsEvent +*eventRef);
+typedef struct
+{
+AmsMsgHandler msgHandler;
+void *msgHandlerUserData;
+AmsRegistrationHandler registrationHandler;
+void *registrationHandlerUserData; AmsUnregistrationHandler +unregistrationHandler;
+void *unregistrationHandlerUserData;
+AmsInvitationHandler invitationHandler;
+void *invitationHandlerUserData; AmsDisinvitationHandler +disinvitationHandler;
+void *disinvitationHandlerUserData; AmsSubscriptionHandler +subscriptionHandler;
+void *subscriptionHandlerUserData; AmsUnsubscriptionHandler +unsubscriptionHandler;
+void *unsubscriptionHandlerUserData;
+AmsUserEventHandler userEventHandler;
+void *userEventHandlerUserData;
+AmsMgtErrHandler errHandler;
+void *errHandlerUserData;
+} AmsEventMgt;
+/* Predefined term values for ams_query and ams_get_event. */
+define AMS_POLL (0) /* Return immediately. */ #define AMS_BLOCKING
+(-1) /* Wait forever. */
+
++int [ams_register]{.underline}(char *mibSource, char *tsorder, char +*applicationName, char *authorityName, char *unitName, char +*roleName, AmsModule
+*module);
+This function is used to initiate the application's participation as a +module in the message space identified by specified application and +authority names, within the local AMS continuum.
+mibSource indicates the location of the Management Information Base +(MIB) information that will enable the proposed new module to +participate in its chosen message space. Nominally it is the name of +an XML file in the current working directory; if NULL, mibSource +defaults to roleName.xml. (A future version of loadmib.c might load +MIB information from an ICI "sdr" database rather than from a file.)
+tsorder is the applicable overriding transport service selection +order string. This capability is not yet fully supported; for now, +tsorder should always be NULL.
+applicationName identifies the AMS application within which the +proposed new module is designed to function. The application must be +declared in the MIB.
+authorityName, together with applicationName, identifies the +message space in which the new module proposes to register. The +message space must be declared in the MIB.
+unitName identifies the cell, within the specified message space, in +which the new module proposes to register. The unit must be declared +in the MIB for ventures containing the specified message space, and a +registrar for this cell of this message space must currently be +running in order for the ams_register function to succeed.
+roleName identifies the functional role that the proposed new module +is designed to perform within the indicated application. The role must +be declared in the MIB for that application, and its name will serve +as the name of the module.
+module points to the variable in which the applicable AMS service +access point will be returned upon successful registration of the new +module.
+The function returns 0 on success, -1 on any error.
+The application thread that invoked ams_register is assumed by AMS to +be the main application thread for the module, or "prime thread". +Following successful completion of ams_register all threads of the +application process may commence invoking AMS services -- inviting +messages, publishing messages, etc. -- except that only the prime +thread may receive AMS events, e.g., process incoming messages.
+int [ams_get_module_nbr]{.underline}(AmsModule module);
+The function returns the unique identifying number (within its chosen +cell) assigned to the indicated module as a result of successful +registration.
+int [ams_get_unit_nbr]{.underline}(AmsModule module);
+The function returns the number that uniquely (within the message +space) identifies the cell in which the module registered. The +combination of unit number and module number uniquely identifies the +module within its message space.
+int [ams_set_event_mgr]{.underline}(AmsModule module, AmsEventMgt +*rules);
+The function starts a background "event manager" thread that +automatically receives and processes all AMS events (messages, notices +of message space configuration change, etc.) enqueued for the +indicated module.
+The thread processes each event according to the indicated rules +structure; any event for which a NULL callback function is provided is +simply discarded. For details of the rules structure and prototype +definitions for the callback functions that the rules point to, see +6.1 above. Some notes on this +interface:
+
++While the event manager thread is running, the prime thread is +prohibited from receiving any AMS events itself, i.e., ams_get_event +will always fail.
+Only the prime thread may call ams_set_event_mgr. The function returns +0 on success, -1 on any error.
+void [ams_remove_event_mgr]{.underline}(AmsModule module);
+The function stops the background event manager thread for this +module, if any is running. Only the prime thread may call +ams_remove_event_mgr. Following completion of this function the prime +thread is once again able to receive and process AMS events.
+int [ams_unregister]{.underline}(AmsModule module);
+The function terminates the module's registration, ending the ability +of any thread of the application process to invoke any AMS services; +it automatically stops the background event manager thread for this +module, if any is running.
+Only the prime thread may call ams_unregister. The function returns 0 +on success, -1 on any error.
+
++int [ams_invite]{.underline}(AmsModule module, int roleNbr, int +continuumNbr, int unitNbr, int subjectNbr, int priority, unsigned char +flowLabel, AmsSequence sequence, AmsDiligence diligence);
+This function establishes the module's willingness to accept messages +on a specified subject, under specified conditions, and states the +quality of service at which the module would prefer those messages to +be sent. Invitations are implicitly constrained by venture number: +only messages from modules registered in messages spaces characterized +by the same application and authority names as the message space in +which the inviting module itself is registered are included in any +invitation.
+module must be a valid AMS service access point as returned from +ams_register.
+roleNbr identifies the role that constrains the invitation: only +messages from modules registered as performing the indicated role are +included in this invitation. If zero, indicates "all modules".
+continuumNbr identifies the continuum that constrains the +invitation: only messages from modules operating within the indicated +continuum are included in this invitation. If -1, indicates "the local +continuum". If zero, indicates "all continua".
+unitNbr identifies the unit that constrains the invitation: only +messages from modules registered in cells identified by the indicated +number -- or in cells that are contained within such cells -- are +included in this invitation. A reminder: cell zero is the "root cell", +encompassing the entire message space.
+subjectNbr identifies the subject that constrains the invitation: +only messages on the indicated subject are included in this +invitation.
+priority indicates the level of priority (from 1 to 15, where 1 is +the highest priority indicating greatest urgency) at which the +inviting module prefers that messages responding to this invitation be +sent.
+flowLabel specifies the flow label (a number from 1 to 255, which +AMS may pass through to transport service adapters for +quality-of-service specification purposes) that the inviting module +asks issuing modules to cite when sending messages in response to this +invitation. Flow label 0 signifies "no flow label."
+sequence indicates the minimum level of transmission order +preservation that the inviting module requires for messages responding +to this invitation.
+diligence indicates the minimum level of reliability (based on +acknowledgement and retransmission) that the inviting module requires +for messages responding to this invitation.
+The function returns 0 on success, -1 on any error. When successful, +it causes the invitation to be propagated automatically to all modules +in the inviting module's own message space.
+int [ams_disinvite]{.underline}(AmsModule module, int roleNbr, int +continuumNbr, int unitNbr, int subjectNbr);
+This function terminates the module's prior invitation for messages on +a specified subject under specified conditions. roleNbr, +continuumNbr, unitNbr, and subjectNbr must be identical to those +that characterized the invitation that is to be terminated. The +function returns 0 on success, -1 on any error. When successful, it +causes cancellation of the invitation to be propagated automatically +to all modules in the inviting module's own message space.
+int [ams_subscribe]{.underline}(AmsModule module, int roleNbr, int +continuumNbr, int unitNbr, int subjectNbr, int priority, unsigned char +flowLabel, AmsSequence sequence, AmsDiligence diligence);
+This function establishes the module's request to receive a copy of +every future message published on a specified subject, under specified +conditions, and states the quality of service at which the module +would prefer those messages to be sent. Subscriptions are implicitly +constrained by venture number: only messages from modules registered +in messages spaces characterized by the same application and authority +names as the message space in which the subscribing module itself is +registered are included in any subscription.
+module must be a valid AMS service access point as returned from +ams_register.
+roleNbr identifies the role that constrains the subscription: only +messages from modules registered as performing the indicated role are +included in this subscription. If zero, indicates "all modules".
+continuumNbr identifies the continuum that constrains the +subscription: only messages from modules operating within the +indicated continuum are included in this subscription. If -1, +indicates "the local continuum". If zero, indicates "all continua".
+unitNbr identifies the unit that constrains the subscription: only +messages from modules registered in cells identified by the indicated +number -- or in cells that are contained within such cells -- are +included in this subscription. A reminder: cell zero is the "root +cell", encompassing the entire message space.
+subjectNbr identifies the subject that constrains the subscription: +only messages on the indicated subject are included in this +subscription. subjectNbr may be zero to indicate that messages +published on all subjects are requested; in this case, continuumNbr +must be -1.
+priority indicates the level of priority (from 1 to 15, where 1 is +the highest priority indicating greatest urgency) at which the +subscribing module prefers that messages responding to this +subscription be sent.
+flowLabel specifies the flow label (a number from 1 to 255, which +AMS may pass through to transport service adapters for +quality-of-service specification purposes) that the subscribing module +asks issuing modules to cite when publishing messages in response to +this subscription. Flow label 0 signifies "no flow label."
+sequence indicates the minimum level of transmission order +preservation that the subscribing module requires for messages +responding to this subscription.
+diligence indicates the minimum level of reliability (based on +acknowledgement and retransmission) that the subscribing module +requires for messages responding to this subscription.
+The function returns 0 on success, -1 on any error. When successful, +it causes the subscription to be propagated automatically to all +modules in the subscribing module's own message space.
+int [ams_unsubscribe]{.underline}(AmsModule module, int roleNbr, int +continuumNbr, int unitNbr, int subjectNbr);
+This function terminates the module's prior subscription to messages +on a specified subject under specified conditions. roleNbr, +continuumNbr, unitNbr, and subjectNbr must be identical to those +that characterized the subscription that is to be terminated. The +function returns 0 on success, -1 on any error. When successful, it +causes cancellation of the subscription to be propagated automatically +to all modules in the subscribing module's own message space.
+
++int [ams_lookup_unit_nbr]{.underline}(AmsModule module, char +*unitName);
+The function returns the unit number corresponding to the indicated +unitName, in the context of the venture encompassing the message +space in which the invoking module is registered. Returns -1 if this +unitName is undefined in this venture.
+int [ams_lookup_role_nbr]{.underline}(AmsModule module, char +*roleName);
+The function returns the role number corresponding to the indicated +roleName, in the context of the application characterizing the +message space in which the invoking module is registered. Returns -1 +if this roleName is undefined in this application.
+int [ams_lookup_subject_nbr]{.underline}(AmsModule module, char +*subjectName);
+The function returns the subject number corresponding to the indicated +subjectName, in the context of the application characterizing the +message space in which the invoking module is registered. Returns -1 +if this subjectName is undefined in this application.
+int [ams_lookup_continuum_nbr]{.underline}(AmsModule module, char +*continuumName);
+The function returns the continuum number corresponding to the +indicated continuumName, Returns -1 if the named continuum is +unknown.
+char *[ams_lookup_unit_name]{.underline}(AmsModule module, int +unitNbr);
+The function returns the unit name corresponding to the indicated +unitNbr, in the context of the venture encompassing the message +space in which the invoking module is registered. Returns NULL if this +unitNbr is undefined in this venture.
+char *[ams_lookup_role_name]{.underline}(AmsModule module, int +roleNbr);
+The function returns the role name corresponding to the indicated +roleNbr, in the context of the application characterizing the +message space in which the invoking module is registered. Returns NULL +if this roleNbr is undefined in this application.
+char *[ams_lookup_subject_name]{.underline}(AmsModule module, int +subjectNbr);
+The function returns the subject name corresponding to the indicated +subjectNbr, in the context of the application characterizing the +message space in which the invoking module is registered. Returns NULL +if this subjectNbr is undefined in this application.
+char *[ams_lookup_continuum_name]{.underline}(AmsModule module, int +continuumNbr);
+The function returns the continuum name corresponding to the indicated +continuumNbr. Returns NULL if the specified continuum is unknown.
+char *[ams_get_role_name]{.underline}(AmsModule module, int unitNbr, +int moduleNbr);
+The function returns the name of the role under which the module +identified by unitNbr and moduleNbr is registered, within the +invoking module's own message space. Returns NULL if no module +identified by unitNbr and moduleNbr is known to be currently +registered within this message space.
+Lyst [ams_list_msgspaces]{.underline}(AmsModule module);
+The function returns a Lyst (see the documentation for lyst) of the +numbers of all AMS continua in which there is known to be another +message space for the venture in which module is registered. Returns +NULL if there is insufficient free memory to create this list. NOTE: +be sure to use lyst_destroy to release the memory occupied by this +list when you're done with it.
+int [ams_subunit_of]{.underline}(AmsModule module, int argUnitNbr, int +refUnitNbr);
+The function returns 1 if the unit identified by argUnitNbr is a +subset of (or is identical to) the unit identified by refUnitNbr. +Otherwise it returns 0.
+int [ams_get_continuum_nbr]{.underline}();
+The function returns the local continuum number.
+int [ams_continuum_is_neighbor]{.underline}(int continuumNbr);
+The function returns 1 if the continuum identified by continuumNbr +is a neighbor (within the RAMS network) of the local continuum. +Otherwise it returns 0.
+int [ams_rams_net_is_tree]{.underline}(AmsModule module);
+The function returns 1 if the RAMS network is configured as a tree. +Otherwise it returns 0.
+
++int [ams_publish]{.underline}(AmsModule module, int subjectNbr, int +priority, unsigned char flowLabel, int contentLength, char *content, +int context);
+This function causes an AMS message to be constructed on the indicated +subject, encapsulating the indicated content and characterized by the +indicated processing context token, and causes one copy of that +message to be sent to every module in the message space that currently +asserts a subscription for messages on this subject such that the +invoking module satisfies the constraints on that subscription.
+priority may be any value from 1 to 15, overriding the priority +preference(s) asserted by the subscriber(s), or it may be zero +indicating "use each subscriber's preferred priority." flowLabel may +be any value from 1 to 255, overriding the flow label preference(s) +asserted by the subscriber(s), or it may be zero indicating "use each +subscriber's preferred flow label."
+The function returns 0 on success, -1 on any error.
+int [ams_send]{.underline}(AmsModule module, int continuumNbr, int +unitNbr, int moduleNbr, int subjectNbr, int priority, unsigned char +flowLabel, int contentLength, char *content, int context);
+This function causes an AMS message to be constructed on the indicated +subject, encapsulating the indicated content and characterized by the +indicated processing context token, and causes that message to be sent +to the module identified by unitNbr and moduleNbr within the indicated +continuum, provided that this module currently asserts an invitation +for messages on this subject such that the invoking module satisfies +the constraints on that invitation.
+If continuumNbr is -1, the local continuum is inferred.
+priority may be any value from 1 to 15, overriding the priority +preference asserted by the destination module, or it may be zero +indicating "use the destination module's preferred priority." +flowLabel may be any value from 1 to 255, overriding the flow label +preference asserted by the destination module, or it may be zero +indicating "use the destination module's preferred flow label."
+The function returns 0 on success, -1 on any error.
+int [ams_query]{.underline}(AmsModule module, int continuumNbr, int +unitNbr, int moduleNbr, int subjectNbr, int priority, unsigned char +flowLabel, int contentLength, char *content, int context, int term, +AmsEvent *event);
+This function is identical to ams_send in usage and effect except +that, following issuance of the message, the function blocks (that is, +does not return control to the invoking function) until either (a) a +message that is a specific reply to this message is received or (b) +the time period indicated by term -- in microseconds -- elapses. +Upon return of control to the invoking function, the AMS event pointer +referenced by event points to the AMS event that caused the return +of control, either a reply message or a timeout or (possibly) a notice +of processing error. If term is 0, the function returns control to the +invoking function immediately and *event always points to a timeout +event. If term is -1, the function never returns control until a reply +message is received.
+The function returns 0 on success, -1 on any error.
+int [ams_reply]{.underline}(AmsModule module, AmsEvent msg, int +subjectNbr, int priority, unsigned char flowLabel, int contentLength, +char
+*content);
+This function is identical to ams_send in usage and effect except that +the destination of the reply message is not stated explicitly by the +invoking function; instead, the invoking function provides a pointer +to the AMS message (an AmsEvent whose event type is AMS_MSG_EVT) whose +sender is the destination of the reply message.
+The function returns 0 on success, -1 on any error.
+int [ams_announce]{.underline}(AmsModule module, int roleNbr, int +continuumNbr, int unitNbr, int subjectNbr, int priority, unsigned char +flowLabel, int contentLength, char *content, int context);
+This function causes an AMS message to be constructed on the indicated +subject, encapsulating the indicated content and characterized by the +indicated processing context token, and causes one copy of that +message to be sent to every module in the domain of the +announcement that currently asserts an invitation for messages on +this subject such that the invoking module satisfies the constraints +on that invitation. The domain of the announcement is the set of all +modules such that:
+
++continuumNbr indicates "the local continuum"; a value of zero +indicates "all continua".
+
++priority may be any value from 1 to 15, overriding the priority +preference(s) asserted by the destination module(s), or it may be zero +indicating "use each destination module's preferred priority." +flowLabel may be any value from 1 to 255, overriding the flow label +preference(s) asserted by the destination module(s), or it may be zero +indicating "use each destination module's preferred flow label."
+The function returns 0 on success, -1 on any error.
+
++int [ams_get_event]{.underline}(AmsModule module, int term, AmsEvent +*event);
+This function acquires the next AMS event currently in the queue of +AMS events that have yet to be handled by the application. The +function blocks (that is, does not return control to the invoking +function) until either (a) an event is available to be acquired or (b) +the time period indicated by term -- in microseconds -- elapses. +Upon return of control to the invoking function, the AMS event pointer +referenced by event points to the AMS event that caused the return +of control: a message, a notice of message space configuration change, +a user-defined event, or a timeout. If term is 0, the function +returns control to the invoking function immediately. If term is -1, +the function never returns control until a non-timeout event can be +acquired.
+The function returns 0 on success, -1 on any error. Following +acquisition of an event, the application program should:
+
++int [ams_get_event_type]{.underline}(AmsEvent event);
+This function returns the event type of the indicated event, enabling +the event to be properly parsed by the application program. The +possible event types are AMS_MSG_EVT, TIMEOUT_EVT, NOTICE_EVT, and +USER_DEFINED_EVT.
+int [ams_parse_msg]{.underline}(AmsEvent event, int *continuumNbr, +int *unitNbr, int
+*moduleNbr, int *subjectNbr, int *contentLength, char **content, +int *context, AmsMsgType *msgType, int *priority, unsigned char
+*flowLabel);
+This function extracts the content of an AMS event that is a received +message, inserting values into the variables that the function's +arguments point to. continuumNbr, unitNbr, and moduleNbr +identify the module that sent the message. Returns 0 unless one or +more of the arguments provided to the function are NULL, in which case +the function returns -1.
+int [ams_parse_notice]{.underline}(AmsEvent event, AmsStateType +*state,
+AmsChangeType *change, int *unitNbr, int *moduleNbr, int *roleNbr, +int *domainContinuumNbr, int *domainUnitNbr, int *subjectNbr, int +*priority, unsigned char *flowLabel, AmsSequence *sequence, +AmsDiligence *diligence);
+This function extracts the content of an AMS event that is a notice of +change in message space configuration, inserting values into the +variables that the function's arguments point to.
+state and change indicate the nature of the change.
+unitNbr and moduleNbr identify the module to which the change +pertains.
+roleNbr is provided in the event that the change is the registration +of a new module (in which case it indicates the functional nature of +the new module) or is a subscription, unsubscription, invitation, or +disinvitation (in which case it indicates the role constraining the +subscription or invitation).
+For a notice of subscription, unsubscription, invitation, or +disinvitation:
+
++For a notice of subscription or invitation, priority, flowLabel, +sequence, and diligence indicate the quality of service requested +by the module for this subscription or invitation.
+Returns 0 unless one or more of the arguments provided to the function +are NULL, in which case the function returns -1.
+int [ams_parse_user_event]{.underline}(AmsEvent event, int *code, int +*dataLength, char **data);
+This function extracts the content of a user-defined AMS event, +inserting values into the variables that the function's arguments +point to. Returns 0 unless one or more of the arguments provided to +the function are NULL, in which case the function returns -1.
+int [ams_recycle_event]{.underline}(AmsEvent event);
+This function simply releases all memory occupied by the indicated +event. Returns 0 unless event is NULL, in which case the function +returns -1.
+
++int [ams_post_user_event]{.underline}(AmsModule module, int +userEventCode, int userEventDataLength, char *userEventData, int +priority);
+This function posts a user-defined event into the queue of AMS events +that have yet to be handled by the application. userEventCode is an +arbitrary, user-defined numeric value; userEventData, if not NULL, +is assumed to be an arbitrary, user-defined character string of length +userEventDataLength.
+priority may be any value from 0 to 16. Note that this enables the +application to post an event to itself that is guaranteed to be of +higher priority than any message -- assuring that it will be processed +before any message that is currently enqueued or that arrives in the +future -- or, alternatively, to post an event that is guaranteed to be +of lower priority than any message and will therefore only be +processed during a lull in message reception.
+Returns 0 on success, -1 on any error.
+
++The JPL implementation of Remote AMS comprises a library +(librams.c) and a sample RAMS gateway program (ramsTest.c) +that uses that library.
+
++The RAMS library implements a very simple API, which is defined by the +header file
+rams.h and comprises just a single function:
+int [rams_run]{.underline}(char *mibSource, char *tsorder, char +*applicationName, char *authorityName, char *unitName, char +*roleName, int lifetime);
+This function initiates a RAMS gateway operations loop. mibSource, +tsorder, mName, memory, mSize, applicationName, +authorityName, unitName, and roleName are as discussed in the +documentation of [ams_register]{.underline} above; they are used to +register the RAMS gateway process as an AMS module. lifetime is the +user-specified maximum time to live for all DTN bundles issued by the +RAMS gateway in the course of its communications over the RAMS +network.
+Note that the priority assigned to any DTN bundle that conveys a +published or privately sent AMS message over the RAMS network will be +computed as a function of the flow label specified at the time the +message was originally published or sent. If no overriding flow label +was specified, then the bundle priority will be 1 (standard). +Otherwise, the number represented by the two low-order bits of the +flow label will be used as the bundle priority and two "extended class +of service" parameters will be derived from the next two higher-order +bits of the flow label: bit 5 (the 3^rd^-lowest-order bit) will be +used as the value of the "minimum latency" flag, with a value of 1 +indicating that the bundle is critical, and bit 4 (the +4^th^-lowest-order bit) will be used as the value of the "best-effort" +flag, with a value of 1 indicating that the bundle should be sent over +an unacknowledged convergence-layer protocol. All bundles issued by +the RAMS gateway that don't carry AMS messages will be assigned bundle +priority 1.
+This function runs indefinitely until it fails or is interrupted by a +SIGINT signal. Returns -1 on any failure, 0 on normal termination.
+
++The sample RAMS gateway program ramsgate provides basic RAMS +gateway functionality. To run it, enter a command of the following +form at a terminal window prompt:
+ramsgate application_name authority_name_name lifetime +[memory_size [memory_manager_name]]
+application_name, authority_name, lifetime, +memory_manager_name, and memory_size are as discussed in the +documentation of ams_register. If not specified, memory size defaults +to 200000. The gateway process always registers in the role named +"RAMS".
+Note that ramsgate relies on the ION implementation of DTN; be +sure an ION node is operating on the local computer before starting +ramsgate. (See the ION Design and Operations Guide for details.)
+To terminate operation of the gateway process, just use CTRL-C to +interrupt the program.
+
++In order to operate correctly, every AMS application process -- and +amsd as well -- must initially load Management Information Base +(MIB) values that are universally consistent.
+Currently this is accomplished automatically during registration: MIB +value declaration commands in XML format are read from a file, parsed, +and processed automatically.
+
++The elements of the XML files used to load AMS MIBs are as follows: +[ams_mib_load]{.underline}: contains a series of MIB load commands.
+Attributes: none
+[ams_mib_init]{.underline}: command that initializes the MIB. +Attributes:
+continuum_nbr: the number identifying the local continuum ptsname: the +name of the primary transport service
+[ams_mib_add]{.underline}: contains a series of elements that add +items to the MIB.
+Attributes: none
+[continuum]{.underline}:
+Attributes:
+nbr: the number that identifies this continuum name: the name that +identifies this continuum
+neighbor: a Boolean indication ("1" or "0") of whether or not this is +a neighboring continuum [If omitted, the continuum is by default +assumed to be a neighbor -- that is, an implicit neighbor="1" +attribute is the default.]
+desc: a brief textual description of this continuum
+[csendpoint]{.underline}: configuration server endpoint specification +(i.e., network location of configuration server)
+Attributes:
+epspec: PTS-specific endpoint specification string (to be more fully +documented in a later edition of this [Programmer's +Guide]{.underline})
+[application]{.underline}: defines an application Attributes:
+name: name of application
+[venture]{.underline}: defines a venture (an instance of an +application) Attributes:
+nbr: the number that identifies this venture
+appname: the name of the application served by this venture
+authname: the name of the authority responsible for this instance of +this application
+net_config: the configuration ("mesh" or "tree") of the RAMS network +comprising all AMS continua that participate in this venture. [If +omitted, the RAMS network configuration is by default assumed to be a +mesh.]
+gweid: a string identifying the endpoint for the local continuum's +RAMS gateway within the RAMS network for this venture; default is +"bp@ipn:local_continuum_nbr.venture_nbr"
+root_cell_resync_period: the period (expressed as a count of registrar +heartbeats) on which the configuration of the root unit of this +venture will automatically be resynchronized. If omitted or set to +zero, automatic resync is disabled within the root unit.
+[role]{.underline}: defines a role within a venture Attributes:
+nbr: the number that identifies this role name: the name that +identifies this role
+[subject]{.underline}: defines a message subject within a venture +Attributes:
+nbr: the number that identifies this subject name: the name that +identifies this subject
+desc: a brief textual description of this message subject
+[element]{.underline}: defines one of the elements (fields) of a +message on a given subject (note that elements must be defined in the +order in which they appear in the message, without omission)
+Attributes:
+type: a number that specifies the data type of this element (1 = long, +2 = int, 3 = short, 4 = char, 5 = string)
+name: the name that identifies this element
+desc: a brief textual description of this message field. +[unit]{.underline}: defines a unit of a venture
+Attributes:
+nbr: the number that identifies this unit name: the name that +identifies this unit
+resync_period: the period (expressed as a count of registrar +heartbeats) on which the configuration of this unit will automatically +be resynchronized. If omitted or set to zero, automatic resync is +disabled within this unit.
+[msgspace]{.underline}: identifies a remote continuum that contains a +message space that is part of this venture
+Attributes:
+nbr: the number that identifies the continuum containing this message +space
+gweid: a string identifying the endpoint for the indicated continuum's +RAMS gateway, within the RAMS network for this venture; default is +"bp@ipn:continuum_nbr.venture_nbr"
+neighbor: Signifies adjacency of the message space with the parent +continuum using a Boolean value (i.e. 1 is adjacent, 0 is +non-adjacent). Note that in this context message space is analogous to +continuum. If the neighbor attribute is omitted the default value is +1.
+
++\<?xml version=\"1.0\" standalone=\"yes\"?>
+\<ams_mib_load>
+\<ams_mib_init continuum_nbr=\"1\" ptsname=\"dgr\"/>
+\<ams_mib_add>
+\<continuum nbr=\"2\" name=\"gsfc\"\" desc=\"Goddard Space Flight +Center\"/>
+\<csendpoint epspec=\"amroc.net:2502\"/>
+\<application name=\"demo\"/>
+\<venture nbr=\"9\" appname=\"demo\" authname=\"test\" >
+\<role nbr=\"2\" name=\"shell\"/>
+\<role nbr=\"3\" name=\"log\"/>
+\<role nbr=\"4\" name=\"pitch\"/>
+\<role nbr=\"5\" name=\"catch\"/>
+\<role nbr=\"6\" name=\"benchs\"/>
+\<role nbr=\"7\" name=\"benchr\"/>
+\<subject nbr=\"1\" name=\"text\" desc=\"ASCII text\"/>
+\<subject nbr=\"2\" name=\"noise\" desc=\"more ASCII text\"/>
+\<subject nbr=\"3\" name=\"bench\" desc=\"numbered messages\"/>
+\<unit nbr=\"1\" name=\"orbiters\"/>
+\<unit nbr=\"2\" name=\"orbiters.near\"/>
+\<unit nbr=\"3\" name=\"orbiters.far\"/>
+\<msgspace nbr=\"2\" neighbor=1/>
+\</venture>
+\</ams_mib_add>
+\</ams_mib_load>
+
++AMS applications (i.e. "modules") are custom software built using AMS' +application programming interface (API). These AMS modules serve as +the interfaces between AMS and mission-specific processes or +applications.
+
++A general overview of the required steps follows this sequence:
+
++Applications created with the API must be compiled with links to ION's +dynamic libraries.
+The following two-step build procedure provides an example of how this +works (note that the default path for installed ION library files +(Linux) is used: /usr/local/lib/):
+
Applications must self-register with an AMS registrar. To facilitate +this the following information is required by the application (i.e. +using the ams_register() function):
+++Given this requirement, it may be useful (but not necessary) for the +application to accept command line arguments.
+An example command to start a custom AMS module might look like the +following:
+./your_module_name ' ' your_module_mib_role_name amsdemo test
+Note: the empty quotes (' ') above are used to specify registration +with the root unit registrar.
+
++Follows is the complete source code for 'amshello', a simple +Unix-based AMS application installed alongside AMS. It is a complete +distributed system comprising two AMS application modules that +functions as follows.
+When the amshello program starts it fork()s into two processes, a +"pitcher" and a "catcher". Both processes then register as modules in +the root cell of an "amsdemo/test" message space. The catcher invites +messages on subject "text" and waits for the first such message to +arrive. The pitcher waits for an invitation to send messages of +subject "text" and, when it arrives, sends one such message and +terminates. When the catcher receives the message it prints the text +of the message and then terminates.
+To run the 'amshello' demonstration application the following 2 steps +are required.
+
++Important points regarding the above amsd command (see man amsd +for additional details):
+
++/*
+amshello.c
+\"Hello world\" demonstration using AMS - Unix platform (only)
+Copyright (c) 2023, California Institute of Technology.
+Sky DeBaun, Jet Propulsion Laboratory.
+This program assumes the following +conditions---------------
+1.) ION is running
+2.) An AMS Registrar is running
+3.) An AMS Configuration Server is running
+4.) An MIB configuration file has been created and is specified for +use (see note below)
+*NOTE: the following command completes steps 2, 3, and 4 above (run +this command after other ION processes start, then run the 'amshello' +command from terminal to run the program):
+amsd @ @ amsdemo test \"\" &
+*/
+include \"ams.h\"
+static int runPitcher()
+{
+AmsModule me;
+AmsEvent evt;
+AmsStateType state;
+AmsChangeType change;
+int zn, nn, rn, dcn, dzn, sn, pr, textlen;
+unsigned char fl;
+AmsSequence sequence;
+AmsDiligence diligence;
+char buffer[80];
+isprintf(buffer, sizeof buffer, \"Hello from process %d\", (int) +getpid());
+textlen = strlen(buffer) + 1;
+//register pitch module using default in-memory MIB (i.e. using the @)
+oK(ams_register(\"@\", NULL, \"amsdemo\", \"test\", \"\", \"pitch\", +&me));
+while (1)
+{
+if (ams_get_event(me, AMS_BLOCKING, &evt) \< 0)
+{
+return 0;
+}
+else
+{
+ams_parse_notice(evt, &state, &change, &zn, &nn, &rn, &dcn,
+&dzn, &sn, &pr, &fl, &sequence, &diligence);
+ams_recycle_event(evt);
+}
+if (state == AmsInvitationState && sn == 1)
+{
+printf(\"Process %d sending: \'%s\'\n\", (int) getpid(), buffer);
+fflush(stdout);
+ams_send(me, -1, zn, nn, 1, 0, 0, textlen, buffer, 0);
+ams_unregister(me);
+return 0;
+}
+}
+}
+static int runCatcher()
+{
+AmsModule me;
+AmsEvent evt;
+int cn, zn, nn, sn, len, ct, pr;
+unsigned char fl;
+AmsMsgType mt;
+char *txt;
+//register catch module using default in-memory MIB (i.e. @)
+oK(ams_register(\"@\", NULL, \"amsdemo\", \"test\", \"\", \"catch\", +&me));
+ams_invite(me, 0, 0, 0, 1, 8, 0, AmsArrivalOrder, AmsAssured);
+while (1)
+{
+if (ams_get_event(me, AMS_BLOCKING, &evt) \< 0) return 0;
+if (ams_get_event_type(evt) == AMS_MSG_EVT) break;
+else ams_recycle_event(evt);
+}
+ams_parse_msg(evt, &cn, &zn, &nn, &sn, &len, &txt, &ct, &mt, &pr, +&fl);
+printf(\"Process %d received: \'%s\'\n\", (int) getpid(), txt); +fflush(stdout);
+ams_recycle_event(evt); ams_unregister(me); return 0;
+}
+int main(void)
+{
+pid_t pid = fork();
+if (pid == -1) {
+fprintf(stderr, \"Failed to create child process.\n\");
+return EXIT_FAILURE;
+}
+if (pid == 0)
+//child process runs +transmitter----------------------
+runPitcher();
+else
+{
+//parent process runs +receiver------------------------
+runCatcher();
+}
+return 0;
+}
+
++ + + + + + + + + + + + + +The research described in this publication was carried out at the Jet +Propulsion Laboratory, California Institute of Technology, under a +contract with the National Aeronautics and Space Administration.
+
The following tools are available to you after ION is built:
+Daemon and Configuration +- ionadmin is the administration and configuration interface for the local ION node contacts and manages shared memory resources used by ION. +- ltpadmin is the administration and configuration interface for LTP operations on the local ION node. +- bsspadmin is the administrative interface for operations of the Bundle Streaming Service Protocol on the local ion node. +- bpadmin is the administrative interface for bundle protocol operations on the local ion node. +- ipnadmin is the administration and configuration interface for the IPN addressing system and routing on the ION node. (ipn:) +- dtn2admin is the administration and configuration interface for the DTN addressing system and routing on the ION node. (dtn://) +- killm is a script which tears down the daemon and any running ducts on a single machine (use ionstop instead). +- ionstart is a script which completely configures an ION node with the proper configuration file(s). +- ionstop is a script which completely tears down the ION node. +- ionscript is a script which aides in the creation and management of configuration files to be used with ionstart.
+Simple Sending and Receiving + - bpsource and bpsink are for testing basic connectivity between endpoints. bpsink listens for and then displays messages sent by bpsource. + - bpsendfile and bprecvfile are used to send files between ION nodes.
+Testing and Benchmarking + - bpdriver benchmarks a connection by sending bundles in two modes: request-response and streaming. + - bpecho issues responses to bpdriver in request-response mode. + - bpcounter acts as receiver for streaming mode, outputting markers on receipt of data from bpdriver and computing throughput metrics.
+It is important to note that, by default, the administrative programs will all trigger the creation of a log file called ion.log in the directory where the program is called. This means that write-access in your current working directory is required. The log file itself will contain the expected log information from administrative daemons, but it will also contain error reports from simple applications such as bpsink. This is important to note since the BP applications may not be reporting all error information to stdout or stderr.
+A script has been created which allows a more streamlined configuration and startup of an ION node. This script is called ionstart, and it has the following syntax. Don't run it yet; we still have to configure it!
+ionstart -I <filename>
filename: This is the name for configuration file which the script will attempt to use for the various configuration commands. The script will perform a sanity check on the file, splitting it into command sections appropriate for each of the administration programs.
+Configuration information (such as routes, connections, etc) can be specified one of two ways for any of the individual administration programs:
+(Recommended) Creating a configuration file and passing it to ionadmin, bpadmin, ipnadmin... either directly or via the ionstart helper script. +Manually typing configuration commands into the terminal for each administration program.
+You can find appropriate commands in the following sections.
+There are five configuration files about which you should be aware.
+The first, ionadmin's configuration file, assigns an identity (node number) to the node, optionally configures the resources that will be made available to the node, and specifies contact bandwidths and one-way transmission times. Specifying the "contact plan" is important in deep-space scenarios where the bandwidth must be managed and where acknowledgments must be timed according to propagation delays. It is also vital to the function of contact-graph routing.
+The second, ltpadmin's configuration file, specifies spans, transmission speeds, and resources for the Licklider Transfer Protocol convergence layer.
+The third, ipnadmin's configuration file, maps endpoints at "neighboring" (topologically adjacent, directly reachable) nodes to convergence-layer addresses. Our examples use TCP/IP and LTP (over IP/UDP), so it maps endpoint IDs to IP addresses. This file populates the ION analogue to an ARP cache for the "ipn" naming scheme.
+The fourth, bpadmin's configuration file, specifies all of the open endpoints for delivery on your local end and specifies which convergence layer protocol(s) you intend to use. With the exception of LTP, most convergence layer adapters are fully configured in this file.
+The fifth, dtn2admin's configuration file, populates the ION analogue to an ARP cache for the "dtn" naming scheme.
+Given to ionadmin either as a file or from the daemon command line, this file configures contacts for the ION node. We will assume that the local node's identification number is 1.
+This file specifies contact times and one-way light times between nodes. This is useful in deep-space scenarios: for instance, Mars may be 20 light-minutes away, or 8. Though only some transport protocols make use of this time (currently, only LTP), it must be specified for all links nonetheless. Times may be relative (prefixed with a + from current time) or absolute. Absolute times, are in the format yyyy/mm/dd-hh:mm:ss. By default, the contact-graph routing engine will make bundle routing decisions based on the contact information provided.
The configuration file lines are as follows:
+1 1 ''
This command will initialize the ion node to be node number 1.
+1 refers to this being the initialization or ''first'' command. +1 specifies the node number of this ion node. (IPN node 1). +'' specifies the name of a file of configuration commands for the node's use of shared memory and other resources (suitable defaults are applied if you leave this argument as an empty string).
+s
This will start the ION node. It mostly functions to officially "start" the node in a specific instant; it causes all of ION's protocol-independent background daemons to start running.
+a contact +1 +3600 1 1 100000
specifies a transmission opportunity for a given time duration between two connected nodes (or, in this case, a loopback transmission opportunity).
+a adds this entry in the configuration table. +contact specifies that this entry defines a transmission opportunity. ++1 is the start time for the contact (relative to when the s command is issued). ++3600 is the end time for the contact (relative to when the s command is issued). +1 is the source node number. +1 is the destination node number. +100000 is the maximum rate at which data is expected to be transmitted from the source node to the destination node during this time period (here, it is 100000 bytes / second).
+a range +1 +3600 1 1 1
specifies a distance between nodes, expressed as a number of light seconds, where each element has the following meaning:
+a adds this entry in the configuration table. +range declares that what follows is a distance between two nodes. ++1 is the earliest time at which this is expected to be the distance between these two nodes (relative to the time s was issued). ++3600 is the latest time at which this is still expected to be the distance between these two nodes (relative to the time s was issued). +1 is one of the two nodes in question. +1 is the other node. +1 is the distance between the nodes, measured in light seconds, also sometimes called the "one-way light time" (here, one light second is the expected distance).
+m production 1000000
specifies the maximum rate at which data will be produced by the node.
+m specifies that this is a management command. +production declares that this command declares the maximum rate of data production at this ION node. +1000000 specifies that at most 1000000 bytes/second will be produced by this node.
+m consumption 1000000
specifies the maximum rate at which data can be consumed by the node.
+m specifies that this is a management command. +consumption declares that this command declares the maximum rate of data consumption at this ION node. +1000000 specifies that at most 1000000 bytes/second will be consumed by this node.
+This will make a final configuration file host1.ionrc which looks like this:
+1 1 ''
+s
+a contact +1 +3600 1 1 100000
+a range +1 +3600 1 1 1
+m production 1000000
+m consumption 1000000
+Given to ltpadmin as a file or from the command line, this file configures the LTP engine itself. We will assume the local IPN node number is 1; in ION, node numbers are used as the LTP engine numbers.
+1 32
This command will initialize the LTP engine:
+1 refers to this being the initialization or ''first'' command. +32 is an estimate of the maximum total number of LTP ''block'' transmission sessions - for all spans - that will be concurrently active in this LTP engine. It is used to size a hash table for session lookups.
+a span 1 32 32 1400 10000 1 'udplso localhost:1113'
This command defines an LTP engine 'span':
+a indicates that this will add something to the engine.
+span indicates that an LTP span will be added.
+1 is the engine number for the span, the number of the remote engine to which LTP segments will be transmitted via this span. In this case, because the span is being configured for loopback, it is the number of the local engine, i.e., the local node number. This will have to match an outduct in Section 2.6.
+32 specifies the maximum number of LTP ''block'' transmission sessions that may be active on this span. The product of the mean block size and the maximum number of transmission sessions is effectively the LTP flow control ''window'' for this span: if it's less than the bandwidth delay product for traffic between the local LTP engine and this spa's remote LTP engine then you'll be under-utilizing that link. We often try to size each block to be about one second's worth of transmission, so to select a good value for this parameter you can simply divide the span's bandwidth delay product (data rate times distance in light seconds) by your best guess at the mean block size.
+The second 32specifies the maximum number of LTP ''block'' reception sessions that may be active on this span. When data rates in both directions are the same, this is usually the same value as the maximum number of transmission sessions.
+1400 is the number of bytes in a single segment. In this case, LTP runs atop UDP/IP on ethernet, so we account for some packet overhead and use 1400.
+1000 is the LTP aggregation size limit, in bytes. LTP will aggregate multiple bundles into blocks for transmission. This value indicates that the block currently being aggregated will be transmitted as soon as its aggregate size exceeds 10000 bytes.
+1 is the LTP aggregation time limit, in seconds. This value indicates that the block currently being aggregated will be transmitted 1 second after aggregation began, even if its aggregate size is still less than the aggregation size limit.
+'udplso localhost:1113' is the command used to implement the link itself. The link is implemented via UDP, sending segments to the localhost Internet interface on port 1113 (the IANA default port for LTP over UDP).
+s 'udplsi localhost:1113'
Starts the ltp engine itself:
+s starts the ltp engine.
+'udplsi localhost:1113' is the link service input task. In this case, the input ''duct' is a UDP listener on the local host using port 1113.
+This means that the entire configuration file host1.ltprc looks like this:
+ +Given to bpadmin either as a file or from the daemon command line, this file configures the endpoints through which this node's Bundle Protocol Agent (BPA) will communicate. We will assume the local BPA's node number is 1; as for LTP, in ION node numbers are used to identify bundle protocol agents.
+1
This initializes the bundle protocol:
+1 refers to this being the initialization or ''first'' command.
+a scheme ipn 'ipnfw' 'ipnadminep'
This adds support for a new Endpoint Identifier (EID) scheme:
+a means that this command will add something.
+scheme means that this command will add a scheme.
+ipn is the name of the scheme to be added.
+'ipnfw' is the name of the IPN scheme's forwarding engine daemon.
+'ipnadminep' is the name of the IPN scheme's custody transfer management daemon.
+a endpoint ipn:1.0 q
This command establishes this BP node's membership in a BP endpoint:
+a means that this command will add something.
+endpoint means that this command adds an endpoint.
+ipn is the scheme name of the endpoint.
+1.0 is the scheme-specific part of the endpoint. For the IPN scheme the scheme-specific part always has the form nodenumber:servicenumber. Each node must be a member of the endpoint whose node number is the node's own node number and whose service number is 0, indicating administrative traffic.
+q means that the behavior of the engine, upon receipt of a new bundle for this endpoint, is to queue it until an application accepts the bundle. The alternative is to silently discard the bundle if no application is actively listening; this is specified by replacing q with x.
+a endpoint ipn:1.1 q
a endpoint ipn:1.2 q
These specify two more endpoints that will be used for test traffic.
+a protocol ltp 1400 100
This command adds support for a convergence-layer protocol:
+a means that this command will add something.
+protocol means that this command will add a convergence-layer protocol.
+ltp is the name of the convergence-layer protocol.
+1400 is the estimated size of each convergence-layer protocol data unit (in bytes); in this case, the value is based on the size of a UDP/IP packet on Ethernet.
+100 is the estimated size of the protocol transmission overhead (in bytes) per convergence-layer procotol data unit sent.
+a induct ltp 1 ltpcli
This command adds an induct, through which incoming bundles can be received from other nodes:
+a means that this command will add something.
+induct means that this command will add an induct.
+ltp is the convergence layer protocol of the induct.
+1 is the identifier of the induct, in this case the ID of the local LTP engine.
+ltpcli is the name of the daemon used to implement the induct.
+a outduct ltp 1 ltpclo
This command adds an outduct, through which outgoing bundles can be sent to other nodes:
+a means that this command will add something.
+outduct means that this command will add an outduct.
+ltp is the convergence layer protocol of the outduct.
+1 is the identifier of the outduct, the ID of the convergence-layer protocol induct of some remote node. See Section 2.5 for remote LTP engine IDs.
+ltpclo is the name of the daemon used to implement the outduct.
+s
This command starts the bundle engine including all daemons for the inducts and outducts.
+That means that the entire configuration file host1.bprc looks like this:
+1
+a scheme ipn 'ipnfw' 'ipnadminep'
+a endpoint ipn:1.0 q
+a endpoint ipn:1.1 q
+a endpoint ipn:1.2 q
+a protocol ltp 1400 100
+a induct ltp 1 ltpcli
+a outduct ltp 1 ltpclo
+s
+As noted earlier, this file is used to build ION's analogue to an ARP cache, a table of ''egress plans.'' It specifies which outducts to use in order to forward bundles to the local node's neighbors in the network. Since we only have one outduct, for forwarding bundles to one place (the local node), we only have one egress plan.
+a plan 1 ltp/1
This command defines an egress plan for bundles to be transmitted to the local node:
+a means this command adds something.
+plan means this command adds an egress plan.
+1 is the node number of the remote node. In this case, that is the local node's own node number; we're configuring for loopback.
+ltp/1 is the identifier of the outduct through which to transmit bundles in order to convey them to this ''remote'' node.
+This means that the entire configuration file host1.ipnrc looks like this:
+a plan 1 ltp/1
Assuming no errors occur with the configuration above, we are now ready to test loopback communications. In one terminal, we have to run the start script (the one we said that you would have to have earlier). It's right here, in case you forgot to write it down:
+ionstart -i host1.ionrc -l host1.ltprc -b host1.bprc -p host1.ipnrc
This command will run the appropriate administration programs, in order, with the appropriate configuration files. Don't worry that the command is lengthy and unwieldly; we will show you how to make a more clean single configuration file later.
+Once the daemon is started, run:
+bpsink ipn:1.1
This will begin listening on the Endpoint ID with the endpoint_number 1 on service_number 1, which is used for testing.
+Now open another terminal and run the command:
+bpsource ipn:1.1
This will begin sending messages you type to the Endpoint ID ipn:1.1, which is currently being listened to by bpsink. Type messages into bpsource, press enter, and see if they are reported by bpsink.
+If so, you're ready for bigger and better things. If not, check the following:
+Do you have write permissions for your current directory? If not, you will not be able to start the daemon as it has to write out to the ion.log file. +Are your config files exactly as specified, except for IP address changes? +Are you running it on one of our supported platforms? Currently, those are the only supported distributions.
+If you are still having problems, you can ask for help on the ION users' list or file an ION bug report.
+As the daemon launches many ducts and helper applications, it can be complicated to turn it all off. To help this, we provided a script. The script similar to ionstart exists called ionstop, which tears down the ion node in one step. You can call it like so:
+ionstop
After stopping the daemon, it can be restarted using the same procedures as outlined above. Do remember that the ion.log file is still present, and will just keep growing as you experiment with ION.
+IMPORTANT: The user account that runs ionstart must also run ionstop. If that account does not, no other accounts can successfully start the daemon, as the shared memory vital to ION's functionality will already be occupied.
+Detailed documentation of ION and its applications are available via the man pages. It is suggested that you start with man ion , as this is an overview man page listing all available ION packages.
+The most difficult and cumbersome method of starting an ION node is to manually run the various administration programs in order, manually typing configuration commands all the way. It is much more efficient and less error-prone to place the configuration commands into a configuration file and using that as input to the administration program, but this is still cumbersome as you must type in each administration program in order. The ionstart program will automatically execute the appropriate administration programs with their respective configuration files in order. Unfortunately, as seen in the previous sections, the command is lengthy. This is why the ionscript script was added to make things even easier.
+The ionscript will basically concatenate the configuration files into one large file. The format of this large configuration file is simply to bookend configuration sections with the lines: ## begin PROGRAM and ## end PROGRAM, where PROGRAM is the name of the administration program for which the configuration commands should be sent (such as ionadmin, bpadmin, ipnadmin).
+To create a single file host1.rc out of the various configuration files defined in the previous section, run this command:
+ionscript -i host1.ionrc -p host1.ipnrc -l host1.ltprc -b host1.bprc -O host1.rc
The command can also be used to split the large host1.rc into the individual configuration files (so long as the large file is formatted correctly). Just run this command to revert the process:
+ionscript -i host1.ionrc -p host1.ipnrc -l host1.ltprc -b host1.bprc -I host1.rc
This isn't very practical in this specific case (as you already have the individual files) but if you start with a single configuration file, this can be helpful.
+Once you have a single configuration file, starting the ION node is a single command:
+ionstart -I host1.rc
Note that ionstart and ionscript require sed and awk, but those are almost universally available on Unix-based systems. The two scripts will always sanity-check the large configuration file to ensure that it interprets the bookend lines correctly- and it will warn you of any errors you have made in the file. Consult the USAGE for each script for further help, by attempting to run the script with no arguments or the -h argument.
+For a simple single-node ION configuration - running multiple instances of ION in the same host, see the tutorial here.
+For a two-node configuration, see the tutorial here.
+For a multi-hop and also multi-network configuration, see this page.
+ + + + + + + + + + + + + +exit commandLab testing of ION-based DTN network often uses only a single network. However, during deployment or operational testing, ION network must operate over multiple networks. To clarify how to configure ION, we consider the following hypothetical network configuration.
+The basic topology is illustrated here:
+ +-------+ protocol a protocol b +-------+
+ | | | |
+ | SC1 +-----+ +--->+ MOC1 |
+ | 21 | | | | 24 |
+ +-------+ | +-------+ | +-------+
+ +---->+ +----+
+ rfnet | GS | gsnet
+ +---->+ 23 +----+
+ +-------+ | +-------+ | +-------+
+ | | | | | |
+ | SC1 +-----+ +--->+ MOC1 |
+ | 22 | | 25 |
+ +-------+ +-------+
+
+subnet: 192.168.100.0/24 subnet:192.168.200.0/24
+rfnet (192.168.100.0/24) and gsnet (192.168.200.0/24).rfnet is protocol A and in gsnet, protocol B.ION associates each neighbor with an convergence layer protocol and an outduct. With the exception of UDP convergence layer, each outduct is associated with an induct as well.
+When there are multiple neighbors using the same convergence layer protocol, only one induct is used to 'pair' with multiple outducts of the same protocol.
+If neighbors using the same protocol are all within the same network, then the induct is associated with the IP address of the ION node on that particular network.
+If the neighbors using the same protocol are from multiple networks, then the induct will need to be associated with INADDR_ANY, 0.0.0.0:port defined for protocols such as TCP, UDP, and STCP.
+For LTP, however, multiple inducts can be defined for each network using the IP address of each network separately, and the induct for a network is called seat (see manual page for ltprc).
In this case, SC1 and SC2 communicates with GS using LTP, while MOC1 and MOC2 communicate with GS using TCP. The port we used is 4556.
+For GS, it defines TCP in this manner in the .bprc file:
a protocol tcp
+
+a induct tcp 192.168.200.23:4556 tcpcli
+
+a outduct tcp 192.168.200.24:4556 tcpclo
+a outduct tcp 192.168.200.25:4556 tcpclo
+
+a plan ipn:24.0
+a planduct ipn:24.0 tcp 192.168.200.24:4556
+a planduct ipn:25.0 tcp 192.168.200.25:4556
+There is only induct for the two outducts. Since node 23, 24, and 25 are in the 192.168.200.0/24 subnet, the induct for 23 can simply use its statically assigned IP address of 192.168.200.23:4556.
+For MOC1, TCP is specified in this manner in the .bprc file:
a protocol tcp
+
+a induct tcp 192.168.200.24:4556 tcpcli
+
+a outduct tcp 192.168.200.23:4556 tcpclo
+
+a plan ipn:23.0
+
+a planduct ipn:23.0 tcp 192.168.200.23:4556
+Since MOC1 has only 1 neighbor and uses TCP, the induct/outduct and egress plans are very much the standard configuration we see typically in a single network configuration.
+Similar configuration can be written for MOC2.
+For LTP, the configuration for GS is:
+# in bprc file
+a protocol ltp
+
+a induct ltp 23 ltpcli
+
+a outduct ltp 21 ltpclo
+a outduct ltp 22 ltpclo
+
+a plan ipn:21.0
+a plan ipn:22.0
+
+a planduct ipn:21.0 ltp 21
+a planduct ipn:22.0 ltp 22
+
+# in .ltprc file
+a span 21 100 100 1482 100000 1 'udplso 192.168.100.21:1113'
+a span 22 100 100 1482 100000 1 'udplos 192.168.100.22:1113'
+a seat 'udplsi 192.168.100.23:1113'
+
+s
+For ltp, a single induct is specified for the 192.168.100.0/24 subnet using the a seat (add seat) command. The older syntax is s 'udplsi 192.168.100.23:1113', which works only for the case of a single network and port combination. However, when extending LTP to multiple seats (inducts) when there are multiple networks or when there are multiple ports, the seat command offers the flexibility to support more complex configurations.
The syntax for LTP/STCP is identical, except replacing tcp with stcp, tcpcli and tcpclo with stcpcli and stcpclo in the configuration files.
When runing TCP or STCP over both networks, the only change is that for the GS node, the induct definitions in .bprc are replaced by:
+a induct tcp 0.0.0.0:4556 tcpcli and a induct stcp 0.0.0.0:4556 tcpcli
When running LTP over both networks, the only key difference is that in the .ltprc file for the GS node, two seats are defined:
a span 21 100 100 1482 100000 1 'udplso 192.168.100.21:1113'
+a span 22 100 100 1482 100000 1 'udplos 192.168.100.22:1113'
+a span 24 100 100 1482 100000 1 'udplos 192.168.200.24:1113'
+a span 25 100 100 1482 100000 1 'udplos 192.168.200.25:1113'
+a seat 'udplsi 192.168.100.23:1113'
+a seat 'udplsi 192.168.200.23:1113'
+
+s
+Of course the bprc file must also be updated to add reflect the additional LTP neighbors, but that extension is straightforward so we will not be listing them here.
+For ION, the use contact graph is optional when there is one hop. In that case, the data rate, normally defined in the contact graph, is provided through commands plan command in .bprc file.
When contact graph is present, the information in the contact graph supersedes the data rate specified in the plan command.
If there are no contact graph and the data rate is either 0 or omitted in the plan command, then there is no bundle level throttling of data.
exit commandWhen no contact graph is provided, only immediate neighbor can exchange data. If relay operation is stil desired, an exit command can be used. In the case of the topology presented earlier, the node GS can be a gateway between the rfnet and gsnet. So GS can be added as an exit node for identified pair of source-destination.
The following configurations can be downloaded (see file attachment)
+These configuration files are provided to give you a basic functional setup; they may not be sufficient to support all features and throughput performance you want to achieve for your network. So please use them as a template and apply update as necessary.
+ + + + + + + + + + + + + +Version 4.1.3
+Jay Gao, Jet Propulsion Laboratory, California Institute of +Technology
+Sky DeBaun, Jet Propulsion Laboratory, California Institute of +Technology
+Document Change Log
+| Ver No. | +Date | +Description | +Note | +
|---|---|---|---|
| V4.1.3 | +11/6/2023 | +Add LTP Performance Test | +Converted to markd down | +
| V4.1.2 | +1/5/2023 | +Added notes on SDR file and CGRM | ++ |
The effort required to deploy the Interplanetary Overlay Network (ION) +software in an operational setting may vary widely depending on the +scope of the deployment and the degree to which the required ION +functionality coincides with the capability provided by default in the +software as distributed. This effort will be expended in two general +phases: initial infusion and ongoing operation.
+Even in the best case, some minimal degree of configuration will be +required. Many elements of ION behavior are managed at run time by +decisions recorded in ION's protocol state databases, as populated by a +variety of administration utility programs. Others are managed at +compile time by means of compiler command-line switches selected when +the software is built. These compile-time configuration options are +described in the Configuration section below.
+In some cases, mission-specific behavior that goes beyond the options +built into ION must be enabled during ION deployment. The intent of the +ION design is to minimize -- to eliminate, if possible -- any need to +modify ION source code in order to enable mission-specific behavior. Two +general strategies are adopted for this purpose.
+First, ION includes a number of conditionally defined functions that can +be cleanly replaced with mission-specific alternative source code by +setting a compiler command-line switch at build time. Setting such a +switch causes the mission-specific source code, written in C, to be +simply included within the standard ION source code at the time of +compilation.
+Second, more generally it is always possible to add new application +executables, new startup/shutdown/monitor/control utilities or scripts, +and even entirely new route computation systems, BP convergence-layer +adapters, and/or LTP link service adapters without ever altering the +distributed ION source code. A few rough guidelines for making these +kinds of modifications are described in the Adaptation section +below.
+Finally, in rare cases it may be necessary to execute ION in an +operating-system environment to which it has not yet been ported. +Guidance for porting ION to new platforms will be provided in a future +edition of this Deployment Guide.
+On an ongoing basis, an ION deployment may require reconfiguration from +time to time and/or may require troubleshooting to resolve performance +or stability problems. Some suggestions for reconfiguration and +troubleshooting procedures are offered in the Operation section +below.
+Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DFSWSOURCE or +--DSM_SEMBASEKEY=0xff13), will alter the functionality of ION as noted +below.
+PRIVATE_SYMTAB
This option causes ION to be built for VxWorks 5.4 or RTEMS with +reliance on a small private local symbol table that is accessed by means +of a function named sm_FindFunction. Both the table and the function +definition are, by default, provided by the symtab.c source file, which +is automatically included within the platform_sm.c source when this +option is set. The table provides the address of the top-level function +to be executed when a task for the indicated symbol (name) is to be +spawned, together with the priority at which that task is to execute and +the amount of stack space to be allocated to that task.
+PRIVATE_SYMTAB is defined by default for RTEMS but not for VxWorks 5.4.
+Absent this option, ION on VxWorks 5.4 must successfully execute the +VxWorks symFindByName function in order to spawn a new task. For this +purpose the entire VxWorks symbol table for the compiled image must be +included in the image, and task priority and stack space allocation must +be explicitly specified when tasks are spawned.
+FSWLOGGER
This option causes the standard ION logging function, which simply +writes all ION status messages to a file named ion.log in the current +working directory, to be replaced (by #include) with code in the source +file fswlogger.c. A file of this name must be in the inclusion path for +the compiler, as defined by --Ixxxx compiler option parameters.
+FSWCLOCK
This option causes the invocation of the standard time function within +getUTCTime (in ion.c) to be replaced (by #include) with code in the +source file fswutc.c, which might for example invoke a mission-specific +function to read a value from the spacecraft clock. A file of this name +must be in the inclusion path for the compiler.
+FSWWDNAME
This option causes the invocation of the standard getcwd function within +cfdpInit (in libcfdpP.c) to be replaced (by #include) with code in the +source file wdname.c, which must in some way cause the mission-specific +value of current working directory name to be copied into +cfdpdbBuf.workingDirectoryName. A file of this name must be in the +inclusion path for the compiler.
+FSWSYMTAB
If the PRIVATE_SYMTAB option is also set, then the FSWSYMTAB option +causes the code in source file mysymtab.c to be included in +platform_sm.c in place of the default symbol table access implementation +in symtab.c. A file named mysymtab.c must be in the inclusion path for +the compiler.
+FSWSOURCE
This option simply causes FSWLOGGER, FSWCLOCK, FSWWDNAME, and FSWSYMTAB +all to be set.
+GDSLOGGER
This option causes the standard ION logging function, which simply +writes all ION status messages to a file named ion.log in the current +working directory, to be replaced (by #include) with code in the source +file gdslogger.c. A file of this name must be in the inclusion path for +the compiler, as defined by --Ixxxx compiler option parameters.
+GDSSOURCE
This option simply causes GDSLOGGER to be set.
+TRACKRFXEVENTS
This option causes user-written code, in a file named rfxtracker.c, to +be executed every time the rfxclock daemon dispatches a schedule RFX +event such as the start or end of a transmission contact. A file of this +name must be in the inclusion path for the compiler, as defined by +--Ixxxx compiler option parameters.
+ION_OPS_ALLOC=*xx*
This option specifies the percentage of the total non-volatile storage +space allocated to ION that is reserved for protocol operational state +information, i.e., is not available for the storage of bundles or LTP +segments. The default value is 40.
+ION_SDR_MARGIN=*xx*
This option specifies the percentage of the total non-volatile storage +space allocated to ION that is reserved simply as margin, for +contingency use. The default value is 20.
+The sum of ION_OPS_ALLOC and ION_SDR_MARGIN defines the amount of +non-volatile storage space that is sequestered at the time ION +operations are initiated: for purposes of congestion forecasting and +prevention of resource oversubscription, this sum is subtracted from the +total size of the SDR "heap" to determine the maximum volume of space +available for bundles and LTP segments. Data reception and origination +activities fail whenever they would cause the total amount of data store +space occupied by bundles and segments to exceed this limit.
+HEAP_PTRS
This is an optimization option for the SDR non-volatile data management +system: when set, it enables the value of any variable in the SDR heap +to be accessed directly by means of a pointer into the dynamic memory +that is used as the data store storage medium, rather than by reading +the variable into a location in local stack memory. Note that this +option must not be enabled if the data store is configured for file +storage only, i.e., if the SDR_IN_DRAM flag was set to zero at the time +the data store was created by calling sdr_load_profile. See the +ionconfig(5) man page in Appendix A for more information.
+NO_SDR_TRACE
This option causes non-volatile storage utilization tracing functions to +be omitted from ION when the SDR system is built. It disables a useful +debugging option but reduces the size of the executable software.
+NO_PSM_TRACE
This option causes memory utilization tracing functions to be omitted +from ION when the PSM system is built. It disables a useful debugging +option but reduces the size of the executable software.
+IN_FLIGHT
This option controls the behavior of ION when an unrecoverable error is +encountered.
+If it is set, then when an unrecoverable error is encountered the status +message "Unrecoverable SDR error" is logged and the SDR non-volatile +storage management system is globally disabled: the current data store +access transaction is ended and (provided transaction reversibility is +enabled) rolled back, and all ION tasks terminate.
+Otherwise, the ION task that encountered the error is simply aborted, +causing a core dump to be produced to support debugging.
+SM_SEMKEY=0x*XXXX*
This option overrides the default value (0xee01) of the identifying +"key" used in creating and locating the global ION shared-memory system +mutex.
+SVR4_SHM
This option causes ION to be built using svr4 shared memory as the +pervasive shared-memory management mechanism. svr4 shared memory is +selected by default when ION is built for any platform other than MinGW +(for which File Mapping objects are used), VxWorks 5.4, or RTEMS. (For +the latter two operating systems all memory is shared anyway, due to the +absence of a protected-memory mode.)
+POSIX1B_SEMAPHORES
This option causes ION to be built using POSIX semaphores as the +pervasive semaphore mechanism. POSIX semaphores are selected by default +when ION is built for RTEMS but are otherwise not used or supported; +this option enables the default to be overridden.
+SVR4_SEMAPHORES
This option causes ION to be built using svr4 semaphores as the +pervasive semaphore mechanism. svr4 semaphores are selected by default +when ION is built for any platform other than MinGW (for which Windows +event objects are used), VxWorks 5.4 (for which VxWorks native +semaphores are the default choice), or RTEMS (for which POSIX semaphores +are the default choice).
+SM_SEMBASEKEY=0x*XXXX*
This option overrides the default value (0xee02) of the identifying +"key" used in creating and locating the global ION shared-memory +semaphore database, in the event that svr4 semaphores are used.
+SEMMNI=*xxx*
This option declares to ION the total number of svr4 semaphore sets +provided by the operating system, in the event that svr4 semaphores are +used. It overrides the default value, which is 128. (Changing this value +typically entails rebuilding the O/S kernel.)
+SEMMSL=*xxx*
This option declares to ION the maximum number of semaphores in each +svr4 semaphore set, in the event that svr4 semaphores are used. It +overrides the default value, which is 250. (Changing this value +typically entails rebuilding the O/S kernel.)
+SEMMNS=*xxx*
This option declares to ION the total number of svr4 semaphores that the +operating system can support; the maximum possible value is SEMMNI x +SEMMSL. It overrides the default value, which is 32000. (Changing this +value typically entails rebuilding the O/S kernel.)
+Note that this option is also supported in the MinGW (Windows) port of +ION, with the same default value; changing this value does not +involve an operating system modification.
+ION_NO_DNS
This option causes the implementation of a number of Internet socket I/O +operations to be omitted for ION. This prevents ION software from being +able to operate over Internet connections, but it prevents link errors +when ION is loaded on a spacecraft where the operating system does not +include support for these functions.
+ERRMSGS_BUFSIZE=*xxxx*
This option set the size of the buffer in which ION status messages are +constructed prior to logging. The default value is 4 KB.
+SPACE_ORDER=*x*
This option declares the word size of the computer on which the compiled +ION software will be running: it is the base-2 log of the number of +bytes in an address. The default value is 2, i.e., the size of an +address is 2^2^ = 4 bytes. For a 64-bit machine, SPACE_ORDER must be +declared to be 3, i.e., the size of an address is 2^3^ = 8 bytes.
+NO_SDRMGT
This option enables the SDR system to be used as a data access +transaction system only, without doing any dynamic management of +non-volatile data. With the NO_SDRMGT option set, the SDR system library +can (and in fact must) be built from the sdrxn.c source file alone.
+DOS_PATH_DELIMITER
This option causes ION_PATH_DELIMITER to be set to '\' (backslash), for +use in the construction of path names. The default value of +ION_PATH_DELIMITER is '/' (forward slash, as is used in Unix-like +operating systems).
+Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DUDP_MULTISEND, will +alter the functionality of LTP as noted below.
+UDP_MULTISEND
The UDP_MULTISEND option can improve LTP performance by sharply reducing +system call overhead: multiple LTP segments encapsulated in UDP +datagrams may be transmitted with a single sendmmsg() call rather than +multiple sendmsg() calls. This reduces the cost of sending LTP blocks in +small segments, which in turn can limit IP fragmentation for LTP +traffic.
+Note that sendmmsg() has no built-in rate control and offers no +opportunity to exercise the rate control algorithm that minimizes UDP +congestion loss in non-MULTISEND LTP. In order to achieve similar +reduction in UDP congestion loss, a node that receives data sent by +sendmmsg() may need to be configured for larger socket buffers. The +sysctl operating system utility may be used for this purpose, setting +new values for net.core.rmem_max and _default and net.core.wmem_max and +_default.
+Note also that not all operating systems support the sendmmsg() system +call. ION currently enables UDP_MULTISEND only for flavors of Linux +other than bionic.
+MULTISEND_SEGMENT_SIZE
By default, ION LTP in UDP_MULTISEND mode will always limit LTP segment +size to 1450 so that every segment may be encapsulated in an IP packet +whose size does not exceed the standard Ethernet frame size. For +networks in which the MTU is known to be larger, this parameter may be +overridden at compile time.
+MULTISEND_BATCH_LIMIT
By default, the maximum number of UDP datagrams that ION LTP in +UDP_MULTISEND mode will send in a single sendmmsg() call is +automatically computed as the block aggregation size threshold divided +by the maximum segment size; that is, normally the amount of data sent +per sendmmsg() call is about one LTP block. This parameter may be +overridden at compile time.
+MULTIRECV_BUFFER_COUNT
In UDP_MULTISEND mode, ION LTP will also use recvmmsg() to receive +multiple LTP segments (encapsulated in UDP datagrams) in a single system +call. By default, 127 segment reception buffers (each one large enough +to receive a single LTP segment of maximum size) are reserved for this +purpose. This parameter may be overridden at compile time.
+Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DION_NOSTATS or +--DBRSTERM=60), will alter the functionality of BP as noted below.
+TargetFFS
Setting this option adapts BP for use with the TargetFFS flash file +system on the VxWorks operating system. TargetFFS apparently locks one +or more system semaphores so long as a file is kept open. When a BP task +keeps a file open for a sustained interval, subsequent file system +access may cause a high-priority non-BP task to attempt to lock the +affected semaphore and therefore block; in this event, the priority of +the BP task may automatically be elevated by the inversion safety +mechanisms of VxWorks. This "priority inheritance" can result in +preferential scheduling for the BP task -- which does not need it -- at +the expense of normally higher-priority tasks, and can thereby introduce +runtime anomalies. BP tasks should therefore close files immediately +after each access when running on a VxWorks platform that uses the +TargetFFS flash file system. The TargetFFS compile-time option ensures +that they do so.
+MULTIDUCTS
It is possible for multiple outducts to be attached to a single egress
+plan, enabling some bundles to be forwarded to a neighboring node using
+one outduct while others are forwarded using another. Selection of the
+outduct to use for the forwarding of a given bundle is a function of the
+bpclm "convergence-layer manager" daemon; each of a given node's egress
+plans is managed by a single bpclm task. The default outduct selection
+algorithm exercised by bpclm can be overridden by means of the
+MULTIDUCTS compile-time configuration option. Setting the -DMULTIDUCTS
+switch causes the standard outduct configuration logic in the
+outductSelected() function of bpclm.c to be replaced (by #include) with
+code in the source file named selectcla.c. A file of this name must be
+in the inclusion path for the compiler, as defined by --Ixxxx compiler
+option parameters.
The implementation of outductSelected() in ION bpv7 implementation
+differs somewhat from that in the bpv6 implementation. The content of a
+very simple selectcla.c file for a node deploying bpv7 might look like
+this:
if (bundle-\>destination.ssp.ipn.serviceNbr == 99)
+{
+ if (strcmp(protocol-\>name, "bssp") == 0)
+ {
+ return 1; /\* Use a BSSP outduct for this bundle. \*/
+ }
+}
+Note that any element of the state of the bundle may be used to select +an outduct based on any element of the state of the outduct. The intent +is for ION to be able to accommodate virtually any mission-defined +algorithm for selecting among communication channels between +topologically adjacent BP nodes.
+BRSTERM=*xx*
This option sets the maximum number of seconds by which the current time +at the BRS server may exceed the time tag in a BRS authentication +message from a client; if this interval is exceeded, the authentication +message is presumed to be a replay attack and is rejected. Small values +of BRSTERM are safer than large ones, but they require that clocks be +more closely synchronized. The default value is 5.
+ION_NOSTATS
Setting this option prevents the logging of bundle processing statistics +in status messages.
+KEEPALIVE_PERIOD=*xx*
This option sets the number of seconds between transmission of +keep-alive messages over any TCP or BRS convergence-layer protocol +connection. The default value is 15.
+ION_BANDWIDTH_RESERVED
Setting this option overrides strict priority order in bundle +transmission, which is the default. Instead, bandwidth is shared between +the priority-1 and priority-0 queues on a 2:1 ratio whenever there is no +priority-2 traffic.
+Defining the following macros, by setting parameters that are provided +to the C compiler (for example, -DNOEXPAT or --DAMS_INDUSTRIAL), will +alter the functionality of AMS as noted below.
+NOEXPAT
Setting this option adapts AMS to expect MIB information to be presented +to it in "amsrc" syntax (see the amsrc(5) man page) rather than in XML +syntax (as described in the amsxml(5) man page), normally because the +expat XML interpretation system is not installed. Note that the default +syntax for AMS MIB information is now amsrc syntax so the -DNOEXPAT +switch is rarely needed.
+AMS_INDUSTRIAL
Setting this option adapts AMS to an "industrial" rather than +safety-critical model for memory management. By default, the memory +acquired for message transmission and reception buffers in AMS is +allocated from limited ION working memory, which is fixed at ION +start-up time; this limits the rate at which AMS messages may be +originated and acquired. When --DAMS_INDUSTRIAL is set at compile time, +the memory acquired for message transmission and reception buffers in +AMS is allocated from system memory, using the familiar malloc() and +free() functions; this enables much higher message traffic rates on +machines with abundant system memory where flight software constraints +on dynamic system memory allocation are not applicable.
+Defining the following macro, by setting a parameter that is provided to +the C compiler (i.e., --DTargetFFS), will alter the functionality of +CFDP as noted below.
+TargetFFS
Setting this option adapts CFDP for use with the TargetFFS flash file +system on the VxWorks operating system. TargetFFS apparently locks one +or more system semaphores so long as a file is kept open. When a CFDP +task keeps a file open for a sustained interval, subsequent file system +access may cause a high-priority non-CFDP task to attempt to lock the +affected semaphore and therefore block; in this event, the priority of +the CFDP task may automatically be elevated by the inversion safety +mechanisms of VxWorks. This "priority inheritance" can result in +preferential scheduling for the CFDP task -- which does not need it -- +at the expense of normally higher-priority tasks, and can thereby +introduce runtime anomalies. CFDP tasks should therefore close files +immediately after each access when running on a VxWorks platform that +uses the TargetFFS flash file system. The TargetFFS compile-time option +assures that they do so.
+ION requires several runtime configuration settings to be defined at the +time a node is initialized. Most notable are the settings for the Admin +functions of ION. ION provides a variety of administration utilities +including ionadmin, ionsecadmin, ltpadmin, bsspadmin, bpadmin, ipnadmin, +and cfdpadmin. Each of the corresponding modules that is to be used at +runtime will need to be configured. The commands that perform these +configuration tasks are normally presented to the admin utility in an +admin configuration file.
+In the Linux environment, two different styles of configuration files +are possible. Both styles are accepted by the "ionstart" program that +installs as part of the official release, an AWK program. The first +style requires that all configuration commands for all in-use admins +will be stored in one file. This single file is sectioned off internally +to separate the commands of each admin. The ionstart program accepts +this single configuration file's name as a parameter, parses this file +looking for sectioned-off areas for each possible admin function, and +then uses the commands within these sections to configure the +corresponding modules.
+The other style requires that each admin will have its own distinct +configuration file. The ionstart program consumes these files as guided +by command line switches and parameters identifying each configuration +file.
+Some ION configuration parameters are declared only at node +initialization time; they cannot later be revised. In particular, the +ionadmin "1" (the numeral one) initialization command must be executed +just once, before any other configuration command is processed. The +first parameter to this command is required and is a numeric value that +indicates the node number of the DTN node being configured. The second +parameter to this command is optional; if present, it must provide the +full pathname of a local file of immutable configuration parameter +values:
+wmKey (integer)
+wmSize (integer)
+wmAddress (integer)
+sdrName (string)
+
+sdrWmSize (integer)
+# bit pattern in integer form, e.g., 3 for 00000011
+configFlags 3
+heapWords (integer)
+heapKey (integer)
+pathName (string)
+This path name should NOT be enclosed in any type of quotation marks. +The file is a text file with 2 fields per line; lines are processed in +sequence. The first field on each line holds one of the parameter +identifier text strings as shown above. The second field holds the value +that will be placed into the identified parameter. Make sure that the +data type specified in the second field matches the type expected.
+For documentation on these parameters, see the ionconfig(5) man page.
+The configFlags entry controls several features of the Simple Data Recorder (SDR). There are several flags of interest:
#define SDR_IN_DRAM 1 /* Write to & read from memory. */
+#define SDR_IN_FILE 2 /* Write file; read file if nec. */
+#define SDR_REVERSIBLE 4 /* Transactions may be reversed. */
+SDR_IN_DRAM is required for normal ION operation and should virtually +always be specified.
+When SDR_REVERSIBLE is specified, SDR transactions that fail (e.g., due +to memory allocation failure) are rolled back, allowing transactions to +fail gracefully without corrupting the ION databases. If the flag is +not supplied, failed transactions will cause an immediate task failure +and, where supported, a core dump. This feature is intended only as an +aid to debugging; in operations ION should normally be configured with +reversible transactions. When transaction reversibility is enabled, ION +creates & manages a log file in the directory named by \"pathName\" +which must be writable by ION and which tracks the SDR changes and +supports rollback to the last consistent state. The filesystem for this +directory should be high-performance; a ramdisk is usually ideal. The +maximum size of the logfile is dependent upon the largest transaction in +the SDR, and is therefore of a size on the same order of magnitude as +the largest bundle. NOTE that if the directory named by "pathname" does +not exist then transaction reversibility will be disabled automatically; +a message to this effect will be written to the ION log file.
+When SDR_IN_FILE is specified, ION creates a file in the \"pathName\"
+directory, which is maintained as a copy of the SDR heap in DRAM;
+whenever the SDR heap in memory is modified, the changes are also
+written to the sdr heap file. Thus the heap file is always the same
+size as the in-memory heap. Again, if the directory named by "pathname"
+does not exist then retention of the ION SDR heap in a file will be
+disabled automatically; a message to this effect will be written to the
+ION log file. NOTE that
SDR_IN_FILE may have the adverse effect of slowing down
+ all SDR transactions, which can significantly impact transmission,
+ relay, and reception speed of ION. Users should conduct performance
+ testing to ensure that keeping SDR in file can still achieve the
+ operational performance expected.SDR_IN_FILE option is that in the case of ION
+ shutdown due to power reset where the state of SDR is not
+ corrupted, it is possible to start ION with the SDR file and
+ resume data transfer operations such as LTP transaction. However, if
+ ION shuts down due to an internal error, then it is not recommended
+ to keep the SDR file when restarting ION, as the SDR state is not
+ certain to be without corruption.When ION stores a bundle, it typically holds part of the bundle part in +memory (heap) as determined by the maxheap parameter in bprc. The +default value is about 650 bytes. The rest of the payload is placed into +file reference. Also a bundle, before transmission, kept its header and +extensions inside a data structure for quick look up and manipulations; +the bundle is serialized into a chunk of octet according to the standard +just prior to transmission. Therefore, when a bundle is stored in an ION +node, part of its footprint is in the 'heap' and part of it is in the +'working memory.'
+Test shows that leaving the maxHeap parameter to its default value, a
+bundle uses about 1.5KB of space in heap and about 100-150 Byte in
+working memory. Adding a 200% margin, we recommend that following
+relationship between heapWords and wmSize:
where 3 is the margin, 8 is the number of octets per word, 0.4 accounts +for the fact that inbound and outbound heap space is only 40% of the +heap, and 10 accounts for the empirically estimated 10:1 ratio between heap and working memory footprints per bundle.
+Many ION runtime configuration parameters can be declared at node +initialization and later revised dynamically, either by passing +supplementary configuration files to the admin utilities or by +exercising the admin utilities in interactive mode.
+For documentation on the admin commands, see the man pages. The man page
+names are in the form of <xxx>rc, where <xxx> gets replaced by the specific
+module name (bp, dtn2, ion, ionsec, ipn, ltp, bssp, cfdp). The
+directories in which to find these files are: ./ici/doc/pod5,
+./ltp/doc/pod5, ./bssp/doc/pod5, ./bp/doc/pod5, and ./cfdp/doc/pod5.
Normally the instantiation of ION on a given computer establishes a +single ION node on that computer, for which hard-coded values of wmKey +and sdrName (see ionconfig(5)) are used in common by all executables to +assure that all elements of the system operate within the same state +space. For some purposes, however, it may be desirable to establish +multiple ION nodes on a single workstation. (For example, constructing +an entire self-contained DTN network on a single machine may simplify +some kinds of regression testing.) ION supports this configuration +option as follows:
+ION_NODE_LIST_DIR is defined in the
+ environment of every participating ION process. Moreover, the value
+ assigned to this variable must be the same text string in the
+ environments of all participating ION processes. That value must be
+ the name (preferably, fully qualified) of the directory in which the
+ ION multi-node database file "ion_nodes" will reside.ION_NODE_LIST_DIR makes it possible to establish
+ up to one ION node per directory rather than just one ION node on
+ the computer. When ionadmin is used to establish a node, the
+ ionInitialize() function will get that node's wmKey and sdrName from
+ the .ionconfig file, use them to allocate working memory and create
+ the SDR data store, and then write a line to the ion_nodes file
+ noting the nodeNbr, wmKey, sdrName, and wdName for the node it just
+ initialized. wdName is the current working directory in which
+ ionadmin was running at the time it called ionInitialize(); it
+ is the directory within which the node resides.Included in the root directory of the ION distribution is a bash script +named ionstop. This script executes each of the "admin" utilities +and instructs each subsystem to exit, by supplying the dummy command +file name ".". Once all of the utilities have exited, the script calls +another script named killm (likewise located in the root directory +of ion-open-source). The killm script first tries to kill all ION +processes by name, then tries to destroy all of the shared-memory +resources allocated to ION at the time the node was created.
+There are also many "local" versions of ionstop script, stored in test +subdirectories out of which one or multiple ION instances were launched +on the same host. These local versions of ionstop script differ from the +ionstop script in the root directory in that it usually contains (a) +additional, customized scripts to clean up test artifacts such as +ion.log, sdr file, received test files, and temporarily acquisition +files for LTP and BP that remained after a test is completed and (b) it +generally does not execute kilim, which will kill all ION processes, not +just the ones related to the ION instance being terminated.
+If you invoke the ionstop script that is part of the ION root directory, +it does not clean up test artifacts or other products created during +operation and if it detects that there are multiple ION instances +running in the same host, it will NOT execute killm. In that case, the +user is advised to always check that all processes are terminated +properly and that the shared memory is cleared appropriately.
+When running ionstop, various administrative programs will process a
+dummy command file "." that signals shutdown. It will first check the
+value of the environment variable ION_NODE_WDNAME, defined in the
+current shell, to determine which instance of ION must be taken down.
+The ION instance that was shutdown does not depend on the current
+directory the shell is in. Therefore it is possible to use either the
+ionstop script provided in ION's root directory or a local, customized
+version to shutdown an individual ION instance.
If you are having trouble shutting an ION node down, see the notes on +"Destroying a Node" later in this Guide.
+It has been pointed out that if you are running ION in a Docker +container inside a Kubernetes pod, the system is likely to assign +process ID 1 to one of the ION processes at startup; since process 1 +cannot be killed, the ionstop script can't complete and your node +will not be cleanly destroyed. One solution seems to be to use dumb-init +for the docker container:
+++[https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html]{.underline}
+
To make this work, you may have to override your entry point in the +manifest file used by the Kubectl "apply" command.
+When you define a DTN node, you do so using ionadmin and its Initialize +command (using the token '1'). This node is then referenced by its node +number throughout the rest of the configuration file.
+## begin ionadmin
+1 1 /home/spwdev/cstl/ion-configs/23/badajoz/3node-udp-ltp/badajoz.ionconfig
+s
+
+a contact +1 +86400 25 25 50000000
+a contact +1 +84600 25 101 50000000
+a contact +1 +84600 25 1 50000000
+
+a contact +1 +86400 101 25 50000000
+a contact +1 +86400 101 101 50000000
+a contact +1 +86400 101 1 50000000
+
+a contact +1 +86400 1 25 50000000
+a contact +1 +86400 1 101 50000000
+a contact +1 +86400 1 1 50000000
+
+
+a range +1 +86400 25 25 1
+a range +1 +86400 25 101 10
+a range +1 +86400 25 1 10
+
+a range +1 +86400 101 25 10
+a range +1 +86400 101 101 1
+a range +1 +86400 101 1 10
+
+a range +1 +86400 1 25 10
+a range +1 +86400 1 101 10
+a range +1 +86400 1 1 1
+
+m production 50000000
+m consumption 50000000
+
+## end ionadmin
+##########################################################################
+## begin ltpadmin
+1 32
+
+a span 25 1 1 1400 1400 1 'udplso 192.168.1.25:1113 1000000000'
+a span 101 1 1 1400 1400 1 'udplso 192.168.1.101:1113 1000000000'
+a span 1 1 1 1400 1400 1 'udplso 192.168.1.1:1113 1000000000'
+
+s 'udplsi 192.168.1.1:1113'
+
+## end ltpadmin
+##########################################################################
+## begin bpadmin
+1
+
+a scheme ipn 'ipnfw' 'ipnadminep'
+
+a endpoint ipn:1.0 q
+a endpoint ipn:1.1 q
+a endpoint ipn:1.2 q
+
+a protocol ltp 1400 100
+a protocol tcp 1400 100
+
+a outduct ltp 25 ltpclo
+a outduct ltp 101 ltpclo
+a outduct ltp 1 ltpclo
+
+a induct ltp 1 ltpcli
+
+s
+## end bpadmin
+##########################################################################
+## begin ipnadmin
+
+a plan 25 ltp/25
+a plan 101 ltp/101
+a plan 1 ltp/1
+
+## end ipnadmin
+The "ipn" scheme for URI formation is generally used to form the +endpoint IDs of endpoints in an ION deployment. Any transmission using +endpoints formed in the "ipn" scheme will have endpoints IDs of this +form:
+++ipn:nodenumber. servicenumber.
+
The Add Scheme command on line 51 below specifies that the "ipn" +endpoint naming scheme is supported; the names of three endpoints formed +in this scheme are shown in lines 53 thru 55.
+The two remaining parameters on this command are used to define the +software functions that will act as data forwarder and administrative +data receiver.
+++a scheme scheme_name forwarder_command admin_app_command
+
The add scheme command. This command declares an endpoint naming +\"scheme\" for use in endpoint IDs, which are structured as URIs: +scheme_name:scheme-specific_part. forwarder_command will be +executed when the scheme is started on this node, to initiate operation +of a forwarding daemon for this scheme. admin_app_command will also be +executed when the scheme is started on this node, to initiate operation +of a daemon that opens an administrative endpoint identified within this +scheme so that it can receive and process custody signals and bundle +status reports.
+Starting at line 71, the egress plans are defined. These determine the +outducts by which data are sent to nodes that are topologically adjacent +to the current node in the DTN-based network.
+++a plan node_nbr default_duct_expression
+
The add plan command. This command establishes an egress plan for the +bundles that must be transmitted to the neighboring node identified by +node_nbr. Each duct expression is a string of the form
+++protocol_name/outduct_name
+
signifying that the bundle is to be queued for transmission via the +indicated convergence layer protocol outduct.
+The ltpadmin utility allows the features of the LTP protocol to become +available. For details of the LTP protocol, see RFC 5325.
+The first command that must be issued to ltpadmin is the Initialize +command (see line number 38 below, the command token is the '1' (one)). +The sole parameter passed to this command is est_max_export_sessions.
+This command uses est_max_export_sessions to configure the hash table +it will use to manage access to export transmission sessions that are +currently in progress. (For optimum performance, +est_max_export_sessions should normally equal or exceed the summation +of max_export_sessions over all spans as discussed below.)
+Appropriate values for this parameter and for the parameters configuring +each \"span\" of potential LTP data exchange between the local LTP and +neighboring engines are non-trivial to determine. See the ION LTP +configuration spreadsheet and accompanying documentation for details.
+- Essentially, the "max export sessions" must be >= the total number +of export sessions on all the spans. If it is expected that new spans +will be added during an ION session, then max export sessions figure +should be large enough to cover the maximum # of sessions possible.
+- Next to be defined are the Spans. They define the interconnection +between two LTP engines. There are many parameters associated with the +Spans.
+a span peer_engine_nbr max_export_sessions max_import_sessions +max_segment_size aggregation_size_threshold aggregation_time_limit +'LSO_command' [queuing_latency]
+The "add span" command. This command declares that a span of potential +LTP data interchange exists between the local LTP engine and the +indicated (neighboring) LTP engine.
+The max_segment_size and the aggregation_size_threshold are +expressed as numbers of bytes of data. max_segment_size limits the +size of each of the segments into which each outbound data block will be +divided; typically this limit will be the maximum number of bytes that +can be encapsulated within a single transmission frame of the underlying +link service. max_segment_size specifies the largest LTP segment that +this span will produce.
+aggregation_size_threshold limits the number of LTP service data units +(e.g., bundles) that can be aggregated into a single block: when the sum +of the sizes of all service data units aggregated into a block exceeds +this limit, aggregation into this block must cease and the block must be +segmented and transmitted. When numerous small bundles are outbound, +they are aggregated into a block of at least this size instead of being +sent individually.
+aggregation_time_limit alternatively limits the number of seconds that +any single export session block for this span will await aggregation +before it is segmented and transmitted, regardless of size. The +aggregation time limit prevents undue delay before the transmission of +data during periods of low activity. When a small number of small +bundles are outbound, they are collected until this time limit is met, +whereupon the aggregated quantity is sent as a single, larger block.
+max_export_sessions constitutes the size of the local LTP engine's +retransmission \"window\" for this span. The retransmission windows of +the spans impose flow control on LTP transmission, preventing the +allocation of all available space in the ION node's data store to LTP +transmission sessions.
+The max_import_sessions parameter is simply the neighboring engine's +own value for the corresponding export session parameter.
+LSO_command is script text that will be executed when LTP is started +on this node, to initiate operation of a link service output task for +this span. Note that peer_engine_nbr will automatically be appended to +LSO_command by ltpadmin before the command is executed, so only the +link-service-specific portion of the command should be provided in the +LSO_command string itself.
+queuing_latency is the estimated number of seconds that we expect to +lapse between reception of a segment at this node and transmission of an +acknowledging segment, due to processing delay in the node. (See the 'm +ownqtime' command below.) The default value is 1.
+If queuing_latency is a negative number, the absolute value of this +number is used as the actual queuing latency and session purging is +enabled; otherwise session purging is disabled. If session purging is +enabled for a span then at the end of any period of transmission over +this span all of the span's export sessions that are currently in +progress are automatically canceled. Notionally this forces +re-forwarding of the DTN bundles in each session's block, to avoid +having to wait for the restart of transmission on this span before those +bundles can be successfully transmitted.
+Additional notes:
+- A "session block" is filled by outbound bundles until its aggregation +size threshold is reached, or its aggregation time limit is reached, +whereupon it is output as a series of segments (of size bounded by +max_segment_size). This series of segments is reliably transferred via +a LTP protocol session with the remote node, one session per block. By +adjusting the size of the session block, the rate of arrival of response +segments from the remote node can be controlled. Assuming a bundle rate +sufficient to fill the session block, a large session block size means a +lot of LTP segments per session (good for a high-rate return, low-rate +forward link situation). A small session block size means the number of +segments per session is smaller and the LTP protocol will complete the +block transfer more quickly because the number of segment retries is +generally smaller.
+- A good starting point for a configuration is to set the aggregation +size threshold to the number of bytes that will typically be transmitted +in one second, so that blocks are typically clocked out about once per +second. The maximum number of export sessions then should be at least +the total number of seconds in the round-trip time for traffic on this +LTP span, to prevent transmission from being blocked due to inability to +start another session while waiting for the LTP acknowledgment that can +end one of the current sessions.
+- The multiplicity of session blocks permits bundles to stream; while +one session block is being transmitted, a second can be filled (and +itself transmitted) before the first is completed. By increasing the +number of blocks, high latency links can be filled to capacity (provided +there is adequate bandwidth available in the return direction for the +LTP acknowledgments). But it is desirable to reduce the +max_export_sessions to a value where "most" of the sessions are employed +because each session allocates an increment of buffer memory from the +SDR whether it is used or not.
+- When a session block is transmitted, it is emitted as a series of +back-to-back LTP segments that are simply queued for transmission; LTP +does not meter segment issuance in any way. The underlying link layer is +expected to pop segments from the queue and transmit them at the current +rate as indicated in the contact plan. The udplso task does limit the +task's rate of segment transmission over UDP/IP to the transmission rate +declared in the contact plan, reducing the incidence of UDP congestion +loss.
+- Note that an LTP session can only be concluded (enabling space +occupied by the block to be recycled) when all segments have been +successfully received -- or retransmission limits have been reached and +the session is canceled. High bit error rates on the link correlate to +high rates of data loss when segments are large and/or blocks comprise +large numbers of segments; this typically results in larger numbers of +NACK/retransmit cycles, retarding session completion. When bit error +rates are high, LTP performance can be improved by reducing segment size +and/or aggregation size threshold.
+++s 'LSI command'
+
This command starts link service output tasks for all LTP spans (to +remote engines) from the local LTP engine, and it starts the link +service input task for the local engine.
+The sole command on line number 44 below starts two main operations +within LTP. The first of these operations starts all of the link service +output tasks, the ones defined for each LTP span (see the LSO_command +parameter of the Add Span command). In this example, each task +instantiates the same function (named 'udplso'). Each 'udplso' needs a +destination for its transmissions and these are defined as hostname or +IP Address (192.168.1.1) and port number (nominally 1113, the +pre-defined default port number for all LTP traffic).
+The second operation started by this command is to instantiate the link +service input task. In this instance, the task is named "udplsi". It is +through this task that all LTP input traffic will be received. Similar +to the output tasks, the input task also needs definition of the +interface on which LTP traffic will arrive, namely hostname or IP +address (192.168.1.1) and port number (1113). If it is necessary for +udplsi to listen on multiple network interfaces simultaneously, \'udplsi +0.0.0.0[:port]\' can be invoked. This instructs udplsi to listen to +the UDP broadcast address, which aggregates traffic from all available +network interfaces, including localhost.
+Once the LTP engine has been defined, initialized and started, we need a +definition as to how data gets routed to the Convergence Layer Adaptors. +Defining a protocol via bpadmin is the first step in that process.
+++a protocol protocol_name payload_bytes_per_frame +overhead_bytes_per_frame
+
The "add protocol" command. This command establishes access to the named +convergence layer protocol at the local node. As noted earlier, the +payload_bytes_per_frame and overhead_bytes_per_frame arguments were +previously used in calculating the estimated transmission capacity +consumption of each bundle, to aid in route computation and congestion +forecasting; in later versions of ION they are not needed and may be +omitted.
+Once the protocol has been defined, it can be used to define ducts, both +inducts and outducts, as seen in lines 76 thru 80 below. The Add "duct" +commands associate a protocol (in this case, LTP) with individual node +numbers (in this case, 25, 101 and 1) and a task designed to handle the +appropriate Convergence Layer output operations. A similar scenario +applies for the induct where the LTP protocol and node number 13 get +connected with "ltpcli" as the input Convergence Layer function.
+++a outduct protocol_name duct_name 'CLO_command' +[max_payload_length]
+
The "add outduct" command. This command establishes a \"duct\" for +transmission of bundles via the indicated CL protocol. The duct's data +transmission structure is serviced by the \"outduct\" task whose +operation is initiated by CLO_command at the time the duct is started. +max_payload_length, if specified, causes ION to fragment bundles +issued via this outduct (as necessary) to ensure that all such bundles +have payloads that are no larger than max_payload_length.
+++a induct protocol_name duct_name 'CLI_command'
+
The "add induct" command. This command establishes a \"duct\" for +reception of bundles via the indicated CL protocol. The duct's data +acquisition structure is used and populated by the \"induct\" task whose +operation is initiated by CLI_command at the time the duct is started.
+++Note that only a single induct is needed for all bundle +reception via any single protocol at any single node, and in fact ION +may operate poorly if multiple inducts are established for any single +protocol. For any induct whose duct name includes an IP address, use +IP address 0.0.0.0 (INADDR_ANY) if the machine on which the node +resides is multihomed and you want the node to be reachable via all of +the machine's network interfaces.
+
Once all of this has been defined, the last piece needed is the egress +plan -- namely how do packets get transmitted to DTN nodes that are the +local node's "neighbors" in the topology of the network.
+As you can see from line numbers 6 thru 29, the only network neighbor to +node 1 is node 101. Node 25 has not been defined (because the commands +in lines 8, 14, 21 and 27 have been commented). In line numbers 15 and +16, we see that the only destinations for data beginning at node 1 are +nodes 101 and 1 (a loopback as such). Therefore, in order to get data +from node 1 to node 25, our only choice is to send data to node 101. Out +best hope of reaching node 25 is that the configurations for node 101 +define a connection to node 25 (either a one-hop direct connection, or +more multi-hop assumptions). This is where egress plans come into play.
+On line numbers 87 thru 89, this configuration defines the only choices +that can be made regarding destinations. For a destination of node 25, +which is not a neighbor, all node 1 can do is pass the data to its only +neighbor, namely node 101; the "exit" command enables this operation. +For destinations of nodes 101 and 1, the scenario is pretty simple.
+++a exit first_node_nbr last_node_nbr gateway_endpoint_ID
+
The "add exit" command. This command establishes an "exit" for static +routing. An exit is an association of some defined routing behavior with +some range of node numbers identifying a set of nodes. Whenever a bundle +is to be forwarded to a node whose number is in the exit's node number +range and it has not been possible to compute a dynamic route to +that node from the contact schedules that have been provided to the +local node and that node is not a neighbor to which the bundle can +be directly transmitted, BP will forward the bundle to the gateway node +associated with this exit.
+++a plan node_nbr duct_expression [nominal_data_rate]
+
The "add plan" command. This command establishes an egress plan for the +bundles that must be transmitted to the neighboring node identified by +node_nbr.
+Each duct expression is a string of the form
+++\"protocol_name/outduct_name\"
+
signifying that the bundle is to be queued for transmission via the +indicated convergence layer protocol outduct.
+The duct expression used in these examples has "ltp" being the protocol +name and 101 and 1 being the outduct names.
+As of ION 4.1.0, bprc's "plan" and "planduct" commands supersede and +generalize the egress plan commands documented in the ipnrc(5) and +dtn2rc(5) man pages, which are [retained for backward +compatibility]{.underline}. The syntax of the egress plan commands +consumed by bpadmin is DIFFERENT from that of the commands consumed by +ipnadmin and dtn2admin. Please see the man page for bprc (5) for +details.
+For some purposes it may be helpful to encapsulate a bundle inside +another bundle -- that is, to let the serialized representation of a +bundle be part of the payload of another bundle. This mechanism is +called "Bundle-in-Bundle Encapsulation" (BIBE) and is defined in +Internet Draft draft-burleigh-dtn-bibect-00.txt (which will likely be +renamed at some point and ideally will become an IETF standards-track +Request For Comments in due course).
+By way of overview, here is an excerpt from that document:
+++Each BP node that conforms to the BIBE specification provides a BIBE +convergence-layer adapter (CLA) that is implemented within the +administrative element of the BP node\'s application agent. Like any +convergence-layer adapter, the BIBE CLA provides:
+
++The BIBE CLA performs these services by:
+
++Bundle-in-bundle encapsulation may have broad utility, but the +principal motivating use case is the deployment of \"cross domain +solutions\" in secure communications. Under some circumstances a +bundle may arrive at a node that is on the frontier of a region of +network topology in which augmented security is required, from which +the bundle must egress at some other designated node. In that case, +the bundle may be encapsulated within a bundle to which the requisite +additional BP Security (BPSEC) extension block(s) can be attached, +whose source is the point of entry into the insecure region (the +\"security source\") and whose destination is the point of egress from +the insecure region (the \"security destination\").
+Note that:
+
++The protocol includes a mechanism for recovery from loss of an +encapsulating bundle, called \"custody transfer\". This mechanism is +adapted from the custody transfer procedures described in the +experimental Bundle Protocol specification developed by the +Delay-Tolerant Networking Research group of the Internet Research Task +Force and documented in RFC 5050. Custody transfer is a convention by +which the loss or corruption of BIBE encapsulating bundles can be +mitigated by the exchange of other bundles, which are termed \"custody +signals\".
+
BIBE is implemented in ION, but configuring ION nodes to employ BIBE is +not as simple as one might think. That is because BIBE functions as both +a BP application and a convergence-layer adapter; coercing the Bundle +Protocol to function in both capacities, offering services to itself at +two different layers of the protocol stack, requires careful +configuration.
+Like any convergence-layer protocol, BIBE is used to copy a bundle from +one BP node (the sending node) to another node (the receiving node), +over one segment of the end-to-end path from the bundle's source node to +its destination node. Somewhat confusingly, in BIBE the copying of the +bundle is accomplished by issuing a second encapsulating bundle, which +has its own source node and destination node:
+Each pair of sending and receiving nodes can be thought of as a "tunnel" +which requires specific configuration. These tunnels constitute the +communication relationships that must be implemented as "outducts" in +ION.
+While the node IDs of the source and destination nodes of encapsulating +bundles are necessary parameters for BIBE transmission, they are not +sufficient: encapsulating bundles are characterized by quality of +service, lifetime, etc., just like other bundles. For this purpose we +use an additional BIBE administration utility program -- bibeadmin +-- that consumes a file of .bprc commands; these commands add, +revise, and delete BIBE convergence layer adapter objects (bclas) that +are managed in a BIBE database. For example:
+++a bcla ipn:3.0 20 20 300 2 128
+
This command adds a bcla identified by "ipn:3.0" -- the ID of the +destination node of all encapsulating bundles formed according to this +bcla -- which asserts that the expected latency for each encapsulating +bundle to reach this destination node is 20 seconds, the expected +latency for a responding custody signal bundle is likewise 20 seconds, +the encapsulating bundle's time-to-live is 300 seconds, its class of +service is 2 (expedited), and its ordinal sub-priority is 128.
+Note that other configuration elements may also be implicitly associated +with this bcla. For example, BPSEC security rules may map this BIBE +source/destination node pair to security block configurations that will +pertain to all encapsulating bundles formed according to this bcla.
+Since BIBE is a convergence-layer protocol, each BIBE tunnel must be +configured by means of BP administration (bpadmin) using .bprc +commands; BIBE must be added as a protocol, the local node must be added +as the BIBE induct, and each supported BIBE tunnel must be added as a +BIBE outduct. For example:
+ +The "a outduct" command states that the BIBE outduct (tunnel) identified +by node ID "ipn:4.0" (the receiving node) is serviced by a BIBE +convergence-layer output daemon operating according to the bcla +identified by "ipn:3.0" as described above. The destination node ipn:3.0 +is responsible for forwarding each extracted (encapsulated) bundle to +the receiving node ipn:4.0. The sending node and the source node of the +encapsulating bundles are both, implicitly, the local node.
+Note that for most convergence-layer adapters the node ID of the +receiving node for a given outduct is implicit; for example, an stcp +outduct explicitly identifies only the socket address of the receiving +node's socket -- that is, the convergence-layer protocol endpoint ID -- +not the node ID of the receiving node. BIBE differs only in that the +convergence-layer protocol endpoint ID is, explicitly, the node ID of +the receiving node, simply because BP is being used as the +convergence-layer protocol.
+In order to cause bundles to be conveyed to a specified receiving node +via a BIBE outduct, that outduct must be associated with that node in an +egress plan. For example, in the .ipnrc file:
+ +The first command asserts that all bundles destined for node "ipn:4.0" +are to be forwarded using BIBE outduct "ipn:4.0". The second asserts +that all bundles destined for node "ipn:3.0" (here, all BIBE +encapsulating bundles formed according to the bcla identified by +"ipn:3.0") are to be forwarded using the stcp outduct connected to TCP +socket "91.7.31.134:4546".
+Finally, in order for data to flow to receiving node ipn:4.0 via the +bibe/ipn:4.0 outduct, a contact object must be added to the contact plan +enabling the transmissions:
+++a contact +0 +1000000000 2 4 100000
+
This command states that data flow from node 2 (here, the local node) to +node 4 (the receiving node) is continuously enabled, but the rate of +transmission is limited to 100,000 bytes per second.
+Under some circumstances, successful forwarding of BIBE bundles requires +that outduct overrides be applied. See the biberc(5) man page for +details.
+ION contains a flexible system that allows its code to display errors in +several different ways. At the core of this system is a typedef that +defines a data type named "Logger" (with upper case "L") that is a +function variable that accepts a character pointer (string) parameter +and returns a value of type void.
+typedef void (* Logger)(char *);
+In ION, there is one variable defined to be of this type. Its identifier +is "logger" (with lower case "L") and it is initialized to a value of +"logToStdout". The function "logToStdout" is defined and its contents +cause the string parameter to be printed to the stdout device. +Therefore, any call to the function variable "logger" will have same +effects as a call to the function "logToStdout".
+However, remember that "logger" is a variable and is allowed to change +its value to that of other functions that accept string parameters and +return void. This is how ION allows for flexibility in logging errors.
+At startup, ION makes a call to "ionRedirectMemos". This function makes +a call to "setLogger" which eventually changes the value of the "logger" +variable. The new value of the variable named "logger" is +"writeMemoToIonLog". This function writes strings to a file named +"ion.log".
+It is through this mechanism that any calls to the functions +"writeMemo", "writeMemoNote" or "writeErrMemo" eventually pass their +parameters to the function "writeMemoToIonLog". This is how the +Linux-based ION's operate.
+Check out the FSWLOGGER macro option as documented in section 2.1.1 of +the Design Guide.
+What types of memory does ION use and how is memory +allocated/controlled?
+For an introductory description of the memory resources used by ION, see +Section 1.5 of the ION Design and Operation guide entitled "Resource +Management in ION".
+Section 1.5 of the Design and Operation guide makes reference to +parameters called "wmSize" and "heapWords". Discussion on these and all +of the parameters can be found in this document under the section +entitled "Runtime Parameters".
+ION allocates its large blocks of memory via calls to malloc. Should the +need ever arise to place these large blocks of memory at known, fixed +addresses, it would be possible to modify the function memalign, in the +file platform.c. A better approach would be to create a shared-memory +segment for each pre-allocated memory block (possibly using ION's +sm_ShmAttach() function to do this) and pass the applicable +shared-memory key values to ION at startup, in the "heapKey" and/or +"wmKey" runtime parameters.
+Any code that references the function "sm_ShmAttach" will be looking to +acquire some block of memory. These would include the Space Management +Trace features and standalone programs such as "file2sm", "sm2file" and +"smlistsh".
+ION is generally optimized for continuous operational use rather than +research. In practice, this means that a lot more attention, both in the +code and in the documentation, has been paid to the care and feeding of +an existing ION-based network than to the problem of setting up a new +network in the first place. (The unspoken expectation is that you're +only going to do it once anyway.)
+Unfortunately this can make ION somewhat painful for new users to work +with. The notes in this section are aimed at reducing this pain, at +least a little.
+ION is based on shared access to a common data store in memory (and/or +in a file), and the objects in that data store are intended to persist +across multiple restarts of network activity in a continuously +operational network. That's okay for Space Station operations, but it's +not helpful while you're still struggling to get the network running in +the first place. For this purpose you are probably creating and +destroying one or more nodes repetitively.
+A key concept:
+++Each time you run the standard ionstart script provided with ION, +you are creating a new network from scratch. To minimize confusion, be +sure to clear out the old data store first.
+
If you don't wipe out the old system before trying to start the new one, +then either you will pick up where you left off in testing the old +system (and any endpoints, ducts, etc. you try to add will be rejected +as duplicates) or -- in the event that you have changed something +fundamental in the configuration, or are using an entirely different +configuration file -- you'll see the "Wrong profile for this SDR" +message and won't be able to continue at all.
+In most cases the ionstop script should terminate the node for you. +Invoke it once for every node of your network. To verify that you're +starting from a clean slate, run the ipcs command after ionstop: +the list of Semaphore Arrays should be empty. If it's not, you've got +one or more leftover processes from the previous network still running; +use ps ax to find them and kill -9 to get rid of them. The +process names to look for are:
+Then run the killm script again to make sure the node's +shared-memory resources have been released; run ipcs again to +verify, and review your leftover processes again if those resources +still haven't been released.
+An additional wrinkle: if you configure ION to manage your ION data +store in a file as well as (or instead of) managing it in shared memory, +then in addition to calling killm to destroy the semaphores and the +copy of the data store that resides in shared memory, you also need to +delete the data store file; this destroys the copy of the data store +that resides in the file system. If the data store isn't deleted, then +when you restart ION using your standard configuration file the +file-system copy of the data store will automatically be reloaded into +shared memory and all the config file commands that create new schemes, +endpoints, etc. will fail, because they're still in the data store that +you were using before.
+Another habit that can be helpful: whenever you restart ION from +scratch, delete all the ion.log files in all of the directories in which +you're configuring your ION nodes. This isn't mandatory -- ION will +happily append new log messages to existing log files, and the messages +are time-tagged anyway, so it's always possible to work out what +happened when. But starting fresh with new log files removes a lot of +clutter so that it's easy to see exactly what's happening in this +particular iteration of your network research. ION will create new log +files automatically if they don't exist; if there's something +particularly interesting in the log from a prior system, copy that log +file with a different name so you can come back to it if you need to.
+This message just means that the directory whose name you've provided as +the value of pathName in the ION configuration file does not exist, +and therefore the ION operations that rely on being able to write files +in that directory are disabled. It's strictly informative; nearly +everything in ION will work just fine even if this message is printed +every time you run.
+But if you do care about transaction reversibility, for example, or if +you just want to get rid of the annoying message, simply create the +directory that is named in pathName (it can be any path name you like) +and make sure it's world-writable. The ionconfig(5) man page discusses +this parameter and others that affect the fundamental character of the +system you're configuring.
+These messages are just warnings, but they are annoying. We're still +struggling to work out a way to support bundle security protocol as +fully and readily as possible but still let people run ION without it, +if they want, without too much hassle.
+For now, the best answer might be to insert the following lines into +each host.rc file immediately after the "##end ionadmin" line. They +should create an empty ION security database on each host, which should +shut down all those warnings:
+++## begin ionsecadmin
+1
+## end ionsecadmin
+
Several key elements of ION (notably LTP transmission and bundle +expiration) rely on the clocks of all nodes in the network being +synchronized to within a few seconds. NTP is a good way to accomplish +this, if you've got access to an NTP server. If you can't get your +clocks synchronized, stick to the TCP or UDP convergence-layer adapters, +don't count on using contact graph routing, and use long lifetimes on +all bundles to prevent premature bundle expiration.
+In ION we always use the same numeric value for LTP (and BSSP) engine +number and BP node number -- and for CFDP entity number and AMS +continuum number as well. The idea is that a given ION node has a single +identifying number, which by convention we use wherever a protocol +endpoint identifier is needed for any local protocol agent. This is not +a DTN or CCSDS requirement, but it doesn't violate any of the protocol +specifications and it does marginally simplify both implementation and +configuration.
+The bprc(5) man page explains the general format of the commands for +adding convergence-layer inducts and outducts, but it doesn't provide +the syntax for duct names, since duct name syntax is different for +different CL protocols. Here's a summary of duct name syntax for the CL +protocols supported as of ION 3.6.1:
+Here are some other points to bear in mind as you debug your ION node +configuration:
+In this section, we present LTP throughput measurements collected on +different computing platforms. The goal of these tests is to provide a +set of data points that give ION users a sense of the achievable LTP +throughput for a given level of computing resources, ranging from +single-board computers (SBC) to medium-level or high-end servers +connected via 10Gbps Ethernet. We made no attempt to match any +particular user's computing environment in this test. Users must +exercise their own good engineering sense when generalizing and applying +these data points to make predictions regarding the performance of their +own systems. The users are encouraged to install ION on the target +platform and configure ION - using some of the configuration +recommendations in this report - when conducting their own tests.
+Since our focus is to explore the speed limitation caused by software +processing within ION, we try to eliminate external factors that can +slow down throughput, such as a poor network connection or other +processing-intensive software running concurrently on the host machine +that compete for CPU cycles, etc. We also eliminated the impact of +round-trip delay and packet error by testing LTP over a high-speed, +low-error, direct Ethernet connection between two LTP peers.
+Given that LTP is designed for space links, not terrestrial links, LTP +segment sizes much larger than typical terrestrial network MTUs +(nominally 1,500 to 9,000 bytes) are considered in our testing. For LTP +configuration in space, the CCSDS Packet Encapsulation service enables +LTP segments of variable sizes to be transmitted over different CCSDS +space link protocols.
+Most of our testing was conducted with the SDR in DRAM (configuration 1) +to achieve higher data processing speed. However, we did collect data on +several cases where reversibility and SDR object boundness checks were +turned on.
+In the interest of efficiency, we also favor selecting larger bundles, +potentially much larger than the LTP aggregation block size. In previous +TCPCL testing, it was observed that a larger bundle size improves +throughput since more data can be transferred per logical operation. For +the same reason, we believe that a larger bundle size will improve LTP +performance.
+ION performs bundle-level metering to throttle the speed with which data +is presented to LTP engines for transmission. The throttle rate is set +by the contact plan and should not exceed the line rate of the physical +Ethernet connection. In many cases, we configure ION with a contact plan +rate that is lower than the Ethernet line rate to allow BP/LTP to +operate as fast as possible without creating a destructive level of +congestion. To facilitate better testing, we also use the bpdriver +utility program with the \'i\' option to control the source data +injection rate. For some tests, we find data metering unnecessary, and +ION can buffer and handle local congestion and deliver the maximum +possible throughput.
+As stated earlier, our goal is to test the ION processing rate +limitation, not the host system\'s memory availability. Therefore, we +configure ION SDR with a generous amount of heap and working memory to +ensure that data storage is not a limiting factor.
+Now, we present our test cases.
+B = byte
+b = bit
+M = mega
+K = kilo
+G = giga
+ION Configuration:
+Hardware Specification and Operating System:
+Throughput Measured:
+ION Configuration:
+Hardware Specification and Operating System:
+Throughput Measured:
+In this test case, we considered several SDR configuration combinations +and assessed their impact.
+We do not include the "SDR in file" or any combination with that since +file operation will slow down performance significantly.
+Base ION Memory Configuration
+BP/LTP Configuration
+The following is representational of .ltprc file on both nodes
+1 50
+Contact Plan Data Rate (1 Gb/sec)
+Hardware Specification and Operating System:
+Network link reported by iperf as follows (PTL Orange Testbed)
+UDP: 1.35 GB/sec
+Throughput Measured
+General observation is that SDR boundedness checks (each write operation +must ensure that the location where data is written is occupied by an +object of the same size of the write operation) introduce about 11% of +throughput degradation. Adding reversibility will substantially slow +down the system since the reversibility, by default, saves transaction +operations record in a file until the transaction is complete or until +when the transaction is canceled and must be reversed. Although it is +possible to store transaction record in ION's working memory, we didn't +consider this case in our testing due to time constraint.
+In this 10Gbps case study, we measured LTP performance between two +machines physically connected by a 10Gbps Ethernet switch. Initial +testing with iperf showed that although the physical connection was +10Gbps, the actual throughput maxed out at 2.5Gbps. Improved throughput +was attained by increasing the kernel buffer sizes to 8MB. Additionally, +increasing the MTU (Maximum Transmission Unit) size from 1500 to 9600 +resolved some caching issues seen at the receiving node.
+UDP Configuration Details
+The following kernel buffer size settings were used to enable full +utilization of the 10Gbps Ethernet on the host machine. These are +provided for your reference. Depending on your host system's +configuration, you may not need to adjust any parameters to make sure +the full capacity of the Ethernet connection is achievable. Even in +cases where you do find it necessary to make such adjustments, the +actual parameters values may not be the same.
+To resolve the caching issue, which allows the LTP engine to clean up +after the test quickly, we set the MTU to 9600. This is not strictly +required but we find it helpful when the MTU is set to 9600 Bytes +instead of the typical value of 1500 Bytes (we observed improved LTP +session cleanup times with the higher MTU). After applying these +updates, iperf testing showed 9.9Gbps throughput on the Ethernet +connection between the two hosts.
+Test Network Details
+The test network consists of 2 host machines physically connected via 10 +Gb Network Interface Card
+Hardware
+ION Memory Configuration Details
+LTP Configuration Details
+Throughput Measurement
+The first series of tests provided some insights into the impact of +bundle size on throughput. In general, using a larger bundle size allows +ION to transfer more data per logical operation since the overhead of a +bundle is relatively fixed regardless of the size of the payload. In our +tests, we controlled the size of all bundles injected into ION for LTP +transfer. In real operations, bundle size will vary, but for bulk data +transfer, the user is generally able to dictate the size of the bundle +it sends. To avoid processing smaller bundles individually (which occurs +in real operations), we turned on LTP block aggregation and set the size +to 64KB.
+Figure 1: LTP Throughput as a Function of Bundle Size
+
In Figure 1, we can immediately observe that bundle size has a +significant impact on LTP throughput. This is because the bundle is the +basic unit of an LTP block. When LTP block aggregation is turned on, a +block may consist of one or multiple bundles. When LTP block aggregation +is not applied, each block is one bundle. When the bundle size is less +than the aggregation size, LTP will accumulate several bundles before +creating a block. While this will limit LTP overhead, the use of small +bundles still has an impact on the bundle protocol level processing, +both before LTP transmission and during post-LTP-reception +reconstruction. Therefore, as we can see, when the bundle size is +dropped below the LTP aggregation threshold, the throughput is still +impacted by bundle size.
+While it may seem that the aggregation size limit does not have a strong +impact on throughput, it does for long delay-bandwidth space links, +where it dictates the maximum number of import/export sessions that ION +must support simultaneously. That will be another investigation for a +future study. For now, we focus solely on testing the limit of ION\'s +data processing speed in a low latency lab environment.
+We also conducted a second series of tests to look at the impact of LTP +segment sizes on throughput. The results are in Figure 2 below.
+Figure 2: Impact of LTP Segment Size on Throughput
+
In this test, we looked at bundle sizes that are 1MB or lower, with +segment sizes ranging from 64KB to 1,500 bytes. Again, we observed that +segment size has a stronger impact on throughput when it is less than +10% of the bundle size; once it goes above 10%, the impact is noticeably +diminished. This has to do with the fact that each segment levies a +minimal amount of logical operation. Using a large segment size can help +reduce LTP processing overhead. However, since the segment is LTP\'s +standard protocol data unit and it determines the +vulnerability/likelihood of data loss (large segments expose more data +to loss due to corruption), it is not advised to arbitrarily increase +the segment size in a real flight environment with a substantial data +loss probability. The key point here is to illustrate that the +choice of segment size can impact the processing overhead and speed of +LTP.
+In real life operation, we expect the users to generate a wide mixture +of large and small bundles. Although we don't have a commonly agreed on +"profile" of how a typical DTN user will generate bundles, it is +nonetheless valuable for us to get a sense of how BP/LTP in ION would +perform when handling bundles of random sizes.
+For a quick study, we leveraged the same 2.1GHz Xeon Sandy Bridge +processor configuration with 1MB LTP aggregation limit and injected +bundles whose payload size is a uniformly distributed random value +between 1024 bytes and 62464 bytes. We found that the throughput is +approximately 260Mbps for segment size of 9600 B, and 300Mbps when +segment size is increased to 64,000B. For the second test, we increased +the bundle size ranges to be between 1KB and 1MB, the measured +throughput is 2.08Gbps.
+This performance is higher than we expected. For the same amount of data +delivery, using 1MB bundle vs an average of 31KB per bundle (uniform +between 1K and 62K) would increase bundle process overhead by a factor +of 32. Holding all other parameters constant, the 300Mbps throughput is +only a factor of 9.6 lower compared to the 1MB bundle case with +throughput of 2.9Gbps. The 32-fold increase of bundle overhead didn't +result in a 32-fold reduction of speed. The reason for this +better-than-expected result is, we believe, due to the use of LTP block +aggregation. Similarly, for the second test, we increased the average +bundle overhead by a factor 2, but the data rate reduction is only about +29 percent. By keeping the block aggregation to 1MB, we keep the number +of LTP sessions and handshaking overhead low, which mitigated some of +the impact of the presence of smaller bundles.
+Our initial assessment is that the mixed use of larger and smaller +bundles will reduce throughput but not as substantially as one would +expect based on a linear interpolation of the bundle processing +overhead. The use of LTP block aggregation can maintain a higher +efficiency under such circumstances. Additional investigation in this +area will be conducted and reported in the near future.
+We conducted a series of tests, documenting the performance of BP/LTP +for a range of hardware and ION configuration options. At the lower end, +we tested two stock Raspberry Pi 4B single-board computers running ION +4.1.2 and achieved 60 Mbps one-way data transfer without any hardware or +OS optimization. At the higher end of our tests, we measured ION +performance between two Linux servers (see spec in Test Case 4; 2012 era +Xeon Sandy Bridge Processors) and showed that [ION\'s BP/LTP +implementation can support up to 3.7Gbps throughput over a 10Gbps +Ethernet physical connection]{.underline}. We also presented a +discussion on the performance trades regarding various LTP configuration +parameters.
+We hope that these data points will provide users with a sense of how to +configure ION, BP, and LTP to achieve the highest possible throughput on +their own systems. We acknowledge that these tests focus on exploring +the performance envelope of ION\'s data processing speed and do not +emulate specific flight configurations, nor do they cover long +round-trip delay and high error rate space links. For specific link +conditions and computing the recommended LTP settings, please consult +the LTP Configuration Tool spreadsheet provided with each ION +open-source package on SourceForge.
+Some of the technology described in this Deployment Guide was developed +at the Jet Propulsion Laboratory, California Institute of Technology, +under a contract with the National Aeronautics and Space Administration.
+Copyright © 2021 California Institute of Technology
+The ION team would like to acknowledge the following individuals for +contributed to the earlier versions of this Guide: Jane Marquart, NASA; +Greg Menke, Columbus; Larry Shackelford, Microtel LLC; Scott Burleigh +(retired), Jet Propulsion Laboratory, California Institute of Technology
+ + + + + + + + + + + + + +Version 4.1.3 +JPL D-48259
+Document Change Log
+| Ver No. | +Date | +Description | +Note | +
|---|---|---|---|
| V4.1.3 | +12/08 /2023 | +update to MarkDown | ++ |
| V4.0.1 | +11/20/2020 | +ION 4.0.1 | ++ |
| V3.6.2 | +11/19/2018 | +ION 3.6.2 release features | +Skipped V3.6.1. | +
| V3.6 | +12/31/2017 | +ION 3.6 release features | +Skipped V3.5. | +
| V3.4 | +3/28/2016 | +ION 3.4 release features | ++ |
| V3.3 | +3/4/2015 | +ION 3.3 release features | ++ |
| V3.2 | +12/17/2013 | +ION 3.2 release features | ++ |
| V3.1 | +9/28/2012 | +ION 3.1 release features | ++ |
| V3.0 | +3/22/2012 | +Align with ION 3.0 release | ++ |
| V1.13 | +10/13/2011 | +Updates for Source Forge Release | ++ |
| V1.12 | +6/11/2010 | +Updates for second open source release (2.2) | ++ |
| V1.11 | +12/11/2009 | +BRS updates, multi-node config | ++ |
| V1.10 | +10/23/2009 | +Final additions prior to DINET 2 experiment | ++ |
| V1.9 | +6/29/2009 | +Add updates for DINET 2, including CFDP, ionsec | ++ |
| V1.8 | +2/6/2009 | +Update discussion of Contact Graph Routing; document status msg formats | ++ |
| V1.7 | +12/1/2008 | +Add documentation for OWLT simulator, BP extension | ++ |
| V1.6 | +10/03/2008 | +Add documentation of sm_SemUnend | ++ |
| V1.5 | +09/20/2008 | +Revisions requested SQA | ++ |
| V1.4 | +07/31/2008 | +Add a section on optimizing ION-based network; tuning | ++ |
| V1.3 | +07/08/2008 | +Revised some details of CGR | ++ |
| V1.2 | +05/24/2008 | +Revised man pages for bptrace, ltprc, bprc. | ++ |
| V1.1 | +05/18/2008 | +Some additional diagrams | ++ |
| V1.0 | +04/28/2008 | +Initial version of ION design and ops manual | ++ |
The Interplanetary Overlay Network (ION) software distribution is an +implementation of Delay-Tolerant Networking (DTN) architecture as +described in Internet RFC 4838. It is designed to enable inexpensive +insertion of DTN functionality into embedded systems such as robotic +spacecraft. The intent of ION deployment in space flight mission systems +is to reduce cost and risk in mission communications by simplifying the +construction and operation of automated digital data communication +networks spanning space links, planetary surface links, and terrestrial +links.
+A comprehensive overview of DTN is beyond the scope of this document. +Very briefly, though, DTN is a digital communication networking +technology that enables data to be conveyed between two communicating +entities automatically and reliably even if one or more of the network +links in the end-to-end path between those entities is subject to very +long signal propagation latency and/or prolonged intervals of +unavailability.
+The DTN architecture is much like the architecture of the Internet, +except that it is one layer higher in the familiar ISO protocol "stack". +The DTN analog to the Internet Protocol (IP), called "Bundle Protocol" +(BP), is designed to function as an "overlay" network protocol that +interconnects "internets" -- including both Internet-structured networks +and also data paths that utilize only space communication links as +defined by the Consultative Committee for Space Data Systems (CCSDS) -- +in much the same way that IP interconnects "subnets" such as those built +on Ethernet, SONET, etc. By implementing the DTN architecture, ION +provides communication software configured as a protocol stack that +looks like this:
+
Figure 1 DTN protocol stack
+Data traversing a DTN are conveyed in DTN bundles -- which are +functionally analogous to IP packets -- between BP endpoints which are +functionally analogous to sockets. Multiple BP endpoints may be accessed +at a single DTN node -- functionally analogous to a network interface +card -- and multiple nodes may reside on the same computer just as a +single computer (host or router) in the Internet may have multiple +network interface cards.
+BP endpoints are identified by Universal Record Identifiers (URIs), +which are ASCII text strings of the general form:
+scheme_name:scheme_specific_part
+For example:
+dtn://topquark.caltech.edu/mail
+But for space flight communications this general textual representation +might impose more transmission overhead than missions can afford. For +this reason, ION is optimized for networks of endpoints whose IDs +conform more narrowly to the following scheme:
+ipn:node_number.service_number
+This enables them to be abbreviated to pairs of unsigned binary integers +via a technique called Compressed Bundle Header Encoding (CBHE). +CBHE-conformant BP endpoint IDs (EIDs) are not only functionally +similar to Internet socket addresses but also structurally similar: node +numbers are roughly analogous to Internet node numbers (IP addresses), +in that they typically identify the flight or ground data system +computers on which network software executes, and service numbers are +roughly analogous to TCP and UDP port numbers.
+More generally, the node numbers in CBHE-conformant BP endpoint IDs are +one manifestation of the fundamental ION notion of network node +number: in the ION architecture there is a natural one-to-one mapping +not only between node numbers and BP endpoint node numbers but also +between node numbers and:
+LTP engine IDs
+AMS continuum numbers
+CFDP entity numbers
+Starting with version 3.1 of ION, this endpoint naming rule is +experimentally extended to accommodate bundle multicast, i.e., the +delivery of copies of a single transmitted bundle to multiple nodes at +which interest in that bundle's payload has been expressed. Multicast in +ION -- "Interplanetary Multicast" (IMC) -- is accomplished by simply +issuing a bundle whose destination endpoint ID conforms to the following +scheme:
+imc:group_number.service_number
+A copy of the bundle will automatically be delivered at every node that +has registered in the destination endpoint.
+(Note: for now, the operational significance of a given group number +must be privately negotiated among ION users. If this multicast +mechanism proves useful, IANA may at some point establish a registry for +IMC group numbers. Also note that a new mechanism for bundle multicast +is introduced in ION 4.0.1, along with support for Bundle Protocol +version 7. This new mechanism vastly simplifies bundle multicast; +chiefly, the imcadmin utility is deprecated.)
+The ION distribution comprises the following software packages:
+ici (Interplanetary Communication Infrastructure), a set of + general-purpose libraries providing common functionality to the + other packages. The ici package includes a security policy component + that supports the implementation of security mechanisms at multiple + layers of the protocol stack.
+ltp (Licklider Transmission Protocol), a core DTN protocol that + provides transmission reliability based on delay-tolerant + acknowledgments, timeouts, and retransmissions. The LTP + specification is defined in Internet RFC 5326.
+bp (Bundle Protocol), a core DTN protocol that provides + delay-tolerant forwarding of data through a network in which + continuous end-to-end connectivity is never assured, including + support for delay-tolerant dynamic routing. The BP specification is + defined in Internet RFC 5050.
+dgr (Datagram Retransmission), an alternative implementation of LTP + that is designed for use on the Internet. Equipped with algorithms + for TCP-like congestion control, DGR enables data to be transmitted + via UDP with reliability comparable to that provided by TCP. The dgr + system is provided primarily for the conveyance of Meta-AMS (see + below) protocol traffic in an Internet-like environment.
+ams (Asynchronous Message Service), an application-layer service + that is not part of the DTN architecture but utilizes underlying DTN + protocols. AMS comprises three protocols supporting the distribution + of brief messages within a network:
+The core AAMS (Application AMS) protocol, which does message + distribution on both the publish/subscribe model and the + client/server model, as required by the application.
+The MAMS (Meta-AMS) protocol, which distributes control + information enabling the operation of the Application AMS + protocol.
+The RAMS (Remote AMS) protocol, which performs aggregated + message distribution to end nodes that may be numerous and/or + accessible only over very expensive links, using an aggregation + tree structure similar to the distribution trees used by + Internet multicast technologies.
+cfdp (CCSDS File Delivery Protocol), another application-layer + service that is not part of the DTN architecture but utilizes + underlying DTN protocols. CFDP performs the segmentation, + transmission, reception, reassembly, and delivery of files in a + delay-tolerant manner. ION's implementation of CFDP conforms to the + "class 1" definition of the protocol in the CFDP standard, utilizing + DTN (BP, nominally over LTP) as its "unitdata transport" layer.
+bss (Bundle Streaming Service), a system for efficient data + streaming over a delay-tolerant network. The bss package + includes (a) a convergence-layer protocol (bssp) that preserves + in-order arrival of all data that were never lost en route, yet + ensures that all data arrive at the destination eventually, and (b) + a library for building delay-tolerant streaming applications, which + enables low-latency presentation of streamed data received in real + time while offering rewind/playback capability for the entire stream + including late-arriving retransmitted data.
+tc (Trusted Collective), a system for propagating critical yet + non-confidential information in a trustworthy manner. tc can be + thought of as a delay-tolerant functional analog to the servers in + client/server architectures. Multiple applications may make use of + the tc system, but currently only one tc application is bundled with + ION: dtka (delay-tolerant key administration), which provides + delay-tolerant public key infrastructure.
+Taken together, the packages included in the ION software distribution +constitute a communication capability characterized by the following +operational features:
+Reliable conveyance of data over a delay-tolerant network (dtnet), + i.e., a network in which it might never be possible for any node to + have reliable information about the detailed current state of any + other node.
+Built on this capability, reliable data streaming, reliable file + delivery, and reliable distribution of short messages to multiple + recipients (subscribers) residing in such a network.
+Management of traffic through such a network, taking into + consideration:
+requirements for data security
+scheduled times and durations of communication opportunities
+fluctuating limits on data storage and transmission resources
+data rate asymmetry
+the sizes of application data units
+and user-specified final destination, priority, and useful + lifetime for those data units.
+Facilities for monitoring the performance of the network.
+Robustness against node failure.
+Portability across heterogeneous computing platforms.
+High speed with low overhead.
+Easy integration with heterogeneous underlying communication + infrastructure, ranging from Internet to dedicated spacecraft + communication links.
+A DTN implementation intended to function in an interplanetary network +environment -- specifically, aboard interplanetary research spacecraft +separated from Earth and from one another by vast distances -- must +operate successfully within two general classes of design constraints: +link constraints and processor constraints.
+All communications among interplanetary spacecraft are, obviously, +wireless. Less obviously, those wireless links are generally slow and +are usually asymmetric.
+The electrical power provided to on-board radios is limited and antennae +are relatively small, so signals are weak. This limits the speed at +which data can be transmitted intelligibly from an interplanetary +spacecraft to Earth, usually to some rate on the order of 256 Kbps to 6 +Mbps.
+The electrical power provided to transmitters on Earth is certainly much +greater, but the sensitivity of receivers on spacecraft is again +constrained by limited power and antenna mass allowances. Because +historically the volume of command traffic that had to be sent to +spacecraft was far less than the volume of telemetry the spacecraft were +expected to return, spacecraft receivers have historically been +engineered for even lower data rates from Earth to the spacecraft, on +the order of 1 to 2 Kbps.
+As a result, the cost per octet of data transmission or reception is +high and the links are heavily subscribed. Economical use of +transmission and reception opportunities is therefore important, and +transmission is designed to enable useful information to be obtained +from brief communication opportunities: units of transmission are +typically small, and the immediate delivery of even a small part +(carefully delimited) of a large data object may be preferable to +deferring delivery of the entire object until all parts have been +acquired.
+The computing capability aboard a robotic interplanetary spacecraft is +typically quite different from that provided by an engineering +workstation on Earth. In part this is due, again, to the limited +available electrical power and limited mass allowance within which a +flight computer must operate. But these factors are exacerbated by the +often intense radiation environment of deep space. In order to minimize +errors in computation and storage, flight processors must be +radiation-hardened and both dynamic memory and non-volatile storage +(typically flash memory) must be radiation-tolerant. The additional +engineering required for these adaptations takes time and is not +inexpensive, and the market for radiation-hardened spacecraft computers +is relatively small; for these reasons, the latest advances in +processing technology are typically not available for use on +interplanetary spacecraft, so flight computers are invariably slower +than their Earth-bound counterparts. As a result, the cost per +processing cycle is high and processors are heavily subscribed; +economical use of processing resources is very important.
+The nature of interplanetary spacecraft operations imposes a further +constraint. These spacecraft are wholly robotic and are far beyond the +reach of mission technicians; hands-on repairs are out of the question. +Therefore the processing performed by the flight computer must be highly +reliable, which in turn generally means that it must be highly +predictable. Flight software is typically required to meet "hard" +real-time processing deadlines, for which purpose it must be run within +a hard real-time operating system (RTOS).
+One other implication of the requirement for high reliability in flight +software is that the dynamic allocation of system memory may be +prohibited except in certain well-understood states, such as at system +start-up. Unrestrained dynamic allocation of system memory introduces a +degree of unpredictability into the overall flight system that can +threaten the reliability of the computing environment and jeopardize the +health of the vehicle.
+The design of the ION software distribution reflects several core +principles that are intended to address these constraints.
+
Figure 2 ION inter-task communication
+Since ION must run on flight processors, it had to be designed to +function successfully within an RTOS. Many real-time operating systems +improve processing determinism by omitting the support for +protected-memory models that is provided by Unix-like operating systems: +all tasks have direct access to all regions of system memory. (In +effect, all tasks operate in kernel mode rather than in user mode.) ION +therefore had to be designed with no expectation of memory protection.
+But universally shared access to all memory can be viewed not only as a +hazard but also as an opportunity. Placing a data object in shared +memory is an extremely efficient means of passing data from one software +task to another.
+ION is designed to exploit this opportunity as fully as possible. In +particular, virtually all inter-task data interchange in ION follows the +model shown in Figure 2:
+The sending task takes a mutual exclusion semaphore (mutex) + protecting a linked list in shared memory (either DRAM or + non-volatile memory), appends a data item to the list, releases the + mutex, and gives a "signal" semaphore associated with the list to + announce that the list is now non-empty.
+The receiving task, which is already pending on the linked list's + associated signal semaphore, resumes execution when the semaphore is + given. It takes the associated mutex, extracts the next data item + from the list, releases the mutex, and proceeds to operate on the + data item from the sending task.
+Semaphore operations are typically extremely fast, as is the storage and +retrieval of data in memory, so this inter-task data interchange model +is suitably efficient for flight software.
+Given ION's orientation toward the shared memory model, a further +strategy for processing efficiency offers itself: if the data item +appended to a linked list is merely a pointer to a large data object, +rather than a copy, then we can further reduce processing overhead by +eliminating the cost of byte-for-byte copying of large objects.
+Moreover, in the event that multiple software elements need to access +the same large object at the same time, we can provide each such +software element with a pointer to the object rather than its own copy +(maintaining a count of references to assure that the object is not +destroyed until all elements have relinquished their pointers). This +serves to reduce somewhat the amount of memory needed for ION +operations.
+The efficiency of inter-task communications based on shared memory makes +it practical to distribute ION processing among multiple relatively +simple pipelined tasks rather than localize it in a single, somewhat +more complex daemon. This strategy has a number of advantages:
+The simplicity of each task reduces the sizes of the software + modules, making them easier to understand and maintain, and thus it + can somewhat reduce the incidence of errors.
+The scope of the ION operating stack can be adjusted incrementally + at run time, by spawning or terminating instances of configurable + software elements, without increasing the size or complexity of any + single task and without requiring that the stack as a whole be + halted and restarted in a new configuration. In theory, a module + could even be upgraded with new functionality and integrated into + the stack without interrupting operations.
+The clear interfaces between tasks simplify the implementation of + flow control measures to prevent uncontrolled resource consumption.
+Designs based on these kinds of principles are foreign to many software +developers, who may be far more comfortable in development environments +supported by protected memory. It is typically much easier, for example, +to develop software in a Linux environment than in VxWorks 5.4. However, +the Linux environment is not the only one in which ION software must +ultimately run.
+Consequently, ION has been designed for easy portability. POSIX™ API +functions are widely used, and differences in operating system support +that are not concealed within the POSIX abstractions are mostly +encapsulated in two small modules of platform-sensitive ION code. The +bulk of the ION software runs, without any source code modification +whatsoever, equally well in Linux™ (Red Hat®, Fedora™, and Ubuntu™, so +far), FreeBSD®, Solaris® 9, Microsoft Windows (the MinGW environment), +OS/X®, VxWorks® 5.4, and RTEMS™, on both 32-bit and 64-bit processors. +Developers may compile and test ION modules in whatever environment they +find most convenient.
+Two broad overviews of the organization of ION may be helpful at this +point. First, here is a summary view of the main functional dependencies +among ION software elements:
+
Figure 3 ION software functional dependencies
+That is, BP and LTP invoke functions provided by the sdr, zco, psm, and +platform elements of the ici package, in addition to functions provided +by the operating system itself; the zco functions themselves also invoke +sdr, psm, and platform functions; and so on.
+Second, here is a summary view of the main line of data flow in ION's +DTN protocol implementations:
+
Figure 4 Main line of ION data flow
+Note that data objects residing in shared memory, many of them in a +nominally non-volatile SDR data store, constitute the central organizing +principle of the design. Here as in other diagrams showing data flow in +this document:
+Ordered collections of data objects are shown as cylinders.
+Darker greyscale data entities indicate data that are managed in the + SDR data store, while lighter greyscale data entities indicate data + that are managed in volatile DRAM to improve performance.
+Rectangles indicate processing elements (tasks, processes, threads), + sometimes with library references specifically identified.
+A few notes on this main line data flow:
+For simplicity, the data flow depicted here is a "loopback" flow in + which a single BP "node" is shown sending data to itself (a useful + configuration for test purposes). To depict typical operations over + a network we would need two instances of this node diagram, such + that the \<LSO> task of one node is shown sending data to the + \<LSI> task of the other and vice versa.
+A BP application or application service (such as Remote AMS) that + has access to the local BP node -- for our purposes, the "sender" -- + invokes the bp_send function to send a unit of application data to a + remote counterpart. The destination of the application data unit is + expressed as a BP endpoint ID (EID). The application data unit is + encapsulated in a bundle and is queued for forwarding.
+The forwarder task identified by the "scheme" portion of the + bundle's destination EID removes the bundle from the forwarding + queue and computes a route to the destination EID. The first node on + the route is termed the "proximate node" for the computed route. The + forwarder appends the bundle to the transmission queue for the + convergence-layer manager (CLM) daemon that is responsible for + transmission to the proximate node.
+The CLM daemon removes the bundle from the transmission queue and + imposes rate control, fragments the bundle as necessary, and appends + the bundle to the transmission buffer for some underlying + "convergence layer" (CL) protocol interface to the proximate node, + termed an outduct. In the event that multiple outducts are + available for transmission to that node (e.g., multiple radio + frequency bands), the CLM invokes mission-supplied code to select + the appropriate duct. Each outduct is serviced by some CL-specific + output task that communicates with the proximate node -- in this + case, the LTP output task ltpclo. (Other CL protocols supported + by ION include TCP and UDP.)
+The output task for LTP transmission to the selected proximate node + removes the bundle from the transmission buffer and invokes the + ltp_send function to append it to a block that is being assembled + for transmission to the proximate node. (Because LTP acknowledgement + traffic is issued on a per-block basis, we can limit the amount of + acknowledgement traffic on the network by aggregating multiple + bundles into a single block rather than transmitting each bundle in + its own block.)
+The ltpmeter task for the selected proximate node divides the + aggregated block into multiple segments and enqueues them for + transmission by underlying link-layer transmission software, such as + an implementation of the CCSDS AOS protocol.
+Underlying link-layer software at the sending node transmits the + segments to its counterpart at the proximate node (the receiver), + where they are used to reassemble the transmission block.
+The receiving node's input task for LTP reception extracts the + bundles from the reassembled block and dispatches them: each bundle + whose final destination is some other node is queued for forwarding, + just like bundles created by local applications, while each bundle + whose final destination is the local node is queued for delivery to + whatever application "opens" the BP endpoint identified by the + bundle's final destination endpoint ID. (Note that a multicast + bundle may be both queued for forwarding, possibly to multiple + neighboring nodes, and also queued for delivery.)
+The destination application or application service at the receiving + node opens the appropriate BP endpoint and invokes the bp_receive + function to remove the bundle from the associated delivery queue and + extract the original application data unit, which it can then + process.
+Finally, note that the data flow shown here represents the sustained +operational configuration of a node that has been successfully +instantiated on a suitable computer. The sequence of operations +performed to reach this configuration is not shown. That startup +sequence will necessarily vary depending on the nature of the computing +platform and the supporting link services. Broadly, the first step +normally is to run the ionadmin utility program to initialize the +data management infrastructure required by all elements of ION. +Following this initialization, the next steps normally are (a) any +necessary initialization of link service protocols, (b) any necessary +initialization of convergence-layer protocols (e.g., LTP -- the +ltpadmin utility program), and finally (c) initialization of the +Bundle Protocol by means of the bpadmin utility program. BP +applications should not try to commence operation until BP has been +initialized.
+Successful Delay-Tolerant Networking relies on retention of bundle +protocol agent state information -- including protocol traffic that is +awaiting a transmission opportunity -- for potentially lengthy +intervals. The nature of that state information will fluctuate rapidly +as the protocol agent passes through different phases of operation, so +efficient management of the storage resources allocated to state +information is a key consideration in the design of ION.
+Two general classes of storage resources are managed by ION: volatile +"working memory" and non-volatile "heap".
+ION's "working memory" is a fixed-size pool of shared memory (dynamic +RAM) that is allocated from system RAM at the time the bundle protocol +agent commences operation. Working memory is used by ION tasks to store +temporary data of all kinds: linked lists, red-black trees, transient +buffers, volatile databases, etc. All intermediate data products and +temporary data structures that ought not to be retained in the event of +a system power cycle are written to working memory.
+Data structures residing in working memory may be shared among ION tasks +or may be created and managed privately by individual ION tasks. The +dynamic allocation of working memory to ION tasks is accomplished by the +Personal Space Management (PSM) service, described later. All of the +working memory for any single ION bundle protocol agent is managed as a +single PSM "partition". The size of the partition is specified in the +wmSize parameter of the ionconfig file supplied at the time ION is +initialized.
+ION's "heap" is a fixed-size pool of notionally non-volatile storage +that is likewise allocated at the time the bundle protocol agent +commences operation. This notionally non-volatile space may occupy a +fixed-size pool of shared memory (dynamic RAM, which might or might not +be battery-backed), or it may occupy only a single fixed-size file +in the file system, or it may occupy both. In the latter case, all heap +data are written both to memory and to the file but are read only from +memory; this configuration offers the reliable non-volatility of file +storage coupled with the high performance of retrieval from dynamic RAM.
+We characterize ION's heap storage as "notionally" non-volatile because +the heap may be configured to reside only in memory (or, for that +matter, in a file that resides in the file system of a RAM disk). When +the heap resides only in memory, its contents are truly non-volatile +only if that memory is battery-backed. Otherwise heap storage is in +reality as volatile as working memory: heap contents will be lost upon a +system power cycle (which may in fact be the preferred behavior for any +given deployment of ION). However, the heap should not be thought of as +\"memory\" even when it in fact resides only in DRAM, just as a disk +device should not be thought of as \"memory\" even when it is in fact a +RAM disk.
+
{width="4.738575021872266in"
+height="3.338542213473316in"}
Figure 5 ION heap space use
+The ION heap is used for storage of data that (in at least some +deployments) would have to be retained in the event of a system power +cycle to ensure the correct continued operation of the node. For +example, all queues of bundles awaiting route computation, transmission, +or delivery reside in the node's heap. So do the non-volatile databases +for all of the protocols implemented within ION, together with all of +the node's persistent configuration parameters.
+The dynamic allocation of heap space to ION tasks is accomplished by the +Simple Data Recorder (SDR) service, described later. The entire heap for +any single ION bundle protocol agent is managed as a single SDR "data +store".
+Space within the ION heap is apportioned as shown in Figure 5. The total +number of bytes of storage space in the heap is computed as the product +of the size of a "word" on the deployment platform (normally the size of +a pointer) multiplied by the value of the heapWords parameter of the +ionconfig file supplied at the time ION is initialized. Of this total, +20% is normally reserved as margin and another 40% is normally reserved +for various infrastructure operations. (Both of these percentages are +macros that may be overridden at compile time.) The remainder is +available for storage of protocol state data in the form of "zero-copy +objects", described later. At any given moment, the data encapsulated in +a zero-copy object may "belong" to any one of the protocols in the ION +stack (AMS, CFDP, BP, LTP), depending on processing state; the available +heap space is a single common resource to which all of the protocols +share concurrent access.
+Because the heap is used to store queues of bundles awaiting processing, +blocks of LTP data awaiting transmission or reassembly, etc., the heap +for any single ION node must be large enough to contain the maximum +volume of such data that the node will be required to retain during +operations. Demand for heap space is substantially mitigated if most of +the application data units passed to ION for transmission are +file-resident, as the file contents themselves need not be copied into +the heap. In general, however, computing the optimum ION heap size for a +given deployment remains a research topic.
+The ICI package in ION provides a number of core services that, from +ION's point of view, implement what amounts to an extended POSIX-based +operating system. ICI services include the following:
+1. Platform
+The platform system contains operating-system-sensitive code that +enables ICI to present a single, consistent programming interface to +those common operating system services that multiple ION modules +utilize. For example, the platform system implements a standard +semaphore abstraction that may invisibly be mapped to underlying POSIX +semaphores, SVR4 IPC semaphores, Windows Events, or VxWorks semaphores, +depending on which operating system the package is compiled for. The +platform system also implements a standard shared-memory abstraction, +enabling software running on operating systems both with and without +memory protection to participate readily in ION's shared-memory-based +computing environment.
+2. Personal Space Management (PSM)
+Although sound flight software design may prohibit the uncontrolled +dynamic management of system memory, private management of assigned, +fixed blocks of system memory is standard practice. Often that private +management amounts to merely controlling the reuse of fixed-size rows in +static tables, but such techniques can be awkward and may not make the +most efficient use of available memory. The ICI package provides an +alternative, called PSM, which performs high-speed dynamic allocation +and recovery of variable-size memory objects within an assigned memory +block of fixed size. A given PSM-managed memory block may be either +private or shared memory.
+3. Memmgr
+The static allocation of privately-managed blocks of system memory for +different purposes implies the need for multiple memory management +regimes, and in some cases a program that interacts with multiple +software elements may need to participate in the private shared-memory +management regimes of each. ICI's memmgr system enables multiple memory +managers -- for multiple privately-managed blocks of system memory -- to +coexist within ION and be concurrently available to ION software +elements.
+4. Lyst
+The lyst system is a comprehensive, powerful, and efficient system for +managing doubly-linked lists in private memory. It is the model for a +number of other list management systems supported by ICI; as noted +earlier, linked lists are heavily used in ION inter-task communication.
+5. Llcv
+The llcv (Linked-List Condition Variables) system is an inter-thread +communication abstraction that integrates POSIX thread condition +variables (vice semaphores) with doubly-linked lists in private memory.
+6. Smlist
+Smlist is another doubly-linked list management service. It differs from +lyst in that the lists it manages reside in shared (rather than private) +DRAM, so operations on them must be semaphore-protected to prevent race +conditions.
+7. SmRbt
+The SmRbt service provides mechanisms for populating and navigating +"red/black trees" (RBTs) residing in shared DRAM. RBTs offer an +alternative to linked lists: like linked lists they can be navigated as +queues, but locating a single element of an RBT by its "key" value can +be much quicker than the equivalent search through an ordered linked +list.
+8. Simple Data Recorder (SDR)
+SDR is a system for managing non-volatile storage, built on exactly the +same model as PSM. Put another way, SDR is a small and simple +"persistent object" system or "object database" management system. It +enables straightforward management of linked lists (and other data +structures of arbitrary complexity) in non-volatile storage, notionally +within a single file whose size is pre-defined and fixed.
+SDR includes a transaction mechanism that protects database integrity by +ensuring that the failure of any database operation will cause all other +operations undertaken within the same transaction to be backed out. The +intent of the system is to assure retention of coherent protocol engine +state even in the event of an unplanned flight computer reboot in the +midst of communication activity.
+9. Sptrace
+The sptrace system is an embedded diagnostic facility that monitors the +performance of the PSM and SDR space management systems. It can be used, +for example, to detect memory "leaks" and other memory management +errors.
+10. Zco
+ION's zco (zero-copy objects) system leverages the SDR system's storage +flexibility to enable user application data to be encapsulated in any +number of layers of protocol without copying the successively augmented +protocol data unit from one layer to the next. It also implements a +reference counting system that enables protocol data to be processed +safely by multiple software elements concurrently -- e.g., a bundle may +be both delivered to a local endpoint and, at the same time, queued for +forwarding to another node -- without requiring that distinct copies of +the data be provided to each element.
+11. Rfx
+The ION rfx (R/F Contacts) system manages lists of scheduled +communication opportunities in support of a number of LTP and BP +functions.
+12. Ionsec
+The IONSEC (ION security) system manages information that supports the +implementation of security mechanisms in the other packages: security +policy rules and computation keys.
+The ION implementation of LTP conforms fully to RFC 5326, but it also +provides two additional features that enhance functionality without +affecting interoperability with other implementations:
+The service data units -- nominally bundles -- passed to LTP for + transmission may be aggregated into larger blocks before + segmentation. By controlling block size we can control the volume of + acknowledgement traffic generated as blocks are received, for + improved accommodation of highly asynchronous data rates.
+The maximum number of transmission sessions that may be concurrently + managed by LTP (a protocol control parameter) constitutes a + transmission "window" -- the basis for a delay-tolerant, + non-conversational flow control service over interplanetary links.
+In the ION stack, LTP serves effectively the same role that is performed +by an LLC protocol (such as IEEE 802.2) in the Internet architecture, +providing flow control and retransmission-based reliability between +topologically adjacent bundle protocol agents.
+All LTP session state is safely retained in the ION heap for rapid +recovery from a spacecraft or software fault.
+The ION implementation of BP conforms fully to RFC 5050, including +support for the following standard capabilities:
+Prioritization of data flows
+Proactive bundle fragmentation
+Bundle reassembly from fragments
+Flexible status reporting
+Custody transfer, including re-forwarding of custodial bundles upon + timeout interval expiration or failure of nominally reliable + convergence-layer transmission
+The system also provides three additional features that enhance +functionality without affecting interoperability with other +implementations:
+Rate control provides support for congestion forecasting and + avoidance.
+Bundle headers are encoded into compressed form (CBHE, as noted + earlier) before issuance, to reduce protocol overhead and improve + link utilization.
+Bundles may be "multicast" to all nodes that have registered within + a given multicast group endpoint.
+In addition, ION BP includes a system for computing dynamic routes +through time-varying network topology assembled from scheduled, bounded +communication opportunities. This system, called "Contact Graph +Routing," is described later in this Guide.
+In short, BP serves effectively the same role that is performed by IP in +the Internet architecture, providing route computation, forwarding, +congestion avoidance, and control over quality of service.
+All bundle transmission state is safely retained in the ION heap for +rapid recovery from a spacecraft or software fault.
+The ION implementation of the CCSDS AMS standard conforms fully to CCSDS +735.0-B-1. AMS is a data system communications architecture under which +the modules of mission systems may be designed as if they were to +operate in isolation, each one producing and consuming mission +information without explicit awareness of which other modules are +currently operating. Communication relationships among such modules are +self-configuring; this tends to minimize complexity in the development +and operations of modular data systems.
+A system built on this model is a "society" of generally autonomous +inter-operating modules that may fluctuate freely over time in response +to changing mission objectives, modules' functional upgrades, and +recovery from individual module failure. The purpose of AMS, then, is to +reduce mission cost and risk by providing standard, reusable +infrastructure for the exchange of information among data system modules +in a manner that is simple to use, highly automated, flexible, robust, +scalable, and efficient.
+A detailed discussion of AMS is beyond the scope of this Design Guide. +For more information, please see the [AMS Programmer's +Guide]{.underline}.
+The DGR package in ION is an alternative implementation of LTP that is +designed to operate responsibly -- i.e., with built-in congestion +control -- in the Internet or other IP-based networks. It is provided as +a candidate "primary transfer service" in support of AMS operations in +an Internet-like (non-delay-tolerant) environment. The DGR design +combines LTP's concept of concurrent transmission transactions with +congestion control and timeout interval computation algorithms adapted +from TCP.
+The ION implementation of CFDP conforms fully to Service Class 1 +(Unreliable Transfer) of CCSDS 727.0-B-4, including support for the +following standard capabilities:
+Segmentation of files on user-specified record boundaries.
+Transmission of file segments in protocol data units that are + conveyed by an underlying Unitdata Transfer service, in this case + the DTN protocol stack. File data segments may optionally be + protected by CRCs. When the DTN protocol stack is configured for + reliable data delivery (i.e., with BP custody transfer running over + a reliable convergence-layer protocol such as LTP), file delivery is + reliable; CFDP need not perform retransmission of lost data itself.
+Reassembly of files from received segments, possibly arriving over a + variety of routes through the delay-tolerant network. The integrity + of the delivered files is protected by checksums.
+User-specified fault handling procedures.
+Operations (e.g., directory creation, file renaming) on remote file + systems.
+All CFDP transaction state is safely retained in the ION heap for rapid +recovery from a spacecraft or software fault.
+The BSS service provided in ION enables a stream of video, audio, or +other continuously generated application data units, transmitted over a +delay-tolerant network, to be presented to a destination application in +two useful modes concurrently:
+In the order in which the data units were generated, with the least + possible end-to-end delivery latency, but possibly with some gaps + due to transient data loss or corruption.
+In the order in which the data units were generated, without gaps + (i.e., including lost or corrupt data units which were omitted from + the real-time presentation but were subsequently retransmitted), but + in a non-real-time "playback" mode.
+The TC service provided in ION enables critical but non-confidential +information (such as public keys, for asymmetric cryptography) to be +provided in a delay-tolerant, trustworthy manner. An instance of TC +comprises:
+A distributed Authority, the members of which must reach consensus + on database content and must collaborate on the proactive + distribution of that content.
+Any number of Clients, which:
+Announce new content to the Authority via authenticated bundle + multicast, and/or
+Receive trustworthy bulletins multicast by the members of the + Authority.
+| Acronyms | +Description | +
|---|---|
| BP | +Bundle Protocol | +
| BSP | +Bundle Security Protocol | +
| BSS | +Bundle Streaming Service | +
| CCSDS | +Consultative Committee for Space Data Systems | +
| CFDP | +CCSDS File Delivery Protocol | +
| CGR | +Contact Graph Routing | +
| CL | +convergence layer | +
| CLI | +convergence layer input | +
| CLO | +convergence layer output | +
| DTKA | +Delay-Tolerant Key Administration | +
| DTN | +Delay-Tolerant Networking | +
| ICI | +Interplanetary Communication Infrastructure | +
| ION | +Interplanetary Overlay Network | +
| LSI | +link service input | +
| LSO | +link service output | +
| LTP | +Licklider Transmission Protocol | +
| OWLT | +one-way light time | +
| RFC | +request for comments | +
| RFX | +Radio (R/F) Contacts | +
| RTT | +round-trip time | +
| TC | +Trusted Collective | +
| TTL | +time to live | +
A small number of network operation design elements -- fragmentation and +reassembly, bandwidth management, and delivery assurance +(retransmission) -- can potentially be addressed at multiple layers of +the protocol stack, possibly in different ways for different reasons. In +stack design it's important to allocate this functionality carefully so +that the effects at lower layers complement, rather than subvert, the +effects imposed at higher layers of the stack. This allocation of +functionality is discussed below, together with a discussion of several +related key concepts in the ION design.
+To minimize transmission overhead and accommodate asymmetric links +(i.e., limited "uplink" data rate from a ground data system to a +spacecraft) in an interplanetary network, we ideally want to send +"downlink" data in the largest possible aggregations -- [coarse-grained +transmission]{.underline}.
+But to minimize head-of-line blocking (i.e., delay in transmission of a +newly presented high-priority item) and minimize data delivery latency +by using parallel paths (i.e., to provide fine-grained partial data +delivery, and to minimize the impact of unexpected link termination), we +want to send "downlink" data in the smallest possible aggregations -- +[fine-grained transmission]{.underline}.
+We reconcile these impulses by doing both, but at different layers of +the ION protocol stack.
+First, at the application service layer (AMS and CFDP) we present +relatively small application data units (ADUs) -- on the order of 64 KB +-- to BP for encapsulation in bundles. This establishes an upper bound +on head-of-line blocking when bundles are de-queued for transmission, +and it provides perforations in the data stream at which forwarding can +readily be switched from one link (route) to another, enabling partial +data delivery at relatively fine, application-appropriate granularity.
+(Alternatively, large application data units may be presented to BP and +the resulting large bundles may be proactively fragmented at the time +they are presented to the convergence-layer manager. This capability is +meant to accommodate environments in which the convergence-layer manager +has better information than the application as to the optimal bundle +size, such as when the residual capacity of a contact is known to be +less than the size of the bundle.)
+Then, at the BP/LTP convergence layer adapter lower in the stack, we +aggregate these small bundles into blocks for presentation to LTP:
+Any continuous sequence of bundles that are to be shipped to the same +LTP engine and all require assured delivery may be aggregated into a +single block, to reduce overhead and minimize report traffic.
+However, this aggregation is constrained by an aggregation size limit +rule: aggregation must stop and the block must be transmitted as soon as +the sum of the sizes of all bundles aggregated into the block exceeds +the block aggregation threshhold value declared for the applicable +span (the relationship between the local node's LTP engine and the +receiving LTP engine) during LTP protocol configuration via +ltpadmin.
+Given a preferred block acknowledgment period -- e.g., a preferred +acknowledgement traffic rate of one report per second -- the nominal +block aggregation threshold is notionally computed as the amount of data +that can be sent over the link to the receiving LTP engine in a single +block acknowledgment period at the planned outbound data rate to that +engine.
+Taken together, application-level fragmentation (or BP proactive +fragmentation) and LTP aggregation place an upper limit on the amount of +data that would need to be re-transmitted over a given link at next +contact in the event of an unexpected link termination that caused +delivery of an entire block to fail. For example, if the data rate is 1 +Mbps and the nominal block size is 128 KB (equivalent to 1 second of +transmission time), we would prefer to avoid the risk of having wasted +five minutes of downlink in sending a 37.5 MB file that fails on +transmission of the last kilobyte, forcing retransmission of the entire +37.5 MB. We therefore divide the file into, say, 1200 bundles of 32 KB +each which are aggregated into blocks of 128 KB each: only a single +block failed, so only that block (containing just 4 bundles) needs to be +retransmitted. The cost of this retransmission is only 1 second of link +time rather than 5 minutes. By controlling the cost of convergence-layer +protocol failure in this way, we avoid the overhead and complexity of +"reactive fragmentation" in the BP implementation.
+Finally, within LTP itself we fragment the block as necessary to +accommodate the Maximum Transfer Unit (MTU) size of the underlying link +service, typically the transfer frame size of the applicable CCSDS link +protocol.
+The allocation of bandwidth (transmission opportunity) to application +data is requested by the application task that's passing data to DTN, +but it is necessarily accomplished only at the lowest layer of the stack +at which bandwidth allocation decisions can be made -- and then always +in the context of node policy decisions that have global effect.
+The transmission queue interface to a given neighbor in the network is +actually three queues of outbound bundles rather than one: one queue for +each of the defined levels of priority ("class of service") supported by +BP. When an application presents an ADU to BP for encapsulation in a +bundle, it indicates its own assessment of the ADU's priority. Upon +selection of a proximate forwarding destination node for that bundle, +the bundle is appended to whichever of the queues corresponds to the +ADU's priority.
+Normally the convergence-layer manager (CLM) task servicing a given +proximate node extracts bundles in strict priority order from the heads +of the three queues. That is, the bundle at the head of the +highest-priority non-empty queue is always extracted.
+However, if the ION_BANDWIDTH_RESERVED compiler option is selected at +the time ION is built, the convergence-layer manager task servicing a +given proximate node extracts bundles in interleaved fashion from the +heads of the node's three queues:
+Whenever the priority-2 ("express") queue is non-empty, the bundle + at the head of that queue is the next one extracted.
+At all other times, bundles from both the priority-1 queue and the + priority-0 queue are extracted, but over a given period of time + twice as many bytes of priority-1 bundles will be extracted as bytes + of priority-0 bundles.
+Following insertion of the extracted bundles into transmission buffers, +CLO tasks other than ltpclo simply segment the buffered bundles as +necessary and transmit them using the underlying convergence-layer +protocols. In the case of ltpclo, the output task aggregates the +buffered bundles into blocks as described earlier and a second daemon +task named ltpmeter waits for aggregated blocks to be completed; +ltpmeter, rather than the CLO task itself, segments each completed +block as necessary and passes the segments to the link service protocol +that underlies LTP. Either way, the transmission ordering requested by +application tasks is preserved.
+In the Internet, protocol operations can be largely driven by currently +effective information that is discovered opportunistically and +immediately, at the time it is needed, because the latency in +communicating this information over the network is negligible: distances +between communicating entities are small and connectivity is continuous. +In a DTN-based network, however, ad-hoc information discovery would in +many cases take so much time that it could not be completed before the +information lost currency and effectiveness. Instead, protocol +operations must be largely driven by information that is pre-placed at +the network nodes and tagged with the dates and times at which it +becomes effective. This information takes the form of contact plans +that are managed by the R/F Contacts (rfx) services of ION's ici +package.
+
Figure 6 RFX services in ION
+The structure of ION's RFX (contact plan) database, the rfx system +elements that populate and use that data, and affected portions of the +BP and LTP protocol state databases are shown in Figure 6. (For +additional details of BP and LTP database management, see the BP/LTP +discussion later in this document.)
+To clarify the notation of this diagram, which is also used in other +database structure diagrams in this document:
+Data objects of defined structure are shown as circles. Dark + greyscale indicates notionally non-volatile data retained in "heap" + storage, while lighter greyscale indicates volatile data retained in + dynamic random access memory.
+Solid arrows connecting circles indicate one-to-many cardinality.
+A dashed arrow between circles indicates a potentially many-to-one + reference mapping.
+Arrows from processing elements (rectangles) to data entities + indicate data production, while arrows from data entities to + processing elements indicate data retrieval.
+A contact is here defined as an interval during which it is expected +that data will be transmitted by DTN node A (the contact's transmitting +node) and most or all of the transmitted data will be received by node B +(the contact's receiving node). Implicitly, the transmitting mode will +utilize some "convergence-layer" protocol underneath the Bundle Protocol +to effect this direct transmission of data to the receiving node. Each +contact is characterized by its start time, its end time, the identities +of the transmitting and receiving nodes, and the rate at which data are +expected to be transmitted by the transmitting node throughout the +indicated time period.
+(Note that a contact is specifically not an episode of activity on +a link. Episodes of activity on different links -- e.g., different radio +transponders operating on the same spacecraft -- may well overlap, but +contacts by definition cannot; they are bounded time intervals and as +such are innately "tiled". For example, suppose transmission on link X +from node A to node B, at data rate RX, begins at time T1 and ends at +time T2; also, transmission on link Y from node A to node B, at data +rate RY begins at time T3 and ends at time T4. If T1 = T3 and T2 = T4, +then there is a single contact from time T1 to time T2 at data rate RX + +RY. If T1 \< T3 and T2 = T4, then there are two contiguous contacts: one +from T1 to T3 at data rate RX, then one from T3 to T2 at data rate RX + +RY. If T1 \< T3 and T3\<T2 \< T4, then there are three contiguous +contacts: one from T1 to T3 at data rate RX, then one from T3 to T2 at +data rate RX + RY, then one from T2 to T4 at data rate RY. And so on.)
+A range interval is a period of time during which the displacement +between two nodes A and B is expected to vary by less than 1 light +second from a stated anticipated distance. (We expect this information +to be readily computable from the known orbital elements of all nodes.) +Each range interval is characterized by its start time, its end time, +the identities of the two nodes to which it pertains, and the +anticipated approximate distance between those nodes throughout the +indicated time period, to the nearest light second.
+The topology timeline at each node in the network is a time-ordered +list of scheduled or anticipated changes in the topology of the network. +Entries in this list are of two types:
+• Contact entries characterize scheduled contacts.
+• Range entries characterize anticipated range intervals.
+Each node to which, according to the RFX database, the local node +transmits data directly via some convergence-layer protocol at some time +is termed a neighbor of the local node. Each neighbor is associated +with one or more outduct for the applicable BP convergence-layer (CL) +protocol adapter(s), so bundles that are to be transmitted directly to +this neighbor can simply be queued for transmission by outduct (as +discussed in the Bandwidth Management notes above).
+At startup, and at any time while the system is running, ionadmin +inserts and removes Contact and Range entries in the topology timeline +of the RFX database. Inserting or removing a Contact or Range entry will +cause routing tables to be recomputed for the destination nodes of all +subsequently forwarded bundles, as described in the discussion of +Contact Graph Routing below.
+Once per second, the rfxclock task (which appears in multiple +locations on the diagram to simplify the geometry) applies all topology +timeline events (Contact and Range start, stop, purge) with effective +time in the past. Applying a Contact event that cites a neighboring node +revises the transmission or reception data rate between the local node +and that Neighbor. Applying a Range event that cites a neighboring node +revises the OWLT between the local node and that neighbor. Setting data +rate or OWLT for a node with which the local node will at some time be +in direct communication may entail creation of a Neighbor object.
+ION's computation of a route for a given bundle with a given destination +endpoint is accomplished by one of several methods, depending on the +destination. In every case, the result of successful routing is the +insertion of the bundle into an outbound transmission queue (selected +according to the bundle's priority) for one or more neighboring nodes.
+But before discussing these methods it will be helpful to establish some +terminology:
+Egress plans
ION can only forward bundles to a neighboring node by queuing them on +some explicitly specified transmission queue. Specifications that +associate neighboring nodes with outducts are termed egress plans. +They are retained in ION's unicast forwarding database.
+Static routes
ION can be configured to forward to some specified node all bundles that +are destined for a given node to which no dynamic route can be +discovered from an examination of the contact graph, as described later. +Static routing is implemented by means of the "exit" mechanism described +below.
+Unicast
When the destination of a bundle is a single node that is registered +within a known "singleton endpoint" (that is, an endpoint that is known +to have exactly one member), then transmission of that bundle is termed +unicast. For this purpose, the destination endpoint ID must be a URI +formed in either the "dtn" scheme (e.g., dtn://bobsmac/mail) or the +"ipn" scheme (e.g., ipn:913.11).
+Exits
When unicast routes must be computed to nodes for which no contact plan +information is known (e.g., the size of the network makes it impractical +to distribute all Contact and Range information for all nodes to every +node, or the destination nodes don't participate in Contact Graph +Routing at all), the job of computing routes to all nodes may be +partitioned among multiple exit nodes. Each exit is responsible for +managing routing information (for example, a comprehensive contact +graph) for some subset of the total network population -- a group +comprising all nodes whose node numbers fall within the range of node +numbers assigned to the exit. A bundle destined for a node for which no +dynamic route can be computed from the local node's contact graph may be +routed to the exit node for the group within whose range the +destination's node number falls. Exits are defined in ION's unicast +forwarding database. (Note that the exit implements static routes in +ION in addition to improving scalability.)
+Multicast
When the destination of a bundle is all nodes that are registered within +a known "multicast endpoint" (that is, an endpoint that is not known to +have exactly one member), then transmission of that bundle is termed +multicast. For this purpose (in ION), the destination endpoint ID must +be a URI formed in the "imc" scheme (e.g., imc:913.11).
+Multicast Groups
A multicast group is the set of all nodes in the network that are +members of a given multicast endpoint. Forwarding a bundle to all +members of its destination multicast endpoint is the responsibility of +all of the multicast-aware nodes of the network. These nodes are +additionally configured to be nodes of a single multicast spanning tree +overlaid onto the dtnet. A single multicast tree serves to forward +bundles to all multicast groups: each node of the tree manages petitions +indicating which of its "relatives" (parent and children) are currently +interested in bundles destined for each multicast endpoint, either +natively (due to membership in the indicated group) or on behalf of more +distant relatives.
+We begin unicast route computation by attempting to compute a dynamic +route to the bundle's final destination node. The details of this +algorithm are described in the section on Contact Graph Routing, +below.
+If no dynamic route can be computed, but the final destination node is a +"neighboring" node that is directly reachable, then we assume that +taking this direct route is the best strategy unless transmission to +that neighbor is flagged as "blocked" for network operations purposes.
+Otherwise we must look for a static route. If the bundle's destination +node number is in one of the ranges of node numbers assigned to exit +nodes, then we forward the bundle to the exit node for the smallest such +range. (If the exit node is a neighbor and transmission to that neighbor +is not blocked, we simply queue the bundle for transmission to that +neighbor; otherwise we similarly look up the static route for the exit +node until eventually we resolve to some egress plan.)
+If we can determine neither a dynamic route nor a static route for this +bundle, but the reason for this failure was transmission blockage that +might be resolved in the future, then the bundle is placed in a "limbo" +list for future re-forwarding when transmission to some node is +"unblocked."
+Otherwise, the bundle cannot be forwarded. If custody transfer is +requested for the bundle, we send a custody refusal to the bundle's +current custodian; in any case, we discard the bundle.
+Multicast route computation is much simpler.
+When an endpoint for the "imc" scheme is added on an ION node -- + that is, when the node joins that multicast endpoint -- BP + administrative records noting the node's new interest in the + application topic corresponding to the endpoint's group number are + multicast to other network nodes as needed, using a "built-in" + multicast group of which all nodes of the network are implicitly + members. On receipt of such a record, each node notes the sending + relative's interest and forwards the record to other nodes as + necessary, and so on. (Deletion of endpoints results in similar + propagation of cancelling administrative records.)
+A bundle whose destination endpoint cites a multicast group, whether + locally sourced or received from another node:
+Is delivered immediately, if the local node is a member of the + indicated endpoint.
+Is queued for direct transmission to all other nodes in the + local "region" of network topology that are members of the + multicast group. Passageway nodes forward the bundle as + necessary into other regions that are topologically adjacent to + the local region.
+End-to-end delivery of data can fail in many ways, at different layers +of the stack. When delivery fails, we can either accept the +communication failure or retransmit the data structure that was +transmitted at the stack layer at which the failure was detected. ION is +designed to enable retransmission at multiple layers of the stack, +depending on the preference of the end user application.
+At the lowest stack layer that is visible to ION, the convergence-layer +protocol, failure to deliver one or more segments due to segment loss or +corruption will trigger segment retransmission if a "reliable" +convergence-layer protocol is in use: LTP "red-part" transmission or TCP +(including Bundle Relay Service, which is based on TCP)1.
+Segment loss may be detected and signaled via NAK by the receiving +entity, or it may only be detected at the sending entity by expiration +of a timer prior to reception of an ACK. Timer interval computation is +well understood in a TCP environment, but it can be a difficult problem +in an environment of scheduled contacts as served by LTP. The round-trip +time for an acknowledgment dialogue may be simply twice the one-way +light time (OWLT) between sender and receiver at one moment, but it may +be hours or days longer at the next moment due to cessation of scheduled +contact until a future contact opportunity. To account for this timer +interval variability in retransmission, the ltpclock task infers the +initiation and cessation of LTP transmission, to and from the local +node, from changes in the current xmit and recv data rates in the +corresponding Neighbor objects. This controls the dequeuing of LTP +segments for transmission by underlying link service adapter(s) and it +also controls suspension and resumption of timers, removing the effects +of contact interruption from the retransmission regime. For a further +discussion of this mechanism, see the section below on LTP Timeout +Intervals.
+Note that the current OWLT in Neighbor objects is also used in the +computation of the nominal expiration times of timers and that +ltpclock is additionally the agent for LTP segment retransmission +based on timer expiration.
+It is, of course, possible for the nominally reliable convergence-layer +protocol to fail altogether: a TCP connection might be abruptly +terminated, or an LTP transmission might be canceled due to excessive +retransmission activity (again possibly due to an unexpected loss of +connectivity). In this event, BP itself detects the CL protocol failure +and re-forwards all bundles whose acquisition by the receiving entity is +presumed to have been aborted by the failure. This re-forwarding is +initiated in different ways for different CL protocols, as implemented +in the CL input and output adapter tasks. If immediate re-forwarding is +impossible because transmission to all potentially viable neighbors is +blocked, the affected bundles are placed in the limbo list for future +re-forwarding when transmission to some node is unblocked.
+In addition to the implicit forwarding failure detected when a CL +protocol fails, the forwarding of a bundle may be explicitly refused by +the receiving entity, provided the bundle is flagged for custody +transfer service. A receiving node's refusal to take custody of a bundle +may have any of a variety of causes: typically the receiving node either +(a) has insufficient resources to store and forward the bundle, (b) has +no route to the destination, or (c) will have no contact with the next +hop on the route before the bundle's TTL has expired. In any case, a +"custody refusal signal" (packaged in a bundle) is sent back to the +sending node, which must re-forward the bundle in hopes of finding a +more suitable route.
+Alternatively, failure to receive a custody acceptance signal within +some convergence-layer-specified or application-specified time interval +may also be taken as an implicit indication of forwarding failure. Here +again, when BP detects such a failure it attempts to re-forward the +affected bundle, placing the bundle in the limbo list if re-forwarding +is currently impossible.
+In the worst case, the combined efforts of all the retransmission +mechanisms in ION are not enough to ensure delivery of a given bundle, +even when custody transfer is requested. In that event, the bundle's +"time to live" will eventually expire while the bundle is still in +custody at some node: the bpclock task will send a bundle status +report to the bundle's report-to endpoint, noting the TTL expiration, +and destroy the bundle. The report-to endpoint, upon receiving this +report, may be able to initiate application-layer retransmission of the +original application data unit in some way. This final retransmission +mechanism is wholly application-specific, however.
+In the Internet, the rate of transmission at a node can be dynamically +negotiated in response to changes in level of activity on the link, to +minimize congestion. On deep space links, signal propagation delays +(distances) may be too great to enable effective dynamic negotiation of +transmission rates. Fortunately, deep space links are operationally +reserved for use by designated pairs of communicating entities over +pre-planned periods of time at pre-planned rates. Provided there is no +congestion inherent in the contact plan, congestion in the network can +be avoided merely by adhering to the planned contact periods and data +rates. Rate control in ION serves this purpose.
+While the system is running, transmission and reception of bundles is +constrained by the current capacity in the throttle of each +convergence-layer manager. Completed bundle transmission activity +reduces the current capacity of the applicable throttle by the capacity +consumption computed for that bundle. This reduction may cause the +throttle's current capacity to become negative. Once the current +capacity of the applicable throttle goes negative, activity is blocked +until non-negative capacity has been restored by bpclock.
+Once per second, the bpclock task increases the current capacity of +each throttle by one second's worth of traffic at the nominal data rate +for transmission to that node, thus enabling some possibly blocked +bundle transmission and reception to proceed.
+bpclock revises all throttles' nominal data rates once per second in +accord with the current data rates in the corresponding Neighbor +objects, as adjusted by rfxclock per the contact plan.
+Note that this means that, for any neighboring node for which there are +planned contacts, ION's rate control system will enable data flow only +while contacts are active.
+A further constraint on rates of data transmission in an ION-based +network is LTP flow control. LTP is designed to enable multiple block +transmission sessions to be in various stages of completion +concurrently, to maximize link utilization: there is no requirement to +wait for one session to complete before starting the next one. However, +if unchecked this design principle could in theory result in the +allocation of all memory in the system to incomplete LTP transmission +sessions. To prevent complete storage resource exhaustion, we set a firm +upper limit on the total number of outbound blocks that can be +concurrently in transit at any given time. These limits are established +by ltpadmin at node initialization time.
+The maximum number of transmission sessions that may be concurrently +managed by LTP therefore constitutes a transmission "window" -- the +basis for a delay-tolerant, non-conversational flow control service over +interplanetary links. Once the maximum number of sessions are in flight, +no new block transmission session can be initiated -- regardless of how +much outduct transmission capacity is provided by rate control -- until +some existing session completes or is canceled.
+Note that this consideration emphasizes the importance of configuring +the aggregation size limits and session count limits of spans during LTP +initialization to be consistent with the maximum data rates scheduled +for contacts over those spans.
+Congestion in a dtnet is the imbalance between data enqueuing and +dequeuing rates that results in exhaustion of queuing (storage) +resources at a node, preventing continued operation of the protocols at +that node.
+In ION, the affected queuing resources are allocated from notionally +non-volatile storage space in the SDR data store and/or file system. The +design of ION is required to prevent resource exhaustion by simply +refusing to enqueue additional data that would cause it.
+However, a BP router's refusal to enqueue received data for forwarding +could result in costly retransmission, data loss, and/or the "upstream" +propagation of resource exhaustion to other nodes. Therefore the ION +design additionally attempts to prevent [potential]{.underline} resource +exhaustion by forecasting levels of queuing resource occupancy and +reporting on any congestion that is predicted. Network operators, upon +reviewing these forecasts, may revise contact plans to avert the +anticipated resource exhaustion.
+The non-volatile storage used by ION serves several purposes: it +contains queues of bundles awaiting forwarding, transmission, and +delivery; it contains LTP transmission and reception sessions, including +the blocks of data that are being transmitted and received; it contains +queues of LTP segments awaiting radiation; it may contain CFDP +transactions in various stages of completion; and it contains protocol +operational state information, such as configuration parameters, static +routes, the contact graph, etc.
+Effective utilization of non-volatile storage is a complex problem. +Static pre-allocation of storage resources is in general less efficient +(and also more labor-intensive to configure) than storage resource +pooling and automatic, adaptive allocation: trying to predict a +reasonable maximum size for every data storage structure and then +rigidly enforcing that limit typically results in underutilization of +storage resources and underperformance of the system as a whole. +However, static pre-allocation is mandatory for safety-critical +resources, where certainty of resource availability is more important +than efficient resource utilization.
+The tension between the two approaches is analogous to the tension +between circuit switching and packet switching in a network: circuit +switching results in underutilization of link resources and +underperformance of the network as a whole (some peaks of activity can +never be accommodated, even while some resources lie idle much of the +time), but dedicated circuits are still required for some kinds of +safety-critical communication.
+So the ION data management design combines these two approaches (see 1.5 +above for additional discussion of this topic):
+A fixed percentage of the total SDR data store heap size (by + default, 40%) is statically allocated to the storage of protocol + operational state information, which is critical to the operation of + ION.
+Another fixed percentage of the total SDR data store heap size (by + default, 20%) is statically allocated to "margin", a reserve that + helps to insulate node management from errors in resource allocation + estimates.
+The remainder of the heap, plus all pre-allocated file system space, + is allocated to protocol traffic2.
+The maximum projected occupancy of the node is the result of computing a +congestion forecast for the node, by adding to the current occupancy +all anticipated net increases and decreases from now until some future +time, termed the horizon for the forecast.
+The forecast horizon is indefinite -- that is, "forever" -- unless +explicitly declared by network management via the ionadmin utility +program. The difference between the horizon and the current time is +termed the interval of the forecast.
+Net occupancy increases and decreases are of four types:
+Bundles that are originated locally by some application on the node, + which are enqueued for forwarding to some other node.
+Bundles that are received from some other node, which are enqueued + either for forwarding to some other node or for local delivery to an + application.
+Bundles that are transmitted to some other node, which are dequeued + from some forwarding queue.
+Bundles that are delivered locally to an application, which are + dequeued from some delivery queue.
+The type-1 anticipated net increase (total data origination) is computed +by multiplying the node's projected rate of local data production, as +declared via an ionadmin command, by the interval of the forecast. +Similarly, the type-4 anticipated net decrease (total data delivery) is +computed by multiplying the node's projected rate of local data +consumption, as declared via an ionadmin command, by the interval of +the forecast. Net changes of types 2 and 3 are computed by multiplying +inbound and outbound data rates, respectively, by the durations of all +periods of planned communication contact that begin and/or end within +the interval of the forecast.
+Congestion forecasting is performed by the ionwarn utility program. +ionwarn may be run independently at any time; in addition, the +ionadmin utility program automatically runs ionwarn immediately +before exiting if it executed any change in the contact plan, the +forecast horizon, or the node's projected rates of local data production +or consumption. Moreover, the rfxclock daemon program also runs +ionwarn automatically whenever any of the scheduled reconfiguration +events it dispatches result in contact state changes that might alter +the congestion forecast.
+If the final result of the forecast computation -- the maximum projected +occupancy of the node over the forecast interval -- is less than the +total protocol traffic allocation, then no congestion is forecast. +Otherwise, a congestion forecast status message is logged noting the +time at which maximum projected occupancy is expected to equal the total +protocol traffic allocation.
+Congestion control in ION, then, has two components:
+First, ION's congestion [detection]{.underline} is anticipatory (via +congestion forecasting) rather than reactive as in the Internet.
+Anticipatory congestion detection is important because the second +component -- congestion [mitigation]{.underline} -- must also be +anticipatory: it is the adjustment of communication contact plans by +network management, via the propagation of revised schedules for future +contacts.
+(Congestion mitigation in an ION-based network is likely to remain +mostly manual for many years to come, because communication contact +planning involves much more than orbital dynamics: science operations +plans, thermal and power constraints, etc. It will, however, rely on the +automated rate control features of ION, discussed above, which ensure +that actual network operations conform to established contact plans.)
+Rate control in ION is augmented by admission control. ION tracks the +sum of the sizes of all zero-copy objects currently residing in the heap +and file system at any moment. Whenever any protocol implementation +attempts to create or extend a ZCO in such a way that total heap or file +occupancy would exceed an upper limit asserted for the node, that +attempt is either blocked until ZCO space becomes available or else +rejected altogether.
+ION is designed to deliver critical data to its final destination with +as much certainty as possible (and optionally as soon as possible), but +otherwise to try to maximize link utilization. The delivery of critical +data is expedited by contact graph routing and bundle prioritization as +described elsewhere. Optimizing link utilization, however, is a more +complex problem.
+If the volume of data traffic offered to the network for transmission is +less than the capacity of the network, then all offered data should be +successfully delivered3. But in that case the users of the network +are paying the opportunity cost of whatever portion of the network +capacity was not used.
+Offering a data traffic volume that is exactly equal to the capacity of +the network is in practice infeasible. TCP in the Internet can usually +achieve this balance because it exercises end-to-end flow control: +essentially, the original source of data is blocked from offering a +message until notified by the final destination that transmission of +this message can be accommodated given the current negotiated data rate +over the end-to-end path (as determined by TCP's congestion control +mechanisms). In a delay-tolerant network no such end-to-end negotiated +data rate may exist, much less be knowable, so such precise control of +data flow is impossible.4
+The only alternative: the volume of traffic offered by the data source +must be greater than the capacity of the network and the network must +automatically discard excess traffic, shedding lower-priority data in +preference to high-priority messages on the same path.
+ION discards excess traffic proactively when possible and reactively +when necessary.
+Proactive data triage occurs when ION determines that it
+cannot compute a route that will deliver a given bundle to its final
+destination prior to expiration of the bundle's Time To Live (TTL). That
+is, a bundle may be discarded simply because its TTL is too short, but
+more commonly it will be discarded because the planned contacts to
+whichever neighboring node is first on the path to the destination are
+already fully subscribed: the queue of bundles awaiting transmission to
+that neighbor is already so long as to consume the entire capacity of
+all announced opportunities to transmit to it. Proactive data triage
+causes the bundle to be immediately destroyed as one for which there is
+"No known route to destination from here."
The determination of the degree to which a contact is subscribed is +based not only on the aggregate size of the queued bundles but also on +the estimated aggregate size of the overhead imposed by all the +convergence-layer (CL) protocol data units -- at all layers of the +underlying stack -- that encapsulate those bundles: packet headers, +frame headers, etc. This means that the accuracy of this overhead +estimate will affect the aggressiveness of ION's proactive data triage:
+If CL overhead is overestimated, the size of the bundle transmission + backlog for planned contacts will be overstated, unnecessarily + preventing the enqueuing of additional bundles -- a potential + under-utilization of available transmission capacity in the network.
+If CL overhead is underestimated, the size of the bundle + transmission backlog for planned contacts will be understated, + enabling the enqueuing of bundles whose transmission cannot in fact + be accomplished by the network within the constraints of the current + contact plan. This will eventually result in reactive data triage.
+Essentially, all reactive data triage -- the destruction
+of bundles due to TTL expiration prior to successful delivery to the
+final destination -- occurs when the network conveys bundles at lower
+net rates than were projected during route computation. These
+performance shortfalls can have a variety of causes:
As noted above, underestimating CL overhead causes CL overhead to + consume a larger fraction of contact capacity than was anticipated, + leaving less capacity for bundle transmission.
+Conversely, the total volume of traffic offered may have been + accurately estimated but the amount of contact capacity may be less + than was promised: a contact might be started late, stopped early, + or omitted altogether, or the actual data rate on the link might be + less than was advertised.
+Contacts may be more subtly shortened by the configuration of ION + itself. If the clocks on nodes are known not to be closely + synchronized then a "maximum clock error" of N seconds may be + declared, causing reception episodes to be started locally N seconds + earlier and stopped N seconds later than scheduled, to avoid missing + some transmitted data because it arrived earlier or later than + anticipated. But this mechanism also causes transmission episodes to + be started N seconds later and stopped N seconds earlier than + scheduled, to avoid transmitting to a neighbor before it is ready to + receive data, and this contact truncation ensures transmission of + fewer bundles than planned.
+Flow control within the convergence layer underlying the bundle + protocol may constrain the effective rate of data flow over a link + to a rate that's lower than the link's configured maximum data rate. + In particular, mis-configuration of the LTP flow control window can + leave transmission capacity unused while LTP engines are awaiting + acknowledgments.
+Even if all nodes are correctly configured, a high rate of data loss + or corruption due to unexpectedly high R/F interference or + underestimated acknowledgment round-trip times may cause an + unexpectedly high volume of retransmission traffic. This will + displace original bundle transmission, reducing the effective + "goodput" data rate on the link.
+Finally, custody transfer may propagate operational problems from + one part of the network to other nodes. One result of reduced + effective transmission rates is the accumulation of bundles for + which nodes have taken custody: the custodial nodes can't destroy + those bundles and reclaim the storage space they occupy until + custody has been accepted by "downstream" nodes, so abbreviated + contacts that prevent the flow of custody acceptances can increase + local congestion. This reduces nodes' own ability to take custody of + bundles transmitted by "upstream" custodians, increasing queue sizes + on those nodes, and so on. In short, custody transfer may itself + ultimately impose reactive data triage simply by propagating + congestion.
+Some level of data triage is essential to cost-effective network +utilization, and proactive triage is preferable because its effects can +be communicated immediately to users, improving user control over the +use of the network. Optimizing an ION-based network therefore amounts to +managing for a modicum of proactive data triage and as little reactive +data triage as possible. It entails the following:
+Estimating convergence-layer protocol overhead as accurately as + possible, erring (if necessary) on the side of optimism -- that is, + underestimating a little.
+As an example, suppose the local node uses LTP over CCSDS Telemetry to +send bundles. The immediate convergence-layer protocol is LTP, but the +total overhead per CL "frame" (in this case, per LTP segment) will +include not only the size of the LTP header (nominally 5 bytes) but also +the size of the encapsulating space packet header (nominally 6 bytes) +and the overhead imposed by the outer encapsulating TM frame.
+Suppose each LTP segment is to be wrapped in a single space packet, +which is in turn wrapped in a single TM frame, and Reed-Solomon encoding +is applied. An efficient TM frame size is 1115 bytes, with an additional +160 bytes of trailing Reed-Solomon encoding and another 4 bytes of +leading pseudo-noise code. The frame would contain a 6-byte TM frame +header, a 6-byte space packet header, a 5-byte LTP segment header, and +1098 bytes of some LTP transmission block.
+So the number of "payload bytes per frame" in this case would be 1098 +and the number of "overhead bytes per frame" would be 4 + 6 + 6 + 5 + +160 = 181. Nominal total transmission overhead on the link would be 181 +/ 1279 = about 14%.
+Synchronizing nodes' clocks as accurately as possible, so that + timing margins configured to accommodate clock error can be kept as + close to zero as possible.
+Setting the LTP session limit and block size limit as generously as + possible (whenever LTP is at the convergence layer), to assure that + LTP flow control does not constrain data flow to rates below those + supported by BP rate control.
+Setting ranges (one-way light times) and queuing delays as + accurately as possible, to prevent unnecessary retransmission. Err + on the side of pessimism -- that is, overestimate a little.
+Communicating changes in configuration -- especially contact plans + -- to all nodes as far in advance of the time they take effect as + possible.
+Providing all nodes with as much storage capacity as possible for + queues of bundles awaiting transmission.
+Although the operation of BP/LTP in ION is complex in some ways, +virtually the entire system can be represented in a single diagram. The +interactions among all of the concurrent tasks that make up the node -- +plus a Remote AMS task or CFDP UT-layer task, acting as the application +at the top of the stack -- are shown below. (The notation is as used +earlier but with semaphores added. Semaphores are shown as small +circles, with arrows pointing into them signifying that the semaphores +are being given and arrows pointing out of them signifying that the +semaphores are being taken.)
+
Figure 7 ION node functional overview
+Further details of the BP/LTP data structures and flow of control and +data appear on the following pages. (For specific details of the +operation of the BP and LTP protocols as implemented by the ION tasks, +such as the nature of report-initiated retransmission in LTP, please see +the protocol specifications. The BP specification is documented in +Internet RFC 5050, while the LTP specification is documented in Internet +RFC 5326.)
+
Figure 8: Bundle protocol database
+
Figure 9: Licklider transmission protocol database
+
Figure 10 BP forwarder
+
Figure 11 BP convergence layer output
+
Figure 12 LTP transmission metering
+
Figure 13 LTP link service output
+
Figure 14 LTP link service input
+CGR is a dynamic routing system that computes routes through a +time-varying topology of scheduled communication contacts in a DTN +network. It is designed to support operations in a space network based +on DTN, but it also could be used in terrestrial applications where +operation according to a predefined schedule is preferable to +opportunistic communication, as in a low-power sensor network.
+The basic strategy of CGR is to take advantage of the fact that, since +communication operations are planned in detail, the communication routes +between any pair of "bundle agents" in a population of nodes that have +all been informed of one another's plans can be inferred from those +plans rather than discovered via dialogue (which is impractical over +long-one-way-light-time space links).
+CGR relies on accurate contact plan information provided in the form of +contact plan messages that currently are only read from ionrc files +and processed by ionadmin, which retains them in a non-volatile +contact plan in the RFX database, in ION's SDR data store.
+Contact plan messages are of two types: contact messages and range +messages.
+Each contact message has the following content:
+The starting UTC time of the interval to which the message pertains.
+The stop time of this interval, again in UTC.
+The Transmitting node number.
+The Receiving node number.
+The planned rate of transmission from node A to node B over this + interval, in bytes per second.
+Each range message has the following content:
+The starting UTC time of the interval to which the message pertains.
+The stop time of this interval, again in UTC.
+Node number A.
+Node number B.
+The anticipated distance between A and B over this interval, in + light seconds.
+Note that range messages may be used to declare that the "distance" in +light seconds between nodes A and B is different in the B🡪A +direction from the distance in the A🡪B direction. While direct radio +communication between A and B will not be subject to such asymmetry, +it's possible for connectivity established using other convergence-layer +technologies to take different physical paths in different directions, +with different signal propagation delays.
+Each node uses Range and Contact messages in the contact plan to build a +\"routing table\" data structure.
+The routing table constructed locally by each node in the network is a +list of entry node lists, one route list for every other node D in the +network that is cited in any Contact or Range in the contact plan. Entry +node lists are computed as they are needed, and the maximum number of +entry node lists resident at a given time is the number of nodes that +are cited in any Contacts or Ranges in the contact plan. Each entry in +the entry node list for node D is a list of the neighbors of local node +X; included with each entry of the entry node list is a list one or more +routes to D through the indicated neighbor, termed a route list.
+Each route in the route list for node D identifies a path to destination +node D, from the local node, that begins with transmission to one of the +local node's neighbors in the network-- the initial receiving node for +the route, termed the route's entry node.
+For any given route, the contact from the local node to the entry node +constitutes the initial transmission segment of the end-to-end path to +the destination node. Additionally noted in each route object are all of +the other contacts that constitute the remaining segments of the route's +end-to-end path.
+Each route object also notes the forwarding cost for a bundle that is +forwarded along this route. In this version of ION, CGR is configured to +deliver bundles as early as possible, so best-case final delivery time +is used as the cost of a route. Other metrics might be substituted for +final delivery time in other CGR implementations. NOTE, however, that if +different metrics are used at different nodes along a bundle's +end-to-end path it becomes impossible to prevent routing loops that can +result in non-delivery of the data.
+Finally, each route object also notes the route's termination time, the +time after which the route will become moot due to the termination of +the earliest-ending contact in the route.
+Every bundle transmitted via DTN has a time-to-live (TTL), the length of +time after which the bundle is subject to destruction if it has not yet +been delivered to its destination. The expiration time of a bundle is +computed as its creation time plus its TTL. When computing the next-hop +destination for a bundle that the local bundle agent is required to +forward, there is no point in selecting a route that can\'t get the +bundle to its final destination prior to the bundle's expiration time.
+One-way light time (OWLT) -- that is, distance -- is obviously a factor +in delivering a bundle to a node prior to a given time. OWLT can +actually change during the time a bundle is en route, but route +computation becomes intractably complex if we can\'t assume an OWLT +\"safety margin\" -- a maximum delta by which OWLT between any pair of +nodes can change during the time a bundle is in transit between them.
+We assume that the maximum rate of change in distance between any two +nodes in the network is about 150,000 miles per hour, which is about 40 +miles per second. (This was the speed of the Helios spacecraft, the +fastest man-made object launched to date.)
+At this speed, the distance between any two nodes that are initially +separated by a distance of N light seconds will increase by a maximum of +80 miles per second of transit (in the event that they are moving in +opposite directions). This will result in data arrival no later than +roughly (N + 2Q) seconds after transmission -- where the "OWLT margin" +value Q is (40 * N) divided by 186,000 -- rather than just N seconds +after transmission as would be the case if the two nodes were stationary +relative to each other. When computing the expected time of arrival of a +transmitted bundle we simply use N + 2Q, the most pessimistic case, as +the anticipated total in-transit time.
+The capacity of a contact is the product of its data transmission rate +(in bytes per second) and its duration (stop time minus start time, in +seconds).
+The size of a bundle is the sum of its payload size and its header +size5, but bundle size is not the only lien on the capacity of a +contact. The total estimated volume consumption (or "EVC") for a bundle +is the sum of the sizes of the bundle's payload and header and the +estimated convergence-layer overhead. For a bundle whose header is of +size M and whose payload is of size N, the estimated convergence-layer +overhead is defined as 3% of (M+N), or 100 bytes, whichever is larger.
+The residual capacity of a given contact between the local node and +one of its neighbors, as computed for a given bundle, is the sum of the +capacities of that contact and all prior scheduled contacts between the +local node and that neighbor, less the sum of the ECCs of all bundles +with priority equal to or higher than the priority of the subject bundle +that are currently queued on the outduct for transmission to that +neighbor.
+A neighboring node C that refuses custody of a bundle destined for some +remote node D is termed an excluded neighbor for (that is, with +respect to computing routes to) D. So long as C remains an excluded +neighbor for D, no bundles destined for D will be forwarded to C -- +except that occasionally (once per lapse of the RTT between the local +node and C) a custodial bundle destined for D will be forwarded to C as +a "probe bundle". C ceases to be an excluded neighbor for D as soon as +it accepts custody of a bundle destined for D.
+A Critical bundle is one that absolutely has got to reach its +destination and, moreover, has got to reach that destination as soon as +is physically possible6.
+For an ordinary non-Critical bundle, the CGR dynamic route computation +algorithm uses the routing table to select a single neighboring node to +forward the bundle through. It is possible, though, that due to some +unforeseen delay the selected neighbor may prove to be a sub-optimal +forwarder: the bundle might arrive later than it would have if another +neighbor had been selected, or it might not even arrive at all.
+For Critical bundles, the CGR dynamic route computation algorithm causes +the bundle to be inserted into the outbound transmission queues for +transmission to [all]{.underline} neighboring nodes that can plausibly +forward the bundle to its final destination. The bundle is therefore +guaranteed to travel over the most successful route, as well as over all +other plausible routes. Note that this may result in multiple copies of +a Critical bundle arriving at the final destination.
+Given a bundle whose destination is node D, we proceed as follows.
+First, if no contacts in the contact plan identify transmission to node +D, then we cannot use CGR to find a route for this bundle; CGR route +selection is abandoned.
+Next, if the contact plan has been modified in any way since routes were +computed for any nodes, we discard all routes for all nodes and +authorize route recomputation. (The contact plan changes may have +invalidated any or all of those earlier computations.)
+We create an empty list of Proximate Nodes (network neighbors) to send +the bundle to.
+We create a list of Excluded Nodes, i.e., nodes through which we will +[not]{.underline} compute a route for this bundle. The list of Excluded +Nodes is initially populated with:
+the node from which the bundle was directly received (so that we + avoid cycling the bundle between that node and the local node) -- + [unless]{.underline} the Dynamic Route Selection Algorithm is being + re-applied due to custody refusal as discussed later;
+all excluded neighbors for the bundle's final destination node.
+If all routes computed for node D have been discarded due to contact +plan modification, then we must compute a new list of all routes from +the local node to D. To do so:
+We construct an abstract contact graph, a directed acyclic graph + whose root is a notional contact from the local node to itself and + whose other vertices are all other contacts representing + transmission "from" some node such that a contact "to" that node + already exists in the graph, excluding contacts representing + transmission "to" some node such that a contact "from" that node + already exists in the graph. A terminal vertex is also included in + the graph, constituting a notional contact from node D to itself.
+We perform several Dijkstra searches within this graph, one search + for each of the local node's neighbors. On each search we find the + lowest-cost route that begins at the root of the graph and ends at + the terminal vertex. Each time a route is computed, we add it to the + list of routes for that route's entry node and then remove from + further consideration all contacts from the local node to the entry + node of that route.
+The lowest-cost route computed during a search is the one that + is found to have the earliest best-case delivery time, where the + best-case delivery time characterizing a route is given by the + time at which a bundle would arrive at node D if transmitted at + the earliest possible moment of the last contact in the route + prior to the terminal vertex.
+Any contact whose end time is before the earliest possible time + that the bundle could arrive at the contact's sending node is + ignored.
+The earliest possible arrival time for the bundle on a given + contact is pessimistically computed as the sum of the bundle's + earliest possible transmission time plus the range in light + seconds from the contact's sending node to its receiving node, + plus the applicable one-way light time margin.
+The earliest possible transmission time for the bundle on a + given contact is the start time of the contact or bundle's + earliest possible arrival time at the contact's sending node, + whichever is later.
+If node D's list of entry nodes (route lists) is still empty, then + we cannot use CGR to find a route for this bundle; CGR route + selection is abandoned.
+We next examine all of the routes that are currently computed for +transmission of bundles to node D.
+Any route whose termination time is in the past is deleted from the + list, and all contacts in that route whose termination time is in + the past are also deleted. But we then run another Dijkstra search + to compute the best route through the affected entry node given the + remaining contacts; if this search finds a route, the new route is + inserted into the appropriate location in the list.
+Any route whose best-case final delivery time is after the bundle's + expiration time is ignored, as is any route whose entry node is in + the list of Excluded Nodes. Loopback routes are also ignored unless + the local node is the bundle's final destination.
+For each route, the aggregate radiation time for this bundle on this + route is computed by summing the product of payload size and contact + transmission rate over all contacts in the route. Any route for + which the sum of best-case delivery time and aggregate radiation + time is after the bundle's expiration time is ignored.
+For each route that is not ignored, the route's entry node is added to +the list of Proximate Nodes for this bundle. Associated with the entry +node number in this list entry are the best-case final delivery time of +the route, the total number of "hops" in the route's end-to-end path, +and the forfeit time for transmission to this node. Forfeit time is the +route's termination time, the time by which the bundle must have been +transmitted to this node in order to have any chance of being forwarded +on this route.
+If, at the end of this procedure, the Proximate Nodes list is empty, +then we have been unable to use CGR to find a route for this bundle; CGR +route selection is abandoned.
+Otherwise:
+If the bundle is flagged as a critical bundle, then a cloned copy of + this bundle is enqueued for transmission to every node in the + Proximate Nodes list.
+Otherwise, the bundle is enqueued for transmission on the outduct to + the most preferred neighbor in the Proximate Nodes list:
+If one of the nodes in this list is associated with a best-case + delivery time that is earlier than that of all other nodes in + the list, then it is the most preferred neighbor.
+Otherwise, if one of the nodes with the earliest best-case + delivery time is associated with a smaller hop count than every + other node with the same best-case delivery time, then it is the + most preferred neighbor.
+Otherwise, the node with the smallest node number among all + nodes with the earliest best-case delivery time and smallest hop + count is arbitrarily chosen as the most preferred neighbor.
+Conveyance of a bundle from source to destination through a DTN can fail +in a number of ways, many of which are best addressed by means of the +Delivery Assurance mechanisms described earlier. Failures in Contact +Graph Routing, specifically, occur when the expectations on which +routing decisions are based prove to be false. These failures of +information fall into two general categories: contact failure and +custody refusal.
+A scheduled contact between some node and its neighbor on the end-to-end +route may be initiated later than the originally scheduled start time, +or be terminated earlier than the originally scheduled stop time, or be +canceled altogether. Alternatively, the available capacity for a contact +might be overestimated due to, for example, diminished link quality +resulting in unexpectedly heavy retransmission at the convergence layer. +In each of these cases, the anticipated transmission of a given bundle +during the affected contact may not occur as planned: the bundle might +expire before the contact's start time, or the contact's stop time might +be reached before the bundle has been transmitted.
+For a non-Critical bundle, we handle this sort of failure by means of a +timeout: if the bundle is not transmitted prior to the forfeit time for +the selected Proximate Node, then the bundle is removed from its +outbound transmission queue and the Dynamic Route Computation Algorithm +is re-applied to the bundle so that an alternate route can be computed.
+A node that receives a bundle may find it impossible to forward it, for +any of several reasons: it may not have enough storage capacity to hold +the bundle, it may be unable to compute a forward route (static, +dynamic, or default) for the bundle, etc. Such bundles are simply +discarded, but discarding any such bundle that is marked for custody +transfer will cause a custody refusal signal to be returned to the +bundle's current custodian.
+When the affected bundle is non-Critical, the node that receives the +custody refusal re-applies the Dynamic Route Computation Algorithm to +the bundle so that an alternate route can be computed -- except that in +this event the node from which the bundle was originally directly +received is omitted from the initial list of Excluded Nodes. This +enables a bundle that has reached a dead end in the routing tree to be +sent back to a point at which an altogether different branch may be +selected.
+For a Critical bundle no mitigation of either sort of failure is +required or indeed possible: the bundle has already been queued for +transmission on all plausible routes, so no mechanism that entails +re-application of CGR's Dynamic Route Computation Algorithm could +improve its prospects for successful delivery to the final destination. +However, in some environments it may be advisable to re-apply the +Dynamic Route Computation Algorithm to all Critical bundles that are +still in local custody whenever a new Contact is added to the contact +graph: the new contact may open an additional forwarding opportunity for +one or more of those bundles.
+The CGR routing procedures respond dynamically to the changes in network +topology that the nodes are able know about, i.e., those changes that +are subject to mission operations control and are known in advance +rather than discovered in real time. This dynamic responsiveness in +route computation should be significantly more effective and less +expensive than static routing, increasing total data return while at the +same time reducing mission operations cost and risk.
+Note that the non-Critical forwarding load across multiple parallel +paths should be balanced automatically:
+Initially all traffic will be forwarded to the node(s) on what is + computed to be the best path from source to destination.
+At some point, however, a node on that preferred path may have so + much outbound traffic queued up that no contacts scheduled within + bundles' lifetimes have any residual capacity. This can cause + forwarding to fail, resulting in custody refusal.
+Custody refusal causes the refusing node to be temporarily added to + the current custodian's excluded neighbors list for the affected + final destination node. If the refusing node is the only one on the + path to the destination, then the custodian may end up sending the + bundle back to its upstream neighbor. Moreover, that custodian node + too may begin refusing custody of bundles subsequently sent to it, + since it can no longer compute a forwarding path.
+The upstream propagation of custody refusals directs bundles over + alternate paths that would otherwise be considered suboptimal, + balancing the queuing load across the parallel paths.
+Eventually, transmission and/or bundle expiration at the + oversubscribed node relieves queue pressure at that node and enables + acceptance of custody of a "probe" bundle from the upstream node. + This eventually returns the routing fabric to its original + configuration.
+Although the route computation procedures are relatively complex they +are not computationally difficult. The impact on computation resources +at the vehicles should be modest.
+Suppose we've got Earth ground station ES that is currently in view of +Mars but will be rotating out of view ("Mars-set") at some time T1 and +rotating back into view ("Mars-rise") at time T3. Suppose we've also got +Mars orbiter MS that is currently out of the shadow of Mars but will +move behind Mars at time T2, emerging at time T4. Let\'s also suppose +that ES and MS are 4 light-minutes apart (Mars is at its closest +approach to Earth). Finally, for simplicity, let's suppose that both ES +and MS want to be communicating at every possible moment (maximum link +utilization) but never want to waste any electricity.
+Neither ES nor MS wants to be wasting power on either transmitting or +receiving at a time when either Earth or Mars will block the signal.
+ES will therefore stop transmitting at either T1 or (T2 - 4 minutes), +whichever is earlier; call this time T~et0~. It will stop receiving -- +that is, power off the receiver -- at either T1 or (T2 + 4 minutes), +whichever is earlier; call this time T~er0~. It will resume transmitting +at either T3 or (T4 - 4 minutes), whichever is [later]{.underline}, and +it will resume reception at either T3 or (T4 + 4 minutes), whichever is +later; call these times T~et1~ and T~er1~.
+Similarly, MS will stop transmitting at either T2 or (T1 - 4 minutes), +whichever is earlier; call this time T~mt0~. It will stop receiving -- +that is, power off the receiver -- at either T2 or (T1 + 4 minutes), +whichever is earlier; call this time T~mr0~. It will resume transmitting +at either T4 or (T3 - 4 minutes), whichever is later, and it will resume +reception at either T4 or (T3 + 4 minutes), whichever is later; call +these times T~mt1~ and T~mr1~.
+By making sure that we don't transmit when the signal would be blocked, +we guarantee that anything that is transmitted will arrive at a time +when it can be received. Any reception failure is due to data corruption +en route.
+So the moment of transmission of an acknowledgment to any message is +always equal to the moment the original message was sent plus some +imputed outbound queuing delay QO1 at the sending node, plus 4 minutes, +plus some imputed inbound and outbound queuing delay QI1 + QO2 at the +receiving node. The nominally expected moment of reception of this +acknowledgment is that moment of transmission plus 4 minutes, plus some +imputed inbound queuing delay QI2 at the original sending node. That is, +the timeout interval is 8 minutes + QO1 + QI1 + QO2 + QO2 -- unless +this moment of acknowledgement transmission is during an interval when +the receiving node is not transmitting, for whatever reason. In this +latter case, we want to suspend the acknowledgment timer during any +interval in which we know the remote node will not be transmitting. More +precisely, we want to add to the timeout interval the time difference +between the moment of message arrival and the earliest moment at which +the acknowledgment could be sent, i.e., the moment at which transmission +is resumed7.
+So the timeout interval Z computed at ES for a message sent to MS at +time T~X~ is given by:
+Z = QO1 + 8 + QI1 + ((T~A~ = T~X~ + 4) > T~mt0~ && T~A~ \< T~mt1~) ? +T~mt1~ -- T~A~: 0) + QI2 +QO2;
+This can actually be computed in advance (at time T~X~) if T1, T2, T3, +and T4 are known and are exposed to the protocol engine.
+If they are not exposed, then Z must initially be estimated to be (2 * +the one-way light time) + QI + QO. The timer for Z must be dynamically +suspended at time T~mt0~ in response to a state change as noted by +ltpclock. Finally, the timer must be resumed at time T~mt1~ (in +response to another state change as noted by ltpclock), at which +moment the correct value for Z can be computed.
+The ION implementation of CFDP is very simple, because only Class-1 +(Unacknowledged) functionality is implemented: the store-and-forward +routing performed by Bundle Protocol makes the CFDP Extended Procedures +unnecessary and the inter-node reliability provided by the CL protocol +underneath BP -- in particular, by LTP -- makes the CFDP Acknowledged +Procedures unnecessary. All that CFDP is required to do is segment and +reassemble files, interact with the underlying Unitdata Transfer layer +-- BP/LTP -- to effect the transmission and reception of file data +segments, and handle CFDP metadata including filestore requests. +CFDP-ION does all this, including support for cancellation of a file +transfer transaction by cancellation of the transmission of the bundles +encapsulating the transaction's protocol data units.
+Note that all CFDP data transmission is "by reference", via the ZCO +system, rather than "by value": the retransmission buffer for a bundle +containing CFDP file data is an extent of the original file itself, not +a copy retained in the ION database, and data received in bundles +containing CFDP PDU is written immediately to the appropriate location +in the reconstituted file rather than stored in the ION database. This +minimizes the space needed for the database. In general, file +transmission via CFDP is the most memory-efficient way to use ION in +flight operations.
+
+Figure 15 A CFDP-ION entity
{width="6.0in"
+height="3.6805555555555554in"}
Figure 16 ION list data structures
+
Figure 17 psm partition structure
+
Figure 18 psm and sdr block structures
+
Figure 19 sdr heap structure
+The ION source distribution contains a README.TXT file with details on +building ION from source. For installations starting with the +Sourceforge distribution, the standard sequence of
+./configure
+make
+sudo make install
+will build ION and install it under /usr/local.
+Users building from a clone of the repository need to use the command
+before starting the installation.
+The "Build" instructions shown in the following sections for each +package are the instructions for building each package individually, for +ION development purposes. The default installation target for the +individual package build commands is /opt.
+One compile-time option is applicable to all ION packages: the platform +selection parameters --DVXWORKS and --DRTEMS affect the manner in which +most task instantiation functions are compiled. For VXWORKS and RTEMS, +these functions are compiled as library functions that must be +identified by name in the platform's symbol table, while for Unix-like +platforms they are compiled as main()functions.
+Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DFSWSOURCE or +--DSM_SEMBASEKEY=0xff13), will alter the functionality of ION as noted +below.
+PRIVATE_SYMTAB
This option causes ION to be built for VxWorks 5.4 or RTEMS with +reliance on a small private local symbol table that is accessed by means +of a function named sm_FindFunction. Both the table and the function +definition are, by default, provided by the symtab.c source file, which +is automatically included within the platform_sm.c source when this +option is set. The table provides the address of the top-level function +to be executed when a task for the indicated symbol (name) is to be +spawned, together with the priority at which that task is to execute and +the amount of stack space to be allocated to that task.
+PRIVATE_SYMTAB is defined by default for RTEMS but not for VxWorks 5.4.
+Absent this option, ION on VxWorks 5.4 must successfully execute the +VxWorks symFindByName function in order to spawn a new task. For this +purpose the entire VxWorks symbol table for the compiled image must be +included in the image, and task priority and stack space allocation must +be explicitly specified when tasks are spawned.
+FSWLOGGER
This option causes the standard ION logging function, which simply +writes all ION status messages to a file named ion.log in the current +working directory, to be replaced (by #include) with code in the source +file fswlogger.c. A file of this name must be in the inclusion path for +the compiler, as defined by --Ixxxx compiler option parameters.
+FSWCLOCK
This option causes the invocation of the standard time function within +getUTCTime (in ion.c) to be replaced (by #include) with code in the +source file fswutc.c, which might for example invoke a mission-specific +function to read a value from the spacecraft clock. A file of this name +must be in the inclusion path for the compiler.
+FSWWDNAME
This option causes the invocation of the standard getcwd function within +cfdpInit (in libcfdpP.c) to be replaced (by #include) with code in the +source file wdname.c, which must in some way cause the mission-specific +value of the current working directory name to be copied into +cfdpdbBuf.workingDirectoryName. A file of this name must be in the +inclusion path for the compiler.
+FSWSYMTAB
If the PRIVATE_SYMTAB option is also set, then the FSWSYMTAB option +causes the code in source file mysymtab.c to be included in +platform_sm.c in place of the default symbol table access implementation +in symtab.c. A file named mysymtab.c must be in the inclusion path for +the compiler.
+FSWSOURCE
This option simply causes FSWLOGGER, FSWCLOCK, FSWWDNAME, and FSWSYMTAB +all to be set.
+GDSLOGGER
This option causes the standard ION logging function, which simply +writes all ION status messages to a file named ion.log in the current +working directory, to be replaced (by #include) with code in the source +file gdslogger.c. A file of this name must be in the inclusion path for +the compiler, as defined by --Ixxxx compiler option parameters.
+GDSSOURCE
+This option simply causes GDSLOGGER to be set.
+ION_OPS_ALLOC=*xx*
This option specifies the percentage of the total non-volatile storage +space allocated to ION that is reserved for protocol operational state +information, i.e., is not available for the storage of bundles or LTP +segments. The default value is 20.
+ION_SDR_MARGIN=*xx*
This option specifies the percentage of the total non-volatile storage +space allocated to ION that is reserved simply as margin, for +contingency use. The default value is 20.
+The sum of ION_OPS_ALLOC and ION_SDR_MARGIN defines the amount of +non-volatile storage space that is sequestered at the time ION +operations are initiated: for purposes of congestion forecasting and +prevention of resource oversubscription, this sum is subtracted from the +total size of the SDR "heap" to determine the maximum volume of space +available for bundles and LTP segments. Data reception and origination +activities fail whenever they would cause the total amount of data store +space occupied by bundles and segments to exceed this limit.
+USING_SDR_POINTERS
This is an optimization option for the SDR non-volatile data management +system: when set, it enables the value of any variable in the SDR data +store to be accessed directly by means of a pointer into the dynamic +memory that is used as the data store storage medium, rather than by +reading the variable into a location in local stack memory. Note that +this option must not be enabled if the data store is configured for +file storage only, i.e., if the SDR_IN_DRAM flag was set to zero at the +time the data store was created by calling sdr_load_profile. See the +ionconfig(5) man page in Appendix A for more information.
+NO_SDR_TRACE
This option causes non-volatile storage utilization tracing functions to +be omitted from ION when the SDR system is built. It disables a useful +debugging option but reduces the size of the executable software.
+NO_PSM_TRACE
This option causes memory utilization tracing functions to be omitted +from ION when the PSM system is built. It disables a useful debugging +option but reduces the size of the executable software.
+IN_FLIGHT
This option controls the behavior of ION when an unrecoverable error is +encountered.
+If it is set, then the status message "Unrecoverable SDR error" is +logged and the SDR non-volatile storage management system is globally +disabled: the current database access transaction is ended and (provided +transaction reversibility is enabled) rolled back, and all ION tasks +terminate.
+Otherwise, the ION task that encountered the error is simply aborted, +causing a core dump to be produced to support debugging.
+SM_SEMKEY=0x*XXXX*
This option overrides the default value (0xee01) of the identifying +"key" used in creating and locating the global ION shared-memory system +mutex.
+SVR4_SHM
This option causes ION to be built using svr4 shared memory as the +pervasive shared-memory management mechanism. svr4 shared memory is +selected by default when ION is built for any platform other than MinGW, +VxWorks 5.4, or RTEMS. (For these latter operating systems all memory is +shared anyway, due to the absence of a protected-memory mode.)
+POSIX1B_SEMAPHORES
This option causes ION to be built using POSIX semaphores as the +pervasive semaphore mechanism. POSIX semaphores are selected by default +when ION is built for RTEMS but are otherwise not used or supported; +this option enables the default to be overridden.
+SVR4_SEMAPHORES
This option causes ION to be built using svr4 semaphores as the +pervasive semaphore mechanism. svr4 semaphores are selected by default +when ION is built for any platform other than MinGW (for which Windows +event objects are used), VxWorks 5.4 (for which VxWorks native +semaphores are the default choice), or RTEMS (for which POSIX semaphores +are the default choice).
+SM_SEMBASEKEY=0x*XXXX*
This option overrides the default value (0xee02) of the identifying +"key" used in creating and locating the global ION shared-memory +semaphore database, in the event that svr4 semaphores are used.
+SEMMNI=*xxx*
This option declares to ION the total number of svr4 semaphore sets +provided by the operating system, in the event that svr4 semaphores are +used. It overrides the default value, which is 10 for Cygwin and 128 +otherwise. (Changing this value typically entails rebuilding the O/S +kernel.)
+SEMMSL=*xxx*
This option declares to ION the maximum number of semaphores in each +svr4 semaphore set, in the event that svr4 semaphores are used. It +overrides the default value, which is 6 for Cygwin and 250 otherwise. +(Changing this value typically entails rebuilding the O/S kernel.)
+SEMMNS=*xxx*
This option declares to ION the total number of svr4 semaphores that the +operating system can support; the maximum possible value is SEMMNI x +SEMMSL. It overrides the default value, which is 60 for Cygwin and 32000 +otherwise. (Changing this value typically entails rebuilding the O/S +kernel.)
+ION_NO_DNS
This option causes the implementation of a number of Internet socket I/O +operations to be omitted for ION. This prevents ION software from being +able to operate over Internet connections, but it prevents link errors +when ION is loaded on a spacecraft where the operating system does not +include support for these functions.
+ERRMSGS_BUFSIZE=*xxxx*
This option set the size of the buffer in which ION status messages are +constructed prior to logging. The default value is 4 KB.
+SPACE_ORDER=*x*
This option declares the word size of the computer on which the compiled +ION software will be running: it is the base-2 log of the number of +bytes in an address. The default value is 2, i.e., the size of an +address is 2^2^ = 4 bytes. For a 64-bit machine, SPACE_ORDER must be +declared to be 3, i.e., the size of an address is 2^3^ = 8 bytes.
+NO_SDRMGT
This option enables the SDR system to be used as a data access +transaction system only, without doing any dynamic management of +non-volatile data. With the NO_SDRMGT option set, the SDR system library +can (and in fact must) be built from the sdrxn.c source file alone.
+DOS_PATH_DELIMITER
This option causes ION_PATH_DELIMITER to be set to '\' (backslash), for +use in construction path names. The default value of ION_PATH_DELIMITER +is '/' (forward slash, as is used in Unix-like operating systems).
+To build ICI for a given deployment platform:
+Decide where you want ION's executables, libraries, header files, + etc. to be installed. The ION makefiles all install their build + products to subdirectories (named bin, lib, include, + man, man/man1, man/man3, man/man5) of an ION root + directory, which by default is the directory named /opt. If you + wish to use the default build configuration, be sure that the + default directories (/opt/bin, etc.) exist; if not, select + another ION root directory name -- this document will refer to it as + $OPT -- and create the subdirectories as needed. In any case, + make sure that you have read, write, and execute permission for all + of the ION installation directories and that:
+The directory /$OPT/bin is in your execution path.
+The directory /$OPT/lib is in your $LD_LOADLIB_PATH.
+Edit the Makefile in ion/ici:
+Make sure PLATFORMS is set to the appropriate platform name, e.g., + x86-redhat, sparc-sol9, etc.
+Set OPT to the directory where you want to install the ici packages + you build, if other than "/opt" (for example: /usr/local).
+Then:
+Three types of files are used to provide the information needed to +perform global configuration of the ION protocol stack: the ION system +configuration (or ionconfig) file, the ION administration command +(ionrc) file, and the ION security configuration (ionsecrc) +file. For details, see the man pages for ionconfig(5), ionrc(5), and +ionsecrc(5) in Appendix A.
+Normally the instantiation of ION on a given computer establishes a +single ION node on that computer, for which hard-coded values of wmKey +and sdrName (see ionconfig(5)) are used in common by all executables to +assure that all elements of the system operate within the same state +space. For some purposes, however, it may be desirable to establish +multiple ION nodes on a single workstation. (For example, constructing +an entire self-contained DTN network on a single machine may simplify +some kinds of regression testing.) ION supports this configuration +option as follows:
+Multi-node operation on a given computer is enabled if and only if + the environment variable ION_NODE_LIST_DIR is defined in the + environment of every participating ION process. Moreover, the value + assigned to this variable must be the same text string in the + environments of all participating ION processes. That value must be + the name (preferably, fully qualified) of the directory in which the + ION multi-node database file "ion_nodes" will reside.
+The definition of ION_NODE_LIST_DIR makes it possible to establish + up to one ION node per directory rather than just one ION node on + the computer. When ionadmin is used to establish a node, the + ionInitialize() function will get that node's wmKey and sdrName from + the .ionconfig file, use them to allocate working memory and create + the SDR database, and then write a line to the ion_nodes file noting + the nodeNbr, wmKey, sdrName, and wdName for the node it just + initialized. wdName is the current working directory in which + ionadmin was running at the time ionInitialize()is called; it is + the directory within which the node resides.
+This makes it easy to connect all the node\'s daemon processes -- + running within the same current working directory -- to the correct + working memory partition and SDR database: the ionAttach() function + simply searches the ion_nodes file for a line whose wdName matches + the current working directory of the process that is trying to + attach, then uses that line\'s wmKey and sdrName to link up.
+It is also possible to initiate a process from within a directory + other than the one in which the node resides. To do so, define the + additional environment variable ION_NODE_WDNAME in the shell from + which the new process is to be initiated. When ionAttach() is called + it will first try to get \"current working directory\" (for ION + attachment purposes only) from that environment variable; only + if ION_NODE_WDNAME is undefined will it use the actual cwd that it + gets from calling igetcwd().
+The executable programs used in operation of the ici component of ION +include:
+The ionadmin system configuration utility and ionsecadmin + security configuration utility, invoked at node startup time and as + needed thereafter.
+The rfxclock background daemon, which affects scheduled network + configuration events.
+The sdrmend system repair utility, invoked as needed.
+The sdrwatch and psmwatch utilities for resource utilization + monitoring, invoked as needed.
+Each time it is executed, ionadmin computes a new congestion +forecast and, if a congestion collapse is predicted, invokes the node's +congestion alarm script (if any). ionadmin also establishes the node +number for the local node and starts/stops the rfxclock task, among +other functions. For further details, see the man pages for ionadmin(1), +ionsecadmin(1), rfxclock(1), sdrmend(1), sdrwatch(1), and psmwatch(1) in +Appendix A.
+Six test executables are provided to support testing and debugging of +the ICI component of ION:
+The file2sdr and sdr2file programs exercise the SDR system.
+The psmshell program exercises the PSM system.
+The file2sm, sm2file, and smlistsh programs exercise the + shared-memory linked list system.
+For details, see the man pages for file2sdr(1), sdr2file(1), +psmshell(1), file2sm(1), sm2file(1), and smlistsh(1) in Appendix A.
+To build LTP:
+Make sure that the "ici" component of ION has been built for the + platform on which you plan to run LTP.
+Edit the Makefile in ion/ltp:
+As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run LTP.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local).
+Then:
+The LTP administration command (ltprc) file provides the information +needed to configure LTP on a given ION node. For details, see the man +page for ltprc(5) in Appendix A.
+The executable programs used in operation of the ltp component of ION +include:
+The ltpadmin protocol configuration utility, invoked at node + startup time and as needed thereafter.
+The ltpclock background daemon, which affects scheduled LTP + events such as segment retransmissions.
+The ltpmeter block management daemon, which segments blocks and + effects LTP flow control.
+The udplsi and udplso link service input and output tasks, + which handle transmission of LTP segments encapsulated in UDP + datagrams (mainly for testing purposes).
+ltpadmin starts/stops the ltpclock and ltpmeter tasks and, as +mandated by configuration, the udplsi and udplso tasks.
+For details, see the man pages for ltpadmin(1), ltpclock(1), +ltpmeter(1), udplsi(1), and udplso(1) in Appendix A.
+Two test executables are provided to support testing and debugging of +the LTP component of ION:
+ltpdriver is a continuous source of LTP segments.
+ltpcounter is an LTP block receiver that counts blocks as they + arrive.
+For details, see the man pages for ltpdriver(1) and ltpcounter(1) in +Appendix A.
+To build BSSP:
+Make sure that the "ici" component of ION has been built for the + platform on which you plan to run BSSP.
+Edit the Makefile in ion/bssp:
+As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run BSSP.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local).
+Then: +
+The BSSP administration command (bssprc) file provides the +information needed to configure BSSP on a given ION node. For details, +see the man page for bssprc(5) in Appendix A.
+The bssprc file has a command option specifying the max_block_size. This +is to prevent retransmission inefficiency when the blocks size of a +stream data is too large. The unit of retransmission for BSSP is the +block, so if the block size is too large, it is very expensive to the +network to provide retransmission. If one needs bulk data transfer, +instead of streaming, one should use BP with reliability LTP instead of +using BSSP. If you are using udpbso and udpbsi as the underlying +convergence layer, then the max_block_size parameter for bssprc cannot +be larger than 65507 bytes, because each UDP datagram can only be as +large as 65507 bytes (payload) + 20 (IP Header) + 8 (UDP Header) = 65535 +byte.
+The executable programs used in operation of the bssp component of ION +include:
+The bsspadmin protocol configuration utility, invoked at node + startup time and as needed thereafter.
+The bsspclock background daemon, which affects scheduled BSSP + events such as segment retransmissions.
+The udpbsi and udpbso link service input and output tasks, + which handle transmission of BSSP segments encapsulated in UDP + datagrams (mainly for testing purposes; in space domain, the + appropriate CCSDS link layer will be used instead of UDP).
+bsspadmin starts/stops the bsspclock task and, as mandated by +configuration, the udpbsi and udblso tasks.
+For details, see the man pages for bsspadmin(1), bsspclock(1), +bsspmeter(1), udpbsi(1), and udpbso(1) in Appendix A.
+Declaring values for the following variables, by setting parameters that +are provided to the C compiler (for example, --DION_NOSTATS or +--DBRSTERM=60), will alter the functionality of BP as noted below.
+[TargetFFS]{.underline}
+Setting this option adapts BP for use with the TargetFFS flash file +system on the VxWorks operating system. TargetFFS apparently locks one +or more system semaphores so long as a file is kept open. When a BP task +keeps a file open for a sustained interval, subsequent file system +access may cause a high-priority non-BP task to attempt to lock the +affected semaphore and therefore block; in this event, the priority of +the BP task may automatically be elevated by the inversion safety +mechanisms of VxWorks. This "priority inheritance" can result in +preferential scheduling for the BP task -- which does not need it -- at +the expense of normally higher-priority tasks, and can thereby introduce +runtime anomalies. BP tasks should therefore close files immediately +after each access when running on a VxWorks platform that uses the +TargetFFS flash file system. The TargetFFS compile-time option ensures +that they do so.
+[BRSTERM=xx]{.underline}
+This option sets the maximum number of seconds by which the current time +at the BRS server may exceed the time tag in a BRS authentication +message from a client; if this interval is exceeded, the authentication +message is presumed to be a replay attack and is rejected. Small values +of BRSTERM are safer than large ones, but they require that clocks be +more closely synchronized. The default value is 5.
+[ION_NOSTATS]{.underline}
+Setting this option prevents the logging of bundle processing statistics +in status messages.
+[KEEPALIVE_PERIOD=xx]{.underline}
+This option sets the number of seconds between transmission of +keep-alive messages over any TCP or BRS convergence-layer protocol +connection. The default value is 15.
+[ION_BANDWIDTH_RESERVED]{.underline}
+Setting this option overrides strict priority order in bundle +transmission, which is the default. Instead, bandwidth is shared between +the priority-1 and priority-0 queues on a 2:1 ratio whenever there is no +priority-2 traffic.
+[ENABLE_BPACS]{.underline}
+This option causes Aggregate Custody Signaling source code to be +included in the build. ACS is alternative custody transfer signaling +mechanism that sharply reduces the volume of custody acknowledgment +traffic.
+[ENABLE_IMC]{.underline}
+This option causes IPN Multicast source code to be included in the +build. IMC is discussed in section 1.8.4 above.
+To build BP:
+Make sure that the "ici", "ltp", "dgr", and "bssp" components of ION + have been built for the platform on which you plan to run BP.
+Edit the Makefile in ion/bp:
+As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run BP.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local).
+Then: +
+The BP administration command (bprc) file provides the information +needed to configure generic BP on a given ION node. The IPN scheme +administration command (ipnrc) file provides information that +configures static and default routes for endpoints whose IDs conform to +the "ipn" scheme. The DTN scheme administration command (dtn2rc) +file provides information that configures static and default routes for +endpoints whose IDs conform to the "dtn" scheme, as supported by the +DTN2 reference implementation. For details, see the man pages for +bprc(5), ipnrc(5), and dtn2rc(5) in Appendix A.
+The executable programs used in operation of the bp component of ION +include:
+The bpadmin, ipnadmin, and dtn2admin protocol configuration + utilities, invoked at node startup time and as needed thereafter.
+The bpclock background daemon, which effects scheduled BP events + such as TTL expirations and which also implements rate control.
+The ipnfw and dtn2fw forwarding daemons, which compute + routes for bundles addressed to "ipn"-scheme and "dtn"-scheme + endpoints, respectively.
+The ipnadminep and dtn2adminep administrative endpoint + daemons, which handle custody acceptances, custody refusals, and + status messages.
+The bpclm background daemon, which selects convergence-layer + outducts by which bundles are transmitted to neighboring nodes.
+The brsscla (server) and brsccla (client) Bundle Relay + Service convergence-layer adapters.
+The tcpcli (input) TCP convergence-layer adapter, which includes + convergence-layer output functionality in privately managed threads.
+The stcpcli (input) and stcpclo (output) simplified TCP + convergence-layer adapters.
+The udpcli (input) and udpclo (output) UDP convergence-layer + adapters.
+The ltpcli (input) and ltpclo (output) LTP convergence-layer + adapters.
+The dgrcla Datagram Retransmission convergence-layer adapter.
+The bpsendfile utility, which sends a file of arbitrary size, + encapsulated in a single bundle, to a specified BP endpoint.
+The bpstats utility, which prints a snapshot of currently + accumulated BP processing statistics on the local node.
+The bptrace utility, which sends a bundle through the network to + enable a forwarding trace based on bundle status reports.
+The lgsend and lgagent utilities, which are used for remote + administration of ION nodes.
+The hmackeys utility, which can be used to create hash keys + suitable for use in bundle authentication blocks and BRS + convergence-layer protocol connections.
+bpadmin starts/stops the bpclock task and, as mandated by +configuration, the ipnfw, dtn2fw, ipnadminep, +dtn2adminep, bpclm, brsscla, brsccla, tcpcli, stcpcli, +stcpclo, udpcli, udpclo, ltpcli, ltpclo, and +dgrcla tasks.
+For details, see the man pages for bpadmin(1),ipnadmin(1), dtn2admin(1), +bpclock(1), bpclm(1), ipnfw(1), dtn2fw(1), ipnadminep(1), +dtn2adminep(1), brsscla(1), brsccla(1),tcpcli(1), stcpcli(1), +stcpclo(1), udpcli(1), udpclo(1), ltpcli(1), ltpclo(1), dgrcla(1), +bpsendfile(1), bpstats(1), bptrace(1), lgsend(1), lgagent(1), and +hmackeys(1) in Appendix A.
+Five test executables are provided to support testing and debugging of +the BP component of ION:
+bpdriver is a continuous source of bundles.
+bpcounter is a bundle receiver that counts bundles as they + arrive.
+bpecho is a bundle receiver that sends an "echo" acknowledgment + bundle back to bpdriver upon reception of each bundle.
+bpsource is a simple console-like application for interactively + sending text strings in bundles to a specified DTN endpoint, + nominally a bpsink task.
+bpsink is a simple console-like application for receiving + bundles and printing their contents.
+For details, see the man pages for bpdriver(1), bpcounter(1), bpecho(1), +bpsource(1), and bpsink(1) in Appendix A.
+To build DGR:
+Make sure that the "ici" component of ION has been built for the + platform on which you plan to run DGR.
+Edit the Makefile in ion/dgr:
+As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run DGR.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local).
+Then:
+No additional configuration files are required for the operation of the +DGR component of ION.
+No runtime executables are required for the operation of the DGR +component of ION.
+Two test executables are provided to support testing and debugging of +the DGR component of ION:
+For details, see the man pages for file2dgr(1) and dgr2file(1) in +Appendix A.
+Note that, by default, the syntax by which AMS MIB information is +presented to AMS is as documented in the "amsrc" man page. Alternatively +it is possible to use an XML-based syntax as documented in the "amsxml" +man page. To use the XML-based syntax instead, be sure that the "expat" +XML interpretation system is installed and pass the argument +"--with-expat" to "./configure" when building ION.
+Defining the following macros, by setting parameters that are provided +to the C compiler (for example, DAMS_INDUSTRIAL), will alter the +functionality of AMS as noted below.
+AMS_INDUSTRIAL
Setting this option adapts AMS to an "industrial" rather than +safety-critical model for memory management. By default, the memory +acquired for message transmission and reception buffers in AMS is +allocated from limited ION working memory, which is fixed at ION +start-up time; this limits the rate at which AMS messages may be +originated and acquired. When --DAMS_INDUSTRIAL is set at compile time, +the memory acquired for message transmission and reception buffers in +AMS is allocated from system memory, using the familiar malloc() and +free() functions; this enables much higher message traffic rates on +machines with abundant system memory.
+To build AMS:
+Make sure that the "bp" component of ION has been built for the + platform on which you plan to run AMS.
+Edit the Makefile in ion/cfdp:
+Just as for bp, make sure PLATFORMS is set to the name of the + platform on which you plan to run AMS.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local).
+Then:
+There is no central configuration of AMS; each AMS entity (configuration +server, registrar, or application module) is individually configured at +the time its initial MIB is loaded at startup. Note that a single MIB +may be shared between multiple AMS entities without issue.
+For details of MIB file syntax, see the man pages for amsrc(5) and +amsxml(5) in Appendix A.
+The executable programs used in operation of the AMS component of ION +include:
+The amsd background daemon, which serves as configuration server + and/or as the registrar for a single application cell.
+The ramsgate application module, which serves as the Remote AMS + gateway for a single message space.
+The amsstop utility, which terminates all AMS operation + throughout a single message space.
+The amsmib utility, which announces supplementary MIB + information to selected subsets of AMS entities without interrupting + the operation of the message space.
+For details, see the man pages for amsd(1), ramsgate(1), amsstop(1), and +amsmib(1) in Appendix A.
+Seven test executables are provided to support testing and debugging of +the AMS component of ION:
+amsbenchs is a continuous source of messages.
+amsbenchr is a message receiver that calculates bundle + transmission performance statistics.
+amshello is an extremely simple AMS "hello, world" demo program + -- a self-contained distributed application in a single source file + of about seventy lines.
+amsshell is a simple console-like application for interactively + publishing, sending, and announcing text strings in messages.
+amslog is a simple console-like application for receiving + messages and piping their contents to stdout.
+amslogprt is a pipeline program that simply prints AMS message + contents piped to it from amslog.
+amspubsub is a pair of functions for rudimentary testing of AMS + functionality in a VxWorks environment.
+For details, see the man pages for amsbenchs(1), amsbenchr(1), +amshello(1), amsshell(1), amslog(1), amslogprt(1), amspub(1), and +amssub(1) in Appendix A.
+For further operational details of the AMS system, please see sections 4 +and 5 of the [AMS Programmer's Guide]{.underline}.
+Defining the following macro, by setting a parameter that is provided to +the C compiler (i.e., --DTargetFFS), will alter the functionality of +CFDP as noted below.
+[TargetFFS]{.underline}
+Setting this option adapts CFDP for use with the TargetFFS flash file +system on the VxWorks operating system. TargetFFS apparently locks one +or more system semaphores so long as a file is kept open. When a CFDP +task keeps a file open for a sustained interval, subsequent file system +access may cause a high-priority non-CFDP task to attempt to lock the +affected semaphore and therefore block; in this event, the priority of +the CFDP task may automatically be elevated by the inversion safety +mechanisms of VxWorks. This "priority inheritance" can result in +preferential scheduling for the CFDP task -- which does not need it -- +at the expense of normally higher-priority tasks, and can thereby +introduce runtime anomalies. CFDP tasks should therefore close files +immediately after each access when running on a VxWorks platform that +uses the TargetFFS flash file system. The TargetFFS compile-time option +ensures that they do so.
+To build CFDP:
+Make sure that the "bp" component of ION has been built for the + platform on which you plan to run CFDP.
+Edit the Makefile in ion/cfdp:
+Just as for bp, make sure PLATFORMS is set to the name of the + platform on which you plan to run CFDP.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed.
+Then:
+The CFDP administration command (cfdprc) file provides the +information needed to configure CFDP on a given ION node. For details, +see the man page for cfdprc(5) in Appendix A.
+The executable programs used in operation of the CFDP component of ION +include:
+The cfdpadmin protocol configuration utility, invoked at node + startup time and as needed thereafter.
+The cfdpclock background daemon, which affects scheduled CFDP + events such as check timer expirations. The cfdpclock task also + affects CFDP transaction cancellations, by canceling the bundles + encapsulating the transaction's protocol data units.
+The bputa UT-layer input/output task, which handles transmission + of CFDP PDUs encapsulated in bundles.
+cfdpadmin starts/stops the cfdpclock task and, as mandated by +configuration, the bputa task.
+For details, see the man pages for cfdpadmin(1), cfdpclock(1), and +bputa(1) in Appendix A.
+A single executable, cfdptest, is provided to support testing and +debugging of the DGR component of ION. For details, see the man page for +cfdptest(1) in Appendix A.
+Defining the following macro, by setting a parameter that is provided to +the C compiler (e.g., --DWINDOW=10000), will alter the functionality of +BSS as noted below.
+[WINDOW=xx]{.underline}
+Setting this option changes the maximum number of seconds by which the +BSS database for a BSS application may be "rewound" for replay. The +default value is 86400 seconds, which is 24 hours.
+To build BSS:
+Make sure that the "bp" component of ION has been built for the + platform on which you plan to run BSS.
+Edit the Makefile in ion/bss:
+As for ici, make sure PLATFORMS is set to the name of the platform + on which you plan to run BSS.
+Set OPT to the directory containing the bin, lib, include, etc. + directories where the ici package is installed (for example: + /usr/local).
+Then:
+No additional configuration files are required for the operation of the +BSS component of ION.
+No runtime executables are required for the operation of the BSS +component of ION.
+Four test executables are provided to support testing and debugging of +the BSS component of ION:
+bssdriver sends a stream of data to bsscounter for + non-interactive testing.
+bssStreamingApp sends a stream of data to bssrecv for + graphical, interactive testing.
+For details, see the man pages for bssdriver(1), bsscounter(1), +bssStreamingApp(1), and bssrecv(1) in Appendix A.
+In ION, reliable convergence-layer protocols (where available) are +by default used for every bundle. The application can instead +mandate selection of "best-effort" service at the convergence layer +by setting the BP_BEST_EFFORT flag in the "extended class of service +flags" parameter, but this feature is an ION extension that is not +supported by other BP implementations at the time of this writing. ↩
+Note that, in all occupancy figures, ION data management accounts +not only for the sizes of the payloads of all queued bundles but +also for the sizes of their headers. ↩
+Barring data loss or corruption for which the various +retransmission mechanisms in ION cannot compensate. ↩
+Note that ION may indeed block the offering of a message to the +network, but this is local admission control -- assuring that the +node's local buffer space for queuing outbound bundles is not +oversubscribed -- rather than end-to-end flow control. It is always +possible for there to be ample local buffer space yet insufficient +network capacity to convey the offered data to their final +destination, and vice versa. ↩
+The minimum size of an ION bundle header is 26 bytes. Adding +extension blocks (such as those that effect the Bundle Security +Protocol) will increase this figure. ↩
+In ION, all bundles are by default non-critical. The application +can indicate that data should be sent in a Critical bundle by +setting the BP_MINIMUM_LATENCY flag in the "extended class of +service" parameter, but this feature is an ION extension that is not +supported by other BP implementations at the time of this writing. ↩
+If we wanted to be extremely accurate we could also +[subtract]{.underline} from the timeout interval the imputed inbound +queuing delay QI, since inbound queuing would presumably be +completed during the interval in which transmission was suspended. +But since we're guessing at the queuing delays anyway, this +adjustment doesn't make a lot of sense. ↩
+To build and install the entire ION system on a Linux, MacOS, or Solaris platform, cd into ion-open-source and enter the following commands:
+./configure
If configure is not present run: autoreconf -fi first
make
sudo make install
sudo ldconfig
For MacOS, the ldconfig command is not present and not necessary to run.
If you want to set overriding compile-time switches for a build, the place to do this is in the ./configure command. For details,
./configure -h
By default, Bundle Protocol V7 will be built and installed, but BPv6 source code is still available. The BPv6 implementation is essentially the same as that of ION 3.7.4, with only critical bugs being updated going forward. All users are encouraged to switch to BPV7.
+To build BPv6, run
+./configure --enable-bpv6
To clean up compilation artifacts such as object files and shared libraries stored within the ION open-source directory, cd to the ION open-source directory and run:
+make clean
To remove executables and shared libraries installed in the system, run:
+sudo make uninstall
To install ION for Windows, please download the Windows installer.
+It's also possible to build the individual packages of ION, using platform-specific Makefiles in the package subdirectories. Currently the only actively maintained platform-specific Makefile is for 64-bits Linux under the "i86_48-fedora" folder. If you choose this option, be aware of the dependencies among the packages:
+make and make install) before any other package.For more detailed instruction on building ION, see section 2 of the "ION Design and Operation Guide" document that is distributed with this package.
+Also, be aware that these Makefiles install everything into subdirectories of /usr/local. To override this behavior, change the value of OPT in the top-level Makefile of each package.
Additional details are provided in the README.txt files in the root directories of some of the subsystems.
+Note that all Makefiles are for gmake; on a FreeBSD platform, be sure to install gmake before trying to build ION.
+Before running ION, let's confirm which version of Bundle Protocol is installed by running:
+bpversion
You will see a simple string on the terminal windows indicating either "bpv6" or "bpv7".
+Also check the ION version installed by running:
+ionadmin
At the ":" prompt, please enter the single character command 'v' and you should see a response like this:
+ +Then type 'q' to quit ionadmin. While ionadmin quits, it may display certain error messages like this:
+at line 427 of ici/library/platform_sm.c, Can't get shared memory segment: Invalid argument (0)
+at line 312 of ici/library/memmgr.c, Can't open memory region.
+at line 367 of ici/sdr/sdrxn.c, Can't open SDR working memory.
+at line 513 of ici/sdr/sdrxn.c, Can't open SDR working memory.
+at line 963 of ici/library/ion.c, Can't initialize the SDR system.
+Stopping ionadmin.
+This is normal due to the fact that ION has not launched yet.
+The tests directory contains regression tests used by system integrator to check ION before issuing each new release. To make sure ION is operating properly after installation, you can also manually run the bping test:
First enter the test directory: cd tests
Enter the command: ./runtests bping/
This command invokes one of the simplest test whereby two ION instances are created and a ping message is sent from one to the other and an echo is returned to the sender of the ping.
+During test, ION will display the configuration files used, clean the system of existing ION instances, relaunch ION according to the test configuration files, execute bping actions, display texts that indicates what the actions are being executed in real-time, and then shutdown ION, and display the final test status message, which looks like this:
+ION node ended. Log file: ion.log
+TEST PASSED!
+
+passed: 1
+ bping
+
+failed: 0
+
+skipped: 0
+
+excluded by OS type: 0
+
+excluded by BP version: 0
+
+obsolete tests: 0
+In this case, the test script confirms that ION is able to execute a bping function properly.
+Under the demos folder of the ION code directory, there are benchmark tests for various ION configurations. These tests also provide a template of how to configure ION.
Take the example of the bench-udp demo:
Go into the demos/bench-udp/ folder, you will see two subfolders: 2.bench.udp and 3.bench.udp, these folders configures two ION nodes, one with node numbers 2 and 3.
Looking inside the 2.bench.udp folder, you will see specific files used to configure ION. These include:
bench.bprc is the configuration file for the bundle protocol. To study the command options contained in this file, run man bprc.bench.ionconfig is the configuration file for the storage configuration of ION. See man ionconfig for details.bench.ionrc is the configuration file for ION. See man ionrc for details.bench.ionsecrc is the configuration file for ION security administration. See man ionsecrc for details.bench.ipnrc is the configuration file for the IPN scheme. See man ipnrc for details.ionstart and ionstop are scripts to launch and shutdown ION. One must note that ION distribution comes with a separate, global ionstart and ionstop scripts installed in /usr/local/bin that can launch and stop ION. The advantage of using local script is that it allows you customize the way you launch and stop ION, for example add helpful text prompt, perform additional checks and clean up activities, etc.
To run this demo test, first go into the test directory bench-udp, then run the dotest script:
+./dotest
You can also study the test script to understand better what is happening.
+If you study the test script under the "tests" and the "demos" folders, you will realize that these tests often will launch 2 or 3 ION nodes on the same host to conduct the necessary tests. While this is necessary to simplify and better automate regression testing for ION developer and integration, it is not a typical, recommended configuration for new users.
+In order to run multiple ION instances in one host, specific, different IPCS keys must be used for each instance, and several variables must be set properly in the shell environment. Please see the ION Deployment Guide (included with the ION distribution) for more information on how to do that.
+We recommend that most users, unless due to specific contrain that they must run multiple ION instance on one host, to run each ION instance on a separate host or (VM).
+Once you have studied these scripts, you can try to run it on two different machines running ION.
+First, install ION in host A with an IP address of, for example, 192.168.0.2, and host B with an IP address of 192.168.0.3. Verify your installation based on earlier instructions.
+Copy the 2.bench.udp folder into host A and the 3.bench.udp folder into host B.
Also copy the file global.ionrc from the bench.udp folder into the same folder where you placed 2.bench.udp and 3.bench.udp
Then you need to modify the IP addresses in the UDP demo configuration files to match the IP addresses of hosts A and B.
+For example, the bprc files copied into host A is:
+1
+a scheme ipn 'ipnfw' 'ipnadminep'
+a endpoint ipn:2.0 x
+a endpoint ipn:2.1 x
+a endpoint ipn:2.2 x
+a endpoint ipn:2.64 x
+a endpoint ipn:2.65 x
+a protocol udp 1400 100
+a induct udp 127.0.0.1:2113 udpcli
+a outduct udp 127.0.0.1:3113 'udpclo 1'
+r 'ipnadmin bench.ipnrc'
+s
+To make it work for host A, you need to replace the induct ip address 127.0.0.1:2113 to 192.168.0.2:2113 - this is where host A's ION will receive incoming UDP traffic.
Similarly for outduct, you want to change the ip address from 127.0.0.1:3113 to 192.168.0.3:3113 - this is where UDP traffic will go out to host B.
You can make similar modifications to the ipnrc file as well.
+In the ionconfig file, you want to comment out or delete the wmKey entry. Since we are running these two nodes on different hosts, we recommend not specifying any IPC key values but let ION use the default value.
Repeat the same updates for host B by appropriately substituting old IP address to that of the new hosts.
+After updating the configuration files on host A and B to reflect the new IP addresses and using default wmKey (by not specifying any), we are new ready to try launching ION.
+Before you try to launch ION, it is recommended that you:
+global.ionrc file, change the data rates for the a contact command down to something similar to your connection speed. Remember, the unit in the global.ionrc file is Bytes per second, not bits per second, which is typically what iperf test report uses. Once you are ready to launch ION on both host A and B, open a terminal and go to the directory where the configuration files are stored, and run the local ionstart script:
+./ionstart
Note: do not run ionstart since that will trigger the global script in the execution PATH
You should see some standard output confirming that ION launch has completed. For example you might see something like this:
+Starting ION...
+wmSize: 5000000
+wmAddress: 0
+sdrName: 'ion2'
+sdrWmSize: 0
+configFlags: 1
+heapWords: 100000000
+heapKey: -1
+logSize: 0
+logKey: -1
+pathName: '/tmp'
+Stopping ionadmin.
+Stopping ionadmin.
+Stopping ionsecadmin.
+Stopping ltpadmin.
+Stopping ipnadmin.
+Stopping bpadmin.
+You can also see additional status information in the ion.log file in the same directory.
Launch ION on both host A and B.
+Now that we have launched ION on both host A and B, it's time to send some data.
+We can repeat the bping test at this point. But since you have already seen that before, let's try something different.
+Let's use the bpdriver-bpcounter test utilities. This pair of utility programs simply sends a number of data in bundles from one node to another and provides a measurement on the throughput.
+On host B, run this command:
+bpcounter ipn:3.2 3
This command tells ION node number 3 to be ready to receive three bundles on the end-point ID ipn:3.2 which was specified in the .bprc file.
After host B has launched bpcounter, then on host A, run this command:
+bpdriver 3 ipn:2.2 ipn:3.2 -10000
This command tells ION running in host A to send 3 bundles from EID 2.2 to EID 3.2, which is waiting for data (per bpcounter command.) And each bundle should be 10,000 bytes in size.
+Why use the "-" sign in front of the size parameter? It's not a typo. The "-" indicates that bpdriver should keep sending bundles without waiting for any response from the receiver. The feature where bpdriver waits for the receiver is available in BPv6 but no longer part of BPv7.
+When the test completed, you should see output indicating that all the data were sent, how many bundles were transmitted/received, and at what rate.
+Please note that on the sending side the transmission may appear to be almost instantaneous. That is because bpdriver, as an application, is pushing data into bundle protocol which has the ability to rate buffer the data. So as soon as the bpdriver application pushes all data into the local bundle protocol agent, it considers the transmission completed and it will report a very high throughput value, one that is far above the contact graph's data rate limit. This is not an error; it simple report the throughput as experienced by the sending application, knowing that the data has not yet delivered fully to the destination.
+Throughput reported by bpcounter, on the other hand, is quite accurate if a large number of bundles are sent. To accurately measure the time it takes to send the bundles, bpdriver program will send a "pilot" bundle just before sending the test data to signals to the bpcounter program to run its throughput calculation timer. This allows the user to run bpcounter and not haveing to worry about immediately send all the bundles in order to produce an accurate throughput measurement.
+If you want to emulate the action of a constant rate source, instead of having bpdriver pushing all data as fast as possible, then you can use the 'i' option to specify a data rate throttle in bits per second.
+If you want to know more about how bpdriver and bpcounter work, look up their man pages for details on syntax and command line options. Other useful ION test utility commands include bpecho, bping, bpsource, bpsink, bpsendfile, bprecvfile, etc.
To confirm whether ION is running properly or has experienced an error, the first thing to do is to check the ion.log, which is a file created in the directory from which ION was launched. If an ion.log file exists when ION starts, it will simply append additional log entries into that file. Each entry has a timestamp to help you determine the time and the relative order in which events occurred.
+When serious error occurs, ion.log will have detailed messages that can pinpoint the name and line number of the source code where the error was reported or triggered.
+Sometimes after operating ION for a while, you will notice a number of files with names such as "bpacq" or "ltpacq" followed by a number. These are temporary files created by ION to stage bundles or LTP blocks during reception and processing. Once a bundle or LTP block is completely constructed, delivered, or cancelled properly, these temporary files are automatically removed by ION. But if ION experiences an anomalous shutdown, then these files may remain and accumulate in the local directory.
+It is generally safe to remove these files between ION runs. Their presence does not automatically imply issues with ION but can indicate that ION operations were interrupted for some reason. By noting their creation time stamp, it can provide clues on when these interruptions occurred. Right now there are no ION utilty program to parse them because these files are essentially bit buckets and do not contain internal markers or structure and allows user to parse them or extract information by processes outside the bundle agents that created them in the first place.
+Sometimes shutting down ION does not go smoothly and you can't seem to relaunch ION properly. In that case, you can use the global ionstop script (or the killm script) to kill all ION processes that did not terminate using local ionstop script. The global ionstop or killm scripts also clears out the IPC shared memory and semaphores allocations that were locked by ION processes and would not terminate otherwise.
To learn about the configuration files and the basic set of command syntax and functions: +ION Config File Tutorial
+To learn more about the design principle of ION and how to use it, a complete series of tutorials is available here: +NASA ION Course
+If you use the ION Dev Kit mentioned in the NASA ION Course, you can find some additional helpful files here: +Additional DevKit Files
+Use the Summary or the Files tab to download point releases
+Track the "stable" branch to match the ION releases
+Track the "current" branch for bug fixes and small updates between releases
+If you plan to contribute to the ION project, please keep these in mind:
+Submitted code should adhere to the ION coding style found in the current code. We plan to add a more formal coding style guide in the future.
+Provide documentation describing the contributed code’s features, its inputs and outputs, dependencies, behavior (provide a high-level state machine or flowchart if possible), and API description. Please provide a draft of a man page.
+Provide canned tests (ION configuration and script) that can be executed to verify and demonstrate the proper functioning of the features. Ideally it should demonstrate nominal operation and off-nominal scenarios.
+The NASA team will review these contributions and determine to either
+incorporate the code into the baseline, or
+not incorporate the code into the baseline but make it available in the /contrib folder (if possible) as experimental modules, or
+not incorporate it at all.
+All baselined features will be supported with at least bug-fixes until removed
+All /contrib folder features are provided ”as is,” and no commitment is made regarding bug-fixes.
+The contributor is expected to help with regression testing.
+Due to resource constraints, we cannot make any commitment as to response time. We will do our best to review them on a best effort basis.
+Fork this repository
+Starting with the "current" branch, create a named feature or bugfix branch and develop/test your code in this branch
+Generate a pull request (called Merge Request on Source Forge) with
+Your feature or bugfix branch as the Source branch
+"current" as the destination branch
+[[project_admins]]
+ + + + + + + + + + + + + +ION Version: 4.1.3
+Bundle Protocol Version:7
+Watch characters, when activated, provide immediate feedback on ION operations by printing various characters standard output (terminal). By examing the watch characters, and the order in which they appear, operators can quickly confirm proper operation or detect configuration or run-time errors.
+This document will list all watch characters currently supported by ION.
+Enhanced watch characters were added ION 4.1.3 to provide detailed state information at Bundle Protocol (BP) and LTP levels and can be activated at compile time by:
+ +Enhanced watch characters prepends additional state information to the standard watch characters inside a pair of parenthesis. In this document, we use the following notion regarding enhanced watch characters information.
+nnn = source node numbersss = service numberttt = bundle creation time in milliseconds Epoch(2000)ccc = bundle sequence numberxxx (LTP session number)Each field can be longer or shorter than 3 digits/characters.
+Besides real-time monitoring of the watch characters on standard out, ION can redirect the watch characters to customized user applications for network monitoring purposes.Prior to and including ION 4.1.2, watch characters are single character. Starting from ION 4.1.3 release, a watch character is now generalized to a string of type char*.
To activate customized processing, there are two steps:
+gdswatcher.c that defines a functions to process watch characters, and pass that function to ION to handle watch character:static void processWatchChar(char* token)
+{
+ //your code goes here
+}
+
+static void ionRedirectWatchCharacters()
+{
+ setWatcher(processWatchChar);
+}
+a - new bundle is queued for forwarding; (nnn,sss,tttt,cccc)a
b - bundle is queued for transmission; (nnn,sss,ccc)b
c - bundle is popped from its transmission queue; (nnn,sss,ccc)c
m - custody acceptance signal is received
w - custody of bundle is accepted
x - custody of bundle is refused
y - bundle is accepted upon arrival; (nnn,sss,ccc)y
z - bundle is queued for delivery to an application; (nnn,sss,ccc)z
~ - bundle is abandoned (discarded) on attempt to forward it
! - bundle is destroyed due to TTL expiration; (nnn,sss,ccc)!
& - custody refusal signal is received
# - bundle is queued for re-forwarding due to CL protocol failure
j - bundle is placed in \"limbo\" for possible future re-forwarding
k - bundle is removed from \"limbo\" and queued for re-forwarding
d - bundle appended to block for next session
e - segment of block is queued for transmission
f - block has been fully segmented for transmission; (xxxx)f
g - segment popped from transmission queue;
(cpxxx)g -- checkpoint, this could be a data segment or a standalone
+ check point
(dsxxx)g -- non-check point data segment
(rcpxxx)g -- retransmitted checkpoint
(prsxxx)g -- positive report (all segments received)
(nrsxxx)g -- negative report (gaps)
(rrsxxx)g -- retransmitted report (either positive or negative)
(rasxxx)g -- a report ack segment
(csxxx)g -- cancellation by block source
(crxxx)g -- cancellation by block receiver
(caxxx)g -- cancellation ack for either CS or CR
h - positive ACK received for block, session ended; (xxx)h
s - segment received
t - block has been fully received
@ - negative ACK received for block, segments retransmitted; (xxx)@
= - unacknowledged checkpoint was retransmitted; (xxx)=
+ - unacknowledged report segment was retransmitted; (xxx)+
{ - export session canceled locally (by sender)
} - import session canceled by remote sender
[ - import session canceled locally (by receiver)
] - export session canceled by remote receiver
w - custody request is accepted (by receiving entity)
m - custody acceptance signal is received (by requester)
x - custody of bundle has been refused
& - custody refusal signal is received (by requester)
$ - bundle retransmitted due to expiration of custody request timer
D - bssp send completed
E - bssp block constructed for issuance
F - bssp block issued
G - bssp block popped from best-efforts transmission queue
H - positive ACK received for bssp block, session ended
S - bssp block received
T - bssp block popped from reliable transmission queue
- - unacknowledged best-efforts block requeued for reliable transmission
* - session canceled locally by sender
NO WARRANTY:
+DISCLAIMER
+THE SOFTWARE AND/OR RELATED MATERIALS ARE PROVIDED "AS-IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ANY WARRANTIES OF PERFORMANCE OR MERCHANTABILITY OR FITNESS FOR A PARTICULAR USE OR PURPOSE (AS SET FORTH IN UCC 2312-2313) OR FOR ANY PURPOSE WHATSOEVER, FOR THE LICENSED PRODUCT, HOWEVER USED.
+IN NO EVENT SHALL CALTECH/JPL BE LIABLE FOR ANY DAMAGES AND/OR COSTS, INCLUDING BUT NOT LIMITED TO INCIDENTAL OR CONSEQUENTIAL DAMAGES OF ANY KIND, INCLUDING ECONOMIC DAMAGE OR INJURY TO PROPERTY AND LOST PROFITS, REGARDLESS OF WHETHER CALTECH/JPL SHALL BE ADVISED, HAVE REASON TO KNOW, OR IN FACT SHALL KNOW OF THE POSSIBILITY.
+USER BEARS ALL RISK RELATING TO QUALITY AND PERFORMANCE OF THE SOFTWARE AND/OR RELATED MATERIALS.
+Copyright 2002-2013, by the California Institute of Technology. ALL RIGHTS RESERVED. U.S. Government Sponsorship acknowledged.
+This software and/or related materials may be subject to U.S. export control laws. By accepting this software and related materials, the user agrees to comply with all applicable U.S. export laws and regulations. User has the responsibility to obtain export licenses or other export authority as may be required before exporting the +software or related materials to foreign countries or providing access to foreign persons.
+The QCBOR code included is distributed with the following condition
+Copyright (c) 2016-2018, The Linux Foundation. +Copyright (c) 2018-2019, Laurence Lundblade. +All rights reserved.
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + * Neither the name of The Linux Foundation nor the names of its contributors, nor the name "Laurence Lundblade" may be used to endorse or promote products derived from this software without specific prior written permission.
+THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+This page is under construction
+ + + + + + + + + + + + + +Scott Burleigh, Jay Gao, and Leigh Torgerson
+Jet Propulsion Laboratory, California Institute of Technology
+Version 4.1.3
+ION open source comes with an Excel spreadsheet to help users configure the LTP protocol to optimize performance based on each user's unique use case.
+ION's implementation of LTP is challenging to configure: there are a lot +of configuration parameters to set, because the design is intended to +support a very wide variety of deployment scenarios that are optimized +for a variety of different figures of merit (utility metrics).
+LTP-ION is managed as a collection of "spans", that is, +transmission/reception relationships between the local LTP engine (the +engine -- or DTN "node" -- that you are configuring) and each other LTP +engine with which the local engine can exchange LTP protocol segments. +Spans are managed using functions defined in libltpP.c that are offered +to the operator by the ltpadmin program.
+ltpadmin can be used to add a span, update an existing span, delete a +span, provide current information on a specified span, or list all +spans. The span configuration parameters that must be set when you add +or update a span are as follows:
+remote LTP engine number identifying the span. For ION, this
+ is by convention the same as the BP node number as established when
+ the ION database was initialized.maximum number of export sessions that can be held open on
+ this span at any one time. This implements LTP flow control across
+ the span: since no new data can be transmitted until it is appended
+ to a block -- the data to be conveyed in a single export session --
+ and no new session can be started until the total number of open
+ sessions drops below the maximum, the closure of export sessions
+ regulates the rate at which LTP can be used to transmit data.maximum number of import sessions that will be open on this
+ span at any one time. This value is simply the remote engine's own
+ value for the "maximum number of export sessions" parameter.Maximum LTP segment size. This value is typically the maximum
+ permitted size of the payload of each link-layer protocol data unit
+ -- nominally a frame.Aggregation size limit. This is the "nominal" size for blocks to
+ be sent over this span: normally LTP will concatenate multiple
+ service data units (such as BP bundles) into a single block until
+ the aggregate size of those service data units exceeds the
+ aggregation size limit, and only then will it divide the block into
+ segments and use the underlying link service to transmit the
+ segments. (Note that it is normal for the aggregation size limit to
+ be exceeded. In this sense, the word "limit" is really a misnomer;
+ "threshold" would be a better term.)Aggregation time limit. This parameter establishes an alternate
+ means of terminating block aggregation and initiating segment
+ transmission: in the event that service data units are not being
+ presented to LTP rapidly enough to promptly fill blocks of nominal
+ size, LTP will arbitrarily terminate aggregation when the length of
+ time that the oldest service data units in the block have been
+ waiting for transmission exceeds the aggregation time limit.The Link Service Output command. This parameter declares the
+ command that will be used to start the link service output task for
+ this span. The value of this parameter is a string, typically
+ enclosed in single quote marks and typically beginning with the name
+ of the executable object for the task. When the "udplso" link
+ service output module is to be used for a given span, the module
+ name is followed by the IPAddress:Port of the remote engine and
+ (optionally) the UDP transmission rate limit in bits per second.In addition, at the time you initialize LTP (normally at the start of +the ltpadmin configuration file) you must set one further configuration +parameter:
+Estimated total number of export sessions, for all spans: this
+ value is used to size the hash table that LTP uses for storing and
+ retrieving export session information.In many cases, the best values for these configuration parameters will +not be obvious to the DTN network administrator. To simplify this task, +an LTP Configuration Worksheet has been developed.
+The LTP configuration worksheet is designed to aid in the configuration +of a single span -- that is, the worksheet +for the span between engines X and Y will provide configuration +parameter values for use in commanding ltpadmin on both engine X and +engine Y.
+The cells of the worksheet are of two general types, Input Cells and
+Calculated Cells.
Input Cells are cells in which the network administrator must supply
+ values based on project decisions. These cells are yellow-filled.Calculated Cells are cells that are computed by the worksheet based
+ on LTP configuration principles. These cells are grey-filled. The
+ cells are protected from modification (though you can unprotect them
+ if you want by selecting "Unprotect Sheet" on the Excel "Review"
+ tab).Some of these cells are used as span configuration parameters or are +figures of merit for network administrators:
+Note: Configuration parameters that are described in detail in this +document are numbered. To ease cross referencing between this document +and the worksheet, the parameter numbers are placed next to the title +cells in the worksheet.*
+This section provides guidance on the values that must be supplied by +the network administrator. Global parameters affect calculated values +and configuration file parameters for all spans involving the local LTP +engine.
+Maximum bit error rate is the maximum bit error rate that the LTP
+should provide for in computing the maximum number of transmission
+efforts to initiate in the course of transmitting a given block. (Note
+that this computation is also sensitive to data segment size and to the
+size of the block that is to be transmitted.) The default value is
+.000001, i.e., 10^-6^, one uncorrected (but detected) bit error per
+million bits transmitted.
The size - estimated size of an LTP report segment in bytes - may vary
+slightly depending on the sizes of the session numbers in use. 25 bytes
+is a reasonable estimate.
Values for the following parameters must be provided by the network +administrator in order for the worksheet to guide the configuration. +Values must be provided for both engine "X" and engine "Y".
+OWLT between engines (sec) is the maximum one-way light time
+ over this span, i.e., the distance between the engines. (Note that
+ this value is assumed to be symmetrical.)engine number for each engine.IP address of each engine. (Assuming udplso will be used as the
+ link service output daemon.)LTP reception port number for each engine. (Again assuming
+ udplso will be used as the link service output daemon.)mean size of the LTP service data units
+ (nominally bundles) sent from this engine over this span.Link service overhead. The expected number of bytes of link service
+ protocol header information per LTP segment.Aggregation size limit - this is the service data unit aggregation
+ size limit for LTP. Note that a suggested
+ value for this parameter is automatically computed as described
+ below, based on available return channel capacity.scheduled transmission rate (in bytes per second) at which this
+ engine will transmit data over this span when the two engines are in
+ contact.Maximum percentage of channel capacity that may be consumed by LTP report segments. A warning will be displayed if other configuration
+ parameters cause this limit to be breached. There are no actual
+ mechanism to enforce this limit in ION. This only set in order to
+ check the estimated report traffic for the current configuration.
+ It is provided as an aid to LTP link designer.estimate of the percentage of all data sent over this span that will be red data, i.e., will be subject to positive and negative
+ LTP acknowledgment.Aggregation time limit. The minimum value is 1 second. Increasing
+ this limit can marginally reduce the number of blocks transmitted,
+ and hence protocol overhead, at times of low communication activity.
+ However, it reduces the "responsiveness" of the protocol, increasing
+ the maximum possible delay before transmission of any given service
+ data unit. (This delay is referred to as "data aggregation
+ latency".)
Low communication activity is defined as a rate of
+ presentation of service data to LTP that is less than the
+ aggregation size limit divided by the aggregation time limit.LTP segment size (bytes) is the maximum LTP segment size sent over
+this span by this engine. Typically, this is the maximum permitted
+size of the payloads of link-layer protocol data units (frames).The maximum number of export sessions. This implements a form of
+flow control by placing a limit on the number of concurrent LTP
+sessions used to transmit blocks. Smaller numbers will result in
+slower transmission, while higher numbers increase storage resource
+occupancy. Note that a suggested value for this parameter is
+automatically computed as described below, based on transmission
+rate and one-way light time.This section provides further information on the methods used to compute
+the Calculated Cells and also guidance for Input Cell values.
The following parameters are automatically computed based on the values +of the basic input parameters.
+Estimated "red" data transmission rate (bytes/sec) is simply the
+ scheduled transmission rate multiplied by the estimated "red" data
+ percentage.Maximum export data in transit (bytes) is the product of the
+ estimated red data transmission rate and the round-trip light time
+ (which is twice the one-way light time between the engines). This is
+ the maximum amount of red data that cannot yet have been positively
+ acknowledged by the remote engine and therefore must be retained in
+ storage for possible retransmission.Values for the following parameters must be chosen by the network +administrator on the basis of (a) known project requirements or +preferences. (b) the first-order computed parameters, and (c) the +computed values of figures of merit that result from tentative parameter +value selections, as noted.
+#6 Aggregation size limit (revisited). Reducing this parameter
+ tends to increase the number of blocks transmitted, increasing total
+ protocol overhead. The suggested value for this parameter is
+ computed as follows:The threshold block size, expressed in bytes per block, is then + given by dividing the local engine's transmission data rate (in + bytes per second) by the maximum number of blocks to be transmitted + per second.
+Est. mean export block size is computed as follows:
a. If the mean service data unit size is so large that aggregation +of multiple service data units into a block is never necessary, +then that mean service data unit size will in effect determine +the mean export block size (one service data unit per block).
+b. Otherwise, the mean export block size will be determined by +aggregation. If the red data transmission rate is so high that +the aggregation time limit will normally never be reached, then +the aggregation size limit constitutes the mean export block +size. Otherwise, block size will be constrained by aggregation +time limit expiration: the estimated mean export block size will +be approximated by multiplying the red data transmission rate by +the number of seconds in the aggregation time limit.
+c. So estimated mean export block size is computed as larger of
+mean service data unit size and "expected aggregate block size",
+where expected aggregate block size is the lesser of block
+aggregation size limit and the product of red data transmission
+rate and aggregation time limit.
+16. Estimated blocks transmitted per second are computed as Estimated red data xmit rate (bytes/sec) (parameter 13) divided by Est. mean export block size (parameter 15).
+17. Est. Report bytes/sec sent by the remote engine in response to these transmitted blocks is computed as the product of Est. blocks transmitted per second (parameter 16) and Size (mean) of LTP acknowledgment (bytes) (a global parameter). When mean service data unit size is less than the aggregation size limit and the red data transmission rate is high enough to prevent the aggregation time limit from ever being reached, this value will be about the same as the maximum number of bytes of LTP report content that the remote engine may transmit per second as computed above.
Note: increasing the aggregation size limit reduces the block transmission rate at the local engine, reducing the rate of transmission of acknowledgment data at the remote engine; this can be a significant consideration on highly asymmetrical links.
+18. Est. segments per block is computed as Est. mean export block size (parameter 15) divided by LTP segment size (bytes) (parameter 11).
+19. Est. LTP delivery efficiency on the span is calculated by dividing Est. blocks delivered per second by Est blocks transmitted per second. Reducing the aggregation size limit indirectly improves delivery efficiency by reducing block size, thus reducing the percentage of transmitted blocks that will be affected by the loss of a given number of frames.
#12 Maximum number of export sessions (revisited). Increasing the maximum number of export sessions will tend to improve link bandwidth utilization but will increase the amount of storage space needed for span state retention. The suggested value for this parameter is computed as the maximum export data in transit (bytes) (Parameter 14) divided by Est. mean export block size (parameter 15) as determined above. Configuring the span for a maximum export session count that is less than this limit will make it impossible to fully utilize the link even if all blocks are of estimated mean size.Nominal export SDU's in transit is computed by dividing
+Nominal export data in transit (bytes) by the Size (mean) of service data units (bytes) (parameter 4).Expected link utilization is then computed by dividing Nominal export data in transit (bytes) by Maximum export data in transit (bytes) (parameter 14). Note that a low value of expected link utilization indicates that a high percentage of the span's transmission capacity is not being used. Utilization can be improved by increasing estimated mean export block size (e.g., by increasing aggregation size limit) or by increasing the maximum number of export sessions.Max data aggregation latency (sec) is simply the value supplied
+for Aggregation time limit (sec) (parameter 10) as this time
+limit is never exceeded.Finally, the remaining LTP initialization parameter can be computed when +all span configuration decisions have been made.
+Maximum number of import sessions is automatically taken from
+ the remote engine's maximum number of export sessions.This research was carried out at the Jet Propulsion Laboratory, +California Institute of Technology, under a contract with the National +Aeronautics and Space Administration.
+This section describes the following features added to the configuration +tool as of May 2021:
+The recommended workflow for using the LTP configuration tool is to +first establish the space link configuration using the link worksheet +before attempting to generate a LTP configuration under the main +worksheet. The link worksheet has the following input and computed +cells:
+Select CCSDS Frame Size (bits) \[user input\]-- this cell allows
+ the user to select a standard CCSDS AOS/TM frame size from a drop
+ down list that includes LDPC, Turbo, and Reed-Solomon codes.CCSDS Frame Size (bytes) \[computed\]-- converts frame size from
+ bits to bytes.Desired Frame Error Rate \[user input\] -- this parameter sets
+ the expected frame error rate of the LTP link in operation. This
+ parameter could be derived from link budget analysis or a mission
+ requirement document.Segment size (byte) \[user input\] -- this parameter sets the
+ maximum segment payload size used by LTP. The size of the segment,
+ in relation to the underlying CCSDS frame, will determine the
+ segment error rate and the probability that LTP will need to request
+ retransmission.Ethernet Frame Size (byte) \[user input\] -- this is the
+ Ethernet frame size used in a laboratory environment to simulate
+ space link frame losses.Segment Error Rate Computation \[computed\] -- this is the LTP
+ segment error rate derived from the frame error rate and the segment
+ and CCSDS frame size selections.*maxBER* Computation \[computed\] -- this is the computed
+ maxBER parameter for LTP. The maxBER parameter is what LTP uses
+ to estimate segment error rate, which in turn will affect how LTP
+ handles handshaking failure and repeated retransmission requests. To
+ properly operate LTP, the maxBER value provided must result in the
+ same segment error rate as one expects to encounter in real space
+ link.Ethernet Error Rate Computation \[computed\] -- this is the
+ recommended setting for using laboratory Ethernet frame error
+ software/hardware to simulate space link loss. This value is
+ translated from the segment error rate to Ethernet frame error to
+ make sure that laboratory testing provides a statistically
+ equivalent impact on LTP.In the main worksheet described in Section 3, we made the following +enhancements:
+Aggregation size limit (bytes) -- a green icon is
+ displayed when the input parameter is greater or equal to the
+ suggested value; a red icon is displayed when this parameter is
+ below the suggested value. The suggested value aggregation size
+ limit upper bounds the LTP block rate such that the acknowledgement
+ traffic (report segments) from the receiver to the sender can be
+ supported.Aggregation time limit (sec) -- there are two factors
+ affecting LTP block aggregation: time limit and size limit. The
+ aggregation process stops as soon as one of the two limits is
+ reached. A green icon is displayed if the time limit value in this
+ cell is sufficiently large such that the aggregation process will be
+ size limited, i.e., on average the LTP block aggregation process
+ will reach the size limit before the time limit. This is the nominal
+ and desired configuration unless there is a strict latency
+ requirement that forces one to use a very low aggregation time
+ limit. A red icon is displayed if the time limit will be driving,
+ which means the LTP block size will generally be smaller than the
+ aggregation size limit and the block rate will be higher than
+ desired. If a latency requirement forces the use of a low
+ aggregation time limit, one must check to make sure there is still
+ sufficient bandwidth to support the acknowledgement (report segment)
+ traffic.Est. report bytes/sec sent - this field estimates the
+ bandwidth required to support LTP report segment traffic up to 95
+ percentile of all cases involving retransmission of missing
+ segments. The segment error rate was derived from the link
+ worksheet. The green icon indicates that estimated report bandwidth
+ is feasible based on current configuration.A simple HeapWord size estimate calculation is added to the main +worksheet, based on the following assumptions:
+Longest Expected Period to Buffer Data Period (sec) --
+ this is the expected longest period of time one expects ION will
+ buffer user data. The data accumulation rate is the same as the LTP
+ red data data rate.(32/64) bit system -- this is platform dependent
+ parameter. Heap space is specified in the number of words. For a
+ 32-bit system, each word has 32 bits; for a 64-bit system, each word
+ has 64 bits.Additional Margin -- adds more margin to the model per
+ user's discretionRecommended heapWords value (with 40% for ZCO) -- this
+ is the suggested heapWords value for ionconfig.Recommended heapWords value - iif source data completely in memory -- this is the suggested heapWords value for
+ ionconfig assuming the source data is copied into heap space at
+ the time of bundle creation without using file-resident reference.wmSize recommendation - this is the suggested wmSize
+ parameter to use to support the staging of large quantities of
+ bundles in the heap. This recommended value includes an additional
+ 200% margin. The rationale for the calculation is derived from
+ analysis summarized in the ION Deployment Guide and based on
+ previous studies.In this section, we summarize the finding documented in a powerpoint +presentation titled, "ION DTN/LTP Configuration and ION Memory Usage +Analysis", dated January 2021, that is used as the basis for estimating +the heap space required for BP/LTP operation in ION:
+With 1 bundle in the system that is under active LTP session, the + minimum heap space needed can be approximated as: heap space = + S + base
+S is the bundle size / segment size x segment header size x 10
+If N \<= M, heap space usage is approximately S x N + 1560 bytes x + N + base
+S x N is LTP related heap usage, 1560 x N is bundle level heap + usage
+If N > M, heap space usage is approximately S x M + 1560 bytes x + N + base
+For bundle in LTP transmission, we count both LTP and bundle + level heap usage
+Nik Ansell co-authored/contributed to the 2016 version of this document, +which has been updated and revised in 2021.
+© 2016 California Institute of Technology. Government sponsorship +acknowledged.
+ + + + + + + + + + + + + +