DETAILS.txt

***TABLE OF CONTENTS***

    LICENSE
    PACKAGE CONTENTS
    LAUNCHING PROCESSES
    DEFINING ENDPOINTS
    CLEANER
    BUILDING THE IMPLEMENTATION
    CONFIGURATION FILE
    THREAD SAFETY
    DEPENDENCIES
    INTERNAL STRUCTURE OF IMPLEMENTATION
    IMPLEMENTATION SPECIFIC BEHAVIOR
    IMPLEMENTATION SPECIFIC ERRORS
    TEST ENVIRONMENTS

***LICENSE***

FOR THE LICENSE OF THIS SOFTWARE SEE LICENSE.txt

AS AN EXCEPTION, HEADER FILES mcapi.h AND mcapi_impl_spec.h HAVE DIFFERENT
LICENCES COPYRIGHTED TO MULTICORE ASSOCIATION. HOWEVER, THESE ARE OPEN SOURCE.

***PACKAGE CONTENTS***

Includes a POSIX message queue based MCAPI implementation. The observed version
of the API specification is 2.015. A "hello world" level example is provided as
well as unit tests and integration tests. Files and folders are explained in 
more detail below:

FILES:

LICENSE.txt     | THE LICENSE OF THIS SOFTWARE
README.txt      | Instructions for running the provided "hello world" example
DETAILS.txt     | This document: Contains more detailed information about this
                  MCAPI implementation and provided files.
launch.sh       | Script to build application processes.
                  More details in chapter LAUNCHING PROCESSES
launcher.sh     | Script to launch application processes.
                  More details in chapter LAUNCHING PROCESSES
makefile.mcapi  | Script to build the implementation object files
                  More details in chapter BUILDING THE IMPLEMENTATION
makefile.cleaner| Script to build and run the cleaner executable.
                  More details in chapter CLEANER

FOLDERS:

/example        | Contains the provided "hello world" example.
                  More details within README.TXT
/include        | Header files of both MCAPI specification and implementation
/src            | Source files of the MCAPI implementation
/obj            | Object files of MCAPI implementation.
                  ***WARNING*** provided makefiles may delete contents.
                  The folder may be recreated by makefiles.
/utests         | Unit tests: Contains its own README with more details
/itests         | Integration tests: Contains its own README with more details

***LAUNCHING PROCESSES***

It is highly recommended to use launch.sh or other script to launch processes.
While launch.sh is supposed to be called by user, launcher.sh is used by
launch.sh. Below example illustrates how the launcher script is executed.

    example: ./launch.sh

If one application process crashes or exits, the other processes should also 
should also do so before restart. The provided script helps with that by
killing the old processes. You should not execute the launcher again before the
previous instance have launched the application processes, which is indicated 
with print "PROCESSES LAUNCHED". Furthermore, launcher.sh runs the required
cleaner executable. See chapter CLEANER for more details.

When the launch.sh is executed, names of the launched processes are inserted
to a file called "hitlist.txt". When the script is ran again, the script goes 
over the list and kills the listed processes. For this reason, you may want
make sure that no two processes use the same name when launched. Signal SIGTSTP
(Ctrl + Z) will also kill the processes, but it will not work before processes 
are launched.

In beginning of launch.sh there is place for application executables.
Insert them there.

    example: PROCESSES=(ITcyan.exe ITblue.exe)

The next line allows to choose if executables are built while executing the 
script. The section for build rules is further below within the script. See
more details in chapter BUILDING THE IMPLEMENTATION.

***DEFINING ENDPOINTS***

Each endpoint must be defined in file "includes/endpointlist.h" which is
statically linked with the implementation, the cleaner executable and
application executable. Each of them needs to refer to the content to access
endpoints. Every time this file is changed, implementation object files and
every executable must be rebuilt. In addition to defining the endpoints, the
definitions must be added to the list in the define file. PLEASE NOTICE THAT
THE FORMAT OF THE FILE IS NOT DESIGNED TO BE USER FRIENDLY, BUT INSTEAD SHOULD
BE GENERATED BY A THIRD PARTY TOOL.

example of definition usage in application:
    
    point = mcapi_endpoint_get( DEFINED_DOMAIN_ID, DEFINED_NODE_ID,
    DEFINED_PORT_ID, TIMEOUT, &status );

For defining endpoint identifier use following format:

    #define DEFINED_DOMAIN_ID FOO
    #define DEFINED_NODE_ID BAR
    #define DEFINED_PORT_ID BAZ

For each endpoint receiving or sending messages use following format:

    #define ENDPOINT_NAME_DEF { { DEFINED_DOMAIN_ID, DEFINED_NODE_ID, \
    DEFINED_PORT_ID }, MCAPI_NO_CHAN, CHAN_NO_DIR }

For scalar channels use following format. Replace SCALAR_SIZE_IN_BYTES with
1, 2, 4 or 8. Both endpoints of the channel must have the same scalar size.

    #define RECEIVING_NAME_DEF { { RECEIVING_DOMAIN_ID, RECEIVING_NODE_ID, \ 
    RECEIVING_PORT_ID }, MCAPI_SCL_CHAN, CHAN_DIR_RECV, { SENDING_DOMAIN_ID, \
    SENDING_NODE_ID, SENDING_PORT_ID }, SCALAR_SIZE_IN_BYTES }

    #define SENDING_NAME_DEF { { SENDING_DOMAIN_ID, SENDING_NODE_ID, \
    SENDING_PORT_ID }, MCAPI_SCL_CHAN, CHAN_DIR_SEND, { RECEIVING_DOMAIN_ID, 
    RECEIVING_NODE_ID, RECEIVING_PORT_ID }, SCALAR_SIZE_IN_BYTES }

In case of packet channels, replace "MCAPI_SCL_CHAN" with "MCAPI_PKT_CHAN".
Also, packet channels do not need definition of the scalar size.

To form the mandatory list of endpoints, use following format:

    #define DEF_LIST { ENDPOINT_NAME_DEF, RECEIVING_NAME_DEF, \
    SENDING_NAME_DEF, A_DEF_IN_SECOND_ROW }

example of endpoint identifier definition:

    #define THE_DOMAIN 1
    #define YELLOW_NODE 13
    #define YELLOW_MSG_IN 0

example of message endpoint definition:

    #define YELLOW_MSG { THE_DOMAIN, YELLOW_NODE, YELLOW_MSG_IN }
    #define YELLOW_MSG_DEF { YELLOW_MSG, MCAPI_NO_CHAN, CHAN_NO_DIR }

example of scalar channel definition:

    #define YELLOW_SIN {1, 13, 1}
    #define RED_SIN {1, 14, 1}
    #define YELLOW_SIN_DEF { YELLOW_SIN, MCAPI_SCL_CHAN, CHAN_DIR_RECV, \
    RED_SIN, 2 }
    #define RED_SIN_DEF { RED_SIN, MCAPI_SCL_CHAN, CHAN_DIR_SEND, \
    YELLOW_SIN, 2 }

example of packet channel definition:

    #define MAG_CYAN_SEND {1, 10, 0}
    #define MAG_CYAN_RECV {1, 11, 0}
    #define MAG_CYAN_SEND_DEF { MAG_CYAN_SEND, MCAPI_PKT_CHAN, CHAN_DIR_SEND, \
    MAG_CYAN_RECV, 0 }
    #define MAG_CYAN_RECV_DEF { MAG_CYAN_RECV, MCAPI_PKT_CHAN, CHAN_DIR_RECV, \ 
    MAG_CYAN_SEND, 0 }

example of list definition:

    #define DEF_LIST { YELLOW_MSG_DEF, YELLOW_SIN_DEF, RED_SIN_DEF, \
    MAG_CYAN_SEND_DEF, MAG_CYAN_RECV_DEF }

***CLEANER***

Cleaner is a simple executable, which will reconstruct names of all POSIX
message queues used by defined endpoints and call "mq_unlink" with said
name as a parameter. This will remove the name, but the message queue proper
will still exist if any process uses it. Thus, executing the cleaner while
application processes are running will result in undefined behavior.
Futhermore, if cleaner is not ran before application processes, they may be
unable to open the new POSIX message queues properly.

Since the cleaner needs access to said defined endpoints, it needs to be
rebuilt every time they are changed. A makefile, "makefile.cleaner",
contains instructions to build and run the cleaner.

    example: ./clean

***BUILDING THE IMPLEMENTATION***

Other makefile, "makefile.mcapi", contains the build instructions for the
implementation object files. It is not intended to produce any kind of
standalone executable, but instead produce the object files to be statically
linked with the applications. The file has also a clean command, which will 
remove the implementation object files from directory "obj".

    example: make -f makefile.mcapi
    example: make -f makefile.mcapi clean

Object files in folder "obj" are supposed to be implementation object files,
which are statically linked to every application executable, while folder
"include" is needed to include their header files, so that the application may
access the functionality. Flag -lrt must be provided to GCC so that it will be
able to access the POSIX message queues.

    example: gcc -o esender example/example_sender.c $ODIR/*.o -I$IDIR -lrt

***CONFIGURATION FILE***

The include/mcapi_config.h is a file containing configurations definable by
application. This includes amount of packet receive buffers and maximum sizes
for both messages and packets. Both implementation and application must be
rebuilt every time these values are changed or undefined behavior will
occur. The file contains more details about each define.

***THREAD SAFETY***

Thread safety is an opt-in feature, turned on and off in configuration file.
For more details on configuration file, see chapter CONFIGURATION FILE.

Configuration with thread safety will not achieve same performance as
configuration without. Furthermore, thread safety is not a thoroughly tested
feature.

***DEPENDENCIES***

THIS IMPLEMENTATION SHOULD RUN ON ANY LINUX FEATURING POSIX MESSAGE QUEUES,
C-COMPILER AND MAKE.

However, tests were executed successfully in systems described in chapter TEST
ENVIRONMENTS. Flag -lrt must be provided to GCC so that it will be able to
access the POSIX message queues. The implementation expects that the comment 
notation "//" is supported by the compiler. Also usleep, clock_gettime and 
CLOCK_REALTIME are used. Files sys/stat.h and sys/fcntl.h are needed for some 
defines. POSIX semaphores are needed for thread safe implementation.

***INTERNAL STRUCTURE OF IMPLEMENTATION***

The graph below illustrates dependencies between modules. Modules below are
dependent on those modules above which they are connected to with a line.
Indirect dependencies are culled out for simplicity. Names with extension ".h"
are merely headers, while extensionless names are modules with their own
implemented functions.

The POSIX message queue specific functions were centered to pmq_layer-module.
Datatypes endPointData and endPointDef also contain variables specific to
POSIX message queues. External dependencies are not included in the graph.

                   mca.h
                    |
                    |---------------------------    endpointlist.h
                    |                           \   /
                    |                     endpointdef   mcapi_config.h
                    |                           |       /
                    |   ------------------mcapi_impl_spec.h
                    |  /
                mcapi.h         pmq_layer
                    |            /
      --------------|------------
     /              |            \
    msg          channel        node
                    |             |
                 ---------     endpoint
                /         \
            scalar      packet

***IMPLEMENTATION SPECIFIC BEHAVIOR***

Within this implementation each node is supposed to be one process. Domains are
user defined, but within the same processor. Endpoints must be defined before
compilation. See chapter DEFINING ENDPOINTS for more details.

This implementation allows use of channel endpoints for message based
communication, even if both ends of channels are open and connected.
Communication between scalar endpoints of different size may be allowed,
but is strongly discouraged. Channels may work even without connection.

NOTICE: none of above relaxations to specification should be utilized, as
it will result in non-portable application.

Most of of the non-blocking functions are not implemented. Notable exceptions
are calls oriented with connecting, opening and closing channels.

The only supported endpoint attribute is MCAPI_ENDP_ATTR_TIMEOUT, which will
set the timeout for the blocking send and receive calls. As such, the only
implemented attribute function is mcapi_endpoint_set_attribute. Node attributes
are not implemented either.

Other non-implemented functions include mcapi_pktchan_release_test,
mcapi_wait_any and mcapi_cancel.

The timeout data type mcapi_timeout_t is defined as milliseconds.

The pool of receive buffers for packets are shared within a node. The number of
buffers is configurable. See more details in chapter CONFIGURATION FILE.

***IMPLEMENTATION SPECIFIC ERRORS***

All error codes used may be found within MCAPI specification. However, some 
errors may occur on implementation specific situations. As a general rule, code
MCAPI_ERR_GENERAL is produced, if there was some bug in implementation, an unexpected external disturbance or application misused the API or 
implementation. MCAPI_ERR_TRANSMISSION is produced usually for same reasons. 
Usually MCAPI_ERR_GENERAL is accompanied by a print to standard error stream.

Calling mcapi_wait may also cause error prints, but in those cases it does
not return any error code. Trying to close a channel with close pending will
yield error MCAPI_ERR_CHAN_CLOSEPENDING.

More specifics listed below:

mcapi_endpoint_get:
    MCAPI_ERR_GENERAL, if failed for other POSIX-call related reason than
    non-existing POSIX message queue.
    MCAPI_ERR_PORT_INVALID, if the endpoint was undefined before compilation.

mcapi_endpoint_create:
    MCAPI_ERR_GENERAL, when failed to open the POSIX message queue, obtain its
    attributes or obtained attributes did not match specified attributes.
    MCAPI_ERR_PORT_INVALID, if the endpoint was undefined before compilation.

mcapi_msg_send:
mcapi_msg_recv:
mcapi_pktchan_send:
mcapi_pktchan_recv:
    MCAPI_ERR_TRANSMISSION, when POSIX call failed for other reason than 
    timeout. An error print will accompany this status code.

mcapi_sclchan_recv_uint64:
mcapi_sclchan_recv_uint32:
mcapi_sclchan_recv_uint16:
mcapi_sclchan_recv_uint8:
mcapi_sclchan_send_uint64:
mcapi_sclchan_send_uint32:
mcapi_sclchan_send_uint16:
mcapi_sclchan_send_uint8:
    MCAPI_TIMEOUT, when POSIX call failed for timeout.
    MCAPI_ERR_TRANSMISSION, when POSIX call failed for other reason than 
    timeout. An error print will accompany this status code.

mcapi_msg_available:
mcapi_pktchan_available:
mcapi_sclchan_available:
    MCAPI_ERR_GENERAL, when could not obtain the POSIX message queue
    attributes to check the count of received items.

***TEST ENVIRONMENTS***

The tests were successfully executed in the test environments listed below. The
first in the list was also the implementation environment.

SYSTEM #1:
    Operating system: CentOS Release 6.5 (Final)
    Kernel: Linux 2.6.32-431.17.1.el6.x86_64

    Processor: 4x Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHZ
    Memory: 3.6 Gib

    Compiler: gcc (GCC) 4.8.1
    Build automation: GNU Make 3.81

SYSTEM #2:
    Additional layer: Oracle VM VirtualBox 4.2.6 Edition
    Operating system: Linux Mint 14 Nadia
    Kernel: Linux 3.5.0-17-generic (x86_64)

    Processor: 4x Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz
    Memory: 2051 MB

    Compiler: gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2
    Build automation: GNU Make 3.81

SYSTEM #3:
    Operating system: Linux Mint 15 Olivia
    Kernel: Linux 3.8.0-19-generic (x86_64)

    Processor: 2x Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz
    Memory: 8 Gib

    Compiler: gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
    Build automation: GNU Make 3.81

SYSTEM #4:
    Operating system: Ubuntu 13.04
    Kernel: Linux 3.8.0-19-generic

    Processor: Intel(R) Core(TM) Duo CPU P7350 @ 2.00GHz
    Memory: 3.9 GiB

    Compiler: gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
    Build automation: GNU Make 3.81

SYSTEM #5:
    Additional layer: Oracle VM VirtualBox 4.1.16 Edition
    Operating system: Ubuntu 12.10 (quantal) 32-bit
    Kernel: Linux 3.5.0-17-generic

    Processor: Intel(R) Core(TM) Duo CPU P7350 @ 2.00GHz
    Memory: 496.4 MiB

    Compiler: gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2
    Build automation: GNU Make 3.81

SYSTEM #6:
    Additional layer: Cygwin 1.7.28
    Kernel: CYGWIN_NT-6.1

    Processor: Intel(R) Core(TM) Duo CPU P7350 @ 2.00GHz
    Memory: 3066 MiB

    Compiler: gcc (GCC) 4.8.2
    Build automation: GNU Make 4.0

SYSTEM #7:
    Operating system: Red Hat Enterprise Linux Server release 5.10
    Kernel: Linux 2.6.18-371.3.1.el5

    Processor: 8x Westmere E56xx/L56xx/X56xx (Nehalem-C)
    Memory: 14.4 GiB

    Compiler: gcc (GCC) 4.8.1
    Build automation: GNU Make 3.81

SYSTEM #8:
    Operating system: Red Hat Enterprise Linux Workstation release 6.5
    Kernel: Linux 2.6.32-431.5.1.el6.x86_64

    Processor: 2x Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
    Memory: 3.6 GiB

    Compiler: gcc (GCC) 4.8.1
    Build automation: GNU Make 3.81

SYSTEM #9:
    Additional layer: Oracle VM VirtualBox 4.3.6 Edition
    Operating system: archlinux-2014.03.01-x86_64
    Kernel: Linux archiso 3.13.5-1-ARCH
    
    Processor: 2x Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHZ
    Memory: 1 GiB

    Compiler: gcc (GCC) 4.8.2 20140206 (prerelease)
    Build automation: GNU Make 4.0

SYSTEM #10:
    Additional layer: Oracle VM VirtualBox 4.3.6 Edition
    Operating system: Linux Mint 16 Petra
    Kernel: Linux 3.11.0-12-generic
    
    Processor: Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHZ
    Memory: 1 GiB

    Compiler: gcc (Ubuntu/Linaro 4.8.1-10ubuntu8) 4.8.1
    Build automation: GNU Make 3.81

SYSTEM #11:
    Name: Cyclone V SOC Development Board
    Operating system: Linux socfpga 
    Kernel: 3.9.0
    
    Processor: ARM Cortex-A9 MPCore
    Memory: 1 GiB

    Compiler: Linaro GCC 4.7.3
    Build automation: GNU Make 3.81
    NOTICE: Was cross compiled on SYSTEM #1 and thus Make is the same