Update fleet-ms and Lat/Long CSV capabilities to be more dynamic

[osowski]

Initially the CSV coordinate files under fleet-ms/src/main/resources were supposed to allow ships to navigate a somewhat realistic path across the water, as a straight line between two ports was not usually possible.

The CSV pattern used should be updated to either pull this information from an external source, import a new CSV file for pairs of Ports, or create a new microservice for route management.

[djones6]

Having a new 'route' data type (and possibly a new microservice to manage it) seems to make sense.

I'd imagine defining that data type as a list of lat/long pairs (the endpoints of which should fall within the bounding boxes of a port), plus an ID and/or name.

There's going to be a relationship between the 'voyage' data type and the 'route'(s). We should think about what that's going to be, particularly if the simulation becomes perpetual. Here are some ideas:

1. The voyage data type could change, to reference a route (or routes?) rather than having a start/end point:

• We could interpret a single route as a path that a ship just goes back and forth along indefinitely
• We could allow a list of routes so that a ship can do A -> B -> C -> A

2. We could leave the voyage data type as specifying a start and end port (or even a list of ports, if we didn't want a simple back-and-forth), and have the routes MS provide a search for a route between two given locations.

3. Rather than having 'routes' and 'voyages', we could just have one type, and associate a ship with that, since a perpetual simulation no longer needs the concept of a voyage start and end date. This would probably limit us to ships going back and forth on a set route, but could be sufficient for example purposes.

For ease of creating / defining routes, rather than requiring routes to have a bunch of lat/long coordinates, we could include some concept of time ('steps' or 'days') that the route represents. Then the routes microservice could create the internal representation of those routes by interpolating between coordinates in order to create the required number of steps. This would simplify the data, but add complexity to the code, so I'm not sure if this is a good idea or not yet.

[osowski]

I would like @jbcodeforce to chime in here as well, since he will have the original context as to how far this may or may not diverge from the original brainstorming session (and what downstream scenario documentation we would need to update).

I think it makes sense (your bullet point 1 above) to have a data model which supports Ships taking Voyages and Voyages going along a Route. This ends up creating a 1:N relationship for Ships to Voyages (but only 1:1 at any given point in time) and an 1:N relationship for Routes being traveled by many Voyages.

This would require the creation of a new Routes MS with CRUD capabilities and the inclusion of Route information into the Voyages data model (which shouldn't complicate things too much). We can then perpetually simulate the travel status of each individual Voyage along a Route, based upon the series of Lat/Long pairs stored in a given Route instance. This updated simulation would make the most sense to be contained in the existing Voyage MS (evolving into a voyage movement simulator), with the Reefer Simulator functionality existing outside of that and providing the exception/anomaly simulation to generate errors.

Does this make sense? Are we thinking the same thing?

[jesusmah]

What you @osowski say above about the data model Ships --> Voyages --> Routes makes sense to me. However, I have some doubts:

1. What is the purpose of Routes MS? If it is only to serve static routes through REST then it would not make sense to me to have yet another microservice which is simply serving information (either loaded in memory from a CSV or from a DB). It would make sense if these routes from Port A to Port B could change based on some AI and Analytics on weather prediction for providing the most suitable route for a specific voyage (which is route a ship will cove from port A to port B at a specific point in time). Of course, I'm not suggesting to implement such weather prediction analytics but trying to understand why routes would need to be a separate MS.
2. With the data model you suggest where a ship will carry out a voyage that is a route covered at specific point in time, Im worried we could be over complicating things that "only" provide completeness to our application in order to have a more visual end-to-end scenario but that could be deviating us from a more EDA technical work... Im not saying I dont like the idea at all. In fact, I'd love to have a more visual and complete solution so that when we demo it we dont simply show json messages in the terminal here and there. I guess Im just simply making the point of trying to keep this a simple as possible :(

[osowski]

Definitely agree on point number 2 above @jesusmah , but clarifying your point number 1 should help.

Routes are a separate data domain and would be handle by a separate Actor in the overall application. Ship Captains would need to work with Port Authorities and use their knowledge of international shipping lanes to register the route that their ship will travel for a specific voyage. By providing a Route MS serving up static routes (via lat/long lists), we are separating the ownership of the domain of route management from the other Actors in the system, while still providing them a way to interact with that set of necessary data.

The Route MS could very easily evolve into using AI & Analytics to refine the lat/long pair lists - but that is when we start to veer away from the KISS principle for the time being. We could definitely introduce additional eventing into the system based on Weather Service updates, with the Voyage microservice checking back in for the next N bearings and if there is an affect on weather etc. But that is going way beyond what we need here....

So keeping it as simple as possible, yet still staying true to our separation of concerns, bounded contexts, and domain models, a Routes MS serving up static route lat/long pairs (with basic CRUD APIs) allows us to integrate Voyages to simulate the real-world example of the coordination of Actors across domains. When I mention CRUD APIs, this would allow us to create new routes via POST actions (starting simple with Point A to Point B, then scaling up to N points) instead of just a static CSV file or DB call. This would go more inline with ibm-cloud-architecture/refarch-kc#56 to remove the pre-canned data and encourage all demo / system-priming data to be inserted via REST APIs.

[djones6]

@osowski @jesusmah For the sake of keeping things simple, I propose that I start off by extending either the voyages or fleets microservice to add the Routes functionality (I think it probably makes sense more in voyages?). It could be split out into a separate microservice later if we decided that was better (eg. if we did decide we wanted to make routes more dynamic).

[osowski]

@djones6 I agree with your proposal. Based off of the Summary of microservice scopes, it makes sense to extend Voyages with the new Routes functionality and data model updates initially.

Once this has been implemented, the fleet-ms microservice can be updated to take advantage of the new Routes data for end-to-end simulation (since the simulation is currently being driven at the Ship/Fleet-level from within the fleet-ms microservice)

[josiemundi]

I understand the thinking to have all hardcoded data removed and thus not having the static csvs. However, my intial feeling is that the movement of ships should be kept reasonably simple as it's kind of a given, in a shipping scenario, that there would be a fleet of ships moving.

As another idea is that we could source a few historical shipping routes for a small fleet of ships in one specific area (e.g North Atlantic) and then use this to populate the initial routes? Mostly ships keep to the same shipping lane routes, these are quite well defined. We can then pick the corresponding ports (just a few at first) and these can be the ones we initially start with.

This doesn't limit the ability to feed in new and more complex routes in the future but does give a mechanism for quickly starting up a realistic demo that could quickly highlight some of the other services that are more demonstrative of the eventing benefits we are trying to show.

[jesusmah]

@djones6 yes, it does make sense to me as well to use the Voyage microservice to begin with the Routes functionality extension.
@josiemundi yeah, the idea here as in with any type of development we do is to start small and iterate if needed. So I agree with you that we should begin with few routes and ports (we are not pretending to simulate 100s of ships at any given point) and then extend if we thing that brings any value to our goals. As to how to populate the initial routes, Im afraid I dont know how to get those coordinates, etc.

[jbcodeforce]

I'm late in the discussion sorry.
voyage has multiple legs, and when the voyage MS get to assign an Order to the voyage it takes the source and destination zipcode and match to two closed harbor. So a voyage with have one leg from source to intermediates harbors to final harbors. We can name route = leg. It is fine. To demonstrate streaming to Maersk I need to have the Vessel separated. So for me a Vessel microservice will support moving a Vessel from two points on the map and send vessel-course-event to kafka so an real time aggregator can compute ETA and time to destination, and average boat speed. So the route object will be what the vessel follows, and we can define a route with 5 points, so 3 intermediate points on the sea the boat will follow. in the vessel simulator it will play random modification of his speed so it can report different latitude and longitude as it progresses over those 4 section of its route. I think I agree to group voyage and route inside Voyage MS and just expose an new API to get the routes of a given voyage id. For the data, as we ship fresh food from california to china, we limit to north pacific ocean and we can have 2 or 3 routes defined with 5 points each. the complexity of the simulator will be to compute the new vessel position on the map using dot vector computation at every tick of the game loop. A game loop representing x hours of real life. So scaling is needed if we want to run the demo for 2 minutes to ship from oakland to beijing.