FTOT 2024_3_1 Patch

FTOT 2024.3.1

New: The FTOT 2024.3.1 patch includes two minor code updates to accommodate Python library versions associated with ArcGIS Pro 3.3. The changes resolve two instances when the SQLite database built during an FTOT scenario was locked, resulting in an error. The patch has been tested to ensure compatibility with ArcGIS Pro 3.2 and 3.3 versions.

No other changes were made in comparing FTOT 2024.3.1 and FTOT 2024.3. We recommend all users download the 2024.3.1 patch, though users with ArcGIS Pro 3.2 or older may continue to use the 2024.3 version without issue.

The changelog for FTOT 2024.3 is included below for reference.

FTOT 2024.3

The FTOT 2024.3 public release includes updates related to customization of first-mile and last-mile costs, custom demand fulfillment incentives by destination facility and commodity, default processor minimum capacities, the Tableau dashboard, as well as minor bug fixes and back-end improvements. The following changes have been made:

• Added functionality to allow users to model several first-mile, last-mile related costs. By default, FTOT models transport to and from user-defined facilities via short artificial links that connect to the modal networks. By default, transport on these links follows local road costs and impedances. The 2024.3 release allows users to (1) specify different transport costs on these links through the XML, (2) impede artificial links differently than modal network links, (3) define additional facility unit costs by facility, commodity phase of matter, and in vs. out movements through the facility-commodity CSV files. These customizations are reflected in FTOT reporting outputs.
• Expanded FTOT’s unmet demand penalty parameter (UDP) to allow for facility-specific and commodity-specific values. The default UDP parameter remains in the XML input file, which the user can overwrite or supplement with values in the destination facility-commodity CSV file. This allows the user to incentivize meeting demand at certain facilities (for certain commodities) by setting a higher UDP for that facility (or commodity).
• Adjusted the default minimum processor capacity to be zero when the user specifies a maximum capacity but no minimum capacity. Previously, FTOT used a default minimum capacity of half the maximum capacity—the user can still define this by explicitly defining minimum capacity in an optional column in the processor facility-commodity CSV file.
• In the Tableau workbook, fixed a bug in the bar chart tooltips in the By Supply & Demand dashboard and made various aesthetic improvements.
• Other updates:
  • Updated the ArcGIS dependency to require version 3.0 or higher.
  • Created an automated testing suite for the development team to ensure functionality of the codebase and improve development processes.
  • Fixed a logger warning that incorrectly indicated "Impedance weights file not specified", when an impedance weight file was specified.
  • Fixed a bug with processor capacities that caused FTOT to overwrite a minimum total input or output capacity when both minimum and maximum total capacities were provided for inputs or outputs to a facility.

See documentation files for additional details.

FTOT 2024_3 Public Release

The FTOT 2024.3 public release includes updates related to customization of first-mile and last-mile costs, custom demand fulfillment incentives by destination facility and commodity, default processor minimum capacities, the Tableau dashboard, as well as minor bug fixes and back-end improvements. The following changes have been made:

• Added functionality to allow users to model several first-mile, last-mile related costs. By default, FTOT models transport to and from user-defined facilities via short artificial links that connect to the modal networks. By default, transport on these links follows local road costs and impedances. The 2024.3 release allows users to (1) specify different transport costs on these links through the XML, (2) impede artificial links differently than modal network links, (3) define additional facility unit costs by facility, commodity phase of matter, and in vs. out movements through the facility-commodity CSV files. These customizations are reflected in FTOT reporting outputs.
• Expanded FTOT’s unmet demand penalty parameter (UDP) to allow for facility-specific and commodity-specific values. The default UDP parameter remains in the XML input file, which the user can overwrite or supplement with values in the destination facility-commodity CSV file. This allows the user to incentivize meeting demand at certain facilities (for certain commodities) by setting a higher UDP for that facility (or commodity).
• Adjusted the default minimum processor capacity to be zero when the user specifies a maximum capacity but no minimum capacity. Previously, FTOT used a default minimum capacity of half the maximum capacity—the user can still define this by explicitly defining minimum capacity in an optional column in the processor facility-commodity CSV file.
• In the Tableau workbook, fixed a bug in the bar chart tooltips in the By Supply & Demand dashboard and made various aesthetic improvements.
• Other updates:
  • Updated the ArcGIS dependency to require version 3.0 or higher.
  • Created an automated testing suite for the development team to ensure functionality of the codebase and improve development processes.
  • Fixed a logger warning that incorrectly indicated "Impedance weights file not specified", when an impedance weight file was specified.
  • Fixed a bug with processor capacities that caused FTOT to overwrite a minimum total input or output capacity when both minimum and maximum total capacities were provided for inputs or outputs to a facility.
    See documentation files for additional details.

FTOT 2024_2_1 Patch

FTOT 2024.2.1

New: The FTOT 2024.2.1 patch includes a small update to simple_setup.bat, the batch file used to set up the FTOT Python environment and required dependencies when first installing FTOT. The updated file names specific Python package versions to ensure compatibility of packages in the FTOT Python environment and has been tested with ArcGIS Pro v3.1.3.

No other files were updated in comparing FTOT 2024.2.1 and FTOT 2024.2. For users new to FTOT, we recommend downloading 2024.2.1 to ensure the FTOT Python environment is set up correctly. For users with an existing FTOT Python environment, FTOT 2024.2 should function as expected.

The changelog for FTOT 2024.2 is included below for reference.

FTOT 2024.2

The FTOT 2024.2 public release includes updates related to cost reporting outputs and visualizations, modeling of intermodal movement costs, scenario input validation, and back-end improvements to how the transportation network is processed and translated into NetworkX. The following changes have been made:

• Developed additional FTOT outputs to explain and visualize costs associated with the optimal routing solution of a scenario. A new CSV report summarizes both scaled and unscaled costs by commodity and by mode. The scaled costs account for user-specified transport and CO2 cost scalars and are used in the optimization. Costs are categorized as movement costs (broken out into costs from transport, transloading, first mile/last mile, mode short haul penalties, and impedances), emissions costs (broken out into CO2 and CO2 first mile/last mile), processor build costs, and unmet demand penalties. A new Cost Breakdown dashboard in the Tableau workbook visualizes the scaled and unscaled cost components.
• Updated the transport cost and routing cost methodology for intermodal movements. In addition to the transloading cost applied per unit moved between modes, transportation costs along the edges connecting the transloading facility to the rest of the FTOT network are now applied using the default per ton-mile (or thousand gallon-mile) costs of the mode that the transloading facility is connected to. The routing cost component from transport for intermodal movements is equivalent to the (unimpeded) transport cost.
• Added new input validation checks to confirm alignment between facility geodatabase and facility-commodity input files. The user is provided log messages when all facilities in the facility-commodity input files fail to match with a facility location in the corresponding feature class of the geodatabase.
• Updated method for hooking user-specified facilities into the network to ensure that all network segment attributes are passed down to split links (previously, only attributes that were part of the FTOT network specification were retained).
• Other updates:
  • Generalized FTOT code used to translate the network geodatabase into a NetworkX object to allow network segment attributes to be passed through the G step to support future extensions cost varying link cost based on other network attributes.
  • Fixed a logging bug that was consistently printing out a warning to users that certain log files were not successfully added to the FTOT text report.
    See documentation files for additional details.

FTOT 2024_2 Public Release

The FTOT 2024.2 public release includes updates related to cost reporting outputs and visualizations, modeling of intermodal movement costs, scenario input validation, and back-end improvements to how the transportation network is processed and translated into NetworkX. The following changes have been made:

• Developed additional FTOT outputs to explain and visualize costs associated with the optimal routing solution of a scenario. A new CSV report summarizes both scaled and unscaled costs by commodity and by mode. The scaled costs account for user-specified transport and CO2 cost scalars and are used in the optimization. Costs are categorized as movement costs (broken out into costs from transport, transloading, first mile/last mile, mode short haul penalties, and impedances), emissions costs (broken out into CO2 and CO2 first mile/last mile), processor build costs, and unmet demand penalties. A new Cost Breakdown dashboard in the Tableau workbook visualizes the scaled and unscaled cost components.
• Updated the transport cost and routing cost methodology for intermodal movements. In addition to the transloading cost applied per unit moved between modes, transportation costs along the edges connecting the transloading facility to the rest of the FTOT network are now applied using the default per ton-mile (or thousand gallon-mile) costs of the mode that the transloading facility is connected to. The routing cost component from transport for intermodal movements is equivalent to the (unimpeded) transport cost.
• Added new input validation checks to confirm alignment between facility geodatabase and facility-commodity input files. The user is provided log messages when all facilities in the facility-commodity input files fail to match with a facility location in the corresponding feature class of the geodatabase.
• Updated method for hooking user-specified facilities into the network to ensure that all network segment attributes are passed down to split links (previously, only attributes that were part of the FTOT network specification were retained).
• Other updates:
  • Generalized FTOT code used to translate the network geodatabase into a NetworkX object to allow network segment attributes to be passed through the G step to support future extensions cost varying link cost based on other network attributes.
  • Fixed a logging bug that was consistently printing out a warning to users that certain log files were not successfully added to the FTOT text report.
    See documentation files for additional details.

FTOT 2024_1 Public Release

The FTOT 2024.1 public release includes updates related to the pipeline network, waterway background volume and capacity handling, FTOT Tools and the Scenario Setup Template, the Tableau routes dashboard, and network resilience analysis. The following changes have been made:

• The default FTOT multimodal network has been refreshed to incorporate updated pipeline tariff data for crude and petroleum products effective as of January 2024. Aside from updated tariff costs, some pipeline tariffs that are no longer effective have been removed, and others that are newly effective or have been newly geolocated have been added.
• Updated the optimization problem to enable use of lock volumes, capacity, and volume/capacity ratio attributes. Lock data are applied to adjacent waterway links when available and when capacity and/or background flows are enabled for the water network. Added a “locks” feature class schema to the FTOT network specification and improved documentation to clarify how waterway capacity and background volumes can be applied to the network.
• Renamed the XLSX-based Scenario Setup Template and complementary Scenario Setup Conversion Tool, which provide a user-friendly process for entering scenario data into corresponding FTOT input files. Additional input data validation has been added to confirm required data have been entered. The tool currently does not create required GIS inputs or optional CSV files. Example XLSX workbooks included with the FTOT codebase have been updated. A tutorial video explaining how to fill out the template and run the conversion tool has been published on the FTOT landing page. See the User Guide for more information.
• Updated default values for modal base transport costs in the scenario XML to align with 2020 data published by BTS.
• Redesigned and added CO2 emissions to the Tableau routes dashboard, which is generated when scenarios are run with network density reduction (NDR) turned on.
• Aligned the FTOT rank and removal tool with FTOT version 2024.1. Refactored for usability and added new visualizations to the output HTML report.
• Other updates:
  • Added a BAT file to launch FTOT Tools.
  • Fixed a bug in generating the Tableau routes dashboard in scenario comparison workbooks.
  • Removed the “network used” metric from the Tableau dashboard to ensure consistency across reports.
  • Removed ArcGIS dependency from scenarios that leverage multiprocessing (a potential cause of license-related crashes of FTOT).
  • Removed use of intermediate shapefiles in converting the GIS-based FTOT network to a NetworkX graph.
    See documentation files for additional details.

FTOT 2023_4 Public Release

The FTOT 2023.4 public release includes updates related to scenario input validation, FTOT tools for data preparation and validation, reporting metrics, and network resilience testing. The following changes have been made:

• Input file validation around the facility-commodity CSV files (e.g., rmp.csv, dest.csv) and the schedules CSV file has been streamlined and updated. Validation checks and error messaging warn the user to issues with their scenario input files related to required columns, improper data formats, and mismatches between files. This development continues input validation work done in the 2023.2 public release; it will continue to be expanded in future releases.
• Enhancements to the network validation helper tool, including network connectivity checks and additional summary statistics. For more details, see the User Guide.
• Updated the XLSX-based input data template and complementary FTOT Tool to convert user scenario inputs into corresponding FTOT input files. Additional input data validation has been added to confirm required data have been entered. The tool currently does not create required GIS inputs or optional CSV files. Several example XLSX workbooks are included with the FTOT codebase. See the User Guide for more information.
• Added CO2 emissions to the all_routes CSV file, which is a supplementary report generated for scenarios run with Network Density Reduction (NDR) on. Also added a new database table with the same information provided in this report.
• Corrected reporting metrics for (1) amount of network used to avoid double-counting links used by multiple commodities and (2) processor utilization of commodity flow through a processor facility as a fraction of processor capacity when specified.
• Aligned the FTOT rank and removal tool with FTOT version 2023.1 and updated example scenario in the tool instructions to Reference Scenario 7.
• Minor updates:
  • Corrected a bug in certain versions of ArcGIS Pro preventing the completion of the M2 step (time and commodity mapping).
  • Corrected a bug in assigning CO2 emission factors to artificial links for non-road modes, specifically for use in CO2 optimization scenarios.
  • Moved Tableau template file from the supplementary materials to the “lib” folder in the FTOT code base.
  • Updated the optimization problem formulation in the technical documentation to align with the codebase.
  • Added high-level guidance for how to conduct end-to-end source tracking using database tables.
    See documentation files for additional details.

FTOT 2023_3 Public Release

The FTOT 2023.3 public release includes major updates related to emissions-based optimization, a new North American multimodal network, user preparation of FTOT input files, and supply chain resilience. The following changes have been made:

• Added new functionality to incorporate carbon dioxide (CO2) emissions-related costs into the FTOT route optimization problem. Users can now set what share of (i) impeded transport cost and (ii) CO2 cost to combine in a composite routing cost for route optimization using optional Transport_Cost_Scalar, CO2_Cost_Scalar, and CO2_Unit_Cost elements in the scenario XML. Users can specify the cost per unit of CO2 emissions and separate scaling factors (in a range from 0.0 to 1.0) for the impeded transport cost component and CO2 cost component. The default is to use the full value of impeded transport cost (scaling factor of 1.0) and to zero out CO2 emissions cost (scaling factor of 0.0), which leads to a transportation cost-only optimization approach.
• Created a North American multimodal network by adding available Canadian and Mexican modal data to FTOT’s default contiguous U.S. multimodal network. The draft network includes road, rail, waterway and intermodal facility data for Canada, along with rail and road data for Mexico. The network is designed to facilitate North American scenarios with a scope beyond the continental United States and is available upon request from the FTOT Team. More details are available in Appendix B of the Technical Documentation.
• Built an XLSX-based input data template and complementary FTOT Tool to convert user scenario inputs into corresponding FTOT input files. The FTOT Tool takes an XLSX workbook filled out by the user as input and creates a batch file, a scenario XML, and all facility-commodity CSV files. The tool currently does not create required GIS inputs or optional CSV files. Several example XLSX workbooks have been included with the FTOT codebase. See the User Guide for more information.
• Aligned FTOT-SCR with FTOT version 2023.2, bringing in new FTOT base functionality from the last four releases.
• Updated methodology for calculating CO2 emissions for movements on road to use partial truckloads. All other emissions for movements on road use nearest full truckload. For more details, see the Technical Documentation.
• Minor updates:
  • Corrected a bug related to use of network density reduction (NDR) for scenarios involving multiple processes, a subset of which are candidate generation processes.
  • Corrected a bug in how detailed emissions factors were assigned to road types missing values for either the limited_access or urban attributes.
  • Made the impedance weights CSV file optional. Weights will default to 1.0 if the impedance CSV file is not found, meaning FTOT will not distinguish transport cost on different link types (e.g., local roads versus highways) in the optimization.
  • Corrected a bug in how impedances were assigned to unrecognized link types. If an impedance weights CSV file is provided but a link type in the modal feature class is not recognized, FTOT now correctly applies the maximum weight listed for that mode.
  • Corrected a bug in the Tableau workbook summary graphs for transport cost, network used, vehicle-distance traveled, fuel burn, and CO2 and made other minor fixes and improvements.

See documentation files for additional details.

FTOT 2023_2 Public Release

The FTOT 2023.2 public release includes updates related to scenario input file validation, candidate generation methodology, first-mile/last-mile reporting, and processor capacities. The following changes have been made:

• Added input validation functionality to check that required scenario files exist, required fields are found in each file, and field values are in the correct format. Expanded documentation in the FTOT User Guide and Reference Scenarios Document in coordination with input validation. This functionality will continue to be expanded in future releases.
• Created a network validation tool within the FTOT Tools suite to help users confirm that custom networks follow the network schema and meet other requirements.
• Expanded candidate generation methodology for both NDR on and off cases to support candidate processes with multiple output commodities and difference in allowed modes between candidate process input and output commodities.
• Enhanced reporting for artificial links. Expanded the artificial links report with additional summary metrics for links used in the optimal solution. Added a toggle to the scenario XML so users optionally can include artificial links in summary calculations for the main CSV and text reports. Emissions on all artificial links are calculated using truck specifications to represent typical first/last-mile transport by road.
• In the Tableau workbook, added a “time period” filter to the “By Commodity & Mode” dashboard to better visualize optimal flows in scenarios using schedules. Made other minor changes in Tableau.
• Corrected a few bugs:
  • Liquid capacities and commodity quantities are now handled correctly in the processor facility-commodity CSV file.
  • Capacity rows in the processor facility-commodity CSV file are now able to handle blank values in the commodity quantity field, as expected.
  • Processor candidate slate CSV files generated during candidate generation scenarios correctly implement facility capacity values.

See documentation files for additional details.

FTOT 2023_1 Public Release

The FTOT 2023.1 public release includes a major update with the creation of a general network specification that enables users to run FTOT with a custom multimodal network. This is a breaking change; users will need to update their supplementary data and scenario files (e.g., scenario XML file) in order to use FTOT 2023.1. This release also includes updates related to candidate generation using network density reduction (NDR), commodity-specific minimum and maximum processing capacities, first mile/last mile costs, and minor Tableau enhancements and bug fixes. The following changes have been made:

• Creates an FTOT general network specification schema for each component of the FTOT multimodal network (road, rail, water, locks, pipeline, multimodal facilities). Generalizes FTOT codebase to align with any transportation network that follows the schema.
• Updates FTOT’s default multimodal network to the general network specification. The new default network is named “FTOT_Public_US_Contiguous_Network_v2023.gdb” and is located with the supplementary data and scenarios download. For users bringing custom networks to FTOT, see Section 2.2 in the FTOT User Guide for network schema requirements and Appendix B in FTOT’s technical documentation for the full network specification schema.
• Updates the XML schema to version 7 to align with the new network specification. To run FTOT 2023.1, users must update any pre-existing scenario XML files to this new version. Changes include:
  • New elements for default distance and currency units
  • Relocation of network impedance weights to a new impedance weights CSV file
  • Relocation of CO2 emission factors by specific road type to the detailed emission factors CSV
  • Switch from currency-based short haul penalties to distance-based penalties
  • Updated XML template and corresponding updates to quick start and reference scenario XMLs
• Enables commodity-specific processor capacities through optional “min_capacity” and “max_capacity” columns in the processor CSV file. Users can still apply facility_level capacity by including a row in the CSV file with “total” as the commodity; input or output must be specified in the “io” column for the “total” row. “Max_processor_input” and “min_processor_input” columns are still supported and interpreted as a “total” input row. This change resolves a bug where commodities were being erroneously combined across different units when checking processor capacity.
• Resolves issues with network density reduction (NDR) for candidate processor generation in scenarios with large max transport distances or with a commodity mode CSV file.
• Resolves a bug in the creation of processor flow constraints where an incorrect input/output ratio was being calculated for processors with multiple input commodities and different required quantities.
• Updates transport cost and routing cost on artificial links for all modes to reflect the first mile/last mile costs of traveling by local roads to/from facilities.
• Generalizes metric names in the reports and Tableau workbook to accommodate user-specified distance and currency units. Renamed metrics include transport cost (formerly dollar cost), network used (formerly miles), and vehicle-distance traveled (former vehicle-miles traveled).
• Corrects minor bugs in (1) assigning urban/rural code and limited/nonlimited access information to road edges split by artificial links, (2) removing links in the optimization problem incorrectly coded as bidirectional that should be one-direction only, (3) minimizing offset of facility locations (raw material producers, processors, destinations) on the network, and (4) running NDR scenarios with the pipeline network.
• Releases the “Customizing Scenarios” instruction video. All video tutorials are posted to the FTOT landing page.

See documentation files for additional details.

FTOT 2022_4 public release

The FTOT 2022.4 public release includes updates related to candidate generation using network density reduction (NDR), Tableau and reporting enhancements, detailed reporting on shortest paths between origin and destination facilities, and other minor bug fixes. The following changes have been made:

• Implements a new method for candidate generation scenarios through network density reduction. The new method incorporates NetworkX shortest path algorithms into (1) selection of candidate processor locations and (2) calculation of the optimal routing solution. Running candidate generation using NDR significantly decreases runtime by pre-solving the network for shortest paths; see documentation for details on methodology and known issues.
• Adds metrics for processor capacity and utilization to the text report and CSV report.
• Updates the Tableau workbook with new information on facility sizes, processor efficiencies and utilizations, and other minor aesthetic improvements.
• Expands reporting on shortest path routes between facilities in the "all_routes" CSV report (formerly "optimal_routes") and a new Tableau routes dashboard when NDR is enabled. Reporting allows users to analyze routes across different scenarios, modes, facility pairs, and optimal vs non-optimal routes. The scenario comparison dashboard has also been updated to reflect these enhancements.
• Corrects a minor bug on converting candidate processor build costs to default units.
• Replaces "kgal" with "thousand_gallon" as default liquid units in the template and example scenario XML files.

See documentation files for additional details.