quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "54120e63-b53a-4cd5-86ae-e09a70002509",
   "metadata": {},
   "source": [
    "# SSScoring interactive quickstart\n",
    "\n",
    "SSScoring provides analsysis tools for individual or bulk processing of FlySight GPS competition data gathered during speed skydiving training and competition.  Scoring methodology adheres to International Skydiving Commission (ISC), International Speed Skydiving Association (ISSA), and United States Parachute Association (USPA) published competition and scoring rules.  SSScoring libraries and tools also operate with track data files produced by these devices:\n",
    "\n",
    "- FlySight 1\n",
    "- FlySight 2\n",
    "- SkyTrax GPS and barometric device\n",
    "\n",
    "SSScoring leverages data manipulation tools in the pandas and NumPy data analysis libraries.  All the SSScoring code is written in pure Python, but the implementation leverages libraries that may require native code for GPU and AI chipset support like Nvidia and M-chipsets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9feac21-6c23-4fb2-acb8-376cf462b609",
   "metadata": {},
   "source": [
    "---\n",
    "## Requirements\n",
    "\n",
    "- A data lake containing track data files\n",
    "- Some prior Python programming to make changes to this live document or to write new implementations\n",
    "- Knowledge of pandas and NumPy is a nice-to-have, but not a show-stopper\n",
    "\n",
    "To start let's create a data lake and copy one or more of these:\n",
    "\n",
    "- FlySight version 1 files in the form `HH-mm-ss.CSV`\n",
    "- FlySight version 2 data container in the form `YY-MM-dd/HH-mm-ss/*CSV`\n",
    "\n",
    "The `./quickstart` directory will be our data lake.  It could contain zero files or everything on a whole disk.\n",
    "\n",
    "Check if this prerequisite is fulfilled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec6d0d22-4b6f-4c80-9bca-da804a3419cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p \"./quickstart-example\" && if [[ -d \"./quickstart-example\" ]]; then ls -Al \"./quickstart-example\"; fi"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a4923f-a4fe-4691-9cb8-7f4187cf7030",
   "metadata": {},
   "source": [
    "It doesn't matter where the data lake is located.  It could be on the local device, the FlySight device itself (though reading files from it is cumbersome and super slow), a cloud drive, Amazon S3, or anywhere the system may access.  The quickstart data lake already contains a mix of FlySight 1 and 2 files.  Feel free to replace them with your files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eefae148-5271-4052-9090-3958c2ef1637",
   "metadata": {},
   "source": [
    "---\n",
    "## Use case:  scoring all the jumps of a competition in 12 lines of code or less\n",
    "\n",
    "As a speed skydiving judge, I have a directory containing a competitor's FlySight files.  I want to get the score for each jump, the aggregate score, the mean speed, and the maximum speed.  It was a short competition consisting of only 3 jumps."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4347df4c-b3c4-48bc-934e-c2c2a4831115",
   "metadata": {},
   "source": [
    "### Imports constants, and getting all tracks\n",
    "\n",
    "The previous command shows that our data lake contains files of various types, including CSVs.  SSScoring can scan a directory and its subdirectories and returns a list of all valid speed skydiving files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7abb6718-6f66-46ee-b7f0-36168437f6a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.flysight import getAllSpeedJumpFilesFrom\n",
    "\n",
    "import pathlib\n",
    "\n",
    "DATA_LAKE = pathlib.Path('quickstart-example/')\n",
    "\n",
    "allFiles = getAllSpeedJumpFilesFrom(DATA_LAKE)\n",
    "\n",
    "print('\\n'.join((f.as_posix() for f in allFiles.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48e6f8f9-55d7-49d6-bf8f-bd1f7d8b882c",
   "metadata": {},
   "source": [
    "`allFiles` is an ordered dictionary in which each entry has these attributes set:\n",
    "\n",
    "- a `libpath.Path` object associated with the track file\n",
    "- a FlySight version tag set to 1 or 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5f763b9-5430-40ed-9bd5-1d03615d957a",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(allFiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3642b041-8d95-4c29-aa5f-8ba1aa17ee34",
   "metadata": {},
   "source": [
    "`allFiles` includes what looks like a valid FlySight file - the one with the lower case `.csv` extension.  Compatible third-party devices like SkyTrax have a different output format and use a lower case file extension.  That's okay.  SSScoring will process their output as long as the track has the correct data headers and the file extension is CSV or csv."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb285367-d3e0-4bcb-ac09-ed72861f1075",
   "metadata": {},
   "source": [
    "### Get all the results\n",
    "\n",
    "This is super easy--barely an inconvenience.  SSScoring sets AGL == MSL if the caller omits the optional `altitudeDZMeters` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0066bbe3-5b09-4be6-b92e-5fd1936dd3d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import processAllJumpFiles\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=UserWarning) # FNV, conda issue\n",
    "jumpResults = processAllJumpFiles(allFiles, altitudeDZMeters = 616)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "596d6568-f016-4465-9518-e30c4c49fa84",
   "metadata": {},
   "source": [
    "The results are stored in a dictionary that uses a tagged track name as the key, and a **<a href='https://pr3d4t0r.github.io/SSScoring/ssscoring/datatypes.html#JumpResults' target='_blank'>JumpResult object</a>**:\n",
    "\n",
    "Results keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c188e22-8574-458d-8243-186414c78767",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('\\n'.join((r for r in jumpResults.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c011ea5a-7610-440a-b644-d698108473ef",
   "metadata": {},
   "source": [
    "Viewing a specific result can be overwhelming.  This use case is about reporting the results of the competition with as little fuss and as fast as possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "807afe24-a233-4cc0-82ad-84ca1e084527",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import aggregateResults\n",
    "from ssscoring.calc import roundedAggregateResults\n",
    "\n",
    "summary = aggregateResults(jumpResults)\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c170bf6-0058-488c-80dc-05b007986c03",
   "metadata": {},
   "source": [
    "Results can also be rounded, for convenience:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a15dcc67-2eb7-45e9-a3b9-6280cd9a9d2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "roundedAggregateResults(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d467993-9aac-40ab-ac40-0fb1a8604647",
   "metadata": {},
   "source": [
    "And of course, we can resolve final results from the summary table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e8ded5d-58e5-4e9e-9230-b3ece625aeec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import totalResultsFrom\n",
    "totalResultsFrom(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c4775d6-6113-4fa0-90fc-efe6a44494d7",
   "metadata": {},
   "source": [
    "### Putting it all together\n",
    "\n",
    "There was a lot of information to unpack for this use case because the software is doing a lot of work.  In everyday use, however, usage is quite simple.  A complete scoring system can be developed in 10 lines of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b585d91b-646a-4682-99a0-9073de077fa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import aggregateResults\n",
    "from ssscoring.calc import processAllJumpFiles\n",
    "from ssscoring.calc import roundedAggregateResults\n",
    "from ssscoring.flysight import getAllSpeedJumpFilesFrom\n",
    "\n",
    "import pathlib\n",
    "\n",
    "DATA_LAKE = pathlib.Path('quickstart-example/')\n",
    "allFiles = getAllSpeedJumpFilesFrom(DATA_LAKE)\n",
    "jumpResults = processAllJumpFiles(allFiles, altitudeDZMeters = 616)\n",
    "display(aggregateResults(jumpResults))\n",
    "display(totalResultsFrom(aggregateResults(jumpResults)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89bd99bd-b12c-4b04-b570-4685e4ff01f8",
   "metadata": {},
   "source": [
    "If we wanted to get technical the whole implementation can be done in fewer than 5 lines of code.  The import statements do nothing other than make the functions available for processing.  Actual coding starts with the data lake definition."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c0f720-4993-4a37-9fc1-b271bbcb10cf",
   "metadata": {},
   "source": [
    "---\n",
    "## Use case:  analyzing and scoring a single jump"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb9ddb70-e33f-4fb1-adbb-da6bbf9ed1e8",
   "metadata": {},
   "source": [
    "As a speed skydiver in training, I want to process a single speed jump and get the results, with the ability to view the jump score, summary, and all data points throughout the jump."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a140ae7-8719-419c-b666-a63ce2a7e0d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "\n",
    "DATA_LAKE = 'quickstart-example/'\n",
    "jumpFile = pathlib.Path(DATA_LAKE) / 'R1_09-20-26.CSV'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "748b8f2e-2773-4272-a0b2-50a8a2bb5c4c",
   "metadata": {},
   "source": [
    "### Getting data from a single jump"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0b05b39-f390-4ae6-a575-d10c2eb2390f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import getFlySightDataFromCSV\n",
    "\n",
    "rawData, tag = getFlySightDataFromCSV(jumpFile)\n",
    "display(rawData.head(5))\n",
    "display('FlySight data: %s' % tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fde1aca-38a6-44d7-b78f-e231bdf345b7",
   "metadata": {},
   "source": [
    "### FlySight vs SSScoring data representation\n",
    "\n",
    "Track files have a unique general purpose format.  SSScoring transforms the generic FlySight headers and data into dataframes with meaningful metadata and actionable units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e9b993b-1f82-49fa-8366-1ad77a6a72fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import convertFlySight2SSScoring\n",
    "data = convertFlySight2SSScoring(rawData, altitudeDZFt=616) # At Skydive Chicago\n",
    "display(data.head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14251b8f-7d68-4ed2-8cac-374a26fdc6f4",
   "metadata": {},
   "source": [
    "The SSScoring columns representation is more \"programmer friendly\" and uses longer, more descriptive names.  It also calculates the altitude AGL, the ISC speed accuracy value for each data point, and provides SI and Imperial units on the same dataframe.  As a user, feel free to use meters or feet directly from the SSScoring jump data representation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563b13a6-4493-415d-9558-6f931021a89e",
   "metadata": {},
   "source": [
    "### Evaluating the jump"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6033ad70-496b-4f52-8963-4ce0bec16f3e",
   "metadata": {},
   "source": [
    "While the API provides individual functions for validating the jump, separating the skydive from the ground data, separating the speed jump window from the rest of the skydive, and so on, the easiest way to score a jump is to let SSScoring process the jump and return an object with all a dataset of results-related data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b962165-1945-4df7-ab08-419f06a2965b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.calc import processJump\n",
    "from ssscoring.datatypes import JumpStatus\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=UserWarning) # FNV, conda in Jupyter/Lucyfer issue\n",
    "jumpResult = processJump(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c629c6-7b67-402f-903f-bb346a8bd8f0",
   "metadata": {},
   "source": [
    "A <a href='https://pr3d4t0r.github.io/SSScoring/ssscoring/datatypes.html#JumpResults' target='_blank'>`JumpResults` object</a> reports everything there is to know about the jump:\n",
    "\n",
    "- `data` contains the speed skydive data, stripped of all other FlySight data prior to exit or below the `BREAKOFF_ALTITUDE`\n",
    "- `maxSpeed` is the maximum speed attained during the jump\n",
    "- `score` is the 3-second window max speed, per ISC/FAI/USPA rules\n",
    "- `scores` contains all the results for every 3-second window during the speed skydive\n",
    "- `table` has a results table showing the speed, altitude, other data at 5-second intervals\n",
    "- `window` is a <a href='https://pr3d4t0r.github.io/SSScoring/ssscoring/datatypes.html#PerformanceWindow' target='_blank'>`PerformanceWindow`</a> object with data elements for the start, end, and validation altitude\n",
    "- `status` is a <a href='https://pr3d4t0r.github.io/SSScoring/ssscoring/datatypes.html#JumpStatus' target='_blank'>`JumpStatus`</a>  object that reports if a jump was successful or not.  If not, its value describes the issue with the current jump."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3d56687-c5b8-4fd9-898b-83a2c39918a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "if jumpResult.status == JumpStatus.OK:\n",
    "    display('Score = %.2f, max speed = %.2f' % (jumpResult.score, jumpResult.maxSpeed))\n",
    "    display(jumpResult.table)\n",
    "    display(jumpResult.window)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d9a1ddc-711b-477a-ae6e-4ec0ab4f4aff",
   "metadata": {},
   "source": [
    "### Output representation\n",
    "\n",
    "The output in this example is a mishmash of strings.  If using Lucyfer/Jupyter notebooks, the cleanest way to display tabular data is to use dataframes and the built-in `display()` function.  Lucyfer/Jupyter details are outside the scope of this quickstart guide, but here's an example based on the `jumpResult` we have so far:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b20dc191-9448-49d7-968d-dff6e811a59c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "display(pd.DataFrame({ 'Score': [ round(jumpResult.score, 2), ], 'maxSpeed': [ round(jumpResult.maxSpeed, 2), ] }))\n",
    "display(jumpResult.table)\n",
    "display(pd.DataFrame([ jumpResult.window._asdict() ]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34da9ef-a1f0-4f95-bbb4-b3ed6910ece4",
   "metadata": {},
   "source": [
    "---\n",
    "## Intepreting the results\n",
    "\n",
    "SSScoring converts and interprets most speed jump data (new requirements appear every day).  Based on recommendations from many top athletes, one of the most common ways to track progress is to check the performance data on a 1- to 5-second intervals from exit to the end of the speed skydive.  That's how the `table` object came about in the SSScoring `JumpResults`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb591947-e253-41e3-a391-1f1285908e5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(jumpResult.table)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ff3015b-a97d-43ab-8e83-df9679010cf7",
   "metadata": {},
   "source": [
    "Most of the data points are self-explanatory.  Here's how to interpret the other fields:\n",
    "\n",
    "- `speedAngle` is the skydiver's glide angle with respect to the ground\n",
    "- `distanceFromExit` is the absolute distance traveled from exit, across the ground; it uses the `lat`, `lon` from the FlySight data and the Haversine formula for calculating the distance between two points on the surface of a sphere\n",
    "- `netVectorKMh` is the resultant of v-speed and h-speed at every point of the skydive, as reported by the FlySight device, **not** using the ISC/USPA/FAI 3-second window speed calculation\n",
    "\n",
    "### Jump interpretation\n",
    "\n",
    "This jumper had a slow score because the angle of attack to the ground was very shallow.  The jumper traveled almost a whole km from exit across the Earth because they were in a steep track.  The data shows that in future jumps it might be a good idea to dive steeper between 5 and 10 seconds, and maintain an angle of at least 80º.  The horizontal speed should trend to the low double- to single-digits (in km/h) toward the end of the skydive.  The net vector and the v-vector should be close in the last 5-10 seconds of the skydive."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27ffc5bf-ff5d-47ea-af9a-cd7bfbe85a43",
   "metadata": {},
   "source": [
    "---\n",
    "## Plotting the results\n",
    "\n",
    "Plots simplify data visualization.  This quickstart uses the Bokeh framework for speed jump visualization.  SSScoring implements a few convenience functions for handling plot initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89ff9724-a797-4cc0-9ff8-65d35bacc343",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ssscoring.notebook import SPEED_COLORS\n",
    "from ssscoring.notebook import graphAltitude\n",
    "from ssscoring.notebook import graphAngle\n",
    "from ssscoring.notebook import graphJumpResult\n",
    "from ssscoring.notebook import initializeExtraYRanges\n",
    "from ssscoring.notebook import initializePlot\n",
    "import bokeh.models as bm\n",
    "\n",
    "plot = initializePlot(jumpFile.as_posix())\n",
    "plot = initializeExtraYRanges(plot, startY=min(jumpResult.data.altitudeAGLFt)-500.0, endY=max(jumpResult.data.altitudeAGLFt)+500.0)\n",
    "graphAltitude(plot, jumpResult)\n",
    "graphAngle(plot, jumpResult)\n",
    "hoverValue = bm.HoverTool(tooltips=[('Y-val', '@y{0.00}',),])\n",
    "plot.add_tools(hoverValue)\n",
    "graphJumpResult(plot, jumpResult, lineColor=SPEED_COLORS[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "278cbcf6-432f-4b54-8419-6a0de10d204c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}