update dark vessel detection example #351
Conversation
| We can run this UDF a single time to make sure it works: | ||
|
|
||
| ```python showLineNumbers | ||
| single_ais_month = fused.run(read_ais_from_noaa_udf, datestr="2024_09_01") | ||
| ``` | ||
|
|
||
| {/* we have /mnt/cache/ here but /tmp/fused in 3.1 */} |
There was a problem hiding this comment.
This is actually a bit of a confusing part. To give you context:
/tmp/-> When running code locally on your computer/mount/(which used to be/mnt/cache/) is when running on Fused server
We should also mention that we're running some of this code in workbench & some of it in a jupyter notebook, I think that might not be that obvious
There was a problem hiding this comment.
I wrote a sentence towards the end of 3.0 (line 115) that we're doing section 3 locally.
Ok, I played around with local vs udf code and I think I understand now. I will make a note of that in the doc
There was a problem hiding this comment.
In this note in a comment: NOTE: As of Jan 2024, fused.file_path() doesn't work reliably across real time, local & run_remote()
What is unreliable about it?
There was a problem hiding this comment.
This was related to this: https://www.notion.so/fusedio/fused-file_path-has-inconsistent-behaviour-on-local-real-time-and-batch-173899d3b76380068339ec755bbc5b8f?pvs=4
We just need to explain better how fused.file_path() works
But I think this should also be done in the top level functions
| ::: | ||
|
|
||
| Fused provides a simple way to do this with the [ingestion process](/core-concepts/data_ingestion/). Our ingestion pipeline will then slice the dataset in order to make | ||
| {/* not sure what this sentence was going for... Our ingestion pipeline will then slice the dataset in order to make */} |
There was a problem hiding this comment.
Oh good catch, let's remove
| @@ -362,9 +338,11 @@ We're going to use the [Microsoft Planetary Computer Sentinel-1 Ground Range Det | |||
|
|
|||
| Let's start with a basic UDF just returning our area of interest: | |||
|
|
|||
| {/* why was this function named s1_udf */} | |||
There was a problem hiding this comment.
An incomplete copy + paste! Good catch
Feel free to remove this comment too :)
Co-authored-by: Maxime Lenormand <44133329+MaxLenormand@users.noreply.github.com>
Co-authored-by: Maxime Lenormand <44133329+MaxLenormand@users.noreply.github.com>
sterfd
force-pushed
the
example_dark_vessel_updates
branch
from
bc72161 to
0aa075e
Compare
sterfd
changed the title
Co-authored-by: Maxime Lenormand <44133329+MaxLenormand@users.noreply.github.com>
Co-authored-by: Maxime Lenormand <44133329+MaxLenormand@users.noreply.github.com>
| - [`fused`](/python-sdk/#install) | ||
| - [`pandas`](https://pandas.pydata.org/) | ||
| - [`geopandas`](https://geopandas.org/en/stable/index.html) | ||
| - [`notebook`](https://pypi.org/project/notebook/) |
There was a problem hiding this comment.
That's not really a requirement :)
| - [`notebook`](https://pypi.org/project/notebook/) |
We can add here that we'll be running things through a notebook so recommend people have Jupyter notebooks of some sort, but we don't directly use notebook (unless you added it?)
|
|
||
| :::note | ||
| Fused UDFs by default run on serverless instances, so their local storage changes at every run. To keep data persistent across runs we use shared mounted storage across all the instances of your team | ||
| ::: | ||
|
|
||
| `fused.file_path()` returns the mount path of any file we'd like to create | ||
| `fused.file_path()` returns the mount path of any file we'd like to create. When run locally, it returns the local `/tmp/fused/` directory on your machine. |
There was a problem hiding this comment.
This is a wonderful explanation! Great use of the tabs
We can leave this here for now, but we should (in a next PR) move this to its own dedicated page as this is useful not only in this example but for every use case
There was a problem hiding this comment.
Turned this into a new card
No description provided.