Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syncing dev and main branches #206

Merged
merged 13 commits into from
Nov 27, 2024
Merged

Syncing dev and main branches #206

merged 13 commits into from
Nov 27, 2024

Conversation

rafaspadilha
Copy link
Contributor

@rafaspadilha rafaspadilha commented Nov 25, 2024

This PR pulls the changes made in dev branch since last sync (May 2024) to the main branch.

farmvibes-ai-cd and others added 12 commits July 5, 2024 23:09
Co-authored-by: Alex Crown <acrown@microsoft.com>
Co-authored-by: Bruno Silva <brunosilva@microsoft.com>
Co-authored-by: Eduardo Rodrigues <edrodrigues@microsoft.com>
Co-authored-by: Jessica Wolk <Jessica.Wolk@microsoft.com>
Co-authored-by: Leonardo Nunes <lnunes@microsoft.com>
Co-authored-by: Naga Bilwanth Gangarapu <Naga@zensa.co>
Co-authored-by: Rafael Padilha <rpadilha@microsoft.com>
Co-authored-by: Renato Luiz de Freitas Cunha <renato.cunha@microsoft.com>
Co-authored-by: Rick Gutierrez <ricardgu@microsoft.com>
Co-authored-by: Roberto de Moura Estevão Filho <robertode@microsoft.com>
Co-authored-by: Roberto Santos <robsantos@microsoft.com>
Co-authored-by: Sara Malvar <saramalvar@microsoft.com>
Add workflow for linting, static type checking, and testing.
- Check each package individually
- Check ops and integration tests
- Check docstrings for `vibe_core`
This code changes are enhancements for deepmc. It help to find the
weather forecast for stations having no historical data or missing
sensor data. It will add new model created using Pytorch Graphical
Neural Network (GNN).

The code changes also having enhancements to existing deepmc scripts
that add datetime in preprocess output. It help to find data belong to
which date while performing GNN model training.

Co-authored-by: Naga Bilwanth Gangarapu <Naga@zensa.co>
Fix for #173.

Planetary Computer does not uses API key anymore. Some of our notebooks are failing because of that (and because the secret has not been added to the cluster).
… raster's boarder (#183)

In the SpaceEye workflow, if the input geometry yields a window that is
smaller than the chip size, we adjust it by splitting the difference
between the chip's and window's width (height) half way to the
right/left (top/bottom) sides of the original window. However if the
input geometry falls close to the border of the raster, we might not be
able to completely increase the window in one of the sides and end up
with a window smaller than the chip size. Because of that, the workflow
fails with "_dim_size cannot be smaller than chip_size. Please consider
reducing the step/chip size or increasing the input geometry._"

This PR updates the `_adjust_roi_window` method, so the window is
adjusted from both sides until reaching the chip size or reaching the
border of the raster.

This fixes #180.
This PR replaces the `ChipWindow` named tuple with a tuple type alias.
This fixes some serialization/deserialization errors that caused the
workflow to break.

Additionally, this PR modifies how we scale and offset the raster values
during normalization before SAM image encoder. After applying the
raster's scale and offset normalization on the RGB bands, we now clip
the values to the range [0,1] before multiplying them by 255. The lack
of clipping operation was leading to inconsistencies in the segmentation
masks outputs.
Add workflow to build and push `dev` and `main` container images.
Workflow that retags images, changes tags in the code, and opens a PR to
main

---------

Co-authored-by: Renato L. de F. Cunha <renato.cunha@microsoft.com>
Create a cluster with local images and run some tests on it
This PR adds a copyright header and license information to all FarmVibes.AI source files.
Add workflow to build and push base images, and then update tags on
relevant files.
If the user does not have [Git LFS](https://git-lfs.com/) installed when
cloning the repo, some of the large files in the repository will not be
available. This causes some workflows to break, due to reading one of
these missing files.

This PR adds Git LFS as one of the requirements for setting up the repo,
and update the Makefile to pull files when building the cluster with
local files (`make local` and `make local-worker`).

---------

Co-authored-by: Renato L. de F. Cunha <renato.cunha@microsoft.com>
Copy link
Contributor

@renatolfc renatolfc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

…207)

This PR fixes an issue in which cluster setup would fail with an `Error:
could not download chart: invalid_reference: invalid tag` error message
when installing rabbitmq. This is related to [this issue in bitnami
repo](bitnami/charts#30582), and fixes issue
#203.

This PR also updates the redis and rabbitmq image tags to the latest
versions to ensure compatibility with the latest dependencies and
security patches.
@rafaspadilha rafaspadilha merged commit 241cb09 into main Nov 27, 2024
20 checks passed
rafaspadilha added a commit that referenced this pull request Nov 29, 2024
renatolfc pushed a commit that referenced this pull request Nov 29, 2024
Reverting previous two commits on `main` branch. Will trigger release
pipeline correctly this time.
renatolfc added a commit that referenced this pull request Nov 29, 2024
This re-releases #206 and #211 now using the release pipeline, which
also updates the prod images in the Container Registry.

----

This PR marks the full release of FarmVibes.AI's source code with an
open source license.

This PR also introduces several significant updates to the development
environment, build workflows, and continuous integration processes. The
most important changes include the addition of a development container
configuration, updates to GitHub workflows for building and testing, and
the introduction of new settings for code linting and formatting.

### Development Environment Updates:
*
[`.devcontainer/devcontainer.json`](diffhunk://#diff-24ad71c8613ddcf6fd23818cb3bb477a1fb6d83af4550b0bad43099813088686R1-R62):
Added a new devcontainer configuration with custom VSCode settings and
extensions, Docker mounts, and environment variables.
*
[`.devcontainer/post-create.sh`](diffhunk://#diff-bcf66a848607b49d2a45edfad434f6808017ee9d18f5a20dde6670745010d89aR1-R71):
Added a post-create script to set up the development environment,
including SSH configuration, Python virtual environment setup, and
installation of necessary tools and dependencies.

### Build and Deployment Workflows:
*
[`.github/workflows/base-build.yml`](diffhunk://#diff-034a91a8f3709fafd1219e587ce389e3a39338ac6adbe543d524bcb1e84b2cf3R1-R64):
Added a new workflow for building and pushing base Docker images,
including steps for Azure CLI login and updating image tags.
*
[`.github/workflows/docker-build.yml`](diffhunk://#diff-3414847e2ad632333f775cabb810f0dc0df61a570365df34750a08b00912fe82R1-R41):
Introduced a workflow for building and pushing service Docker images on
push events to the `dev` and `main` branches.
*
[`.github/workflows/release-to-main.yml`](diffhunk://#diff-b55efbebc7173bd35e8aa625b689d11a521f0ea54f064ae7a2be9cbf5c5fa1a4R1-R54):
Added a workflow to handle the release process, including retagging
Docker images and updating the default image tag in the codebase.

### Continuous Integration:
*
[`.github/workflows/lint-test.yml`](diffhunk://#diff-1dccad827a8e5157df1dcdb70b73e454524c2586ebc64576af82cb348d478892R1-R197):
Created a comprehensive workflow for linting, type checking, and testing
the codebase across multiple packages and environments.

### Code Linting and Formatting:
*
[`.ruff.toml`](diffhunk://#diff-b3236d4cb260ca6acce083493fcdc42816f732fc3d5aefb263a3c3fe9db4f81fR1-R17):
Added configuration for the `ruff` linter, specifying linting rules,
file exclusions, and formatting preferences.

These changes aim to ensure FarmVibes.AI's longevity, to streamline the
development process, ensure consistent code quality, and automate the
build and deployment pipelines.

---------

Co-authored-by: FarmVibes Release Pipeline <farmvibesaicd@microsoft.com>
Co-authored-by: Alex Crown <acrown@microsoft.com>
Co-authored-by: Bruno Silva <brunosilva@microsoft.com>
Co-authored-by: Eduardo Rodrigues <edrodrigues@microsoft.com>
Co-authored-by: Jessica Wolk <Jessica.Wolk@microsoft.com>
Co-authored-by: Leonardo Nunes <lnunes@microsoft.com>
Co-authored-by: Naga Bilwanth Gangarapu <Naga@zensa.co>
Co-authored-by: Rafael Padilha <rpadilha@microsoft.com>
Co-authored-by: Rick Gutierrez <ricardgu@microsoft.com>
Co-authored-by: Roberto de Moura Estevão Filho <robertode@microsoft.com>
Co-authored-by: Roberto Santos <robsantos@microsoft.com>
Co-authored-by: Sara Malvar <saramalvar@microsoft.com>
Co-authored-by: Naga Bilwanth Gangarapu <82965480+v-ngangarapu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants