Skip to content

Conversation

motus
Copy link
Member

@motus motus commented Sep 18, 2025

Pull Request

Title

Handle the case when a script invoked by local_env fails silently (i.e., with error code 0) and produces an empty metrics CSV file.


Description

  • check if the metrics input file is empty;
  • make sure the input CSV file contains all required metrics columns.
  • add unit test for the case
  • make local env unit tests a bit more flexible to check other cases like this one

Type of Change

  • 🛠️ Bug fix

Testing

added a new unit test to cover the case

Closes #998

@motus motus self-assigned this Sep 18, 2025
@motus motus requested a review from a team as a code owner September 18, 2025 23:35
@Copilot Copilot AI review requested due to automatic review settings September 18, 2025 23:35
@motus motus added bug Something isn't working tests Add or fix unit tests mlos-bench labels Sep 18, 2025
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR handles edge cases where benchmark scripts produce empty CSV files despite returning successful exit codes. The changes ensure robust handling of empty metrics files and improve test infrastructure to verify different failure scenarios.

  • Added validation to check for empty CSV files and mark trials as failed
  • Enhanced test framework to support flexible status checking for different test scenarios
  • Fixed column specification in DataFrame creation for optimizer

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
local_env.py Added check for empty CSV files and return FAILED status
init.py Enhanced test helper function to support flexible status expectations
local_env_test.py Added new test case for empty CSV file scenario
local_env_vars_test.py Updated test call to use keyword arguments
mlos_core_optimizer.py Fixed DataFrame creation with explicit column specification

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@motus motus force-pushed the sergiym/storage/null_csv branch from a9c7ac1 to b30ba63 Compare September 18, 2025 23:39
@motus motus enabled auto-merge (squash) September 18, 2025 23:41
@motus motus added the ready for review Ready for review label Sep 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working mlos-bench ready for review Ready for review tests Add or fix unit tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mark trial as failed when local benchmark script produces no results
1 participant