OpenML outage because of the TU/e cyberattack

[joaquinvanschoren]

OpenML went down on Sunday January 12th, 2025, due to a cyber-attack on the Eindhoven University of Technology, which even reached the international press. The OpenML servers were not affected or breached. However, the university took down the entire network out of caution, meaning that the OpenML servers were not accessible, not even to the OpenML system admins. While OpenML is fully containerized, has redundancy against failing servers, and multiple backups, everything was hosted in data centers under the university network, meaning that we could not get access to any of our servers. Below is a list of updates and plans to avoid this from happening in the future.

📢 Updates:

• Sunday January 19th: We gained internal access to our servers again and started setting up a new OpenML instance on a different network to restore access.
• Monday January 20th: We are deploying a read-only instance, which should be available very soon.
• Tuesday January 21th: We have deployed a read-only instance on http://145.38.195.79/. Note that this server has limited functionality (see below). ❤️ Shout out to @PGijsbers and @josvandervelde!
• Wednesday January 22th: The DNS is updated. https://openml.org is accessible again and points to the read-only server. The Python API can be used as usual (R and other APIs should work again as well). There are still SSL issues with https://www.openml.org and https://api.openml.org (used in sklearn.fetch_openml). We'll fix that as soon as possible.
• Thursday January 23rd: Certificates for the subdomains now work.
• Friday January 24th: The production server is accessible again. Most functionality is restored. Remaining known issues are listed below.
• Saturday January 25th: We released openml-python 0.15.1, which allows you to skip downloading parquet files. Update with python -m pip install --upgrade openml.
• Wednesday January 29th: All known issues have now been resolved. If you use the Python API, we recommend clearing your cache.

💻 Use cases:
sklearn.fetch_openml Works :)

from sklearn.datasets import fetch_openml
adult = fetch_openml("adult", version=2)

🐍 OpenML Python:
get_dataset Works :)

import openml
dataset = openml.datasets.get_dataset(1590)

Note

Workarounds specifying environment variable OPENML_SKIP_PARQUET is no longer needed.
In case you experience issues, make sure to try again with a clear cache. If the problem persists, please reach out to us.

Note

Workarounds specifying the ip-based url (http://145.38.195.79/) are no longer needed, and may lead to errors.
In case you experience issues, make sure to try again with a clear cache. If the problem persists, please reach out to us.

Future plans:

• We will set up protocols to very quickly deploy a read-only server in another network to quickly fall back to that server in case anything happens to the university network.
• We will use the lessons learned from this to make sure we always have the OpenML data available outside of the TU/e network.
• We will set up a page with status information and unify communication to better inform the community.

We apologize for the outage and for any inconvenience this may have caused. This was an unprecedented series of events that we did not foresee and had very little control over. We will be better prepared if this ever happens again.

The OpenML team

[brianjester]

I was using a scikit-learn example that stopped working, and used the example on this page to create a workaround, which is provided below:

Code I was attempting to run from scikit-learn:

from sklearn.datasets import fetch_openml
bike_sharing = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True)
df = bike_sharing.frame

Workaround based on example on this page that now downloads:

import openml
openml.config.server = "http://145.38.195.79/api/v1/xml" # Point to the read-only server
bike_sharing = openml.datasets.get_dataset(42712, download_data=True) # version 2 of the dataset
df, *_ = bike_sharing.get_data()

[lesteve]

@joaquinvanschoren honestly there is no rush (we can definitely wait a few more days), but a question out of curiosity and as a scikit-learn maintainer, I was hoping for a work-around in scikit-learn similar to this but this does not work:

from sklearn.datasets import fetch_openml

from sklearn.datasets import _openml
# Point to the read-only server
_openml._OPENML_PREFIX = "http://145.38.195.79/"

bike_sharing = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True)
df = bike_sharing.frame

I get a HTTPError 400:

HTTPError: HTTP Error 400: Bad Request

Here is the curl info:

 ❯ curl 'http://145.38.195.79/data/v1/download/22044627'
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidBucketName</Code><Message>The specified bucket is not valid.</Message><Resource>/v1/download/22044627</Resource><RequestId>181CEF3D5DF63BBE</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>%   

Maybe it's the scikit-learn code that has some assumptions, or relies on something that is not yet working in the read-only server, maybe it's something else, not sure ...

[PGijsbers]

It looks like scikit-learn uses hard-coded path to determine where to fetch the data: https://github.com/scikit-learn/scikit-learn/blob/fef470195d4fcb689bb80c8858d1fcb116e027bc/sklearn/datasets/_openml.py#L40

The correct path on the read-only server is provided by the dataset description url and on the readonly server (currently) looks something like "http://145.38.195.79/data/datasets/0001/1212/dataset_1212.arff" for dataset 1212. However, don't try that right now as I am trying to reconfigure our proxy to work a bit better with minio (it currently doesn't allow for listing calls which breaks the download_all_files functionality of openml-python).

[lesteve]

OK I guess this is an opportunity for scikit-learn code to be improved then 😉 I opened scikit-learn/scikit-learn#30699 to track this (not urgent at all I guess you have bigger fish to fry right now 😉)

[fouodo]

Should R users wait until the primary server is back online?

[sebffischer]

@joaquinvanschoren When I change the server as suggested (and don't pass any API key), I get:

odt(42279)
INFO  [15:16:40.753] Retrieving JSON {url: `http://145.38.195.79/api/v1/json/data/42279`, authenticated: `FALSE`}
Error in curl::curl_fetch_disk(add_auth_string(url, api_key = api_key),  : 
  SSL peer certificate or SSH remote key was not OK: [145.38.195.79] SSL: no alternative certificate subject name matches target ipv4 address '145.38.195.79'

[PGijsbers]

The workaround with the IP adres is no longer needed since we managed to update the domain and we also added certificates. As a consequence, the workaround with the IP adres broke. It very likely works out of the box now (it does for Python), make sure to clear your cache if you continue to see references to the ip adres. If you still experience an issue, let us know and we'll look into it.

[sebffischer]

okay thanks

[sebffischer]

Should the data splits already work?

I get

! Error downloading 'https://openml.org/api_splits/get/359983/Task_359983_splits.arff' (http code: 412, oml code: NA, message: 'failed to perform action generate_folds. Evaluation Engine result send to EMAIL_API_LOG account.'

[joaquinvanschoren]

The production server is back. This should work now. Can you please try again?

[joaquinvanschoren]

For those who want to use image datasets (e.g. MetaAlbum), the cache paths are a bit weird at the moment, but should be normal again after the DNS updates. For now, here's a usage example:

https://colab.research.google.com/drive/1aEZmcmSFFLPFhW7lVAtHGDsbIpqktu5H?usp=sharing

[martinlschumann]

I am getting privacy errors with www.
It seems that the SSL certificate is only valid for https://openml.org and not https://www.openml.org
E.g. the following from https://openml.org/apis (REST API) does not work:
curl -X 'GET' 'https://www.openml.org/api/v1/json/data/3' -H 'accept: application/json'

[joaquinvanschoren]

Confirmed, thanks. We'll fix it as soon as possible.

[PGijsbers]

We believe the issues with the certification for subdomains are resolved now.

[imatiach-msft]

It says in the edited discussion above that the DNS was updated, but I am still seeing errors when calling the fetch_openml api without specifying the server directly in builds (as in, not using server=...):

Wednesday January 22th: The DNS is updated. https://www.openml.org/ is accessible again and points to the read-only server. The Python API can be used as usual (R and other APIs should work again as well). There is still a small issue with the search on the website.

See: https://github.com/microsoft/responsible-ai-toolbox/actions/runs/12795480111/job/36020797462?pr=2594

[hengtanoms]

I'm still having issue when trying to calling fetch_openml:

X, y = fetch_openml("mnist_784", version=1, return_X_y=True, parser="auto", cache=False)

here is the exception:

File /opt/venv/lib/python3.9/site-packages/sklearn/datasets/_openml.py:488, in _load_arff_response(url, data_home, parser, output_type, openml_columns_info, feature_names_to_select, target_names_to_select, shape, md5_checksum, n_retries, delay)
    485     actual_md5_checksum = md5.hexdigest()
    487 if actual_md5_checksum != md5_checksum:
--> 488     raise ValueError(
    489         f"md5 checksum of local file for {url} does not match description: "
    490         f"expected: {md5_checksum} but got {actual_md5_checksum}. "
    491         "Downloaded file could have been modified / corrupted, clean cache "
    492         "and retry..."
    493     )
    495 gzip_file = _open_openml_url(url, data_home, n_retries=n_retries, delay=delay)
    496 with closing(gzip_file):

ValueError: md5 checksum of local file for data/v1/download/52667 does not match description: expected: 0298d579eb1b86163de7723944c7e495 but got fe38182e173709620a43065ffd82bc6b. Downloaded file could have been modified / corrupted, clean cache and retry...

I've checked the xml file https://openml.org/api/v1/data/554 it stated the checksum value is 0298d579eb1b86163de7723944c7e495 but the actual download file checksum is fe38182e173709620a43065ffd82bc6b.

Any idea how we can resolve this? I wasn't able to openml.datasets.get_dataset because I don't have permission to install the openml module (restriction set by online tutorlal jupyter site i'm using), I only allow to use fetch_openml

[joaquinvanschoren]

Hi, see the messages above about fetch_openml. It's currently not yet working again, sorry...
Are you not allowed to run '!pip install openml' in your jupyter notebook?
Can you download the dataset directly (e.g. with curl or wget)? There are .arff and .pq download links in https://openml.org/api/v1/data/554.

[joaquinvanschoren]

fetch_openml is working now 🎉

[hengtanoms]

@joaquinvanschoren very grateful thanks for you help. I can confirm that fetch_openml is working as expected now. Thanks again to you your team for all your hard work for getting opeml server working again :) .

[imatiach-msft]

thanks, I can confirm our tests which use fetch_openml are passing now

[AI-Ahmed]

Hello @joaquinvanschoren,
On colab, I run the following snippet:

#  Satellite dataset https://www.openml.org/d/182
# Switch to the read-only server
openml.config.server = "http://145.38.195.79/api/v1/xml"

# Set the cache dir (for colab)
openml.config.set_root_cache_directory(os.path.expanduser('/content/cache'))

dataset = openml.datasets.get_dataset(182, download_data=True, download_all_files=True) # version 2 of the dataset

I get:

FileNotFoundError: [Errno 2] No such file or directory: '/content/cache/79/195/38/145/datasets/182/description.xml'

SSLError: HTTPSConnectionPool(host='145.38.195.79', port=443): Max retries exceeded with url: /api/v1/xml/data/182 (Caused by SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '145.38.195.79'. (_ssl.c:1006)")))

[AI-Ahmed]

The reason is clear, but I was focusing on reading the comments that contain scripts rather than reading the updates. The today's update declared that there is no need to use server configuration anymore...

Workarounds specifying the ip-based url (http://145.38.195.79/) are no longer needed, and may lead to errors.

[TheRazorace]

Hi everyone and thanks for the support. I am now able to download datasets but still unable to load runs if they are not cached:

Example:
run = openml.runs.get_run(1838227, ignore_cache=True)

Error:
SSLError: HTTPSConnectionPool(host='www.openml.org', port=443): Max retries exceeded with url: /api/v1/xml/run/1838227 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

[joaquinvanschoren]

This works again now.

[PGijsbers]

If you are using openml-python, make sure to upgrade to the latest version (released minutes ago, January 25th, 2025). You can upgrade with python -m pip install --upgrade openml.

With 0.15.1 installed, set environment variable OPENML_SKIP_PARQUET, i.e.,

Linux/MacOS: export OPENML_SKIP_PARQUET=true
Windows: set OPENML_SKIP_PARQUET=true

This stops openml-python from attempting to download parquet files. We recommend to use the feature only until the MinIO production server is operational again.
Without the environment variable set, openml-python is completely functional, but slower.

[TheRazorace]

Hi, just wanted to mention that in Linux, I still have problems with the get_data() function even with the above configuration.

Example:
dataset = openml.datasets.get_dataset(11)
X, y, _, _ = dataset.get_data(dataset_format="dataframe")

Error:
openml.exceptions.OpenMLHashException: Checksum of downloaded file is unequal to the expected checksum 76938608d472f620c170cef9c8c1fa65 when downloading https://openml.org/datasets/0000/0011/dataset_11.arff. Raised when downloading dataset 11.

If I don't use the configuration, even though the dataset is cached, I am still not able to get it.

Error: raise Exception(f"File: {data_file}") from e
Exception: File: /apollo/ida/.cache/openml/org/openml/www/datasets/11/dataset_11.pq

I can do it in MacOS, though, without problems. Is there a reason?

It is not urgent, just reporting it. Thanks

[PGijsbers]

Thanks for the report. That issue is unknown to us. Does it also happen for other datasets? Could you post a full stack trace for both? Please enclose in triple-quoted blocks (```) for formatting. I'll try to have a closer look.

[lesteve]

Looks like a temporary thing maybe (or maybe delete your cache ~/.cache/openml) since the data description seems to point to a different URL https://api.openml.org/data/v1/download/11/balance-scale.arff compared to the one in your error message https://openml.org/datasets/0000/0011/dataset_11.arff.

I can run fine the same snippet:

import openml.datasets
dataset = openml.datasets.get_dataset(11)
X, y, _, _ = dataset.get_data(dataset_format="dataframe")

[TheRazorace]

Thank you both for your replies! I tried again today and it was still the same. I am posting the traceback:

Traceback (most recent call last):
  File "/apollo/ida/Bosch/ml-dataset-recommendation/test.py", line 11, in <module>
    X, y, _, _ = dataset.get_data(dataset_format="dataframe")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 820, in get_data
    data, categorical, attribute_names = self._load_data()
                                         ^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 607, in _load_data
    self._download_data()
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 368, in _download_data
    self.data_file = str(_get_dataset_arff(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/functions.py", line 1312, in _get_dataset_arff
    raise e
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/functions.py", line 1304, in _get_dataset_arff
    openml._api_calls._download_text_file(
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/_api_calls.py", line 275, in _download_text_file
    response = __read_url(source, request_method="get", md5_checksum=md5_checksum)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/_api_calls.py", line 342, in __read_url
    return _send_request(
           ^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/_api_calls.py", line 438, in _send_request
    raise retry_raise_e
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/_api_calls.py", line 401, in _send_request
    raise OpenMLHashException(
openml.exceptions.OpenMLHashException: Checksum of downloaded file is unequal to the expected checksum e013a5e0acb7e141d169198a55b22c51 when downloading https://openml.org/datasets/0000/0004/dataset_4.arff. Raised when downloading dataset 4.

Without the recommended configuration and even though the dataset is cached, I get this:

Traceback (most recent call last):
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 593, in _parse_data_from_pq
    data = pd.read_parquet(data_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/pandas/io/parquet.py", line 667, in read_parquet
    return impl.read(
           ^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/pandas/io/parquet.py", line 274, in read
    pa_table = self.api.parquet.read_table(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 1793, in read_table
    dataset = ParquetDataset(
              ^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 1360, in __init__
    [fragment], schema=schema or fragment.physical_schema,
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 1431, in pyarrow._dataset.Fragment.physical_schema.__get__
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not open Parquet input source '<Buffer>': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/apollo/ida/Bosch/ml-dataset-recommendation/test.py", line 11, in <module>
    X, y, _, _ = dataset.get_data(dataset_format="dataframe")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 820, in get_data
    data, categorical, attribute_names = self._load_data()
                                         ^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 611, in _load_data
    return self._cache_compressed_file_from_file(Path(file_to_load))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 556, in _cache_compressed_file_from_file
    attribute_names, categorical, data = self._parse_data_from_file(data_file)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 586, in _parse_data_from_file
    attribute_names, categorical, data = self._parse_data_from_pq(data_file)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/apollo/ida/miniconda3/envs/exekg/lib/python3.11/site-packages/openml/datasets/dataset.py", line 595, in _parse_data_from_pq
    raise Exception(f"File: {data_file}") from e
Exception: File: /apollo/ida/.cache/openml/org/openml/www/datasets/4/dataset_4.pq

[PGijsbers]

I cannot reproduce this, and both tracebacks point to this being a cache problem (the first almost surely reads a cached description file, the latter reads a corrupted dataset file).
This is a bit nuclear - it will delete your cache - but could you run the following snippet to see if it helps?

python -c "import openml, shutil; shutil.rmtree(openml.config.get_cache_directory())"

[ujohn33]

Hello! Thank you for your work on openml. I wish this resolves quickly and no more attacks happen in the future.

I have tried updating to 15.1 and changed OPENML_SKIP_PARQUET=true
Is there still a workaround if the server it tries downloading pqt file from https://openml1.win.tue.nl/datasets/ ?
Is this issue going to be resolved soon?

The code I use with openml:

    benchmark_suite = openml.study.get_suite(336)  
    task = openml.tasks.get_task(task_id)  # download the OpenML task
    dataset = task.get_dataset()

The warnings and errors:

WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /datasets/0004/44132/dataset_44132.pq
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /datasets/0004/44132/dataset_44132.pq
WARNING:openml.datasets.functions:Could not download file from https://openml1.win.tue.nl/datasets/0004/44132/dataset_44132.pq: HTTPSConnectionPool(host='openml1.win.tue.nl', port=443): Max retries exceeded with url: /datasets/0004/44132/dataset_44132.pq (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
LIGHTGBMLSS
______________________
Traceback (most recent call last):
  File "/rhea/scratch/brussel/105/vsc10528/LSSboost/openml/openml_lssboost_HP_single_run.py", line 238, in <module>
    results = run_single_argument(task_number)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rhea/scratch/brussel/105/vsc10528/LSSboost/openml/openml_lssboost_HP_single_run.py", line 83, in run_single_argument
    X, y, categorical_indicator, attribute_names = dataset.get_data(
                                                   ^^^^^^^^^^^^^^^^^
  File "/rhea/scratch/brussel/105/vsc10528/gpu_friendly/lib/python3.11/site-packages/openml/datasets/dataset.py", line 803, in get_data
    data, categorical, attribute_names = self._load_data()
                                         ^^^^^^^^^^^^^^^^^
  File "/rhea/scratch/brussel/105/vsc10528/gpu_friendly/lib/python3.11/site-packages/openml/datasets/dataset.py", line 594, in _load_data
    return self._cache_compressed_file_from_file(Path(file_to_load))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rhea/scratch/brussel/105/vsc10528/gpu_friendly/lib/python3.11/site-packages/openml/datasets/dataset.py", line 539, in _cache_compressed_file_from_file
    attribute_names, categorical, data = self._parse_data_from_file(data_file)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rhea/scratch/brussel/105/vsc10528/gpu_friendly/lib/python3.11/site-packages/openml/datasets/dataset.py", line 571, in _parse_data_from_file
    raise ValueError(f"Unknown file type for file '{data_file}'.")
ValueError: Unknown file type for file 'None'.

[PGijsbers]

Hi,
Thanks for the kind words. Unfortunately, the code example is incomplete and the task_id is undefined, so we cannot tell which task gives you trouble. It looks like the actual error is produced from a dataset.get_data call, not the provided snippet, and it looks like the error which should have been fixed with the 0.15.1 release. Could you please:

• Ensure that the correct openml-python version is installed? It is possible something went wrong when you tried to upgrade. Find out the version with python -c "import openml; print(openml.__version__)", this should print 0.15.1. If it does not, please install the latest version (python -m pip install --upgrade openml, or similar)
• Provide a minimal example of the code that reproduces the error?