Support pandas 2

[tung-vu-td]

Couple of sanity tests I've done for pandas 2 support:

Setup

import numpy as np; import pandas as pd
def fake_data(n):
   users = np.random.choice([0., 1., 2.], (n, 1))
   items = np.random.choice([0., 1., 2.], (n, 1))
   weight = np.random.rand(n,1)
   return np.concatenate((users, items, weight), axis=1)

d1 = fake_data(10_000_000)
df = pd.DataFrame(d1, columns=["users", "items", "scores"])

import pytd; import os
client=pytd.Client(database="sample_datasets", apikey=os.environ["TD_API_KEY"])

Upload to TD table

client.load_table_from_dataframe(df, "tung_db.pytd_test", writer="bulk_import", if_exists="overwrite", fmt="msgpack", max_workers=6, chunk_record_size=1_000_000)

Results: table is successfully imported to TD

Query TD table

result = client.query('select symbol, count(1) as cnt from nasdaq group by 1 order by 1')
df2 = pd.DataFrame(**result)
print(df2)

Result:

     symbol   cnt
0      AAIT   590
1       AAL    82
2      AAME  9252
3      AAOI   253
4      AAON  5980
...     ...   ...
2827   ZNGA   698
2828   ZOOM  6021
2829   ZSPH    71
2830     ZU   217
2831   ZUMZ  2364

[2832 rows x 2 columns]

Read through pandas 2.0.0 release notes

https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html

Seems like the only change that can potentially break backward compatibility is Construction with datetime64 or timedelta64 dtype with unsupported resolution

Currently, we only support datetime64[ns] for time column. In pandas 1, if pytd users specify a wrong resolution for datetime64, for example, datetime64[s], they can still convert and import the column correctly. But in pandas 2, they won't be able to convert and import the time column

[chezou]

I think testing over two versions would be enough, i.e., 1.5.3 and 2.2.2. Is there any specific reason to test for 1.3.5, 1.4.4, 2.0.3, and 2.1.4?

[tung-vu-td]

No specific reason. I'm just following current convention (1.3, 1.4, 1.5)

If you think it's uneccessary, I'll test major versions only. I think testing minor version has limited benefit as well (unless there is some bug in one of pandas versions). Testing only 2 versions will drastically improve test time (now close to 1 hour 😬 )

[chezou]

Okay, then let's test 2 major versions only.

[tung-vu-td]

Updated in cad4317