Spark Connect display shows generic column names (Field 1, Field 2) instead of actual names

[j1wonpark]

What happens?

When using Spark Connect, query results display generic column names like Field 1, Field 2 instead of the actual column names (e.g., name, age, city).

This is caused by PrettyTable.clear() being called in ResultSet.fetchmany() and ResultSet.fetchall() on the Spark path. clear() removes both rows and field_names, so when rows are re-added, PrettyTable auto-generates generic field names. The fix is to use clear_rows() instead, which only removes rows and preserves field_names.

Root cause: src/sql/run/resultset.py lines 432 and 458 call self._pretty_table.clear() instead of self._pretty_table.clear_rows().

To Reproduce

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()

%load_ext sql
%sql spark

%sql SELECT * FROM my_table
# Expected: columns show actual names (e.g., name, age, city)
# Actual: columns show Field 1, Field 2, Field 3

Environment

OS: macOS

JupySQL Version: 0.11.2dev

Identity Disclosure

Full Name: Jiwon Park

Affiliation: Individual