How to sort or reorder columns selected with selectors

[ozgurkalan]

Hi there,

How to sort columns in this case:
(t .select( s.startswith( ("H", "J") ) ) )

[lostmygithubaccount]

you could re-select and sort the columns:

t = t.select(sorted(t.columns))

and replace the sorted function with custom logic as needed

[cpcloud]

@ozgurkalan Can you share your code so that we might help you figure out what's slow?

[ozgurkalan]

No worries, previous works. It was exactly as below:
appr. 500k records with 59 cols
(t .select(sorted(t.columns)) # this is added with your recommendation and works fast, in miliseconds .select(sel.startswith(("H","J"))) .select(sorted(_.columns)) # this is still working at 5th minute and not responding yet .order_by("H1") .distinct() )

[ozgurkalan]

by the way, a shape-like function would be excellent, I need to t.to_pandas().shape to get the shape, if there is any better way, really appreciate

[cpcloud]

I can make an issue for a shape property.

The cheapest way to compute shape of an arbitrary table expression is:

def shape(expr) -> tuple[int, int]:
	return int(expr.count().execute()), len(expr.columns)

Note that expr.count() may not always be fast, but it will be much faster and more memory efficient than expr.to_pandas().shape.

[cpcloud]

No worries, previous works. It was exactly as below: appr. 500k records with 59 cols (t .select(sorted(t.columns)) # this is added with your recommendation and works fast, in miliseconds .select(sel.startswith(("H","J"))) .select(sorted(_.columns)) # this is still working at 5th minute and not responding yet .order_by("H1") .distinct() )

_.columns is not doing what you think it does: sorted(_.columns) is going to run forever, because it uses __getitem__ to access elements of a list. I'll put up a PR to make this fail fast.

To access .columns you may need to create a variable to hold the expression until we have a way to reorder columns with a relocate selector.