Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provide some IPython implementations #956

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions src/snowflake/snowpark/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,35 @@
WhenNotMatchedClause,
)
from snowflake.snowpark.window import Window, WindowSpec


def register_sql_magic():
try:
Comment on lines +72 to +73
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you describe what is the purpose of this function or point me to some documentation I can follow

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the purpose of this function is that when an import is done it will run init.py and call register_sql_magic. This will function will try to gather the ipython environment and if found it will register the sql magic

import IPython
from IPython.core.magic import register_cell_magic

Check warning on line 75 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L75

Added line #L75 was not covered by tests

def sql(line, cell):
user_ns = IPython.get_ipython().user_ns

Check warning on line 78 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L77-L78

Added lines #L77 - L78 were not covered by tests
if "session" in user_ns:
session = user_ns["session"]

Check warning on line 80 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L80

Added line #L80 was not covered by tests
Comment on lines +78 to +80
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does user_ns get populated with objects? how do we know that snowflake.snowpark.Session will always be mapped to "session"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't know before hand, that why we check and if it is not there we notify the user. We can iterate thru all the objects until we find a session object as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we would either need to document that session object must be named session and no other object must be named session, else these changes won't work

# the following will allow using variables
# for example session.sql("select {variable1}")
formatted_sql = cell.format(**user_ns)
name = None

Check warning on line 84 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L83-L84

Added lines #L83 - L84 were not covered by tests
if line and line.strip():
name = line.strip().split(" ")[0]
sfc-gh-mrojas marked this conversation as resolved.
Show resolved Hide resolved
df = session.sql(formatted_sql)

Check warning on line 87 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L86-L87

Added lines #L86 - L87 were not covered by tests
if name:
user_ns[name] = df

Check warning on line 89 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L89

Added line #L89 was not covered by tests
else:
user_ns["__df"] = df
return df

Check warning on line 92 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L91-L92

Added lines #L91 - L92 were not covered by tests
else:
return "No session was found. Define a Snowpark Session object with the name 'session'"

Check warning on line 94 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L94

Added line #L94 was not covered by tests

register_cell_magic(sql)

Check warning on line 96 in src/snowflake/snowpark/__init__.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/__init__.py#L96

Added line #L96 was not covered by tests
except Exception:
pass


register_sql_magic()
43 changes: 43 additions & 0 deletions src/snowflake/snowpark/dataframe.py
Original file line number Diff line number Diff line change
Expand Up @@ -3772,6 +3772,49 @@
exprs = [convert(col) for col in parse_positional_args_to_list(*cols)]
return exprs

# class property to control the number of rows to diplay
__rows = 50
Comment on lines +3775 to +3776
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this naming convention followed by ipython variables? I think there should be a better way to set this property because users should not be accessing variables that start with _.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I was just thinking on a way to expose a mechanism for setting the number of rows to display by default. Maybe a more explicit value in the session object for example will be better.


def _repr_html_(self):
"""Render a rich DataFrame representation in HTML format.

This method is used for pretty printing DataFrame in Jupyter
notebook or similar environments that support HTML rendering."""
import IPython
import pandas as pd

Check warning on line 3784 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3783-L3784

Added lines #L3783 - L3784 were not covered by tests
Comment on lines +3783 to +3784
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on when this function is called and what happens when pandas is not available

For a safe way to import pandas, you can follow

from snowflake.connector.options import installed_pandas, pandas

we should have dependencies well documented when users want to use IPython magics. My guess is that IPython is always installed when running inside any IPython notebook.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes IPython will always be available in jupyter. For the pandas... yes we should check for that. I see know that there is an existing mechanism for checking pandas presence. I think I can use that and if not just use df.show

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this function is only called by IPython notebook

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


rows_limit = DataFrame.__rows

Check warning on line 3786 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3786

Added line #L3786 was not covered by tests
if "display" in globals():
display = globals()["display"]

Check warning on line 3788 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3788

Added line #L3788 was not covered by tests
elif "display" in IPython.get_ipython().user_ns:
display = IPython.get_ipython().user_ns["display"]

Check warning on line 3790 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3790

Added line #L3790 was not covered by tests
else:
from IPython.display import display
try:
count = self.count()

Check warning on line 3794 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3792-L3794

Added lines #L3792 - L3794 were not covered by tests
if count == 0:
return "No rows to display"

Check warning on line 3796 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3796

Added line #L3796 was not covered by tests
elif count == 1:
df = pd.DataFrame.from_records([x.as_dict() for x in self.collect()])
elif count > rows_limit:
print(

Check warning on line 3800 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3800

Added line #L3800 was not covered by tests
f"There are {count} rows. Showing only {rows_limit}. Change DataFrame.__rows_limit value to display more rows"
)
Comment on lines +3801 to +3802
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
f"There are {count} rows. Showing only {rows_limit}. Change DataFrame.__rows_limit value to display more rows"
)
f"There are {count} rows. Showing only {rows_limit}. Change DataFrame.__rows value to display more rows"
)

the class variable is called __rows

df = self.limit(rows_limit).to_pandas()

Check warning on line 3803 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3803

Added line #L3803 was not covered by tests
else:
df = self.to_pandas()
display(df)
return ""
except Exception as ex:
return str(ex)

Check warning on line 3809 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3805-L3809

Added lines #L3805 - L3809 were not covered by tests
Comment on lines +3808 to +3809
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is the exception displayed on the client side. Is it just a cell with error message?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will just displayed in the cell output. Would you like to edit the traceback?


def _ipython_key_completions_(self) -> List[str]:
"""Provide custom key completions for the DataFrame in IPython or Jupyter Notebook.

Comment on lines +3811 to +3813
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a simple unit test can be added here

This method is used to supply additional keys for tab-completion support in
IPython or Jupyter Notebook when interacting with the DataFrame."""
return self.columns

Check warning on line 3816 in src/snowflake/snowpark/dataframe.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/dataframe.py#L3816

Added line #L3816 was not covered by tests

where = filter

# Add the following lines so API docs have them
Expand Down
18 changes: 18 additions & 0 deletions src/snowflake/snowpark/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -2284,4 +2284,22 @@
else default_value
)

def __repr__(self):
"""
Return a string representation of the Session object."""
snowflake_environment = self.sql("select current_version()").collect()
return (

Check warning on line 2291 in src/snowflake/snowpark/session.py

View check run for this annotation

Codecov / codecov/patch

src/snowflake/snowpark/session.py#L2290-L2291

Added lines #L2290 - L2291 were not covered by tests
sfc-gh-mrojas marked this conversation as resolved.
Show resolved Hide resolved
"\n"
+ f"role : {self.get_current_role()}\n"
+ f"warehouse : {self.get_current_warehouse()}\n"
+ f"database : {self.get_current_database()}\n"
+ f"schema : {self.get_current_schema()}\n"
+ f"db.version : {snowflake_environment[0][0]}\n"
+ f"snowpark.version : {get_version()}\n"
+ f"os.name : {get_os_name()}\n"
+ f"python.connector.version : {get_connector_version()}\n"
+ f"python.connector.session.id : {self._session_id}\n"
+ f"query_tags : {self.query_tag}"
)

createDataFrame = create_dataframe
Loading