-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
provide some IPython implementations #956
base: main
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -67,3 +67,35 @@ | |
WhenNotMatchedClause, | ||
) | ||
from snowflake.snowpark.window import Window, WindowSpec | ||
|
||
|
||
def register_sql_magic(): | ||
try: | ||
import IPython | ||
from IPython.core.magic import register_cell_magic | ||
|
||
def sql(line, cell): | ||
user_ns = IPython.get_ipython().user_ns | ||
if "session" in user_ns: | ||
session = user_ns["session"] | ||
Comment on lines
+78
to
+80
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. how does user_ns get populated with objects? how do we know that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't know before hand, that why we check and if it is not there we notify the user. We can iterate thru all the objects until we find a session object as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we would either need to document that session object must be named |
||
# the following will allow using variables | ||
# for example session.sql("select {variable1}") | ||
formatted_sql = cell.format(**user_ns) | ||
name = None | ||
if line and line.strip(): | ||
name = line.strip().split(" ")[0] | ||
sfc-gh-mrojas marked this conversation as resolved.
Show resolved
Hide resolved
|
||
df = session.sql(formatted_sql) | ||
if name: | ||
user_ns[name] = df | ||
else: | ||
user_ns["__df"] = df | ||
return df | ||
else: | ||
return "No session was found. Define a Snowpark Session object with the name 'session'" | ||
|
||
register_cell_magic(sql) | ||
except Exception: | ||
pass | ||
|
||
|
||
register_sql_magic() |
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -3772,6 +3772,49 @@ | |||||||||
exprs = [convert(col) for col in parse_positional_args_to_list(*cols)] | ||||||||||
return exprs | ||||||||||
|
||||||||||
# class property to control the number of rows to diplay | ||||||||||
__rows = 50 | ||||||||||
Comment on lines
+3775
to
+3776
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this naming convention followed by ipython variables? I think there should be a better way to set this property because users should not be accessing variables that start with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, I was just thinking on a way to expose a mechanism for setting the number of rows to display by default. Maybe a more explicit value in the session object for example will be better. |
||||||||||
|
||||||||||
def _repr_html_(self): | ||||||||||
"""Render a rich DataFrame representation in HTML format. | ||||||||||
|
||||||||||
This method is used for pretty printing DataFrame in Jupyter | ||||||||||
notebook or similar environments that support HTML rendering.""" | ||||||||||
import IPython | ||||||||||
import pandas as pd | ||||||||||
Comment on lines
+3783
to
+3784
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you elaborate on when this function is called and what happens when pandas is not available For a safe way to import pandas, you can follow
we should have dependencies well documented when users want to use IPython magics. My guess is that IPython is always installed when running inside any IPython notebook. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes IPython will always be available in jupyter. For the pandas... yes we should check for that. I see know that there is an existing mechanism for checking pandas presence. I think I can use that and if not just use df.show There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess this function is only called by IPython notebook There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes, it is a ipython method: https://ipython.readthedocs.io/en/stable/config/integrating.html |
||||||||||
|
||||||||||
rows_limit = DataFrame.__rows | ||||||||||
if "display" in globals(): | ||||||||||
display = globals()["display"] | ||||||||||
elif "display" in IPython.get_ipython().user_ns: | ||||||||||
display = IPython.get_ipython().user_ns["display"] | ||||||||||
else: | ||||||||||
from IPython.display import display | ||||||||||
try: | ||||||||||
count = self.count() | ||||||||||
if count == 0: | ||||||||||
return "No rows to display" | ||||||||||
elif count == 1: | ||||||||||
df = pd.DataFrame.from_records([x.as_dict() for x in self.collect()]) | ||||||||||
elif count > rows_limit: | ||||||||||
print( | ||||||||||
f"There are {count} rows. Showing only {rows_limit}. Change DataFrame.__rows_limit value to display more rows" | ||||||||||
) | ||||||||||
Comment on lines
+3801
to
+3802
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
the class variable is called |
||||||||||
df = self.limit(rows_limit).to_pandas() | ||||||||||
else: | ||||||||||
df = self.to_pandas() | ||||||||||
display(df) | ||||||||||
return "" | ||||||||||
except Exception as ex: | ||||||||||
return str(ex) | ||||||||||
Comment on lines
+3808
to
+3809
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. how is the exception displayed on the client side. Is it just a cell with error message? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it will just displayed in the cell output. Would you like to edit the traceback? |
||||||||||
|
||||||||||
def _ipython_key_completions_(self) -> List[str]: | ||||||||||
"""Provide custom key completions for the DataFrame in IPython or Jupyter Notebook. | ||||||||||
|
||||||||||
Comment on lines
+3811
to
+3813
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. a simple unit test can be added here |
||||||||||
This method is used to supply additional keys for tab-completion support in | ||||||||||
IPython or Jupyter Notebook when interacting with the DataFrame.""" | ||||||||||
return self.columns | ||||||||||
|
||||||||||
where = filter | ||||||||||
|
||||||||||
# Add the following lines so API docs have them | ||||||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you describe what is the purpose of this function or point me to some documentation I can follow
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the purpose of this function is that when an import is done it will run init.py and call register_sql_magic. This will function will try to gather the ipython environment and if found it will register the sql magic