No plot output from Plots with plotly() backend (incompatible with recent Jupyter Notebook Versions 7.0.0-current)

[ajw-free]

[UPDATE 2023-11-09: The problem of plotly() backend not displaying plots within Jupyter Notebook started with the release of Jupyter Notebook version 7.xx (which is installed by IJulia). See my comment further down on 9 November which gives a temporary solution via installation of an older version of Jupyter Notebook, version 6.5.x]

Bug report:

The plotly() backend of Plots does not does not display a plot within Jupyter notebook running on Windows [and MAC as reported below by gmantegna].

The bug is present on MS Windows 11/10 running Julia 1.9.3, IJulia 1.24.2 and Plots 1.39.0 [fresh installation] and Jupyter Notebook 7.04.

The following code displays a blank space instead of the plot.

using Plots
plotly()
display(plot(rand(10)))

If I use the gr() backend, the plot output does display in the notebook.

If I use Plots with plotly() from the command line REPL, it does work - a tab on my browser opens and displays the plot.

Oddly, on my Ubuntu Linux 23.04 installation, running the same version of Julia and packages (but Jupyter Notebook 6.4.12), the plotly() backend does work in Jupyter Notebook. I have not tested on MAC. The problem is however definitely present on Windows - I have tested on three different Windows PCs.

The problem is the same as an earlier issue that was closed on 4 June 2023.
"Not seeing any output after running plotly() with julia 1.9" #1073
and also reported here
JuliaPlots/Plots.jl#4739

[ajw-free]

@stevengj asked me:

What is the value of keys(IJulia.display_dict(display(plot(rand(10)))))? This is what MIME types it is trying to display.

On Windows 11/10 (no plot output in Jupyter Notebook, just a blank space) I get:

KeySet for a Dict{String, Union{String, JSON.Writer.JSONText}} with 1 entry. Keys:
"text/plain"

On my Ubuntu Linux, which does display the plot, the keys(...) output gives the same text as Windows:
KeySet for a Dict{String, Union{String, JSON.Writer.JSONText}} with 1 entry. Keys:
"text/plain"

[ajw-free]

On a Windows installation, I compared the html code created by Jupyter Notebook (which displays nothing) with the html created from the REPL (which displays properly in a browser tab).

To do this, I used savefig() to save to an html file which can be viewed in an editor.

There are differences in the first few lines which suggest that some header information is not being written.

(For gr() the files are the same, as expected.)

To see the differences:

(1) Run this following code via the REPL:
using Plots
plotly()
p=plot(1:10)
savefig(p, "plot_from_REPL.html")
display(p) # works and shows plot in a browser tab.

(2) Run this code via Jupyter Notebook:
using Plots
plotly()
p=plot(1:10)
savefig(p, "plot_from_jupyter.html")
display(p) # shows no display.

RESULTS

(1) File "plot_from_REPL.html" starts with:

etc.

(2) File "plot_from_jupyter.html" starts immediately with:

etc.

Clearly text is missing at the start of the html created from within the notebook.
If one loads the created html file into a browser, no plot is visible.

I hope that someone can assist with fixing this bug!

[gmantegna]

Noting that this problem is also present on a Mac (with Apple Silicon processor), even with fresh installations of everything including IJulia 1.24.2.

[ajw-free]

Today I tried using VSCode with the Julia extension to create a notebook.
The same problem is present in VSCode's Notebook on Windows and on Linux.
I.e. using Plots with plotly() backend does not produce an output.

EDIT: I tried saving the plot object as an html file the saved html file looks the same as that created from the REPL i.e. example (1) in my previous post above. The html file can be viewed in a browser, even though it does not show within VS Code's notebook.

I then installed PlotyJS package via: using Pkg; Pkg.add("PlotlyJS").
Using Plots with plotlyjs() backend worked correctly in VSCode's notebook and displayed a plot.
Unfortunately plotlyjs() backend did not work in Jupyter notebook.

One very strange thing that I noticed in VSCode's notebook was that if I first used plotlyjs() backend to create a plot, and then switched backend to plotly(), the plotly() backend started working!
I then tried the same in Jupyter notebook, but both plotlyjs() and plotly() did not display an output.

EDIT: I have now discovered that if I install PlotlyBase as well as Plots, then using Plots with plotly() backend always works in VS Code. PlotlyBase is the "plot-making guts of PlotlyJS.jl". Its installation must somehow change something when Plots is used with plotly() backend (even though one does not explicitly load PlotlyBase). This does not solve the problem within Jupyter Notebook.

I hope that someone can assist in fixing this bug.
Without plotly() and plotlyjs(), I see no way to create interactive plots using Plots within Jupyter Notebook (on Windows).
(I found another way - use WGLMakie)

[ajw-free]

The problem is also present with other versions of Julia on Windows.

I created fresh installations of Julia LTS v1.6.7 and v1.10.0 beta 3, and then installed IJulia and Plots.

On Windows, Plots with plotly() backend shows no output in Jupyter notebook.

In VS Code's notebook, plotly() backend only shows an output if PlotlyBase is also installed (perhaps this is meant to be so).

I have also reported the issue on the Plots github site:
JuliaPlots/Plots.jl#4827

[ajw-free]

After further investigation, I have concluded that the problem is related to an incompatibility with the current version of Jupyter Notebook that is installed by IJulia.

To see which version of Jupyter Notebook one is using:
Selecting Help -> About Jupyter Notebook

Plotly() backend does not work in the current version of Jupyter Notebook being 7.0.6 (or any version from 7.0.0 onwards)

The last version of the notebook that works with plotly() backend is version 6.5.7 (tested 2024-09-14).
Something went wrong between Notebook release 6.5.x and the next release being 7.0.0.

To reproduce the problem on Windows.
Delete (or rename) the ".julia" folder.
Run Julia. It will be a clean start.

Install packages IJulia and Plots:

using Pkg;
Pkg.add("IJulia");
Pkg.add("Plots")

Then at the Julia prompt, execute:
using IJulia
notebook()

It will then ask the question:
Install Jupiter via conda, y/n? [Y]
Answering yes will then install conda (a mini python) and then also jupyter notebook (newest version, being 7.0.6).

The browser tab opens with Jupyters' filemanager. One can then open a notebook.
If one tries to use Plots with plotly() backend, it does not display a plot.

HERE IS A TEMPORARY SOLUTION:

Use conda to install "jupyter 6.5.x" (the last version that works with plotly() backend) as follows:

 using Pkg
 Pkg.add("Conda")
 using Conda
 Conda.add("notebook=6")  

Now you can try to open a new Jupyter notebook (which should be version 6.x.x) and it should work with Plots and plotly() backend.

Note: You can see all available versions of the notebook package via typing at the julia prompt:

 run(`.julia\\conda\\3\\x86_64\\Scripts\\conda search notebook`)

Now that I have finally found a temporary solution, I hope that someone can make current versions of IJulia+Plots+plotly() work correctly with the current version of Jupyter Notebook.

(The problem may be with Jupyter Notebook or it may have something to do with the html code that is produced from plotly() within the notebook. A saved html file has some missing header information which means it cannot be later viewed in a web-browser - see my comment above on 14 September 2023.)

[ajw-free]

A suggestion and request to IJulia software developers

It is a pity that this issue has not been resolved in the 12 months since I first reported it on Sep 14, 2023.

There is great value in having a mouse-interactive plotting backend for Plots that works in Jupyter notebook. Analysis of waveforms requires interactive zooming to inspect detail. Plots + plotly() backend is superb for interactive data visualisation. For new users, this should simply work.

Currently if one loads IJulia and executes notebook(), the code checks to see if Jupyter is installed. If not, it prompts the user "install Jupyter via Conda, y/n? [y]"
If "Y" is selected, it installs the current version 7.x.x which is not compatible with Plots+plotly() backend.

A temporary fix is to modify the existing IJulia code (script jupyter.jl) to allow installation of Jupyter version 6.x.x.

This could be hard-coded until the incompatibility issue with version 7.x.x is resolved.

i.e. Modify IJulia code to state: Conda.add("notebook=6").

Alternatively the prompt could offer a choice:

(1) Install Jupyter version 6.x.x to ensure compatibility with Plots + plotly() backend.
(2) Install the newest version of Jupyter, being 7.x.x.

[gmantegna]

Tagging @ViralBShah @jakobjpeters @fredrikekre @hyrodium

[ViralBShah]

Suggest perhaps bringing this up on discourse?