330 improve spark integration with notebook (#331)

* Add Spark session creation button to NotebookToolbar component

* Refactor hover styles in NotebookToolbar component to use CSS-in-JS approach for SparkIcon, improving code readability and maintainability.

* Enhance NotebookToolbar component by replacing SparkIcon with VscFlame icon and updating hover styles to use inline event handlers, improving user interaction and visual consistency.

* Add Spark session creation functionality and update Spark app configuration handling

- Implemented a new endpoint in `spark_app.py` for creating Spark sessions via a POST request.
- Enhanced `Notebook.js` to include a method for initiating Spark session creation and handling success/error responses.
- Updated `SparkModel.js` to fetch Spark configuration and create a Spark session, improving integration with the backend API.
- Added state management for Spark configuration in the Notebook component.

* Integrate NotebookToolbar into Notebook component and enhance Spark session creation logging

- Added NotebookToolbar component to the Notebook, providing controls for running all cells, saving, deleting, and creating Spark sessions.
- Improved logging in the createSparkSession method to provide better visibility into the session creation process, including notebook path and configuration responses.
- Updated error handling to maintain user feedback during Spark session creation.

* Refactor import path for NotebookToolbar in Notebook component

- Updated the import statement for NotebookToolbar to reflect its new location in the 'content' directory, improving project structure and organization.

* Refactor Notebook component by removing Spark session creation logic and NotebookToolbar integration

- Eliminated the handleCreateSparkSession function to streamline the Notebook component.
- Removed NotebookToolbar component from the Notebook, simplifying the UI and focusing on core functionalities.
- Updated runAllCells prop to directly reference the runAllCells function, enhancing code clarity.

* Add runAllCells function to Notebook component for executing all cells

* Enhance Notebook component by integrating createSparkSession prop

- Added createSparkSession prop to both Notebook and Code components, enabling Spark session creation functionality.
- Updated the Code component to pass createSparkSession to the relevant child components, ensuring consistent functionality across the notebook interface.

* Add handleCreateSparkSession function to Notebook component for improved Spark session management

* Refactor Spark session creation in SparkModel.js to generate unique Spark app IDs and streamline API requests

- Updated the createSparkSession method to generate a unique Spark app ID using the current timestamp.
- Modified the API request to use the new Spark app ID for session creation, enhancing session management.
- Improved error handling and logging for better visibility during the Spark session creation process.

* Enhance demo notebook with Spark session initialization and update Notebook component cell handling

- Added a new code cell in the demo notebook to create a Spark session with detailed configuration settings.
- Updated the Notebook component to append new cells at the bottom instead of the top, improving user experience.
- Adjusted the execution logic to correctly handle the index of newly added cells during execution.

* Enhance demo notebook and Cell component for Spark output handling

- Added a markdown cell to the demo notebook for better documentation.
- Updated the last execution time and execution count in the notebook.
- Refactored the Cell component to include a new renderOutput function for improved handling of Spark output types, specifically integrating SparkOutputBox for Spark session information display.
- Streamlined output rendering logic to enhance code clarity and maintainability.

* Add SparkOutputBox component for rendering Spark output in notebooks

* Refactor SparkOutputBox styling and simplify component structure

- Updated the StyledBox component to use theme-based styling for better responsiveness.
- Changed the CSS selectors to target MUI Card components instead of the previous output area.
- Streamlined the return statement in SparkOutputBox for improved readability.

* Revert "Add SparkOutputBox component for rendering Spark output in notebooks"

This reverts commit 389bdac.

* Revert "Enhance demo notebook and Cell component for Spark output handling"

This reverts commit 0a10aba.

Author: xuwenyihust

examples/user_0@gmail.com/demo.ipynb

@@ -1,5 +1,70 @@
 {
  "cells": [
+  {
+   "cell_type": "code",
+   "isExecuted": true,
+   "lastExecutionResult": "success",
+   "lastExecutionTime": "2024-12-10 03:27:50",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "            <div>\n",
+       "                <p><b>SparkSession - in-memory</b></p>\n",
+       "                \n",
+       "        <div>\n",
+       "            <p><b>SparkContext</b></p>\n",
+       "\n",
+       "            <p><a href=\"http://8e207d700c27:4040\">Spark UI</a></p>\n",
+       "\n",
+       "            <dl>\n",
+       "              <dt>Version</dt>\n",
+       "                <dd><code>v3.5.0</code></dd>\n",
+       "              <dt>Master</dt>\n",
+       "                <dd><code>spark://spark-master:7077</code></dd>\n",
+       "              <dt>AppName</dt>\n",
+       "                <dd><code>spark-1733801092185</code></dd>\n",
+       "            </dl>\n",
+       "        </div>\n",
+       "        \n",
+       "            </div>\n",
+       "        "
+      ],
+      "text/plain": [
+       "<pyspark.sql.session.SparkSession at 0x7fffe2dc9ed0>"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "\n",
+    "from pyspark.sql import SparkSession\n",
+    "\n",
+    "spark = SparkSession.builder\\\n",
+    "    .appName(\"spark-1733801270245\")\\\n",
+    "    .master(\"spark://spark-master:7077\")\\\n",
+    "    .config(\"spark.jars.packages\", \"io.delta:delta-spark_2.12:3.0.0\")\\\n",
+    "    .config(\"spark.sql.extensions\", \"io.delta.sql.DeltaSparkSessionExtension\")\\\n",
+    "    .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.spark.sql.delta.catalog.DeltaCatalog\")\\\n",
+    "    .config(\"spark.eventLog.enabled\", \"true\")\\\n",
+    "    .config(\"spark.eventLog.dir\", \"/opt/data/spark-events\")\\\n",
+    "    .config(\"spark.history.fs.logDirectory\", \"/opt/data/spark-events\")\\\n",
+    "    .config(\"spark.sql.warehouse.dir\", \"/opt/data/spark-warehouse\")\\\n",
+    "    .config(\"spark.executor.memory\", \"1g\")\\\n",
+    "    .config(\"spark.executor.cores\", 1)\\\n",
+    "    .config(\"spark.executor.instances\", 1)\\\n",
+    "    .config(\"spark.driver.memory\", \"1g\")\\\n",
+    "    .config(\"spark.driver.cores\", 1)\\\n",
+    "    .getOrCreate()\n",
+    "\n",
+    "spark\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "isExecuted": true,
@@ -14,7 +79,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "isExecuted": true,
+   "isExecuted": false,
    "lastExecutionResult": "success",
    "lastExecutionTime": "2024-08-04 15:29:17",
    "metadata": {},

server/app/routes/spark_app.py

@@ -25,4 +25,22 @@ def get_spark_app_config(notbook_path):
 def update_spark_app_config(notbook_path):
     logging.info(f"Updating spark app config for notebook path: {notbook_path}")
     data = request.get_json()
-    return SparkApp.update_spark_app_config_by_notebook_path(notbook_path, data)
+    return SparkApp.update_spark_app_config_by_notebook_path(notbook_path, data)
+
+@spark_app_blueprint.route('/spark_app/session', methods=['POST'])
+def create_spark_session():
+    data = request.get_json()
+    notebook_path = data.get('notebookPath')
+    spark_config = data.get('config')
+    
+    try:
+        spark_app_id = SparkApp.create_spark_session(notebook_path, spark_config)
+        return jsonify({
+            'status': 'success',
+            'sparkAppId': spark_app_id
+        })
+    except Exception as e:
+        return jsonify({
+            'status': 'error',
+            'message': str(e)
+        }), 500

webapp/src/components/notebook/Notebook.js

@@ -36,6 +36,8 @@ function Notebook({
     const [cellStatuses, setCellStatuses] = useState(notebookState.content ? notebookState.content.cells.map(() => CellStatus.IDLE) : []);
     const [cellExecutedStatuses, setCellExecutedStatuses] = useState(notebookState.content ? notebookState.content.cells.map(cell => cell.cell_type === 'markdown') : []);
 
+    const [sparkConfig, setSparkConfig] = useState(null);
+    
     const setCellStatus = (index, status) => {
         setCellStatuses(prevStatuses => {
           const newStatuses = [...prevStatuses];
@@ -265,6 +267,57 @@ function Notebook({
         }
     };
 
+    const runAllCells = async () => {
+        console.log('Running all cells');
+        try {
+            await NotebookModel.runAllCells(
+                jupyterBaseUrl,
+                notebookState,
+                kernelId,
+                setKernelId,
+                cellStatuses,
+                setCellStatus,
+                cellExecutedStatuses,
+                setCellExecutedStatus
+            );
+        } catch (error) {
+            console.error('Failed to run all cells:', error);
+        }
+    };
+
+    const handleCreateSparkSession = async () => {
+        console.log('Create Spark session clicked');
+        try {
+            const { sparkAppId, initializationCode } = await SparkModel.createSparkSession(notebookState.path);
+            
+            // Create a new cell with the initialization code
+            const newCell = {
+                cell_type: 'code',
+                source: initializationCode,
+                metadata: {},
+                outputs: []
+            };
+
+            // Add the cell to the bottom of the notebook
+            const cells = [...notebookState.content.cells];
+            cells.push(newCell);
+            setNotebookState({
+                ...notebookState,
+                content: { ...notebookState.content, cells }
+            });
+
+            // Execute the cell (now need to use the last index)
+            const newCellIndex = cells.length - 1;
+            await handleRunCodeCell(newCell, CellStatus.IDLE, (status) => setCellStatus(newCellIndex, status));
+            
+            console.log('Spark session created with ID:', sparkAppId);
+            setSparkAppId(sparkAppId);
+        } catch (error) {
+            console.error('Failed to create Spark session:', error);
+            alert('Failed to create Spark session. Please check the configuration.');
+        }
+    };
+
     return (
         <div>
             {showNotebook && (
@@ -310,18 +363,10 @@ function Notebook({
                             handleCreateCell={handleCreateCell}
                             kernelId={kernelId}
                             setKernelId={setKernelId}
-                            runAllCells={
-                                () => NotebookModel.runAllCells(
-                                    jupyterBaseUrl, 
-                                    notebookState, 
-                                    kernelId, 
-                                    setKernelId, 
-                                    cellStatuses, 
-                                    setCellStatus,
-                                    cellExecutedStatuses,
-                                    setCellExecutedStatus)}
+                            runAllCells={runAllCells}
                             saveNotebook={handleUpdateNotebook}
                             deleteNotebook={handleDeleteNotebook}
+                            createSparkSession={handleCreateSparkSession}
                             /> : contentType === ContentType.Config ?
                             <Config 
                                 notebook={notebook}

webapp/src/components/notebook/content/Code.js

@@ -22,7 +22,8 @@ const Code = ({
   setKernelId,
   runAllCells,
   saveNotebook,
-  deleteNotebook
+  deleteNotebook,
+  createSparkSession
 }) => {
 
   const jupyterBaseUrl= `${config.jupyterBaseUrl}`
@@ -35,7 +36,8 @@ const Code = ({
         notebook={notebook}
         runAllCells={runAllCells}
         saveNotebook={saveNotebook}
-        deleteNotebook={deleteNotebook}/>
+        deleteNotebook={deleteNotebook}
+        createSparkSession={createSparkSession}/>
 
       <Box sx={{ 
           width: '95%',

webapp/src/components/notebook/content/NotebookToolbar.js

@@ -1,4 +1,4 @@
-import { VscSave, VscRunAll, VscTrash } from "react-icons/vsc";
+import { VscSave, VscRunAll, VscTrash, VscFlame } from "react-icons/vsc";
 import Tooltip from '@mui/material/Tooltip';
 import { Card, IconButton } from '@mui/material';
 import MoveButton from "../header/move/MoveButton";
@@ -7,7 +7,8 @@ const NotebookToolbar = ({
   notebook,
   runAllCells, 
   saveNotebook, 
-  deleteNotebook
+  deleteNotebook,
+  createSparkSession
 }) => {
   const headerIconSize = 13;
 
@@ -73,6 +74,27 @@ const NotebookToolbar = ({
           notebook={notebook}
           headerIconSize={headerIconSize}/>
 
+        {/* New Spark Button */}
+        <Tooltip title="Create Spark Session">
+          <IconButton 
+            disableRipple
+            onClick={createSparkSession}
+            aria-label="spark" 
+            sx={{ 
+                width: 'auto',
+                mt: 0.5 }}>
+            <VscFlame
+              size={headerIconSize} 
+              onMouseEnter={(e) => {
+                e.currentTarget.style.color = 'black';
+              }}
+              onMouseLeave={(e) => {
+                e.currentTarget.style.color = 'black';
+              }}
+              style={{ color: 'black' }}/>
+          </IconButton>
+        </Tooltip>
+
         {/* Delete Button */}
         <Tooltip title="Delete Notebook">
           <IconButton 

webapp/src/models/SparkModel.js

@@ -44,6 +44,73 @@ class SparkModel {
     return applicationId;
   }
 
+  static async createSparkSession(notebookPath) {
+    try {
+        console.log('Creating Spark session for notebook:', notebookPath);
+        // First get the config for this notebook
+        const configResponse = await fetch(`${config.serverBaseUrl}/spark_app/${notebookPath}/config`);
+        console.log('Config response:', configResponse);
+        
+        if (!configResponse.ok) {
+            throw new Error('Failed to fetch Spark configuration');
+        }
+        const sparkConfig = await configResponse.json();
+        console.log('Spark config:', sparkConfig);
+
+        // Generate a unique spark app ID
+        const sparkAppId = `spark-${Date.now()}`;
+
+        // Create a cell with Spark initialization code that uses the config
+        const sparkInitCode = `
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder\\
+    .appName("${sparkAppId}")\\
+    .master("spark://spark-master:7077")\\
+    .config("spark.jars.packages", "io.delta:delta-spark_2.12:3.0.0")\\
+    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")\\
+    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")\\
+    .config("spark.eventLog.enabled", "true")\\
+    .config("spark.eventLog.dir", "/opt/data/spark-events")\\
+    .config("spark.history.fs.logDirectory", "/opt/data/spark-events")\\
+    .config("spark.sql.warehouse.dir", "/opt/data/spark-warehouse")\\
+    .config("spark.executor.memory", "${sparkConfig['spark.executor.memory']}")\\
+    .config("spark.executor.cores", ${sparkConfig['spark.executor.cores']})\\
+    .config("spark.executor.instances", ${sparkConfig['spark.executor.instances']})\\
+    .config("spark.driver.memory", "${sparkConfig['spark.driver.memory']}")\\
+    .config("spark.driver.cores", ${sparkConfig['spark.driver.cores']})\\
+    .getOrCreate()
+
+spark
+`;
+
+        // Create the Spark session with this config
+        const response = await fetch(`${config.serverBaseUrl}/spark_app/${sparkAppId}`, {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+            },
+            body: JSON.stringify({
+                notebookPath: notebookPath,
+                sparkInitCode: sparkInitCode
+            }),
+        });
+
+        if (!response.ok) {
+            throw new Error('Failed to create Spark session');
+        }
+
+        const data = await response.json();
+        return {
+            sparkAppId: sparkAppId,
+            initializationCode: sparkInitCode
+        };
+    } catch (error) {
+        console.error('Error creating Spark session:', error);
+        throw error;
+    }
+}
+
 }
 
 export default SparkModel;