Logging and plotting

[normanius]

Feature Request 🚀

A beautiful feature of Thonny is, that if one prints numeric data to stdout in a structured manner, Thonny creates a live plot of the data being. See here. I'd love to see a similar feature for MicroPico.

Is your feature request related to a problem? Please describe.

The feature request is related to logging, monitoring and debugging.

Describe the solution you'd like

• Make it possible to forward the terminal output into a file (so that another tool can pick up the data)
• Offer some automatic plotting feature, similar to the one implemented in Thonny

Describe alternatives you've considered

Thonny IDE

Teachability, Documentation, Adoption, Migration Strategy

--

[normanius]

...is there already possibility to forward output to the terminal (through print statements) into a text file?

[normanius]

I've just implemented a tool that works for me outside of VS Code... Would be great if a feature similar to this one can be embedded in the MicroPico extension.

"""
How to use this tool with your MicroPython code:

    - Use formatted print statements:
        - print("e(t), Ie(t), de(t), u(t)")
        - print("%.2f, %.2f, %.2f, %.2f" % (e, Ie, de, u))
    - Use comma separated values!
    - Use a header at the beginning
    - Don't use any additional formatting
    
Install mpremote on your client machine:
    - pip install mpremote
    - Usage:
        - mpremote --help           # Help
        - mpremote ls               # List the filesystem
        - mpremote connect list     # List available serial ports
        - mpremote run script.py    # Run a script on the board
    - For logging, use the following command:
        - mpremote run script.py | tee logging/log.txt
        
While running the MicroPython code using mpremote, run this script in a 
separate terminal or within an IDE. It will plot the data in real-time.
"""

import sys
import time
import numpy as np
import pandas as pd
from pathlib import Path
import matplotlib.pyplot as plt
from matplotlib import colors as mplc
from itertools import product, cycle

configs = dict(
    path_to_logs = "2024-HS - PCLS/code/pico/week08/solutions/logs/logs.txt",
    sleep_time = 0.05,
    max_samples = 100,
    timeout = 10, # seconds
)

palette = ["#2D8FF3", "#FC585E", "#1AAF54"]
# https://mycolor.space / 3-color-gradient
palette = ["#2d8ff3", "#8682ed", "#bb71d9", "#e05fba", "#f65394", "#fb566f", 
           "#f4634b", "#e37529", "#c38a00", "#9c9b00", "#6da728", "#1aaf54"];
palette = ["#2d8ff3", "#fc585e", "#1aaf54", "#e05fba", "#e37529", "#f65394"]
styles = ["solid", "dashed", "dotted", "dashdot"]
#palette = palette[::6] + [palette[-1]] #+ palette[3::6]
palette = [mplc.to_rgba(c) for c in palette]
# Get list of colors of the Pastel1 colormap
#palette = plt.cm.Pastel1.colors

log_file = Path(configs["path_to_logs"])

start = time.time()
while log_file.is_file() == False:
    time.sleep(0.1) # wait for the file
    current = time.time()
    if current - start > configs.get("timeout", 60):
        print("Timeout reached, no log file found")
        import os
        print(os.getcwd())
        sys.exit()
        
def read_data(path):
    try:
        data = pd.read_csv(path)
    except pd.errors.EmptyDataError:
        return None
    if len(data) == 0:
        return None
    data = data.tail(configs["max_samples"])
    return data

def read_data_until(path, timeout=60):
    start = time.time()
    while True:
        data = read_data(path)
        if data is not None:
            return data
        time.sleep(1.0)
        current = time.time()
        if current - start > timeout:
            return None

def plot_data(data, ax):
    
    colors_styles = cycle(product(styles, palette))
    
    handles = dict()
    for i, col in enumerate(data.columns):
        ls, c = next(colors_styles)
        handle = ax.plot(data[col].values, 
                         color=c,
                         linestyle=ls, 
                         label=col.strip())
        handles[col] = handle
    # Place legend outside the plot
    ax.legend(loc="upper left", bbox_to_anchor=(1.02, 1.0))
    ax.set_title("Data log", fontweight="bold")
    ax.set_xlabel("Sample")
    ax.set_ylabel("Value")
    ax.grid(axis="y", alpha=0.5)
    return handles

def plot_update(data, handles, configs):
    if data is None:
        return
    max_samples = configs["max_samples"]
    x_min = np.inf
    x_max = -np.inf
    y_min = np.inf
    y_max = -np.inf
    for i, col in enumerate(data.columns):
        x = data[col].index
        y = data[col].values
        dtype = data[col].dtype
        if dtype == "object":
            # Likely an i/o error
            continue
        handles[col][0].set_ydata(y)
        handles[col][0].set_xdata(x)
        x_min = min(x_min, x.min())
        x_max = max(x_max, x.max())
        y_min = min(y_min, y.min())
        y_max = max(y_max, y.max())
    ax.set_xlim(x_min, x_max)
    y_min_cur, y_max_cur = ax.get_ylim()
    # Only change the y-axis limits if they have changed 
    low_rel = np.abs(y_min - y_min_cur)/(np.abs(y_min))
    high_rel = np.abs(y_max - y_max_cur)/(np.abs(y_max))
    if (low_rel > 1.5 or low_rel < 0.5) or (high_rel > 1.5 or high_rel < 0.5):
        #ax.set_ylim((y_min-((y_max-y_min)*0.1)**2, y_max+((y_max-y_min)*0.1))**2)
        ax.set_ylim((y_min_cur + (y_min - y_min_cur)*0.1, 
                     y_max_cur + (y_max - y_max_cur)*0.1))
    plt.draw()

plt.ion()
fig, ax = plt.subplots()
data = read_data_until(log_file, timeout=configs["timeout"])
handles = plot_data(data, ax=ax)
fig.tight_layout()

while True:
    data = read_data(log_file)
    plot_update(data, handles, configs)
    plt.pause(configs["sleep_time"])
    # Check if fig is still alive...
    if not plt.fignum_exists(fig.number):
        break

[Josverl]

Another option is micropython-magic,

That allows you to connect a jupyter notebook on your pc, running MicroPython code from the cells, and plotting the returned data.

Youll need to disconnect MicroPico while using it though. Cant have two different captains on the same serial port

[normanius]

Thank you. Will check it out.

In the meantime, I have put the above log visualizer in a GitHub project for reference: Link