Skip to content

Commit

Permalink
[DOC] Updated intro and analyze sections (#2855)
Browse files Browse the repository at this point in the history 
* Updated intro and analyze sections

* Apply suggestions from code review

* Update docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md

* Apply suggestions from code review

* Apply suggestions from code review
  • Loading branch information
@knylander-grafana
knylander-grafana authored Dec 15, 2023
1 parent 9ef1044 commit 3d54b5f
Show file tree
Hide file tree
Showing 10 changed files with 324 additions and 158 deletions.
95 changes: 17 additions & 78 deletions docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,96 +3,35 @@ title: Analyze app performance using Pyroscope
menuTitle: Analyze app performance
description: How to use the Pyroscope UI to analyze performance of your applications.
weight: 40
draft: true
keywords:
- pyroscope
- performance analysis
- flamegraphs
---

# Analyze app performance using Pyroscope


## Continuous profiling and metadata

While code profiling has been a long-standing practice, Continuous Profiling represents a modern and more advanced approach to performance monitoring. This technique adds two critical dimensions to traditional profiles:

- **Time:** Profiling data is collected _continuously_, providing a time-centric view that allows querying performance data from any point in the past
- **Metadata:** Profiles are enriched with metadata, adding contextual depth to the performance data

These dimensions, coupled with the detailed nature of performance profiles, make Continuous Profiling a uniquely valuable tool. Pyroscope's UI enhances this further by offering a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing.

In this UI reference, we'll show how Pyroscope parallels these other modern observability tools by providing a Prometheus-like querying experience. More importantly, you'll learn how to use Pyroscope's extensive UI features for a deeper insight into your application's performance.

## Key features of the Pyroscope UI

### Tag Explorer

The Tag Explorer page is a vital part of Pyroscope's UI, allowing users to navigate and analyze performance data through tags/labels. This feature is crucial for identifying performance anomalies and understanding the behavior of different application segments under various conditions. We intentionally don't include a query language on this page as we built this page to be as intuitive as possible for users to use the UI to navigate and drill down into which tags are most interesting to them.

To use the Tag Explorer:
1. Select a tag to view the corresponding profiling data
2. Analyze the pie chart and the table of descriptive statsitcs to determine which tags if any are behaving abnormally
3. Select a tag to view the corresponding profiling data
4. Make use of the shortcuts to the single, comparison, and diff pages to further identify the root cause of the performance issue
![tag-explorer-page](https://grafana.com/static/img/pyroscope/pyroscope-tag-explorer-cpu-2023-11-30.png)


### Single view

The Single View page in Pyroscope's UI is built for in-depth profile analysis. Here, you can explore a single flamegraph with multiple viewing options and functionalities:

- **Table View:** Breaks down the profiling data into a sortable table format
- **Sandwich View:** Displays both the callers and callees for a selected function, offering a comprehensive view of function interactions
- **Flamegraph View:** Visualizes profiling data in a flamegraph format, allowing easy identification of resource-intensive functions
- **Export & Share:** Options to export the flamegraph for offline analysis or share it via a flamegraph.com link for collaborative review
<!-- This page is unpublished until we have more information. -->

**Visual Placeholder:** *Screenshots demonstrating each view option in the Single View page.*

The screenshot above shows a spike in CPU usage.
Without profiling, we would go from a memory spike to digging through code or guessing what the cause of it is. However, with profiling we can use the flamegraph and table to see exactly which function is most responsible for the spike. Often this will show up as a single node taking up a noticeably disproportionate width in the flamegraph as seen below with the "checkDriverAvailability" function.

![example-flamegraph](https://grafana.com/static/img/pyroscope/pyroscope-ui-single-2023-11-30.png)

However, in some instances it may be a function that is called many times and is taking up a large amount of space in the flamegraph. In this case we can use the sandwich view to see that a logging function called throughout many functions in the codebase is the culprit.

![example-sandwich-view](https://grafana.com/static/img/pyroscope/sandwich-view-2023-11-30.png)

### Comparison Page

**Conducting Comparative Analysis with Label Sets**

The Comparison page facilitates side-by-side comparison of profiles either based on different label sets, different time periods, or both. This feature is extremely valuable for understanding the impact of changes or differences between do distinct queries of your application.

**How to Compare:**
1. Select two different sets of labels (e.g., `env:production` vs. `env:development`) and or time periods, reflected by the sub-timelines above each flamegraph
2. View the resulting flamegraphs side by side to identify disparities in performance

**Examples of Comparative Analysis:**
We see many practical use cases for comparison for companies using Pyroscope. Some examples of labels below experessed as `label:value` are:
- **Feature Flags:** Compare application performance with `feature_flag:a` vs. `feature_flag:b`
- **Deployment Environments:** Contrast `env:production` vs. `env:development`
- **Release Analysis:** Examine `commit:release-1` vs. `commit:release-2`
- **Region:** Compare `region:us-east-1` vs. `region:us-west-1`


Another example whre time is more important than labels is when you want to compare two different time periods. For example, in investigating the cause of a memory leak you would see something like the following where the timeline shows an steadily increasing amount of memory allocations over time. This is a clear indicator of a memory leak.

You can then use the comparison page to compare the memory allocations between two different time periods where allocations were low and where allocations were high which would allow you to identify the function that is causing the memory leak.

![comparison-ui](https://grafana.com/static/img/pyroscope/pyroscope-ui-comparison-2023-11-30.png)
# Analyze app performance using Pyroscope

### Diff Page
Pyroscope's UI is designed to make it easy to visualize and analyze profiling data.
There are several different modes for viewing, analyzing, uploading, and comparing profiling data.
These modes are discussed in the [Pyroscope UI documentation]({{< relref "../pyroscope-ui" >}}).

**Identifying Changes with Differential Analysis**
![Screenshots of Pyroscope's UI](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png)

The Diff page is realy an extension of the comparison page, crucial for more easily visually showing the differences between two profiling data sets. It normalizes the data by comparing the percentage of total time spent in each function so that the resulting flamegraph is comparing the __share__ of time spent in each function rather than the absolute amount of time spent in each function. This is important because it allows you to compare two different queries that may have different total amounts of time spent in each function.
One of the major benefits of continuous profiling is the ability to compare and diff profiling data from two different queries:

Similar to a git diff it takes the flamegraphs from the comparison page and highlights the differences between the two flamegraphs where red represents an increase in cpu usage from the baseline to the comparison and green represents a decrease.
- Comparing two different git commits before and after a code change
- Comparing Staging vs production environments to identify differences in performance
- Comparing performance between two different a/b tests or feature flag experiments
- Comparing memory allocations between two different time periods before and after a memory leak
- etc

Using the same examples from above here is a diff between two label sets:
![diff-ui](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png)
With traditional profiling getting any of this information is much more difficult to organize, properly label, share, or store for later analysis. With Pyroscope, all of this is just a matter of writing the two queries you'd like to compare and clicking a button.

<!-- and a diff between two time periods during a introduction of a memory leak:
![memory leak](https://grafana.com/static/img/pyroscope/pyroscope-memory-leak-2023-11-30.png) -->
## Seamless integration with observability tools

![Flowchart showing Pyroscope integration with other tools](https://grafana.com/static/img/pyroscope/grafana-pyroscope-dashboard-2023-11-30.png)

Pyroscope enhances its value through seamless integration with leading observability tools like Grafana, Prometheus, and Loki. This integration facilitates deeper insights into application performance and aids in addressing issues identified through other monitoring methods.
20 changes: 10 additions & 10 deletions docs/sources/ingest-and-analyze-profile-data/profiling-types/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ Various languages support different profiling types. Pyroscope supports the foll

CPU profiling measures the amount of CPU time consumed by different parts of your application code. High CPU usage can indicate inefficient code, leading to poor performance and increased operational costs. It's used to identify and optimize CPU-intensive functions in your application.

- **When to Use**: To identify and optimize CPU-intensive functions
- **Flamegraph Insight**: The width of blocks indicates the CPU time consumed by each function
- **When to use**: To identify and optimize CPU-intensive functions
- **Flamegraph insight**: The width of blocks indicates the CPU time consumed by each function

![example flamegraph](https://grafana.com/static/img/pyroscope/pyroscope-ui-single-2023-11-30.png)

Expand All @@ -61,8 +61,8 @@ As you can see here the UI is showing a spike in CPU along with the flamegraph a
Memory allocation profiling tracks the amount and frequency of memory allocations by the application. Excessive or inefficient memory allocation can lead to memory leaks and high garbage collection overhead, impacting application efficiency.

- **Types**: Alloc Objects, Alloc Space
- **When to Use**: For identifying and optimizing memory usage patterns
- **Flamegraph Insight**: Highlights functions where memory allocation is high
- **When to use**: For identifying and optimizing memory usage patterns
- **Flamegraph insight**: Highlights functions where memory allocation is high

![memory leak example](https://grafana.com/static/img/pyroscope/pyroscope-memory-leak-2023-11-30.png)

Expand All @@ -74,23 +74,23 @@ Without profiling this is may be something that is exhibited in metrics or OOM l

Goroutines are lightweight threads in Go, used for concurrent operations. Goroutine profiling measures the usage and performance of these threads. Poor management can lead to issues like deadlocks and excessive resource usage.

- **When to Use**: Especially useful in Go applications for concurrency management
- **Flamegraph Insight**: Provides a view of goroutine distribution and issues
- **When to use**: Especially useful in Go applications for concurrency management
- **Flamegraph insight**: Provides a view of goroutine distribution and issues

## Mutex profiling

Mutex profiling involves analyzing mutex (mutual exclusion) locks, used to prevent simultaneous access to shared resources. Excessive or long-duration mutex locks can cause delays and reduced application throughput.

- **Types**: Mutex Count, Mutex Duration
- **When to Use**: To optimize thread synchronization and reduce lock contention
- **Flamegraph Insight**: Shows frequency and duration of mutex operations
- **When to use**: To optimize thread synchronization and reduce lock contention
- **Flamegraph insight**: Shows frequency and duration of mutex operations

## Block profiling

Block profiling measures the frequency and duration of blocking operations, where a thread is paused or delayed. Blocking can significantly slow down application processes, leading to performance bottlenecks.

- **Types**: Block Count, Block Duration
- **When to Use**: To identify and reduce blocking delays
- **Flamegraph Insight**: Identifies where and how long threads are being blocked
- **When to use**: To identify and reduce blocking delays
- **Flamegraph insight**: Identifies where and how long threads are being blocked

<!-- # Next Steps: Exploring Pyroscope's UI(link to ui analysis docs) -->
Loading

0 comments on commit 3d54b5f

Please sign in to comment.