From 3d54b5f4ba7338208b6b746af47e4cdd108288fc Mon Sep 17 00:00:00 2001 From: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com> Date: Fri, 15 Dec 2023 13:00:19 -0500 Subject: [PATCH] [DOC] Updated intro and analyze sections (#2855) * Updated intro and analyze sections * Apply suggestions from code review * Update docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md * Apply suggestions from code review * Apply suggestions from code review --- .../analyze-profiles/_index.md | 95 ++--------- .../profiling-types/_index.md | 20 +-- .../pyroscope-ui.md | 119 +++++++++++-- docs/sources/introduction/_index.md | 2 - .../continuous-profiling/_index.md | 28 +++- docs/sources/introduction/examples.md | 33 ---- docs/sources/introduction/flamegraphs.md | 3 +- .../sources/introduction/profiling-tracing.md | 156 ++++++++++++++++++ docs/sources/introduction/profiling.md | 12 +- .../introduction/pyroscope-in-grafana.md | 14 +- 10 files changed, 324 insertions(+), 158 deletions(-) delete mode 100644 docs/sources/introduction/examples.md create mode 100644 docs/sources/introduction/profiling-tracing.md diff --git a/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md b/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md index 7b59c98951..14c559e4ca 100644 --- a/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md +++ b/docs/sources/ingest-and-analyze-profile-data/analyze-profiles/_index.md @@ -3,96 +3,35 @@ title: Analyze app performance using Pyroscope menuTitle: Analyze app performance description: How to use the Pyroscope UI to analyze performance of your applications. weight: 40 +draft: true keywords: - pyroscope - performance analysis - flamegraphs --- -# Analyze app performance using Pyroscope - - -## Continuous profiling and metadata - -While code profiling has been a long-standing practice, Continuous Profiling represents a modern and more advanced approach to performance monitoring. This technique adds two critical dimensions to traditional profiles: - -- **Time:** Profiling data is collected _continuously_, providing a time-centric view that allows querying performance data from any point in the past -- **Metadata:** Profiles are enriched with metadata, adding contextual depth to the performance data - -These dimensions, coupled with the detailed nature of performance profiles, make Continuous Profiling a uniquely valuable tool. Pyroscope's UI enhances this further by offering a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing. - -In this UI reference, we'll show how Pyroscope parallels these other modern observability tools by providing a Prometheus-like querying experience. More importantly, you'll learn how to use Pyroscope's extensive UI features for a deeper insight into your application's performance. - -## Key features of the Pyroscope UI - -### Tag Explorer - -The Tag Explorer page is a vital part of Pyroscope's UI, allowing users to navigate and analyze performance data through tags/labels. This feature is crucial for identifying performance anomalies and understanding the behavior of different application segments under various conditions. We intentionally don't include a query language on this page as we built this page to be as intuitive as possible for users to use the UI to navigate and drill down into which tags are most interesting to them. - -To use the Tag Explorer: -1. Select a tag to view the corresponding profiling data -2. Analyze the pie chart and the table of descriptive statsitcs to determine which tags if any are behaving abnormally -3. Select a tag to view the corresponding profiling data -4. Make use of the shortcuts to the single, comparison, and diff pages to further identify the root cause of the performance issue -![tag-explorer-page](https://grafana.com/static/img/pyroscope/pyroscope-tag-explorer-cpu-2023-11-30.png) - - -### Single view - -The Single View page in Pyroscope's UI is built for in-depth profile analysis. Here, you can explore a single flamegraph with multiple viewing options and functionalities: - -- **Table View:** Breaks down the profiling data into a sortable table format -- **Sandwich View:** Displays both the callers and callees for a selected function, offering a comprehensive view of function interactions -- **Flamegraph View:** Visualizes profiling data in a flamegraph format, allowing easy identification of resource-intensive functions -- **Export & Share:** Options to export the flamegraph for offline analysis or share it via a flamegraph.com link for collaborative review + -**Visual Placeholder:** *Screenshots demonstrating each view option in the Single View page.* - -The screenshot above shows a spike in CPU usage. -Without profiling, we would go from a memory spike to digging through code or guessing what the cause of it is. However, with profiling we can use the flamegraph and table to see exactly which function is most responsible for the spike. Often this will show up as a single node taking up a noticeably disproportionate width in the flamegraph as seen below with the "checkDriverAvailability" function. - -![example-flamegraph](https://grafana.com/static/img/pyroscope/pyroscope-ui-single-2023-11-30.png) - -However, in some instances it may be a function that is called many times and is taking up a large amount of space in the flamegraph. In this case we can use the sandwich view to see that a logging function called throughout many functions in the codebase is the culprit. - -![example-sandwich-view](https://grafana.com/static/img/pyroscope/sandwich-view-2023-11-30.png) - -### Comparison Page - -**Conducting Comparative Analysis with Label Sets** - -The Comparison page facilitates side-by-side comparison of profiles either based on different label sets, different time periods, or both. This feature is extremely valuable for understanding the impact of changes or differences between do distinct queries of your application. - -**How to Compare:** -1. Select two different sets of labels (e.g., `env:production` vs. `env:development`) and or time periods, reflected by the sub-timelines above each flamegraph -2. View the resulting flamegraphs side by side to identify disparities in performance - -**Examples of Comparative Analysis:** -We see many practical use cases for comparison for companies using Pyroscope. Some examples of labels below experessed as `label:value` are: -- **Feature Flags:** Compare application performance with `feature_flag:a` vs. `feature_flag:b` -- **Deployment Environments:** Contrast `env:production` vs. `env:development` -- **Release Analysis:** Examine `commit:release-1` vs. `commit:release-2` -- **Region:** Compare `region:us-east-1` vs. `region:us-west-1` - - -Another example whre time is more important than labels is when you want to compare two different time periods. For example, in investigating the cause of a memory leak you would see something like the following where the timeline shows an steadily increasing amount of memory allocations over time. This is a clear indicator of a memory leak. - -You can then use the comparison page to compare the memory allocations between two different time periods where allocations were low and where allocations were high which would allow you to identify the function that is causing the memory leak. - -![comparison-ui](https://grafana.com/static/img/pyroscope/pyroscope-ui-comparison-2023-11-30.png) +# Analyze app performance using Pyroscope -### Diff Page +Pyroscope's UI is designed to make it easy to visualize and analyze profiling data. +There are several different modes for viewing, analyzing, uploading, and comparing profiling data. +These modes are discussed in the [Pyroscope UI documentation]({{< relref "../pyroscope-ui" >}}). -**Identifying Changes with Differential Analysis** +![Screenshots of Pyroscope's UI](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png) -The Diff page is realy an extension of the comparison page, crucial for more easily visually showing the differences between two profiling data sets. It normalizes the data by comparing the percentage of total time spent in each function so that the resulting flamegraph is comparing the __share__ of time spent in each function rather than the absolute amount of time spent in each function. This is important because it allows you to compare two different queries that may have different total amounts of time spent in each function. +One of the major benefits of continuous profiling is the ability to compare and diff profiling data from two different queries: -Similar to a git diff it takes the flamegraphs from the comparison page and highlights the differences between the two flamegraphs where red represents an increase in cpu usage from the baseline to the comparison and green represents a decrease. +- Comparing two different git commits before and after a code change +- Comparing Staging vs production environments to identify differences in performance +- Comparing performance between two different a/b tests or feature flag experiments +- Comparing memory allocations between two different time periods before and after a memory leak +- etc -Using the same examples from above here is a diff between two label sets: -![diff-ui](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png) +With traditional profiling getting any of this information is much more difficult to organize, properly label, share, or store for later analysis. With Pyroscope, all of this is just a matter of writing the two queries you'd like to compare and clicking a button. - +## Seamless integration with observability tools +![Flowchart showing Pyroscope integration with other tools](https://grafana.com/static/img/pyroscope/grafana-pyroscope-dashboard-2023-11-30.png) +Pyroscope enhances its value through seamless integration with leading observability tools like Grafana, Prometheus, and Loki. This integration facilitates deeper insights into application performance and aids in addressing issues identified through other monitoring methods. \ No newline at end of file diff --git a/docs/sources/ingest-and-analyze-profile-data/profiling-types/_index.md b/docs/sources/ingest-and-analyze-profile-data/profiling-types/_index.md index a2c1b3ff3a..7ede9074a5 100644 --- a/docs/sources/ingest-and-analyze-profile-data/profiling-types/_index.md +++ b/docs/sources/ingest-and-analyze-profile-data/profiling-types/_index.md @@ -46,8 +46,8 @@ Various languages support different profiling types. Pyroscope supports the foll CPU profiling measures the amount of CPU time consumed by different parts of your application code. High CPU usage can indicate inefficient code, leading to poor performance and increased operational costs. It's used to identify and optimize CPU-intensive functions in your application. -- **When to Use**: To identify and optimize CPU-intensive functions -- **Flamegraph Insight**: The width of blocks indicates the CPU time consumed by each function +- **When to use**: To identify and optimize CPU-intensive functions +- **Flamegraph insight**: The width of blocks indicates the CPU time consumed by each function ![example flamegraph](https://grafana.com/static/img/pyroscope/pyroscope-ui-single-2023-11-30.png) @@ -61,8 +61,8 @@ As you can see here the UI is showing a spike in CPU along with the flamegraph a Memory allocation profiling tracks the amount and frequency of memory allocations by the application. Excessive or inefficient memory allocation can lead to memory leaks and high garbage collection overhead, impacting application efficiency. - **Types**: Alloc Objects, Alloc Space -- **When to Use**: For identifying and optimizing memory usage patterns -- **Flamegraph Insight**: Highlights functions where memory allocation is high +- **When to use**: For identifying and optimizing memory usage patterns +- **Flamegraph insight**: Highlights functions where memory allocation is high ![memory leak example](https://grafana.com/static/img/pyroscope/pyroscope-memory-leak-2023-11-30.png) @@ -74,23 +74,23 @@ Without profiling this is may be something that is exhibited in metrics or OOM l Goroutines are lightweight threads in Go, used for concurrent operations. Goroutine profiling measures the usage and performance of these threads. Poor management can lead to issues like deadlocks and excessive resource usage. -- **When to Use**: Especially useful in Go applications for concurrency management -- **Flamegraph Insight**: Provides a view of goroutine distribution and issues +- **When to use**: Especially useful in Go applications for concurrency management +- **Flamegraph insight**: Provides a view of goroutine distribution and issues ## Mutex profiling Mutex profiling involves analyzing mutex (mutual exclusion) locks, used to prevent simultaneous access to shared resources. Excessive or long-duration mutex locks can cause delays and reduced application throughput. - **Types**: Mutex Count, Mutex Duration -- **When to Use**: To optimize thread synchronization and reduce lock contention -- **Flamegraph Insight**: Shows frequency and duration of mutex operations +- **When to use**: To optimize thread synchronization and reduce lock contention +- **Flamegraph insight**: Shows frequency and duration of mutex operations ## Block profiling Block profiling measures the frequency and duration of blocking operations, where a thread is paused or delayed. Blocking can significantly slow down application processes, leading to performance bottlenecks. - **Types**: Block Count, Block Duration -- **When to Use**: To identify and reduce blocking delays -- **Flamegraph Insight**: Identifies where and how long threads are being blocked +- **When to use**: To identify and reduce blocking delays +- **Flamegraph insight**: Identifies where and how long threads are being blocked diff --git a/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md b/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md index 002f9240b4..dd05422581 100644 --- a/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md +++ b/docs/sources/ingest-and-analyze-profile-data/pyroscope-ui.md @@ -1,7 +1,7 @@ --- -title: Use Pyroscope UI +title: Use the Pyroscope UI to explore profiling data menuTitle: Use the Pyroscope UI -description: How to use the Pyroscope UI to analyze performance of your applications. +description: How to use the Pyroscope UI to explore profile data. weight: 40 keywords: - pyroscope @@ -9,27 +9,112 @@ keywords: - flamegraphs --- -## Pyroscope: Continuous Profiling in Action - -![Screenshots of Pyroscope's UI](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png) +# Use the Pyroscope UI to explore profiling data Pyroscope's UI is designed to make it easy to visualize and analyze profiling data. There are several different modes for viewing, analyzing, uploading, and comparing profiling data. -We will go into more detail about these modes in the [Pyroscope UI documentation]. -For now, it's important to note that one of the major benefits of continuous profiling is the ability to compare and diff profiling data from two different queries: -- Comparing two different git commits before and after a code change -- Comparing Staging vs production environments to identify differences in performance -- Comparing performance between two different a/b tests or feature flag experiments -- Comparing memory allocations between two different time periods before and after a memory leak -- etc +![Screenshots of Pyroscope's UI](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png) + +While code profiling has been a long-standing practice, continuous profiling represents a modern and more advanced approach to performance monitoring. This technique adds two critical dimensions to traditional profiles: + +Time +: Profiling data is collected _continuously_, providing a time-centric view that allows querying performance data from any point in the past. + +Metadata +: Profiles are enriched with metadata, adding contextual depth to the performance data. + +These dimensions, coupled with the detailed nature of performance profiles, make continuous profiling a uniquely valuable tool. +Pyroscope's UI enhances this further by offering a convenient platform to analyze profiles and get insights that are impossible to get from using other traditional signals like logs, metrics, or tracing. + +In this UI reference, you'll learn how Pyroscope parallels these other modern observability tools by providing a Prometheus-like querying experience. More importantly, you'll learn how to use Pyroscope's extensive UI features for a deeper insight into your application's performance. + +## Key features of the Pyroscope UI + +The following sections describe Pyroscope UI capabilities. + + + +## Tag Explorer + +The Tag Explorer page is a vital part of Pyroscope's UI, allowing users to navigate and analyze performance data through tags/labels. This feature is crucial for identifying performance anomalies and understanding the behavior of different application segments under various conditions. We intentionally don't include a query language on this page as we built this page to be as intuitive as possible for users to use the UI to navigate and drill down into which tags are most interesting to them. + +To use the Tag Explorer: + +1. Select a tag to view the corresponding profiling data +2. Analyze the pie chart and the table of descriptive statistics to determine which tags if any are behaving abnormally +3. Select a tag to view the corresponding profiling data +4. Make use of the shortcuts to the single, comparison, and diff pages to further identify the root cause of the performance issue +![tag-explorer-page](https://grafana.com/static/img/pyroscope/pyroscope-tag-explorer-cpu-2023-11-30.png) + +## Single view + +The Single View page in Pyroscope's UI is built for in-depth profile analysis. Here, you can explore a single flamegraph with multiple viewing options and functionalities: + +Table view +: Breaks down the profiling data into a sortable table format. + +Sandwich view +: Displays both the callers and callees for a selected function, offering a comprehensive view of function interactions. + +Flamegraph view +: Visualizes profiling data in a flamegraph format, allowing easy identification of resource-intensive functions. + +Export & share +: Options to export the flamegraph for offline analysis or share it via a flamegraph.com link for collaborative review. + + + +The screenshot above shows a spike in CPU usage. +Without profiling, we would go from a memory spike to digging through code or guessing what the cause of it is. However, with profiling, you can use the flamegraph and table to see exactly which function is most responsible for the spike. Often this will show up as a single node taking up a noticeably disproportionate width in the flamegraph as seen below with the "checkDriverAvailability" function. + +![example-flamegraph](https://grafana.com/static/img/pyroscope/pyroscope-ui-single-2023-11-30.png) + +However, in some instances it may be a function that is called many times and is taking up a large amount of space in the flamegraph. +In this case, you can use the sandwich view to see that a logging function called throughout many functions in the codebase is the culprit. + +![example-sandwich-view](https://grafana.com/static/img/pyroscope/sandwich-view-2023-11-30.png) + +### Comparison page + +The Comparison page facilitates side-by-side comparison of profiles either based on different label sets, different time periods, or both. This feature is extremely valuable for understanding the impact of changes or differences between do distinct queries of your application. + +To run a comparison: + +1. Select two different sets of labels (for exmaple, `env:production` vs. `env:development`) and or time periods, reflected by the sub-timelines above each flamegraph. +2. View the resulting flamegraphs side by side to identify disparities in performance. + +There are many practical use cases for comparison for companies using Pyroscope. +Some examples of labels below expressed as `label:value` are: + +Feature flags +: Compare application performance with `feature_flag:a` vs. `feature_flag:b` + +Deployment environments +: Contrast `env:production` vs. `env:development` + +Release analysis +: Examine `commit:release-1` vs. `commit:release-2` + +Region +: Compare `region:us-east-1` vs. `region:us-west-1` + +Another example where time is more important than labels is when you want to compare two different time periods. For example, in investigating the cause of a memory leak you would see something like the following where the timeline shows an steadily increasing amount of memory allocations over time. This is a clear indicator of a memory leak. + +You can then use the comparison page to compare the memory allocations between two different time periods where allocations were low and where allocations were high which would allow you to identify the function that is causing the memory leak. + +![comparison-ui](https://grafana.com/static/img/pyroscope/pyroscope-ui-comparison-2023-11-30.png) -With traditional profiling getting any of this information is much more difficult to organize, properly label, share, or store for later analysis. With Pyroscope, all of this is just a matter of writing the two queries you'd like to compare and clicking a button. +## Diff page: Identify changes with differential analysis -This UI will also expand over time to better help dig deeper into the data and provide more insights into your application. +The Diff page is an extension of the comparison page, crucial for more easily visually showing the differences between two profiling data sets. +It normalizes the data by comparing the percentage of total time spent in each function so that the resulting flamegraph is comparing the __share__ of time spent in each function rather than the absolute amount of time spent in each function. +This is important because it allows you to compare two different queries that may have different total amounts of time spent in each function. -## Seamless integration with observability tools +Similar to a git diff it takes the flamegraphs from the comparison page and highlights the differences between the two flamegraphs where red represents an increase in CPU usage from the baseline to the comparison and green represents a decrease. -![Flowchart showing Pyroscope integration with other tools](https://grafana.com/static/img/pyroscope/grafana-pyroscope-dashboard-2023-11-30.png) +Using the same examples from above, here is a diff between two label sets: +![diff-ui](https://grafana.com/static/img/pyroscope/pyroscope-ui-diff-2023-11-30.png) -Pyroscope enhances its value through seamless integration with leading observability tools like Grafana, Prometheus, and Loki. This integration facilitates deeper insights into application performance and aids in addressing issues identified through other monitoring methods. \ No newline at end of file + diff --git a/docs/sources/introduction/_index.md b/docs/sources/introduction/_index.md index 0f0c0762fc..9445888bba 100644 --- a/docs/sources/introduction/_index.md +++ b/docs/sources/introduction/_index.md @@ -8,8 +8,6 @@ keywords: - Profiling --- - - # Introduction Grafana Pyroscope is a multi-tenant continuous profiling aggregation system, aligning its architectural design with Grafana Mimir, Grafana Loki, and Grafana Tempo. It facilitates the ingestion, storage, and querying of profiles and seamlessly integrates with Grafana, enabling a cohesive correlation of profiling data with existing metrics, logs, and traces. diff --git a/docs/sources/introduction/continuous-profiling/_index.md b/docs/sources/introduction/continuous-profiling/_index.md index 1fa0da4a3a..bd28134f57 100644 --- a/docs/sources/introduction/continuous-profiling/_index.md +++ b/docs/sources/introduction/continuous-profiling/_index.md @@ -12,17 +12,21 @@ keywords: ## When to use continuous profiling -**Continuous profiling** is a systematic method of collecting and analyzing performance data from production systems. +Continuous profiling is a systematic method of collecting and analyzing performance data from production systems. -Traditionally, profiling has been used more as an ad-hoc debugging tool in languages like Go and Java. +Traditionally, profiling is used as an ad-hoc debugging tool in languages like Go and Java. You are probably used to running a benchmark tool locally and getting a pprof file in Go or maybe connecting into a misbehaving prod instance and pulling a flamegraph from a JFR file in Java. This is great for debugging but not so great for production. ![example flamegraph](https://grafana.com/static/img/pyroscope/pyroscope-ui-single-2023-11-30.png) -Continuous profiling is a modern approach which is safer and more scalable for production environments. -It makes use of low overhead sampling to collect profiles from production systems and stores them in a database for later analysis. -You can get a more holistic view of your application and how it behaves in production. +{{% admonition type="note" %}} +To learn more about flamegraphs, refer to [Flamegraphs: Visualizing performance data]({{< relref "../flamegraphs" >}}). +{{% /admonition %}} + +Continuous profiling is a modern approach which is safer and more scalable for production environments. +It uses low-overhead sampling to collect profiles from production systems and stores the profiles in a database for later analysis. +Using continuous profiling gives you a more holistic view of your application and how it behaves in production. ## Benefits @@ -30,9 +34,9 @@ You can get a more holistic view of your application and how it behaves in produ Why prioritize continuous profiling? -1. **In-Depth Code Insights:** It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance. -2. **Complements Other Observability Tools:** Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy. -3. **Proactive Performance Optimization:** Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications. +1. **In-depth code insights:** It provides granular, line-level insights into how application code utilizes resources, offering the most detailed view of application performance. +2. **Complements other observability tools:** Continuous profiling fills critical gaps left by metrics, logs, and tracing, creating a more comprehensive observability strategy. +3. **Proactive performance optimization:** Regular profiling enables teams to proactively identify and resolve performance bottlenecks, leading to more efficient and reliable applications. ## Use cases @@ -49,6 +53,14 @@ Adopting continuous profiling with tools like Pyroscope can lead to significant Pyroscope's low-overhead profiling enables precise optimization of resource usage, directly impacting various cost centers in technology infrastructure. By providing in-depth insights into application performance, Pyroscope allows teams to identify and eliminate inefficiencies, leading to significant savings in areas like observability, incident management, messaging/queuing, deployment tools, and infrastructure. +By using sampling profilers, Pyroscope is able to collect data with minimal overhead (~2-5% depending on a few factors). +The [custom storage engine](https://pyroscope.io/docs/storage-design/) compresses and stores the data efficiently. +Some advantages of this are: + +- Low CPU overhead thanks to sampling profiler technology +- Control over profiling data granularity (10s to multiple years) +- Efficient compression, low disk space requirements and cost + ### Reduced latency Pyroscope plays a pivotal role in reducing application latency by identifying performance bottlenecks at the code level. diff --git a/docs/sources/introduction/examples.md b/docs/sources/introduction/examples.md deleted file mode 100644 index 4961e0b162..0000000000 --- a/docs/sources/introduction/examples.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Profiling examples -menuTitle: Profiling examples -description: What can you do with profiling data to resolve real-world problems? -weight: 100 -draft: true -keywords: - - Pyroscope - - Profiling ---- - - - -# Profiling examples - - - -Introduce the concept. - - - -## Sections - - - -## Useful links - - diff --git a/docs/sources/introduction/flamegraphs.md b/docs/sources/introduction/flamegraphs.md index 43a7160924..b68050dc69 100644 --- a/docs/sources/introduction/flamegraphs.md +++ b/docs/sources/introduction/flamegraphs.md @@ -1,7 +1,7 @@ --- title: "Flamegraphs: Visualizing performance data" menuTitle: Flamegraphs -description: Learn about flamegraphs to help visualize performance data +description: Learn about flamegraphs to help visualize performance data. weight: 100 keywords: - Pyroscope @@ -34,4 +34,3 @@ The nodes below those represent the functions that were called from those functi This continues until you reach the bottom of the flamegraph. This is a CPU profile, but profiles can represent many other types of resource such as memory, network, disk, etc. -To understand more about how to read a flamegraph, what the different colors mean, and what other types of profiles exist and when to use them see our flamegraph documentation or [profile types documentation](/docs/pyroscope/next/analyzing-profiles-in-pyroscope-ui/) \ No newline at end of file diff --git a/docs/sources/introduction/profiling-tracing.md b/docs/sources/introduction/profiling-tracing.md new file mode 100644 index 0000000000..70d36c5020 --- /dev/null +++ b/docs/sources/introduction/profiling-tracing.md @@ -0,0 +1,156 @@ +--- +title: Profiling and tracing integration +menuTitle: Profiling and tracing +description: Learning about how profiling and tracing work together. +weight: 50 +keywords: + - pyroscope + - continuous profiling + - tracing +--- + +# Profiling and tracing integration + +Profiles, continuous profiling, and distributed traces are all tools that can be used to improve the performance and reliability of applications. +However, each tool has its own strengths and weaknesses, and it is important to choose the right tool for the job as well as understand when to use both. + +## Profiling + +Profiling offers a deep-dive into an application's performance at the code level, highlighting resource usage and performance spots. + + + + + + + + + + + + + + +
Usage + During development, major releases, or upon noticing performance quirks. +
Benefits + +
    + +
  • Business: Boosts user experience through enhanced application performance. + +
  • Technical: Gives clear insights into code performance and areas of refinement. +
    • +
+
Example + A developer uses profiling upon noting slow app performance, identifies a CPU-heavy function, and optimizes it. +
+ + +## Continuous profiling + +Continuous profiling provides ongoing performance insights, capturing long-term trends and intermittent issues. + + + + + + + + + + + + + + +
Usage + Mainly in production, especially for high-priority applications. +
Benefits + +
    + +
  • Business: Preemptively addresses inefficiencies, potentially saving costs. + +
  • Technical: Highlights performance trends and issues like potential memory leaks over time. +
    • +
+
Example + A month-long data from continuous profiling suggests increasing memory consumption, hinting at a memory leak. +
+ + +## Distributed tracing + +Traces requests as they cross multiple services, revealing interactions and service dependencies. + + + + + + + + + + + + + + +
Usage + Essential for systems like microservices where requests touch multiple services. +
Benefits + +
    + +
  • Business: Faster issue resolution, reduced downtimes, and strengthened customer trust. + +
  • Technical: A broad view of the system's structure, revealing bottlenecks and inter-service dependencies. +
    • +
+
Example + In e-commerce, a user's checkout request might involve various services. Tracing depicts this route, pinpointing where time is most spent. +
+ + +## Combined power of tracing and profiling + +When used together, tracing and provide a powerful tool for understanding system and application performance. + + + + + + + + + + + + + + +
Usage + For comprehensive system-to-code insights, especially when diagnosing complex issues spread across services and codebases. +
Benefits + +
    + +
  • Business: Reduces downtime, optimizes user experience, and safeguards revenues. + +
  • Technical: +
      + +
    • Holistic view: Tracing pinpoints bottle-necked services, while profiling delves into the responsible code segments. + +
    • End-to-end insight: Visualizes a request's full journey and the performance of individual code parts. + +
    • Efficient diagnosis: Tracing identifies service latency; profiling zeroes in on its cause, be it database queries, API calls, or specific code inefficiencies. +
      • +
    +
    • +
+
Example + Tracing reveals latency in a payment service. Combined with profiling, it's found that a particular function, making third-party validation calls, is the culprit. This insight guides optimization, refining system efficiency. +
+ diff --git a/docs/sources/introduction/profiling.md b/docs/sources/introduction/profiling.md index 16f332f952..14f119b8e0 100644 --- a/docs/sources/introduction/profiling.md +++ b/docs/sources/introduction/profiling.md @@ -11,7 +11,11 @@ keywords: # Profiling fundamentals -**Profiling** is a technique used in software development to measure and analyze the runtime behavior of a program. By profiling a program, developers can identify which parts of the program consume the most resources, such as CPU time, memory, or I/O operations. This information can then be used to optimize the program, making it run faster or use fewer resources. +Profiling is a technique used in software development to measure and analyze the runtime behavior of a program. +By profiling a program, developers can identify which parts of the program consume the most resources, such as CPU time, memory, or I/O operations. +This information can then be used to optimize the program, making it run faster or use fewer resources. + +Pyroscope can be used for both traditional and continuous profiling. ## Traditional profiling (non-continuous) @@ -27,13 +31,13 @@ Traditional profiling provides: - **Precision**: Offers a deep dive into specific sections of the code. - **Control**: Developers can initiate profiling sessions at their discretion, allowing for targeted optimization efforts. -- **Detailed Reports**: Provides granular data about program execution, making it easier to pinpoint bottlenecks. +- **Detailed reports**: Provides granular data about program execution, making it easier to pinpoint bottlenecks. ## Continuous profiling -As software systems grew in complexity and scale, the limitations of traditional profiling became evident. Issues could arise in production that were not apparent during limited profiling sessions in the development or staging environments. +As software systems grew in complexity and scale, the limitations of traditional profiling became evident. Issues could arise in production that weren't apparent during limited profiling sessions in the development or staging environments. -This led to the development of **continuous profiling**, a method where the profiling data is continuously collected in the background with minimal overhead. By doing so, developers gain a more comprehensive view of a program's behavior over time, helping to identify sporadic or long-term performance issues. +This led to the development of continuous profiling, a method where the profiling data is continuously collected in the background with minimal overhead. By doing so, developers gain a more comprehensive view of a program's behavior over time, helping to identify sporadic or long-term performance issues. ### Benefits diff --git a/docs/sources/introduction/pyroscope-in-grafana.md b/docs/sources/introduction/pyroscope-in-grafana.md index 92edb20388..cf29709e55 100644 --- a/docs/sources/introduction/pyroscope-in-grafana.md +++ b/docs/sources/introduction/pyroscope-in-grafana.md @@ -13,14 +13,20 @@ keywords: # Pyroscope and profiling in Grafana -One of the major benefits of Pyroscope is that it can be used alongside the other Grafana tools such as Loki, Tempo, Mimir, and k6. This allows you to use Pyroscope to get the most granular insight into your application and how you can use it to fix issues that you may have identified via metrics, logs, traces, or anything else. +Pyroscope can be used alongside the other Grafana tools such as Loki, Tempo, Mimir, and k6. +You can use Pyroscope to get the most granular insight into your application and how you can use it to fix issues that you may have identified via metrics, logs, traces, or anything else. -You can use Pyroscope within Grafana by using the Pyroscope datasource plugin. This plugin allows you to query Pyroscope data from within Grafana and visualize it alongside your other Grafana data. +You can use Pyroscope within Grafana by using the Pyroscope data source plugin. +This plugin lets you query Pyroscope data from within Grafana and visualize it alongside your other Grafana data. -For example here is a screenshot of the explore page where we've combined traces and profiles to be able to see granular line-level detail when available for a trace span. This allows you to see the exact function that is causing a bottleneck in your application as well as a specific request. +## Visualize traces and profiles data + +Here is a screenshot of the **Explore** page where combined traces and profiles to be able to see granular line-level detail when available for a trace span. This allows you to see the exact function that's causing a bottleneck in your application as well as a specific request. ![trace-profiler-view](https://grafana.com/static/img/pyroscope/pyroscope-trace-profiler-view-2023-11-30.png) -And here is an example of how you can integrate profiles into your dashboards. In this case we showing memory profiles alongside panels for logs and metrics to be able to debug OOM errors alongside the associated logs and metrics. +## Integrate profiles into dashboards + +Here is an example of how you can integrate profiles into your dashboards. In this case, the screenshot shows memory profiles alongside panels for logs and metrics to be able to debug OOM errors alongside the associated logs and metrics. ![dashboard](https://grafana.com/static/img/pyroscope/grafana-pyroscope-dashboard-2023-11-30.png)