Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic when trying to profile cpu #1008

Closed
Rachelint opened this issue Jun 20, 2023 · 1 comment · Fixed by #1088
Closed

panic when trying to profile cpu #1008

Rachelint opened this issue Jun 20, 2023 · 1 comment · Fixed by #1088
Labels
bug Something isn't working upstream

Comments

@Rachelint
Copy link
Contributor

Describe this problem

I try to profile cpu with ceresdb's http interface, and unfortunately the process panic.

2023-06-20 23:46:37.296 ERRO [common_util/src/panic.rs:42] thread 'ceres-default' panicked 'attempt to add with overflow' at "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/findshlibs-0.10.2/src/lib.rs:261"
   0: common_util::panic::set_panic_hook::{{closure}}
             at /root/code/xunming/ceresdb/common_util/src/panic.rs:41:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:577:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panicking.rs:64:14
   6: core::panicking::panic
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panicking.rs:114:5
   7: pprof::profiler::ProfilerGuardBuilder::blocklist::{{closure}}
      findshlibs::linux::SharedLibrary::callback::{{closure}}
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/findshlibs-0.10.2/src/linux/mod.rs:285:13
      core::ops::function::FnOnce::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/ops/function.rs:250:5
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panic.rs:140:14
      findshlibs::linux::SharedLibrary::callback
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/findshlibs-0.10.2/src/linux/mod.rs:281:15
   8: __GI___dl_iterate_phdr
   9: <findshlibs::linux::SharedLibrary as findshlibs::SharedLibrary>::each
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/findshlibs-0.10.2/src/linux/mod.rs:351:13
      pprof::profiler::ProfilerGuardBuilder::blocklist
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/pprof-0.11.1/src/profiler.rs:88:13
      profile::Profiler::dump_cpu_prof
             at /root/code/xunming/ceresdb/components/profile/src/lib.rs:148:21
  10: server::http::Service<Q>::profile_cpu::{{closure}}::{{closure}}::{{closure}}
             at /root/code/xunming/ceresdb/server/src/http.rs:414:25
      <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/task.rs:42:21
      tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/core.rs:223:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::Core<T,S>::poll
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/core.rs:212:13
      tokio::runtime::task::harness::poll_future::{{closure}}
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:476:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panic.rs:140:14
      tokio::runtime::task::harness::poll_future
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:464:18
      tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:198:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/harness.rs:152:15
      tokio::runtime::task::raw::poll
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/raw.rs:255:5
  11: tokio::runtime::task::raw::RawTask::poll
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/raw.rs:200:18
      tokio::runtime::task::UnownedTask<S>::run
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/task/mod.rs:431:9
      tokio::runtime::blocking::pool::Task::run
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/pool.rs:159:9
      tokio::runtime::blocking::pool::Inner::run
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/pool.rs:513:17
      tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.27.0/src/runtime/blocking/pool.rs:471:13
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys_common/backtrace.rs:121:18
  12: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/thread/mod.rs:558:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panic.rs:140:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/thread/mod.rs:557:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/ops/function.rs:250:5
  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys/unix/thread.rs:108:17
  14: start_thread
  15: clone

Server version

curl 0:5000/debug/profile/cpu/1

Steps to reproduce

curl 0:5000/debug/profile/cpu/1

Expected behavior

CeresDB Server
Version: 1.2.2
Git commit: d26e958
Git branch: main
Opt level: 3
Rustc version: 1.69.0-nightly
Target: x86_64-unknown-linux-gnu
Build date: 2023-06-19T09:29:34.376369881Z

Additional Information

No response

@Rachelint Rachelint added the bug Something isn't working label Jun 20, 2023
@jiacai2050
Copy link
Contributor

https://github.com/gimli-rs/findshlibs/blob/0.10.2/src/lib.rs#L261

There is already an issue for this gimli-rs/findshlibs#77

But unfortunately no progress on this.

ShiKaiWi pushed a commit that referenced this issue Jul 20, 2023
## Rationale
maybe close #1008

## Detailed Changes
Updated version due to bugs in upstream `findshlibs`

## Test Plan
manual
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working upstream
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants