Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running many Halo-enabled processes at once #23

Open
kavon opened this issue Jan 24, 2020 · 0 comments
Open

Running many Halo-enabled processes at once #23

kavon opened this issue Jan 24, 2020 · 0 comments
Labels
limitation Something that works but in a limited way

Comments

@kavon
Copy link
Member

kavon commented Jan 24, 2020

We run into some system-wide perf_events buffer limitations when many halo-enabled processes are running at once. See the comment here: halo-project/llvm@e389423

We had to change the noop test to not spawn 50 clients on the same system because of a pretty low default kernel limit on the number of pages that can be used for perf_events buffers. Now that we have to create one buffer per core, it's quite possible to hit this limit without some care.

One way around this problem would be to only gather sampling data from some CPUs and/or reduce the number of pages in the ring buffer. We can also find out how to increase /proc/sys/kernel/perf_event_mlock_kb and tell users to follow those instructions.

kavon added a commit that referenced this issue Jan 24, 2020
@kavon kavon changed the title System-wide perf_events buffer limitations Running many Halo-enabled processes at once Jan 24, 2020
@kavon kavon added the limitation Something that works but in a limited way label Jan 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
limitation Something that works but in a limited way
Projects
None yet
Development

No branches or pull requests

1 participant