[core] shm-monitoring-performance #1645

rex-schilasky · 2024-07-04T07:50:38Z

Description

shm_monitoring logic simplified, based on "network_enabled" only
parallel shm_monitoring AND udp_monitoring feature removed massive_pub_sub sample added to test creation performance of pub/sub for large scenarios and monitoring performance
no more remvoval of expired values (erase_expired) in ApplyPub/Sub/Server/ClientDescription because of bad performance

parallel shm_monitoring AND udp_monitoring feature removed massive_pub_sub sample added to test creation performance of pub/sub for large scenarios and monitoring performance no more remvoval of expired values (erase_expired) in ApplyPub/Sub/Server/ClientDescription because of bad performance

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-04T08:59:17Z

ecal/core/src/registration/ecal_registration_provider.cpp

-    m_use_registration_udp = !Config::Experimental::IsNetworkMonitoringDisabled();
-    m_use_registration_shm = Config::Experimental::IsShmMonitoringEnabled();
+    // send registration over udp or shared memory
+    m_use_registration_udp = Config::IsNetworkEnabled;


warning: implicit conversion 'bool (*)()' -> bool [readability-implicit-bool-conversion]

Suggested change

m_use_registration_udp = Config::IsNetworkEnabled;

m_use_registration_udp = (Config::IsNetworkEnabled != nullptr);

Config::IsNetworkEnabled() 😨

github-actions · 2024-07-04T08:59:18Z

ecal/core/src/registration/ecal_registration_receiver.cpp

-    m_use_registration_udp = !Config::Experimental::IsNetworkMonitoringDisabled();
-    m_use_registration_shm     = Config::Experimental::IsShmMonitoringEnabled();
+    // receive registration via udp or shared memory
+    m_use_registration_udp = Config::IsNetworkEnabled;


warning: implicit conversion 'bool (*)()' -> bool [readability-implicit-bool-conversion]

Suggested change

m_use_registration_udp = Config::IsNetworkEnabled;

m_use_registration_udp = (Config::IsNetworkEnabled != nullptr);

@rex-schilasky 🤣 this is really bad, did this not generate a regular warning?

…emove expired values)

…ation refresh)

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-04T13:22:27Z

ecal/tests/cpp/util_test/src/util_getclients.cpp

@@ -22,8 +22,8 @@
 #include <gtest/gtest.h>


warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]

#include <gtest/gtest.h> ^

KerstinKeller

I think we should go a bit further in a next PR. ;)

KerstinKeller · 2024-07-05T09:27:54Z

ecal/core/src/registration/ecal_registration_provider.cpp

-    m_use_registration_shm = Config::Experimental::IsShmMonitoringEnabled();
+    // send registration over udp or shared memory
+    m_use_registration_udp = Config::IsNetworkEnabled();
+    m_use_registration_shm = !m_use_registration_udp;



I'm a bit confused. shouldn't the registration provider / receiver also be passed configuration objects (maybe just registration configuration) in their constructors? But maybe that's for a different PR.

…s pr run)

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-09T08:41:12Z

ecal/core/include/ecal/config/configuration.h

+
+  private:
+    ECAL_API void Init(std::vector<std::string>& args_);


warning: function 'eCAL::Configuration::Init' has a definition with different parameter names [readability-inconsistent-declaration-parameter-name]

ECAL_API void Init(std::vector<std::string>& args_); ^

Additional context

ecal/core/src/config/ecal_config_initializer.cpp:236: the definition seen here

void Configuration::Init(std::vector<std::string>& arguments_) ^

ecal/core/include/ecal/config/configuration.h:77: differing parameters are named here: ('args_'), in definition: ('arguments_')

ECAL_API void Init(std::vector<std::string>& args_); ^

…tions)

commit e13cadb Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Fri Aug 23 14:30:08 2024 +0200 [config] Move host_group_name logic (eclipse-ecal#1720) * Removed HostName in registration config, added usage in attribute builder. ecal/tests/cpp/pubsub_proto_test/src/commit a410305 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Fri Aug 23 14:27:54 2024 +0200 [core] Bugfix: Publishers / Subscribers need to use global configuration, when no configuration is provided. (eclipse-ecal#1721) commit 4f8bad0 Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Thu Aug 22 12:56:37 2024 +0200 [config] Removed quoteStrings and unused variable in config test. (eclipse-ecal#1718) commit bfea308 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Aug 20 16:17:23 2024 +0200 [core] remove monitoring timeout (registration timeout is only valid timeout). Set registration timeout to 10 seconds. (eclipse-ecal#1714) commit bda08a6 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Tue Aug 20 14:37:30 2024 +0200 python monitoring dictionary enhanced (still not complete) (eclipse-ecal#1715) commit 5779c07 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Tue Aug 20 12:00:45 2024 +0200 [sample] sample massive_pub_sub extended (eclipse-ecal#1713) commit d4fd7d6 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Aug 20 11:49:19 2024 +0200 [core] Rewrite GetTopicsParallel test to be more robust. (eclipse-ecal#1706) commit 9db33c8 Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Tue Aug 20 10:05:06 2024 +0200 [config] Monitoring uses SAttributes instead of config object (eclipse-ecal#1710) commit 998b3ff Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Tue Aug 20 09:01:06 2024 +0200 [GH] Hotfix macos build & dependency (Qt5, capnp, python) brew installation (eclipse-ecal#1711) * Python, Qt5, capnp installation via brew (all latest) * Python packages installation with parameter --break-system-packages * Added Qt5 installation path to the CMAKE_PREFIX_PATH in installation commit c974e6a Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Aug 20 08:44:53 2024 +0200 [core] eCAL sample completely internal, only SEntityId public. (eclipse-ecal#1712) commit 6a44dad Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Mon Aug 19 16:49:44 2024 +0200 [core] feature/id-based-descgate (eclipse-ecal#1708) descgate API changed to use separate GetEntityIDs() + GetEntityInfo(id) functions instead of single GetEntities() variants commit fe1fe22 Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Mon Aug 19 15:39:17 2024 +0200 [config] Added config object to registration. (eclipse-ecal#1709) * Added Config object to registration. commit f80ce01 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Aug 13 15:14:56 2024 +0200 [core] warnings and build issue fixes (eclipse-ecal#1707) commit 938c891 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Mon Aug 12 12:54:52 2024 +0200 [core] unregistration of timed out samples in one central place (instead of all over the place). (eclipse-ecal#1675) Modifies also CExpirationMap interface to return keys and values of expired items. commit 09d2cc8 Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Thu Aug 8 12:11:14 2024 +0200 [config] Added config object to monitoring initialization. (eclipse-ecal#1699) * Added config object to monitoring initialization commit ab0870c Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Thu Aug 8 11:19:31 2024 +0200 [core] registration specific functions moved from Util:: to Registration:: (eclipse-ecal#1700) * registration specific functions moved from Util:: to Registration:: commit 2b2e4d7 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Wed Aug 7 13:39:01 2024 +0200 [core] enable loopback for monitoring functionality (eclipse-ecal#1698) commit 0277978 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Wed Aug 7 09:52:22 2024 +0200 [core] Remove SPublicationInfo / SSubscriptionInfo type. (eclipse-ecal#1695) Instead Registration::SampleIdentifier are used in datawriter / datareader. commit 5a4300b Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Aug 6 15:34:24 2024 +0200 [core] Registration::SampleIdentifier and partial sample serialization eCAL Registration samples have a command type, and some member types. Now the serialization is performed only partially, according to the command type. Additionally, every sample now has a unique ID, marking the entity which has produced this sample. This info will be unique within the whole eCAL Ecosystem. commit 6886ee4 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Tue Aug 6 15:06:48 2024 +0200 [core] remove callbacks before stopping tcp protocol layer (eclipse-ecal#1693) commit 9e8ecbb Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Mon Aug 5 08:22:09 2024 +0200 [config] Updated documentation to new configuration structure (eclipse-ecal#1690) Only doc changes. New yaml references and config settings in code snippets. Issue link for current documentation checkup: eclipse-ecal#1672 commit e6dbb82 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Fri Aug 2 16:14:31 2024 +0200 [core] missing include (std::find) (eclipse-ecal#1691) commit a5d9eb4 Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Fri Aug 2 12:12:44 2024 +0200 [config] configuration yaml generation (eclipse-ecal#1680) * Config initialization happens now directly in structure. * Removed eCAL defs except for non configurable defs. * Adapted subscriber configuration to layer::udp as in other structs. * Added yaml creation out of a stringstream while building. * Made yaml-cpp optional again. Added compiler flags depending on ECAL_CORE_CONFIGURATION option to exclude yaml-cpp code. commit 63d1cc2 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Tue Jul 30 16:04:12 2024 +0200 [core] zero-copy-zero-payload-shm-transfer (eclipse-ecal#1683) * zero length payload was not transferred over shm in zero copy mode commit 9b4c0d3 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Jul 30 14:53:07 2024 +0200 [core] Protobuf Publisher Send should return the actual send size. (eclipse-ecal#1681) commit 8f80696 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Mon Jul 29 17:05:01 2024 +0200 [build] ensure compatibility with yaml-cpp < 0.8.0 (eclipse-ecal#1678) commit 0b60453 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Mon Jul 29 14:14:25 2024 +0200 [core] registration-provider-refactoring (eclipse-ecal#1677) * registration provider single threaded sending* * Register(false) calls removed finally * registration provider logic reduced to just send registration samples (connection to descgate, monitoring cut) commit 9110d2a Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Fri Jul 26 16:47:21 2024 +0200 [core] registration-sender-datarace-fix (eclipse-ecal#1674) commit 2c99f61 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Fri Jul 26 16:41:52 2024 +0200 [core] serialization-decode-error-handling (eclipse-ecal#1673) commit 0d361d0 Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Fri Jul 26 14:27:00 2024 +0200 [config] Introduction of YAML format for eCAL configuration (eclipse-ecal#1669) * Changed runtime configuration format from ini to yaml * Implementation of yaml reader * Removing the simple-ini dependency + implementation from core * Implementation of defaults via yaml reading commit 9b5f082 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Wed Jul 24 18:43:05 2024 +0200 [core] active-registration-logic-getter (eclipse-ecal#1671) commit 9211a88 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Wed Jul 24 15:15:01 2024 +0200 [core] further refactoring registration receiver. (eclipse-ecal#1670) commit 21535da Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Mon Jul 22 13:39:01 2024 +0200 [core] registration-loopback-logic-fixed-for-service-registrations (eclipse-ecal#1667) * registration sample processing (local/network mode, loopback not loopback) handle identically for pub/sub and client/server registration Co-authored-by: Kerstin Keller <KerstinKeller@users.noreply.github.com> commit 3441e62 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Thu Jul 18 14:56:46 2024 +0200 [core] new-pub-sub-matching-compatible (eclipse-ecal#1665) commit 4659192 Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Wed Jul 17 18:01:20 2024 +0200 [core] new-pub-sub-matching (eclipse-ecal#1653) Co-authored-by: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> commit 73f5f3c Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Wed Jul 17 15:04:26 2024 +0200 [core] refactor registration receiver and file structure (eclipse-ecal#1662) commit 15d8354 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Wed Jul 17 14:47:05 2024 +0200 [core] sync-with-ecal-core-2024-07-17 (eclipse-ecal#1664) commit 5a835aa Author: Peguen <73380451+Peguen@users.noreply.github.com> Date: Mon Jul 15 16:10:32 2024 +0200 [config] cleanup (eclipse-ecal#1655) * Removed unused methods from ecal_config.h * Removed unused member from transport_layer configuration. * Moved num_executor_reader/writer to publisher/subscriber config. * Added new num_execs to API and uses subscriber options for previous implementation calls. commit 6dd1718 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Mon Jul 15 16:04:30 2024 +0200 kill XProtect to avoid malware scan of dmg image (eclipse-ecal#1659) commit a51ecec Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Mon Jul 15 11:48:23 2024 +0200 [core] innosetup-configuration-path-fix (eclipse-ecal#1656) * innosetup path fix "configuration" -> "_configuration" * innosetup path fix "configinstall" -> "_configinstall" commit 4d67a9f Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Fri Jul 12 10:27:08 2024 +0200 [core] registration provider refactoring (eclipse-ecal#1647) * Split logic for registration providers into different files. commit a58f6af Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Wed Jul 10 17:16:51 2024 +0200 [core] Fix bug in frequency calculator that will reset the frequency to 0.0 even though data is still incoming. (eclipse-ecal#1650) commit bd3bfc3 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Tue Jul 9 14:04:08 2024 +0200 [core] shm-monitoring-performance (eclipse-ecal#1645) * shm_monitoring logic simplified, based on on switch only (needs to be improved further) * parallel shm_monitoring AND udp_monitoring feature removed * massive_pub_sub sample added to test creation performance of pub/sub for large scenarios and monitoring performance no more remvoval of expired values (erase_expired) in ApplyPub/Sub/Server/ClientDescription because of bad performance * erase_expired removal removed (we need to think about when/where to remove expired values) * shm registration configurable separately (interim solution to get this pr run) commit 7b47978 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Fri Jun 28 11:46:39 2024 +0200 [samples] shutdown condition fixed (eclipse-ecal#1641) commit 35c1fd9 Author: Rex Schilasky <49162693+rex-schilasky@users.noreply.github.com> Date: Tue Jun 25 15:06:32 2024 +0200 [core] new IsPublished API for CSubscriber, improved state logic for IsPublished and GetPublisherCount commit 55eb50f Author: KerstinKeller <KerstinKeller@users.noreply.github.com> Date: Tue Jun 25 15:04:08 2024 +0200 [core] Refactoring / Renaming / Documentation of CExpMap (eclipse-ecal#1637)

rex-schilasky added the cherry-pick-to-NONE Don't cherry-pick these changes label Jul 4, 2024

config sample and test fixed

51657a4

github-actions bot reviewed Jul 4, 2024

View reviewed changes

rex-schilasky added 4 commits July 4, 2024 11:56

erase_expired removal removed (we need to think about when/where to r…

70b93ba

…emove expired values)

a function should be called like a function ;-)

2100496

ecal.ini shm monitoring settings removed

ba295e3

test timings adapted (timeout for registration set to 2 times registr…

8e8d9ee

…ation refresh)

github-actions bot reviewed Jul 4, 2024

View reviewed changes

network_enabled configuration moved to registration group

0b9cbe1

KerstinKeller approved these changes Jul 5, 2024

View reviewed changes

shm registration configurable separately (interim solution to get thi…

7618288

…s pr run)

github-actions bot reviewed Jul 9, 2024

View reviewed changes

config test fixed (network enabled no more part of transport_layer op…

6753be5

…tions)

rex-schilasky merged commit bd3bfc3 into master Jul 9, 2024
20 checks passed

rex-schilasky deleted the hotfix/shm_monitoring_performance branch July 9, 2024 12:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] shm-monitoring-performance #1645

[core] shm-monitoring-performance #1645

rex-schilasky commented Jul 4, 2024

github-actions bot left a comment

github-actions bot Jul 4, 2024

KerstinKeller Jul 4, 2024

github-actions bot Jul 4, 2024

KerstinKeller Jul 4, 2024

rex-schilasky Jul 4, 2024

github-actions bot left a comment

github-actions bot Jul 4, 2024

KerstinKeller left a comment

KerstinKeller Jul 5, 2024

github-actions bot left a comment

github-actions bot Jul 9, 2024

	m_use_registration_udp = Config::IsNetworkEnabled;
	m_use_registration_udp = (Config::IsNetworkEnabled != nullptr);


		private:
		ECAL_API void Init(std::vector<std::string>& args_);

[core] shm-monitoring-performance #1645

[core] shm-monitoring-performance #1645

Conversation

rex-schilasky commented Jul 4, 2024

Description

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

KerstinKeller Jul 4, 2024

Choose a reason for hiding this comment

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

KerstinKeller Jul 4, 2024

Choose a reason for hiding this comment

rex-schilasky Jul 4, 2024

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

KerstinKeller left a comment

Choose a reason for hiding this comment

KerstinKeller Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jul 9, 2024

Choose a reason for hiding this comment