Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ConfigItem::CommitNewItems(): pre-sort types by their load dependencies once #10003

Closed
wants to merge 2 commits into from
Closed

ConfigItem::CommitNewItems(): pre-sort types by their load dependencies once #10003

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4037b60
Introduce Type::GetAllConfigTypesSortedByLoadDependencies()
Al2Klimov Sep 2, 2024
a422419
ConfigItem::CommitNewItems(): pre-sort types by their load dependenci…
Al2Klimov Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions lib/base/initialize.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ enum class InitializePriority {
RegisterBuiltinTypes,
RegisterFunctions,
RegisterTypes,
SortTypes,
EvaluateConfigFragments,
Default,
FreezeNamespaces,
Expand Down
66 changes: 66 additions & 0 deletions lib/base/type.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
/* Icinga 2 | (c) 2012 Icinga GmbH | GPLv2+ */

#include "base/type.hpp"
#include "base/atomic.hpp"
#include "base/configobject.hpp"
#include "base/debug.hpp"
#include "base/scriptglobal.hpp"
#include "base/namespace.hpp"
#include "base/objectlock.hpp"
#include <algorithm>
#include <functional>

using namespace icinga;

Expand Down Expand Up @@ -32,6 +37,61 @@ INITIALIZE_ONCE_WITH_PRIORITY([]() {
Type::Register(type);
}, InitializePriority::RegisterTypeType);

static std::vector<Type::Ptr> l_SortedByLoadDependencies;
static Atomic l_SortingByLoadDependenciesDone (false);

INITIALIZE_ONCE_WITH_PRIORITY([] {
auto types (Type::GetAllTypes());

types.erase(std::remove_if(types.begin(), types.end(), [](auto& type) {
return !ConfigObject::TypeInstance->IsAssignableFrom(type);
}), types.end());
Comment on lines +46 to +48
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also inline this check into the visit callback, since you don't have to remove those unwanted types beforehand.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I need to VERIFY(sorted.size() == types.size());


// Depth-first search
std::unordered_set<Type*> unsorted;
std::unordered_set<Type*> visited;
std::vector<Type::Ptr> sorted;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to temporarily cache the sorted ones elsewhere than l_SortedByLoadDependencies? It is guarded by VERIFY(l_SortingByLoadDependenciesDone.load()); so why can't you put the sorted ones directly in their final location.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sorted var and the "atomic" std::swap() at the end (which I consider good practice) has no disadvantages IMAO.


for (auto type : types) {
unsorted.emplace(type.get());
}

std::function<void(Type*)> visit;
visit = ([&visit, &unsorted, &visited, &sorted](Type* type) {
if (unsorted.find(type) == unsorted.end()) {
return;
}
Comment on lines +61 to +63
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If, for whatever reason, you define in the *.ti files that a type should be loaded after a random type that is not registered in the Types namespace, then you can be sure that you will not get a Type* if you load it with Type#GetLoadDependencies(), but a nullptr instead. So I would drop unsorted entirely and perform a ...


VERIFY(visited.emplace(type).second); // Detect cycles in type dependencies

for (auto dep : type->GetLoadDependencies()) {
visit(dep);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... dep != nullptr instead. Because if you put this in the host.ti file, load_after awegqa24gtw34gtw34; it will try to find this bogus type via GetByName("awegqa24gtw34gtw34").get(), which obviously doesn't exist, and store a nullptr in the load dependencies set that is returned when you call host->GetLoadDependencies().

}

unsorted.erase(type);
sorted.emplace_back(type);
});

while (!unsorted.empty()) {
visit(*unsorted.begin());
}

VERIFY(sorted.size() == types.size());

visited.clear();

for (auto& t : sorted) {
for (auto dep : t->GetLoadDependencies()) {
VERIFY(visited.find(dep) != visited.end());
}

VERIFY(visited.emplace(t.get()).second);
}

std::swap(sorted, l_SortedByLoadDependencies);
l_SortingByLoadDependenciesDone.store(true);
}, InitializePriority::SortTypes);

String Type::ToString() const
{
return "type '" + GetName() + "'";
Expand Down Expand Up @@ -72,6 +132,12 @@ std::vector<Type::Ptr> Type::GetAllTypes()
return types;
}

const std::vector<Type::Ptr>& Type::GetAllConfigTypesSortedByLoadDependencies()
{
VERIFY(l_SortingByLoadDependenciesDone.load());
return l_SortedByLoadDependencies;
}

String Type::GetPluralName() const
{
String name = GetName();
Expand Down
1 change: 1 addition & 0 deletions lib/base/type.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ class Type : public Object
static void Register(const Type::Ptr& type);
static Type::Ptr GetByName(const String& name);
static std::vector<Type::Ptr> GetAllTypes();
static const std::vector<Ptr>& GetAllConfigTypesSortedByLoadDependencies();

void SetField(int id, const Value& value, bool suppress_events = false, const Value& cookie = Empty) override;
Value GetField(int id) const override;
Expand Down
201 changes: 76 additions & 125 deletions lib/config/configitem.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,180 +444,131 @@ bool ConfigItem::CommitNewItems(const ActivationContext::Ptr& context, WorkQueue
<< "Committing " << total << " new items.";
#endif /* I2_DEBUG */

std::set<Type::Ptr> types;
std::set<Type::Ptr> completed_types;
int itemsCount {0};

for (const Type::Ptr& type : Type::GetAllTypes()) {
if (ConfigObject::TypeInstance->IsAssignableFrom(type))
types.insert(type);
}

while (types.size() != completed_types.size()) {
for (const Type::Ptr& type : types) {
if (completed_types.find(type) != completed_types.end())
continue;
for (auto& type : Type::GetAllConfigTypesSortedByLoadDependencies()) {
std::atomic<int> committed_items(0);

bool unresolved_dep = false;
{
auto items (itemsByType.find(type.get()));

/* skip this type (for now) if there are unresolved load dependencies */
for (auto pLoadDep : type->GetLoadDependencies()) {
if (types.find(pLoadDep) != types.end() && completed_types.find(pLoadDep) == completed_types.end()) {
unresolved_dep = true;
break;
if (items != itemsByType.end()) {
for (const ItemPair& pair: items->second) {
newItems.emplace_back(pair.first);
}
}

if (unresolved_dep)
continue;

std::atomic<int> committed_items(0);
upq.ParallelFor(items->second, [&committed_items](const ItemPair& ip) {
const ConfigItem::Ptr& item = ip.first;

{
auto items (itemsByType.find(type.get()));
if (!item->Commit(ip.second)) {
if (item->IsIgnoreOnError()) {
item->Unregister();
}

if (items != itemsByType.end()) {
for (const ItemPair& pair: items->second) {
newItems.emplace_back(pair.first);
return;
}

upq.ParallelFor(items->second, [&committed_items](const ItemPair& ip) {
const ConfigItem::Ptr& item = ip.first;
committed_items++;
});

if (!item->Commit(ip.second)) {
if (item->IsIgnoreOnError()) {
item->Unregister();
}

return;
}

committed_items++;
});

upq.Join();
}
upq.Join();
}
}

itemsCount += committed_items;

completed_types.insert(type);
itemsCount += committed_items;

#ifdef I2_DEBUG
if (committed_items > 0)
Log(LogDebug, "configitem")
<< "Committed " << committed_items << " items of type '" << type->GetName() << "'.";
if (committed_items > 0)
Log(LogDebug, "configitem")
<< "Committed " << committed_items << " items of type '" << type->GetName() << "'.";
#endif /* I2_DEBUG */

if (upq.HasExceptions())
return false;
}
if (upq.HasExceptions())
return false;
}

#ifdef I2_DEBUG
Log(LogDebug, "configitem")
<< "Committed " << itemsCount << " items.";
#endif /* I2_DEBUG */

completed_types.clear();
for (auto& type : Type::GetAllConfigTypesSortedByLoadDependencies()) {
std::atomic<int> notified_items(0);

while (types.size() != completed_types.size()) {
for (const Type::Ptr& type : types) {
if (completed_types.find(type) != completed_types.end())
continue;
{
auto items (itemsByType.find(type.get()));

bool unresolved_dep = false;
if (items != itemsByType.end()) {
upq.ParallelFor(items->second, [&notified_items](const ItemPair& ip) {
const ConfigItem::Ptr& item = ip.first;

/* skip this type (for now) if there are unresolved load dependencies */
for (auto pLoadDep : type->GetLoadDependencies()) {
if (types.find(pLoadDep) != types.end() && completed_types.find(pLoadDep) == completed_types.end()) {
unresolved_dep = true;
break;
}
}
if (!item->m_Object)
return;

if (unresolved_dep)
continue;

std::atomic<int> notified_items(0);

{
auto items (itemsByType.find(type.get()));

if (items != itemsByType.end()) {
upq.ParallelFor(items->second, [&notified_items](const ItemPair& ip) {
const ConfigItem::Ptr& item = ip.first;

if (!item->m_Object)
return;
try {
item->m_Object->OnAllConfigLoaded();
notified_items++;
} catch (const std::exception& ex) {
if (!item->m_IgnoreOnError)
throw;

try {
item->m_Object->OnAllConfigLoaded();
notified_items++;
} catch (const std::exception& ex) {
if (!item->m_IgnoreOnError)
throw;
Log(LogNotice, "ConfigObject")
<< "Ignoring config object '" << item->m_Name << "' of type '" << item->m_Type->GetName() << "' due to errors: " << DiagnosticInformation(ex);

Log(LogNotice, "ConfigObject")
<< "Ignoring config object '" << item->m_Name << "' of type '" << item->m_Type->GetName() << "' due to errors: " << DiagnosticInformation(ex);
item->Unregister();

item->Unregister();

{
std::unique_lock<std::mutex> lock(item->m_Mutex);
item->m_IgnoredItems.push_back(item->m_DebugInfo.Path);
}
{
std::unique_lock<std::mutex> lock(item->m_Mutex);
item->m_IgnoredItems.push_back(item->m_DebugInfo.Path);
}
});
}
});

upq.Join();
}
upq.Join();
}

completed_types.insert(type);
}

#ifdef I2_DEBUG
if (notified_items > 0)
Log(LogDebug, "configitem")
<< "Sent OnAllConfigLoaded to " << notified_items << " items of type '" << type->GetName() << "'.";
if (notified_items > 0)
Log(LogDebug, "configitem")
<< "Sent OnAllConfigLoaded to " << notified_items << " items of type '" << type->GetName() << "'.";
#endif /* I2_DEBUG */

if (upq.HasExceptions())
return false;
if (upq.HasExceptions())
return false;

notified_items = 0;
for (auto loadDep : type->GetLoadDependencies()) {
auto items (itemsByType.find(loadDep));
notified_items = 0;
for (auto loadDep : type->GetLoadDependencies()) {
auto items (itemsByType.find(loadDep));

if (items != itemsByType.end()) {
upq.ParallelFor(items->second, [&type, &notified_items](const ItemPair& ip) {
const ConfigItem::Ptr& item = ip.first;
if (items != itemsByType.end()) {
upq.ParallelFor(items->second, [&type, &notified_items](const ItemPair& ip) {
const ConfigItem::Ptr& item = ip.first;

if (!item->m_Object)
return;
if (!item->m_Object)
return;

ActivationScope ascope(item->m_ActivationContext);
item->m_Object->CreateChildObjects(type);
notified_items++;
});
}
ActivationScope ascope(item->m_ActivationContext);
item->m_Object->CreateChildObjects(type);
notified_items++;
});
}
}

upq.Join();
upq.Join();

#ifdef I2_DEBUG
if (notified_items > 0)
Log(LogDebug, "configitem")
<< "Sent CreateChildObjects to " << notified_items << " items of type '" << type->GetName() << "'.";
if (notified_items > 0)
Log(LogDebug, "configitem")
<< "Sent CreateChildObjects to " << notified_items << " items of type '" << type->GetName() << "'.";
#endif /* I2_DEBUG */

if (upq.HasExceptions())
return false;
if (upq.HasExceptions())
return false;

// Make sure to activate any additionally generated items
if (!CommitNewItems(context, upq, newItems))
return false;
}
// Make sure to activate any additionally generated items
if (!CommitNewItems(context, upq, newItems))
return false;
}

return true;
Expand Down
Loading