Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve logging for exceptions #4942

Merged
merged 3 commits into from
Sep 18, 2024
Merged

Improve logging for exceptions #4942

merged 3 commits into from
Sep 18, 2024

Conversation

srikanthjg
Copy link
Contributor

@srikanthjg srikanthjg commented Sep 12, 2024 •
edited
Loading

Description

Add noisy marker to logs that occur repeatedly.

Notes:
setCause(e) is specifically for logging an exception with its full stack trace.
addArgument(e) treats the exception as a regular argument, and typically logs only the exception’s message (or whatever toString() provides).

Issues Resolved

Resolves #4927

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@srikanthjg
Improve logging for exceptions
e0adff4 
Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>
dlvenable
dlvenable requested changes Sep 12, 2024
View reviewed changes
data-prepper-api/src/main/java/org/opensearch/dataprepper/logging/DataPrepperMarkers.java Outdated

static {
EVENT.add(SENSITIVE);
NOISY.add(SENSITIVE);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want this relationship. A log may be noisy without being sensitive.

...ugins/common/src/main/java/org/opensearch/dataprepper/plugins/processor/StringProcessor.java Outdated
@@ -84,7 +83,7 @@ public Collection<Record<Event>> execute(final Collection<Record<Event>> records
.build();
modifiedRecords.add(new Record<>(newRecordEvent));
} catch (JsonProcessingException e) {
LOG.error(EVENT, "Unable to process Event data: {}", eventJson, e);
LOG.error(NOISY, "Unable to process Event data: {}", eventJson, e);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update this to use fluent logging such that it applies both markers.

See the description in #4927.

It will probably look like:

LOG.atError()
  .addMarker(SENSITIVE)
  .addMarker(NOISY)
  .setMessage("Unable to process Event data: {}")
  .addArgument(eventJson)
  .addArgument(e)
  .log();

...v-processor/src/main/java/org/opensearch/dataprepper/plugins/processor/csv/CsvProcessor.java Outdated
@@ -105,7 +104,7 @@ public Collection<Record<Event>> doExecute(final Collection<Record<Event>> recor
}
} catch (final IOException e) {
csvInvalidEventsCounter.increment();
LOG.error(EVENT, "An exception occurred while reading event [{}]", event, e);
LOG.error(NOISY, "An exception occurred while reading event [{}]", event, e);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

...sor/src/main/java/org/opensearch/dataprepper/plugins/processor/dissect/DissectProcessor.java Outdated
@@ -72,7 +72,7 @@ public Collection<Record<Event>> doExecute(Collection<Record<Event>> records) {
}
}
} catch (Exception ex){
LOG.error(EVENT, "Error dissecting the event [{}] ", record.getData(), ex);
LOG.error(NOISY, "Error dissecting the event [{}] ", record.getData(), ex);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

...processor/src/main/java/org/opensearch/dataprepper/plugins/processor/grok/GrokProcessor.java Outdated
@@ -153,11 +152,11 @@ public Collection<Record<Event>> doExecute(final Collection<Record<Event>> recor

} catch (final TimeoutException e) {
event.getMetadata().addTags(tagsOnTimeout);
LOG.error(EVENT, "Matching on record [{}] took longer than [{}] and timed out", record.getData(), grokProcessorConfig.getTimeoutMillis());
LOG.error(NOISY, "Matching on record [{}] took longer than [{}] and timed out", record.getData(), grokProcessorConfig.getTimeoutMillis());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

...c/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/MapToListProcessor.java Outdated
recordEvent.getMetadata().addTags(config.getTagsOnFailure());
}
} catch (final Exception e) {
LOG.error(EVENT, "There was an exception while processing Event [{}]", recordEvent, e);
LOG.error(NOISY, "There was an exception while processing Event [{}]", recordEvent, e);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

...c/main/java/org/opensearch/dataprepper/plugins/processor/mutateevent/RenameKeyProcessor.java Outdated
@@ -70,7 +69,7 @@ public Collection<Record<Event>> doExecute(final Collection<Record<Event>> recor
}
}
} catch (final Exception e) {
LOG.error(EVENT, "There was an exception while processing Event [{}]", recordEvent, e);
LOG.error(NOISY, "There was an exception while processing Event [{}]", recordEvent, e);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need both the log statements for the same error?

.../java/org/opensearch/dataprepper/plugins/processor/mutatestring/AbstractStringProcessor.java
@@ -37,7 +36,7 @@ public Collection<Record<Event>> doExecute(final Collection<Record<Event>> recor
try {
performStringAction(recordEvent);
} catch (final Exception e) {
LOG.error(EVENT, "There was an exception while processing Event [{}]", recordEvent, e);
LOG.error(NOISY, "There was an exception while processing Event [{}]", recordEvent, e);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need both the log lines for the same message?

...src/main/java/org/opensearch/dataprepper/plugins/processor/translate/TranslateProcessor.java Outdated
@@ -67,7 +66,7 @@ public Collection<Record<Event>> doExecute(Collection<Record<Event>> records) {
translateSource(sourceObject, recordEvent, targetConfig);
}
} catch (Exception ex) {
LOG.error(EVENT, "Error mapping the source [{}] of entry [{}]", mappingConfig.getSource(),
LOG.error(NOISY, "Error mapping the source [{}] of entry [{}]", mappingConfig.getSource(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment about using fluent logging to apply both markers.

...lugins/common/src/main/java/org/opensearch/dataprepper/plugins/processor/DelayProcessor.java Outdated
@@ -34,7 +35,7 @@ public Collection<Record<?>> execute(final Collection<Record<?>> records) {
try {
Thread.sleep(delayDuration.toMillis());
} catch (final InterruptedException ex) {
LOG.error("Interrupted during delay processor", ex);
LOG.error(NOISY,"Interrupted during delay processor", ex);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the formatting to include a space after the comma. ...NOISY, "Interrupted....

@srikanthjg
Address comments
a444078 
Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>
dlvenable
dlvenable previously approved these changes Sep 14, 2024
View reviewed changes
sb2k16
sb2k16 reviewed Sep 17, 2024
View reviewed changes
...main/java/org/opensearch/dataprepper/plugins/processor/obfuscation/ObfuscationProcessor.java
@@ -99,7 +99,12 @@ public ObfuscationProcessor(final PluginMetrics pluginMetrics,
Pattern p = Pattern.compile(rawPattern);
patterns.add(p);
} catch (Exception e) {
LOG.error(e.getMessage());
LOG.error(NOISY,e.getMessage());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.

sb2k16
sb2k16 requested changes Sep 17, 2024
View reviewed changes
Copy link
Member

@sb2k16 sb2k16 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a few comments. Please take a look.

@srikanthjg
Address duplicate log comment
718081a 
Signed-off-by: Srikanth Govindarajan <srigovs@amazon.com>
@dlvenable dlvenable merged commit 87c560a into opensearch-project:main Sep 18, 2024
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sb2k16 sb2k16 sb2k16 approved these changes

@san81 san81 san81 approved these changes

@dlvenable dlvenable dlvenable approved these changes

@chenqi0805 chenqi0805 Awaiting requested review from chenqi0805 chenqi0805 is a code owner

@engechas engechas Awaiting requested review from engechas engechas is a code owner

@graytaylor0 graytaylor0 Awaiting requested review from graytaylor0 graytaylor0 is a code owner

@dinujoh dinujoh Awaiting requested review from dinujoh dinujoh is a code owner

@kkondaka kkondaka Awaiting requested review from kkondaka kkondaka is a code owner

@KarstenSchnitter KarstenSchnitter Awaiting requested review from KarstenSchnitter KarstenSchnitter is a code owner

@oeyh oeyh Awaiting requested review from oeyh oeyh is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add a NOISY SLF4J marker
4 participants
@srikanthjg @dlvenable @san81 @sb2k16