Skip to content

Fix excluded tags lookup to use correct key type #417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seanstory merged 2 commits into from
Feb 6, 2026
Merged

Fix excluded tags lookup to use correct key type #417

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
72dc46f
Fix excluded tags lookup to use right type for key
seanstory Feb 6, 2026
8fa3ec5
Fix excluded tags lookup to use right type for key
seanstory Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion lib/crawler/data/crawl_result/html.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,10 @@ def headings(limit: 10)

def get_body_tag(exclude_tags)
exclude_tags ||= {}
tags_to_exclude_for_domain = exclude_tags.fetch(url, [])
# Config stores exclude_tags keyed by domain URL (e.g., "https://example.com")
# Try site first (scheme + host), which is the standard format
tags_to_exclude_for_domain = exclude_tags.fetch(url.site, nil) ||
exclude_tags.fetch(url.to_s, [])

if tags_to_exclude_for_domain.empty?
parsed_content.body
Expand Down
31 changes: 30 additions & 1 deletion spec/lib/crawler/data/crawl_result/html_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,13 +329,42 @@
end

context 'when given a list of tags to exclude' do
let(:body_text) { crawl_result.document_body(exclude_tags: { url => ['h1'] }) }
# Keys must be site strings (scheme + host) to match how config stores them
let(:body_text) { crawl_result.document_body(exclude_tags: { url.site => ['h1'] }) }

it 'should remove content associated with those tags, even if there is a data-elastic-include' do
expect(body_text).to_not match('Page header')
end
end

context 'when given a list of tags to exclude with string keys (as from config)' do
let(:html) do
<<~HTML
<html>
<body>
<header>HEADER TEXT Should not be indexed</header>
<h2>title</h2>
<p>BODY content</p>
<address>main street 123 to be ignored too</address>
<footer>FOOTER TEXT</footer>
</body>
</html>
HTML
end

# This reproduces the bug from https://github.com/elastic/crawler/issues/416
# The config stores exclude_tags with string keys (site URLs like "https://example.com"),
# so the lookup must use url.site to match the config format
let(:body_text) { crawl_result.document_body(exclude_tags: { url.site => %w[header address] }) }

it 'should remove content associated with those tags when keys are strings' do
expect(body_text).to_not match('HEADER TEXT')
expect(body_text).to_not match('main street')
expect(body_text).to match('BODY content')
expect(body_text).to match('FOOTER TEXT')
end
end

it 'should remove empty spaces from the content' do
expect(body_text).to match('Something something else')
end
Expand Down