Add support for removing HTML elements using XPath selectors

[michaelmcmillan]

Similarly to how XPath selectors can be used to extract matching HTML elements, it would also be useful to remove HTML elements that match an XPath selector.

[dgtlmoon]

thanks so much :)

[michaelmcmillan]

@dgtlmoon: Added support for those now. Mind taking another look?

[dgtlmoon]

sorry about the test failure, it is unrelated to your changes

[dgtlmoon]

sorry, one more change, can you change the logic so its the same as this

• use futures so that we capture any memory flood caused by anything that used LXML
• use the same logic for finding what xpath filter to use (because xpath 2+ uses a different implementation)

xpath: or // assume xpath2+ compat

xpath1: should be only xpath1

changedetection.io/changedetectionio/processors/text_json_diff/processor.py

Lines 174 to 202 in 19f3851

	for filter_rule in include_filters_rule:
	# For HTML/XML we offer xpath as an option, just start a regular xPath "/.."
	if filter_rule[0] == '/' or filter_rule.startswith('xpath:'):
	with ProcessPoolExecutor() as executor:
	# Use functools.partial to create a callable with arguments - anything using bs4/lxml etc is quite "leaky"
	future = executor.submit(partial(html_tools.xpath_filter, xpath_filter=filter_rule.replace('xpath:', ''),
	html_content=self.fetcher.content,
	append_pretty_line_formatting=not watch.is_source_type_url,
	is_rss=is_rss))
	html_content += future.result()
	elif filter_rule.startswith('xpath1:'):
	with ProcessPoolExecutor() as executor:
	# Use functools.partial to create a callable with arguments - anything using bs4/lxml etc is quite "leaky"
	future = executor.submit(partial(html_tools.xpath1_filter, xpath_filter=filter_rule.replace('xpath1:', ''),
	html_content=self.fetcher.content,
	append_pretty_line_formatting=not watch.is_source_type_url,
	is_rss=is_rss))
	html_content += future.result()
	else:
	with ProcessPoolExecutor() as executor:
	# Use functools.partial to create a callable with arguments - anything using bs4/lxml etc is quite "leaky"
	# CSS Filter, extract the HTML that matches and feed that into the existing inscriptis::get_text
	future = executor.submit(partial(html_tools.include_filters, include_filters=filter_rule,
	html_content=self.fetcher.content,
	append_pretty_line_formatting=not watch.is_source_type_url))
	html_content += future.result()
	if not html_content.strip():

[dgtlmoon]

@michaelmcmillan please update your fork with our master and repush

[michaelmcmillan]

sorry, one more change, can you change the logic so its the same as this

• use futures so that we capture any memory flood caused by anything that used LXML
• use the same logic for finding what xpath filter to use (because xpath 2+ uses a different implementation)

xpath: or // assume xpath2+ compat

xpath1: should be only xpath1

That was easier said than done. Been going at it for two hours, just getting errors... Can you give it a shot?