feat: support for setting UA for URL previews #17968
Conversation
| Example configuration: | ||
| ```yaml | ||
| url_preview_user_agent: "Hello Matrix" | ||
| ``` |
There was a problem hiding this comment.
We should put this next to the other url_preview_xxx options
| or "Synapse (bot; +https://github.com/matrix-org/synapse)" | ||
| ) | ||
| if len(self.url_preview_user_agent.strip()) == 0: | ||
| raise ConfigError("The 'url_preview_user_agent' must be a valid User-Agent") |
There was a problem hiding this comment.
| raise ConfigError("The 'url_preview_user_agent' must be a valid User-Agent") | |
| raise ConfigError( | |
| "Must be a valid User-Agent string", | |
| ("url_preview_user_agent",), | |
| ) |
| self.url_preview_user_agent: str = ( | ||
| config.get("url_preview_user_agent") | ||
| or "Synapse (bot; +https://github.com/matrix-org/synapse)" | ||
| ) | ||
| if len(self.url_preview_user_agent.strip()) == 0: | ||
| raise ConfigError("The 'url_preview_user_agent' must be a valid User-Agent") | ||
|
|
There was a problem hiding this comment.
We should do this processing alongside the other url_preview_config in synapse/config/repository.py
| config.get("url_preview_user_agent") | ||
| or "Synapse (bot; +https://github.com/matrix-org/synapse)" |
There was a problem hiding this comment.
| config.get("url_preview_user_agent") | |
| or "Synapse (bot; +https://github.com/matrix-org/synapse)" | |
| config.get("url_preview_user_agent", "Synapse (bot; +https://github.com/matrix-org/synapse)") |
| b"User-Agent": [ | ||
| "Synapse (bot; +https://github.com/matrix-org/synapse)" | ||
| ], | ||
| b"User-Agent": [self.hs.config.server.url_preview_user_agent], |
There was a problem hiding this comment.
Similar to what we do for max_spider_size in __init__, we can do the same for url_preview_user_agent
| b"User-Agent": [self.hs.config.server.url_preview_user_agent], | |
| b"User-Agent": [self.url_preview_user_agent], |
| @@ -0,0 +1 @@ | |||
| Add support for setting User-Agent for URL previewing. | |||
There was a problem hiding this comment.
The option itself seems relatively straightforward. Our sane default feels like it should be good enough. I'm going to add this to the To-Discuss board to talk with the team on Monday whether we want to introduce this.
The workaround use case seems kinda meh but I appreciate the context. (adding yet another option for a problem that will go away/change in the future)
There was a problem hiding this comment.
Discussed with the team and we're not keen on pretending to be someone else. If someone doesn't want to serve content to Synapse, that's their prerogative.
This might be a XY problem. The goal is make YouTube URL previews work. Do they have some API/flow that they prefer people to use? Should we be using OpenGraph/oEmbed instead of scraping, etc? We'd rather have PR's for that instead of this workaround.
There was a problem hiding this comment.
he goal is make YouTube URL previews work. Do they have some API/flow that they prefer people to use? Should we be using OpenGraph/oEmbed instead of scraping, etc?
See #17462.
Fix #17873
Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.(run the linters)
close #17873