u3d available getting a 403

[a-maurice]

Issue Checklist

• I'm using the latest u3d version
• I ran u3d --help
• I read the README
• I made sure that a similar issue doesn't exit

Issue Description

Running u3d available from a Mac, I'm seeing 403 errors, though I'm not sure which underlying call is having the actual problem. I see there was a similar issue a couple years ago: #402, and I wonder if some page has been made unavailable recently. From a previous run a few days ago that did work, we had gotten results like:
Version 2020.3.41f1: https://download.unity3d.com/download_unity/7c19dc9acfda/
I see that the link gets access denied when I try it locally, I don't know if that was the case before or if that is new.

Current output from the command:

$ u3d available --trace --verbose
INFO Fetching central 'versions.json' cache
DEBUG Using http opts: {:use_ssl=>#<MatchData "https">, :read_timeout=>300}
WARN Cache is out of date. Updating cache for Mac OSX
INFO Loading Unity lts releases
DEBUG Using http opts: {:use_ssl=>#<MatchData "https">, :read_timeout=>300}
/Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/utils.rb:82:in `follow_redirects': Request failed with status 403 (RuntimeError)
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/utils.rb:51:in `page_content'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/utils.rb:47:in `get_ssl'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:168:in `fetch_version_paged'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:274:in `block in fetch_some'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:273:in `each'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:273:in `fetch_some'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:298:in `fetch_all_channels'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:313:in `list_available'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:151:in `list_available'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/cache.rb:140:in `update_cache'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/cache.rb:115:in `overwrite_cache'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/cache.rb:72:in `initialize'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands.rb:275:in `new'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands.rb:275:in `cache_versions'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands.rb:113:in `list_available'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands_generator.rb:142:in `block (2 levels) in run'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/command.rb:187:in `call'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/command.rb:157:in `run'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/runner.rb:444:in `run_active_command'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/runner.rb:58:in `run!'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/delegates.rb:18:in `run!'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands_generator.rb:265:in `run'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands_generator.rb:43:in `start'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/exe/u3d:9:in `<top (required)>'
	from /usr/local/bin/u3d:23:in `load'
	from /usr/local/bin/u3d:23:in `<main>'

[chrico84]

Unfortunately, we are also getting 403 error when using u3d.

No unity version specified. If the current directory is a Unity project, we try to install the one it requires

Fetching central 'versions.json' cache

�[33mCache is out of date. Updating cache for Windows�[0m

Loading Unity lts releases

error: Request failed with status 403. Use --trace to view backtrace

Failed with error 1.

[DellaBitta]

Same issue.

Reinstalling u3d doesn't seem to help.

$ u3d install --trace --verbose 2020.3.34f1 -p Unity,Android,iOS,tvOS
INFO Fetching central 'versions.json' cache
DEBUG Using http opts: {:use_ssl=>#<MatchData "https">, :read_timeout=>300}
WARN Cache is out of date. Updating cache for Mac OSX
INFO Loading Unity lts releases
DEBUG Using http opts: {:use_ssl=>#<MatchData "https">, :read_timeout=>300}
/Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/utils.rb:82:in `follow_redirects': Request failed with status 403 (RuntimeError)
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/utils.rb:51:in `page_content'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/utils.rb:47:in `get_ssl'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:168:in `fetch_version_paged'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:274:in `block in fetch_some'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:273:in `each'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:273:in `fetch_some'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:298:in `fetch_all_channels'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:313:in `list_available'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/unity_versions.rb:151:in `list_available'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/cache.rb:140:in `update_cache'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/cache.rb:115:in `overwrite_cache'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/cache.rb:72:in `initialize'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands.rb:275:in `new'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands.rb:275:in `cache_versions'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands.rb:148:in `install'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands_generator.rb:178:in `block (2 levels) in run'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/command.rb:187:in `call'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/command.rb:157:in `run'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/runner.rb:444:in `run_active_command'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/runner.rb:58:in `run!'
	from /Library/Ruby/Gems/2.6.0/gems/commander-4.6.0/lib/commander/delegates.rb:18:in `run!'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands_generator.rb:265:in `run'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/lib/u3d/commands_generator.rb:43:in `start'
	from /Library/Ruby/Gems/2.6.0/gems/u3d-1.3.2/exe/u3d:9:in `<top (required)>'
	from /usr/local/bin/u3d:25:in `load'
	from /usr/local/bin/u3d:25:in `<main>'

[mwegner]

This appears to be Unity blocking default Ruby user agent. This line can set a real user agent:

https://github.com/DragonBox/u3d/blob/master/lib/u3d/utils.rb#L46

i.e. this works for me:

def get_ssl(url, redirect_limit: 10, request_headers: {'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0'})

[chrico84]

Nice find. Thanks, it also works for me.

[niezbop]

Thanks @mwegner for the find! I've opened a PR that does mostly the same, I'll try to release this fix ASAP!

[vladyslav-androshchuk]

Hi! May I ask for an approximate release date with a fix?

[vladyslav-androshchuk]

@niezbop Hi! Would be you kind to merge this fix?

[niezbop]

Hey everyone. Sorry for the time it took.

u3d has been updated to 1.3.3! It's available on RubyGems and will be available on Github once I manage to fix my setup on my machine