Clean up cache manager #187

Job	Run time
build-macos	4m 33s
build-android (aarch64-linux-android)	2m 41s
build-linux-musl (x86_64-unknown-linux-musl)	3m 31s
build-android (armv7-linux-androideabi)	2m 57s
build-linux (x86_64-unknown-linux-gnu)	3m 36s
build-linux-musl (aarch64-unknown-linux-musl)	3m 30s
build-windows (x86_64-pc-windows-msvc)	6m 5s
build-linux (aarch64-unknown-linux-gnu)	3m 26s
build-android (i686-linux-android)	2m 53s
build-linux-musl (armv7-unknown-linux-musleabihf)	3m 43s
build-windows (i686-pc-windows-msvc)	8m 0s
build-linux (armv7-unknown-linux-gnueabihf)	3m 32s
build-android (x86_64-linux-android)	3m 2s
	51m 29s

Provide feedback