It turns out it only takes a couple minutes for first build, and the time savings on cached environments is negligible. A recent experiment found that the biggest task in an initial book build from template (61 s total) is the installation of dependencies, which was 44 s, and this is not even part of the cache!
Some packages require cloning a repository (those with git+... as in Issue #85 ) perhaps it would be good to add a few of these to the cache to save some time.