generate_new_metadata in tidy_backup_clean.rb needs to preserve metadata.json

[lamont-granquist]

If the cookbook has a metadata.json that file should never be stripped, removed or tampered with at all (no matter if metadata.rb exists or not, client programs should never attempt to regenerate metadata.json from the metadata.rb in the cookbook if it is present, the metadata.json file MUST be considered authoritative. The metadata.rb file MAY require build time artifacts which are not present in the shipping cookbook and trying to recompile it after the metadata.json has been generated is guaranteed to fail on some cookbooks.

If the cookbook has a metadata.rb file that should also not be tampered with, but it should also be ignored completely if the metadata.json file is present (so if the metadata.json and metadata.rb files are present then nothing should be done with either file THE metadata.rb file may not be parseable and THIS IS NOT A BUG TO FIX -- downstream utilities must be modified to prefer the metadata.json file.

ONLY IF one of the files exists and the other does not should knife tidy attempt to generate or reconstruct one from the other.

I can't sort out precisely what knife tidy is actually doing, but I'm reasonably confident it isn't following these guidelines.

[lamont-granquist]

I think the only thing that should happen here is:

1. if metadata.rb and metadata.json both exist, do nothing at all.
2. if metadata.json only exists, try to construct a simplified metadata.rb
3. if metadata.rb only exists, generate metadata.json using the knife cookbook metadata approach most likely (this should never happen, don't worry about error conditions, just let it fail hard if it doesn't work)
4. if metadata.rb and metadata.json both don't exist, hard fail (this should never ever happen)

Most of the logic around hacking up metadata.rb in knife tidy should be eliminated from the codebase.

[lamont-granquist]

Okay so I now have a bit more complete information on what is going on.

The problem is that this runs on a checkout of the filesystem from "knife ec backup".

For old chef servers there may have been no enforcement of the metadata.json file being uploaded to the manifest of the cookbook. All the information is accessible from the API, but knife ec backup will only download the filesystem contents. Since it is throwing away necessary data, this seems to violate the principle that it should be a "backup" tool.

So what needs to happen is that knife ec backup should GET <org>/cookbooks/NAME/version and dump the metadata field in the response to metadata.json and use that as the source of truth of the metadata.json of the cookbook instead of the filesystem
contents.

If knife download/upload aka chef_fs need to be changed at the same time that would probably be a good way to implement this.

Once that is done THEN we can assume in knife tidy that cookbooks MUST always have a valid metadata.json (and can raise and suggest the user upgrade their knife-ec-backup gem if the cookbook does not).