E2E: change timing of vaultsecrets test to guarantee lease window (#…

…20200)

We've been getting a couple of errors from this test on nightly where the
template hasn't rendered by the time we expect it to. I've run some tests
locally and this may be a timing issue introduced by recent code changes to
templates.

Move the start of the timer to after we're guaranteed that we've got a secret
lease TTL started, to eliminate this as a source of flakiness. In my tests this
adds another ~5s to a test that already takes over a minute to run anyways.

e2e/vaultsecrets/vaultsecrets_test.go

@@ -107,17 +107,17 @@ func TestVaultSecrets(t *testing.T) {
 	writePolicy(t, policyID, "./input/policy-good.hcl", testID)
 	submission.Rerun(jobs3.ReplaceInJobSpec("FIRST", "SECOND"))
 
-	// record the rough start of vault lease TTL window, so that we don't have
-	// to wait excessively later on
-	ttlStart := time.Now()
-
 	// job should be now unblocked
 	err = e2e.WaitForAllocStatusExpected(jobID, ns, []string{"running", "complete"})
 	must.NoError(t, err, must.Sprint("expected running->complete allocation"))
 
 	renderedCert := waitForAllocSecret(t, submission, "/secrets/certificate.crt", "BEGIN CERTIFICATE")
 	waitForAllocSecret(t, submission, "/secrets/access.key", secretValue)
 
+	// record the earliest we can guaranteee that the vault lease TTL has
+	// started, so we don't have to wait excessively later on
+	ttlStart := time.Now()
+
 	var re = regexp.MustCompile(`VAULT_TOKEN=(.*)`)
 
 	// check vault token was written and save it for later comparison