Unable to acquire wildcard certificate from Entrust but the process works fine for single and multi-domain certificates

[jamiekowalczik]

I'm trying to acquire a wildcard certificate from Entrust but the process is not working. I have separate EAB accounts for single, multi and wildcard certificates. The process works fine for single and multi-domain certificates but is failing for wildcard certificates. Below is the debug output, any information you can provide to assist would be greatly appreciated. I also have Entrust involved to rule out issues on there end.

--- snippets of the script ---

$pArgs = @{ 
      EIPUsername = $EIPUsername
      EIPPassword = ($EIPPassword | ConvertTo-SecureString -asPlainText -Force)
      EIPHostname = $EIPHostname
      EIPDNSName = $EIPDNSName
      EIPView = $EIPView
   };
   $aArgs = @{
         Contact = $EmailContact
         AcceptTOS = $true
         Force = $true
   };
   $rArgs = @{ 
         Domain = $DNSNamesArray
         Contact = $EmailContact
         CertKeyLength = $CertKeyLength
         Subject = $Subject
         AcceptTOS = $true
         AlwaysNewKey = $true
         DirectoryUrl = $PAServerURL
         Plugin = 'EfficientIP'
         PluginArgs = $pArgs
         DnsAlias = $DNSAliasArray
         Force = $true
         LifetimeDays = $LifetimeDays
         Verbose = $True
   };

If((Get-PAAccount).id -eq $IDWC -And (Get-PAAccount).status -eq "valid") { Write-Host "$($IDWC) Account exists" } Else { 
            Set-PAServer $PAServerURL
            New-PAAccount -ID $IDWC -ExtAcctKID $ExtAcctKIDWC -ExtAcctHMACKey $ExtAcctHMACKeyWC @aArgs | Out-Null         
}

$Certificate = New-PACertificate @rArgs
Return $Certificate

--- output below ---

VERBOSE: Acquring certificate
DEBUG: Loading PAServer list from disk
DEBUG: Requesting nonce from https://acme.entrust.net/acme2/new-nonce
DEBUG: Saving PAServer to disk
DEBUG: Loading PAServer list from disk
DEBUG: Enabling cert validation
DEBUG: Creating new ec-256 account with contact: mailto:me@here.com
DEBUG: Creating new EC 256 key
DEBUG: ACME Header:
{
  "alg": "HS256",
  "kid": "uniquekey",
  "url": "https://acme.entrust.net/acme2/new-account"
}
DEBUG: ACME Payload:
{"crv":"P-256","kty":"EC","x":"redacted","y":"redacted"}
DEBUG: Signing message using HMAC with hash size 256
DEBUG: ACME Header:
{
  "alg": "ES256",
  "url": "https://acme.entrust.net/acme2/new-account",
  "nonce": "redacted",
  "jwk": {
    "crv": "P-256",
    "kty": "EC",
    "x": "redacted",
    "y": "redacted"
  }
}
DEBUG: ACME Payload:
{"termsOfServiceAgreed":true,"contact":["mailto:me@here.com"],"externalAccountBinding":{"payload":"redacted","protected":"redacted","signature":"redacted"}}
DEBUG: Signing message using EC with SHA256
DEBUG: POST https://acme.entrust.net/acme2/new-account
{"payload":"redacted","protected":"redacted","signature":"redacted"}
VERBOSE:
Name                           Value
----                           -----
UserAgent                      Posh-ACME/4.25.1 PowerShell/7.3.12 Platform/Linux
Headers                        {[Accept-Language, en-us,en;q=0.5]}
Method                         Post
Body                           {"payload":"redacted
ErrorAction                    Stop
ContentType                    application/jose+json
Uri                            https://acme.entrust.net/acme2/new-account
Verbose                        False


DEBUG: ACME Response:
{"status":"valid","contact":["mailto:me@here.com"],"orders":"https://acme.entrust.net/acme2/orders/redacted"}
DEBUG: Updated nonce: redacted
DEBUG: Loading PAServer list from disk
VERBOSE: Updating directory info from https://acme.entrust.net/acme2/directory
DEBUG: Requesting nonce from https://acme.entrust.net/acme2/new-nonce
DEBUG: Saving PAServer to disk
DEBUG: Loading PAServer list from disk
DEBUG: Enabling cert validation
DEBUG: Loading PAAccount Wildcard-Test from disk
VERBOSE: Using ACME Server https://acme.entrust.net/acme2/directory
DEBUG: Loading PAAccount list from disk
VERBOSE: Using account Wildcard-Test
VERBOSE: Order name not specified, using '!.domain.com'
DEBUG: Loading PAOrder list from disk
VERBOSE: Creating a new order '!.domain.com' for *.domain.com
DEBUG: New order params:
{
  "OCSPMustStaple": {
    "IsPresent": false
  },
  "PfxPass": "poshacme",
  "Plugin": [
    "EfficientIP"
  ],
  "Subject": "CN=*.domain.com",
  "AlwaysNewKey": {
    "IsPresent": true
  },
  "Install": {
    "IsPresent": false
  },
  "KeyLength": "2048",
  "LifetimeDays": 395,
  "UseModernPfxEncryption": {
    "IsPresent": false
  },
  "FriendlyName": "*.domain.com",
  "Domain": [
    "*.domain.com"
  ],
  "PluginArgs": {
    "EIPPassword": {
      "Length": 2255
    },
    "EIPView": "aview",
    "EIPHostname": "efficientip.local",
    "EIPUsername": "user",
    "EIPDNSName": "smart.local"
  },
  "DnsAlias": [
    "alias.for.star.domain.com"
  ],
  "Name": "!.domain.com"
}
DEBUG: Loading PAOrder list from disk
DEBUG: Creating new 2048 order with domains: *.domain.com
DEBUG: ACME Header:
{
  "alg": "ES256",
  "kid": "https://acme.entrust.net/acme2/acct/redacted",
  "url": "https://acme.entrust.net/acme2/new-order",
  "nonce": "redacted"
}
DEBUG: ACME Payload:
{"notAfter":"2025-11-02T20:06:12Z","notBefore":"2024-10-03T20:06:12Z","identifiers":[{"value":"*.domain.com","type":"dns"}]}
DEBUG: Signing message using EC with SHA256
DEBUG: POST https://acme.entrust.net/acme2/new-order
{"payload":"redacted","protected":"redacted","signature":"redacted"}
VERBOSE:
Name                           Value
----                           -----
UserAgent                      Posh-ACME/4.25.1 PowerShell/7.3.12 Platform/Linux
Headers                        {[Accept-Language, en-us,en;q=0.5]}
Method                         Post
Body                           {"payload":"redacted
ErrorAction                    Stop
ContentType                    application/jose+json
Uri                            https://acme.entrust.net/acme2/new-order
Verbose                        False


DEBUG: ACME Response:
{"status":"pending","expires":"2024-10-10T20:06:12Z","identifiers":[{"type":"dns","value":"domain.com"}],"notBefore":"2024-10-03T20:06:12Z","notAfter":"2025-11-02T20:06:12Z","authorizations":["https://acme.entrust.net/acme2/authz/redacted"],"finalize":"https://acme.entrust.net/acme2/order/redacted/finalize"}
DEBUG: Updated nonce: redacted
DEBUG: Adding location https://acme.entrust.net/acme2/order/redacted
DEBUG: Loading PAOrder list from disk
DEBUG: ACME Header:
{
  "alg": "ES256",
  "kid": "https://acme.entrust.net/acme2/acct/redacted",
  "url": "https://acme.entrust.net/acme2/authz/redacted",
  "nonce": "redacted"
}
DEBUG: ACME Payload: (empty)
DEBUG: Signing message using EC with SHA256
DEBUG: POST https://acme.entrust.net/acme2/authz/redacted
{"payload":"","protected":"redacted","signature":"redacted"}
VERBOSE:
Name                           Value
----                           -----
UserAgent                      Posh-ACME/4.25.1 PowerShell/7.3.12 Platform/Linux
Headers                        {[Accept-Language, en-us,en;q=0.5]}
Method                         Post
Body                           {"payload":"","protected":"redacted
ErrorAction                    Stop
ContentType                    application/jose+json
Uri                            https://acme.entrust.net/acme2/authz/redacted
Verbose                        False


DEBUG: ACME Response:
{"identifier":{"type":"dns","value":"domain.com"},"status":"pending","challenges":[{"type":"dns-01","url":"https://acme.entrust.net/acme2/chall/redacted","status":"pending","token":"redacted"}],"wildcard":false}
DEBUG: Updated nonce: redacted
DEBUG: Retry-After = 1800
DEBUG: Exporting plugin args for order '!.domain.com' with plugins EfficientIP
DEBUG: Loading PAOrder list from disk
DEBUG: Attempting to load plugin EfficientIP
DEBUG: Adding new value for EIPPassword
DEBUG: Adding new value for EIPView
DEBUG: Adding new value for EIPHostname
DEBUG: Adding new value for EIPUsername
DEBUG: Adding new value for EIPDNSName
Get-PAAuthorization: /path/Submit-ChallengeValidation.ps1:64
Line |
  64 |          $allAuths = @($Order | Get-PAAuthorization)
     |                                 ~~~~~~~~~~~~~~~~~~~
     | Cannot bind argument to parameter 'AuthURLs' because it is null.
DEBUG: Plugin: EfficientIP
DEBUG: DnsAlias: alias.for.star.domain.com
DEBUG: Loading PAOrder list from disk
DEBUG: Refreshing order '!.domain.com'
DEBUG: ACME Header:
{
  "alg": "ES256",
  "kid": "https://acme.entrust.net/acme2/acct/redacted",
  "url": "https://acme.entrust.net/acme2/order/redacted",
  "nonce": "redacted"
}
DEBUG: ACME Payload: (empty)
DEBUG: Signing message using EC with SHA256
DEBUG: POST https://acme.entrust.net/acme2/order/redacted
{"payload":"","protected":"redacted","signature":"redacted"}
VERBOSE:
Name                           Value
----                           -----
UserAgent                      Posh-ACME/4.25.1 PowerShell/7.3.12 Platform/Linux
Headers                        {[Accept-Language, en-us,en;q=0.5]}
Method                         Post
Body                           {"payload":"","protected":"redacted
ErrorAction                    Stop
ContentType                    application/jose+json
Uri                            https://acme.entrust.net/acme2/order/redacted
Verbose                        False


DEBUG: ACME Response:
{"status":"pending","expires":"2024-10-10T20:06:12Z","identifiers":[{"type":"dns","value":"domain.com"}],"notBefore":"2024-10-03T20:06:12Z","notAfter":"2025-11-02T20:06:12Z","authorizations":["https://acme.entrust.net/acme2/authz/redacted"],"finalize":"https://acme.entrust.net/acme2/order/redacted/finalize"}
DEBUG: Updated nonce: redacted
DEBUG: Loading PAOrder list from disk

-------
> Get-PAAccount | fl

id        : redacted
status    : valid
contact   : {mailto:me@here.com}
location  : https://acme.entrust.net/acme2/acct/redacted
key       : @{crv=P-256; d=redacted; kty=EC; x=redacted
            ; y=redacted}
alg       : ES256
KeyLength : ec-256
orders    : https://acme.entrust.net/acme2/orders/redacted
sskey     :
Folder    : /path/Posh-ACME/acme.entrust.net/redacted


> Get-PAOrder | fl

status              : pending
expires             : 2024-10-10T20:52:14Z
identifiers         : {System.Collections.Hashtable}
authorizations      : {$null}
finalize            : https://acme.entrust.net/acme2/order/redacted/finalize
location            : https://acme.entrust.net/acme2/order/redacted
UseSerialValidation : False
Subject             :
SANs                : {}
DnsSleep            : 120
PreferredChain      :
OCSPMustStaple      : False
PfxPass             : poshacme
AlwaysNewKey        : False
KeyLength           : 2048
LifetimeDays        :
Install             : False
RenewAfter          :
DnsAlias            :
ValidationTimeout   : 60
MainDomain          : *.domain.com
CertExpires         :
Plugin              : {Manual}
FriendlyName        : *.domain.com
certificate         :
Name                : !.domain.com
Folder              : /path/!.domain.com


> Get-PAOrder | Get-PAAuthorization
Get-PAAuthorization: Cannot bind argument to parameter 'AuthURLs' because it is null.
>

[rmbolger]

Hey @jamiekowalczik, thanks for reaching out and thanks a ton for the debug output as well. For future reference, you don't need to redact the nonce values. They're just random strings generated by the ACME server that are included to prevent replay attacks.

The first thing I notice from the logs is that the ACME response following the POST https://acme.entrust.net/acme2/new-order request is different than the response you get from other ACME servers like Let's Encrypt. Specifically, the *. has been stripped from the identifier value. So this is what was requested:

"identifiers":[{"value":"*.domain.com","type":"dns"}]

But the response order object had:

"identifiers":[{"type":"dns","value":"domain.com"}]

Normally, I'd assume this was just an accidental typo in the log redaction. But then later when the module requests the authorization object (POST https://acme.entrust.net/acme2/authz/redacted) for the order's identifier, the ACME response has "wildcard":false where I would expect to see "wildcard":true since you did indeed request a wildcard identifier.

It seems like this Entrust ACME endpoint is purposefully stripping the wildcard portion of identifiers as if they're not allowed. I'm pretty sure the errors stem from the fact that the identifiers in the requested order don't match the identifiers in the resulting order object that was returned by Entrust. There's some internal code that needs to match the authz URLs with their associated identifier. But since the authz data doesn't match any of the identifiers in the module's copy of the order object, things go wonky.

[jamiekowalczik]

Thanks @rmbolger! Appreciate the quick response. I shared this link with support. Hoping for a resolution. I'll keep you posted.

[jamiekowalczik]

The Entrust support ticket got escalated. Still waiting for a response.

I made the following adjustment as a workaround that got me a wildcard certificate, however as you are likely aware, if I tried to acquire a certificate with both *.domain.com and domain.com in the SAN field, no bueno.
New-PAOrder.ps1

  From:
        $auth = $auths | Where-Object { $_.fqdn -eq $order.identifiers[$i].value }        
  To:
        $auth = $auths | Where-Object { $_.fqdn -eq $order.identifiers[$i].value -Or "*." + $_.fqdn -eq $order.identifiers[$i].value } 

After trying to determine how to maintain order when converting a json string to a powershell object manually, assuming that was the problem with associating the order name with an authorization URL, I stumbled on the following note here (I'm using 7.4.3): https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertfrom-json?view=powershell-7.4
"The PSObject type maintains the order of the properties as presented in the JSON string"

I followed up with returning the code back to how it was and commented out the section below. After that, I've had repeated success with acquiring single, multi and wildcard domain certificates from Entrust.

#$auths = Get-PAAuthorization $order.authorizations
#for ($i=0; $i -lt $order.identifiers.Count; $i++) {
#    $auth = $auths | Where-Object { $_.fqdn -eq $order.identifiers[$i].value -Or "*." + $_.fqdn -eq $order.identifiers[$i].value }        
#    $order.authorizations[$i] = $auth.location
#}

UPDATE: It had a good run.. ..and then I ran into some occasions of "Timed out waiting 60 seconds for authorizations to become valid". Really hoping Entrust can make some adjustments on their end.

[rmbolger]

It's still so weird that they'd deviate from the spec like that. Like, I get maybe a bug in their implementation that strips the *. from the order object, but to also return "wildcard":false seems almost intentional.

You said you were able to get a cert with both bare apex and wildcard apex in the same cert after commenting out the identifier matching code? Did that order still require validation on those names? Or did it go through automatically because the previous authorizations had been cached? I'm really curious what the resulting order object JSON response looked like on that one too.

I suppose even if the resulting order had 2 identical identifiers probably would have worked because the order doesn't matter since the TXT records ultimately get created at the same FQDN for both. Things might get wonky if there were other identifiers in the order as well. The more distinct names, the higher the chance that the TXT records get set to the wrong FQDNs.

I knew about the PowerShell 7.3 JSON ordering thing. But sadly doesn't actually help in this case because it's the order sent back by the ACME CA that we can't count on.