code 10 windows

[platinum55]

Description

Hey,

sorry I think I forgot to run "make build" after git pull. I deleted everything, cloned fresh and made a new build. DMA still not working in Windows.

Vivado shows the three ".hex" files as "Unknown", maybe I need to change the type to "Memory initialization file"?

I tried to read out the BAR0 in Linux, but its not possible, it seems to be gone/broken. I get Operation not permitted with the same python script that was working before and showed ff ff ff ff ff ff with the dummy stuff.

Also I noticed that:

• a DSN (Device Serial Number) is emulated, but the original device does not have this feature
• donor has 16k BAR memory but the clone has only 8k "[size=16K]" vs "[size=8K]"
• MSIX is reported different, probably causing errors too

Original Donor output of lspci -vv:

	Subsystem: Samsung Electronics Co Ltd SSD 980 PRO
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	IOMMU group: 14
	Region 0: Memory at 81400000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr+ FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 16GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
			 10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
			 EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: Upstream Port
	Capabilities: [b0] MSI-X: Enable- Count=130 Masked-
		Vector table: BAR=0 offset=00003000
		PBA: BAR=0 offset=00002000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq+ ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr+ HeaderOF+
		AERCap:	First Error Pointer: 14, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 40000001 0000000f 81400000 ffffffff
	Capabilities: [168 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [178 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [198 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [1bc v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady+ MargSoftReady+
	Capabilities: [214 v1] Latency Tolerance Reporting
		Max snoop latency: 15728640ns
		Max no snoop latency: 15728640ns
	Capabilities: [21c v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=30720ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [3a0 v1] Data Link Feature <?>
	Kernel driver in use: vfio-pci
	Kernel modules: nvme

and here is the flashed DMA Card output under linux:

    Subsystem: Samsung Electronics Co Ltd SSD 980 PRO
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 32
    Region 0: Memory at fcd00000 (64-bit, non-prefetchable) [size=8K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 26W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x1
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
    Kernel modules: nvme

Error Type

Code 10 (device cannot start)

Device Context File

device_context.json

Windows Event Log

0000: 0F 00 24 00 01 00 00 00   ..$.....
0008: 00 00 00 00 0B 00 04 C0   .......À
0010: 02 00 00 00 00 00 00 00   ........
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 01 00 00 00 20 00 04 00   .... ...
0030: 0B 00 04 C0 02 00 00 00   ...À....
0038: 00 00 00 00 00 00 00 00   ........
0040: 00 00 00 00 00 00 00 00   ........
0048: 04 00 00 00               ....

Device Manager Error Details

No response

Terminal Output

[sercanarga]

your samsung 980 pro doesn't have a dsn in its config space, but the fpga firmware was always advertising one because DSN_Enabled was hardcoded to true in the vivado pcie ip config. windows stornvme driver noticed the mismatch and refused to start the device.

i've pushed a fix that:

• disables DSN_Enabled in the pcie ip when the donor device doesn't have a dsn
• zeros out the hardcoded default dsn value in the sv source

please pull the latest, run make build, regenerate your firmware and try again.

one note about the bar size your 980 pro has a 16kb bar0 but the dma card is showing 8kb. that's a hardware limitation of the fpga's bram capacity and can't be fixed in software. depending on your fpga board it might not cause issues, but if it does there's not much we can do on the software side.

let me know if the code 10 goes away after rebuilding.

[platinum55]

Hello sercan,

done, still error 10 in windows, unchanged.

device_context.json

I wonder why the coe files are listed multiple times in vivado project. please see attached screenshot from the project:

The DSN is gone now but BAR still not readable from linux, the lspci -vvv shows:

04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
    Subsystem: Samsung Electronics Co Ltd SSD 980 PRO
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 32
    Region 0: Memory at fcd00000 (64-bit, non-prefetchable) [size=8K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 26W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x1
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported

[sercanarga]

the main issue was that the pcie ip wasn't configured with msi-x capability at all, so the nvme driver couldn't set up interrupts. also fixed:

• msi-x now configured from donor data (130 vectors in your case)
• msi vector count matched to donor (1/32 instead of 1/1)
• dsn removed when donor doesn't have it
• bar0 size preserved correctly

please pull latest, rebuild and reflash. let me know if the code 10 persists.

[platinum55]

hi sercan,

the config failed to load, I had to modify the following parts in vivado_generate_project.tcl to make it load without errors:

  # MSI vector capability from donor device
  set_property -dict [list \
    CONFIG.Multiple_Message_Capable 8_vectors \
  ] $pcie_ip

  # MSI-X capability from donor device (130 vectors)
  set_property CONFIG.MSIx_Enabled {true} $pcie_ip
  set_property -dict [list \
    CONFIG.MSI_Enabled       {true} \
    CONFIG.MSIx_Enabled      {true} \
    CONFIG.MSIx_Table_Size   {082} \
    CONFIG.MSIx_Table_BIR    {BAR_1:0} \
    CONFIG.MSIx_Table_Offset {00003000} \
    CONFIG.MSIx_PBA_BIR      {BAR_1:0} \
    CONFIG.MSIx_PBA_Offset   {00002000} \
  ] $pcie_ip

otherwise there were errors like:

ERROR: [Vivado 12-4371] Cannot find parameter 'MSIx_Cap_Table_Size' on IP 'pcie_7x_0'.
ERROR: [Vivado 12-1342] Failed to set property 'CONFIG.MSIx_Cap_Table_Size' on IP 'pcie_7x_0'.

or

WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'MSIx_Table_Size' from '1' to '082' has been ignored for IP 'pcie_7x_0'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'MSIx_Table_Offset' from '0' to '00003000' has been ignored for IP 'pcie_7x_0'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'MSIx_Table_BIR' from 'BAR_1:0' to 'BAR_0' has been ignored for IP 'pcie_7x_0'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'MSIx_PBA_Offset' from '0' to '00002000' has been ignored for IP 'pcie_7x_0'
WARNING: [IP_Flow 19-3374] An attempt to modify the value of disabled parameter 'MSIx_PBA_BIR' from 'BAR_1:0' to 'BAR_0' has been ignored for IP 'pcie_7x_0'

or

ERROR: [IP_Flow 19-3461] Value 'BAR_0' is out of the range for parameter 'BAR Indicator(MSIx_Table_BIR)' for IP 'pcie_7x_0' . Valid values are - BAR_1:0
ERROR: [IP_Flow 19-3461] Value 'BAR_0' is out of the range for parameter 'PBA BAR Indicator(MSIx_PBA_BIR)' for IP 'pcie_7x_0' . Valid values are - BAR_1:0

also the CONFIG.Multiple_Message_Capable 5_vectors \ was wrong, I assume that 8_vectors is correct (that was chosable from the GUI dropdown)?

after modifying it was loading and I was able to generate the bitstream without errors. However the Captain_75T stays on red-LED, the firmware seems not to be able to boot/enable the card

[platinum55]

the three .hex files are still "unknown", shall I set them to type "Memory initialization file" or does it not matter?

[sercanarga]

you were right on all points. i've pushed fixes for everything you ran into:

what changed:

• MSIx_Cap_Table_Size was the wrong parameter name, renamed to MSIx_Table_Size
• msix is now enabled before setting the sub-parameters (vivado ignores them otherwise)
• msi vector format fixed (32_vectors instead of 5_vectors etc)
• bar indicator now uses BAR_1:0 for 64-bit bar0 instead of BAR_0
• hex files are now properly typed as "Memory Initialization Files" so they won't show as "Unknown"

about the bar size: your 980 pro has 16k bar0 but the fpga can only expose 8k due to bram limits. this is a hardware constraint we can't fix in software, but most nvme drivers handle smaller bars fine it shouldn't cause code 10 on its own.

please pull latest, rebuild and reflash. the red led issue was most likely caused by the msix config failing silently during ip generation. let me know how it goes.

[platinum55]

thanks a lot for your endless efforts! I'm getting the following error now when loading the project into vivado:

ERROR: [Vivado 12-4371] Cannot find parameter 'Coe_File' on IP 'drom_pcie_cfgspace_writemask'.
ERROR: [Vivado 12-1342] Failed to set property 'CONFIG.Coe_File' on IP 'drom_pcie_cfgspace_writemask'.
INFO: [Common 17-17] undo 'set_property'
ERROR: [Common 17-39] 'set_property' failed due to earlier errors.

    while executing
"rdi::add_properties -dict {CONFIG.Coe_File /home/ast/Desktop/shared/PCILeechGen/pcileech_datastore/pcileech_cfgspace_writemask.coe} drom_pcie_cfgspace..."
    invoked from within
"set_property -dict [list CONFIG.Coe_File $coe_path] $writemask_bram"
    invoked from within
"if { $writemask_bram != "" } {
  set coe_path [file normalize "${origin_dir}/pcileech_cfgspace_writemask.coe"]
  if {[file exists $coe_path]} {
    se..."
    (file "vivado_generate_project.tcl" line 108)

I will try to fix it somehow and post the result here :)

[sercanarga]

i appreciate your contributions. here posted an update. check it out. it should fix this problem :)

[platinum55]

thank you!
I'm now getting thru and the new bitstream was built, sadly still error 10 with slightly different bytecode in eventlog (last line starts with 08 instead of 04):

0000: 0F 00 24 00 01 00 00 00   ..$.....
0008: 00 00 00 00 0B 00 04 C0   .......À
0010: 02 00 00 00 00 00 00 00   ........
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 01 00 00 00 20 00 04 00   .... ...
0030: 0B 00 04 C0 02 00 00 00   ...À....
0038: 00 00 00 00 00 00 00 00   ........
0040: 00 00 00 00 00 00 00 00   ........
0048: 08 00 00 00               ....

I noticed vivado warnings of overwritten files/content:

[HDL 9-3756] overwriting previous definition of module 'pcileech_tlps128_bar_controller' ["/home/ast/Desktop/shared/PCILeechGen/pcileech_datastore/CaptainDMA_75T/CaptainDMA_75T.srcs/sources_1/imports/src/pcileech_tlps128_bar_controller.sv":42]
[filemgmt 20-1318] Duplicate Design Unit 'pcileech_tlps128_bar_controller()' found in library 'xil_defaultlib'
Duplicate found at line 12 of file "/home/ast/Desktop/shared/PCILeechGen/pcileech_datastore/CaptainDMA_75T/CaptainDMA_75T.srcs/sources_1/imports/pcileech_datastore/pcileech_tlps128_bar_controller.sv"
	(Active) Duplicate found at line 42 of file "/home/ast/Desktop/shared/PCILeechGen/pcileech_datastore/CaptainDMA_75T/CaptainDMA_75T.srcs/sources_1/imports/src/pcileech_tlps128_bar_controller.sv"

maybe thats the issue?

Under Linux I tried to read out the BAR content but got "Operation not permitted" again (I'm root)

lspci output:

04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
    Subsystem: Samsung Electronics Co Ltd SSD 980 PRO
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 32
    Region 0: Memory at fcd00000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/32 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 26W
        DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x1
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp- ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [9c] MSI-X: Enable- Count=129 Masked-
        Vector table: BAR=0 offset=00003000
        PBA: BAR=0 offset=00002000