Unknown NtStatus error returned 'STATUS_FS_DRIVER_REQUIRED'

[andrewsiemer]

I am running into the following issue when trying to copy a file from a DFS share.

Traceback (most recent call last):
   File "/Users/andrewsiemer/Documents/NVIDIA/asiemer_macbook/sw/main/bios/brf/Scripts/Python/Library/netutils.py", line 155, in copy_file
     with smbclient.open_file(src, mode="rb") as infile:
   File "/Users/andrewsiemer/Library/Python/3.9/lib/python/site-packages/smbclient/_os.py", line 389, in open_file
     raw_fd.open()
   File "/Users/andrewsiemer/Library/Python/3.9/lib/python/site-packages/smbclient/_io.py", line 463, in open
     transaction.commit()
   File "/Users/andrewsiemer/Library/Python/3.9/lib/python/site-packages/smbclient/_io.py", line 349, in commit
     raise failures[0]
 smbprotocol.exceptions.SMBOSError: [Error 0] [NtStatus 0xc000019c] Unknown NtStatus error returned 'STATUS_FS_DRIVER_REQUIRED': '\\share\builds\path\to\file.run'

The code is:

import smbclient.shutil

smbclient.shutil.copyfile(
    "\\share\builds\path\to\remote\file.zip",
    '\path\to\local\file.zip',
    username='adminuser',
    password='mypassword')

I use this code to copy many files and it seems to only throw this error with one share. When I mount via the CLI, I can copy the file manually.

Thanks in advance!

[jborean93]

What is the service you are using here, is it a Windows DFS implementation/something else? I've seen a few reports for this but was never able to figure out what was happening. Typically you see STATUS_FS_DRIVER_REQUIRED when a DFS referral request was sent to a service that cannot handle that IOCTL code but in this case it seems like the open method. If it's possible it would be great to see the Wireshark trace for this but I can understand if the data is sensitive and cannot be shared.

[andrewsiemer]

Yes, sadly this is not a share that I set up. This is the result of smbutil statshares -a if that's even helpful. Sorry I cannot share much more. Is there any other non-sensitive info I can share?

                              SERVER_NAME                   test
                              USER_ID                       501
                              SMB_NEGOTIATE                 SMBV_NEG_SMB1_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB2_ENABLED
                              SMB_NEGOTIATE                 SMBV_NEG_SMB3_ENABLED
                              SMB_VERSION                   SMB_3.1.1
                              SMB_ENCRYPT_ALGORITHMS        AES_128_CCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_128_GCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_256_CCM_ENABLED
                              SMB_ENCRYPT_ALGORITHMS        AES_256_GCM_ENABLED
                              SMB_CURR_ENCRYPT_ALGORITHM    OFF
                              SMB_SHARE_TYPE                DISK
                              SIGNING_SUPPORTED             TRUE
                              EXTENDED_SECURITY_SUPPORTED   TRUE
                              LARGE_FILE_SUPPORTED          TRUE
                              OS_X_SERVER                   TRUE
                              FILE_IDS_SUPPORTED            TRUE
                              DFS_SUPPORTED                 TRUE
                              FILE_LEASING_SUPPORTED        TRUE
                              MULTI_CREDIT_SUPPORTED        TRUE
                              PERSISTENT_HANDLES_SUPPORTED  TRUE
                              DFS_SHARE                     TRUE

[jborean93]

Thanks for the info, can you share the SMB server implementation, i.e. is it Samba, some NAS box? Can you try out this low level code to see what happens?

import uuid

from smbprotocol.connection import Connection
from smbprotocol.tree import TreeConnect
from smbprotocol.session import Session
from smbprotocol.open import (
    Open,
    ImpersonationLevel,
    FileAttributes,
    FilePipePrinterAccessMask,
    ShareAccess,
    CreateDisposition,
)

server = "server-name"
username = "adminuser"
password = "mypassword"
share = "share"
path = r"builds\path\to\remote\file.zip"

c = Connection(uuid.uuid4(), server)
c.connect()
try:
    print(f"Connection - Capabilities: {c.server_capabilities}")

    s = Session(c, username, password)
    s.connect()

    t = TreeConnect(s, rf"\\{server}\{share}")
    t.connect()
    print(f"Tree - IsDfsShare {t.is_dfs_share}")

    o = Open(t, path)
    o.create(
        impersonation_level=ImpersonationLevel.Impersonation,
        desired_access=FilePipePrinterAccessMask.FILE_READ_DATA,
        file_attributes=FileAttributes.FILE_ATTRIBUTE_NORMAL,
        share_access=ShareAccess.FILE_SHARE_READ,
        create_disposition=CreateDisposition.FILE_OPEN,
        create_options=0,
        create_contexts=None,
    )
    print("File opened")

finally:
    c.disconnect()

Comparing it with one that works would also be great. If this does fail what happens if you do t.is_dfs_share = False straight after t.connect()?

[andrewsiemer]

@jborean93 Thanks for the help, here are the results using the known good and known bad share.

Known bad (same result with t.is_dfs_share = False):

Connection - Capabilities: (23) SMB2_GLOBAL_CAP_DFS, SMB2_GLOBAL_CAP_LARGE_MTU, SMB2_GLOBAL_CAP_LEASING, SMB2_GLOBAL_CAP_PERSISTENT_HANDLES
Tree - IsDfsShare True  # or False when t.is_dfs_share = False
Traceback (most recent call last):
  File "/Users/andrewsiemer/Desktop/test.py", line 35, in <module>
    o.create(
  File "/Users/andrewsiemer/Library/Python/3.9/lib/python/site-packages/smbprotocol/open.py", line 1160, in create
    return self._create_response(request)
  File "/Users/andrewsiemer/Library/Python/3.9/lib/python/site-packages/smbprotocol/open.py", line 1168, in _create_response
    response = self.connection.receive(request)
  File "/Users/andrewsiemer/Library/Python/3.9/lib/python/site-packages/smbprotocol/connection.py", line 1095, in receive
    raise SMBResponseException(response)
smbprotocol.exceptions.PathNotCovered: Received unexpected status from the server: The contacted server does not support the indicated part of the DFS namespace. (3221226071) STATUS_PATH_NOT_COVERED: 0xc0000257

Known good (same result with t.is_dfs_share = False):

Connection - Capabilities: (23) SMB2_GLOBAL_CAP_DFS, SMB2_GLOBAL_CAP_LARGE_MTU, SMB2_GLOBAL_CAP_LEASING, SMB2_GLOBAL_CAP_PERSISTENT_HANDLES
Tree - IsDfsShare True  # or False when t.is_dfs_share = False
File opened

[jborean93]

The extra flag added on a request when the tree connect has marked it as a DFS share was introduced with #190 which was designed to replicate the Windows behaviour for the issue #170.

I'm not really sure what the best thing to do here, it seems like some server like this while others do not. You could use Wireshark to see how Windows opens the same file as well other macOS' SMB client does as well. Comparing them all to see what flags are set for the request and how they handle things would definitely help to see what the next steps should be.

[andrewsiemer]

Do you have any quick resources I can reference for checking the request in Wireshark? It has been a few years 😅

[jborean93]

Essentially install Wireshark and capture port 445. You want to compare the headers for the TreeConnect and Open requests. The code I gave you lets you do it with smbprotocol, for macOS you can have a look at mounting the path and doing a cat on the file, for Windows just open the file with Get-Item in PowerShell.

[andrewsiemer]

I have captured the following packets for the good/bad case and they seem to have similar headers (up to the encrypted ones). I do not see any of the error codes that I see from the python script. Am I missing something?

I did realize that mounting manually and opening the file from the command line produces "Create File Request" packets instead of Encrypted SMB3.

[jborean93]

Ah my apologies, I forgot smbprotocol defaults to enabling encryption. You can disable it to see the same Tree and Open/Create packets by doing Session(..., require_encryption=False) when creating the session.

[andrewsiemer]

Thanks, that did it. So now the only difference is in the Create Response packet from the server. Although other than the error code, the flag fields match.

Create Response, Error: STATUS_PATH_NOT_COVERED

Another thing I noticed is it seems to only be an issue in this directory. Other directories on the same share don't have the error.

//server/share/a/b/c/d/e/file1.txt     <--- anything beyond the //server/share/a/b/c directory throws the error
//server/share/a/b/f/g/h/file2.txt     <--- no issues

[jborean93]

Have you compared the flags on the header part of the message? It seems like having the SMB2_FLAGS_DFS_OPERATIONS flag (that smbprotocol is adding) is causing the STATUS_FS_DRIVER_REQUIRED. Does your macOS tool also add them, have you been able to test with Windows to see if it also sets that flag?

The STATUS_PATH_NOT_COVERED is somewhat expected for a DFS path as the server is telling the client the path isn't covered by this server and to send a DFS request to figure out the correct server and it's path that can process that request. Knowing the error flow for this type of path and the subsequent messages that are sent after t3he error help to identify what needs to be done for DFS operations here.

[andrewsiemer]

All requests have the flag (when is_dfs_share == True) but same result when the flag is not set.

...1 .... .... .... .... .... .... .... = DFS operation: This is a DFS OPERATION

All responses always:

...0 .... .... .... .... .... .... .... = DFS operation: This is a normal operation

Mac + windows CLI utility do not use the SMB2_FLAGS_DFS_OPERATIONS flag.

[jborean93]

I think I'm going to have to play around with some DFS scenarios a bit more and get a better understanding of SMB2_FLAGS_DFS_OPERATIONS. IIRC I saw WIndows set that when the tree reported itself as a DFS share but maybe that's not the case.

[andrewsiemer]

Anything I can do to help here?

After printing out some debug messages I have found that the following line causes the unknown exception.

Line 307 of smbclient/_io.py:

for smb_open in _resolve_dfs(self.raw):

Are there any changes to dfs_request that might help?

[jborean93]

Sorry I haven't gotten back to this, I just haven't found the time needed to dig into the SMB protocol. Ultimately there's a problem in how I interpret the SMB2_FLAGS_DFS_OPERATIONS in the SMB headers and what to do when the tree reports IsDfsShare. At the time I thought my current behaviour was based on how Windows works but based on your responses I need to revisit it and figure out what exactly is happening.

[andrewsiemer]

No worries, in the meantime I am happy to help provide debug info if needed.

[andrewsiemer]

Could it be due to path normalization required by dfs?

If the request received has SMB2_FLAGS_DFS_OPERATIONS set in the Flags field of the SMB2 header, and TreeConnect.Share.IsDfs is TRUE, the server MUST verify the value of IsDfsCapable:

• If IsDfsCapable is TRUE, the server MUST invoke the interface defined in [MS-DFSC] section 3.2.4.1 to normalize the path name by supplying the target path name.
• If IsDfsCapable is FALSE, the server MUST fail the request with STATUS_FS_DRIVER_REQUIRED.

Source: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/8c61e928-9242-44ed-96a0-98d1032d0d39

As specified in [MS-SMB2] section 3.3.5.9 and [MS-SMB] section 3.3.5.5, the SMB server invokes the DFS server to normalize the path name.

• If the DFS namespace initialization (as specified in section 3.2.3) corresponding to the share in the path is not yet complete, the DFS server MUST fail the path normalization request with STATUS_DFS_UNAVAILABLE.
• Otherwise, the DFS server matches the path name against DFS metadata. If the path matches or contains a DFS link, the DFS server MUST respond to the path normalization request with STATUS_PATH_NOT_COVERED, indicating to the client to resolve the path by using a DFS link referral request. Otherwise, the DFS server MUST change the path name to a path relative to the root of the namespace and return STATUS_SUCCESS. For example, if the path name is "\MyDomain\MyDfs\MyDir\file1", then the DFS server MUST change the path name to "MyDir\file1"

Source: https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-dfsc/c52dba85-adbd-407e-963d-21253d702ca5

[jborean93]

Yea the problem is that the request from smbprotocol is including SMB2_FLAGS_DFS_OPERATIONS when talking to a host that doesn't know how to resolve the path. Currently smbprotocol sets this header flag when the TreeConnect IsDfs flag is set, introduced with #190. Why I did that I unfortunately cannot remember and the PR/comments/issue don't seem to indicate why unfortunately. It could be I misread that particular entry you shared or it could be behaviour I saw with a real DFS share at the time but obviously based on this issue it's not correct. The trick is figuring out when it should be set and when it should not be.

[jborean93]

Hi I'm trying to jump back into this issue and figure out the problem once and for all. If you still have the environment around and are willing to test things it would be great if you could share the following:

• What the target SMB server software might be
  • I'm not sure if you are unaware, can't tell, or I never asked all good with either but wanted to clarify
• If you know the DFS topology in question that might make it easier to replicate locally
  • Is the DFS root a Windows Server implementation?
  • Is //server/share/a/b/c a DFS target
  • Does that target point to another server
  • Is that the server that fails with this error
  • Do you know the server implementation of this target
• Whether the tree connect response for your native macOS Windows scenarios set SMB2_SHAREFLAG_DFS in the tree flags
• Whether the Negotiate Response on the server it is failing on supports the DFS capability

I'm trying to setup a similar environment to test out so I can play around with it all but Windows seems to handle whatever I throw at it just fine.

[jborean93]

I don't know if it will solve the issue in your case but #253 makes setting this flag a bit more selective and maybe will help you. If you get time to test it out that would be great.

[andrewsiemer]

I have tried #253 with no luck (smbprotocol.exceptions.SMBOSError: [Error 0] [NtStatus 0xc000019c] Unknown NtStatus error returned 'STATUS_FS_DRIVER_REQUIRED':). It still works if I use the DFS referral path just not with the actual DFS path.

[jborean93]

That's a pity.

It still works if I use the DFS referral path just not with the actual DFS path.

So something is going on with the referral process that's causing it to send the the open to potentially the DFS server(?) causing it to fail. Unfortunately at this point in time I still cannot replicate it in my environment so short of seeing the network traces there's not much else I can do here. I essentially need to compare the network operations likes

• What server the DFS referral is being sent to
• What servers are returned
• How servers it is trying to open it on
• What servers are reporting PATH_NO_COVERED
• How does Windows act with the same path

Unless you can share more about the network environment such as the DFS servers used, the namespace and target setup, target paths, etc so I can potentially replicate it I'm nearing the end of my abilities here sorry.

[andrewsiemer]

No worries, thank you so much for all the help!