From 8cf069f559c047b0fbcae17a06baa89e029714df Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Wed, 10 Jan 2024 21:32:56 +0100 Subject: [PATCH 1/9] Add Swift interop tooling components and layout --- proposed/swift-interop.md | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 4879c018b..64ebc88c0 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -152,33 +152,46 @@ If possible, Swift tuples should be represented as `ValueTuple`s in .NET. If thi #### Projection Tooling Components -The projection tooling should be split into these components: +The projection tooling will be based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift) (BTfS). The BTfS contains a set of tools that can consume a compiled Apple Swift library and generate wrappers (bindings) that allow it to be surfaced as a .NET library. This tool will be implemented as a self-hosted .NET CLI tool, automating the generation of wrappers by parsing the Swift library interface. It generates C# bindings that can be utilized as a .NET library and Swift bindings that C# bindings invoke in cases where direct P/Invoke into Swift library is not possible. + +The projection tooling should be split into these components. ##### Importing Swift into .NET -1. A tool that takes in a `.swiftinterface` file or Swift sources and produces C# code. -2. A library that provides the basic support for Swift interop that the generated code builds on. -3. User tooling to easily generate Swift projections for a given set of `.framework`s. - - This tooling would build a higher-level interface on top of the tool in item 1 that is more user-friendly and project-system-integrated. -4. (optional) A NuGet package, possibly referencable by `FrameworkReference` or automatically included when targeting macOS, Mac Catalyst, iOS, or tvOS platforms that exposes the platform APIs for each `.framework` that is exposed from Swift to .NET. - - This would be required to provide a single source of truth for Swift types so they can be exposed across an assembly boundary. +Importing Swift into .NET is done through bindings generated by the following components: + - `Demangler`: This component analyzes a source Swift library and maps entry points to mangled names. It consumes a source Swift library and creates `TLDefinition` objects that contain type, mangled name, demangled name, module name, and offset within a module. + - `Swiftmodule parser`: This component aggregates the public API. It takes in a `.swiftmodule` file generated by the Swift compiler, similar to a C header file, and combines it with the `TLDefinition` objects to generate a module declaration. + - `WrappingCompiler` and `OverrideBuilder`: These components generate Swift wrappers for cases where direct P/Invoke from C# into Swift is not possible. + - `NewClassCompiler`: This component generates C# bindings based on the generated `TLDefinition` and module declarations. It should implement type mapping and register lowering. Key considerations include: + - Splitting out tuples into individual parameters + - Determining the return buffer size using the value witness table for the struct type + - Managing enum/struct arguments through registers vs. passing by reference + - Error handling + - `XamGlue`: This component is a library that provides basic support for Swift interop that the generated code builds on. It includes bindings for common built-in types like arrays and dictionaries. It provides a single source of truth for Swift types so that they can be exposed across an assembly boundary. This can be a NuGet package, possibly referencable by `FrameworkReference` or automatically included when targeting macOS, Mac Catalyst, iOS, or tvOS platforms that exposes the platform APIs for each `.framework` that is exposed from Swift to .NET. + - `Tom-Swifty`: This component orchestrates other components and allows users to easily generate Swift projections for a given set of `.framework`s. It presents an interface on top of the tool that is user-friendly and project-system-integrated. + +The flow starts with the source Swift library, which is processed by the Demangler to generate `TLDefinition` objects. Then, the `Swiftmodule parser` aggregates the public API from `.swiftmodule` file and generates a module declaration. Both the `TLDefinition` objects and module declaration are then used as inputs for two distinct components: the `WrappingCompiler` and `OverrideBuilder`. These components generate Swift wrappers source code. The wrappers source code is subsequently processed again by the `Demangler` and `Swiftmodule parser` leading to the generation of wrapper library `TLDefinition` objects and its module declaration. Finally, `TLDefinition` and module declarations, both for the source Swift library and Swift wrappers, are consumed by the `NewClassCompiler` to generate C# bindings. -##### Exporting .NET to Swift +##### Exporting .NET to Swift There are two components to exporting .NET to Swift: Implementing existing Swift types in .NET and passing instances of those types to Swift, and exposing novel types from .NET code to Swift code to be created from Swift. Exposing novel types from .NET code to Swift code is considered out of scope at this time. For implementing existing Swift types in .NET, we will require one of the following tooling options: 1. A Roslyn source generator to generate any supporting code needed to produce any required metadata, such as type metadata and witness tables, to pass instances of Swift-type-implementing .NET types defined in the current project to Swift. -2. An IL-post-processing tool to generate the supporting code and metadata from the compiled assembly. +2. An IL-post-processing tool to generate the supporting code and metadata from the compiled assembly. -If we were to use an IL-post-processing tool here, we would break Hot Reload in assemblies that implement Swift types, even for .NET-only code, due to introducing new tokens that the Hot Reload "client" (aka Roslyn) does not know about. As a result, we should prefer the Roslyn source generator approach. +If we were to use an IL-post-processing tool here, we would break Hot Reload in assemblies that implement Swift types, even for .NET-only code, due to introducing new tokens that the Hot Reload "client" (aka Roslyn) does not know about. As a result, we should prefer the Roslyn source generator approach. The existing tool [supports exporting .NET types to Swift](https://github.com/xamarin/binding-tools-for-swift/blob/main/docs/FutureIdeas.md#exposing-c-to-swift) that can work as a source generator. + +For a C# class that can be inherited gets implemented in C#, the tool generates a mirror class in Swift and overrides it with another class that vectors all the calls through a simulated vtable. This class implements the constructor and overrides the virtual methods by sending calls to a function pointer which ends up in C#. ### Distribution -The calling convention work will be implemented by the .NET runtimes in dotnet/runtime. +The calling convention work will be implemented by the .NET runtimes in dotnet/runtime. The projection tooling will be implemented and shipped as part of the Xamarin publishing infrastructure. The projection tooling will be accessible as a .NET CLI tool and distributed as a NuGet package. It should either be automatically included as part of the TPMs for Apple platforms or should be easily referencable. + +### Validation -The projection tooling will not ship as part of the runtime. It should be available as a separate NuGet package, possibly as a .NET CLI tool package. The projections should either be included automatically as part of the TPMs for macOS, iOS, and tvOS, or should be easily referenceable. +The interop will be showcased through MAUI samples. List of libraries and MAUI samples to support in .NET 9 are outlined in https://github.com/dotnet/runtime/issues/95636. ## Q & A From 9c218640fabb9a86a3ef0fcd3af2691182ab2d69 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Thu, 11 Jan 2024 12:24:15 +0100 Subject: [PATCH 2/9] Simplify importing Swift into .NET; instead of tool details outline the flow with artifacts --- proposed/swift-interop.md | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 64ebc88c0..fc6f6dfb3 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -158,19 +158,13 @@ The projection tooling should be split into these components. ##### Importing Swift into .NET -Importing Swift into .NET is done through bindings generated by the following components: - - `Demangler`: This component analyzes a source Swift library and maps entry points to mangled names. It consumes a source Swift library and creates `TLDefinition` objects that contain type, mangled name, demangled name, module name, and offset within a module. - - `Swiftmodule parser`: This component aggregates the public API. It takes in a `.swiftmodule` file generated by the Swift compiler, similar to a C header file, and combines it with the `TLDefinition` objects to generate a module declaration. - - `WrappingCompiler` and `OverrideBuilder`: These components generate Swift wrappers for cases where direct P/Invoke from C# into Swift is not possible. - - `NewClassCompiler`: This component generates C# bindings based on the generated `TLDefinition` and module declarations. It should implement type mapping and register lowering. Key considerations include: - - Splitting out tuples into individual parameters - - Determining the return buffer size using the value witness table for the struct type - - Managing enum/struct arguments through registers vs. passing by reference - - Error handling - - `XamGlue`: This component is a library that provides basic support for Swift interop that the generated code builds on. It includes bindings for common built-in types like arrays and dictionaries. It provides a single source of truth for Swift types so that they can be exposed across an assembly boundary. This can be a NuGet package, possibly referencable by `FrameworkReference` or automatically included when targeting macOS, Mac Catalyst, iOS, or tvOS platforms that exposes the platform APIs for each `.framework` that is exposed from Swift to .NET. - - `Tom-Swifty`: This component orchestrates other components and allows users to easily generate Swift projections for a given set of `.framework`s. It presents an interface on top of the tool that is user-friendly and project-system-integrated. - -The flow starts with the source Swift library, which is processed by the Demangler to generate `TLDefinition` objects. Then, the `Swiftmodule parser` aggregates the public API from `.swiftmodule` file and generates a module declaration. Both the `TLDefinition` objects and module declaration are then used as inputs for two distinct components: the `WrappingCompiler` and `OverrideBuilder`. These components generate Swift wrappers source code. The wrappers source code is subsequently processed again by the `Demangler` and `Swiftmodule parser` leading to the generation of wrapper library `TLDefinition` objects and its module declaration. Finally, `TLDefinition` and module declarations, both for the source Swift library and Swift wrappers, are consumed by the `NewClassCompiler` to generate C# bindings. +Importing Swift into .NET is done through the following steps: + +1. The tool analyzes a source `.swift` library and maps entry points to mangled names. +2. Then,the tool aggregates the public API; it takes in a `.swiftmodule` file (similar to a C header file) generated by the Swift compiler, creating a module declaration. +3. For cases where direct P/Invoke from C# into Swift is not possible, the tool generates `.swift` source code wrappers along with a `.dylib` compiled native library. +4. Subsequently, the tool aggregates the public API and generates module declarations for the generated wrappers. +5. Finally, the tool generates C# bindings in the form of a `.dll` assembly based on the generated module declarations. ##### Exporting .NET to Swift @@ -187,7 +181,7 @@ For a C# class that can be inherited gets implemented in C#, the tool generates ### Distribution -The calling convention work will be implemented by the .NET runtimes in dotnet/runtime. The projection tooling will be implemented and shipped as part of the Xamarin publishing infrastructure. The projection tooling will be accessible as a .NET CLI tool and distributed as a NuGet package. It should either be automatically included as part of the TPMs for Apple platforms or should be easily referencable. +The calling convention work will be implemented by the .NET runtimes in [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs). The projection tooling will be implemented and shipped as part of the Xamarin publishing infrastructure. The projection tooling will be accessible as a .NET CLI tool and distributed as a NuGet package. It should either be automatically included as part of the TPMs for Apple platforms or should be easily referencable. ### Validation From 76d1433d6fcc8248f68f0994a2fa08d012b38d2e Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Fri, 12 Jan 2024 12:55:53 +0100 Subject: [PATCH 3/9] Avoid tool details and add limitations --- proposed/swift-interop.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index fc6f6dfb3..0ee36fe08 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -160,11 +160,11 @@ The projection tooling should be split into these components. Importing Swift into .NET is done through the following steps: -1. The tool analyzes a source `.swift` library and maps entry points to mangled names. -2. Then,the tool aggregates the public API; it takes in a `.swiftmodule` file (similar to a C header file) generated by the Swift compiler, creating a module declaration. -3. For cases where direct P/Invoke from C# into Swift is not possible, the tool generates `.swift` source code wrappers along with a `.dylib` compiled native library. -4. Subsequently, the tool aggregates the public API and generates module declarations for the generated wrappers. -5. Finally, the tool generates C# bindings in the form of a `.dll` assembly based on the generated module declarations. +1. The tool analyzes a source library `.swift` and maps entry points to mangled names. +2. Then, the tool aggregates the public API; it take in a `.swiftmodule` file (similar to a C header file) generated by the Swift compiler. +3. For cases where direct P/Invoke from C# into Swift is not possible, the tool generates source code `.swift` wrappers and compiles them. +4. Subsequently, the tool aggregates the public API for the generated wrappers. +5. Finally, the tool generates C# bindings source code based on the aggregated public API and compiles them. ##### Exporting .NET to Swift @@ -179,6 +179,13 @@ If we were to use an IL-post-processing tool here, we would break Hot Reload in For a C# class that can be inherited gets implemented in C#, the tool generates a mirror class in Swift and overrides it with another class that vectors all the calls through a simulated vtable. This class implements the constructor and overrides the virtual methods by sending calls to a function pointer which ends up in C#. +##### Limitations of the tool + +The current version of the projection tooling has limitations and, in some cases, requires Swift wrappers in addition to C# bindings. Here are some examples: +1. Passing a struct by value in more registers than P/Invoke will allow. In this case, it requires "unpacking" from the Swift side. +2. Exporting .NET types into Swift. +3. Handling closures. + ### Distribution The calling convention work will be implemented by the .NET runtimes in [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs). The projection tooling will be implemented and shipped as part of the Xamarin publishing infrastructure. The projection tooling will be accessible as a .NET CLI tool and distributed as a NuGet package. It should either be automatically included as part of the TPMs for Apple platforms or should be easily referencable. From ea4d636bea00e31027eccdb6d680f5ff65bce7b1 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Tue, 16 Jan 2024 13:37:10 +0100 Subject: [PATCH 4/9] Add projecting Swift into .NET for various types based on the current support --- proposed/swift-interop.md | 43 +++++++++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 8 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 0ee36fe08..1ea9e0973 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -132,6 +132,40 @@ The majority of the work for Swift/.NET interop is determining how a type that e All designs in this section should be designed such that they are trimming and AOT-compatible by construction. We should work to ensure that no feature requires whole-program analysis (such as custom steps in the IL Linker) to be trim or AOT compatible. +The following subheadings describe the current projection of Swift types into C#, and can be used as a starting point for potential design improvements. + +#### Structs/Enums + +Swift structs and enums can be projected into C# as either original types or C# classes. + +Some Swift structs are of scalar types which are essentially structs with one or more fields that contain identical blittable types, making them directly translatable to C# as structs. On the other hand, there are Swift structs that do not fit the scalar model. These non-scalar structs require representation in C# as IDisposable classes. The reason for using IDisposable is that Swift structs behave differently from C# value types. They have distinct semantics when they enter and exit scope, potentially involving changes to reference counts or destructors. + +Swift enums can be classified as integral when they have an integral raw type or when every element has an integral payload. If an enum's payload types are all identical, it's considered homogeneous. An enum is trivial when it lacks inheritance, raw types, and none of its elements have payloads, or when it's both homogeneous and integral. These trivial enums can be represented as C# enums. Non-trivial enums are represented in C# as IDisposable classes. + +These C# classes typically have a single property, SwiftData, which holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by marshaler in cases when it is a return value from a function. The projection tooling implements Swift types as ISwiftEnum and ISwiftStruct. + +Additionally, scalar structs and trivial enums require a lowering process. During this process, the layout of the struct or enum type is recursively flattened into a sequence of primitives. If this sequence has a length of 4 or less, the values of this type are split into the elements of this sequence for parameter passing, rather than passing the struct as a whole. + +#### Class/Protocols + +The Swift classes can be mapped to C# classes, but there are distinctions between final and virtual classes. In C#, public Swift classes are represented as final classes, while virtual Swift classes are represented as internal classes. Final classes have a straightforward inheritance model, while virtual classes introduce more complexity, particularly related to subclassing and simulated vtable method. Additionally, a C# binding class contains two static methods: a static factory method used to construct a C# binding instance from a Swift handle, typically used after marshaling from Swift to C#; and a static method used when a Swift handle needs to be constructed from C#. + +Another important aspect of Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable external to the type. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. + +Swift protocols can be mapped to C# interfaces, but there is not always a 1:1 mapping for each protocol. A Swift object that conforms to a protocol routes its methods through a vtable with function pointers in C#. These functions then marshal the call into a C# object. The C# bindings for a protocol include an interface matching the Swift protocol and a proxy class that contains the interface implementation and implements the interface itself. The proxy class serves as an interface implementation when there is either a C# type that needs to look like a Swift type of a Swift type that needs to look like a C# type. The proxy class in C# implements a static constructor for vtable setup, a vtable parallel to Swift's, static methods for each vtable entry, and a static property for accessing the protocol witness table. + +Swift wrappers define a vtable structure that acts as a proxy for C# to receive and respond to calls made by Swift for each method in a Swift protocol. + +#### Tuples/Closures + +Swift uses two types of closures: escaping and non-escaping. Escaping closures can exist beyond their original context, while non-escaping closures should not persist beyond their declaration context and cannot reference external data. The Swift compiler has a mechanism to convert a non-escaping closure into an escaping one within the scope of another closure. + +Any escaping closure can be mapped into C# as `(args) -> return` into `(UnsafeMutablePointer, OpaquePointer) -> ()`. This transformation allows C# to call it as an `Action`. + +#### Error handling + +The tooling will support the conversion of errors and exceptions between Swift and C#. The Swift runtime library implementation in C# is available at https://github.com/xamarin/binding-tools-for-swift/tree/main/SwiftRuntimeLibrary. + #### Swift to .NET Language Feature Projections ##### Structs/Value Types @@ -144,7 +178,7 @@ Unlike .NET, Swift's struct types have strong lifetime semantics more similar to Structs that are non-bitwise-movable are more difficult. They cannot be moved by copying their bits; their copy constructors must be used in all copy scenarios. When mapping these structs to C#, we must take care that we do not copy the underlying memory and to call the deallocate function when the C# usage of the struct falls out of scope. These use cases best match up to C# class semantics, not struct semantics. -We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. The size and layout information of a struct is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. +We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. When a value type has opaque layout, the actual size and layout of fields is not known at compilation time, but only at runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. ##### Tuples @@ -179,13 +213,6 @@ If we were to use an IL-post-processing tool here, we would break Hot Reload in For a C# class that can be inherited gets implemented in C#, the tool generates a mirror class in Swift and overrides it with another class that vectors all the calls through a simulated vtable. This class implements the constructor and overrides the virtual methods by sending calls to a function pointer which ends up in C#. -##### Limitations of the tool - -The current version of the projection tooling has limitations and, in some cases, requires Swift wrappers in addition to C# bindings. Here are some examples: -1. Passing a struct by value in more registers than P/Invoke will allow. In this case, it requires "unpacking" from the Swift side. -2. Exporting .NET types into Swift. -3. Handling closures. - ### Distribution The calling convention work will be implemented by the .NET runtimes in [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs). The projection tooling will be implemented and shipped as part of the Xamarin publishing infrastructure. The projection tooling will be accessible as a .NET CLI tool and distributed as a NuGet package. It should either be automatically included as part of the TPMs for Apple platforms or should be easily referencable. From 0bb0e3cc08bdb1fa04c9fe194232b07e5ee6e9f4 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Tue, 23 Jan 2024 15:05:14 +0100 Subject: [PATCH 5/9] Update binding components based on feedback --- proposed/swift-interop.md | 63 +++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 33 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 1ea9e0973..934c4484a 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -132,61 +132,58 @@ The majority of the work for Swift/.NET interop is determining how a type that e All designs in this section should be designed such that they are trimming and AOT-compatible by construction. We should work to ensure that no feature requires whole-program analysis (such as custom steps in the IL Linker) to be trim or AOT compatible. -The following subheadings describe the current projection of Swift types into C#, and can be used as a starting point for potential design improvements. - -#### Structs/Enums +#### Swift to .NET Language Feature Projections -Swift structs and enums can be projected into C# as either original types or C# classes. +The following subheadings describe the intended projection of Swift types into C#, and can be used as a starting point for potential design improvements. -Some Swift structs are of scalar types which are essentially structs with one or more fields that contain identical blittable types, making them directly translatable to C# as structs. On the other hand, there are Swift structs that do not fit the scalar model. These non-scalar structs require representation in C# as IDisposable classes. The reason for using IDisposable is that Swift structs behave differently from C# value types. They have distinct semantics when they enter and exit scope, potentially involving changes to reference counts or destructors. +##### Structs/Enums -Swift enums can be classified as integral when they have an integral raw type or when every element has an integral payload. If an enum's payload types are all identical, it's considered homogeneous. An enum is trivial when it lacks inheritance, raw types, and none of its elements have payloads, or when it's both homogeneous and integral. These trivial enums can be represented as C# enums. Non-trivial enums are represented in C# as IDisposable classes. +Unlike .NET, Swift's struct types have strong lifetime semantics more similar to C++ types than .NET structs. At the Swift ABI layer, there are broadly three types of structs/enums: "POD/Trivial" structs, "Bitwise Takable/Movable" structs, and non-bitwise movable structs. The [Swift documentation](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#layout-and-properties-of-types) covers these different kinds of structs. Let's look at how we could map each of these categories of structs into .NET. -These C# classes typically have a single property, SwiftData, which holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by marshaler in cases when it is a return value from a function. The projection tooling implements Swift types as ISwiftEnum and ISwiftStruct. +"POD/Trivial" structs have no memory management required and no special logic for copying/moving/deleting the struct instance. Structs of this category can be represented as C# structs with the same field layout. -Additionally, scalar structs and trivial enums require a lowering process. During this process, the layout of the struct or enum type is recursively flattened into a sequence of primitives. If this sequence has a length of 4 or less, the values of this type are split into the elements of this sequence for parameter passing, rather than passing the struct as a whole. +"Bitwise Takable/Movable" structs have some memory management logic and require calls to Swift's ref-counting machinery to maintain expected lifetimes. Structs of this category can be projected into C# as a struct. When creating this C# struct, we would semantically treat each field as a separate local, create the C# projection of it, and save this "local" value into a field in the C# struct. -#### Class/Protocols +Structs that are non-bitwise-movable are more difficult. They cannot be moved by copying their bits; their copy constructors must be used in all copy scenarios. When mapping these structs to C#, we must take care that we do not copy the underlying memory and to call the deallocate function when the C# usage of the struct falls out of scope. These use cases best match up to C# class semantics, not struct semantics. -The Swift classes can be mapped to C# classes, but there are distinctions between final and virtual classes. In C#, public Swift classes are represented as final classes, while virtual Swift classes are represented as internal classes. Final classes have a straightforward inheritance model, while virtual classes introduce more complexity, particularly related to subclassing and simulated vtable method. Additionally, a C# binding class contains two static methods: a static factory method used to construct a C# binding instance from a Swift handle, typically used after marshaling from Swift to C#; and a static method used when a Swift handle needs to be constructed from C#. +We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. When a value type has opaque layout, the actual size and layout of fields is not known at compilation time, but only at runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. -Another important aspect of Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable external to the type. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. +Swift structs and enums can be projected into C# as IDisposable classes. -Swift protocols can be mapped to C# interfaces, but there is not always a 1:1 mapping for each protocol. A Swift object that conforms to a protocol routes its methods through a vtable with function pointers in C#. These functions then marshal the call into a C# object. The C# bindings for a protocol include an interface matching the Swift protocol and a proxy class that contains the interface implementation and implements the interface itself. The proxy class serves as an interface implementation when there is either a C# type that needs to look like a Swift type of a Swift type that needs to look like a C# type. The proxy class in C# implements a static constructor for vtable setup, a vtable parallel to Swift's, static methods for each vtable entry, and a static property for accessing the protocol witness table. +Some Swift structs are of scalar types which are essentially structs with one or more fields that contain identical blittable types. On the other hand, there are Swift structs that do not fit the scalar model. Both types are translated to C# as IDisposable classes. The reason for using IDisposable is that Swift structs behave differently from C# value types. They have distinct semantics when they enter and exit scope, potentially involving changes to reference counts or destructors. -Swift wrappers define a vtable structure that acts as a proxy for C# to receive and respond to calls made by Swift for each method in a Swift protocol. +Swift enums can be classified as integral when they have an integral raw type or when every element has an integral payload. If an enum's payload types are all identical, it's considered homogeneous. An enum is trivial when it lacks inheritance, raw types, and none of its elements have payloads, or when it's both homogeneous and integral. These trivial enums can be mapped to C# enums, but for simplicity they are instead translated into IDisposable classes. Non-trivial enums are represented in C# as IDisposable classes. -#### Tuples/Closures +These C# classes typically have a single property, SwiftData, which holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by marshaler in cases when it is a return value from a function. To manage memory lifetimes, the class also includes a finalizer to ensure proper memory release. -Swift uses two types of closures: escaping and non-escaping. Escaping closures can exist beyond their original context, while non-escaping closures should not persist beyond their declaration context and cannot reference external data. The Swift compiler has a mechanism to convert a non-escaping closure into an escaping one within the scope of another closure. +Additionally, scalar structs and trivial enums require a lowering process. During this process, the layout of the struct or enum type is recursively flattened into a sequence of primitives. If this sequence has a length of 4 or less, the values of this type are split into the elements of this sequence for parameter passing, rather than passing the struct as a whole. -Any escaping closure can be mapped into C# as `(args) -> return` into `(UnsafeMutablePointer, OpaquePointer) -> ()`. This transformation allows C# to call it as an `Action`. +##### Classes/Protocols -#### Error handling +The Swift classes can be mapped to C# classes, but there are distinctions between final and virtual classes. In C#, public Swift classes are represented as final classes, while virtual Swift classes are represented as internal classes. Final classes have a straightforward inheritance model, while virtual classes introduce more complexity, particularly related to subclassing and simulated vtable method. Additionally, a C# binding class contains two static methods: a static factory method used to construct a C# binding instance from a Swift handle, typically used after marshaling from Swift to C#; and a static method used when a Swift handle needs to be constructed from C#. The class implements static abstract interface to avoid use of reflection for methods discovering and to ensure linker-safe construction/compile time resolution. -The tooling will support the conversion of errors and exceptions between Swift and C#. The Swift runtime library implementation in C# is available at https://github.com/xamarin/binding-tools-for-swift/tree/main/SwiftRuntimeLibrary. +Another important aspect of Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable external to the type. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. -#### Swift to .NET Language Feature Projections +Swift protocols can be mapped to C# interfaces, but there is not always a 1:1 mapping for each protocol. A Swift object that conforms to a protocol routes its methods through a vtable with function pointers in C#. These functions then marshal the call into a C# object. The C# bindings for a protocol include an interface matching the Swift protocol and a proxy class that contains the interface implementation and implements the interface itself. The proxy class serves as an interface implementation when there is either a C# type that needs to look like a Swift type of a Swift type that needs to look like a C# type. The proxy class in C# implements a static constructor for vtable setup, a vtable parallel to Swift's, static methods for each vtable entry, and a static property for accessing the protocol witness table. -##### Structs/Value Types +Swift wrappers define a vtable structure that acts as a proxy for C# to receive and respond to calls made by Swift for each method in a Swift protocol. -Unlike .NET, Swift's struct types have strong lifetime semantics more similar to C++ types than .NET structs. At the Swift ABI layer, there are broadly three types of structs/enums: "POD/Trivial" structs, "Bitwise Takable/Movable" structs, and non-bitwise movable structs. The [Swift documentation](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#layout-and-properties-of-types) covers these different kinds of structs. Let's look at how we could map each of these categories of structs into .NET. +##### Tuples/Closures -"POD/Trivial" structs have no memory management required and no special logic for copying/moving/deleting the struct instance. Structs of this category can be represented as C# structs with the same field layout. +If possible, Swift tuples should be represented as `ValueTuple`s in .NET. If this is not possible, then they should be represented as types with a `Deconstruct` method similar to `ValueTuple` to allow a tuple-like experience in C#. -"Bitwise Takable/Movable" structs have some memory management logic and require calls to Swift's ref-counting machinery to maintain expected lifetimes. Structs of this category can be projected into C# as a struct. When creating this C# struct, we would semantically treat each field as a separate local, create the C# projection of it, and save this "local" value into a field in the C# struct. +Swift uses two types of closures: escaping and non-escaping. Escaping closures can exist beyond their original context, while non-escaping closures should not persist beyond their declaration context and cannot reference external data. The Swift compiler has a mechanism to convert a non-escaping closure into an escaping one within the scope of another closure. -Structs that are non-bitwise-movable are more difficult. They cannot be moved by copying their bits; their copy constructors must be used in all copy scenarios. When mapping these structs to C#, we must take care that we do not copy the underlying memory and to call the deallocate function when the C# usage of the struct falls out of scope. These use cases best match up to C# class semantics, not struct semantics. +Any escaping closure can be mapped into C# as `(args) -> return` into `(UnsafeMutablePointer, OpaquePointer) -> ()`. This transformation allows C# to call it as an `Action`. -We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. When a value type has opaque layout, the actual size and layout of fields is not known at compilation time, but only at runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. +##### Error handling -##### Tuples +The tooling will support the conversion of errors and exceptions between Swift and C#. The Swift runtime library implementation in C# is available at https://github.com/xamarin/binding-tools-for-swift/tree/main/SwiftRuntimeLibrary.å -If possible, Swift tuples should be represented as `ValueTuple`s in .NET. If this is not possible, then they should be represented as types with a `Deconstruct` method similar to `ValueTuple` to allow a tuple-like experience in C#. #### Projection Tooling Components -The projection tooling will be based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift) (BTfS). The BTfS contains a set of tools that can consume a compiled Apple Swift library and generate wrappers (bindings) that allow it to be surfaced as a .NET library. This tool will be implemented as a self-hosted .NET CLI tool, automating the generation of wrappers by parsing the Swift library interface. It generates C# bindings that can be utilized as a .NET library and Swift bindings that C# bindings invoke in cases where direct P/Invoke into Swift library is not possible. +The projection tooling will be based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift) (BTfS). The BTfS contains a set of tools that can consume a compiled Apple Swift library and generate wrappers (bindings) that allow it to be surfaced as a .NET library. This tool will be implemented as a self-hosted .NET CLI tool, automating the generation of wrappers by parsing the Swift library interface. It generates C# bindings that can be utilized as a .NET library. The projection tooling will utilize the runtime core interop source-gen infrastructure to implement marshalling codegen. Sharing the code will allow the Swift interop story to gain/share improvements and features with the rest of the interop source generators. The projection tooling should be split into these components. @@ -194,11 +191,11 @@ The projection tooling should be split into these components. Importing Swift into .NET is done through the following steps: -1. The tool analyzes a source library `.swift` and maps entry points to mangled names. -2. Then, the tool aggregates the public API; it take in a `.swiftmodule` file (similar to a C header file) generated by the Swift compiler. -3. For cases where direct P/Invoke from C# into Swift is not possible, the tool generates source code `.swift` wrappers and compiles them. -4. Subsequently, the tool aggregates the public API for the generated wrappers. -5. Finally, the tool generates C# bindings source code based on the aggregated public API and compiles them. +1. The tool analyzes a `.swiftmodule` file (similar to a C header file) and maps entry points to mangled names. +2. Then, the tool aggregates the public API; it take in a `.swiftmodule` file generated by the Swift compiler. +3. Finally, the tool generates C# bindings source code based on the aggregated public API and compiles them. + +The projection tooling currently generates Swift wrappers when direct P/Invoke from C# into Swift is not feasible. One of the objectives of this effort is to reduce the need for Swift wrappers. In cases where it is impossible to avoid using Swift wrappers, the responsibility will be on users to generate them. ##### Exporting .NET to Swift From 722a72c2f79ac2d67054304d82148849714d6643 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Thu, 8 Feb 2024 14:50:40 +0100 Subject: [PATCH 6/9] Update tooling layout and reduce set of C# types to illustrate the main aspects of projections --- proposed/swift-interop.md | 54 +++++++++++++++------------------------ 1 file changed, 21 insertions(+), 33 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index f0f80669a..914a35fd7 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -142,7 +142,7 @@ All designs in this section should be designed such that they are trimming and A #### Swift to .NET Language Feature Projections -The following subheadings describe the intended projection of Swift types into C#, and can be used as a starting point for potential design improvements. +The following subheadings describe projections of Swift types into C#. This section illustrates general mechanisms and practices applied in the tooling projection. The complete documentation of the projection tooling will be available at https://github.com/dotnet/runtimelab/tree/feature/swift-bindings/docs. ##### Structs/Enums @@ -154,39 +154,27 @@ Unlike .NET, Swift's struct types have strong lifetime semantics more similar to Structs that are non-bitwise-movable are more difficult. They cannot be moved by copying their bits; their copy constructors must be used in all copy scenarios. When mapping these structs to C#, we must take care that we do not copy the underlying memory and to call the deallocate function when the C# usage of the struct falls out of scope. These use cases best match up to C# class semantics, not struct semantics. -We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. When a value type has opaque layout, the actual size and layout of fields is not known at compilation time, but only at runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. +We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. -Swift structs and enums can be projected into C# as IDisposable classes. +Swift structs and enums have richer semantics than in .NET and are projected as C# classes which implement `IDisposable` interface to streamline handling of both simple blittable and more complex scenarios. These C# classes have a single property that holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by the marshaler in cases when it is a return value from a function. -Some Swift structs are of scalar types which are essentially structs with one or more fields that contain identical blittable types. On the other hand, there are Swift structs that do not fit the scalar model. Both types are translated to C# as IDisposable classes. The reason for using IDisposable is that Swift structs behave differently from C# value types. They have distinct semantics when they enter and exit scope, potentially involving changes to reference counts or destructors. - -Swift enums can be classified as integral when they have an integral raw type or when every element has an integral payload. If an enum's payload types are all identical, it's considered homogeneous. An enum is trivial when it lacks inheritance, raw types, and none of its elements have payloads, or when it's both homogeneous and integral. These trivial enums can be mapped to C# enums, but for simplicity they are instead translated into IDisposable classes. Non-trivial enums are represented in C# as IDisposable classes. - -These C# classes typically have a single property, SwiftData, which holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by marshaler in cases when it is a return value from a function. To manage memory lifetimes, the class also includes a finalizer to ensure proper memory release. - -Additionally, scalar structs and trivial enums require a lowering process. During this process, the layout of the struct or enum type is recursively flattened into a sequence of primitives. If this sequence has a length of 4 or less, the values of this type are split into the elements of this sequence for parameter passing, rather than passing the struct as a whole. +Swift primitive types are implemented as frozen structs that conform to Swift-specific lowering processes handled by the runtime. However, most of these types are below the size limit for being passed by reference and can fit within the underlying calling convention. ##### Classes/Protocols -The Swift classes can be mapped to C# classes, but there are distinctions between final and virtual classes. In C#, public Swift classes are represented as final classes, while virtual Swift classes are represented as internal classes. Final classes have a straightforward inheritance model, while virtual classes introduce more complexity, particularly related to subclassing and simulated vtable method. Additionally, a C# binding class contains two static methods: a static factory method used to construct a C# binding instance from a Swift handle, typically used after marshaling from Swift to C#; and a static method used when a Swift handle needs to be constructed from C#. The class implements static abstract interface to avoid use of reflection for methods discovering and to ensure linker-safe construction/compile time resolution. - -Another important aspect of Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable external to the type. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. +A public Swift class is projected as a final C# class, while a virtual Swift class is projected as an internal C# class. A final class has a straightforward inheritance model, while a virtual class introduces more complexity, particularly related to subclassing and simulated vtable methods. Another important type in Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable for each conformance. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. -Swift protocols can be mapped to C# interfaces, but there is not always a 1:1 mapping for each protocol. A Swift object that conforms to a protocol routes its methods through a vtable with function pointers in C#. These functions then marshal the call into a C# object. The C# bindings for a protocol include an interface matching the Swift protocol and a proxy class that contains the interface implementation and implements the interface itself. The proxy class serves as an interface implementation when there is either a C# type that needs to look like a Swift type of a Swift type that needs to look like a C# type. The proxy class in C# implements a static constructor for vtable setup, a vtable parallel to Swift's, static methods for each vtable entry, and a static property for accessing the protocol witness table. +Projections can utilize [`IUnmanagedVirtualMethodTableProvider` interface](https://learn.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.marshalling.iunmanagedvirtualmethodtableprovider?view=net-8.0) to retrieve vtable for a given target type from an object. -Swift wrappers define a vtable structure that acts as a proxy for C# to receive and respond to calls made by Swift for each method in a Swift protocol. +This subheading will be updated with more details on projection once simpler types are reviewed, like structs and enums. ##### Tuples/Closures -If possible, Swift tuples should be represented as `ValueTuple`s in .NET. If this is not possible, then they should be represented as types with a `Deconstruct` method similar to `ValueTuple` to allow a tuple-like experience in C#. +A Swift tuple can be projected as `ValueTuple` type in C#. Swift uses two types of closures: escaping and non-escaping. Escaping closures can exist beyond their original context, while non-escaping closures should not persist beyond their declaration context and cannot reference external data. The Swift compiler has a mechanism to convert a non-escaping closure into an escaping one within the scope of another closure. -Swift uses two types of closures: escaping and non-escaping. Escaping closures can exist beyond their original context, while non-escaping closures should not persist beyond their declaration context and cannot reference external data. The Swift compiler has a mechanism to convert a non-escaping closure into an escaping one within the scope of another closure. +This subheading will be updated with more details on projection once simpler types are reviewed, like structs and enums. -Any escaping closure can be mapped into C# as `(args) -> return` into `(UnsafeMutablePointer, OpaquePointer) -> ()`. This transformation allows C# to call it as an `Action`. - -##### Error handling - -The tooling will support the conversion of errors and exceptions between Swift and C#. The Swift runtime library implementation in C# is available at https://github.com/xamarin/binding-tools-for-swift/tree/main/SwiftRuntimeLibrary.å +##### SIMD types Swift has its own built-in SIMD types; however they're named based on the number of elements, not based on the width of the vector type. For example, Swift has `SIMD2`, `SIMD4`, up to `SIMD64`. When the instantiations of these types correspond to an intrinsic vector type, they are treated as that type. Otherwise, they are treated as a struct of vectors. In .NET, our vector types are named based on their vector with, so `Vector128`, `Vector256`, etc. @@ -199,19 +187,19 @@ As mentioned in the calling-convention section above, none of the libraries we a #### Projection Tooling Components -The projection tooling will be based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift) (BTfS). The BTfS contains a set of tools that can consume a compiled Apple Swift library and generate wrappers (bindings) that allow it to be surfaced as a .NET library. This tool will be implemented as a self-hosted .NET CLI tool, automating the generation of wrappers by parsing the Swift library interface. It generates C# bindings that can be utilized as a .NET library. The projection tooling will utilize the runtime core interop source-gen infrastructure to implement marshalling codegen. Sharing the code will allow the Swift interop story to gain/share improvements and features with the rest of the interop source generators. +The projection tooling will be based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift). The tooling will contain components that can consume a compiled Apple Swift library interface (`.swiftinterface`) and generate C# bindings that allow it to be surfaced as a .NET library. The tool will not generate any Swift wrappers and it's users responsibility to provide Swift wrappers for cases where direct binding is not possible. The projection tooling will utilize the runtime core interop source-gen infrastructure to implement marshalling codegen. -The projection tooling should be split into these components. +The projection tooling is split into these components. ##### Importing Swift into .NET Importing Swift into .NET is done through the following steps: -1. The tool analyzes a `.swiftmodule` file (similar to a C header file) and maps entry points to mangled names. -2. Then, the tool aggregates the public API; it take in a `.swiftmodule` file generated by the Swift compiler. +1. The tool analyzes a `.swiftinterface` file (similar to a C header file) and maps entry points to mangled names. +2. Then, the tool aggregates the public API; it take in a `.swiftinterface` file generated by the Swift compiler. 3. Finally, the tool generates C# bindings source code based on the aggregated public API and compiles them. -The projection tooling currently generates Swift wrappers when direct P/Invoke from C# into Swift is not feasible. One of the objectives of this effort is to reduce the need for Swift wrappers. In cases where it is impossible to avoid using Swift wrappers, the responsibility will be on users to generate them. +The projection tooling should provide projections of Swift primitive types and expose runtime functions for metadata handling. ##### Exporting .NET to Swift @@ -220,19 +208,19 @@ There are two components to exporting .NET to Swift: Implementing existing Swift For implementing existing Swift types in .NET, we will require one of the following tooling options: 1. A Roslyn source generator to generate any supporting code needed to produce any required metadata, such as type metadata and witness tables, to pass instances of Swift-type-implementing .NET types defined in the current project to Swift. -2. An IL-post-processing tool to generate the supporting code and metadata from the compiled assembly. +2. An IL-post-processing tool to generate the supporting code and metadata from the compiled assembly. -If we were to use an IL-post-processing tool here, we would break Hot Reload in assemblies that implement Swift types, even for .NET-only code, due to introducing new tokens that the Hot Reload "client" (aka Roslyn) does not know about. As a result, we should prefer the Roslyn source generator approach. The existing tool [supports exporting .NET types to Swift](https://github.com/xamarin/binding-tools-for-swift/blob/main/docs/FutureIdeas.md#exposing-c-to-swift) that can work as a source generator. - -For a C# class that can be inherited gets implemented in C#, the tool generates a mirror class in Swift and overrides it with another class that vectors all the calls through a simulated vtable. This class implements the constructor and overrides the virtual methods by sending calls to a function pointer which ends up in C#. +If we want to use an IL-post-processing tool here, we would break Hot Reload in assemblies that implement Swift types, even for .NET-only code, due to introducing new tokens that the Hot Reload "client" (aka Roslyn) does not know about. ### Distribution -The calling convention work will be implemented by the .NET runtimes in [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs). The projection tooling will be implemented and shipped as part of the Xamarin publishing infrastructure. The projection tooling will be accessible as a .NET CLI tool and distributed as a NuGet package. It should either be automatically included as part of the TPMs for Apple platforms or should be easily referencable. +The projection tooling will be implemented as a .NET CLI tool and integrated into the Xamarin publishing infrastructure. It will be included in the macios workload for Apple platforms and also available as a standalone package independent of MAUI framework. + +The calling convention work will be implemented in the [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs) repository. ### Validation -The interop will be showcased through MAUI samples. List of libraries and MAUI samples to support in .NET 9 are outlined in https://github.com/dotnet/runtime/issues/95636. +The interop will be showcased through CryptoKit library in the runtime repository and MAUI libraries and samples: https://github.com/dotnet/runtime/issues/95636. ## Q & A From 4ccb5013b08dccc709c34e8bae0816b71b32c163 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Thu, 22 Feb 2024 14:42:22 +0100 Subject: [PATCH 7/9] Update documentation based on initial implementation of the tooling --- proposed/swift-interop.md | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 914a35fd7..3d9eb78c1 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -142,7 +142,11 @@ All designs in this section should be designed such that they are trimming and A #### Swift to .NET Language Feature Projections -The following subheadings describe projections of Swift types into C#. This section illustrates general mechanisms and practices applied in the tooling projection. The complete documentation of the projection tooling will be available at https://github.com/dotnet/runtimelab/tree/feature/swift-bindings/docs. +The following subheadings describe projections of Swift types into C#. This section illustrates general mechanisms and practices applied in the tooling projection. The complete documentation of the projection tooling is available at https://github.com/dotnet/runtimelab/tree/feature/swift-bindings/docs. + +##### Primitive types + +Swift primitive types are implemented as frozen structs that conform to Swift-specific lowering processes handled by the runtime. However, most of these types are below the size limit for being passed by reference and can fit within the underlying calling convention. ##### Structs/Enums @@ -158,8 +162,6 @@ We plan to interop with Swift's Library Evolution mode, which brings an addition Swift structs and enums have richer semantics than in .NET and are projected as C# classes which implement `IDisposable` interface to streamline handling of both simple blittable and more complex scenarios. These C# classes have a single property that holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by the marshaler in cases when it is a return value from a function. -Swift primitive types are implemented as frozen structs that conform to Swift-specific lowering processes handled by the runtime. However, most of these types are below the size limit for being passed by reference and can fit within the underlying calling convention. - ##### Classes/Protocols A public Swift class is projected as a final C# class, while a virtual Swift class is projected as an internal C# class. A final class has a straightforward inheritance model, while a virtual class introduces more complexity, particularly related to subclassing and simulated vtable methods. Another important type in Swift is protocols. Swift allows any type to implement a protocol and supports retroactive modeling through extensions. Since the protocol's implementation can't be part of the object, Swift uses a [Protocol Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#protocol-witness-tables), which functions like a vtable for each conformance. A protocol type in Swift is represented using an [existential container](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#existential-metadata) that includes payload, type metadata pointer, and a protocol witness table pointer. @@ -187,19 +189,7 @@ As mentioned in the calling-convention section above, none of the libraries we a #### Projection Tooling Components -The projection tooling will be based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift). The tooling will contain components that can consume a compiled Apple Swift library interface (`.swiftinterface`) and generate C# bindings that allow it to be surfaced as a .NET library. The tool will not generate any Swift wrappers and it's users responsibility to provide Swift wrappers for cases where direct binding is not possible. The projection tooling will utilize the runtime core interop source-gen infrastructure to implement marshalling codegen. - -The projection tooling is split into these components. - -##### Importing Swift into .NET - -Importing Swift into .NET is done through the following steps: - -1. The tool analyzes a `.swiftinterface` file (similar to a C header file) and maps entry points to mangled names. -2. Then, the tool aggregates the public API; it take in a `.swiftinterface` file generated by the Swift compiler. -3. Finally, the tool generates C# bindings source code based on the aggregated public API and compiles them. - -The projection tooling should provide projections of Swift primitive types and expose runtime functions for metadata handling. +The projection tooling is based on the [Binding Tools for Swift](https://github.com/xamarin/binding-tools-for-swift). The tooling contain components that can consume a compiled Apple Swift library interface and generate C# source code bindings that allow it to be surfaced as a .NET library. The tool will not generate any Swift wrappers and it's users responsibility to provide Swift wrappers for cases where direct binding is not possible. The projection tooling will utilize the runtime core interop source-gen infrastructure to implement marshalling codegen. ##### Exporting .NET to Swift @@ -216,7 +206,7 @@ If we want to use an IL-post-processing tool here, we would break Hot Reload in The projection tooling will be implemented as a .NET CLI tool and integrated into the Xamarin publishing infrastructure. It will be included in the macios workload for Apple platforms and also available as a standalone package independent of MAUI framework. -The calling convention work will be implemented in the [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs) repository. +The calling convention work will be implemented in the [dotnet/runtime](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/Swift/SwiftTypes.cs) repository. ### Validation From b9325c61d5941b0a48fb7370cdf0adda6886fe1c Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Fri, 23 Feb 2024 11:56:58 +0100 Subject: [PATCH 8/9] Update Swift interop documentation for memory management --- proposed/swift-interop.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 3d9eb78c1..69a146597 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -130,7 +130,11 @@ CoreCLR and NativeAOT currently block the `VectorX` types from P/Invokes as t ##### Automatic Reference Counting and Lifetime Management -Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. Unmanaged objects from C# should either implement `IDisposable` or utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the `NativeMemory` class, to explicitly release memory. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. +Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. + +The `IDisposable` provides an explicit mechanism for releasing unmanaged resources. Destructors are managed by the GC and offer a way to release unmanaged resources when an object is collected by the GC. While destructors abstract away memory management from the user, the `Idisposable` pattern provides deterministic control over when resources and can lead to better performance as it prevents the need for GC collection cycles. The `IDisposable` pattern is the typical .NET approach for dealing with unmanaged resources and thus is selected as default option at initial stage. If it is determined that the `IDisposable` pattern introduces unnecessary overhead for users, and that destructors can adequately manage the release of unmanaged resources, appropriate updates to the memory management approach will be made. + +Unmanaged objects from C# should either implement `IDisposable` or utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the `NativeMemory` class, to explicitly release memory. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. The Binding Tools for Swift tooling handles these explicit lifetime semantics with generated Swift code. In the new Swift/.NET interop, management of these lifetime semantics will be done by the Swift projection layer and not by the raw calling-convention support. If any GC interaction is required to handle the lifetime semantics correctly, we should take an approach more similar to the `ComWrappers` support (higher-level, less complex interop interface) rather than the Objective-C interop support (lower-level, basically only usable by the ObjCRuntime implementation). From 84b2fa4494c23d52c206dcbf38613c495dbeef45 Mon Sep 17 00:00:00 2001 From: Milos Kotlar Date: Fri, 8 Mar 2024 12:38:16 +0100 Subject: [PATCH 9/9] Update memory management destructors and IDisposable section --- proposed/swift-interop.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/proposed/swift-interop.md b/proposed/swift-interop.md index 69a146597..ffd0008a3 100644 --- a/proposed/swift-interop.md +++ b/proposed/swift-interop.md @@ -132,11 +132,13 @@ CoreCLR and NativeAOT currently block the `VectorX` types from P/Invokes as t Swift has a strongly-defined lifetime and ownership model. This model is specified in the Swift ABI and is similar to Objective-C's ARC (Automatic Reference Counting) system. When .NET calls into Swift, the .NET GC is responsible for managing all managed objects. -The `IDisposable` provides an explicit mechanism for releasing unmanaged resources. Destructors are managed by the GC and offer a way to release unmanaged resources when an object is collected by the GC. While destructors abstract away memory management from the user, the `Idisposable` pattern provides deterministic control over when resources and can lead to better performance as it prevents the need for GC collection cycles. The `IDisposable` pattern is the typical .NET approach for dealing with unmanaged resources and thus is selected as default option at initial stage. If it is determined that the `IDisposable` pattern introduces unnecessary overhead for users, and that destructors can adequately manage the release of unmanaged resources, appropriate updates to the memory management approach will be made. +There are two strategies for managing native memory in .NET: destructors and `IDisposable`. Destructors abstract away memory management from the user and are managed by the GC. They provide a way to release unmanaged resources when an object is collected by the GC. `IDisposable` offers an explicit mechanism for releasing unmanaged resources with deterministic control over when resources are released. The preferred behavior for general cases would be to implement destructors. This approach aligns with the .NET pattern and offers codegen benefits by avoiding excessive `using` statements. Ideally, the tooling should only use `IDisposable` for custom deinit. -Unmanaged objects from C# should either implement `IDisposable` or utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the `NativeMemory` class, to explicitly release memory. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. +Using `IDisposable` can lead to better overall performance as it prevents the need for GC collection cycles. However, implementing Swift types as `IDisposable` in .NET can be confusing for customers. In that case, every Swift type would come with `IDisposable`, and without knowing the details of the type, it may be challenging for the caller to determine whether to explicitly dispose it. -The Binding Tools for Swift tooling handles these explicit lifetime semantics with generated Swift code. In the new Swift/.NET interop, management of these lifetime semantics will be done by the Swift projection layer and not by the raw calling-convention support. If any GC interaction is required to handle the lifetime semantics correctly, we should take an approach more similar to the `ComWrappers` support (higher-level, less complex interop interface) rather than the Objective-C interop support (lower-level, basically only usable by the ObjCRuntime implementation). +Swift type projections in C# should implement destructors and utilize a designated thin wrapper over the Swift memory allocator, currently accessible through the NativeMemory class, to explicitly release memory. In custom scenarios where more than just memory is being freed, they will implement `IDisposable` pattern. It's important to ensure that when a Swift callee function allocates an "unsafe" or "raw" pointer types, such as UnsafeMutablePointer and UnsafeRawPointer, where explicit control over memory is needed, and the pointer is returned to .NET, the memory is not dereferenced after the call returns. Also, if a C# managed object is allocated in a callee function and returned to Swift, the .NET GC will eventually collect it, but Swift will keep track using ARC, which represents an invalid case and should be handled by projection tools. + +In the new Swift/.NET interop, management of these lifetime semantics will be done by the Swift projection layer and not by the raw calling-convention support. If any GC interaction is required to handle the lifetime semantics correctly, we should take an approach more similar to the `ComWrappers` support (higher-level, less complex interop interface) rather than the Objective-C interop support (lower-level, basically only usable by the ObjCRuntime implementation). ### Projecting Swift into .NET @@ -164,7 +166,7 @@ Structs that are non-bitwise-movable are more difficult. They cannot be moved by We plan to interop with Swift's Library Evolution mode, which brings an additional wrinkle into the Swift struct story. Swift's Library Evolution mode abstracts away all type layout and semantic information unless a type is explicitly marked as `@frozen`. In the Library Evolution case, all structs have "opaque" layout, meaning that their exact layout and category cannot be determined until runtime. The size and layout information of concrete types is available in its [Value Witness Table](https://github.com/apple/swift/blob/main/docs/ABIStabilityManifesto.md#value-witness-table), so we can look up this information at runtime for allocating struct instances and manipulating struct memory correctly. As a result, we need to treat all "opaque" layout structs as possibly non-bitwise-movable at compile time as we will not know until runtime what the exact layout is. Swift/C++ interop is not required to use the Library Evolution mode in all cases as it can statically link against Swift libraries, so it is not limited by opaque struct layouts in every case. Every concrete type in Swift has a structure that provides information about how to manipulate values of that type. -Swift structs and enums have richer semantics than in .NET and are projected as C# classes which implement `IDisposable` interface to streamline handling of both simple blittable and more complex scenarios. These C# classes have a single property that holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by the marshaler in cases when it is a return value from a function. +Swift structs and enums have richer semantics than in .NET and are projected as C# classes to streamline handling of both simple blittable and more complex scenarios. These C# classes have a single property that holds the data payload for the type. They typically include two constructors: one that corresponds to the init method in the Swift class, and another internal constructor used to create uninitialized types invoked by the marshaler in cases when it is a return value from a function. ##### Classes/Protocols