Use of `BuildToolPlugin`

[zamderax]

I noticed in the README.md that it says:

SwiftPM (command line) cannot build the Metal shaders so the ultimate build has to be done via Xcode.

and

Although SwiftPM (command line) cannot build the Metal shaders, xcodebuild can and it can be used to do command line builds:

# build and run tests
xcodebuild test -scheme mlx-swift-Package -destination 'platform=OS X'
# build Tutorial
xcodebuild build -scheme Tutorial -destination 'platform=OS X'

Can we use BuildToolPlugin ?

[davidkoski]

Maybe. The limitation with plugins is they can't produce executable code (like a .o) file. They can be used to preprocess and produce swift files that are built and they can produce resources. The metallib is (I think) just a resource.

[zamderax]

Option A Swift PM Build Tool

If I read this correct:
swiftlang/swift-package-manager@0147f71
Then it seems that they can produce executable code and then we can use .binaryTarget to link to the artifact?

Option B

In Swift 6.2 we can link static binary targets for Linux and Windows. Does it make sense just to create a static library from the mlx repo for all the triples and then use the artifact bundles to link to them?

A separate note

1. Out of curiosity, I noticed that the python package, builds the C++ mlx library locally, then uses nanobind to generate a mlx.core Python module. It also links this module against the core mlx C++ library. Is this the right approach for us?
2. In addition, I've noticed that we are using mlx-c, is there a way that we don't need mlx-c and just straight use the C++ headers? Thoughts?

[davidkoski]

I am not sure that Option A helps, see:

Remaining limitations:

• generated files aren't treated the same way as non-generated ones, e.g. we will treat anything unknown as a resource to be processed and we won't respect declarations from the manifest

That is what I found -- you can make binary resources but not something that is linked in. This should work for metal shaders, but it would not work e.g. for running CMake on the C++ code.

Option B: perhaps, I am not sure what the tradeoffs are there.

Out of curiosity, I noticed that the python package, builds the C++ mlx library locally, then uses nanobind to generate a mlx.core Python module. It also links this module against the core mlx C++ library. Is this the right approach for us?

This isn't so far off of what we have -- in fact most of the integration with mlx-c code is 1:1 with the nanobind code. And this does build the C++ library, it is just in a separate repository. SwiftPM doesn't allow pulling down additional repositories as part of the build, so we have the submodule hookup. Since we can't run CMake as part of the build (well, not usefully anyway, see above) we have the header build as well.

In addition, I've noticed that we are using mlx-c, is there a way that we don't need mlx-c and just straight use the C++ headers? Thoughts?

At the time when we started there were C++ binding s for swift but they had limitations like being unable to call virtual methods. Some of those are improved with newer versions of Swift, but we haven't re-checked to see if it could be done. For the exceptions alone we probably need something like mlx-c and mlx-c is pretty lightweight. It also opens a path for other languages that can integration with C but not C++.

So these are all reasonable questions but there were (and I think still are) issues that prevent us from doing it this way. But I do think that we could probably use a plugin to build the metal shaders.

[zamderax]

Got it! Thank you for replying. So this is your thought?

• So if BuildToolPlugin can only help with Metal Shaders, we still have a problem where running swift build cannot call cmake in the mlx git submodule right?

• So cmake remains the reason we can't have mlx-swift run with SPM on Linux?

[davidkoski]

So if BuildToolPlugin can only help with Metal Shaders, we still have a problem where running swift build cannot call cmake in the mlx git submodule right?

Yes, but I don't think this is really a problem -- we already have it building. I think if we used a plugin we might be able to relax the requirement that it be built with xcodebuild and it would work with swift build. I don't think this is a problem for typical macOS/iOS developers.

So cmake remains the reason we can't have mlx-swift run with SPM on Linux?

I think the cuda build should deal with that. Linux can only run swift build , but as long as it doesn't need to build metal shaders (it doesn't as there is no metal on linux) it should be fine. Now there may be other issues that we will encounter, but we will have to see what they are.

[zamderax]

Yes, but I don't think this is really a problem -- we already have it building. I think if we used a plugin we might be able to relax the requirement that it be built with xcodebuild and it would work with swift build. I don't think this is a problem for typical macOS/iOS developers.

Right but it does help that downstream users of the package just need swift build regardless of what machine they're using

I think the cuda build should deal with that. Linux can only run swift build , but as long as it doesn't need to build metal shaders (it doesn't as there is no metal on linux) it should be fine. Now there may be other issues that we will encounter, but we will have to see what they are.

Could you clarify what you mean by "the cuda build"? I think the mlx with the CUDA backend there are .cu files that need get compiled no? And to compile .cu you need nvcc filename.cu -o output. I think this similar to the shader compilation step.

[davidkoski]

Could you clarify what you mean by "the cuda build"? I think the mlx with the CUDA backend there are .cu files that need get compiled no? And to compile .cu you need nvcc filename.cu -o output. I think this similar to the shader compilation step.

OK, so this may require a plugin to compile those -- same sort of technique that Metal would require. If this produces a .o then we may be in trouble. If it produces a resource (non executable code) then we just need to make sure it lands in a place where it can be loaded.

[zamderax]

1. It looks like the BuildToolPlugin can compile these .cu on the fly.
2. Without the ability for swift build to invoke the cmake inside of the mlx git submodule, it looks like the only path is to leverage Swift 6.2's new binaryTargets for non-apple frameworks which allows cross platform artifact bundles.

So it seems that maybe the plan is

1. fork mlx-swift and rename it mlx-swift-static
2. leverage the mlx-c headers for interop between Swift and MLX-C
3. Then create a github action to generate the artifact bundles whenever there is a new release

Thoughts?

[davidkoski]

I am not sure what it takes to build linux command line tools using Cuda. I still hope that we can accommodate this in this repo without forking (though perhaps that is a path to develop it).

I do think the BuildToolPlugin would be a good addition for both .cu and .mtl files, though there are some drawbacks related to permissions (but I think they are roughly the same issues that macros have, so this isn't a blocker).

I think your point 3 is interesting -- providing binary artifacts. To date we have not done so because SwiftPM allows for easy referencing and building of dependencies. What problem does 3 solve? If it is just another distribution mechanism then I don't know if it is a good idea to support that (more maintenance and more places to break). If there is something that it enables then it may be worth it.

[Joannis]

@davidkoski Unfortunately, producing a .cu or .mtl file is not enough for SwiftPM to actually compile this on Linux. SwiftPM lacks the awareness on how to use those artifacts - or even what kind of artifacts they are. I think .mtl support on macOS is an exception here, and only because SwiftPM is loaded through Xcode. If you try to build the package through an (Open Source) Swift toolchain from the command line, you'll see what I mean.

Regarding point 3; There have been more use cases where binary artifacts would've been useful, but until Swift 6.2 there was no way to use a static library with SwiftPM on Linux or Windows. The only way to build a CMake dependency with SwiftPM is to update it to support a SwiftPM structure.

There's one more thing I was curious about; Right now MLX-Swift uses the MLX-C library through C interop. Is there a reason it's not using C++ interop with MLX itself directly?

[davidkoski]

Unfortunately, producing a .cu or .mtl file is not enough for SwiftPM to actually compile this on Linux. SwiftPM lacks the awareness on how to use those artifacts - or even what kind of artifacts they are. I think .mtl support on macOS is an exception here, and only because SwiftPM is loaded through Xcode. If you try to build the package through an (Open Source) Swift toolchain from the command line, you'll see what I mean.

Yeah, that is exactly why the docs say it requires xcodebuild. However, a BuildTool plugin ought to be able to do the same. Maybe :-)

Regarding point 3; There have been more use cases where binary artifacts would've been useful, but until Swift 6.2 there was no way to use a static library with SwiftPM on Linux or Windows. The only way to build a CMake dependency with SwiftPM is to update it to support a SwiftPM structure.

Well, we are actually doing that here, but it does require a bit of work. We use submodules to map in the source and have SwiftPM do an equivalent build as CMake. It does require some support -- there are shell scripts that prepare headers, etc and we have to adjust things when the CMake side changes. Doable, but not trivial.

There's one more thing I was curious about; Right now MLX-Swift uses the MLX-C library through C interop. Is there a reason it's not using C++ interop with MLX itself directly?

Yeah, at the time this was built the C++ interop wasn't capable enough. For example it couldn't call virtual methods (among many other things). The C layer was built for this reason and it also supports other languages that have C bindings.