Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-12626: [C++] Support toolchain xsimd, update toolchain version to version 8.1.0 #13244

Merged
merged 43 commits into from
Jun 14, 2022

Conversation

wesm
Copy link
Member

@wesm wesm commented May 26, 2022

This also updates pinned vcpkg to use xsimd 8.1.0.

This also implements auto python-wheel-windows-vs2017 image update mechanism.
We have a problem of "docker build" on Windows. "docker build" doesn't reuse pulled
image as caches. "docker build" always rebuilds an image. This implements manual
reuse mechanism like the following:

if ! docker pull; then
   docker build # build only when built images don't exist
fi
docker run

But this doesn't work when ci/docker/python-wheel-windows-vs2017.dockerfile is updated
but pinned vcpkg revision isn't changed. In the case, "docker build" isn't run because
"docker pull" is succeeded.

To work this mechanism, this introduces "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION".
We must bump it manually when we update ci/docker/python-wheel-windows-vs2017.dockerfile.
"PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" is used in tag name. So "docker pull"
is failed with new "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" and then "docker build"
is used.

@github-actions
Copy link

@cyb70289
Copy link
Contributor

@kou, is there convenient way to verify if this pr may break packaging and release in github UI?

@kou
Copy link
Member

kou commented May 30, 2022

We can use @github-actions crossbow submit -g nightly-packaging -g nightly-release for it.

@cyb70289
Copy link
Contributor

@github-actions crossbow submit -g nightly-packaging -g nightly-release

@github-actions

This comment was marked as outdated.

@cyb70289 cyb70289 force-pushed the ARROW-12626-toolchain-xsimd branch 2 times, most recently from 9af5ea8 to 4a006ad Compare May 30, 2022 07:38
@cyb70289
Copy link
Contributor

@github-actions crossbow submit -g nightly-packaging -g nightly-release

@github-actions
Copy link

Revision: 4a006ad

Submitted crossbow builds: ursacomputing/crossbow @ actions-8103008ca7

Task Status
almalinux-8-amd64 Github Actions
almalinux-8-arm64 TravisCI
amazon-linux-2-amd64 Github Actions
centos-7-amd64 Github Actions
centos-8-stream-amd64 Github Actions
centos-8-stream-arm64 TravisCI
conan-maximum Github Actions
conan-minimum Github Actions
conda-clean Azure
conda-linux-gcc-py310-arm64 Azure
conda-linux-gcc-py310-cpu Azure
conda-linux-gcc-py310-cuda Azure
conda-linux-gcc-py310-ppc64le Azure
conda-linux-gcc-py37-arm64 Azure
conda-linux-gcc-py37-cpu-r40 Azure
conda-linux-gcc-py37-cpu-r41 Azure
conda-linux-gcc-py37-cuda Azure
conda-linux-gcc-py37-ppc64le Azure
conda-linux-gcc-py38-arm64 Azure
conda-linux-gcc-py38-cpu Azure
conda-linux-gcc-py38-cuda Azure
conda-linux-gcc-py38-ppc64le Azure
conda-linux-gcc-py39-arm64 Azure
conda-linux-gcc-py39-cpu Azure
conda-linux-gcc-py39-cuda Azure
conda-linux-gcc-py39-ppc64le Azure
conda-osx-arm64-clang-py310 Azure
conda-osx-arm64-clang-py38 Azure
conda-osx-arm64-clang-py39 Azure
conda-osx-clang-py310 Azure
conda-osx-clang-py37-r40 Azure
conda-osx-clang-py37-r41 Azure
conda-osx-clang-py38 Azure
conda-osx-clang-py39 Azure
conda-win-vs2017-py310 Azure
conda-win-vs2017-py37-r40 Azure
conda-win-vs2017-py37-r41 Azure
conda-win-vs2017-py38 Azure
conda-win-vs2017-py39 Azure
debian-bookworm-amd64 Github Actions
debian-bookworm-arm64 TravisCI
debian-bullseye-amd64 Github Actions
debian-bullseye-arm64 TravisCI
debian-buster-amd64 Github Actions
debian-buster-arm64 TravisCI
homebrew-cpp Github Actions
homebrew-r-autobrew Github Actions
homebrew-r-brew Github Actions
java-jars Github Actions
nuget Github Actions
python-sdist Github Actions
r-nightly-packages Github Actions
ubuntu-bionic-amd64 Github Actions
ubuntu-bionic-arm64 TravisCI
ubuntu-focal-amd64 Github Actions
ubuntu-focal-arm64 TravisCI
ubuntu-impish-amd64 Github Actions
ubuntu-impish-arm64 TravisCI
ubuntu-jammy-amd64 Github Actions
ubuntu-jammy-arm64 TravisCI
verify-rc-source-cpp-linux-almalinux-8-amd64 Github Actions
verify-rc-source-cpp-linux-conda-latest-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-cpp-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-cpp-macos-amd64 Github Actions
verify-rc-source-cpp-macos-arm64 Github Actions
verify-rc-source-cpp-macos-conda-amd64 Github Actions
verify-rc-source-csharp-linux-almalinux-8-amd64 Github Actions
verify-rc-source-csharp-linux-conda-latest-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-csharp-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-csharp-macos-amd64 Github Actions
verify-rc-source-csharp-macos-arm64 Github Actions
verify-rc-source-go-linux-almalinux-8-amd64 Github Actions
verify-rc-source-go-linux-conda-latest-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-go-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-go-macos-amd64 Github Actions
verify-rc-source-go-macos-arm64 Github Actions
verify-rc-source-integration-linux-almalinux-8-amd64 Github Actions
verify-rc-source-integration-linux-conda-latest-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-integration-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-integration-macos-amd64 Github Actions
verify-rc-source-integration-macos-arm64 Github Actions
verify-rc-source-integration-macos-conda-amd64 Github Actions
verify-rc-source-java-linux-almalinux-8-amd64 Github Actions
verify-rc-source-java-linux-conda-latest-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-java-macos-amd64 Github Actions
verify-rc-source-js-linux-almalinux-8-amd64 Github Actions
verify-rc-source-js-linux-conda-latest-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-js-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-js-macos-amd64 Github Actions
verify-rc-source-js-macos-arm64 Github Actions
verify-rc-source-python-linux-almalinux-8-amd64 Github Actions
verify-rc-source-python-linux-conda-latest-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-python-macos-amd64 Github Actions
verify-rc-source-python-macos-arm64 Github Actions
verify-rc-source-python-macos-conda-amd64 Github Actions
verify-rc-source-ruby-linux-almalinux-8-amd64 Github Actions
verify-rc-source-ruby-linux-conda-latest-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-18.04-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-20.04-amd64 Github Actions
verify-rc-source-ruby-linux-ubuntu-22.04-amd64 Github Actions
verify-rc-source-ruby-macos-amd64 Github Actions
verify-rc-source-ruby-macos-arm64 Github Actions
verify-rc-source-windows Github Actions
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp310-universal2 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-big-sur-cp39-universal2 Github Actions
wheel-macos-high-sierra-cp310-amd64 Github Actions
wheel-macos-high-sierra-cp37-amd64 Github Actions
wheel-macos-high-sierra-cp38-amd64 Github Actions
wheel-macos-high-sierra-cp39-amd64 Github Actions
wheel-macos-mavericks-cp310-amd64 Github Actions
wheel-macos-mavericks-cp37-amd64 Github Actions
wheel-macos-mavericks-cp38-amd64 Github Actions
wheel-macos-mavericks-cp39-amd64 Github Actions
wheel-manylinux2010-cp310-amd64 Github Actions
wheel-manylinux2010-cp37-amd64 Github Actions
wheel-manylinux2010-cp38-amd64 Github Actions
wheel-manylinux2010-cp39-amd64 Github Actions
wheel-manylinux2014-cp310-amd64 Github Actions
wheel-manylinux2014-cp310-arm64 TravisCI
wheel-manylinux2014-cp37-amd64 Github Actions
wheel-manylinux2014-cp37-arm64 TravisCI
wheel-manylinux2014-cp38-amd64 Github Actions
wheel-manylinux2014-cp38-arm64 TravisCI
wheel-manylinux2014-cp39-amd64 Github Actions
wheel-manylinux2014-cp39-arm64 TravisCI
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp37-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@cyb70289
Copy link
Contributor

There are many xsimd related errors, @kou, will you comment perferred fixes? Thanks.

@wesm
Copy link
Member Author

wesm commented May 31, 2022

As I recall, the project was pinned to a bundled version that was a few weeks before 8.1 was released. Feel free to take control of this PR from me

@kou
Copy link
Member

kou commented May 31, 2022

I'll push fixes later. Please wait for a while.

@kou kou force-pushed the ARROW-12626-toolchain-xsimd branch from 4a006ad to 902e81c Compare June 1, 2022 20:45
@kou
Copy link
Member

kou commented Jun 1, 2022

@github-actions crossbow submit conan-* conda-linux-gcc-*

@github-actions
Copy link

github-actions bot commented Jun 1, 2022

Revision: 902e81c

Submitted crossbow builds: ursacomputing/crossbow @ actions-76b5b8c64b

Task Status
conan-maximum Github Actions
conan-minimum Github Actions
conda-linux-gcc-py310-arm64 Azure
conda-linux-gcc-py310-cpu Azure
conda-linux-gcc-py310-cuda Azure
conda-linux-gcc-py310-ppc64le Azure
conda-linux-gcc-py37-arm64 Azure
conda-linux-gcc-py37-cpu-r40 Azure
conda-linux-gcc-py37-cpu-r41 Azure
conda-linux-gcc-py37-cuda Azure
conda-linux-gcc-py37-ppc64le Azure
conda-linux-gcc-py38-arm64 Azure
conda-linux-gcc-py38-cpu Azure
conda-linux-gcc-py38-cuda Azure
conda-linux-gcc-py38-ppc64le Azure
conda-linux-gcc-py39-arm64 Azure
conda-linux-gcc-py39-cpu Azure
conda-linux-gcc-py39-cuda Azure
conda-linux-gcc-py39-ppc64le Azure

@kou
Copy link
Member

kou commented Jun 1, 2022

@github-actions crossbow submit conda-osx-* conda-win-*

@github-actions
Copy link

github-actions bot commented Jun 1, 2022

Revision: addd3ac

Submitted crossbow builds: ursacomputing/crossbow @ actions-603614ae49

Task Status
conda-osx-arm64-clang-py310 Azure
conda-osx-arm64-clang-py38 Azure
conda-osx-arm64-clang-py39 Azure
conda-osx-clang-py310 Azure
conda-osx-clang-py37-r40 Azure
conda-osx-clang-py37-r41 Azure
conda-osx-clang-py38 Azure
conda-osx-clang-py39 Azure
conda-win-vs2017-py310 Azure
conda-win-vs2017-py37-r40 Azure
conda-win-vs2017-py37-r41 Azure
conda-win-vs2017-py38 Azure
conda-win-vs2017-py39 Azure

@kou
Copy link
Member

kou commented Jun 1, 2022

I've opened a pull request to update xsimd in vcpkg: microsoft/vcpkg#25030

@kou
Copy link
Member

kou commented Jun 1, 2022

@github-actions crossbow submit conan-*

@github-actions
Copy link

github-actions bot commented Jun 1, 2022

Revision: 309e42a

Submitted crossbow builds: ursacomputing/crossbow @ actions-983084a5d3

Task Status
conan-maximum Github Actions
conan-minimum Github Actions

@kou
Copy link
Member

kou commented Jun 1, 2022

conda-win-* failures aren't related to this. They are failed at least since 2022-05-30: https://lists.apache.org/thread/s6ggw17162cgoyk8wm1yg5g5mqv60j5q
@xhochy Could you confirm the conda-win-* failures?

@kou
Copy link
Member

kou commented Jun 3, 2022

@github-actions crossbow submit wheel-*

@github-actions
Copy link

github-actions bot commented Jun 3, 2022

Revision: d100b39

Submitted crossbow builds: ursacomputing/crossbow @ actions-9de9fb80c7

Task Status
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp310-universal2 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-big-sur-cp39-universal2 Github Actions
wheel-macos-high-sierra-cp310-amd64 Github Actions
wheel-macos-high-sierra-cp37-amd64 Github Actions
wheel-macos-high-sierra-cp38-amd64 Github Actions
wheel-macos-high-sierra-cp39-amd64 Github Actions
wheel-macos-mavericks-cp310-amd64 Github Actions
wheel-macos-mavericks-cp37-amd64 Github Actions
wheel-macos-mavericks-cp38-amd64 Github Actions
wheel-macos-mavericks-cp39-amd64 Github Actions
wheel-manylinux2010-cp310-amd64 Github Actions
wheel-manylinux2010-cp37-amd64 Github Actions
wheel-manylinux2010-cp38-amd64 Github Actions
wheel-manylinux2010-cp39-amd64 Github Actions
wheel-manylinux2014-cp310-amd64 Github Actions
wheel-manylinux2014-cp310-arm64 TravisCI
wheel-manylinux2014-cp37-amd64 Github Actions
wheel-manylinux2014-cp37-arm64 TravisCI
wheel-manylinux2014-cp38-amd64 Github Actions
wheel-manylinux2014-cp38-arm64 TravisCI
wheel-manylinux2014-cp39-amd64 Github Actions
wheel-manylinux2014-cp39-arm64 TravisCI
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp37-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@kou kou force-pushed the ARROW-12626-toolchain-xsimd branch from da2798f to 160ec95 Compare June 3, 2022 02:01
@kou
Copy link
Member

kou commented Jun 3, 2022

@github-actions crossbow submit wheel-*

@github-actions
Copy link

github-actions bot commented Jun 3, 2022

Revision: 160ec95

Submitted crossbow builds: ursacomputing/crossbow @ actions-082fa4d9b1

Task Status
wheel-macos-big-sur-cp310-arm64 Github Actions
wheel-macos-big-sur-cp310-universal2 Github Actions
wheel-macos-big-sur-cp38-arm64 Github Actions
wheel-macos-big-sur-cp39-arm64 Github Actions
wheel-macos-big-sur-cp39-universal2 Github Actions
wheel-macos-high-sierra-cp310-amd64 Github Actions
wheel-macos-high-sierra-cp37-amd64 Github Actions
wheel-macos-high-sierra-cp38-amd64 Github Actions
wheel-macos-high-sierra-cp39-amd64 Github Actions
wheel-macos-mavericks-cp310-amd64 Github Actions
wheel-macos-mavericks-cp37-amd64 Github Actions
wheel-macos-mavericks-cp38-amd64 Github Actions
wheel-macos-mavericks-cp39-amd64 Github Actions
wheel-manylinux2010-cp310-amd64 Github Actions
wheel-manylinux2010-cp37-amd64 Github Actions
wheel-manylinux2010-cp38-amd64 Github Actions
wheel-manylinux2010-cp39-amd64 Github Actions
wheel-manylinux2014-cp310-amd64 Github Actions
wheel-manylinux2014-cp310-arm64 TravisCI
wheel-manylinux2014-cp37-amd64 Github Actions
wheel-manylinux2014-cp37-arm64 TravisCI
wheel-manylinux2014-cp38-amd64 Github Actions
wheel-manylinux2014-cp38-arm64 TravisCI
wheel-manylinux2014-cp39-amd64 Github Actions
wheel-manylinux2014-cp39-arm64 TravisCI
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp37-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 3, 2022

@github-actions crossbow java-jars

@kou
Copy link
Member

kou commented Jun 12, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: 21b3427

Submitted crossbow builds: ursacomputing/crossbow @ actions-729a5ecccd

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 12, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: 083fdba

Submitted crossbow builds: ursacomputing/crossbow @ actions-1fd3333176

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 12, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: 2c2cf17

Submitted crossbow builds: ursacomputing/crossbow @ actions-f57e2a50a4

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 12, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: 080fc8f

Submitted crossbow builds: ursacomputing/crossbow @ actions-c871540910

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 12, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: be9daf2

Submitted crossbow builds: ursacomputing/crossbow @ actions-d8403e63aa

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 12, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: e5bf8ca

Submitted crossbow builds: ursacomputing/crossbow @ actions-c62ae2d49f

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 13, 2022

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions
Copy link

Revision: 4edd5b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-3056f2001d

Task Status
wheel-windows-cp38-amd64 Github Actions

@kou
Copy link
Member

kou commented Jun 13, 2022

@github-actions crossbow submit wheel-windows-*

@github-actions
Copy link

Revision: 4edd5b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-aa101b6926

Task Status
wheel-windows-cp310-amd64 Github Actions
wheel-windows-cp37-amd64 Github Actions
wheel-windows-cp38-amd64 Github Actions
wheel-windows-cp39-amd64 Github Actions

@assignUser
Copy link
Member

2h 51min vs 1h 🥳 🚀

@kou
Copy link
Member

kou commented Jun 13, 2022

docker build didn't use pulled images as cache as @kszucs said.
I added the following workflow as a workaround (see the description of this pull request for details, I've updated):

if ! docker pull; then
  docker build
fi
docker run

This is ready to merge now. I'll merge this tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants