Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SHA1 Re-write #163

Merged
merged 96 commits into from
Dec 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
7afc2b0
Reduce more CI runs
mborland Dec 13, 2024
aa01137
Add config
mborland Dec 13, 2024
4ae2d79
A clear memory function
mborland Dec 13, 2024
72d63fc
Add export macro
mborland Dec 13, 2024
dfdbec8
Upgrade quick test to sha family
mborland Dec 13, 2024
b4a12ff
Disable all old tests
mborland Dec 13, 2024
ba2c95a
Add clang-19 safe buffers flag
mborland Dec 13, 2024
4ee6966
Make module safe
mborland Dec 13, 2024
f4a5307
Restart module
mborland Dec 13, 2024
de2055c
Add state enum class
mborland Dec 16, 2024
3d59843
Make a generic version of clear mem
mborland Dec 16, 2024
ded7577
First draft of new sha1 & sha2 base class
mborland Dec 16, 2024
86868d7
Implement sha1
mborland Dec 17, 2024
8319917
Enforce const correctness
mborland Dec 17, 2024
9ee4148
Remove memset_s
mborland Dec 17, 2024
4f1309a
Modern SHA1 testing
mborland Dec 17, 2024
7d225dc
Use empty instead of 0 size
mborland Dec 17, 2024
00f04cb
Add override to virtual block
mborland Dec 17, 2024
e7c2253
Add update file reader class
mborland Dec 17, 2024
c20aead
Add one shot functions
mborland Dec 17, 2024
614b326
Add warning suppression
mborland Dec 17, 2024
39cdb28
Make file reader construction more generic
mborland Dec 17, 2024
a830310
Finish up testing changes
mborland Dec 17, 2024
091ecf7
Fix rebase items
mborland Dec 17, 2024
f57758f
Fix clang-16 incompatibility
mborland Dec 17, 2024
b2f815b
Remove use of Secure Clear
mborland Dec 17, 2024
38c7d3c
Replace Ubuntu 20.04 cmake runs
mborland Dec 17, 2024
54104e5
Fix macos clang missing header
mborland Dec 17, 2024
57dd593
Update sanitizers
mborland Dec 17, 2024
717b798
Update fuzzing
mborland Dec 17, 2024
df6e658
Add no discard to get_digest
mborland Dec 17, 2024
0ff4440
Remove unneeded fuzzer runs
mborland Dec 17, 2024
15cd393
Update SHA1 NVCC test
mborland Dec 17, 2024
8f954e9
Remove inclusion errors
mborland Dec 17, 2024
26fd1c6
Update NVCC CI
mborland Dec 17, 2024
e9c23bd
Add additional CUDA headers
mborland Dec 17, 2024
e9e0350
Install additional packages
mborland Dec 17, 2024
f2d7249
Add CUDA constexpr
mborland Dec 17, 2024
0fc1177
Try direct install from NVIDIA
mborland Dec 17, 2024
5f391c6
Debug output
mborland Dec 17, 2024
720e80d
Revert CMAKE change
mborland Dec 17, 2024
f3a5878
Change include directory
mborland Dec 17, 2024
8069b9e
Remove thrust usage
mborland Dec 17, 2024
552964b
Pass c++20 flag to NVCC
mborland Dec 18, 2024
746cf49
Add array clear mem for cuda
mborland Dec 18, 2024
661fe5f
Fix macro error
mborland Dec 18, 2024
e17d896
Fix inclusion of stl headers in CUDA mode
mborland Dec 18, 2024
84a18bf
Fix missing headers
mborland Dec 18, 2024
a039c31
Ignore buffer sizing warning
mborland Dec 18, 2024
1af32cb
Re-arrange headers to avoid internal errors
mborland Dec 18, 2024
6a01ab0
Missing semi-colon
mborland Dec 18, 2024
2a61dbc
Change flags for boost_test_jamfile
mborland Dec 18, 2024
bf9d0d1
More flags
mborland Dec 18, 2024
e61264e
Begin compat file
mborland Dec 19, 2024
b5d0ace
Add in further required utilites
mborland Dec 20, 2024
2e31985
Add make span helper functions
mborland Dec 20, 2024
f18d8a3
Use compatibility file
mborland Dec 20, 2024
9e0366f
Disable old runs
mborland Dec 20, 2024
a8a9089
Add MMD flag
mborland Dec 20, 2024
6f6e2fe
Fix build errors
mborland Dec 20, 2024
98c55ab
Move CUDA config to top-level
mborland Dec 23, 2024
fbb17da
Remove unneeded failing step
mborland Dec 23, 2024
807b6ed
Workaround old cmake
mborland Dec 23, 2024
83e0348
Only pass new version
mborland Dec 23, 2024
1db57b0
Use newer CMake
mborland Dec 23, 2024
c1b6bbf
Fix 0 byte under CUDA
mborland Dec 23, 2024
a1e7f76
Fix use of span for non-cuda version
mborland Dec 23, 2024
dd50421
Delete unneeded using decl
mborland Dec 23, 2024
8332dfe
Fix bad impl of compat utils
mborland Dec 23, 2024
931b69c
Apply compat utils to SHA1
mborland Dec 23, 2024
0771f2a
Fix definition of rotl to match STL
mborland Dec 23, 2024
29b2ffb
Add unsafe buffer usage for clang-20 runs
mborland Dec 23, 2024
dbfaf4d
Ignore clang-20 unsafe buffer warning
mborland Dec 23, 2024
e2d767d
Simplify CUDA cmake update
mborland Dec 23, 2024
1bd0f93
Add missing STD flag in coverage
mborland Dec 23, 2024
1b83695
More CODECOV fixes
mborland Dec 23, 2024
fcc66aa
More sign conversion fixes
mborland Dec 23, 2024
61ac620
Update from apt-kitware instructions
mborland Dec 23, 2024
52da1b3
Ignore unused errors with LCOV
mborland Dec 23, 2024
d89b390
Get Kitware key from ubuntu keyring
mborland Dec 23, 2024
77eddbc
Fix macos clang warning
mborland Dec 23, 2024
dc24641
Use NVIDIA container
mborland Dec 23, 2024
64a4b56
Fix sudo usage
mborland Dec 23, 2024
c37eff2
Exclude lines that imply fs failure
mborland Dec 23, 2024
61b92f1
Secure clear the intermediate hash on destruction
mborland Dec 23, 2024
bbb8307
Simplified test
mborland Dec 23, 2024
d4fcd9a
Fix CUDA secure memset types
mborland Dec 23, 2024
e245b02
Cover corruption lines
mborland Dec 23, 2024
86e9b8b
Fix constexpr coverage
mborland Dec 24, 2024
202ace4
Fix crypt v1 errors
mborland Dec 24, 2024
053e115
Enable SHA1 SHAVS testing
mborland Dec 24, 2024
06b138c
Update to crypt2
mborland Dec 24, 2024
285fc6e
Update SHAVS test harness for crypt2
mborland Dec 24, 2024
f6b6e63
Change CODECOV method
mborland Dec 24, 2024
b3f0f27
Exclude NIST CAVS test harness for now
mborland Dec 24, 2024
c8e11ef
Cover last lines of diff
mborland Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,14 @@ jobs:
compiler: gcc-13, cxxstd: '03,11,14,17,20', os: ubuntu-24.04 }
# Clang isn't compatible with libstdc++-13, so use the slightly older one
- { compiler: clang-15, cxxstd: '03,11,14,17,20', os: ubuntu-22.04, install: 'clang-15 g++-12', gcc_toolchain: 12 }
- { compiler: clang-16, cxxstd: '11,14,17,20,2b', os: ubuntu-24.04 }
- { compiler: clang-16, cxxstd: '11,14,17,20,2b', os: ubuntu-24.04, install: 'clang-16 g++-12', gcc_toolchain: 12 }
# https://github.com/llvm/llvm-project/issues/59827: disabled 2b/23 for clang-17 with libstdc++13 in 24.04
- { compiler: clang-17, cxxstd: '11,14,17,20', os: ubuntu-24.04 }
- { compiler: clang-18, cxxstd: '11,14,17,20,23,2c', os: ubuntu-24.04 }

- { name: Clang w/ sanitizers, sanitize: yes,
compiler: clang-18, cxxstd: '03,11,14,17,20', os: ubuntu-24.04 }

- { name: MacOS w/ clang and sanitizers,
compiler: clang, cxxstd: '03,11,14,17,20,2b', os: macos-13, sanitize: yes }
- { compiler: clang, cxxstd: '03,11,14,17,20,2b', os: macos-14 }
Expand Down Expand Up @@ -348,8 +351,8 @@ jobs:
fail-fast: false
matrix:
include:
- { os: ubuntu-20.04, build_shared: ON, build_type: Debug, generator: 'Unix Makefiles' }
- { os: ubuntu-20.04, build_shared: OFF, build_type: Debug, generator: 'Unix Makefiles' }
- { os: ubuntu-24.04, build_shared: ON, build_type: Debug, generator: 'Unix Makefiles' }
- { os: ubuntu-24.04, build_shared: OFF, build_type: Debug, generator: 'Unix Makefiles' }
- { os: windows-2019, build_shared: ON, build_type: Debug, generator: 'Visual Studio 16 2019' }
- { os: windows-2019, build_shared: OFF, build_type: Debug, generator: 'Visual Studio 16 2019' }

Expand Down
230 changes: 178 additions & 52 deletions .github/workflows/codecov.yml
Original file line number Diff line number Diff line change
@@ -1,76 +1,202 @@
# ------------------------------------------------------------------------------
# Copyright Matt Borland 2023 - 2024.
# Copyright Christopher Kormanyos 2023 - 2024.
# Distributed under the Boost Software License,
# Version 1.0. (See accompanying file LICENSE_1_0.txt
# or copy at http://www.boost.org/LICENSE_1_0.txt)
# ------------------------------------------------------------------------------

# Copyright 2020-2021 Peter Dimov
# Copyright 2021 Andrey Semashev
# Copyright 2021 Alexander Grund
# Copyright 2022 James E. King III
# Copyright 2023 Matt Borland
#
# Distributed under the Boost Software License, Version 1.0.
# (See accompanying file LICENSE_1_0.txt or copy at http://boost.org/LICENSE_1_0.txt)
---
name: codecov

on:
pull_request:
push:
branches:
- master
- develop
- bugfix/**
- feature/**
pull_request:
types: [opened, synchronize, reopened]
- fix/**
- pr/**

env:
GIT_FETCH_JOBS: 8
NET_RETRY_COUNT: 5
B2_CI_VERSION: 1
B2_VARIANT: debug,release
B2_LINK: shared,static
LCOV_BRANCH_COVERAGE: 0
CODECOV_NAME: Github Actions
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

jobs:
gcc-gcov-native:
runs-on: ubuntu-latest
posix:
defaults:
run:
shell: bash

strategy:
fail-fast: false
matrix:
standard: [ c++20 ]
compiler: [ g++ ]
include:
- { name: Collect coverage, coverage: yes,
compiler: gcc-13, cxxstd: '23', os: ubuntu-24.04, install: 'g++-13-multilib', address-model: '32,64' }

timeout-minutes: 120
runs-on: ${{matrix.os}}
container: ${{matrix.container}}
env: {B2_USE_CCACHE: 1}

steps:
- uses: actions/checkout@v4
- name: Setup environment
run: |
if [ -f "/etc/debian_version" ]; then
echo "DEBIAN_FRONTEND=noninteractive" >> $GITHUB_ENV
export DEBIAN_FRONTEND=noninteractive
fi
if [ -n "${{matrix.container}}" ] && [ -f "/etc/debian_version" ]; then
apt-get -o Acquire::Retries=$NET_RETRY_COUNT update
apt-get -o Acquire::Retries=$NET_RETRY_COUNT install -y sudo software-properties-common curl
# Need (newer) git, and the older Ubuntu container may require requesting the key manually using port 80
curl -sSL --retry ${NET_RETRY_COUNT:-5} 'http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xE1DD270288B4E6030699E45FA1715D88E1DF1F24' | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/git-core_ubuntu_ppa.gpg
for i in {1..${NET_RETRY_COUNT:-3}}; do sudo -E add-apt-repository -y ppa:git-core/ppa && break || sleep 10; done
apt-get -o Acquire::Retries=$NET_RETRY_COUNT update
osver=$(lsb_release -sr | cut -f1 -d.)
pkgs="g++ git"
# Ubuntu 22+ has only Python 3 in the repos
if [ -n "$osver" ] && [ "$osver" -ge "22" ]; then
pkgs+=" python-is-python3 libpython3-dev"
else
pkgs+=" python libpython-dev"
fi
apt-get -o Acquire::Retries=$NET_RETRY_COUNT install -y $pkgs
fi
# For jobs not compatible with ccache, use "ccache: no" in the matrix
if [[ "${{ matrix.ccache }}" == "no" ]]; then
echo "B2_USE_CCACHE=0" >> $GITHUB_ENV
fi
git config --global pack.threads 0

- uses: actions/checkout@v3
with:
# For coverage builds fetch the whole history, else only 1 commit using a 'fake ternary'
fetch-depth: ${{ matrix.coverage && '0' || '1' }}

- name: Cache ccache
uses: actions/cache@v3
if: env.B2_USE_CCACHE
with:
path: ~/.ccache
key: ${{matrix.os}}-${{matrix.container}}-${{matrix.compiler}}-${{github.sha}}
restore-keys: ${{matrix.os}}-${{matrix.container}}-${{matrix.compiler}}-

- name: Fetch Boost.CI
uses: actions/checkout@v3
with:
fetch-depth: '0'
- name: update-tools
repository: boostorg/boost-ci
ref: master
path: boost-ci-cloned

- name: Get CI scripts folder
run: |
# Copy ci folder if not testing Boost.CI
[[ "$GITHUB_REPOSITORY" =~ "boost-ci" ]] || cp -r boost-ci-cloned/ci .
rm -rf boost-ci-cloned

- name: Install packages
if: startsWith(matrix.os, 'ubuntu')
run: |
sudo apt install lcov locales
SOURCE_KEYS=(${{join(matrix.source_keys, ' ')}})
SOURCES=(${{join(matrix.sources, ' ')}})
# Add this by default
SOURCES+=(ppa:ubuntu-toolchain-r/test)
for key in "${SOURCE_KEYS[@]}"; do
for i in {1..$NET_RETRY_COUNT}; do
keyfilename=$(basename -s .key $key)
curl -sSL --retry ${NET_RETRY_COUNT:-5} "$key" | sudo gpg --dearmor > /etc/apt/trusted.gpg.d/${keyfilename} && break || sleep 10
done
done
for source in "${SOURCES[@]}"; do
for i in {1..$NET_RETRY_COUNT}; do
sudo add-apt-repository $source && break || sleep 10
done
done
sudo apt-get -o Acquire::Retries=$NET_RETRY_COUNT update
if [[ -z "${{matrix.install}}" ]]; then
pkgs="${{matrix.compiler}}"
pkgs="${pkgs/gcc-/g++-}"
else
pkgs="${{matrix.install}}"
fi
sudo apt-get -o Acquire::Retries=$NET_RETRY_COUNT install -y $pkgs locales
sudo locale-gen de_DE.UTF-8
sudo update-locale

- name: clone-submods-bootstrap-headers-boost-develop
- name: Setup GCC Toolchain
if: matrix.gcc_toolchain
run: |
git clone -b develop --depth 1 https://github.com/boostorg/boost.git ../boost-root
cd ../boost-root
git submodule update --init tools
git submodule update --init libs/assert
git submodule update --init libs/config
git submodule update --init libs/core
git submodule update --init libs/math
git submodule update --init libs/multiprecision
git submodule update --init libs/predef
git submodule update --init libs/static_assert
git submodule update --init libs/test
git submodule update --init libs/uuid
git submodule update --init libs/throw_exception
git submodule update --init libs/type_traits
./bootstrap.sh
./b2 headers
- name: gcc-gcov-native
GCC_TOOLCHAIN_ROOT="$HOME/gcc-toolchain"
echo "GCC_TOOLCHAIN_ROOT=$GCC_TOOLCHAIN_ROOT" >> $GITHUB_ENV
if ! command -v dpkg-architecture; then
apt-get install -y dpkg-dev
fi
MULTIARCH_TRIPLET="$(dpkg-architecture -qDEB_HOST_MULTIARCH)"
mkdir -p "$GCC_TOOLCHAIN_ROOT"
ln -s /usr/include "$GCC_TOOLCHAIN_ROOT/include"
ln -s /usr/bin "$GCC_TOOLCHAIN_ROOT/bin"
mkdir -p "$GCC_TOOLCHAIN_ROOT/lib/gcc/$MULTIARCH_TRIPLET"
ln -s "/usr/lib/gcc/$MULTIARCH_TRIPLET/${{matrix.gcc_toolchain}}" "$GCC_TOOLCHAIN_ROOT/lib/gcc/$MULTIARCH_TRIPLET/${{matrix.gcc_toolchain}}"

- name: Setup multiarch
if: matrix.multiarch
run: |
cd test/cover
echo "build and run gcov/lcov/genhtml"
echo "make prepare -f make_gcov_01_generic.gmk MY_ALL_COV=0 MY_BOOST_ROOT=../../../boost-root MY_CC=${{ matrix.compiler }} MY_STD=${{ matrix.standard }}"
echo
make prepare -f make_gcov_01_generic.gmk MY_ALL_COV=0 MY_BOOST_ROOT=../../../boost-root MY_CC=${{ matrix.compiler }} MY_STD=${{ matrix.standard }}
echo
echo "make gcov -f make_gcov_01_generic.gmk --jobs=8 MY_ALL_COV=0 MY_BOOST_ROOT=../../../boost-root MY_CC=${{ matrix.compiler }} MY_STD=${{ matrix.standard }}"
echo
make gcov -f make_gcov_01_generic.gmk --jobs=8 MY_ALL_COV=0 MY_BOOST_ROOT=../../../boost-root MY_CC=${{ matrix.compiler }} MY_STD=${{ matrix.standard }}
echo
- name: upload-codecov
sudo apt-get install --no-install-recommends -y binfmt-support qemu-user-static
sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
git clone https://github.com/jeking3/bdde.git
echo "$(pwd)/bdde/bin/linux" >> ${GITHUB_PATH}
echo "BDDE_DISTRO=${{ matrix.distro }}" >> ${GITHUB_ENV}
echo "BDDE_EDITION=${{ matrix.edition }}" >> ${GITHUB_ENV}
echo "BDDE_ARCH=${{ matrix.arch }}" >> ${GITHUB_ENV}
echo "B2_WRAPPER=bdde" >> ${GITHUB_ENV}

- name: Setup Boost
env:
B2_ADDRESS_MODEL: ${{matrix.address-model}}
B2_COMPILER: ${{matrix.compiler}}
B2_CXXSTD: ${{matrix.cxxstd}}
B2_SANITIZE: ${{matrix.sanitize}}
B2_STDLIB: ${{matrix.stdlib}}
B2_CXXFLAGS: ${{matrix.cxxflags}}
# More entries can be added in the same way, see the B2_ARGS assignment in ci/enforce.sh for the possible keys.
# B2_DEFINES: ${{matrix.defines}}
# Variables set here (to non-empty) will override the top-level environment variables, e.g.
# B2_VARIANT: ${{matrix.variant}}
# Set the (B2) target(s) to build, defaults to the test folder of the current library
# Can alternatively be done like this in the build step or in the build command of the build step, e.g. `run: B2_TARGETS=libs/$SELF/doc ci/build.sh`
# B2_TARGETS: libs/foo/test//bar
run: source ci/github/install.sh

- name: Setup coverage collection
if: matrix.coverage
run: ci/github/codecov.sh "setup"

- name: Run tests
if: '!matrix.coverity'
run: ci/build.sh

- name: Upload coverage
if: matrix.coverage
run: ci/codecov.sh "upload"
env:
BOOST_CI_CODECOV_IO_UPLOAD: skip

- name: Upload coverage
if: matrix.coverage
uses: codecov/codecov-action@v4
with:
plugin: gcov
file: ${{ runner.workspace }}/crypt/test/cover/coverage.info
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: true
verbose: false
disable_search: true
file: coverage.info
name: Github Actions
token: ${{secrets.CODECOV_TOKEN}}
verbose: true
42 changes: 27 additions & 15 deletions .github/workflows/cuda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,25 +23,18 @@ jobs:
fail-fast: false

runs-on: gpu-runner-1
container:
image: nvidia/cuda:12.6.3-devel-ubuntu24.04
options: --gpus all

steps:
- uses: Jimver/cuda-toolkit@v0.2.19
id: cuda-toolkit
with:
cuda: '12.5.0'
method: 'network'
sub-packages: '["nvcc"]'

- name: Output CUDA information
run: |
echo "Installed cuda version is: ${{steps.cuda-toolkit.outputs.cuda}}"+
echo "Cuda install location: ${{steps.cuda-toolkit.outputs.CUDA_PATH}}"
nvcc -V
- uses: actions/checkout@v4

- name: Install Packages
run: |
sudo apt-get install -y cmake make
apt-get update
apt-get install -y wget cmake git python3 build-essential

- name: Setup Boost
run: |
echo GITHUB_REPOSITORY: $GITHUB_REPOSITORY
Expand All @@ -62,15 +55,34 @@ jobs:
cp -r $GITHUB_WORKSPACE/* libs/$LIBRARY
git submodule update --init tools/boostdep
python3 tools/boostdep/depinst/depinst.py --git_args "--jobs 3" $LIBRARY

- name: Check CUDA Installation
run: |
nvidia-smi
nvcc --version
ls -l /usr/local/cuda-12.6/bin/nvcc

- name: Configure
run: |
cd ../boost-root
mkdir __build__ && cd __build__
cmake -DBOOST_INCLUDE_LIBRARIES=$LIBRARY -DBUILD_TESTING=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DBOOST_CRYPT_ENABLE_CUDA=1 -DCMAKE_CUDA_ARCHITECTURES=70 -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.5 ..
cmake -DBOOST_INCLUDE_LIBRARIES=$LIBRARY \
-DBUILD_TESTING=ON \
-DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.6/bin/nvcc \
-DBOOST_CRYPT_ENABLE_CUDA=1 \
-DCMAKE_CUDA_ARCHITECTURES=70 \
-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.6 \
-DCUDA_PATH=/usr/local/cuda-12.6 \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
.. \
--debug-output

- name: Build tests
run: |
cd ../boost-root/__build__
cmake --build . --target tests -j $(nproc)
cmake --build . --target tests -j $(nproc) --verbose

- name: Run tests
run: |
cd ../boost-root/__build__
Expand Down
Loading
Loading