mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-28 17:05:53 +08:00
When a Release native test executable instantiates CRT endpoint code, static AWS archives can embed private CRT/JSON state in both the executable and shared libraries. `Aws::InitAPI` initializes one copy while another copy can abort in `aws_mem_acquire` with an uninitialized allocator. Compiler inlining can hide this failure in other builds. Use one shared AWS SDK/CRT/C-library graph for Milvus, milvus-storage, Arrow and tests: - Enable shared libraries for the SDK, CRT C++, every `aws-c-*` package and `aws-checksums`. Root Conan validation always rejects static overrides. - Keep the direct CRT requirement needed by tests to expose link libraries, without `force=True`; document that its version must track the SDK recipe. Resolve its target through `find_package(AWSSDK)` without a redundant CRT find. - Add `MILVUS_AUDIT_AWS_RUNTIME` for Linux linker-map and full, unstripped symbol checks on milvus-storage, milvus_core and all_tests. Normal development builds default OFF; unit/coverage builds and `make install` / `make gpu-install` default ON. Successful FetchContent storage checks use a stamp, invalidated by library/checker changes, instead of rescanning on every no-op build. Conan validation stays mandatory with auditing OFF. - Run the checker CTest from both full C++ unit and coverage runners, with missing/failed tests treated as failures. Focused gtest invocations retain their existing behavior. The separate real CRT RuleEngine regression remains in all_tests. - Balance the checksum test's SDK initialization after its client configuration is destroyed, so CRT workers finish before SDK logger teardown. - Document rebuilding Arrow/native consumers and shipping the complete AWS shared-library/SONAME closure. Existing imports and installation handle that closure; no milvus-common or milvus-storage source-pin update is required. Validation for the review changes (Linux x86-64, GCC 14, C++20): - Conan graph resolves with the unforced direct CRT dependency; static SDK, CRT, C-common and checksums overrides are rejected in both master and 3.0 recipes. - A consumer using only `find_package(AWSSDK)` configures, links direct CRT API calls and runs with the regenerated root graph. Removing the direct CRT requirement in a negative-control experiment exposes headers/targets but fails linking with `DSO missing from command line`. - Build-wiring fixtures passed with Ninja, Unix Makefiles and Ninja Multi-Config: defaults/overrides, all three audit targets, no-op builds, script/library invalidation, hidden-runtime and unused-static-archive rejection, and separate Debug/Release stamps. - Both actual runner scripts passed dispatch checks with the real checker CTest. Failed tests, absent registrations and missing build directories fail the runner, including when the caller directory contains an unrelated passing test. Focused gtest dispatch is preserved. Makefile package-default inheritance and core_build.sh ON/OFF argument propagation passed. - Shell syntax, Conan recipe syntax and `git diff --check` passed. A fresh complete master native rebuild/full test suite was not repeated for these build-only edits; new-head CI is pending. Earlier full master verification used Linux x86-64, GCC 14, C++20, Release, coverage disabled and `WITH_CRT=OFF`. All default native targets rebuilt and the three native audits, 10 endpoint/CRT tests, 20 related storage tests, installed-library CGO and live-MinIO checks passed. The direct CRT regression aborts with the old static-runtime executable. The installed output contains 26 AWS DSOs with valid SONAME links; the complete installed test executable passed the same 10 endpoint/CRT tests in a container with no Conan cache. The live-MinIO CRUD check exercises both the storage Arrow data plane and legacy Minio control plane, including missing-object classification. The reporter's Ubuntu 20/GCC 11.4 configuration, macOS, `WITH_CRT=ON` and full Go server startup were not validated locally. CI on the preceding head `3316a06` passed build/unit coverage, default E2E and Go SDK checks; those results do not replace CI on this update. Prebuilt shared Conan binaries have not been published by this work; fresh builders may still need `--build=missing`. Remote binary publication is a separate follow-up. Fixes #54101 --------- Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
109 lines
3.4 KiB
Bash
Executable File
109 lines
3.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
# Licensed to the LF AI & Data foundation under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
set -e
|
|
|
|
SOURCE="${BASH_SOURCE[0]}"
|
|
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
|
|
DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
|
|
SOURCE="$(readlink "$SOURCE")"
|
|
[[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
|
|
done
|
|
SCRIPTS_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
|
|
|
|
MILVUS_CORE_DIR="${SCRIPTS_DIR}/../internal/core"
|
|
CORE_INSTALL_PREFIX="${MILVUS_CORE_DIR}/output"
|
|
UNITTEST_DIRS=("${CORE_INSTALL_PREFIX}/unittest")
|
|
|
|
# currently core will install target lib to "internal/core/output/lib"
|
|
if [ -d "${CORE_INSTALL_PREFIX}/lib" ]; then
|
|
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${CORE_INSTALL_PREFIX}/lib
|
|
fi
|
|
|
|
# Suppress known LeakSanitizer false positives from uninstrumented shared libraries
|
|
LSAN_SUPPRESSIONS="${MILVUS_CORE_DIR}/lsan_suppressions.txt"
|
|
if [ -f "${LSAN_SUPPRESSIONS}" ]; then
|
|
export LSAN_OPTIONS="suppressions=${LSAN_SUPPRESSIONS}"
|
|
fi
|
|
|
|
# run unittest
|
|
arg="$1"
|
|
filter_value="${arg#*=}"
|
|
|
|
for UNITTEST_DIR in "${UNITTEST_DIRS[@]}"; do
|
|
if [ ! -d "${UNITTEST_DIR}" ]; then
|
|
echo "The unittest folder does not exist!"
|
|
exit 1
|
|
fi
|
|
|
|
|
|
if [[ $filter_value ]]; then
|
|
if [ $filter_value == "--gtest_list_tests" ]; then
|
|
${UNITTEST_DIR}/all_tests $filter_value
|
|
exit 0
|
|
else
|
|
${UNITTEST_DIR}/all_tests --gtest_filter=$filter_value
|
|
exit 0
|
|
fi
|
|
fi
|
|
|
|
if [[ "$(uname -s)" == "Linux" ]]; then
|
|
echo "Running AWS runtime link checker self-test"
|
|
(
|
|
cd "${MILVUS_CMAKE_BUILD_DIR:-${SCRIPTS_DIR}/../cmake_build}/unittest" &&
|
|
ctest -R '^aws_runtime_link_check$' --output-on-failure --no-tests=error
|
|
) || exit 1
|
|
fi
|
|
|
|
echo "Running all unittest ..."
|
|
${UNITTEST_DIR}/all_tests
|
|
if [ $? -ne 0 ]; then
|
|
echo ${UNITTEST_DIR}/all_tests "run failed"
|
|
exit 1
|
|
fi
|
|
echo "Running index unittest"
|
|
INDEX_TEST_BINARY="${UNITTEST_DIR}/index_tests" "${SCRIPTS_DIR}/run_index_unittest.sh"
|
|
|
|
if [ -f "${UNITTEST_DIR}/dynamic_simd_test" ]; then
|
|
echo "Running dynamic simd test"
|
|
${UNITTEST_DIR}/dynamic_simd_test
|
|
if [ $? -ne 0 ]; then
|
|
echo ${UNITTEST_DIR}/dynamic_simd_test "run failed"
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
echo "Running json_stats unittest"
|
|
${UNITTEST_DIR}/json_stats_test
|
|
if [ $? -ne 0 ]; then
|
|
echo ${UNITTEST_DIR}/json_stats_test "run failed"
|
|
exit 1
|
|
fi
|
|
|
|
done
|
|
|
|
# run cwrapper unittest
|
|
if [ -f ${CWRAPPER_UNITTEST} ];then
|
|
echo "Running cwrapper unittest: ${CWRAPPER_UNITTEST}"
|
|
${CWRAPPER_UNITTEST}
|
|
if [ $? -ne 0 ]; then
|
|
echo ${CWRAPPER_UNITTEST} " run failed"
|
|
exit 1
|
|
fi
|
|
fi
|