mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-28 17:05:53 +08:00
eb7e63ebcf
## What this PR does Adds dedicated Spark-Milvus Backfill E2E coverage and the runtime tooling needed to execute it against real Storage V3 and Storage V2 segments. - Adds opt-in, serial-only pytest coverage for Backfill modes, null semantics, snapshot boundaries, schema fencing, manifest versions, compaction, and invalid inputs. - Adds Kubernetes Job and reusable Spark Toolbox runners with credential redaction, RBAC validation, evidence collection, and cleanup. - Builds `zilliztech/spark-milvus` in the toolbox pod and runs Spark locally through the validated wrapper and native-library runtime. - Separates Storage V3 and Storage V2 execution through pytest markers and independent Milvus deployments. - Uses the dedicated `SparkBackfill` tag, so the normal `L0 L1 L2 ClusterOnly` nightly selection does not include this suite. - Documents manual toolbox deployment and dedicated-suite execution. Fixes #52130 ## Pipeline isolation The suite is ignored during ordinary Python client collection unless `--run-spark-backfill` is explicitly provided. The dedicated pipeline additionally selects `--tags SparkBackfill`; existing PR, nightly, and weekly tag sets therefore select zero Spark Backfill cases. V3 and V2 remain separate marker selections because they require different Milvus storage configurations and must run serially without pytest-xdist. ## Test plan - [x] Dedicated Spark Backfill pipeline completed successfully. - [x] Storage V3: 30 cases collected; 26 passed and 4 strict xfails track the known schema-fencing gap in #51318. - [x] Dedicated `SparkBackfill` tag collects 31 cases: 30 V3 and 1 V2. - [x] Normal nightly tags `L0 L1 L2 ClusterOnly` select 0 Spark Backfill cases, even when the opt-in collection gate is opened. - [x] `ruff check tests/python_client/common/common_type.py tests/python_client/conftest.py tests/python_client/spark_backfill` - [x] `git diff upstream/master...HEAD --check` ## Operational notes - Spark execution uses `spark-submit --master local[2]` in one toolbox pod. - The toolbox is discovered by `app=spark-milvus-toolbox` and is managed outside pytest. - V3 and V2 must use independent Milvus deployments. --------- Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
28 lines
1.2 KiB
INI
28 lines
1.2 KiB
INI
[pytest]
|
|
|
|
|
|
addopts = -p no:locust -v
|
|
# python3 -W ignore -m pytest
|
|
|
|
log_format = [%(asctime)s - %(levelname)s - %(name)s]: %(message)s (%(filename)s:%(lineno)s)
|
|
log_date_format = %Y-%m-%d %H:%M:%S
|
|
markers =
|
|
tags: custom tags for test cases
|
|
CDC: CDC sync test cases (not run in regular e2e pipeline)
|
|
spark_e2e: requires a remote Spark runtime
|
|
spark_backfill_v3: Spark-Milvus Backfill tests requiring real Storage V3 segments
|
|
spark_backfill_v2: Spark-Milvus Backfill tests requiring real Storage V2 segments
|
|
spark_backfill_core: core read, backfill, commit, and visibility scenarios
|
|
spark_backfill_negative: invalid Parquet, schema, mode, and batch-size scenarios
|
|
spark_backfill_compaction: Snapshot compaction-protection and stale-result scenarios
|
|
spark_backfill_known_gap: tracked expected failures for known server protection gaps
|
|
compaction_data_integrity_serial: requires the dedicated non-xdist compaction data-integrity stage
|
|
|
|
filterwarnings =
|
|
ignore::DeprecationWarning
|
|
ignore::pymilvus.decorators.PyMilvusDeprecationWarning
|
|
ignore:invalid escape sequence:SyntaxWarning
|
|
|
|
asyncio_default_fixture_loop_scope = function
|
|
timeout_method = thread
|