## What this PR does
Adds dedicated Spark-Milvus Backfill E2E coverage and the runtime
tooling needed to execute it against real Storage V3 and Storage V2
segments.
- Adds opt-in, serial-only pytest coverage for Backfill modes, null
semantics, snapshot boundaries, schema fencing, manifest versions,
compaction, and invalid inputs.
- Adds Kubernetes Job and reusable Spark Toolbox runners with credential
redaction, RBAC validation, evidence collection, and cleanup.
- Builds `zilliztech/spark-milvus` in the toolbox pod and runs Spark
locally through the validated wrapper and native-library runtime.
- Separates Storage V3 and Storage V2 execution through pytest markers
and independent Milvus deployments.
- Uses the dedicated `SparkBackfill` tag, so the normal `L0 L1 L2
ClusterOnly` nightly selection does not include this suite.
- Documents manual toolbox deployment and dedicated-suite execution.
Fixes#52130
## Pipeline isolation
The suite is ignored during ordinary Python client collection unless
`--run-spark-backfill` is explicitly provided. The dedicated pipeline
additionally selects `--tags SparkBackfill`; existing PR, nightly, and
weekly tag sets therefore select zero Spark Backfill cases.
V3 and V2 remain separate marker selections because they require
different Milvus storage configurations and must run serially without
pytest-xdist.
## Test plan
- [x] Dedicated Spark Backfill pipeline completed successfully.
- [x] Storage V3: 30 cases collected; 26 passed and 4 strict xfails
track the known schema-fencing gap in #51318.
- [x] Dedicated `SparkBackfill` tag collects 31 cases: 30 V3 and 1 V2.
- [x] Normal nightly tags `L0 L1 L2 ClusterOnly` select 0 Spark Backfill
cases, even when the opt-in collection gate is opened.
- [x] `ruff check tests/python_client/common/common_type.py
tests/python_client/conftest.py tests/python_client/spark_backfill`
- [x] `git diff upstream/master...HEAD --check`
## Operational notes
- Spark execution uses `spark-submit --master local[2]` in one toolbox
pod.
- The toolbox is discovered by `app=spark-milvus-toolbox` and is managed
outside pytest.
- V3 and V2 must use independent Milvus deployments.
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #53307
Protocol dependency: milvus-io/milvus-proto#668 (merged).
## What changed
- Reuse the existing RPCs to expose retained segment lifecycle metadata
and direct compaction sources; retain the existing sealed/loaded segment
reporting contract without adding Growing segments to
GetQuerySegmentInfo.
- Expose collection-scoped retained compaction tasks, with complete task
identity, public type/state enums, source/target sets, and failure
reasons, while preserving manual job-ID lookup.
- Preserve failure reasons across worker cleanup without changing task
cleanup or metadata GC semantics.
- Add seven **L3-only** data-integrity workloads: fixed V2/V3 Insert,
fixed V2/V3 Import, V2 add/drop-field evolution through MixCompaction
with schema bump disabled, V3 schema evolution, and V2-to-V3-to-V2
storage transitions.
- Validate deterministic full-row/full-field data against an independent
oracle, fenced persistence/serving checkpoints, and actual lineage
transitions, allowing either automatic or manual compaction.
- Provide `tests/scripts/ci_compaction_integrity.sh` as a dedicated,
single-invocation, serial entry; leave the shared
`tests/scripts/ci_e2e.sh` unchanged from the PR base.
- Retain successful audit records only for the dedicated workload
marker, preserving ordinary tests' JSON, HTML, and worker-report
formats.
## Compatibility and execution boundary
- Omitted segment states preserve the existing persistent-state
behavior; task and dropped-segment history are subject to existing
retention/GC, not a fixed response-size or record-count limit.
- These workloads are not added to per-PR E2E or ordinary Nightly
pipelines.
- The dedicated job must own the deployment/config namespace
exclusively, install a compatible PyMilvus build, and provide connection
options plus a unique `CI_LOG_PATH`.
- The entry enforces `-n 0`, the L3 workload marker, and explicit
opt-in, while forwarding connection options, filters, and pytest's exit
status.
- Insert/Import workloads use three rounds by default; insert load units
are batches, not a promise of a physical segment count, and dense vector
dimensions are 16.
## Observability semantics and follow-ups
- **Segment listing is not an atomic snapshot.** The Proxy first calls
`GetSegmentsByStates`, then `GetSegmentInfo`; returned details are
filtered again against the requested states (or the legacy default:
Flushing, Flushed, Sealed). A segment whose fetched state no longer
matches is omitted, and a segment becoming eligible after ID enumeration
may be absent. Consumers must not treat a single response as a
consistent lifecycle snapshot.
- **Existing lookup errors are preserved.** `IncludeUnHealthy` is
enabled only when Dropped is explicitly requested. In the default
healthy-only path, a selected source becoming Dropped before detail
lookup still returns `SegmentNotFound`, as before; that error is not
converted into a successful partial list. A history segment being GC'd
between the two RPCs can also fail the lookup.
- **Aggregate `Completed` means no running tasks, not universal
success.** This preserves the existing aggregation semantics:
all-failed/all-timeout retained tasks, or an empty task set, can report
Completed. Inspect each task's `state` and `failure_reason` for its
outcome; proving an actual data transition additionally requires
lineage, active-set adoption, and serving-set evidence. This PR does not
redefine task retry or cleanup behavior.
- **History capacity is a follow-up.** Dropped-segment queries enumerate
the retained history and fetch its details, while collection-scoped
compaction queries return retained tasks across triggers. Neither path
currently provides pagination or a fixed count/size cap, so scan, copy,
and response costs grow with the retained record count despite
retention/GC. Pagination and capacity controls for both paths are
deferred; they must preserve the ability to enumerate complete retained
history rather than silently truncate lineage or task evidence.
## Previously recorded verification
The results below are earlier recorded runs, including the original
six-workload suite before the V2 DDL workload was added; the
documentation/review-response update did not rerun tests.
- 22 server-independent CI boundary tests passed, covering
ordinary/special audit output, worker report merging, scoped execution
guards, and the dedicated entry's argument/exit-code behavior.
- 20 differential checks against the PR-base logging plugin produced
byte-identical ordinary worker JSON, report JSON, and HTML across all
five result outcomes.
- 14 existing DataIntegrity and etcd-helper unit tests passed.
- The three original L1 DataIntegrity parameter instances passed with
`-n 2 --dist loadgroup` in 59.39 seconds; their merged successful JSON
records retained the original metadata-only shape.
- Ruff lint/format, shell syntax, and scoped PR-base whitespace checks
passed; the shared runner is byte-identical to the PR base.
- The new dedicated entry ran all six workloads serially against the
existing local cluster: **6 passed in 3373.52 seconds**, with three
rounds for each Insert/Import workload and the storage-transition
workload, plus five DDL checkpoints over 25,000 rows.
- All six successful reports retained their audit evidence (403
structured events), including every expected round/DDL checkpoint, full
segment/task snapshots, serving frontiers, and matching
expected/retrieved PK digests.
## Design and job entry
- [Data-integrity design
contract](https://github.com/MrPresent-Han/milvus/blob/codex/compaction-e2e-observability/docs/design-docs/design_docs/compaction_data_integrity_e2e_design.md)
- [Dedicated serial
entry](https://github.com/MrPresent-Han/milvus/blob/codex/compaction-e2e-observability/tests/scripts/ci_compaction_integrity.sh)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
issue: #51544Reopens#51755, whose author has left the project. The branch is rebased
onto current master and the conflicts are resolved.
## What changed
- Derive the default Python client test namespace from `MILVUS_NS` or
`MILVUS_HELM_NAMESPACE` before falling back to `chaos-testing`.
- Add an optional external-source MinIO endpoint override
(`MILVUS_EXTERNAL_SOURCE_MINIO_HOST`) for external table tests, so the
client upload endpoint and the server-side `external_source` endpoint
can be configured independently.
- Resolve the bulk insert MinIO endpoint from the existing `minio_host`
/ `minio_bucket` fixtures instead of looking up the MinIO pod IP through
the k8s API, so these cases no longer require in-cluster access.
- Extend TEXT LOB layout validation to read Vortex segment data files in
addition to the existing parquet layout. The Vortex files are decoded
with `vortex-data`; when the installed reader cannot decode Milvus
StorageV3's custom encoding the case skips rather than asserting on
synthesized references.
## Dropped from the original PR
Two changes from #51755 have since landed on master independently and
are not included here:
- The multi-primary-key invalid schema assertion — master already
validates at `add_field` time, via the framework's `CheckTasks.err_res`
helper rather than `pytest.raises`.
- The `Date` / `Time` / `Decimal` entries in the external table
add-field matrix.
## Verification
- `test_text_lob_bulk_import_json_csv_parquet`
- `test_text_lob_inline_vs_lob_object_layout`
- `test_milvus_client_external_table_add_all_supported_field_types`
- `test_milvus_client_xgboost_supports_all_numeric_feature_types`
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: nico <cheng.yuan@zilliz.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- Expand Python MilvusClient external_table E2E coverage for schema
validation, lifecycle, DQL, refresh, indexing, read-only/write-blocked
behavior, and multi-format datasets.
- Cover parquet, lance-table, iceberg-table, and vortex basic format
flows with shared data builders and stricter assertions.
- Add REST v2 external collection coverage for external job APIs,
nullable vectors, custom database flow, and format matrix behavior.
- Move Vortex data generation to the Python 3.12 test environment and
remove the old sidecar helper scripts.
- Add Python 3.12 environment setup in `tests/scripts` so E2E scripts
create and activate the expected test runtime.
## Notes
- Vortex full-matrix coverage excludes fields that the current
`vortex-data==0.56.0` writer or Milvus refresh path cannot represent
consistently (`FixedSizeBinary`-backed vector fields and known
string/JSON fields).
- Chaos fault-injection coverage is included in the PR but was not run
against the shared validation instance because it mutates the
deployment.
## Validation
- `uv venv -p 3.12 .venv`
- `uv pip install --python .venv/bin/python -r requirements.txt`
- `pytest -q -x -n 6 milvus_client/test_milvus_client_external_table.py
--host 10.100.36.207 --port 19530 --minio_host 10.100.36.172
--minio_bucket external-table-pr49378-master`
- `162 passed, 2 skipped in 176.78s`
- `pytest -q -x -n 6 testcases/test_external_collection_operations.py
--endpoint http://10.100.36.207:19530 --token root:Milvus --minio_host
10.100.36.172 --bucket_name external-table-pr49378-master`
- `22 passed, 4 xfailed in 66.04s`
- `git diff --check`
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Add 40 E2E tests for FileResource CRUD APIs and analyzer integration
(jieba, synonym, stop words, decompounder)
- Verify tokenization correctness with baseline comparison (with vs
without file resources)
- Test data generated at runtime, no static files committed
/kind improvement
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Related issue: #32653
1. Remove some meaningless common types
2. Refine error assertion in negative cases
3. Remove some dup tests
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
add freshness checker
insert/upsert --> query: Get the time when it can be queried
delete --> query: Get the time when it can not be queried
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
