mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-28 17:05:53 +08:00
master
2464
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zhuwenxing
|
3e7b80b5f0 |
test: avoid duplicate recovery continuation workloads (#54740)
The post-chaos collection verification stage can exceed its 60-minute timeout because each concurrent worker creates a dedicated recovery continuation collection. With five concurrent workers and one single-request workload, verification includes five ordinary collections and six recovery collections. At three workers and at least 16 minutes per case, this requires at least 64 minutes before other overhead. Keep recovery collection creation in the single-request workload and remove duplicate creation from concurrent workers. The concurrent workload still resumes existing recovery collections and verifies their retained input journals. For this job configuration, post-chaos verification is reduced from 11 cases to 6; recovery continuation coverage uses one collection instead of six. Validation: - Ruff lint and formatting checks passed. - Python compilation passed. - Isolated initializer checks with mocked checker constructors passed: five concurrent workers plus one single-request worker create one recovery checker; existing recovery collections resume exclusively; ordinary collections create no recovery checker. - Jenkins/E2E was not rerun. Actual runtime improvement remains unverified. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhuwenxing
|
ec6c06079a |
test: accept both whitespace formats in query pagination errors (#54751)
The invalid query limit/offset tests now accept both `[ ]` and `[]` when the input is a single space. Older redact versions remove that space while newer versions preserve it; #53910 upgraded the dependency on master, but 3.0 still uses the older version. Keep the existing error-response checks and validate the parameter name, displayed value, and `is invalid` clause. All other parameter values retain their original expectations. This makes the expectations introduced by #54680 and #54681 compatible with both dependency versions. Validation: - Ruff lint/format, Python compilation, and git diff checks passed. - All 8 parameterized test instances collected successfully. - Offline replay through the actual testcase methods and response checker accepted 10 compatible error responses and rejected 40 negative controls, including successful responses, wrong parameters, wrong values, and unrelated errors. - Both original whitespace failures were reproduced before the change. - Live-server E2E was not run. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhuwenxing
|
e6aff936b7 |
test: expand CDC API contract and boundary coverage (#54721)
CDC tests miss several mutation and control APIs, and some existing checks can pass after a failed read or by inspecting only the primary. This change verifies replicated state, intentionally local operations, and rejected requests against their respective contracts. - Cover schema/function mutations, partial updates, truncate, RBAC, resource groups, import commit/abort, snapshot/refresh boundaries, and unsupported RPCs with state and subsequent-write checks. - Strengthen existing load/release, index, alias, partition, permission, and switchover assertions. Mark cases that require a CDC topology as L3. - Use MilvusClient for supported SDK operations and remove MilvusSys/global ORM setup from CDC paths. Retain authenticated protocol coverage where public SDK wrappers are unavailable. Document configuration requirements and the explicit instance-name option for stability tests. Validation: - 72 selected cases passed across focused runs against Milvus commit `7abbd0c5f7f` with PyMilvus `3.1.0rc98`; this is an aggregate, not a full-suite run. - 11 cases passed with ORM connections and MilvusSys initialization forbidden. The corrected local restore/refresh boundaries and separate secondary-refresh rejection case also passed. - Ruff checks and formatting passed. After rebasing onto master, L3 collection selected 106 of 171 CDC cases. - No new Milvus restarts or panic signatures were observed in the focused verification windows. Long-duration stability workloads were not rerun; their connection initialization was checked separately. Recurring Jenkins selection is not verified. Setup and execution details are documented in `tests/python_client/cdc/README.md`. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhuwenxing
|
abf9482c98 |
test: add recovery continuation checker to chaos workloads (#54682)
## What does this PR do? Add a recovery continuation checker to the existing chaos workloads without new runtime options. It alternates small and large payloads in an independent collection, retains write inputs across test phases, and verifies continued writes and data integrity after flush and reload. ## How was this PR tested? Ruff checks and collection of the four existing pytest entrypoints passed. Live cluster recovery has not been validated. ## Related issues Related to #54650. This adds general recovery workload coverage; hitting the issue-specific recovery path still depends on scenario orchestration. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhuwenxing
|
f51f353725 |
test: preserve whitespace in query pagination error expectations (#54680)
Query pagination tests strip whitespace from invalid limit and offset values when constructing expected errors. The server echoes the original values, so the `" "` cases incorrectly expect `[]` instead of `[ ]`. Remove the stripping and its misleading comments while retaining assertions on the parameter name, original value, and invalid-parameter message. Validation: - Ruff, Python compilation, and git diff checks passed. - All 8 parameterized cases collected successfully. - Offline assertion replay passed for all 8 cases; the original two whitespace failures were reproduced, and successful responses and unrelated errors were rejected. - Live-server E2E was not run. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
yanliang567
|
f067780bd8 |
test: strengthen compaction PR CI coverage (#53273)
## What this PR does Strengthens compaction coverage in the PR gate and removes false-positive and flaky paths. - preserves terminal success/failure/timeout outcomes after task cleanup and uses `EndTime` (with a legacy `StartTime` fallback) for post-finish GC retention - resolves target-based manual compaction through durable target metadata: an executable ACTIVE target remains `Executing` across reconciliation batches, INACTIVE completes only after every task is classified, inert ACTIVE targets and unknown/GCed IDs remain undefined, and `compact_id=-1` remains the explicit no-op sentinel - keeps unknown task states and invalid cleaned terminal states fail-closed even when their target is runtime-active - refuses successful aggregation when any task state or cleaned terminal outcome is unclassified, and covers INACTIVE targets with still-executing tasks - makes the Python compaction waiter consume structured plan counts with context metadata and one monotonic deadline across raw RPC attempts, reconnect, and backoff - restores channel recovery for `UNAVAILABLE` through the synchronous, timeout-aware SDK `reconnect()` interface without leaving a recovery worker that can mutate the handler after timeout - polls compaction-eligible segments through a single raw `GetPersistentSegmentInfo` RPC per observation, preserving call context and status mapping without the SDK retry decorator reusing a stale timeout - retries transient gRPC and rate-limit failures within the remaining deadline, and recognizes both current `code=600` and legacy `error_code=SegmentNotFound` wire formats - retries recoverable `Status.CONNECT_FAILED` reconnect-handshake failures in segment polling and PK pagination within the existing deadline/backoff; non-recoverable handshake errors still propagate - disables auto compaction before topology/concurrency setup and validates MixCompaction source/target multiplicity, exact topology, row count, PK uniqueness, and exact PK set - proves explicit ForceMerge `target_size` behavior with a fixed `segment.maxSize == 64MB` fixture: equivalent datasets must produce 3 outputs at 64MB and 2 outputs at 128MB - exposes finite ForceMerge QueryNode/DataNode memory factors in DataCoord system metrics; non-finite dynamic values fall back to the default before serialization - reads the unfiltered system topology and skips the target-size comparison when QueryNode/DataNode metrics are erroneous or the observed machine-safe ceiling cannot preserve the larger effective target - directly regression-tests the L3 machine-safe wrapper with healthy typed nodes plus an untyped metrics error, preventing a future return to filtered topology input - verifies every planned ForceMerge source leaves the active topology, every exposed target becomes active, the complete PK set survives, and physical rewrite bytes are conserved - releases and reloads the collection before MixCompaction and ForceMerge PK validation, then verifies the complete loaded segment set, retired-source exclusion, and node assignments so readable old inputs cannot mask missing compaction outputs - submits reload asynchronously through single-attempt SDK calls and polls loading progress with fresh remaining budgets; release, load submission, progress, and topology checks share one monotonic deadline - replaces `QueryIterator` validation with deterministic PK-range pages; every raw query RPC receives a freshly calculated remaining timeout, retains MilvusClient `cluster_id`, and retries transient gRPC/rate-limit failures within the same total deadline - verifies reconnect receives only the remaining pagination deadline, an over-budget reconnect cannot issue another query RPC, ignored gRPC codes and real rc83 non-rate-limit business statuses fail fast, and both current and legacy `(code=0, compatible_code=49)` rate-limit mappings retry within budget - preserves valid ForceMerge QueryNode/DataNode memory factors, while non-finite values fall back and all finite factors below 1—including positive fractional values—clamp to 1 - sustains checked insert/CreateIndex/search workloads across compaction and propagates every worker failure, including the last in-flight request during cleanup; an Executing sample counts as overlap evidence only when the same workload call spans the observation - waits for two matching Flushed/L1/sorted source snapshots before the index workload, then expects unselected sources plus Mix targets; selected sources must retire and every loaded segment must have an index and node assignment - validates final PKs, indexed targets, and search results before marking an unobserved overlap window as insufficient coverage (pytest skip); repeated CreateIndex requests establish request concurrency, not repeated index builds - validates complete binary-search results against exact Jaccard ground truth - adds DataCoord aggregation/target lifecycle UT and compaction stats-loader UT coverage `CompactionMergeInfo` exposes one legacy `target` field although a ForceMerge task may split into multiple result segments. Split-output validation therefore requires each exposed target to be active and separately requires every planned source to be retired; complete active-output multiplicity is verified through the persistent-segment topology. Fixes #53272 ## Test plan - [x] Partial-plan review follow-up (2026-10-22): 82 focused local tests passed with Python 3.12 / PyMilvus rc98, plus Ruff lint/format and diff checks. The regression invokes the actual index E2E case with an 8-of-9 Mix plan: the prior implementation times out, the fix accepts the valid indexed target plus unselected source. Negative cases reject retained selected sources, missing/unexpected/unindexed/unassigned segments, and missing PKs; the ORM adapter exercises sorting convergence and context preservation. - [ ] Live index concurrency rerun is blocked: both Dev CLI deployment-list and `deploy status compaction-pr-ci` requests return HTTP 500. No live-cluster pass is claimed for this follow-up. - [x] Review follow-up on 2026-10-22: Python 3.12 / pinned PyMilvus 3.1.0rc98 focused suite: 72 passed; Ruff lint/format and diff whitespace checks passed; all three revised concurrency E2E cases collect successfully. Fault injection covers fast completion, unobserved overlap, worker failures before and after terminal observation, and cleanup. Real SDK loading regressions fail on the previous implementation (5-second budget returns at t=8) and pass with fresh per-poll budgets. - [ ] Re-run the three sustained-workload cases and release/reload integrity cases against a live cluster; this follow-up has local regression coverage only. - [ ] Full `make verifiers` attempted on 2026-10-22 but stopped in `build-3rdparty`: this macOS host lacks a supported LLVM 14–18 toolchain. - [x] Review follow-up on 2026-10-20: Python 3.12 / pinned PyMilvus 3.1.0rc98 focused suite: 56 passed; Ruff lint/format, Python syntax, and diff whitespace checks passed. Reconnect regressions exercise the actual SDK reconnect/readiness functions; topology tests inject delayed/partial/stuck loading, reload failures, and late responses. - [ ] Re-run live MixCompaction / ForceMerge integrity E2E with the new release/reload validation; the review follow-up was verified locally with fault-injection tests and server-path inspection. - [x] Go formatting, Python 3.12 compile, Ruff check/format, protobuf submodule compile, and `git diff --check` - [x] Local Python 3.12 waiter/raw-adapter/pagination/machine-safe matrix: 38 passed - [x] Local focused paramtable + metrics UT: `go test -tags dynamic,test -gcflags='all=-N -l'` - [x] Dev CLI exact non-finite memory-factor/serialization UT: `go-ut-local-yanliang-test-compaction-pr-5aa7d24-ce47cff0-u3805` - [x] Dev CLI exact full memory-factor preservation/fallback matrix: `go-ut-local-yanliang-test-compaction-pr-fab7e57-ce47cff0-u3805` - [x] Dev CLI exact fractional-boundary memory-factor matrix: `go-ut-local-yanliang-test-compaction-pr-3152b2a-ce47cff0-u3805` - [ ] Dev CLI targeted retry L0 could not start pytest because the runner Pod could not resolve the COS source host: `pysdk-e2e-local-yanliang-test-compaction-d642871-420126f0-u3805`; the same 3-case rc83 test passes locally - [x] Dev CLI current server image build: `build-local-yanliang-qia-test-compac-5aa7d24-bf331245b298-u3805` - [x] Deploy `harbor.tencent-stage.zilliz.cn/manta/milvus:local-u3805-test-compaction-pr-ci-coverage-5aa7d24-bf331245b298` to `compaction-pr-ci`; verify `Healthy` and `updated=true` - [x] Dev CLI current DataCoord + metrics UT: `go-ut-local-yanliang-test-compaction-pr-d3b086b-57710439-u3805` - [x] Dev CLI exact GC-retention suite: `go-ut-local-yanliang-test-compaction-pr-c9996b9-29281152-u3805` - [x] Dev CLI full `internal/compaction` + `internal/datacoord` Go UT for the broader PR changes: `go-ut-local-yanliang-test-compaction-pr-7b125fe-6ba90598-u3805` - [x] Dev CLI real no-op trigger plus unknown-ID E2E (3/3): `pysdk-e2e-local-yanliang-test-compaction-710c3bc-cef493b1-u3805` - [x] Dev CLI exact binary-search validation (1/1): `pysdk-e2e-local-yanliang-test-compaction-0728a4d-714e12a8-u3805` - [ ] Run the final fixed 64/128MB ForceMerge topology/full-PK E2E. The latest local-source runner still fails before pytest because its Pod cannot resolve the COS source host: `pysdk-e2e-local-yanliang-test-compaction-5aa7d24-4e733021-u3805`; the PR-source channel independently cannot resolve GitHub: `pysdk-e2e-pr-53273-5aa7d24-8424db8b-u3805` - [ ] Re-run the final Mix multiplicity assertion and insert/index result-binding cases after the same source-runner DNS incident is resolved ## Notes Upgrade, chaos, performance, and experimental-configuration coverage remain lower-priority follow-up work. Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
cai.zhangandClaude Opus 5.5
|
df6811a24c |
enhance: allow hyphens in resource group names (#53818)
issue: #52979 ## Hyphens in resource group names `ValidateResourceGroupName` now accepts `-` after the first character, the rule partition names already follow. The first character must still be a letter or an underscore, so `-rg` stays invalid. Nothing that parses resource group names treats `-` specially: they reach etcd keys, node labels and metric label values, and all of them accept `-`. The Python resource group suite drops the hyphenated entry from the shared invalid-name list for its create case only; the shared list is left alone because other objects use it. It gains a case that creates, describes and drops a hyphenated group and still refuses a leading hyphen. ## Testing - `internal/proxy`: the resource-group name tests (`TestValidateResourceGroupName*`) pass; the whole package passes except `TestProxy`, which needs a live message queue and hangs locally with or without this commit. - `golangci-lint` with the repo config reports 0 issues on `internal/proxy`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
zhuwenxing
|
b589299116 |
test: cover CDC partitions created after collection load (#54519)
Related issue: #54467 Related fix: #54517 Add CDC regression coverage for partitions created after the collection has finished loading on both clusters. - Cover inserts with and without flush. - Verify a partition created before the initial load as a control, then create three new partitions after loading. - Check that each new partition becomes Loaded on the secondary and that partition-level and collection-level queries return the exact expected IDs, scalar values, and vectors, without additional load/release calls. Validation: - Ruff lint/format, Python compilation, and pytest collection passed (2 new instances). - E2E: two standalone instances, Woodpecker, 16 physical channels, PyMilvus 3.1.0rc98; identical test code and assertions for both runs. - Master `ac107a4d00f`: 2 failed; the first new partition remained NotLoad for the full 120-second sync timeout in both instances. - PR #54517 head `cb63cfd9324`: 2 passed in 56.43 seconds; all three post-load partitions were verified in each instance. These cases are expected to fail on affected builds until #54517 or an equivalent fix is merged. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
Zhen Ye
|
cf0e5b68a0 |
fix: prepare analyzer file resources through streaming client (#54302)
issue: #54102 - Resource loading: add an internal batch FetchFileResources RPC by bound resource IDs or analyzer resource names. AnalyzerResourceManager owns metadata fetching, per-ID download flights and local file references, with transient failures retried locally and shared downloads canceled only after the last waiter leaves. Retain unused files for common.fileResource.cacheTtl (default 15m), with safe reacquisition, expiration and shutdown cleanup. - Analyzer validation: route coordinator validation through the streaming client, check remote resource names and filenames against fetched metadata, and return validation/preparation failures through business Status. Preserve resource reservation around DDL publication and distinguish invalid input from system failures, including generic Rust IoNotFound handling at the C++ validation boundary. - Runner lifecycle and WAL: pin exact resource bindings through native runner lifetime, include schema resource IDs in runner identity, and initialize resource-dependent runners on demand. Prepare unavailable runners in the redo interceptor after inner locks and admission scopes unwind, then retry materialization while preserving the outermost idempotency guard. - Integration: retain public RPCs, legacy resource distribution, mode selection and the WAL assignment readiness gate as groundwork for qv. The new loading path follows on-demand execution and Milvus's supported upgrade order, with MixCoord upgraded before worker nodes. --------- Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
Zhen Ye
|
d0ce73cabd |
feat: support cluster-local flush on primary and secondary (#54380)
issue: #54278 - Flush APIs: allow cluster-local Flush/FlushAll on either replication role using `_ur`, existing resource locks and AckSyncUp. A local Flush overtaken by replicated Drop completes as an empty best-effort WAL operation. Flush does not promise source catch-up or trigger a remote Flush. - Transactions and CDC: replay complete source transactions after abort; validate nonzero Commit `body_cnt` before WAL append on both roles and deduplicate replicated bodies by source MessageID. CDC preserves historical zero counts without backfilling validation. - WAL and recovery: use one scoped transaction-abort path after successful exclusive appends, including Flush and ForcePromote, and apply the same boundaries during consumption. Failed, deduplicated and explicitly ignored operations preserve sessions. ForcePromote clears old transactions before its all-channel ACK; subsequent ignored topology supplements do not abort new local transactions. Retry backend context errors and TSO allocation failures while the WAL is live; Commit succeeds only after append, with cancellation/fencing outcomes resolved by recovery on either role. - Coordination and documentation: retain the existing broadcaster resource locks and worker pool. Local exclusive appends rely on the outer WAL lock and acquire the replication mutex only for successful prefix cleanup, preserving progress on unrelated VChannels. Document retry/recovery preconditions, the corrected ManualFlush dispatch contract, and user-visible Flush behavior. Operators must upgrade all participating CDC, Proxy, StreamingNode and consumer processes before invoking local Flush on a secondary. Cross-cluster upgrade coordination remains operational; no version-probing gate is added. --------- Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
wei liu
|
42d337e774 |
feat: Support precomputed sparse vectors in external collections (#53893)
issue: #53827 External collections reject precomputed sparse vector fields before reading the source. Allow these fields and normalize Arrow Map/Struct columns to Milvus sparse rows at the shared index, load, Take and sampling boundary. Preserve null versus empty rows, reject malformed indices and weights, and normalize source Binary rows, reusing already ordered Binary arrays. Reuse sparse binary validation in the native row copier while retaining the target branch's length checks and error classification. Add regression cases for Parquet, Lance and Vortex, and update the existing external-table design. Preflight conversion buffers against a startup-only budget of 256 MiB. Encode ordered rows directly, reserve scratch only for unordered rows, and handle BinaryView/LargeBinary before generic canonicalization can allocate. Accept two null Struct child lists as an empty vector. Fixed-budget rejection terminates index tasks through Unsupported; allocation failures retain their existing retry classification. Design: [External table design](https://github.com/weiliu1031/milvus/blob/support-external-sparse/docs/design-docs/design_docs/20260105-external_table.md) Validation (current rebased candidate): - Rebased onto master `4f0f748f1056446b6c84366a7295ff6cd0597685`. Preserved the upstream array-validity parameter; the prior fixture-fix commit was already present upstream and was dropped. - 10 isolated C++ tests passed using the extracted production entry, dispatcher, canonicalizer and sparse conversion with real Arrow. Metadata/error scaffolding and unrelated branches were simplified; this is not a full-core integration run. - The isolated production index scheduler classification test passed with 100% statement coverage. The paramtable common-config test passed. Go tests used `-tags dynamic,test -gcflags="all=-N -l"`. - `git diff --check` passed; source equivalence was checked with `git range-diff`. - Full index package and C++ test compilation remain blocked by stale local native C ABI and generated protobuf dependencies. Full-package and end-to-end runtime validation remain outstanding. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
zhuwenxing
|
d568a53b4b |
test: bump PyMilvus to 3.1.0rc98 (#54331)
Pin the Python client test dependencies to PyMilvus 3.1.0rc98, which
includes milvus-io/pymilvus#3816. This unblocks the nullable StructArray
and sparse-vector bulk import used by the continuous integrity workload
in #54158.
Keep the existing null-preserving test oracle unchanged. This PR updates
both the base package and bulk_writer extra pins.
Add three focused L1 regression cases: mixed-value insert, mixed-value
Parquet bulk import, and all-null Parquet bulk import. Verify explicit
nulls, omitted fields, empty StructArrays, Parquet null counts, and
exact queried values before and after release/load. All-null sparse
insert remains tracked separately in #54330.
Validation against Milvus commit
|
||
|
zhenshan.cao
|
0ebf9f2d1d |
test: test_connection.py asserts legacy port-range error messages that (#54325)
<!-- team2-testfix:54322 --> fix: #54322 ## What Test-only change: the assertion pinned a value generated by a dependency, which changes legitimately when that dependency is upgraded. Product code is untouched. ## Evidence The single failing case was run against one running Milvus instance, alternating unpatched/patched twice each (`testcases/test_connection.py::TestConnectAddressInvalid::test_parameters_with_invalid_address_port[default-65536]`): | run | unpatched | patched | |---|---|---| | 1 | fails with the reported assertion | passes | | 2 | fails with the reported assertion | passes | ## Analysis The case `test_parameters_with_invalid_address_port[default-65536]` asserts that `connect(address="127.0.0.1:65536")` fails with the gRPC-layer message `illegal connection params or server unavailable`. That expectation only held while pymilvus accepted any int-convertible port string and let the connection attempt fail later. Since pymilvus 3.1.0rc95 (pymilvus#3812), `is_legal_port` rejects out-of-range ports up front, so `is_legal_address` raises `Illegal address: 127.0.0.1:65536, should be in form 'localhost:19530'` before any RPC is sent. The Milvus server is never contacted, so there is no server behaviour to fix; rejecting port 65536 at validation time is the correct client behaviour. The test's error-message check is a substring match, so the fix narrows to splitting the port parametrization so that `"100"` keeps the gRPC-layer message while `"65536"` expects `Illegal address`. The sibling case `test_connection_add_connection_kwargs_invalid_host_port` has the same drift for port `"-1"` and should expect `Type of 'port' must be str or int`. --- _Automated draft from the team2 pipeline. Not auto-merged; please review._ Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> |
||
|
yanliang567
|
4f0f748f10 |
fix: Preserve first Data Integrity failure and diagnostics (#54158)
## Summary - Preserve the continuous-integrity design section 6 null contract: import rows keep nullable StructArray and sparse-vector nulls, and the oracle uses the original canonical expectations. Only lossless INT8 physical representation conversion is applied. - Record coordinator failures before executor cancellation/shutdown, and preserve observer first errors throughout final checkpoint, fenced dataset, and post-ingress validation. - Persist bounded first-error stack locations and secondary failures, and add case-phase/progress events without logging exception payloads or locals. - Make the dedicated E2E entrypoint initialize plugin import paths, INFO console output, and a default JSONL destination while preserving caller-supplied diagnostic paths. - Include both diagnostic/continuous regression modules in the dedicated unit entrypoint and update argument/exit-status contracts. issue: #54058 This PR does not claim to fix SDK/import null support. Unsupported null handling must fail visibly; converting nulls and rewriting the oracle would violate the continuous workload contract. The earlier workaround has been removed. ## Review follow-up - Preserve original nullable values and oracle expectations. - Capture coordinator failures before worker cancellation can claim first-error ownership. - Extend first-error recovery across the observer's final-validation lifetime. - Verify diagnostics through the default E2E entrypoint without external log-level or PYTHONPATH overrides. - Update E2E entrypoint argv and exit-status contracts. - Exercise all new regressions through the dedicated unit entrypoint. ## Test Plan - [x] Dedicated unit entrypoint: 221 passed, including controlled coordinator/observer fault scenarios and the default logging entrypoint probe. - [x] Entrypoint contract tests: 32 passed, including caller argument, diagnostic path, Python path, and exit-code checks. - [x] Dedicated L3 entrypoint collect-only: 9 cases selected. - [x] Ruff check/format, Bash syntax checks, and git diff whitespace checks passed. - [ ] Run full continuous E2E with the restored null contract on a dedicated QA release. The previously reported E2E run used the removed null-conversion workaround and does not validate this revised contract. --------- Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
zhenshan.caoandClaude Opus 5.5
|
228dc7800c |
test: give each RemoteBulkWriter a private local_path (#54272)
## Problem Without `local_path`, `RemoteBulkWriter` stages files under `<dirname(sys.argv[0])>/bulk_writer`, so every pytest-xdist worker shares one parent directory. pymilvus creates that parent and the per-writer uuid subdirectory in two separate `mkdir` calls, and `__exit__` removes the parent when it looks empty. When one worker exits between another worker's two `mkdir` calls, the second `mkdir` fails with `FileNotFoundError`. Observed as `test_import_without_embedding[False-numpy]` failing under `-n 32`. ## Fix Pass `local_path=tempfile.mkdtemp(prefix="bulk_writer_")` at every `RemoteBulkWriter` call site in `tests/python_client` so no two writers share a parent directory. pymilvus still removes the empty directory on exit, so nothing is left behind. Test-only change; no product code touched. ## Verification Ran the full PR-tier e2e (`ci_e2e_4am.sh`, L0+L1, `-n 32`) on a standalone built from this branch: 9113 passed / 428 skipped, including all 6 `test_import_without_embedding` parametrizations and all bulk_insert / data_integrity cases. The only failures were 7 `test_milvus_client_pyudf` cases caused by the PyUDF runtime module not being installed on the test host, unrelated to this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Zhen Ye
|
a56ee2a67c |
enhance: decouple analyzer execution from shard delegators (#54107)
issue: #54102 - Analyzer routing: forward requests directly from Proxy without scheduler tasks; execute inline requests on ready StreamingNodes and field requests on the collection WAL owner, independent of delegators and collection load state. - WAL execution: use each WAL’s current schema and function runners with runner pins and existing WAL/gRPC lifetimes; preserve collection identity across drop/recreate. - Streaming transport: expose AnalyzerClient through WALAccesser; keep streaming failures in native gRPC errors with discovery and recovery shared by HandlerClient, and pass analyzer Status directly through responses. Preserve public and legacy RPC signatures; upgrade StreamingNodes before Proxies, with no QueryNode fallback. --------- Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
pymilvus-bot
|
8ad6654d75 |
test: Increase PyMilvus version to 3.1.0rc95 from master for master branch (#54161)
Automated bump from pymilvus master branch to milvus master branch. Updates tests/python_client/requirements.txt. Signed-off-by: pymilvus-bot <pymilvus@zilliz.com> |
||
|
junjiejiangjjj
|
06161a3334 |
feat: Support pyudf (#52001)
doc: docs/design-docs/design_docs/20260624-function-chain-api.md issue: https://github.com/milvus-io/milvus/issues/52319 https://github.com/milvus-io/milvus/issues/46565 Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
yanliang567
|
a46050c742 |
test: Add NGRAM multi-byte LIKE crash regression (#54139)
## Summary - Add a Python E2E regression case for single multi-byte UTF-8 LIKE literals on an NGRAM-indexed VARCHAR field. - Exercise the inner, prefix, postfix, and wildcard-split match shapes from #53661 with `min_gram=2` and verify exact result IDs plus a follow-up query. - Build the index over 1,100 rows and wait for QueryNode index loading so the test covers the indexed server path. issue: #53661 ## Test Plan - [x] Targeted Python E2E on 4am 3.0 image `3.0-20261011-4079b8b0`: 1 passed. - [x] Ruff check and format check for `test_ngram.py`. Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
pymilvus-bot
|
1e943dc23b |
test: Increase PyMilvus version to 3.1.0rc92 from master for master branch (#54068)
Automated bump from pymilvus master branch to milvus master branch. Updates tests/python_client/requirements.txt. Signed-off-by: pymilvus-bot <pymilvus@zilliz.com> |
||
|
junjiejiangjjj
|
8cfeb14750 |
test: Enable hybrid rerank chain test (#54041)
issue: https://github.com/milvus-io/milvus/issues/46565 Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
Jamesandxiaofanluan
|
d1cb277f0c |
test: wait for compactable segments in struct partial-update lifecycle (#53964)
issue: #53962 The Struct Array partial-update lifecycle test can request manual compaction while its freshly flushed segments are still being sorted. DataCoord then returns the valid no-plan result `-1`, and the immediate positive-ID assertion fails before the compaction/reload checks run. Wait for vector-index readiness, then retry only `-1` against a monotonic 120-second deadline, passing the remaining budget to each compact RPC. Require an accepted compaction to complete and check the baseline both immediately afterward and after release/load. RPC failures, unexpected IDs, timeouts, and data mismatches remain failures. This follows the existing Struct nullable lifecycle test's approach from #52108. The observed failure is in [PR #53941's E2E run](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-gcp/61102/console); this change does not modify server compaction behavior. Validation: - Ruff lint and format checks, Python syntax compilation, and `git diff --check`. - Executed the changed statements extracted from the actual file with simulated time/RPC responses: retry then success, persistent no-plan timeout, invalid IDs, RPC failure, index/compaction wait failures, and both post-compaction and post-reload data mismatches. - Live E2E has not been run against this fix locally; CI validation is pending. The separately requested rerun of #53941 uses its original head and does not validate this patch. Signed-off-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> |
||
|
|
15fcd798cf |
test: add continuous streaming data integrity draft (#53307) (#53868)
issue: #53307 ## What changed - Add a Phase 2 draft continuous-streaming DataIntegrity L3 workload with concurrent MutationIngress and ImportIngress lanes, no manual flush, and no manual compaction during streaming. - Extend the Phase 1 framework through opt-in sequence, level, and filtered-verification helpers while keeping existing cases unchanged. - Add the Phase 2 design contract and include the draft workload in the existing dedicated serial DataIntegrity job. - After automatic coverage and final ingress verification pass, request one post-ingress compaction and require Phase 1 lineage/serving checkpoints, fenced full-data verification, and release/load verification; this does not establish per-Delete L0 attribution. - Keep fixed epochs without producer/validator coordination; missing suffix progress fails with explicit coverage diagnostics for CI failure-rate measurement. ## Validation - Related DataIntegrity unit tests: 202 passed. - Ruff, format checks, and scoped git diff checks passed. - This review update was verified in medium mode; the new post-ingress compaction phase has not yet been rerun against a live cluster. Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: GPT-5 <noreply@openai.com> |
||
|
zhuwenxing
|
734acf93f0 |
test: expand import and snapshot restore data integrity coverage (#53820)
## Summary - Add Python client import coverage for nullable values, defaults, row identity, scalar boundaries, vector encodings, dynamic fields, BM25, geometry, and timestamps across supported file formats. - Cover multi-file progress, empty-file commit/abort, malformed input, and preservation of existing data after failed imports. - Correct legacy import fixtures and assertions, including JSON field NULLs and the distinction between non-finite and null array elements. - Verify packed Storage V2 binlog restore after adding nullable dense/sparse vectors. Native Storage V3 uses the snapshot export/restore path instead. - Strengthen the REST v2 L0 snapshot lifecycle test with per-row and schema comparisons, NULL filtering, vector search, restore after dropping the source snapshot and collection, reload checks, and exported-object cleanup. ## Validation Tested against Milvus commit `1ae8642b8b1` using `milvus-dev-cli` E2E runners. - REST v2 snapshot lifecycle: passed on Storage V2 (76.93s) and Storage V3 (77.97s). - Packed Storage V2 binlog restore: 2 passed (dense and sparse nullable vectors). - Nullable timestamp defaults: 3 passed (JSON, CSV, and Parquet). - Targeted import and legacy bulk-insert runs covered nullable values, defaults, partition keys, malformed files, vector encodings, scalar boundaries, dynamic fields, BM25, and invalid array elements. - Python import testcase collection after renaming: 60 cases collected. - Changed snapshot testcase: Ruff formatting/lint, Python compilation, and pytest collection passed. Renamed import testcase: Ruff lint, Python compilation, and pytest collection passed. Validation used targeted runs during development; the complete final branch test selection has not been rerun as one suite. Snapshot validation used serial REST runs. The default CI storage configuration is not changed by this PR. ## Scope Test-only changes. No production behavior changes. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
marcelo-cjl
|
ad4887b38c |
fix: support vector null predicates (#52442)
issue: #52239 ## Summary - Support IS NULL and IS NOT NULL predicates for ordinary vector fields. - Read logical-row validity through vector-index HasValidData/IsRowValid when raw field data is unavailable, remaining compatible with #50524's Knowhere IdMap ownership. - Rebase onto current master while preserving master's broader search-by-PK result reordering and tests. ## Test Plan - [x] Rebase onto master `38c4699891` with exactly two commits and no merge commit. - [x] `git range-diff` confirms both patches are unchanged by the rebase. - [x] `git diff --check origin/master...HEAD` - [x] `go test ./internal/parser/planparserv2` - [x] `go test -c -o /tmp/milvus-pr-52442-rebased-testcases.test ./testcases` (from `tests/go_client`) - [x] C++ clang-format, Rust format, and Go format checks. - [x] `bash scripts/check_segcore_error_boundaries.sh` - [x] `ruff check` for the four changed Python test files. - [x] `python3 -m py_compile` for the four changed Python test files. - [x] Pre-rebase private Release+ASan integration build at `d839d36b`; the rebased commits are patch-equivalent. - [ ] CI on rebased head `a9e9d31ff2`. --------- Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com> |
||
|
Bingyi Sun
|
215c79e293 |
test: wait for indexes in external table source override (#53883)
The external-table add-field source-override test can load a refreshed snapshot before its embedding index is ready. Use a FLAT/L2 index for this data/schema test and wait for `pending_index_rows == 0` before loading each refreshed snapshot. Validation: Python syntax parsing, helper-signature review, clang-format 15, and `git diff --check` pass. The Milvus/MinIO E2E case was not run locally. `make lint-fix` ran Go formatting, then stopped at unrelated native type-check errors in `internal/storagev2` (missing Loon filesystem metrics APIs). Signed-off-by: sunby <sunbingyi1992@gmail.com> |
||
|
dependabot[bot]
|
ca8c70ba6f |
build(deps): bump deepdiff from 8.6.1 to 8.6.2 in /tests/python_client (#53674)
Bumps [deepdiff](https://github.com/qlustered/deepdiff) from 8.6.1 to 8.6.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/qlustered/deepdiff/releases">deepdiff's releases</a>.</em></p> <blockquote> <p>8.6.2 - Fix (CVE-2025-58367)</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/qlustered/deepdiff/commit/0d07ec21d12b46ef4e489383b363eadc22d990fb"><code>0d07ec2</code></a> Merge commit from fork</li> <li><a href="https://github.com/qlustered/deepdiff/commit/791f5aac51b2da7f90375ab204256ef6a6b40206"><code>791f5aa</code></a> updating CVE number</li> <li><a href="https://github.com/qlustered/deepdiff/commit/a6aafea3ba5498aab6f9c77047c54949e7e968ae"><code>a6aafea</code></a> updating docs</li> <li><a href="https://github.com/qlustered/deepdiff/commit/a0950abbe6263298bc4bdbc3ebb57edc1af079b4"><code>a0950ab</code></a> Bump version: 8.6.1 → 8.6.2</li> <li><a href="https://github.com/qlustered/deepdiff/commit/887128abe5e510341cc4a82f6914a82d5ff1b6a7"><code>887128a</code></a> Fix (CVE-2025-58367)</li> <li>See full diff in <a href="https://github.com/qlustered/deepdiff/compare/8.6.1...8.6.2">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/milvus-io/milvus/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
zhuwenxing
|
dde236e5f9 |
test: decouple external add-field type matrix from E2E (#53777)
## What changed
- Move the proto and PyMilvus DataType matrix check into a separate test
so enum changes cannot prevent the external add-field E2E from reaching
Milvus.
- Classify the new proto-only UUID type and check that DataType
categories are disjoint and cover both proto and SDK enums.
- Verify each added field's type, external field mapping, and nullable
setting before refresh and index creation.
## Verification
- Ruff check and format check passed.
- The standalone DataType matrix test passed with PyMilvus 3.1.0rc89.
- The targeted external add-field E2E passed against Milvus master
commit
|
||
|
Zhen YeandClaude Opus 4.6
|
06ba9d2298 |
enhance: refactor asynchronous WAL recovery and summary persistence (#53595)
issue: #40451 - WAL recovery: replace the legacy flusher with asynchronous VChannel/Segment persistence, retained-message completion, and owner-fenced snapshots before a single global checkpoint. Publish schemas before VChannel base metadata and use the base checkpoint to filter incomplete schema publication during recovery; delegate task concurrency to the shared NodeScheduler. Preserve transaction state through RecoveryBarrier with one scanner, reconcile legacy segment state during migration, and mark the WAL unavailable if background checkpoint persistence terminates unexpectedly. - WALSummary and idempotency: keep summaries and keyed-write deduplication always available; persist chunks/manifests independently, restore deduplication windows, provide bounded Delete reads, and manage WAL coverage, retention, and bootstrap replay boundaries. Seal on staged size or WAL tail pressure, report terminal Summary failures to the WAL owner, and perform physical GC outside reader/publication locks. Remove obsolete idempotency retention settings. - Data persistence: materialize missing function outputs before growing persistence, publish growing L1 data to DataCoord before completing Inserts, and track commits with DataVersion. Retire empty L1 segments explicitly through DataCoord without generating a manifest or publishing DataView; install the SN tombstone without a DataVersion only after retirement is confirmed, allowing ManualFlush to complete. Materialize retained Deletes into L0 using size, age, recovery pressure, or explicit WAL boundaries; retain the Summary-based materializer for future QueryView integration. Report complete recovery positions and poison work after WAL ownership loss. - Lifecycle completion: synchronize ManualFlush, FlushAll, and snapshot creation through consuming-side L1/L0 completion and broadcast acknowledgements. Let Broadcaster add CChannel centrally while callers supply the required collection VChannels. Rename the import ID-range update message to UpdateImport while preserving its payload and behavior. Preserve coordinator-owned Import visibility and legacy CommitImport completion during upgrades, and flush surviving partitions on DropPartition. Fence dropped objects in memory, publish durable tombstones only after asynchronous dependencies finish, and retire metadata and historical schemas through checkpoint- and Summary-gated GC. Cancel stale QueryCoord orphan-segment cleanup when a loaded collection reassigns the node to a replica. --------- Signed-off-by: chyezh <chyezh@outlook.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Feilong HouandEric Hou
|
38c4699891 |
fix: Repair Spark backfill toolbox build dependencies (#53770)
The Spark backfill toolbox init container fails when the pinned Avro recipe downloads the unavailable 1.12.1 archive from Apache's current mirror. The source build also needs additional Rust prerequisites, and the clean Spark runtime cannot load JNI unless libaio is included in the artifacts. This change: - Builds Avro C++ 1.12.2 from its official archive with a pinned SHA256. Reuses only the existing pinned recipe's build instructions, exports a distinct local recipe, and forces the storage dependency graph to that exact revision. - Installs pkg-config, libssl-dev, libclang-dev, and protobuf-compiler for the Rust bridge. - Bundles libaio.so.1 alongside the Conan runtime libraries. - Records the Avro recipe reference in build evidence and documents the prerequisites. issue: #52130 ### Validation - Shell syntax, embedded Python syntax, and git diff --check pass. - Verified Conan source download/checksum and repeatable recipe export; the resolved storage graph uses Avro 1.12.2. - Completed a Kubernetes toolbox build with Spark 4.0.1 / Java 21 on Linux AMD64, using spark-milvus commit 37d2217fe64c352048db71e4c928068b8cdaada6. - Verified artifact hashes, all JNI shared-library dependencies (including Avro 1.12.2), Python imports, BackfillApp class loading, and JNI loading in the Spark JVM. - A local Spark/Parquet round trip returned 100 rows with ID sum 4950. The build completed using caches preserved across retries. The final libaio packaging step was tested separately against the built artifacts, followed by the clean-runtime checks above; a fresh full build was not repeated after that packaging-only change. Milvus/S3 backfill was not exercised. Assembly emitted warnings about skipped Java 21/22 multi-release entries; assembly and the listed runtime checks passed. Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> |
||
|
zhuwenxing
|
2dfd51b8d7 |
test: add external table reload checker (#53784)
issue: #53778 ## Changes - Add `ExternalTableLoadChecker` to repeatedly release and load a fixed external collection, then verify its row count without refreshing the source. - Register `external_table_load` as a chaos operation. ## Verification - `ruff check tests/python_client/chaos/checker.py` - Python 3.12 compile and import checks for the checker and operation enum. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
junjiejiangjjj
|
8c317d2ce1 |
test: fix topology-dependent FunctionChain tests (#53743)
Remove three Python E2E cases that assume L1 rerank runs on globally merged candidates. L1 pagination and GroupBy candidate selection depend on how segments are distributed across workers. Add QueryNode regression tests for worker-local L1 pagination and GroupBy candidate selection before L1 rerank. issue: #53737 Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
|
5ddad821c9 |
test: add V2 projection data-integrity case (#53307) (#53631)
issue: #53307 ## Summary - Add an independent L3 Storage V2 vector-projection workload: persist deterministic PK-bound narrow and wide vectors, verify the full dataset, drop the binary-vector field with schema bump disabled, and require an adopted MixCompaction rewrite before checking every retained cell. - Reuse the existing exclusive configuration fixture, lineage/serving checkpoints, full-retrieve fences, canonical byte comparisons, and cleanup; leave ordinary workloads, server code, and shared CI runners unchanged. - Pin both PyMilvus requirement entries to `3.1.0rc89`, whose published wheel includes `list_segments`, `list_compaction_tasks`, `list_loaded_segments`, and the required public enums. - Check dropped-index absence by field ownership (`list_indexes(..., field_name=dropped_field)`), with regression coverage for same-name and custom-name stale indexes and an unrelated index sharing the dropped field's name. - Cover deterministic payloads, row-preserving vector corruption, verification ordering, and configuration restoration across success and failure paths in the explicitly collected helper unit-test module; unequal field widths are adversarial inputs, not proof of a particular native reader refill. ## Verification - Isolated published `pymilvus==3.1.0rc89`, not the local editable SDK: 163 helper/oracle unit tests and 50 configuration/reporting/runner tests passed. - All 8 dedicated L3 workloads collect successfully; server-backed E2E was not rerun for this medium-mode update. - Negative control: restoring the old name-only assertion in memory causes 6 expected unit-test failures. - Scoped Ruff checks, Ruff format checks, and `git diff --check` passed. Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Codex <noreply@openai.com> |
||
|
zhuwenxing
|
3cd8755ee2 |
test: update snapshot restore alias conflict expectation (#53682)
After #53547, restoring a snapshot to an existing alias is rejected during restore admission with code 1100 and an "already exists in database" message. The alias-conflict test still expected the later CreateCollection error. Update `test_restore_target_name_equals_existing_alias_fails` to match the admission error, explicitly assert code 1100, and correct its path documentation. Preserve the snapshot check and subsequent restore to a fresh target. Validation: - Reproduced the original assertion failure against Milvus commit `09d3618c818` with PyMilvus `3.1.0rc83`. - Targeted E2E on the same instance: 3 passed (alias target, existing collection target, and failed-restore resource cleanup). - Ruff lint/format, Python compilation, and `git diff --check` passed. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
|
eb7e63ebcf |
test: add Spark-Milvus backfill E2E coverage (#52155)
## What this PR does Adds dedicated Spark-Milvus Backfill E2E coverage and the runtime tooling needed to execute it against real Storage V3 and Storage V2 segments. - Adds opt-in, serial-only pytest coverage for Backfill modes, null semantics, snapshot boundaries, schema fencing, manifest versions, compaction, and invalid inputs. - Adds Kubernetes Job and reusable Spark Toolbox runners with credential redaction, RBAC validation, evidence collection, and cleanup. - Builds `zilliztech/spark-milvus` in the toolbox pod and runs Spark locally through the validated wrapper and native-library runtime. - Separates Storage V3 and Storage V2 execution through pytest markers and independent Milvus deployments. - Uses the dedicated `SparkBackfill` tag, so the normal `L0 L1 L2 ClusterOnly` nightly selection does not include this suite. - Documents manual toolbox deployment and dedicated-suite execution. Fixes #52130 ## Pipeline isolation The suite is ignored during ordinary Python client collection unless `--run-spark-backfill` is explicitly provided. The dedicated pipeline additionally selects `--tags SparkBackfill`; existing PR, nightly, and weekly tag sets therefore select zero Spark Backfill cases. V3 and V2 remain separate marker selections because they require different Milvus storage configurations and must run serially without pytest-xdist. ## Test plan - [x] Dedicated Spark Backfill pipeline completed successfully. - [x] Storage V3: 30 cases collected; 26 passed and 4 strict xfails track the known schema-fencing gap in #51318. - [x] Dedicated `SparkBackfill` tag collects 31 cases: 30 V3 and 1 V2. - [x] Normal nightly tags `L0 L1 L2 ClusterOnly` select 0 Spark Backfill cases, even when the opt-in collection gate is opened. - [x] `ruff check tests/python_client/common/common_type.py tests/python_client/conftest.py tests/python_client/spark_backfill` - [x] `git diff upstream/master...HEAD --check` ## Operational notes - Spark execution uses `spark-submit --master local[2]` in one toolbox pod. - The toolbox is discovered by `app=spark-milvus-toolbox` and is managed outside pytest. - V3 and V2 must use independent Milvus deployments. --------- Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
zhuwenxing
|
fda7d59e39 |
test: fix missing primary key expectation for AutoID partition upsert (#53611)
AutoID collections support Upsert, but callers must supply a primary key for lookup. `test_partition_upsert_with_auto_id` omits that key, so PyMilvus rejects the request before sending an Upsert RPC. The expectation restored in #53158 incorrectly matched the server-side error. Expect SDK error code `1` and the missing-primary-key message, preserving the existing negative test. Validation: - Ruff lint/format, Python compilation, and pytest collection passed. - Targeted E2E passed with PyMilvus `3.1.0rc83` and Milvus commit `09d3618c818`: the corrected missing-key test passed; two diagnostic cases confirmed Int64/VarChar AutoID Upsert preserves existing IDs, allocates IDs for missing lookup keys, and supports partial updates. Diagnostic cases are not included in this PR. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhikunyao
|
ed7c7ff0d6 |
fix: initialize Docker bind mount ownership (#53549)
issue: #51221 related: #49775 ## What this changes - Add a one-shot volume initializer to the CPU and GPU standalone Compose files. - Resolve the numeric UID/GID from the same Milvus image and migrate `/var/lib/milvus` before the non-root server starts. - Write a UID/GID-specific marker only after a successful recursive migration, so large existing volumes are not scanned on every start and interrupted migrations remain retryable. - Prepare the bind mount in `standalone_embed.sh`, including when a previously created container is stopped after a permission failure. - Teach Docker deployment health checks to exclude one-shot services. - Document the ownership and restore contract. The initializer has no network, uses a read-only root filesystem, and mounts only the Milvus data directory. The Milvus server continues to run as `milvus:milvus`. This fixes both fresh rootful-Docker installations, where Docker creates the bind source as `root:root`, and upgrades from root-running images with root-owned data or cache files. It avoids the ineffective image-layer `chmod` approach from #51226 because a bind mount hides image-layer ownership and mode. ## Compatibility - The recursive ownership migration runs once for each image UID/GID. - A failed or interrupted migration does not create the marker and is retried. - Root-running older images can still read and write the migrated files on rollback. - After restoring files owned by another user, remove `.milvus-volume-owner-*` before starting Milvus to rerun migration. ## Validation - `git diff --check` - `bash -n scripts/standalone_embed.sh tests/python_client/deploy/utils.sh tests/python_client/deploy/check_healthy.sh` - `docker compose -f deployments/docker/standalone/docker-compose.yml config --quiet` - `docker compose -f deployments/docker/gpu/standalone/docker-compose.yml config --quiet` - Verified the deployment health-count expressions return the three long-running services. - Simulated the existing release bump regexes and confirmed both init/server image references and `MILVUS_IMAGE` are updated. Container-level startup was not run locally because the Docker daemon is unavailable; the Linux deployment CI must exercise the fresh-install and root-owned upgrade paths. --------- Signed-off-by: Zhikun Yao <zhikun.yao@zilliz.com> |
||
|
Chun HanandMrPresent-Han
|
33d4d826f3 |
enhance: expose compaction lifecycle observability (#53308)
issue: #53307 Protocol dependency: milvus-io/milvus-proto#668 (merged). ## What changed - Reuse the existing RPCs to expose retained segment lifecycle metadata and direct compaction sources; retain the existing sealed/loaded segment reporting contract without adding Growing segments to GetQuerySegmentInfo. - Expose collection-scoped retained compaction tasks, with complete task identity, public type/state enums, source/target sets, and failure reasons, while preserving manual job-ID lookup. - Preserve failure reasons across worker cleanup without changing task cleanup or metadata GC semantics. - Add seven **L3-only** data-integrity workloads: fixed V2/V3 Insert, fixed V2/V3 Import, V2 add/drop-field evolution through MixCompaction with schema bump disabled, V3 schema evolution, and V2-to-V3-to-V2 storage transitions. - Validate deterministic full-row/full-field data against an independent oracle, fenced persistence/serving checkpoints, and actual lineage transitions, allowing either automatic or manual compaction. - Provide `tests/scripts/ci_compaction_integrity.sh` as a dedicated, single-invocation, serial entry; leave the shared `tests/scripts/ci_e2e.sh` unchanged from the PR base. - Retain successful audit records only for the dedicated workload marker, preserving ordinary tests' JSON, HTML, and worker-report formats. ## Compatibility and execution boundary - Omitted segment states preserve the existing persistent-state behavior; task and dropped-segment history are subject to existing retention/GC, not a fixed response-size or record-count limit. - These workloads are not added to per-PR E2E or ordinary Nightly pipelines. - The dedicated job must own the deployment/config namespace exclusively, install a compatible PyMilvus build, and provide connection options plus a unique `CI_LOG_PATH`. - The entry enforces `-n 0`, the L3 workload marker, and explicit opt-in, while forwarding connection options, filters, and pytest's exit status. - Insert/Import workloads use three rounds by default; insert load units are batches, not a promise of a physical segment count, and dense vector dimensions are 16. ## Observability semantics and follow-ups - **Segment listing is not an atomic snapshot.** The Proxy first calls `GetSegmentsByStates`, then `GetSegmentInfo`; returned details are filtered again against the requested states (or the legacy default: Flushing, Flushed, Sealed). A segment whose fetched state no longer matches is omitted, and a segment becoming eligible after ID enumeration may be absent. Consumers must not treat a single response as a consistent lifecycle snapshot. - **Existing lookup errors are preserved.** `IncludeUnHealthy` is enabled only when Dropped is explicitly requested. In the default healthy-only path, a selected source becoming Dropped before detail lookup still returns `SegmentNotFound`, as before; that error is not converted into a successful partial list. A history segment being GC'd between the two RPCs can also fail the lookup. - **Aggregate `Completed` means no running tasks, not universal success.** This preserves the existing aggregation semantics: all-failed/all-timeout retained tasks, or an empty task set, can report Completed. Inspect each task's `state` and `failure_reason` for its outcome; proving an actual data transition additionally requires lineage, active-set adoption, and serving-set evidence. This PR does not redefine task retry or cleanup behavior. - **History capacity is a follow-up.** Dropped-segment queries enumerate the retained history and fetch its details, while collection-scoped compaction queries return retained tasks across triggers. Neither path currently provides pagination or a fixed count/size cap, so scan, copy, and response costs grow with the retained record count despite retention/GC. Pagination and capacity controls for both paths are deferred; they must preserve the ability to enumerate complete retained history rather than silently truncate lineage or task evidence. ## Previously recorded verification The results below are earlier recorded runs, including the original six-workload suite before the V2 DDL workload was added; the documentation/review-response update did not rerun tests. - 22 server-independent CI boundary tests passed, covering ordinary/special audit output, worker report merging, scoped execution guards, and the dedicated entry's argument/exit-code behavior. - 20 differential checks against the PR-base logging plugin produced byte-identical ordinary worker JSON, report JSON, and HTML across all five result outcomes. - 14 existing DataIntegrity and etcd-helper unit tests passed. - The three original L1 DataIntegrity parameter instances passed with `-n 2 --dist loadgroup` in 59.39 seconds; their merged successful JSON records retained the original metadata-only shape. - Ruff lint/format, shell syntax, and scoped PR-base whitespace checks passed; the shared runner is byte-identical to the PR base. - The new dedicated entry ran all six workloads serially against the existing local cluster: **6 passed in 3373.52 seconds**, with three rounds for each Insert/Import workload and the storage-transition workload, plus five DDL checkpoints over 25,000 rows. - All six successful reports retained their audit evidence (403 structured events), including every expected round/DDL checkpoint, full segment/task snapshots, serving frontiers, and matching expected/retrieved PK digests. ## Design and job entry - [Data-integrity design contract](https://github.com/MrPresent-Han/milvus/blob/codex/compaction-e2e-observability/docs/design-docs/design_docs/compaction_data_integrity_e2e_design.md) - [Dedicated serial entry](https://github.com/MrPresent-Han/milvus/blob/codex/compaction-e2e-observability/tests/scripts/ci_compaction_integrity.sh) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> |
||
|
junjiejiangjjj
|
bf02c32d35 |
enhance: chain supports dynamic field (#53493)
https://github.com/milvus-io/milvus/issues/46565 Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
wei liu
|
f887463b25 |
fix: Serialize snapshot restores targeting the same collection (#53547)
issue: #53513 Concurrent internal restores can both pass Phase 0 and reuse the same target collection through idempotent CreateCollection, producing duplicate copy jobs. Share the existing per-target external restore lock with internal restores and validate target absence under the Phase 0 DDL lock before pinning the source. Hold the target lock through collection creation, job submission and synchronous cleanup, while releasing the DDL lock before nested DDL calls. Normalize an empty database name to `default` so both spellings share the same lock. An existing restore target is now rejected during admission, before the source snapshot is pinned. Its error message changes from "create duplicate collection with different parameters" to "target collection ... already exists in database ..."; the error code remains 1100. Update both existing Python rejection tests to check the new message and explicitly assert the code, since the shared error checker only validates message substrings. Add regression tests for internal/external contention, independent targets, admission failures, rollback retry and registered-broadcast ACK timeouts. Add an L2 Python e2e that starts two restores to the same target and requires one success, one already-exists error, one completed job and matching restored data. ### Validation - Targeted restore Go tests passed with `-tags dynamic,test`, `-gcflags="all=-N -l"` and `-race`. Changed restore functions have 100% statement coverage; this is not whole-package coverage. - `TestMilvusClientSnapshotConcurrency::test_concurrent_restore_same_target` passed with `--count=3`: 3 passed in 33.08s. Each run checks all 256 primary keys, scalar fields and vectors. Test collections were cleaned up. - After rebuilding Go on base `f5475123f2`, the two existing-target rejection tests (`test_snapshot_restore_collection_exist` and `test_snapshot_restore_failure_no_resource_leak`) and the concurrent-target regression passed together: 3 passed in 35.42s. The existing-target tests explicitly verify error code 1100 as well as the admission error message. Independent cleanup verification found no remaining collections. - Ruff lint/format checks and `git diff --check` passed. - Live testing used this worktree's binary in isolated standalone mode with embedded etcd, RocksMQ and local storage. Mixed internal/external contention is unit-tested; the original cluster/S3 topology was not exercised. The lock serializes restore requests within the snapshot manager. It does not add fencing against unrelated CreateCollection/DropCollection operations or coordinator failover. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liu
|
2cbfc53b81 |
feat: preserve primary keys in full AutoID upsert (#53158)
issue: #53157 ## What changed - Full AutoID Upsert shares preparation with Partial Upsert and classifies supplied lookup keys with one PK-only Strong Retrieve using standard Query visibility and TTL handling. It adds no Full-specific fixed MVCC or guarantee-timestamp mechanism. - Existing rows preserve their primary keys, so their delete and insert route to the same VChannel transaction. - Missing rows always receive allocator-owned IDs. No Delete is emitted for a missing lookup key. No insert-on-not-found configuration or new not-found error code is introduced. - Partial Upsert retains master's Strong Query and actual per-channel CAS snapshots. AutoID Partial preserves existing keys and allocates new IDs for missing rows with complete insert fields, retaining them across internal CAS retries. Full Upsert does not collect or bind CAS proofs. - The Go row client retains AutoID lookup keys for Upsert, and the response returns final IDs in request order. - Shared `prepareUpsert` performs Retrieve, PK classification and missing AutoID allocation. Full Upsert returns before Partial-only field merging and CAS processing, preserving the complete request payload and row order. - Missing business PKs are allocated during preparation; internal RowIDs remain initialized in `insertPreExecute`. One PK helper validates keys and rewrites only missing positions in a copy of the PK field. Response IDs also drive Insert routing, without separate routing state. ## Compatibility and scope This preserves the historical create-on-not-found outcome while fixing identity replacement for existing rows. Full AutoID Upsert now depends on the standard Query/load-state path. Retrying a missing lookup key after an ambiguous outcome may create another generated identity; this is not an idempotency mechanism or a cross-VChannel request transaction. No RootCoord behavior, collection property, public protobuf, WAL format, or replication protocol is added. Deployments must use homogeneous Proxy versions during the semantic transition, as described in the design. Design: [Preserve Primary Keys in Full AutoID Upsert](https://github.com/weiliu1031/milvus/blob/refine-autoid-upsert/docs/design-docs/design_docs/20260803-autoid-upsert-primary-key-preservation.md) ## Validation Latest publication (`24e2ae1974`, 2026-09-15): - Rebased onto master `fee4ed4316` without conflicts. Both branch-only commits are equivalent in range-diff; the aggregate patch ID and changed paths are unchanged. - Post-rebase `SKIP_3RDPARTY=1 make milvus` passed in the designated worktree, rebuilding Milvus C++ and Go code while reusing third-party dependencies already built there by the preceding full `make milvus`. - 82 targeted Proxy tests passed after rebase with `-tags dynamic,test`, `-gcflags='all=-N -l'`, `-count=1` and local macOS linker options. Coverage includes shared Full/Partial preparation, PK validation, missing-ID allocation, CAS retries, routing and Query visibility. These mocked tests required no shared-service reset. - Full Proxy golangci-lint 2.11.3 passed with 0 issues. `gofmt` checks and `git diff --check` passed; the worktree was clean before publication. - At the user's direction, full-suite tests, the 99% coverage gate and repository-wide verifiers were skipped. No live E2E or fresh coverage result is claimed for this HEAD. New-head CI remains required. Previous publication (`e6000d8b67`, 2026-09-15; historical): - Rebased onto master `423b8e483d`. Resolved query/upsert conflicts by preserving master's Partial Strong-read/CAS protocol and AutoID fallback, while keeping Full classification on its ordinary visibility path. - The mmap counter collision fix is already upstream and was dropped as a duplicate. No new feature paths were added; the design and tests were aligned with the authorized Partial behavior. - `make build-cpp` passed in the designated worktree. - 56 targeted Proxy tests passed with `-tags dynamic,test`, `-gcflags='all=-N -l'`, `-count=1` and local macOS linker options, covering Full/Partial AutoID, CAS retries, snapshot binding, routing and Query visibility. These mocked tests required no shared-service reset. - Full Proxy golangci-lint 2.11.3 passed with 0 issues; formatter and `git diff --check` passed. - Full-suite tests, coverage and live E2E were not rerun. New-head CI remains required; earlier-head green results do not validate this head. Previous publication (`ffa01a4d1d`, 2026-09-10; historical): - Rebased onto master `13cd0e99f9` without conflicts. All seven commits are equivalent in range-diff; aggregate patch ID and paths are unchanged. - Full merr package tests and the Proxy paramtable subtest passed. - Formatter and git diff checks passed. - Post-rebase Proxy tests and full Proxy lint were attempted but blocked by stale local C++ headers missing master's new initcore C interfaces. Rebuilding this worktree with `make build-cpp` then failed at `LoadAdmissionController.cpp:87,91,131`: returning `future` calls the deleted `folly::coro::Future<LoadAdmissionLease>` copy constructor. That source file is identical to master and is not changed by this PR. No unrelated C++ repair was included. Post-rebase integration package compilation was not reached; Proxy validation remains incomplete. - Mock tests did not require starting or resetting shared services. - Full-suite tests, coverage and live E2E were not run for this publication. New-head CI must still validate the complete branch. Historical validation, not fresh evidence for this HEAD: - Before rebase, the switch-removal candidate passed 19 targeted Proxy tests, full merr tests, the Proxy paramtable subtest, full Proxy lint 2.11.3, formatter checks and HelloMilvus integration package compilation. - Earlier revisions ran targeted CAS/retry/routing tests, Go SDK `TestUpsertAutoIDRows` and `TestUpsertAutoIDMixedBatch`, REST v2 `test_upsert_vector_pk_auto_id` for Int64/VarChar, builds and Python syntax checks. - The prior QueryNode `TestFillOutputFieldsOrdered_NoCMemoryLeak` CI failure was not repaired by this change; its implementation and thresholds remain untouched. New-head CI is needed to establish status. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
junjiejiangjjj
|
c24624e280 |
feat: HybridSearch support l0/l1/l2 rerank chains (#52884)
https://github.com/milvus-io/milvus/issues/46565 doc: docs/design-docs/design_docs/20260818-hybrid-search-function-chain.md Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
wei liu
|
10beedd721 |
enhance: Avoid TimeTick waits in partial update reads (#53336)
issue: #53335 Serial partial updates currently use a fixed MVCC read that can wait for the next TimeTick. Use a fresh Strong query for the first attempt and each safe CAS retry, then bind QueryNode's actual per-channel snapshots to the CAS proof. StreamingNode rejects proofs older than its current WAL tracking lifecycle, including a reopened WAL with the same channel term. Require a nonzero snapshot for every write channel before merging or writing. Existing conflict, retention, and incomplete-transaction guards remain active. All participating Proxy, QueryNode, and StreamingNode instances must implement this protocol; mixed-version rollout is outside this change. Design: [Partial Update Optimistic Lock](docs/design-docs/design_docs/20260601-partial-update-optimistic-lock-design.md). AutoID partial updates now preserve existing PKs and allocate a new AutoID for each missing row with complete insert fields. Each attempt resolves all possible destination terms before one Strong query, then selects the final write channels from its actual snapshots after allocation. Freshly allocated IDs are not queried again. Destination IDs remain stable within internal CAS retries; original fields stay immutable and returned IDs retain input order. Missing source PKs generate no Delete, and successful results return destination IDs in input order. Cross-vchannel transactions remain independent, and this does not add cross-request deduplication. Missing-PK insert failures name the PK and missing required field, without a row index, while retaining the non-retriable parameter error. The design and Python e2e error assertions are synchronized. Earlier validation (before the latest refactor): - Targeted Proxy query/upsert regression tests, QueryNode executed-snapshot tests, and StreamingNode lifecycle-history tests passed locally. - Earlier Strong-read Release build and serial A/B/B/A benchmark (before the AutoID fallback update): batch 16 / 128 / 512 mean latency changed from 199.771 / 199.774 / 199.731 ms to 1.293 / 1.838 / 4.658 ms. Each version/batch has 300 measured requests; 2,160 total requests including warmups succeeded, with full scalar-field and sampled vector-preservation checks. - Benchmark scope: Apple M5, Standalone, RocksMQ, 2 shards, 8,192 rows, 128-dimensional vectors, scalar REPLACE. Both versions used identical local native libraries, including a separate local compiler compatibility adjustment. This is not a distributed or concurrent-load benchmark. - `git diff --check` passed. Distributed WAL failover/fault injection and the full repository test suite were not run. - AutoID fallback: targeted Proxy tests cover Int64/VarChar IDs, absent/existing rows, real message packing with mocked mixed-channel success/CAS failure, stable retry IDs, result order, and missing-field error classification. QueryNode snapshot and StreamingNode lifecycle tests were also rerun on this branch. - Python error assertions were updated and syntax-checked; Python e2e and distributed fault injection were not run for this update. Latest refactor validation: - Updated the design and tests for one query per attempt, stable AutoIDs, and shared PK handling. Added RPC-to-CAS packing cases for distinct channel snapshots, empty destinations, missing/zero snapshots, and fixed namespace routing. - Formatting and diff checks passed. No tests or local builds were run for this update, as requested; earlier test and benchmark results do not validate this refactor. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
lorenzozane
|
2ad9913d41 |
fix: align XGBoost corrupted model assertion with segcore diagnostics (#53401)
issue: #53398 ## Summary The corrupted-XGBoost-UBJ e2e test asserted that `segcoreCode=` never appears in the error message. That contradicts the current error contract: `merr` keeps the original segcore code in the diagnostic, and the code travels the wire directly. Update the assertion to require `segcoreCode=2042` instead of rejecting the decoration. The test still verifies the explicit parse failure, wire code 2042, input-error classification, non-retriable behavior, and successful reranking with the valid model. ## Test Plan - `python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_function_chain.py` - `ruff check --config tests/ruff.toml tests/python_client/milvus_client/test_milvus_client_function_chain.py` - CI on this PR: `ci-v2/e2e-default`, `ci-v2/go-sdk`, and `Ruff (changed files only)` pass. Fixes #53398 Signed-off-by: lorenzozanee <wyz0707@proton.me> |
||
|
|
82bcce8867 |
enhance: classify segcore errors across producers and enforce classification end-to-end (#50768)
## What Consume the producer-owned error classification at the segcore boundary and make the whole C++→Go classification drift-proof, so a segcore error is classified as **input** (caller's fault, non-retriable), **transient** (retriable) or **permanent** (non-retriable) instead of flattening to `UnexpectedError(2001)` or carrying the wrong retry default. Design + tracking: #50903. ## Changes - **T1** — register the storage fallback pair in `pkg/util/merr/segcore.go`: `StorageError(2044)` non-retriable, `StorageTransientError(2045)` retriable. - **T2** — `KnowhereStatusToErrorCode` → a switch with **no `default` + `-Werror=switch`** over the full `knowhere::Status`; add build-path variant `KnowhereBuildStatusToErrorCode` so a build-time OOM / disk read stays **retriable** instead of collapsing into a permanent `IndexBuildError`. - **T3/T4** — `ArrowStatusToErrorCode` delegates to the producer's `milvus_storage::ToSegcoreError` (retires milvus's duplicate mapper); audited and routed **25 storage arrow-status sites** that were collapsing to `2001` through the single mapper (extracted to `storage/StatusToErrorCode.h`), always preserving the arrow sub-code in the message. - **T5** — unmapped-code observability: `UnmappedSegcoreCodeTotal{code}` counter + rate-limited WARN via an observer hook (merr is a leaf package); registered on QueryNode and DataNode. Unknown code degrades to non-retriable, never panics. - **T6** — codegen + compile-time enforcement: a generated `SegcoreCode` type (from milvus-common's `EasyAssert.h`) + an exhaustive `classForCode` switch marked `//exhaustive:enforce`, with the `exhaustive` golangci-lint enabled opt-in — a new C++ code that is not classified fails lint (the C++→Go analog of `-Werror=switch`). - **§3 B-tier** — classify `marisa` and `simdjson` errors (build/load/parse) instead of collapsing to `2001`, sub-code in the message; simdjson optional-access (`NO_SUCH_FIELD`/`INCORRECT_TYPE`) stays a benign skip; the `loon_ffi` FFI boundary is untouched. - **Boundary hardening (adversarial self-review of this PR's own diff)** — closed the escapes that would defeat the mapping above: a `throw e;` slicing rethrow in `LoadWithStrategy` that destroyed the very codes the columnar-read mapping attaches (bare `throw;` now), the same slice in `MinioChunkManager::PreCheck`; `GetCoreMetrics` / `EstimateLoadIndexResource` / init-and-config entry points that could let an exception cross the C ABI and terminate the process; and every remaining extern-C entry that caught only `std::exception` now ends in `catch(...)` via the shared `CGoCatch.h` macros. - **Pin + semantics** — bump `milvus-storage_VERSION` to `11f8a36` (the milvus-io/milvus-storage#574 merge, which also contains #575) and align the no-detail `IOError` expectation with the settled semantics: the producer tags every known-transient failure with a retryable `ExtendStatusDetail`, so a bare `IOError` with no detail is unclassified and deliberately falls back to permanent `StorageError(2044)` — a stripped-detail NotFound now degrades to non-retriable (safe) instead of retriable (retry storm on a permanent 404). - **Wire pass-through (client-visible)** — a segcore error now reaches the client with its ORIGINAL code (2009 stays 2009, 2024 stays 2024) instead of collapsing to the `ErrSegcore(2000)` umbrella with the real code buried in the message. Family identity for `errors.Is` is preserved via inner/Unwrap; input/system/retriable classification unchanged. Guardrails: only in-band (2000-2099) codes pass through (garbage still collapses to 2000); cross-family mappings (2046 → wire 110) keep their sentinel's code. `ErrSegcoreUnsupported`/`ErrSegcorePretendFinished` move to the C++ values they represent (2001→2003, 2002→2033) — their old numbers squatted on C++ UnexpectedError/NotImplemented and would false-match under code-based `errors.Is`. Verified end-to-end on a live standalone (ef<k reaches the client as 2042, unsupported tokenizer as 2001); the three e2e assertions pinning the old 2000 updated. - **Remaining code-destroying sites** — the three classes that still swallowed a producer's classification before the cgo boundary are now gone from `internal/core/src` and `internal/core/thirdparty`: status-consuming `AssertInfo` (104 → 0, incl. ~47 arrow builder paths whose commonest failure is OOM, now retriable `MemAllocateFailed` instead of a permanent 2001), bare `throw std::runtime_error/logic_error/bad_alloc` (68 → 0 — these were not `SegcoreError`, so they collapsed to 2001 *and* falsely fired the untyped-exception observer), and `throw fmt::format(...)` (12 → 0 — it throws a `std::string`, which `catch (std::exception&)` cannot see at all). tantivy's 73 `AssertInfo(res.result_->success, ...)` (plus 10 raw-`RustResult` stragglers found later) now classify the rust error — originally by its Display prefix, since replaced by a proper `#[repr(i32)]` discriminant carried in `RustResult.error_code` (see the Aug-10 update below). Typed `ThrowInfo` sites: 894 → 1081. The ~1500 genuine invariant asserts are untouched — 2001 is correct for them. The long-standing FIXME about `err_code` not surviving the nested LOON FFI boundary is also resolved, delegating to `milvus_storage::ToSegcoreErrorCode` rather than duplicating its table. ## Verification **Verified in this PR:** - **Mapping correctness (unit-tested, in-process):** `test_knowhere_status_mapping.cpp` / `test_storage_error_code.cpp` / `test_exec.cpp` cover every mapper branch (knowhere Status incl. the build variant, arrow/extend status incl. `AwsErrorNotFound→ObjectNotExist(2017)`, permanent-S3 vs transient), plus `FailureCStatus` code preservation and both observer hooks firing. - **Code projection to Go (one hop, unit-tested):** `segcore_test.go` pins `classForCode` for every generated code and asserts `merr.Status(err).GetRetriable()` for transient codes; the T6 generator is idempotent and the `exhaustive` lint fails on an unclassified code. - **Full C++ suite:** 8213/8223 unit tests pass locally (10 skipped; Azure connectivity tests excluded), 8648 in CI, rebased on current master (one pre-existing, unrelated concurrency test excluded: `GrowingConcurrentReopenTest` deadlocks deterministically on current master with or without this PR — rwlock writer starvation in growing-segment reopen code this PR does not touch; reported separately). - **Static audit (grep-verifiable):** every storage arrow-status consumption site on the read path routes through `ArrowStatusToErrorCode`, and every extern-C boundary ends in a `catch(...)` tail. **Explicitly NOT verified here (follow-up):** - **Runtime fault injection.** No S3 throttle / 404 / OOM / corrupt-file failure has been triggered end-to-end in a running cluster. Transient codes reach Go with `retriable=true` (unit-tested projection), but the downstream consumption — `lb_policy` replica reroute on `merr.IsRetryableErr`, index/analyze scheduler retry — is pre-existing logic from #50221 and has **not** been driven by a real segcore transient error in this PR. This PR preserves classification for observability and correct retry defaults; the retry behavior itself is exercised only by its own pre-existing tests. ## Dependencies - ~~milvus-common `StorageTransientError(2045)` — zilliztech/milvus-common#102~~ **merged**. - ~~milvus-storage `ToSegcoreError` / packed `ExtendStatusCode` — milvus-io/milvus-storage#575 + #574~~ **merged; pin bumped in-tree to `11f8a36`**. - ~~knowhere three-way classification — zilliztech/knowhere#1704~~ **merged** (the milvus-side `KnowhereStatusToErrorCode` → thin delegate to knowhere's own `ToSegcoreErrorCode` is a follow-up, gated on a knowhere version bump). - ~~milvus-common untyped-cgo-exception observer — zilliztech/milvus-common#112~~ **merged and released as `1.0.0-1fd1160`; the pin now points at the published package.** All dependencies are in. ## Update (Aug 10) — full-population audit, LOON path, runtime observability The originally deferred FFI/LOON path is now **done on the milvus side**, and the audit was extended from the three grep-able classes to the *entire* 2001-producing population: - **Every remaining 2001 site read.** All 1,517 `AssertInfo` (four sweeps: errno fingerprint, failure-keyword messages, condition morphology, and finally **data provenance** — does the guarded value come from disk/network?) and all 198 explicit `ThrowInfo(UnexpectedError)` sites. ~290 were externally-triggerable and now carry typed codes: file/remote IO -> `FileOpen/Create/Read/WriteFailed` (retriable), mmap/allocation -> `MmapError`/`MemAllocateFailed` (retriable), persisted-format damage (CRC/magic/parquet meta/index-meta keys) -> `DataFormatBroken`, deployment config -> `ConfigInvalid`, request content -> `InvalidParameter`, a cancel-race -> `FollyCancel`. The ~1,400 kept sites are genuine invariants or cgo contracts where 2001 is the correct report. - **Two infinite-retry bugs.** Statically-impossible conditions (index_type x metric blacklist, per-type metric allowlists, json/geometry index gates) threw 2001 -> generic retry -> the build task spun forever; they now throw `Unsupported`, which `getStateFromError` maps to a terminal `JobStateFailed`. Missing `index_type`/`metric_type`/`min_gram`/`max_gram` keys in persisted index meta had the same loop on the load path; they are `DataFormatBroken` now. - **knowhere `expected<>` bypasses closed** (8 sites in `QueryResult.h`/`CachedSearchIterator`): iterator failures went through `AssertInfo` and discarded the Status knowhere had already classified; they now route through `KnowhereStatusToErrorCode`, so an OOM/disk failure during search iteration stays retriable. Preflight rewraps in `segment_c`/`boost_score` similarly preserved the original `SegcoreError` code instead of flattening to 2001+string. - **tantivy discriminant over the FFI.** `RustResult` now carries `error_code` (`#[repr(i32)] TantivyBindingErrorCode`, cbindgen-exported); the C++ mapper switches on the enum instead of parsing the Display text, and the inner `tantivy::TantivyError` is discriminated too (`IoError/Open*Error` -> Io/retriable, `DataCorruption/IncompatibleIndex` -> DataCorruption). Wording changes on the rust side can no longer silently degrade classification. - **LOON / FFI path (the deferred item), milvus side complete.** The Go funnel `HandleLoonFFIResult` dropped `err_code` entirely and wrapped every failure as `ErrLoonTransient` — a 404/access-denied/corrupt-data retried as transient. It now classifies by the producer's own `loon_ffi_is_retryable_errcode`; permanent failures carry the new `ErrLoonPermanent` and terminate retry loops (`pack_writer_v3` via `retry.Unrecoverable`; the external-refresh manager guard extended so behavior does not invert). On the C++ side `LoonErrCodeToErrorCode` is the single classification entry (low band -> hand table, extend band -> producer's `ToSegcoreErrorCode`, unknown -> producer's retryable probe), unifying the two previously-divergent `ThrowIfFFIError` helpers — `LOON_FILE_NOT_FOUND(12)` now converges to `ObjectNotExist(2017)` on both integration paths. Remaining LOON items (e.g. promoting FileNotFound into `ExtendStatusCode`) live in the milvus-storage repo. - **Regression guards.** `scripts/check_segcore_error_boundaries.sh` wired into `make static-check`: every `throw` in `internal/core/src` must carry a milvus ErrorCode (zero-tolerance; currently 0 violations); vendored `fmindex::` is confined to its boundary files; knowhere/arrow/milvus_storage/tantivy are ratcheted by a checked-in file-set baseline (new consumer files fail the check; shrinking is free). - **Runtime observability for what is left.** `milvus_cgo_unexpected_segcore_origin_total{origin="<file>:<line>"}` counts every 2001 crossing the cgo boundary by its C++ source location (parsed from the ` at file:line` suffix `AssertInfo` already emits, build paths collapsed to repo-relative). A site that fires in production names itself — reclassification becomes evidence-driven instead of re-reading ~1,400 asserts. Site count for the 2001 family: 1,955 on master -> 1,525 on this branch; the delta is reclassification into actionable codes, not deletion of checks. ## Deferred - milvus-storage-side LOON improvements: promote `LOON_FILE_NOT_FOUND` into `ExtendStatusCode`, category byte (design §4.7) — tracked in the storage repo. - knowhere-side: thin-delegate `KnowhereStatusToErrorCode` to knowhere's own `ToSegcoreErrorCode`, gated on a knowhere version bump. issue: #50903 --------- Signed-off-by: Zack <noreply@zilliz.com> Co-authored-by: Zack <noreply@zilliz.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> |
||
|
|
48a5f578c0 |
test: update client cases for storage and sdk changes (#53305)
issue: #51544 Reopens #51755, whose author has left the project. The branch is rebased onto current master and the conflicts are resolved. ## What changed - Derive the default Python client test namespace from `MILVUS_NS` or `MILVUS_HELM_NAMESPACE` before falling back to `chaos-testing`. - Add an optional external-source MinIO endpoint override (`MILVUS_EXTERNAL_SOURCE_MINIO_HOST`) for external table tests, so the client upload endpoint and the server-side `external_source` endpoint can be configured independently. - Resolve the bulk insert MinIO endpoint from the existing `minio_host` / `minio_bucket` fixtures instead of looking up the MinIO pod IP through the k8s API, so these cases no longer require in-cluster access. - Extend TEXT LOB layout validation to read Vortex segment data files in addition to the existing parquet layout. The Vortex files are decoded with `vortex-data`; when the installed reader cannot decode Milvus StorageV3's custom encoding the case skips rather than asserting on synthesized references. ## Dropped from the original PR Two changes from #51755 have since landed on master independently and are not included here: - The multi-primary-key invalid schema assertion — master already validates at `add_field` time, via the framework's `CheckTasks.err_res` helper rather than `pytest.raises`. - The `Date` / `Time` / `Decimal` entries in the external table add-field matrix. ## Verification - `test_text_lob_bulk_import_json_csv_parquet` - `test_text_lob_inline_vs_lob_object_layout` - `test_milvus_client_external_table_add_all_supported_field_types` - `test_milvus_client_xgboost_supports_all_numeric_feature_types` Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: nico <cheng.yuan@zilliz.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
zhuwenxing
|
dd75544f37 |
test: fix external refresh and oversized insert expectations (#53148)
issue: #53133 issue: #52648 ## What was changed - Handle both valid outcomes of two back-to-back external collection refresh requests: - reject or reuse the request while the first job is active; - allow a new job only after the first job reaches a terminal state. - Update the oversized insert test to verify the current application-level `maxInsertSize` rejection. - Assert error code `1102` and the stable `exceeds maxInsertSize` message instead of the obsolete gRPC `message larger than max` error. ## Why The external refresh test assumed that the first job would always remain active until the second request arrived. On faster environments, the first job can complete before the second request, making a new refresh job valid. The oversized insert test assumed a 64 MiB gRPC receive limit. The Proxy receive limit is now 128 MiB, while oversized materialized inserts are rejected by the 64 MiB `quotaAndLimits.limits.maxInsertSize` guard. ## Verification - `ruff check` passed for both changed files. - `ruff format --check` passed for both changed files. - Python compilation checks passed. - External refresh regression E2E: passed. - Oversized insert regression E2E: passed with error code `1102`. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
yanliang567
|
3067e2542d |
test: add Go, PyMilvus, and REST e2e for FMINDEX and membership_match (#52629)
## Summary Add end-to-end coverage for FMINDEX and the unified `membership_match` expression across the Go SDK, PyMilvus, and RESTful v2. Related issues: #51375, #52778, #53036 ## Coverage ### FMINDEX - Build parameter defaults, boundaries, combined parameters, and invalid types or ranges. - VARCHAR-only validation, including JSON, scalar, and struct-array sub-fields. - Prefix, infix, suffix, general LIKE recheck, regex fallback, escaped wildcards, NULL, and UTF-8 behavior. - Long-text low-hit fixtures and sealed segments above `minSegmentNumRowsToEnableIndex`, preventing fake-finished index coverage. - Exact indexed-field versus unindexed twin-field PK comparison. - Growing plus sealed visibility, mutation, upsert, named-index drop, and rebuild lifecycle checks. - Query and Search coverage with exact projected fields, distance ordering, index identity, type, and row-progress assertions. - VARCHAR AUTOINDEX resolution is verified against the configured scalar-index policy without hard-coding a specific index type. ### membership_match - Bloom and Roaring wire compatibility through a shared static manifest while each SDK keeps independent collection setup. - Bloom zero-false-negative checks plus a statistically bounded false-positive check tied to the configured FPR; Roaring uses exact PK ground truth. - Query and Search must return the same complete Bloom-filtered PK set when topK covers the fixture. - Integer widths, negative values, INT64 boundaries, VARCHAR, JSON strict typing, nested paths, dynamic fields, NULL semantics, and unsupported top-level BOOL/FLOAT fields. - Positive, negated, empty, malformed, domain-mismatch, delete, growing plus sealed, and struct-array rejection cases. - Explicit `STL_SORT`, `INVERTED`, `BITMAP`, and `TRIE` scalar-index paths. - Internal-only `HYBRID` paths are exercised through the supported `AUTOINDEX` request and verified against DataCoord's resolved build parameters for primitive and JSON-path indexes. - Mutation tests use isolated collections; shared read-only fixtures register cleanup before setup begins. ### RESTful v2 - FMINDEX anchored LIKE Query and Search compared with exact PK ground truth and an unindexed twin field. - Index identity, type, persisted parameters, row progress, completion, and collection load state are verified before assertions. - Search validates exact projected values by primary key. - JSON string and missing membership template parameters are verified as server-side validation failures. - Python `bytes` are verified to fail local JSON serialization before HTTP. A positive REST binary-template case remains blocked by #53036. ## Validation - `go test -tags dynamic,test -gcflags='all=-N -l' -c ./testcases` - `go vet -tags dynamic,test ./testcases` - `go test ./membership/sbbf -count=1` - Ruff check and format check for all changed Python and REST files - Python byte-compilation for all changed Python and REST files - Pytest collection: 68 PyMilvus/FMINDEX tests and 7 REST tests - Cross-SDK membership wire-manifest verification - `git diff --check` The Proxy parameter-resolution unit test and Bloom integration package cannot execute on the host because it lacks the native `milvus_core`, `rdkafka`, and `rocksdb` pkg-config packages. A targeted PyMilvus FMINDEX run reached index creation but timed out after 120 seconds before assertions; its temporary collection was cleaned up. Full server-backed E2E was not completed. Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
zhuwenxing
|
87058e5dc7 |
test: stabilize TTL E2E cases under slow CI (#53175)
issue: #53171 ## What this PR does - Preserve collection TTL expiration coverage while measuring from before the first write with a monotonic deadline. - Verify newly inserted rows remain visible under the original short collection TTL before extending it for slower multi-RPC checks. - Replace setup-sensitive TTL windows in timezone-offset and partition cases with explicit expired and unexpired timestamps. - Preserve real-time entity expiration after release and reload by starting the short TTL window after reload completes. - Use monotonic polling with attempt and elapsed-time diagnostics. ## How was it tested? - Ruff check and format check passed. - Python compilation check passed. - The complete TTL test file passed with pytest-xdist using 6 workers and load-group scheduling: 37 passed. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
|
32b08caeb5 |
fix: fail invalid milvus-table refresh requests (#52304)
issue: #52294 ## Summary - add PyMilvus E2E coverage for using Milvus snapshots as milvus-table external sources - classify invalid snapshot metadata paths as input errors so refresh jobs fail terminally instead of remaining retryable - verify real and virtual primary keys, deletes, refreshes, schema alignment, mixed data types, and failure atomicity - mark the target-owned BM25 output case as a strict xfail pending #52303 ## Design document docs/design-docs/design_docs/20260526-milvus-table-external-source.md ## Test plan - [x] python3 -m compileall -q milvus_client/test_milvus_client_snapshot_external.py - [x] python3 -m ruff check milvus_client/test_milvus_client_snapshot_external.py - [x] targeted BM25 E2E reports XFAIL with the #52303 reason - [x] pytest collection discovers all 17 test variants - [ ] targeted Go unit tests were not runnable locally because milvus_core, milvus-storage, and RocksDB pkg-config dependencies are unavailable; CI will run them --------- Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |