mirror of
https://github.com/milvus-io/milvus.git
synced 2026-10-28 17:05:53 +08:00
Jeng.Gwan
756825980b
Cherry-pick from master pr: #53334 pr: #54419 pr: #54497 pr: #53706 pr: #54652 issue: #53333 issue: #54418 issue: #54490 issue: #54413 issue: #54512 issue: #53702 issue: #54650 ## Summary Backport of Import V3 reshard (#53334) plus the follow-up fragment-release change (#54419) to 3.0. Import V3 is a sort-based re-shard pipeline: DataCoord packs reshard fragments into per-(vchannel, partition) segment plans and drives them through a checker state machine; DataNode runs each plan's reshard, count-only preimport and merge worker tasks under a node slot scheduler. - `Planning` / `Resharding` job states, BFD fragment packing, per-plan segment assignment and a disk-quota-gated planner. - DataNode `importv3` task framework with reshard, count-only preimport (parquet/numpy footer, JSON/CSV scan) and merge workers, plus a node slot scheduler. - #54419 releases a terminal job's reshard fragments and `import_v3/{job_id}/` prefix as soon as the job is terminal instead of waiting for the final cleanup window. - #54497 makes Import V3 task failures deterministic: malformed sources (truncated `.npy`, header parsing, parquet/JSON/WKT decode errors) become input errors, permanent storage/internal failures are terminal, and retries only continue for genuinely transient errors. It also validates `.npy` declared shape against object size and rewinds/probes headers safely. V3 is behind the one-way rollout gate `dataCoord.import.enableImportV3` (default `false`). Snapshot and L0 imports stay on V2. ## Dependencies - #54416 (two-phase `UpdateImport`) — merged into 3.0. - milvus-proto #678 (backport of milvus-proto #663) adds `msgpb.ImportMsg.version = 10` — merged. The released `v3.0.4` tag predates #678, so all four modules (`go.mod`, `pkg`, `client`, `tests/go_client`) are pinned to the 3.0 branch pseudo-version `github.com/milvus-io/milvus-proto/go-api/v3 v3.0.4-0.20261020180144-0dbac7706ae9`, keeping them in lockstep as master does. Import version handling is identical to master (`ImportMsg.Version`, `body.GetVersion()`). - #54497 needs `packed.ErrLoonPermanent`, introduced by master #50768. #50768 is not backported as a whole; only its storage-side piece is taken here. See "Partial backport of #50768" below. ## Partial backport of #50768 (issue #50903) #54497 depends on `packed.ErrLoonPermanent`, which master introduced in #50768. #50768 (211 files, C++/segcore/Go error classification) is not backported as a whole. Only the storage-side piece required by #54497 is taken: - `internal/storagev2/packed/ffi_common.go`: add `ErrLoonPermanent`; `HandleLoonFFIResult` now classifies by `loon_ffi_is_retryable_errcode` (byte-identical to master). - `internal/datacoord/external_collection_refresh_manager.go`: treat `ErrLoonPermanent` as non-retriable in the external-table explore branch (same line as master). - `internal/flushcommon/syncmgr/pack_writer_v3.go`: comment only; the code already maps non-transient loon errors to `retry.Unrecoverable`. - `internal/storagev2/packed/exttable_test.go`: assertion updated to permanent / not-transient. Not backported from #50768: the C++/segcore producer-side error classification, Talon-specific permanent errors, and other call-site changes. Import V3 only touches Go readers and the storage FFI, so its error handling is identical to master after this partial port. ## Partial backports of #53706 and #54652 - **#53706 (issue #53702), partial.** 3.0 already has the broadcaster-side change (#53807): `broadcasterWithRK` adds the control channel to every broadcast. However, #54416 (backport of #53544) later re-added the old DataCoord-side appends, so the Import/CommitImport/RollbackImport/UpdateImport broadcasts carried the control channel twice (the broadcaster de-duplicated it). This PR takes the remaining DataCoord part of #53706: broadcast to the job's data vchannels only and let the broadcaster add the control channel; tests assert the data vchannels only, matching master. - **#54652 (issue #54650), partial.** The branch is rebased onto the 3.0 commit that contains #54652, so the column-split recovery fix is present. This PR additionally restores the master (#54651) branch in `column_group_recovery.go` that treats both `ErrLoonTransient` and `ErrLoonPermanent` as storage errors, using the `ErrLoonPermanent` sentinel backported from #50768. Without it, a permanent FFI failure while reading the column-group layout would lose the storage wire code. ## Impact when Import V3 is disabled (V2 only) With `dataCoord.import.enableImportV3=false` (default), the V2 import state machine is untouched, but V2 imports share code that #54497 changes: - `importutilv2/{numpy,json,parquet}`: malformed `.npy` headers/shapes, JSON record-boundary errors, and parquet/WKT decode failures now fail fast with input errors instead of being retried or mistaken for EOF. - `hookutil` KMS init failures return `ParameterInvalid` instead of an unclassified status. - malformed binlog headers/events return `DataIntegrity` instead of a raw parse error. - DataNode `CreateTask` rejects malformed V2/V3 task payloads with `ParameterInvalid`, and chunk-manager init failures are mapped to typed IO errors. - `syncmgr` fails fast (`retry.Unrecoverable`) on permanent loon FFI errors; transient ones still retry. ## 3.0 adaptation Only structural differences; the Import V3 logic and version handling are identical to master. - `DataCoordCatalog` lives in the consolidated `internal/metastore/catalog.go` on 3.0 (master uses `datacoord_catalog.go`). The nine V3 methods are added there and the mock is regenerated with mockery v2.53.3. - 3.0 has no DataView manager, so the `dataViewGC` wiring and its tests are omitted. V3's own `importJobAlive` / `import_v3/{job}/` orphan-prefix GC is kept. - 3.0 publishes import visibility through the flusher-driven `HandleCommitVchannel`; master's `CommitByCoordinator` callback rewrite is not part of this change (3.0's `ImportMessageHeader` has no such field). - `data_coord.proto` does not import `view.proto` on 3.0 (the file does not exist in the 3.0 proto branch). - 3.0 removed the RLS implementation (#54353) while retaining the RLS proto APIs. Import V3 keeps the master proto (`ReshardTaskPlan.rls_check_predicate`) but does not set or enforce it: the reshard-source `rlsutil` check, planner propagation and their tests are removed, and `TestExecuteReshardPlanIgnoresRetiredRLSPredicate` pins the ignore behavior. - Master's #50328 (`notifySegmentIndexBuild` helper) is not backported to 3.0. The V3 index notifications use 3.0's existing non-blocking `getBuildIndexChSingleton()` send, which is functionally identical to the helper. - Master-only tests that depend on 3.0-absent features are dropped: nullable array-element storage, data-view projection, and the `CommitByCoordinator` V3 commit test. `tests/integration/import/binlog_test.go` keeps 3.0's `GetCollSegIDs` flush API. The branch is rebased onto `50d8e78693` (the v3.0.4 version bump). Parity check performed against the master PRs: - 71 of the 75 files added by #53334 are byte-identical to master. The 4 differences are: - `import_task_import_v3.go`, `import_task_import_v3_test.go`, `import_v3_handler.go`: master's #50328 `notifySegmentIndexBuild` helper is not backported to 3.0; 3.0 uses its existing non-blocking `getBuildIndexChSingleton()` send. - `import_result.go`: 3.0 uses `updateSegStateAndPrepareMetrics` (its metacache API), master uses `updateSegState` from #53579. - 3.0 removed the RLS implementation (#54353) as master did (#54352); the RLS proto is retained and `import_v3.pb.go` is byte-identical. - the 7 files touched by #54419 are identical except `import_job.go`, which lacks the `GetCommitByCoordinator` interface method because the 3.0 proto has no such field. ## Test Plan - [x] `go build ./internal/...` (against the 3.0 C++ output) - [x] DataCoord import/V3 unit tests: checker V2+V3, planning/reshard/preimport, ID-range gate, terminal fragment GC, commit/abort callbacks - [x] DataNode `importv3` unit tests; `importv2` regression tests (excluding the two pre-existing 3.0 snapshot failures) - [x] storage record reader/writer, binlog reader, `metastore/kv/datacoord` and `kv/binlog`, `flushcommon/syncmgr` pack writer, streamingcoord legacy import forwarding - [x] #54497 coverage: `importutilv2/common|numpy|json|parquet`, `hookutil`, DataCoord V3 task tests, DataNode `importv3` tests - [x] `storagev2/packed` column-group recovery tests (with `-gcflags="all=-N -l"` for mockey) --------- Signed-off-by: xaxys <tpnnghd@163.com>
Compile and install milvus cluster
Environment
OS: Ubuntu 20.04
go：1.21
cmake: >=3.18
gcc: >= 11
Install dependencies
Install compile dependencies
$ sudo apt install -y g++ gcc make libssl-dev zlib1g-dev libboost-regex-dev \
libboost-program-options-dev libboost-system-dev libboost-filesystem-dev \
libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev
$ export GO111MODULE=on
$ go get github.com/golang/protobuf/protoc-gen-go@v1.3.2
OpenBLAS is managed by Conan for C++ core builds. Install the compiler toolchain and Fortran runtime needed by Conan, then run the normal third-party build step.
Compile
Generate the go files from proto file
$ make check-proto-product
Check code specifications
$ make verifiers
Compile milvus
$ make milvus
Install docker-compose
refer: https://docs.docker.com/compose/install/
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
$ docker-compose --version
$ docker compose --version
Start service
Start third-party service:
$ cd [milvus project path]/deployments/docker/cluster
$ docker-compose up -d
$ docker compose up -d
Start milvus cluster:
$ cd [milvus project path]
$ ./scripts/start_cluster.sh
Run unittest
Run all unittest including go and cpp cases:
$ make unittest
You also can run go unittest only:
$ make test-go
Run cpp unittest only:
$ make test-cpp
Run code coverage
Run code coverage including go and cpp:
$ make codecov
You also can run go code coverage only:
$ make codecov-go
Run cpp code coverage only:
$ make codecov-cpp