0 Fork 0

mirror of https://github.com/milvus-io/milvus.git synced 2026-10-28 17:05:53 +08:00

Files

T

History

Buqian Zheng e14fbcf578 feat: store element-nullable and nested Array fields as native Arrow lists

Redo Array<Array<T>> and element-level null on top of the released formats
instead of per-row proto blobs. Only the two 2.6/3.0 formats are kept as they
are: single-level element-non-nullable scalar Array (proto Binary per row) and
element-non-nullable ArrayOfVector (list<FixedSizeBinary>).

Schema
- element_nullable is allowed only on Array / ArrayOfVector sub-fields of a
  struct array field; top-level element-nullable, nested and vector arrays
  stay rejected. A nested TypeSchema must mirror FieldSchema.nullable at the
  root and element_nullable at the second level; the leaf nullable flag is
  independent. rootcoord enforces the same rules.
- Element-nullable vector arrays reject MAX_SIM metrics and emb-list indexes,
  which are out of scope for this project. Element-level metrics are in
  scope; index and search with them return "not supported yet" until the
  query layer supports element validity, so collection load does not require
  an index on these fields.
- Bulk import, external collections and Storage V1 reject the new formats.

Data contract
- Row validity is written to both FieldData.valid_data and the field-specific
  ScalarField/VectorField.valid_data (legacy wins on read); inner validity
  lives only in the inner message.
- Scalar arrays are dense at every level after the proxy: a null inner array
  is an empty ScalarField of the leaf type, a null leaf is a zero value, and
  non-nullable levels carry no valid_data. Vector arrays stay compact. The
  proxy validates compact input at every nullable level and expands it.

Storage (Go)
- Element-nullable scalar Array -> list<T>, nested Array -> list<list<T>>,
  element-nullable ArrayOfVector -> list<Binary>. Leaf types use native
  widths, list children are named "item" and carry per-level nullable flags.
  Serialization never emits a null list with a non-empty child segment.
- RecordBuilder copies lists generically for mix/sort compaction; clustering
  value serde, RecordToInsertData and the zero-copy Arrow reader understand
  the new formats.

Segcore (C++)
- FieldMeta::is_native_list_array routes new-format scalar fields to
  ColumnarArrayChunk, which gains leaf validity, native int8/int16 widths and
  direct construction from Arrow ListArray with buffer bounds checks; legacy
  fields keep ArrayChunk. Single-level element-nullable arrays get a
  synthesized internal TypeSchema.
- VectorArrayChunk keeps compact vectors and adds per-row logical lengths and
  an element bitmap; list<FixedSizeBinary> and list<Binary> both load into it.
- Growing insert and load, retrieval, take and Arrow export handle the new
  formats. Expressions, scalar index builds, array offsets and element-level
  ANN on element-nullable fields return NotImplemented for now.

Also updates the element-level-null, recursive-array layout and struct design
docs, and adds an integration suite under tests/integration/elementnullable.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>

2026-10-28 07:40:10 +00:00

agent_guides

fix: write manifest index artifacts under the segment _index directory (#54412 )

2026-10-26 20:59:56 +08:00

README.md

Milvus repository documentation

This directory contains documentation that must stay close to the Milvus source tree. Public product documentation belongs in the milvus-io/milvus-docs repository.

Directory map

Path	Purpose	Status
`agent_guides/`	Code-backed subsystem architecture, invariants, and debugging guidance	Current
`dev/`	Contributor-facing engineering rules and development procedures	Current
`design-docs/`	Design records and their supporting assets	Current and historical
`archive/`	Superseded documentation retained for historical context	Historical, non-normative

Documentation that belongs to one component may remain next to that component. Other repository-wide documentation should use one of the directories above.