### Rationale for this change
MemoryView export of `Arrow::PrimitiveArray` didn't support arrays with non-zero offset (sliced arrays). For example, exporting `Arrow::Int32Array.new([0, 1, 2, 3]).slice(1, 3)` returned wrong values because the data pointer was advanced by `offset` bytes instead of `offset * byte width` bytes.
### What changes are included in this PR?
* Use `offset * bit width / 8` as the byte offset of the data pointer. This supports sliced fixed-width arrays.
* Support sliced boolean arrays only when the offset is byte aligned. MemoryView can't represent a bit offset.
* Implement `available_p` for `Arrow::PrimitiveArray`. It returns `false` for arrays that can't be represented by MemoryView: unsupported types and boolean arrays with not byte aligned offset.
### Are these changes tested?
Ran the complete `test-memory-view.rb` file through the native Ruby/Fiddle interface using Ruby 3.2.3, Fiddle 1.1.8, and Arrow C++/GLib 25.0.1.
The original MemoryView implementation in the 25.0.1 release is identical to the current checkout. The extension was rebuilt with this PR's updated `memory-view.cpp`.
- Revised complete test file: 29 tests, 31 assertions, 0 failures, 0 errors, 0 omissions.
- Both new Boolean cases fail against the previous PR implementation and pass after the revision.
- `git diff --check` passes.
The full repository test suite and a complete build of the 26.0.0-SNAPSHOT dependencies were not run.
### Are there any user-facing changes?
* Sliced fixed-width arrays and byte-aligned sliced boolean arrays can be exported by MemoryView.
* `rb_memory_view_available_p()` returns `false` for unsupported arrays.
### Was AI used for this PR?
AI assisted with the investigation, implementation, regression tests, validation, and PR description.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [ ] Human
- [x] AI
- [ ] Not reviewed
* GitHub Issue: #52021
Lead-authored-by: Edmond Abraham <54119952+gitedmond@users.noreply.github.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
GH-50382 requests explicit value constructors for all existing `ArrowFormat::Type` subclasses before the broader builder API is exposed. Fixed-size lists still require callers to manually construct the parent validity buffer and flattened child array.
This is the focused fixed-size-list prerequisite requested in the GH-50382 maintainer discussion. It is distinct from variable-size `ListArray` support in GH-51262.
### What changes are included in this PR?
- Add `ArrowFormat::FixedSizeListArray.new(type, values)` while preserving the existing low-level four-argument constructor.
- Build the parent validity bitmap and the child array from nested Ruby values.
- Preserve exactly `type.size` child slots for null parent lists.
- Reject non-null lists whose size differs from the declared fixed size.
- Delegate child construction to the declared child field type.
- Add focused coverage for typed construction, parent and child nulls, invalid sizes, empty input, and the low-level constructor.
### Are these changes tested?
Yes.
```console
RUBYLIB=/opt/homebrew/lib/ruby/gems/4.0.0/gems/red-arrow-25.0.1/lib GI_TYPELIB_PATH=/opt/homebrew/lib/girepository-1.0 bundle exec ruby test/run.rb test-fixed-size-list-array.rb
# 6 tests, 12 assertions, 0 failures, 0 errors
RUBYLIB=/opt/homebrew/lib/ruby/gems/4.0.0/gems/red-arrow-25.0.1/lib GI_TYPELIB_PATH=/opt/homebrew/lib/girepository-1.0 bundle exec rake test
# 696 tests, 705 assertions, 0 failures, 0 errors
```
I also wrote an Arrow IPC file containing `[[1, 2], nil, [3, nil]]` with the new constructor and loaded it with the native Arrow reader; the values round-tripped unchanged.
The tests use the current `red-arrow-format` sources with the locally installed Arrow 25.0.1 native extension. Upstream CI remains authoritative for the matching main-branch native runtime.
### Are there any user-facing changes?
Yes. `ArrowFormat::FixedSizeListArray` gains a two-argument values constructor. The existing low-level constructor remains supported.
### AI assistance disclosure
OpenAI Codex assisted with issue research, implementation, test generation, validation commands, and drafting this pull request. The submitted behavior is supported by the focused regression, full package suite, and IPC round-trip results above.
* GitHub Issue: #51273
Lead-authored-by: Yifan Chen <emecii23@gmail.com>
Co-authored-by: Yifan Chen <30335308+emecii@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`ArrowFormat::Date32Array` currently stores `Date#day`, the day of the month, for Ruby `Date` values. Arrow Date32 values are signed day offsets from the Unix epoch, so this can silently produce incorrect data. The same conversion is used by `ArrowFormat::Array.build` when it infers Date32.
### What changes are included in this PR?
* Convert Ruby `Date` values from their Julian day to the Unix epoch day offset.
* Cover dates before, at, and after the Unix epoch, including nulls.
* Make the ArrayBuilder regression assert the encoded payload instead of comparing two arrays built through the same conversion path.
### Are these changes tested?
Yes. The full `red-arrow-format` suite passes locally: 691 tests and 695 assertions. The official rake runner used the current pure-Ruby sources with the locally installed Arrow 25.0.1 native extension; the local system does not have the Arrow C++ 26 dependency needed to build the native Ruby extension from `main`.
### Are there any user-facing changes?
Yes. Ruby `Date` values now produce specification-compliant Date32 day offsets.
**This PR contains a "Critical Fix".** It fixes incorrect Date32 data produced from Ruby `Date` values.
### AI assistance
AI assistance was used to investigate the defect, draft the focused implementation and tests, and run the verification described above. No human review is claimed.
* GitHub Issue: #51271
Authored-by: Yifan Chen <30335308+emecii@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
GH-50382 identified list arrays as a prerequisite for a consistent public typed-array builder API. This adds the separately requested list constructor.
### What changes are included in this PR?
* Add `ArrowFormat::ListArray.new(type, values)` while preserving the existing buffer-based constructor.
* Build list validity, offsets, and the child array from nested Ruby values using the declared child type.
* Cover empty lists, null lists, and null child values.
### Are these changes tested?
Yes. The `red-arrow-format` suite passes locally (694 tests, 705 assertions).
### Are there any user-facing changes?
Yes. Callers can construct an `ArrowFormat::ListArray` directly from a list type and nested Ruby values.
### AI assistance
AI assistance drafted the implementation and tests. The final patch was reviewed against the existing typed constructors and low-level list array representation, and the focused and full package tests were run locally.
* GitHub Issue: #51262
Lead-authored-by: Yifan Chen <emecii23@gmail.com>
Co-authored-by: Yifan Chen <30335308+emecii@users.noreply.github.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`ArrowFormat::RecordBatch.new(values)` infers a schema from Ruby values. This adds the corresponding schema-aware constructor so callers can preserve intended Arrow types while building from Ruby rows or columns.
### What changes are included in this PR?
* Add `ArrowFormat::RecordBatch.new(schema, values)` for row- and column-oriented Ruby values.
* Build each column through its schema field type and preserve null values.
* Cover type preservation, supported value forms, inconsistent column lengths, and invalid extra values.
### Are these changes tested?
Yes. The `red-arrow-format` suite passes locally (690 tests, 693 assertions), including the focused record-batch tests. The existing `red-arrow` record-batch suite also passes (21 tests, 21 assertions).
### Are there any user-facing changes?
Yes.
### AI assistance
AI assistance drafted the implementation and tests. The final scope was checked against the existing two-argument `Arrow::RecordBatch.new` contract and the one-argument `ArrowFormat::RecordBatch` API; the validation commands above were run locally.
* GitHub Issue: #50571
Authored-by: Yifan Chen <emecii23@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It looks like [this change](https://github.com/ruby-gnome/ruby-gnome/pull/1743) has fixed issue #50919.
#50868 disabled Arrow Flight test for GLib on macOS.
### What changes are included in this PR?
Re-enable Arrow Flight tests for GLib and Ruby on macOS.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #51039
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building a fixed size binary Arrow array from Ruby objects is convenient.
### What changes are included in this PR?
Accept:
* `ArrowFormat::FixedSizeBinaryArray.new(byte_width, values)`
* `ArrowFormat::FixedSizeBinaryArray.new(type, values)`
`ArrowFormat::FixedSizeBinaryType.try_convert` is added for the byte width shorthand, in the same way as `ArrowFormat::Time32Array.new(unit, values)`. The existing 4 arguments form is kept.
Decimal arrays are out of scope. `ArrowFormat::DecimalArray#to_a` returns `BigDecimal`, so building one from Ruby objects needs a conversion between `BigDecimal` and two's complement. I would like to open a separate issue for it.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50832
Authored-by: otegami <a.s.takuya1026@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's a convenient API to get values as Ruby objects.
### What changes are included in this PR?
* Add `ArrowFormat::RecordBatch#records`
* Add `ArrowFormat::RecordBatch#each_record`
* Add `ArrowFormat::Record`
* Add `ArrowFormat::RecordBatch#find_column`
* Add `ArrowFormat::Schema#[]`
* Ensure using `String` for `ArrowFormat::Field#name`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50572
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's a convenient API to get a value at the given index.
### What changes are included in this PR?
Add `ArrowFormat::Array#[]`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50588
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's convenient that we can build a record batch from Ruby objects.
### What changes are included in this PR?
* Add `ArrowFormat::RecordBatch.new({name1: column1, name2: column2, ...})` # column mode
* Add `ArrowFormat::RecordBatch.new([{name1: value1_1, name2: value2_1}, {name1: value1_2, name2: value2_2}, ...])` # row mode
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50569
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`ArrowFormat::Array.build(values)` is convenient to build an Arrow array from Ruby objects without specifying an Arrow type explicitly.
This is just for convenient. Users who want to specify Arrow type explicitly should use ArrowFormat::XXXArray.new(values) explicitly instead.
### What changes are included in this PR?
* Add `ArrowFromat::ArrayBuilder`
* Add `ArrowFormat::Array.build(values)`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50528
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Comparing `ArrowFormat::DurationType` instances with the same unit should return `true`.
### What changes are included in this PR?
Add `ArrowFormat::DurationType#==`.
### Are these changes tested?
Yes.
Added tests for:
* equal duration types
* different duration types
### Are there any user-facing changes?
Yes.
GitHub Issue: #50456
* GitHub Issue: #50456
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building variable-size binary Arrow arrays from Ruby objects is convenient.
### What changes are included in this PR?
Accept:
* `ArrowFormat::BinaryArray.new(values)`
* `ArrowFormat::LargeBinaryArray.new(values)`
* `ArrowFormat::UTF8Array.new(values)`
* `ArrowFormat::LargeUTF8Array.new(values)`
### Are these changes tested?
Yes.
Added tests for:
* initialization
* mixed values with `nil`
* equality
* sliced arrays
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50439
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Comparing `ArrowFormat::Time32Type` and `ArrowFormat::Time64Type` instances with the same unit should return `true`.
### What changes are included in this PR?
Add `ArrowFormat::TimeType#==`.
### Are these changes tested?
Yes.
Added tests for:
* `Time32Type`
* equal types
* different units
* `Time64Type`
* equal types
* different units
### Are there any user-facing changes?
Yes.
GitHub Issue: #50455
* GitHub Issue: #50455
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building a duration Arrow array from Ruby objects is convenient.
### What changes are included in this PR?
Accept `ArrowFormat::DurationArray.new(unit, values)`.
### Are these changes tested?
Yes.
Added tests for:
* initialization
* mixed values with `nil`
* equality
* sliced arrays
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50438
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building a year month/day time/month day nano interval Arrow array from Ruby objects is convenient.
### What changes are included in this PR?
* Accept `ArrowFormat::YearMonthIntervalArray.new(values)`
* Accept `ArrowFormat::DayTimetntervalArray.new(values)`
* Add `ArrowFormat::DayTimeIntervalArray#each`
* Accept `ArrowFormat::MonthDayNanoIntervalArray.new(values)`
* Add `ArrowFormat::MonthDayNanoIntervalArray#each`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50437
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building a timestamp Arrow array from Ruby objects is convenient.
### What changes are included in this PR?
Accept `ArrowFormat::TimestampArray.new(unit, values)`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50436
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building a time{32,64} Arrow array from Ruby objects is convenient.
### What changes are included in this PR?
Accept `ArrowFormat::Time{32,64}.new(unit, values)`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50435
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Building a date{32,64} Arrow array from Ruby objects is convenient.
### What changes are included in this PR?
Accept `ArrowFormat::Date{32,64}.new(values)`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50434
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Comparing 2 bitmaps/arrays is a common operation. We should provide content based (not identity based) comparition feature.
### What changes are included in this PR?
* Add `ArrowFormat::Bitmap#==`
* Add `ArrowFormat::Bitmap#offset`
* Add `ArrowFormat::Bitmap#size`
* Add `ArrowFormat::Bitmap#length`
* Make `ArrowFormat::Array` enumerable
* Add `ArrowFormat::Array#==`
* Add `ArrowFormat::NullArray#each`
* Add `ArrowFormat::PrimiveArray#each`
* Add `ArrowFormat::BooleanArray#each`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50432
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We did drop support for Ubuntu 20.04 almost a year ago. We should remove cases that cater for it.
### What changes are included in this PR?
Found a couple places where we still seem to be doing some minor fixes for Ubuntu 20.04. Clean those up.
### Are these changes tested?
CI has been run
### Are there any user-facing changes?
No
* GitHub Issue: #50412
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`gaflight_server_listen()` may use options' auth handler later. So we should refer the given options while the server is alive. If we don't refer the given options, we may touch freed auth handler later.
The argument of `ArrowFlight::Criteria.new`/`ArrowFlight::Ticket.new` should be referred to protect from GC in Ruby.
I could fix some problems by `GC.stress = true` but I couldn't fix some problems that are reproduced only on GitHub Actions. I omit tests for them for now.
### What changes are included in this PR?
* Refer options for `gaflight_server_listen()`
* Refer expression for `ArrowFlight::Criteria.new`
* Refer data for `ArrowFlight::Ticket.new`
* Omit Flight related tests that I can't fix...
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50302
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Int array and float array use similar structure. We can share common code for builder API for them.
### What changes are included in this PR?
Add `ArrowFormat::PrimitiveArray#build_data`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50390
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It can be used for validity bitmap and boolean array.
### What changes are included in this PR?
* Add `ArrowFormat::DenseBitmapBuilder`
* Add `ArrowFormat::SparseBitmapBuilder`
* Add `ArrowFormat::BooleanArray.new(values)`
* Use them in integration test
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #50385
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`rb_memory_view_get()` callers may pass a non-zero-initialized `rb_memory_view_t`. `primitive_array_get()` and `buffer_get()` did not initialize `item_desc.components` and `item_desc.length`, which could cause `rb_memory_view_release()` to attempt to free an invalid pointer and abort the process.
### What changes are included in this PR?
This change initializes `item_desc.components` and `item_desc.length` in both `primitive_array_get()` and `buffer_get()`.
It also adds regression tests that verify releasing a memory view with a non-zero-initialized `rb_memory_view_t` does not crash for both `Arrow::Int32Array` and `Arrow::Buffer`.
### Are these changes tested?
Yes. Added regression tests in `test-memory-view.rb` that reproduced the crash before this change and pass after the fix.
### Are there any user-facing changes?
No.
**This PR contains a "Critical Fix".**
This change fixes a crash that could occur when `rb_memory_view_release()` is called with a memory view whose `item_desc` members were not initialized.
* GitHub Issue: #45187
Lead-authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
See #38584 for original PR. Will be quoted for this PR description.
### Rationale for this change
> support multi sortkey nulls first.
```
order by i nulls first, j, k nulls first;
```
> The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.
### What changes are included in this PR?
> 1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
> 2.Substriait related.
> 3.c_glib related.
> 4.SelectK related.
> 5.RankOptions related.
### Are these changes tested?
> yes, I changed the code inside vector_sort_test.cc and performed additional tests.
### Are there any user-facing changes?
> yes, pg database include null sorting of multiple sort keys.
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
I amended the original PR to be less breaking in public APIs.
Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a `std::optional<NullPlacement>` instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. ~I also need some help with fixing the `c_glib` bindings.~
* GitHub Issue: #38558
Lead-authored-by: Alexander Taepper <alexander.taepper@gmail.com>
Co-authored-by: light-city <455954986@qq.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
`Arrow::Table#merge` and `Arrow::RecordBatch#merge` had similar implementations, resulting in duplicated merge logic.
### What changes are included in this PR?
This PR moves the common merge implementation to `Arrow::ColumnContainable`.
Container-specific behavior such as column conversion and creating the merged container remains implemented in `Arrow::Table` and `Arrow::RecordBatch`.
### Are these changes tested?
Yes.
The existing `RecordBatch` and `Table` test suites were run and passed successfully.
### Are there any user-facing changes?
No.
* GitHub Issue: #50225
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This adds `Arrow::RecordBatch#merge` with behavior analogous to
`Arrow::Table#merge`.
### What changes are included in this PR?
- Add `RecordBatch#merge`
- Support merging with a `Hash`
- Support merging with another `RecordBatch`
- Support column addition, replacement, and removal (via `Hash` values of `nil`)
- Add tests covering the new behavior
### Are these changes tested?
Yes. Added unit tests covering:
- add column
- remove column
- replace column
- merge with another RecordBatch
- invalid merge target
* GitHub Issue: #48294
Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
GH-49621 changed dictionary array related API but we didn't follow the API change in integration test code.
### What changes are included in this PR?
Follow dictionary array related API change in integration test code.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #49634
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
`Message` can attach custom metadata: https://github.com/apache/arrow/blob/61ef6722b01a289ca6e09d3c5aaef782f060f94c/format/Message.fbs#L152-L157
### What changes are included in this PR?
* Add `ArrowFormat::Dictionary` for dictionary
* Add support for reading custom metadata in `Message` and attaching them to `ArrowFormat::Schema`, `ArrowFormat::RecordBatch` and `ArrowFormat::Dictionary`
* Add support for writing custom metadata in `ArrowFormat::Schema`, `ArrowFormat::RecordBatch` and `ArrowFormat::Dictionary` as `Message`'s custom metadata
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49620
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
In file format, Footer can have custom metadata.
### What changes are included in this PR?
* Add `garrow_record_batch_file_reader_get_metadata()`
* Add `garrow_record_batch_file_writer_new_full()`
* Add `ArrowFormat::FileReader#metadata`
* Add `metadata` to `ArrowFormat::FileWriter#finish`
* Add `metadata:` to `Arrow::Table#save`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49576
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
There are some missing features in the Ruby implmentaion for now but we can pass them by skipping some tests in our integration tests.
### What changes are included in this PR?
Archery:
* Add `--with-ruby` to `archery integration`
* Add `archery.integration.tester_ruby.RubyTester`
* Add `no_map_field_names_validate` quirk for GH-49415
* Show environment variables too on external command failure because Ruby tester uses environment variables not command line arguments to pass information to integration tester
* Use `ARCHERY_INTEGRATION_WITH_CPP=1` instead of `ARROW_INTEGRATION_CPP=ON` like other implementations such as `ARCHERY_INTEGRATION_WITH_GO`
Ruby:
* Add `red-arrow-format-integration-test` as the test driver
* This is not included in `.gem` because this is only for development
* Add `ruby/red-arrow-format/lib/arrow-format/integration/` as helpers of the test driver
* This is not included in `.gem` because this is only for development
* Add `ArrowFormat::Array#empty?`
* Add `ArrowFormat::RecordBatch#empty?`
* Add `ArrowFormat::NullArray#n_nulls`
* `ArrowFormat::*Array#to_a`: Add support for empty case
* Fix Apache Arrow decimal <-> `BigDecimal` conversion
* `ArrowFormat::Bitmap#each`: Fix a bug that one bit is ignored
* Move dictionary ID to `ArrowFormat::DictionaryType` from `ArrowFormat::Field`
* Add support for V4 union that has validity bitmap
* Add support for no continuation token message for backward compatibility
* `ArrowFormat::StreamingReader`: Add support for reading schema without calling `#each`
* `ArrowFormat::MapType`: Add support for keys sorted
* `ArrowFormat::MapType`: Always use "key"/"value"/"entries" for field names
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49422
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's a fixed size variant of list array.
### What changes are included in this PR?
* Add `ArrowFormat::FixedSizeListType`
* Add `ArrowFormat::FixedSizeListArray`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49406
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
They are optimized C++ values -> Ruby values converters.
### What changes are included in this PR?
* Add support for `arrow::FixedSizeListArray` in `#values` and `#raw_records` implementation in C++
* Add tests for `Arrow::FixedSizeList#values` and `#raw_values`
* Add tests for `Arrow::LargeList#values` and `#raw_values` because they are missing
* This is not so related to `Arrow::FixedSizeList#values` and `#raw_values`. Sorry.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49400
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
We can add custom metadata to `Field` and `Schema`.
### What changes are included in this PR?
* Add `ArrowFormat::Field#metadata`
* Add `ArrowFormat::Schema#metadata`
* Add support for reading and writing custom metadata in `Field` and `Schema`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49389
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The current reader tests use a sub test case per type but we can use a test per type like the current writer tests.
### What changes are included in this PR?
* Create test data in a test not a setup to use a test no a sub test case per type
* Add `ArrowFormat::Type#to_s`
* Add `ArrowFormat::FileReader#schema`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #49364
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
If `gem install red-arrow` installs Apache Arrow GLib automatically, it's convenient for users.
### What changes are included in this PR?
Add a rubygems-requirements-system configuration for Homebrew.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49316
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
extpp 0.1.2 includes C++20 support.
### What changes are included in this PR?
Update required version information of extpp dependency.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #49318
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Nested types and dictionary type are out of scope of this.
### What changes are included in this PR?
* Fix `ArrowFormat::Array#slice_offsets_buffer`
* Add `ArrowFormat::DayTimeIntervalArray#element_size`
* `ArrowFormat::Bitmap#each`: Add support for offset
* Add support for chunked dictionaries
* Add support for `Arrow::DictionaryArray#raw_records` with large binary and large UTF-8 dictionaries
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49225
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
This focuses on implementing base dictionary delta message support mechanism. So this adds support for only UTF-8 array as dictionary. Other arrays will be supported in follow-up tasks.
### What changes are included in this PR?
* Add support for `ArrowFromat#slice` (But it's not completed. It just works partially.)
* If the second record batch includes an updated dictionary (new entries are appended), these appended entries are sliced and they are only written as delta.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49208
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Delta dictionary message support is out of scope.
### What changes are included in this PR?
* Add `ArrowFormat::DictionaryArray#each_buffer`
* Add `ArrowFormat::DictionaryType#build_fb_type`
* Add support for dictionary message in `ArrowFormat::StreamingWriter`
* Add support for writing dictionary message blocks in footer in `ArrowFormat::FileWriter`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #48132
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's a list based array.
### What changes are included in this PR?
* Add `ArrowFormat::MapType#to_flatbuffers`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49119
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
There are dense and sparse variants.
### What changes are included in this PR?
* Add `garrow_union_array_get_n_fields()`
* Add `ArrowFormat::UnionArray#children`
* Add `ArrowFormat::DenseUnionArray#each_buffer`
* Add `ArrowFormat::SparseUnionArray#each_buffer`
* Add `ArrowFormat::UnionType#to_flatbuffers`
* Add `Arrow::UnionArray#fields`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49117
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It has unit parameter.
### What changes are included in this PR?
* Add `ArrowFormat::DurationType#to_flatbuffers`
* Add duration support to `#values` and `raw_records`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49093
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
It's a nested array.
### What changes are included in this PR?
* Add `ArrowFormat::StructType#to_flatbuffers`
* Add `ArrowFormat::StructArray#each_buffer`
* Add `ArrowFormat::StructArray#children`
* Fix `ArrowFormat::Array#n_nulls`
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #49096
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Raúl Cumplido