### Rationale for this change
Require Numpy 2.0 to support newer features, allow newer numpy API/ABI
### What changes are included in this PR?
- Numpy 2.0+ requirement
- Pandas 2.2.2+ requirement (first wheel compatible with numpy 2.0)
- Pyodide 0.27.1 usage (numpy 2.0 api)
### Are there any user-facing changes?
Yes, in contrast to pyarrow 25, after merging this PR pyarrow 26 can't be used with Numpy 1.x or pandas 2.0-2.2.1
**This PR includes breaking changes to public APIs.**
This PR was created using GPT-5.6-Sol-xhigh, every line read & reviewed by me.
* GitHub Issue: #48473
Lead-authored-by: Adam Lippai <adam@rigo.sk>
Co-authored-by: Ádám Lippai <adam@rigo.sk>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Fixes https://github.com/apache/arrow/issues/37333.
### What changes are included in this PR?
- Replaces the calls to `pandas.util.testing.rands` with existing `pyarrow.tests.util.rands`
- Updated benchmarks.rst to remove a mention of the now-deprecated `asv dev` command.
### Are these changes tested?
I tested locally in a freshly-created python venv and a conda environment and the benchmarks run without error.
### Are there any user-facing changes?
No.
* GitHub Issue: #37333
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Failures:
/arrow/python/benchmarks/parquet.py:18:1: F401 'shutil' imported but unused
/arrow/python/benchmarks/parquet.py:19:1: F401 'tempfile' imported but unused
/arrow/python/benchmarks/parquet.py:23:1: F401 'pandas as pd' imported but unused
### What changes are included in this PR?
* Remove unused imports.
* Add python/benchmarks/ to lint targets.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #40439
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Legacy ParquetDataset has been deprecated for a while now, see
https://github.com/apache/arrow/issues/31529. This PR is removing the
legacy implementation from the code.
### What changes are included in this PR?
The PR is removing:
- `ParquetDatasetPiece `
- `ParquetManifest`
- `_ParquetDatasetMetadata `
- `ParquetDataset`
The PR is renaming `_ParquetDatasetV2` to `ParquetDataset` which was
removed. It is also updating the docstrings.
The PR is updating:
- `read_table`
- `write_to_dataset`
The PR is updating all the tests to not use `use_legacy_dataset` keyword
or legacy parametrisation.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Deprecated code is removed.
* Closes: #31303
### Rationale for this change
We can't maintain Plasma.
### What changes are included in this PR?
These changes remove all Plasma related codes.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* Closes: #33243
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
This also adds a benchmark to be able to see how the performance of reading Parquet files with many columns (up to 10,000) scales.
Closes#5653 from wesm/ARROW-6876 and squashes the following commits:
e97c43009 <Wes McKinney> Revert member to unordered_set
0b8d690e7 <Wes McKinney> Add write benchmark, write more rows
1d604339f <Wes McKinney> Use shared_ptr to avoid copying ReaderContext, fix performance regression with reading many columns. Add asv benchmark
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Antoine Pitrou <pitrou@free.fr>
These benchmarks will help track possible regressions in buffered IO performance from the Python side.
This also allows objects that implement the right APIs (at least according to `hasattr`) to be passed into these functions.
Closes#5356 from wesm/ARROW-4220 and squashes the following commits:
0307cd919 <Wes McKinney> Code review
bba2aa62c <Wes McKinney> Add buffered IO benchmarks with simulated high latency
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
I don't really understand why this is faster, though.
before
```
----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
BM_ArrowBinaryDict/EncodeDictDirectInt8/1048576 7334087 ns 7333876 ns 98 136.354M items/s
BM_ArrowBinaryDict/EncodeDictDirectInt16/1048576 7022430 ns 7022412 ns 100 142.401M items/s
BM_ArrowBinaryDict/EncodeDictDirectInt32/1048576 7061033 ns 7060870 ns 99 141.626M items/s
BM_ArrowBinaryDict/EncodeDictDirectInt64/1048576 7084581 ns 7084398 ns 97 141.155M items/s
```
after
```
----------------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------------
BM_ArrowBinaryDict/EncodeDictDirectInt8/1048576 4387151 ns 4387175 ns 156 227.937M items/s
BM_ArrowBinaryDict/EncodeDictDirectInt16/1048576 4446167 ns 4446074 ns 159 224.918M items/s
BM_ArrowBinaryDict/EncodeDictDirectInt32/1048576 4501028 ns 4500934 ns 156 222.176M items/s
BM_ArrowBinaryDict/EncodeDictDirectInt64/1048576 4635792 ns 4635728 ns 150 215.716M items/s
```
On an i9-9960X CPU before these changes perf reported that `__memmove_avx_unaligned_erms` was taking up a lot of time. In principle `std::vector::reserve` should be correct since memory is not initialized, but something weird seems to be going wrong. If anyone has any ideas I'm interested to learn more. In any case I'll stick with the empirical benchmark evidence on this
I started to refactor to use `TypedBufferBuilder<int32_t>` but I'm not sure about the performance of that for scalar appends vs. `std::vector` so I'll leave that for future experimentation.
Closes#5248 from wesm/ARROW-6411 and squashes the following commits:
b1159ec8a <Wes McKinney> Add C++ benchmarks for DictEncoder<T>::PutIndices
da8cc9d79 <Wes McKinney> Add C++ benchmarks
5a73bf509 <Wes McKinney> Add Python benchmark
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Use BufferBuilder and UnsafeAppend to accelerate writes.
Before (prior to ARROW-6381, which came up while investigating this):
```
----------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------
BM_ArrowBinaryPlain/EncodeArrow/262144 6111690 ns 6109829 ns 465 246.06MB/s
BM_ArrowBinaryPlain/EncodeArrow/1048576 30470849 ns 30451048 ns 85 197.296MB/s
BM_ArrowBinaryPlain/EncodeLowLevel/262144 5352838 ns 5352679 ns 514 280.866MB/s
BM_ArrowBinaryPlain/EncodeLowLevel/1048576 29736017 ns 29735036 ns 94 202.047MB/s
```
After
```
----------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------
BM_ArrowBinaryPlain/EncodeArrow/262144 2020914 ns 2020905 ns 1000 743.918MB/s
BM_ArrowBinaryPlain/EncodeArrow/1048576 11596223 ns 11596094 ns 242 518.096MB/s
BM_ArrowBinaryPlain/EncodeLowLevel/262144 2740316 ns 2740256 ns 1021 548.63MB/s
BM_ArrowBinaryPlain/EncodeLowLevel/1048576 17562138 ns 17560763 ns 157 342.12MB/s
```
Dictionary encoding perf is not really affected by this work, so this will mostly affect that case where we fall back to PLAIN encoding when the dictionary grows large.
Closes#5233 from wesm/ARROW-4398 and squashes the following commits:
3a8c37ac9 <Wes McKinney> Code review comments. Don't box string_view as ByteArray
252b45c4c <Wes McKinney> Fix -Wsign-compare error
f77838cac <Wes McKinney> Improve benchmark
f7e659d29 <Wes McKinney> Add ASV benchmarks for binary writes
844427ca3 <Wes McKinney> More perf improvements, add benchmark for low level vs high level
ab2ad26ef <Wes McKinney> Use unsafe appends for BinaryArray for better performance
e6458d2aa <Wes McKinney> Use BufferBuilder in PlainEncoder, use UnsafeAppend to avoid calling Reserve so much
47f10cde7 <Wes McKinney> Add benchmark of direct Arrow write
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
Warning: hold your nose for this one =) I think this can be made cleaner, but I just wanted to triage all the problems and make sure we don't introduce a hard dependency on pandas again
Also resolves ARROW-4794: make pandas an optional dependency
Author: Wes McKinney <wesm+git@apache.org>
Closes#3893 from wesm/ARROW-4637 and squashes the following commits:
3c353b6b <Wes McKinney> do not override orc global mark in test_orc.py
59fdf8ab <Wes McKinney> Tweak pyarrow._orc import to see if it fixes MSVC
747c8937 <Wes McKinney> Address Python 2.7 unicode interaction issue with Cython
4d94b2e1 <Wes McKinney> Add benchmark from ARROW-4629
f7bf7741 <Wes McKinney> Cythonize pandas API shim for better performance
4a2549aa <Wes McKinney> Remove TF testing from travis_script_manylinux1.sh
5061c6e1 <Wes McKinney> Do not require pandas to run unit tests
385cfe55 <Wes McKinney> Finish pandas API shim; do not eagerly import pandas, add to CI
804587a0 <Wes McKinney> add import test script
bb0240e2 <Wes McKinney> Begin to refactor to make references to pandas more centralized and lazy
This ended up being much more difficult than anticipated due to the spaghetti-like state (as the result of many hacks) of pyarrow/pandas_compat.py.
This is partly a performance and memory use optimization. It has consequences, though, namely tables will have some index data discarded when concatenated from multiple pandas DataFrame objects that were converted to Arrow. I think this is OK, though, since the preservation of pandas indexes is generally something that's handled at the granularity of a single DataFrame. One always has the option of calling `reset_index` to convert a RangeIndex if that's what is desired.
This patch also implements proposed extensions to the serialized pandas metadata to accommodate indexes-as-columns vs. indexes-represented-as-metadata, as described in
https://github.com/pandas-dev/pandas/issues/25672
Author: Wes McKinney <wesm+git@apache.org>
Closes#3868 from wesm/ARROW-1639 and squashes the following commits:
ec929aebd <Wes McKinney> Add pandas_metadata attribute to pyarrow.Schema to make interactions simpler
670dc6f70 <Wes McKinney> Add compatibility tests for pre-0.13 metadata. Add Arrow version to pandas metadata
0ca1bfc58 <Wes McKinney> Add benchmark
9ba413106 <Wes McKinney> Serialize RangeIndex as metadata via Table.from_pandas instead of converting to data column. This affects serialize_pandas and writing to Parquet format
Author: Antoine Pitrou <antoine@python.org>
Closes#3650 from pitrou/ARROW-4576-py-benchmarks-fix and squashes the following commits:
67e8069d1 <Antoine Pitrou> ARROW-4576: Fix error during benchmarks
"conda activate" unfortunately isn't available from a non-interactive shell, and running bash as interactive doesn't look like a workable solution.
Also fix a setup slowness issue in the Parquet benchmarks, and fix a C++ ABI issue by downloading packages from Anaconda rather than conda-forge.
Author: Antoine Pitrou <antoine@python.org>
Closes#3357 from pitrou/ARROW-4118-fix-asv-run and squashes the following commits:
b07b68e61 <Antoine Pitrou> ARROW-4118: Fix benchmark setup for "asv run"
I would guess I'm the first person to try to run the benchmark suite on Windows!
Author: Wes McKinney <wesm+git@apache.org>
Closes#3271 from wesm/benchmark-no-plasma and squashes the following commits:
c99b76fae <Wes McKinney> flake
c7ede9fd4 <Wes McKinney> Revert whitespace change
4938932d2 <Wes McKinney> Check for ImportError in benchmarks/plasma.py
008ae7b98 <Wes McKinney> Don't fail ASV if Plasma extension is not built (e.g. on Windows)
This adds a `deduplicate_objects` option to all of the `to_pandas` methods. It works with string types, date types (when `date_as_object=True`), and time types.
I also made it so that `ScalarMemoTable` can be used with `string_view`, for more efficient memoization in this case.
I made the default for `deduplicate_objects` is True. When the ratio of unique strings to the length of the array is low, not only does this use drastically less memory, it is also faster. I will write some benchmarks to show where the "crossover point" is when the overhead of hashing makes things slower.
Let's consider a simple case where we have 10,000,000 strings of length 10, but only 1000 unique values:
```
In [50]: import pandas.util.testing as tm
In [51]: unique_values = [tm.rands(10) for i in range(1000)]
In [52]: values = unique_values * 10000
In [53]: arr = pa.array(values)
In [54]: timeit arr.to_pandas()
236 ms ± 1.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [55]: timeit arr.to_pandas(deduplicate_objects=False)
730 ms ± 12.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Almost 3 times faster in this case. The different in memory use is even more drastic
```
In [44]: unique_values = [tm.rands(10) for i in range(1000)]
In [45]: values = unique_values * 10000
In [46]: arr = pa.array(values)
In [49]: %memit result11 = arr.to_pandas()
peak memory: 1505.89 MiB, increment: 76.27 MiB
In [50]: %memit result12 = arr.to_pandas(deduplicate_objects=False)
peak memory: 2202.29 MiB, increment: 696.11 MiB
```
As you can see, this is a huge problem. If our bug reports about Parquet memory use problems are any indication, users have been suffering from this issue for a long time.
When the strings are mostly unique, then things are slower as expected, the peak memory use is higher because of the hash table
```
In [17]: unique_values = [tm.rands(10) for i in range(500000)]
In [18]: values = unique_values * 2
In [19]: arr = pa.array(values)
In [20]: timeit result = arr.to_pandas()
177 ms ± 574 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [21]: timeit result = arr.to_pandas(deduplicate_objects=False)
70.1 ms ± 783 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [42]: %memit result8 = arr.to_pandas()
peak memory: 644.39 MiB, increment: 92.23 MiB
In [43]: %memit result9 = arr.to_pandas(deduplicate_objects=False)
peak memory: 610.85 MiB, increment: 58.41 MiB
```
In real world work, many duplicated strings is the most common use case. Given the massive memory use and moderate performance improvements, it makes sense to have this enabled by default.
Author: Wes McKinney <wesm+git@apache.org>
Closes#3257 from wesm/ARROW-3928 and squashes the following commits:
d9a88700 <Wes McKinney> Prettier output
a00b51c7 <Wes McKinney> Add benchmarks for object deduplication
ca88b963 <Wes McKinney> Add Python unit tests, deduplicate for date and time types also when converting to Python objects
7a7873b8 <Wes McKinney> First working iteration of string deduplication when calling to_pandas
https://github.com/apache/arrow/issues/3154
This removes unused plasma interfaces Fetch(), Wait(), Transfer() and Info(), which depend on plasma manager which has already been removed from ray.
Author: Philipp Moritz <pcmoritz@gmail.com>
Author: Zhijun Fu <pingfu.fzj@antfin.com>
Author: Robert Nishihara <robertnishihara@gmail.com>
Closes#3167 from zhijunfu/remove-legacy-interfaces and squashes the following commits:
0efb5005f <Philipp Moritz> fix tensorflow op
be92e9085 <Philipp Moritz> fix java client
9da2cd38b <Philipp Moritz> Update _plasma.pyx
16ec63e9a <Robert Nishihara> More updates
e7413f739 <Robert Nishihara> Update _plasma.pyx
21398b5e7 <Zhijun Fu> merge
bcb320400 <Zhijun Fu> address comments
7967aea09 <Philipp Moritz> Merge branch 'master' into remove-legacy-interfaces
583cd97c4 <Zhijun Fu> ARROW-4015: remove unused interfaces for plasma manager
The ability to define custom install / build / uninstall commands was added in mainline ASV in https://github.com/airspeed-velocity/asv/pull/699
We don't need to use our own fork / PR anymore.
Author: Antoine Pitrou <antoine@python.org>
Closes#2480 from pitrou/ARROW-3125-asv-instructions-update and squashes the following commits:
59ddc4e4 <Antoine Pitrou> ARROW-3125: Update ASV instructions
Author: Antoine Pitrou <antoine@python.org>
Closes#2389 from pitrou/ARROW-2992-fix-parquet-benchmark and squashes the following commits:
9083dc3 <Antoine Pitrou> ARROW-2992: Fix Parquet benchmark
https://issues.apache.org/jira/browse/ARROW-2656 I probably should still write a benchmark for this, but figured I can get some quick feedback on this while working on that.
Author: Robert Gruener <robbieg@uber.com>
Closes#2185 from rgruener/thread-pooling and squashes the following commits:
4af3cdc3 <Robert Gruener> ARROW-2656: Improve the creation time of ParquetManifest for partitioned datasets using a thread pool
Author: Wes McKinney <wesm+git@apache.org>
Closes#2137 from wesm/ARROW-2395 and squashes the following commits:
d9e6e9f3 <Wes McKinney> Fix Cython flake8 warnings in examples
059a6d6b <Wes McKinney> Make functions in benchmarks/common.py py2-compatible
13791863 <Wes McKinney> Fix flake8 warnings outside of pyarrow/ directory. Check in CI
This adds some initial ASV benchmarks for plasma:
- Put latency
- Get latency
- Put throughput for 1KB, 10KB, 100KB, 1MB, 10MB, 100MB
It also includes some minor code restructuring to expose the start_plasma_store method.
Author: Philipp Moritz <pcmoritz@gmail.com>
Closes#2038 from pcmoritz/plasma-asv and squashes the following commits:
34a06845 <Philipp Moritz> measure wallclock time instead of process cpu time
c89256f7 <Philipp Moritz> parametrize tests
3567ddc7 <Philipp Moritz> fix windows build
eca17675 <Philipp Moritz> build plasma in asv
47671b34 <Philipp Moritz> fix
1261177e <Philipp Moritz> fix linting errors
7d4d6854 <Philipp Moritz> Add asv benchmarks for plasma
Also refactor the type inference visitor and remove the superfluous separate SeqVisitor; improve inference visitor performance by 30%; and add a struct type inference benchmark.
Author: Antoine Pitrou <antoine@python.org>
Closes#1935 from pitrou/ARROW-2074-infer-dict-lists and squashes the following commits:
13ed6c30 <Antoine Pitrou> Fix tests on 2.7
3baa2eac <Antoine Pitrou> ARROW-2074: Infer lists of dicts as struct arrays
Also add Decimal fodder for other benchmarks. @cpcloud
Author: Antoine Pitrou <antoine@python.org>
Closes#1798 from pitrou/ARROW-2357-benchmarks-pandas-object-is-null and squashes the following commits:
b1c6460 <Antoine Pitrou> ARROW-2357: Add microbenchmark for PandasObjectIsNull()
Author: Antoine Pitrou <antoine@python.org>
Closes#1665 from pitrou/ARROW-1035-streaming-benchmark and squashes the following commits:
32b1956 <Antoine Pitrou> ARROW-1035: Add streaming dataframe reconstruction benchmark
The goal here is to get the best of both the `pandas_serialization_context` (speed at serializing pandas dataframes containing strings and other objects) and the `default_serialization_context` (correctly serializing a large class of numpy object arrays).
This PR sort of messes up the function `pa.pandas_compat.dataframe_to_serialized_dict`. Is that function just a helper function for implementing the custom pandas serializers? Or is it intended to be used in other places.
TODO in this PR (assuming you think this approach is reasonable):
- [x] remove `pandas_serialization_context`
- [x] make sure this code path is tested
- [x] double check performance/behavior
cc @wesm @pcmoritz @devin-petersohn
Author: Robert Nishihara <robertnishihara@gmail.com>
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#1581 from robertnishihara/pandasserialization and squashes the following commits:
c551ed6c [Wes McKinney] Add benchmark for zero-copy pandas deserialization
e12cd721 [Robert Nishihara] Use highest protocol.
6e3eceb6 [Robert Nishihara] Fix bug.
31b3cbee [Robert Nishihara] Fixes.
0fc9ecfd [Robert Nishihara] Update documentation and testing.
b796fbdb [Robert Nishihara] Handle object arrays directly in pandas serializer.
Author: Antoine Pitrou <antoine@python.org>
Closes#1572 from pitrou/ARROW-2073-struct-from-tuples and squashes the following commits:
0a41ccc [Antoine Pitrou] ARROW-2073: [Python] Create struct array from sequence of tuples
This PR focusses on:
* ASV setup fixes
* splitting the benchmark file
* improving the array from/to pylist conversion benchmarks
Author: Antoine Pitrou <antoine@python.org>
Closes#1543 from pitrou/ARROW-1861-rework-benchmarks and squashes the following commits:
b608579 [Antoine Pitrou] ARROW-1861: [Python] Rework benchmark suite [skip ci]
This also fixes a bug in ChunkedArray::Equals. This is caught by the Python test suite but would benefit from more C++ unit tests.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#274 from wesm/ARROW-438 and squashes the following commits:
1f39568 [Wes McKinney] py3 compatibility
2e76c5e [Wes McKinney] Implement arrow::ConcatenateTables and Python wrapper. Fix bug in ChunkedArray::Equals
f3cb170 [Wes McKinney] Fix Cython compilation, verify pyarrow.Table.from_batches still works
af28755 [Wes McKinney] Implement Table::FromRecordBatches
Change-Id: I948b61d848c178edefad63465a74d9c303ad1f18
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#51 from xhochy/arrow-89 and squashes the following commits:
bd6a7cb [Uwe L. Korn] Split benchmarks and add one for a float64 column with NaNs
8f74528 [Uwe L. Korn] ARROW-89: [Python] Add benchmarks for Arrow<->Pandas conversion
Run them using `asv run --python=same` or `asv dev`.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#44 from xhochy/arrow-79 and squashes the following commits:
d3c6401 [Uwe L. Korn] Move benchmarks to toplevel folder
2737f18 [Uwe L. Korn] ARROW-79: [Python] Add benchmarks
