### Rationale for this change
On the hypothesis tests we treat Warnings as errors and the job has been failing with the latest Pytest version (9.1.0). This is related to:
- https://github.com/pytest-dev/pytest/issues/13409
With the latest pytest using generators for parametrization has been deprecated.
### What changes are included in this PR?
Change the generator that was causing the failures and return a list of supported_factories instead.
### Are these changes tested?
Yes both locally and the archery test that was failing.
### Are there any user-facing changes?
No
* GitHub Issue: #50277
Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
mimalloc is enabled by default but users can disable it. For example, Debian package disables it.
If it's disabled, PyArrow tests are failed.
### What changes are included in this PR?
Remove `"mimalloc"` from `mandatory_backends` in our test.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #49295
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Please check #18036.
### What changes are included in this PR?
Almost everything needed for building and testing Python wheels for musllinux.
The side-effect of these changes are the exclusion of 2 unit tests when running on musl based systems.
The skipped unit tests are:
- test_uwsgi_integration
- test_print_stats
### Are these changes tested?
I was able to successfully generate a musllinux wheel by running the following:
```
docker-compose build python-wheel-musllinux-1-2
docker-compose run python-wheel-musllinux-1-2
```
I was also able to run the following tests with no errors:
```
python-wheel-musllinux-test-imports
python-free-threaded-wheel-musllinux-test-imports
python-wheel-musllinux-test-unittests
python-free-threaded-wheel-musllinux-test-unittests
```
### Are there any user-facing changes?
Yes.
* GitHub Issue: #18036
Authored-by: Nuno Veloso <nveloso21@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Add a MemoryPool method to print allocator-specific statistics to stderr, to help diagnose perceived memory consumption issues.
Also add missing Python bindings for `MemoryPool::total_bytes_allocated` and `MemoryPool::num_allocations`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45358
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
jemalloc may have a problem on ARM.
See also: https://github.com/apache/arrow/issues/44342
### What changes are included in this PR?
* Disable jemalloc by default on ARM.
* Disable jemalloc for manylinux wheel for ARM.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #44342
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
Disable debug memory pool for ASAN and Valgrind so that they can detect more subtle memory issues regarding to buffer tail bytes.
### What changes are included in this PR?
1. Add a `none` option to debug memory pool env var to make other things slightly easier.
2. Change `*_test.sh` scripts to conditionally set debug memory pool env var.
3. Top-level docker compose change to pass none to debug memory pool env var for ASAN and Valgrind.
### Are these changes tested?
The CI should cover it well.
### Are there any user-facing changes?
No.
* Closes: #39973
Authored-by: Ruoxi Sun <zanmato1984@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
When writing out of bounds and ARROW_DEBUG_MEMORY_POOL isn't set, the test subprocess may crash depending on the underlying allocator.
Closes#12595 from pitrou/minor-test-memory-py
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Debug memory checks can be enabled by setting the environment variable ARROW_DEBUG_MEMORY_POOL
to one of the recognized values "abort", "trap" or "warn" (regardless of whether Arrow
was compiled in debug or release mode).
The only implemented check adds a suffix past the allocation and checks that it isn't clobbered on deallocation.
Adding a prefix would be more expensive because of alignment constraints.
Also add a fix by Will Jones for ARROW-14047 (Parquet Arrow reader sets null values in buffer overflow), since otherwise CI would fail.
Closes#12330 from pitrou/ARROW-15550-debug-memory-checks
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
@pitrou What are your thoughts on this. It's not urgent (benchmarking can always use the system allocator in the meantime if they are having RAM issues and I think they reduced already which benchmarks they run on the RAM-limited servers)
With mimalloc the following command has a max RSS of ~16GB.
```
conbench file-read fanniemae_2016Q4 --all=true
```
If I add a call to release_unused between runs then the max RSS drops to ~10GB.
Closes#10186 from westonpace/feature/ARROW-12584--c-python-expose-method-for-benchmarking-tools
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
ARROW_DEFAULT_MEMORY_POOL can take the name of the desired memory pool backend
('jemalloc', 'mimalloc', 'system').
Closes#9105 from pitrou/ARROW-11009-memory-pool-env-var
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
By default, Cython extension classes (defined with "cdef class") don't have a weakref slot, so add one to all of them.
This adds just one memory word to each object, which IMHO is acceptable.
Closes#7758 from pitrou/ARROW-9469-py-weakrefable-objects
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Also expand the exposed API a bit.
Author: Antoine Pitrou <antoine@python.org>
Closes#2725 from pitrou/ARROW-2808-python-memory-pool-tests and squashes the following commits:
7507891d <Antoine Pitrou> Re-use common constructor test
463e52d4 <Antoine Pitrou> ARROW-2808: Add MemoryPool tests
Raúl Cumplido