### Rationale for this change
Switch to std::chrono for MSVC to be able to use the system-provided timezone automatically on Windows.
### What changes are included in this PR?
This adds `chrono_internal.h` that uses C++20 std::chrono timezone/calendar APIs on compilers with support (MSVC only for now) and falls back to vendored `date.h` otherwise.
### Are these changes tested?
Partially tested locally and partially to be tested on CI.
### Are there any user-facing changes?
Yes, Windows users will no longer need to install the IANA tzdb (see [instructions here](https://arrow.apache.org/docs/cpp/build_system.html#runtime-dependencies) and [here](https://arrow.apache.org/docs/python/install.html#tzdata-on-windows)). We possibly have tzdb download set up in CI too and should update it appropriately.
* GitHub Issue: #48593
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
See: #49044
### What changes are included in this PR?
Urllib now request with `"user-agent": "pyarrow"`
### Are these changes tested?
It's a CI fix.
### Are there any user-facing changes?
No, just a CI test fix.
* GitHub Issue: #49044
Authored-by: Rok Mihevc <rok@mihevc.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
### Rationale for this change
We have two Windows issues and this PR is addressing both:
1. PyArrow's `download_tzdata_on_windows` can fail due to TLS issues in certain CI environments.
2. The Python wheel test infrastructure needs a tzinfo database for ORC and the automation fetching that started failing because the URL was made invalid upstream.
These two issues are being solved in one PR simply because they appeared together during the 19.0.1 release process but they're separate.
### What changes are included in this PR?
1. Makes `download_tzdata_on_windows` more robust to TLS errors by attempting to use `requests` if it's available and falling back to urllib otherwise.
2. Switches our Windows wheel test infrastructure to grab a tzinfo database from the tzdata package on PyPi instead of from a mirror URL. This should be much more stable for us over time.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #45295
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
There is a section in the [Arrow C++ documentation with the instructions](https://arrow.apache.org/docs/dev/cpp/build_system.html#runtime-dependencies) on how to download and extract text version of the IANA timezone database and on Windows. We should provide a function in PyArrow that a user would call to download and extract the timezone database from Python.
### What changes are included in this PR?
Function `download_tzdata_on_windows()` added to python/pyarrow/util.py that downloads and extracts timezone database to a standard location in `%USERPROFILE%\Downloads\tzdata` on Widnows.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* Closes: #37328
Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
### What changes are included in this PR?
Spelling fixes to python/
### Are these changes tested?
### Are there any user-facing changes?
* Closes: #38944
Authored-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change
Python classes sometimes duplicate docstrings, but change one word such as class name. Add a decorator function as a utility to help deduplicate docstring descriptions. Only works in Python. Does not work in Cython due to this CPython issue https://github.com/python/cpython/issues/91309.
### What changes are included in this PR?
Add a decorator `@ doc` that can copy, concatenate, and/or format docstrings between classes.
### Are these changes tested?
Tests added.
```
>>> import pyarrow
>>> from pyarrow.filesystem import FileSystem, LocalFileSystem, DaskFileSystem, S3FSWrapper
>>> from pyarrow.hdfs import HadoopFileSystem
>>> for fs in [FileSystem, LocalFileSystem, DaskFileSystem, S3FSWrapper, HadoopFileSystem]:
... print(fs.__name__)
... print(fs.isdir.__doc__)
...
FileSystem
Return True if path is a directory.
Parameters
----------
path : str
Path to check.
LocalFileSystem
Return True if path is a directory.
Parameters
----------
path : str
Path to check.
DaskFileSystem
Return True if path is a directory.
Parameters
----------
path : str
Path to check.
S3FSWrapper
Return True if path is a directory.
Parameters
----------
path : str
Path to check.
HadoopFileSystem
Return True if path is a directory.
Parameters
----------
path : str
Path to check.
```
Note that `FileSystem.isdir.__doc__` is not dedented because it does not use the `@ doc` decorator.
### Are there any user-facing changes?
No
* Closes: #34868
Authored-by: Dane Pitkin <dane@voltrondata.com>
Signed-off-by: Alenka Frim <frim.alenka@gmail.com>
This PR adds support for the home directory in paths. This fixes a problem where `pyarrow.parquet.read_table` fails to find files relative to the home directory.
I added `os.path.expanduser` to the `_stringify_path` function to have the widest effect with minimal changes. Happy to adjust that if there is a better place.
Closes#12675 from cjermain/ARROW-15982
Authored-by: Colin Jermain <cjermain@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Some Python versions have a bug where `signal.getsignal` creates a reference cycle holding execution frames alive (https://bugs.python.org/issue42248).
This would cause excessive lifetimes of the PyArrow table returned by `read_csv`.
Closes#10609 from pitrou/ARROW-13187-signal-refcycle
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This binds InMemoryDataset to Python, allowing us to create and write back out datasets from iterables of record batches and various other objects.
Closes#9802 from lidavidm/arrow-10882
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
TODOs:
- [x] split PRs into two, one with simplified python to arrow conversions with benchmarks
- [x] implement union scalar on the C++ side
- [x] store the index value for dictionary scalar
- [x] more tests
Closes ARROW-9017, ARROW-9153
Closes#7519 from kszucs/ARROW-9153
Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Wes McKinney <wesm@apache.org>
This is some routine scrubbing that deals with some remnants of the prior dual py2/3 codebase.
Closes#7421 from wesm/ARROW-9030
Authored-by: Wes McKinney <wesm@apache.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
In my opinion, these APIs in the top level namespace are unclear and possibly misleading. We should encourage people to use the `pyarrow.ipc` namespace.
Closes#6757 from wesm/ARROW-8232
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This PR implements the following new features:
1. `CudaContext::GetDeviceAddress` method
2. `pyarrow.cuda.Context.get_device_address` method
3. `pyarrow.cuda.Context.buffer_from_object` method to create a CudaBuffer view of arbitrary device memory object that implements `__cuda_array_interface__` attribute or that is `CudaBuffer` or `CudaHostBuffer` or numba `MemoryPointer` object.
4. Tests for `buffer_from_object` method
and the following improvements:
1. `pyarrow.cuda.Context.foreign_buffer` ensures that the used device memory address is valid for the given context.
Author: Pearu Peterson <pearu.peterson@gmail.com>
Closes#3439 from pearu/arrow-4212 and squashes the following commits:
ba905739 <Pearu Peterson> Add 2-D and 3-D tests. buffer_from_object requires contiguous memory input.
355fc035 <Pearu Peterson> Apply more feedback fixes.
22134da4 <Pearu Peterson> Refactor. Add tests for negative strides and reduce test cases.
dbd1cec6 <Pearu Peterson> Moved some buffer_from_objects tests under test_cuda.
562c1805 <Pearu Peterson> Change GetDeviceAddress API to use uint8_t, add C++ tests for it, update docs and apply feedback.
2726a91e <Pearu Peterson> Add buffer_from_object and get_device_address methods.
df8656f8 <Pearu Peterson> Add GetDeviceAddress method to CudaContext.
Author: Wes McKinney <wesm+git@apache.org>
Closes#3180 from wesm/ARROW-3971 and squashes the following commits:
2a367f5d <Wes McKinney> Remove Python deprecations in 0.11 and prior
system. Add unit test label granularity options, ability to add component group
targets like 'make parquet' that build libraries and tests
Change-Id: I250fd10da3a9505952115b3cec18cd7cb5589bdb
- organize parquet test files to be consistent with the recently added orc [subdir](https://github.com/apache/arrow/tree/master/python/pyarrow/tests/data)
- use pathlib paths in parquet and orc tests
I intend to consolidate parquet test suite a bit in followup PRs.
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes#2506 from kszucs/ARROW-3160 and squashes the following commits:
bf1fe1f8 <Krisztián Szűcs> use _has_pathlib
8b023034 <Krisztián Szűcs> compatibility with py35
dc306003 <Krisztián Szűcs> reorganizte utility functions in parquet and filasystem modules; support pathlib.Path in Filesystem API
062ef56a <Krisztián Szűcs> global tempdir fixture returns pathlib object
7c4d72c7 <Krisztián Szűcs> fallback to pathlib2 for older python
c550e1ba <Krisztián Szűcs> update parquet and orc tests to use pathlib
ee494170 <Krisztián Szűcs> use pathlib in orc and parquet tests; move parquet test files to its own directory
I didn't remove the `nthreads` deprecations in Python yet. @pitrou do you support removing them in 0.11?
Author: Wes McKinney <wesm+git@apache.org>
Closes#2478 from wesm/ARROW-2989 and squashes the following commits:
8de36e6a <Wes McKinney> Test non-threaded conversion
bff138a0 <Wes McKinney> Make use_threads=True the default in pandas conversion tests
5ae97a96 <Wes McKinney> Remove nthreads argument where thread pool is being used. Set default for use_threads to True
279453c7 <Wes McKinney> Remove C++/Python API deprecations in 0.10, except for nthreads in Python
There are two areas where `nthreads` cannot be replaced immediately by the global thread pool:
1. when converting Pandas data to Arrow table or record batch, since it uses a Python `ThreadPoolExecutor` from pure Python code (see `dataframe_to_arrays` in `pandas_compat.py`)
2. when reading or writing Parquet data, since `parquet-cpp` relies on parallelization facilities in the stable version of Arrow (see https://github.com/apache/parquet-cpp/pull/467)
Elsewhere, we add a `use_threads` boolean argument and deprecate `nthreads`.
Author: Antoine Pitrou <antoine@python.org>
Closes#2078 from pitrou/ARROW-2568 and squashes the following commits:
91187bf6 <Antoine Pitrou> Move use_threads flag into PandasOptions
a7aeed0e <Antoine Pitrou> Factor out secession predicate
f601d4e9 <Antoine Pitrou> ThreadPool::State pointer is const
4567a2c3 <Antoine Pitrou> Add a two-argument variant of ParallelFor() that uses the global CPU thread pool
d0a527ab <Antoine Pitrou> Restore single-thread path in WriteTableToBlocks()
2171e6e2 <Antoine Pitrou> On Windows, avoid shutting down the global thread pool at process exit
934e5a11 <Antoine Pitrou> Add & operator between Statuses
96397076 <Antoine Pitrou> Factor out deprecation logic
6b685406 <Antoine Pitrou> Fix MSVC warning
6b6f64a1 <Antoine Pitrou> Make ThreadPool capacity an int, not a size_t
61669755 <Antoine Pitrou> Rename CPUThreadPool() to GetCpuThreadPool()
d4eb8d47 <Antoine Pitrou> Export ThreadPool and CPUThreadPool()
6985afe6 <Antoine Pitrou> Lint
fcca62f5 <Antoine Pitrou> Emit FutureWarning (which is visible by default) rather than DeprecationWarning
172fba37 <Antoine Pitrou> Use C++ API, rather than multiprocessing, in pyarrow.{cpu_count,set_cpu_count}
50310b88 <Antoine Pitrou> Add API function to get desired ThreadPool capacity
3417325c <Antoine Pitrou> ARROW-2568: WIP
Author: Antoine Pitrou <antoine@python.org>
Closes#1736 from pitrou/ARROW-2292-py_buffer and squashes the following commits:
41bf5f8f <Antoine Pitrou> ARROW-2292: Rename frombuffer() to py_buffer()
cc @yackoa
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#916 from wesm/ARROW-1213 and squashes the following commits:
f8a0aff1 [Wes McKinney] Add HDFS section to API docs
c54302df [Wes McKinney] Add deprecation warning for HdfsClient
4d3e7222 [Wes McKinney] Auto-wrap s3fs filesystem when using ParquetDataset
0be33bb8 [Wes McKinney] Implement os.walk emulation layer for s3fs
719f806d [Wes McKinney] Progress toward supporting s3fs in Parquet reader
bbd664ed [Wes McKinney] Refactor HdfsClient into pyarrow/hdfs.py. Add connect factory method. Rename to HadoopFilesystem. Add walk implementation for HDFS, base Parquet directory walker on that
4984a9d4 [Wes McKinney] Refactoring slightly
4c0bcf4a [Wes McKinney] Start on Dask filesystem wrapper, S3-Parquet dataset test case
Also fixes a serious bug in which the data source passed to the ParquetReader gets garbage collected prematurely
Also implements ARROW-470
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#296 from wesm/ARROW-475 and squashes the following commits:
894d2a2 [Wes McKinney] Implement Filesystem abstraction, add Filesystem.read_parquet. Implement rudimentary shim on local filesystem
3927c2c [Wes McKinney] Test read multiple Parquet from HDFS, fix premature garbage collection error
4904b3b [Wes McKinney] Implement read_multiple_files function for multiple Parquet files as a single Arrow table
Rok Mihevc