(obviously still needs test, but already putting the fix I was testing locally in a draft PR)
Closes#9128 from jorisvandenbossche/ARROW-11163
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
- organize parquet test files to be consistent with the recently added orc [subdir](https://github.com/apache/arrow/tree/master/python/pyarrow/tests/data)
- use pathlib paths in parquet and orc tests
I intend to consolidate parquet test suite a bit in followup PRs.
Author: Krisztián Szűcs <szucs.krisztian@gmail.com>
Closes#2506 from kszucs/ARROW-3160 and squashes the following commits:
bf1fe1f8 <Krisztián Szűcs> use _has_pathlib
8b023034 <Krisztián Szűcs> compatibility with py35
dc306003 <Krisztián Szűcs> reorganizte utility functions in parquet and filasystem modules; support pathlib.Path in Filesystem API
062ef56a <Krisztián Szűcs> global tempdir fixture returns pathlib object
7c4d72c7 <Krisztián Szűcs> fallback to pathlib2 for older python
c550e1ba <Krisztián Szűcs> update parquet and orc tests to use pathlib
ee494170 <Krisztián Szűcs> use pathlib in orc and parquet tests; move parquet test files to its own directory
These use the example files published in the Apache ORC repository.
Author: Wes McKinney <wesm+git@apache.org>
Author: Antoine Pitrou <antoine@python.org>
Closes#2428 from pitrou/ARROW-1968-orc-tests and squashes the following commits:
6f538923 <Wes McKinney> Update quay.io image used for pandas 0.23.4
28335ff9 <Wes McKinney> Use pandas 0.23.4 everywhere
d5d34e53 <Wes McKinney> Upgrade pandas in manylinux1 image to latest
c51a77f1 <Wes McKinney> Do not use binary read mode
dd456204 <Wes McKinney> Use pandas for ORC tests, delete pickles
43b0208a <Antoine Pitrou> Add decimals test
06c0f5f4 <Antoine Pitrou> Fix tests on py2
42f01864 <Antoine Pitrou> ARROW-1968: Add basic unit tests for ORC reader
This closes [ARROW-1883](https://issues.apache.org/jira/browse/ARROW-1883).
So basically what I did in `_add_any_metadata` was replacing `col = table[i]` with:
```
idx = schema.get_field_index(raw_name)
if idx != -1:
col = table[idx]
```
to check that the column is actually present in the schema. However, that involved some more code to get to `raw_name` (the name how the column is present in the schema), as this does not always match the name in `pandas_metadata['column'][..]['name']`. Not sure if there is a better way to get that name.
(or if it would be better to filter `pandas_metadata` earlier on, instead of checking when actually trying to process the metadata of that column)
Author: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Closes#1386 from jorisvandenbossche/parquet-column-selection and squashes the following commits:
3f605ef3 [Joris Van den Bossche] add test for compatibility for arrow 0.7.1 written parquet files
ea891b25 [Joris Van den Bossche] Use field_name if available
f6bdd1d3 [Joris Van den Bossche] Merge remote-tracking branch 'upstream/master' into parquet-column-selection
5df51f18 [Joris Van den Bossche] Add additional tests
306eaba3 [Joris Van den Bossche] ARROW-1883: [Python] Fix handling of metadata in to_pandas when not all columns are present
Author: Phillip Cloud <cpcloud@gmail.com>
Closes#1298 from cpcloud/ARROW-1787 and squashes the following commits:
6f5fbd55 [Phillip Cloud] Add more index naming tests
8ba06256 [Phillip Cloud] Add test data
56e7fe58 [Phillip Cloud] Use BytesIO
5f50da38 [Phillip Cloud] Implement
d61c43e6 [Phillip Cloud] Add test
9abad95f [Phillip Cloud] Add test data
Junming Chen