Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#238 from xhochy/ARROW-420 and squashes the following commits:
e497d9f [Uwe L. Korn] Add datetime.date parsing for numpy conversion
5c21453 [Uwe L. Korn] Add support for datetime.datetime
6bf346f [Uwe L. Korn] Add datetime.date conversions
6fca4da [Uwe L. Korn] ARROW-420: Align DATE type with Java implementation
Closes#108
Some users will not have a full Java Hadoop distribution and may wish to use the libhdfs3 package from Pivotal (https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3), part of Apache HAWQ (incubating).
In C++, you can switch by setting:
```c++
HdfsConnectionConfig conf;
conf.driver = HdfsDriver::LIBHDFS3;
```
In Python, you can run:
```python
con = arrow.io.HdfsClient.connect(..., driver='libhdfs3')
```
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#244 from wesm/ARROW-243 and squashes the following commits:
7ae197a [Wes McKinney] Refactor HdfsClient code to support both libhdfs and libhdfs3 at runtime. Add driver option to Python interface
These changes are resulting in GitHub producing archive tarballs with non-deterministic contents.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#243 from wesm/ARROW-429 and squashes the following commits:
49f6edb [Wes McKinney] Revert "ARROW-379: Use setuptools_scm for Python versioning"
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#242 from xhochy/ARROW-426 and squashes the following commits:
10739ac [Uwe L. Korn] ARROW-426: Python: Conversion from pyarrow.Array to a Python list
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#241 from xhochy/pyarrow-private-api and squashes the following commits:
dc9b814 [Uwe L. Korn] ARROW-425: Add private API to get python Table from a C++ object
I've been wanting to do this for a while -- it feels cleaner to me. I also am going to promote modules from arrow/util to the top level as well. I'm open to other ideas, too.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#236 from wesm/ARROW-418 and squashes the following commits:
6f556ea [Wes McKinney] Add missing math.h include for clang
9dc2e22 [Wes McKinney] Fix remaining old includes
6f7ae77 [Wes McKinney] Fixes, cpplint
66ac3f7 [Wes McKinney] Promote buffer.h/status.h/memory-pool.h to top level directory
8cdf059 [Wes McKinney] Consolidate Array and Builder classes in array.h, builder.h. Remove arrow/types subdirectory
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#235 from wesm/ARROW-328 and squashes the following commits:
f71decc [Wes McKinney] Return shared_ptr<T> by value instead of const-ref
As we cannot currently build pyarrow on readthedocs, we have to resort
to building the API docs for the latest version of pyarrow on
conda-forge. All other documentation will though be pulled directly from
the source code.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#228 from xhochy/ARROW-346 and squashes the following commits:
6a4bdc1 [Uwe L. Korn] Add license header
b741141 [Uwe L. Korn] ARROW-346: Use conda environment to build API docs
The one downside of this patch is that HdfsFile handles don't get garbage-collected until the cyclic GC runs -- I tried to fix this but couldn't get it working. So bytes don't always get flushed to HDFS until `close()` is called. The flush issue should be addressed on the C++ side
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#230 from wesm/ARROW-404 and squashes the following commits:
3a8e641 [Wes McKinney] Use weakref in _HdfsFileNanny to avoid cyclic gc
274d0c5 [Wes McKinney] amend comment
1539a2c [Wes McKinney] Ensure that HdfsClient does not get closed before an open file does when the last user-accessible client reference goes out of scope
These are better maintained on conda-forge since there's also an associated cross-platform build environment
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#231 from wesm/ARROW-408 and squashes the following commits:
8c58b75 [Wes McKinney] Remove defunct conda recipes
From discussion in ARROW-369, it is more consistent and flexible for the pyarrow.Table API to convert a RecordBatch list first a Table, then Table to pandas.DataFrame. For example:
```
table = pa.Table.from_batches(batches)
df = table.to_pandas()
```
Also updated conversion to print schemas in exception message if not equal.
Author: Bryan Cutler <cutlerb@gmail.com>
Closes#229 from BryanCutler/pyarrow-table-from_batches-ARROW-409 and squashes the following commits:
f5751e0 [Bryan Cutler] fixed schema check to print out if not equal
72ea875 [Bryan Cutler] changed batches conversion to Table instead
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#224 from xhochy/ARROW-379 and squashes the following commits:
3a68d9f [Uwe L. Korn] Remove deprecated version import
15fe9b2 [Uwe L. Korn] Add license header
aa9bd49 [Uwe L. Korn] ARROW-379: Use setuptools_scm for Python versioning
Modified Pandas adapter to handle columns with multiple chunks with `ConvertColumnToPandas`. This modifies the pyarrow public API by adding a class `RecordBatchList` and static method `toPandas` which takes a list of Arrow RecordBatches and outputs a Pandas DataFrame.
Adds unit test in test_table.py to do the conversion for each column with typed specialization.
Author: Bryan Cutler <cutlerb@gmail.com>
Closes#216 from BryanCutler/multi-batch-toPandas-ARROW-369 and squashes the following commits:
b6c9986 [Bryan Cutler] fixed formatting
edf056e [Bryan Cutler] simplified with pyarrow.schema.Schema.equals
068bc1b [Bryan Cutler] Merge remote-tracking branch 'upstream/master' into multi-batch-toPandas-ARROW-369
da65345 [Bryan Cutler] fixed test case for schema checking
9edb0ba [Bryan Cutler] used auto keyword where some typecasting was done in ConvertValues
bd2a720 [Bryan Cutler] added testcase for schema not equal, disabled now
c3d7e8f [Bryan Cutler] Changed conversion to make Table from columns first, now conversion is now just a free function
3ee51e6 [Bryan Cutler] cleanup
398b18d [Bryan Cutler] Fixed case for Integer specialization without nulls
7b29a55 [Bryan Cutler] Initial working version of RecordBatch list to_pandas, need more tests and cleanup
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#214 from xhochy/ARROW-389 and squashes the following commits:
e66c895 [Uwe L. Korn] Switch image to deprecated group
876cd65 [Uwe L. Korn] ARROW-389: Python: Write Parquet files to pyarrow.io.NativeFile objects
Added pyarrow api for `Schema.equals` to check if 2 schema's are equal and corresponding test case.
Author: Bryan Cutler <cutlerb@gmail.com>
Closes#221 from BryanCutler/add-pyarrow-schema_equals-ARROW-396 and squashes the following commits:
910e943 [Bryan Cutler] added test case for pyarrow Schema equals
24cf982 [Bryan Cutler] added pyarrow Schema equals, and related def for CSchema
This also includes format reconciliation as discussed in ARROW-384.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#211 from wesm/ARROW-363 and squashes the following commits:
6982c3c [Wes McKinney] Permit end of buffer IPC reads if length is 0
4d46c8b [Wes McKinney] Fix logical error with offsets array in JsonFileWriter. Add broken string test case to simple.json
36ab5d6 [Wes McKinney] Increment MetadataVersion in flatbuffer
844257e [Wes McKinney] cpplint
a2711f2 [Wes McKinney] Address other format incompatibilities, write vectorLayout to Arrow metadata
13608ef [Wes McKinney] Relax 64 byte padding. Do not write RecordBatch embedded in Message for now
6a66fc8 [Wes McKinney] Write record batch size prefix in Java
72ea42c [Wes McKinney] Note that padding is 64-bytes at start of file (for now)
c2ffde4 [Wes McKinney] More notes about the file format
aef4382 [Wes McKinney] cpplint
85128f7 [Wes McKinney] Refactor IPC/File record batch read/write structure to reflect discussion in ARROW-384
dbd6ed6 [Wes McKinney] Do not embed metadata length in WriteDataHeader
c529d63 [Wes McKinney] Fix JSON integration test example to make it further
d806aa6 [Wes McKinney] Exclude JSON files from Apache RAT checks
a7e2d4b [Wes McKinney] Draft testing harness
* Fix numpydoc compilation
* Add simple examples to the API
* Move away from deprecated Cython-property declaration
* Add basic descriptions with return types to functions
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#208 from xhochy/ARROW-382 and squashes the following commits:
31e0cb3 [Uwe L. Korn] ARROW-382: Extend Python API documentation
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#205 from xhochy/ARROW-371 and squashes the following commits:
1f73e8b [Uwe L. Korn] ARROW-371: Handle pandas-nullable types correctly
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#204 from wesm/ARROW-375 and squashes the following commits:
9e6f2a6 [Wes McKinney] BUG: convert unicode to utf8 bytes for column filtering
Assumes #192.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#193 from xhochy/ARROW-356 and squashes the following commits:
530484f [Uwe L. Korn] Mention new setup instructions
06b2f9c [Uwe L. Korn] Add tables describing dtype support
0467e0e [Uwe L. Korn] Move installation instructions into Sphinx docs
744202a [Uwe L. Korn] Document Pandas<->Arrow conversion
b5b4df5 [Uwe L. Korn] ARROW-356: Add documentation about reading Parquet
close#198
Author: Wes McKinney <wes.mckinney@twosigma.com>
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#200 from wesm/ARROW-362 and squashes the following commits:
99df96b [Wes McKinney] Force gc to avoid non-deterministic failure
d85228f [Wes McKinney] Be more careful about reference counts in zero-copy handoff, add pyarrow.Array.to_pandas method
cc7a6b3 [Uwe L. Korn] ARROW-362: Remove redunant reference count
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#197 from xhochy/ARROW-361 and squashes the following commits:
c1fb939 [Uwe L. Korn] Cache column indices
0c32213 [Uwe L. Korn] ARROW-361: Python: Support reading a column-selection from Parquet files
Added note to use LD_LIBRARY_PATH env var to add $ARROW_HOME/lib path so PyArrow can locate Arrow-Cpp shared libs.
Author: Bryan Cutler <cutlerb@gmail.com>
Closes#199 from BryanCutler/pyarrow-README-note-LD_LIBRARY_PATH-ARROW-368 and squashes the following commits:
15861c4 [Bryan Cutler] Added note for LD_LIBRARY_PATH in Python README
Added a couple ways to do this, either via the `--with-parquet` command line option (preferred) or by passing through an option to CMake
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#194 from wesm/ARROW-323 and squashes the following commits:
07c05cc [Wes McKinney] Update readme to illustrate proper use of with build_ext
3bd9a8d [Wes McKinney] Add --with-parquet option to setup.py
374e254 [Wes McKinney] Add to README about building the parquet extension
cab55cb [Wes McKinney] Opt in to building the pyarrow.parquet extension, do not silently fail
This is not the optimal choice, we should rather have an option to optimise for the underlying block size of the filesystem but without the infrastructure for that in ``parquet-cpp``, writing a single RowGroup is the much better choice.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#192 from xhochy/ARROW-357 and squashes the following commits:
9eccefd [Uwe L. Korn] ARROW-357: Use a single RowGroup for Parquet files as default.
This also resolves ARROW-230.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#187 from wesm/ARROW-348 and squashes the following commits:
3cdaeaf [Wes McKinney] Cast build_type to lowercase in case env variable is uppercase
74bfa71 [Wes McKinney] Pull default build type from environment variable
d0b3154 [Wes McKinney] Tweak readme
6017948 [Wes McKinney] Add built-type command line option to setup.py, build extensions in release type subdirectory to avoid conflicts with setuptools
Author: Christopher C. Aycock <christopher.aycock@twosigma.com>
Closes#185 from chrisaycock/ARROW-350 and squashes the following commits:
c2a4e64 [Christopher C. Aycock] Renamed 'kerb' parameter to 'kerb_ticket'
f1d63de [Christopher C. Aycock] ARROW-350: Added Kerberos to HDFS client
8f1052f [Christopher C. Aycock] ARROW-345: Proper locations of libhdfs and libjvm on Mac
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#179 from xhochy/ARROW-342 and squashes the following commits:
15d0ce3 [Uwe L. Korn] ARROW-342: Set Python version on release
Will try to verify whether this resolves the issue. See https://travis-ci.org/conda-forge/staged-recipes/builds/166897102
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#171 from wesm/ARROW-334 and squashes the following commits:
ed8fa39 [Wes McKinney] Switch by to xcode 6.4
b8224ce [Wes McKinney] Escape dollar sign in ORIGIN
b76b7ac [Wes McKinney] Fix LD_LIBRARY_PATH
3c8d2dd [Wes McKinney] Clean up Travis CI scripts a bit. Put in LD_LIBRARY_PATH
30488d7 [Wes McKinney] Don't conda install arrow-cpp during Travis build
afb1dc0 [Wes McKinney] Remove INSTALL_RPATH_USE_LINK_PATH
There's also a little bit of naming cleanup in `bit-util.h`, pardon the diff noise.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#177 from wesm/ARROW-317 and squashes the following commits:
0666b22 [Wes McKinney] Fix up pyarrow usage of BitUtil
3ab4e7a [Wes McKinney] Add Slice, Copy methods to Buffer
cb9519d [Wes McKinney] Use more conforming names in bit-util.h
Per discussions on the mailing list. This should in theory match the Java implementation.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#176 from wesm/ARROW-261 and squashes the following commits:
dca39ce [Wes McKinney] Make binary/string constants static to avoid memory-access-related segfaults in third party libraries
1e65b01 [Wes McKinney] Deprecate pyarrow::Status in favor of just arrow::Status. Conform pyarrow use of ArrayBuilder::Finish
9a1f77e [Wes McKinney] Add license header to index.rst
bd70cab [Wes McKinney] Complete refactoring, fix up IPC tests for flattened string/binary buffer/metadata layout
ae64f2e [Wes McKinney] Refactoring to reflect collaprsed list-like structure of Binary and String types. Not yet complete
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#175 from xhochy/ARROW-97 and squashes the following commits:
2ec3e11 [Uwe L. Korn] Add license headers
d838e81 [Uwe L. Korn] ARROW-97: API documentation via sphinx-apidoc
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#173 from xhochy/ARROW-191 and squashes the following commits:
278f8b0 [Uwe L. Korn] ARROW-191: Python: Provide infrastructure for manylinux1 wheels
This makes testing and IPC data wrangling a little easier.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#165 from wesm/ARROW-332 and squashes the following commits:
5f19b97 [Wes McKinney] Add simple arrow::Array->NumPy-for-pandas conversion helper and RecordBatch.to_pandas
This also adds some IO scaffolding for interacting with `arrow::Buffer` objects from Python and assorted additions to help with testing.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#164 from wesm/ARROW-312 and squashes the following commits:
7df3e5f [Wes McKinney] Set BUILD_WITH_INSTALL_RPATH on arrow_ipc
be8cee0 [Wes McKinney] Link Cython modules to libarrow* libraries
5716601 [Wes McKinney] Fix accidental deletion
77fb03b [Wes McKinney] Add / test Buffer wrapper. Test that we can write an arrow file to a wrapped buffer. Resize buffer in BufferOutputStream on close
316537d [Wes McKinney] Get ready to wrap Arrow buffers in a Python object
4822d32 [Wes McKinney] Implement RecordBatch::Equals, compare in Python ipc file writes
a931e49 [Wes McKinney] Permit buffers (write padding) in a non-multiple of 64 in an IPC context, to allow zero-copy writing of NumPy arrays
2c49cd4 [Wes McKinney] Some debugging
ca1562b [Wes McKinney] Draft implementations of Arrow file read/write from Python
Author: Julien Le Dem <julien@dremio.com>
Closes#159 from julienledem/fix_licenses and squashes the following commits:
0c97810 [Julien Le Dem] fix NOTICE
1489289 [Julien Le Dem] more licenses
0eb2aeb [Julien Le Dem] more licenses
9ac1159 [Julien Le Dem] more licenses
eafa0e1 [Julien Le Dem] more licenses
30b0fa1 [Julien Le Dem] more licenses
bcfc75f [Julien Le Dem] add ci
51db31b [Julien Le Dem] ARROW-321: fix arrow licenses
Also removes a redundant LICENSE.txt from cpp/
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#157 from wesm/ARROW-318 and squashes the following commits:
9e802f2 [Wes McKinney] Update python/README.md. Remove redundant LICENSE.txt from cpp/
Author: Christopher C. Aycock <christopher.aycock@twosigma.com>
Closes#155 from chrisaycock/ARROW-313 and squashes the following commits:
e47cc01 [Christopher C. Aycock] ARROW-313: Build on any version of XCode
This will enable code (such as arrow IPC or Parquet) that only knows about Arrow's IO subsystem to interact with Python objects in various ways. In other words, when we have in C++:
```
std::shared_ptr<io::ReadableFileInterface> handle = ...;
handle->Read(nbytes, &out);
```
then the C++ file handle could be invoking the `read` method of a Python object. Same goes for `arrow::io::OutputStream` and `write` methods. There's data copying in some places overhead because of the rigid memory ownership semantics of the `PyBytes` type, but this can't be avoided here.
Another nice thing is that if we have some data in a Python bytes object that we want to expose to some other C++ component, we can wrap it in the `PyBytesReader` which provides zero-copy read access to the underlying data.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#152 from wesm/ARROW-302 and squashes the following commits:
2de9f97 [Wes McKinney] Fix compiler warning / bug from OS X
316b845 [Wes McKinney] Code review comments
e791893 [Wes McKinney] Python 2.7 fix
0fc4cf1 [Wes McKinney] cpplint
e9b8c60 [Wes McKinney] Test the size() method and fix bug with missing whence
6481e91 [Wes McKinney] Add a zero-copy reader for PyBytes
7e357eb [Wes McKinney] Get basic Python file read/write working
d470133 [Wes McKinney] Share default implementations of ReadAt, add Buffer-based Read API
737a8db [Wes McKinney] Refactoring, more code sharing with native file interfaces
8be433f [Wes McKinney] Draft PyReadableFile implementation, not yet tested
20a3f28 [Wes McKinney] Draft API for Arrow IO wrappers for Python files
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#148 from xhochy/arrow-305 and squashes the following commits:
93d653b [Uwe L. Korn] ARROW-305: Add compression and use_dictionary options to Parquet interface
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#149 from xhochy/arrow-306 and squashes the following commits:
11a3e66 [Uwe L. Korn] ARROW-306: Add option to pass cmake arguments via environment variable
This patch depends on PARQUET-728 (to run the full test suite, including pyarrow Parquet tests)
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#145 from wesm/ARROW-296 and squashes the following commits:
d67b4f9 [Wes McKinney] Refactor to link against parquet_arrow, fix up cmake files
Several things here
* Clean up IO interface class structure to be able to indicate precise characteristics of an implementation
* Make the IPC reader/writer use more generic interfaces -- writing only needs an output stream, reading only needs a random access reader. This will unblock ARROW-267
* Create a separate arrow_ipc shared library
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#138 from wesm/ARROW-280 and squashes the following commits:
6a59eb6 [Wes McKinney] * Restructure IO interfaces to accommodate more configurations. * Refactor memory mapped IO interfaces to be in line with other arrow::io classes. * Split arrow_ipc into a leaf library * Refactor pyarrow and arrow_parquet to suit. Move BufferReader to arrow_io. Pyarrow parquet tests currently segfault
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#132 from wesm/ARROW-284 and squashes the following commits:
e3410cf [Wes McKinney] Install miniconda in $HOME to avoid long prefix issues in conda-build 2.0
9fd94f5 [Wes McKinney] Do not run death test when valgrind is enabled. Gracefully skip pyarrow.parquet when ARROW_PARQUET=off
ccf56f8 [Wes McKinney] Disable arrow_parquet module in Travis CI
For the Pandas<->Parquet bridge this is a lossy conversion but must be explicitly activated by the user.
Regarding Parquet 1.0: Yes, the logical type is not supported but should be simply ignored by the reader. Implementation for INT96 timestamps is not in the scope of this PR.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#107 from xhochy/arrow-242 and squashes the following commits:
8db6968 [Uwe L. Korn] Add missing include
34126b1 [Uwe L. Korn] ARROW-242: Support Timestamp Data Type
Author: Wes McKinney <wesm@apache.org>
Closes#104 from wesm/ARROW-236 and squashes the following commits:
73648e0 [Wes McKinney] cpplint
f2cd77f [Wes McKinney] Check in io.pxd
94bcd30 [Wes McKinney] Do not let Parquet close an Arrow file
9b9d94d [Wes McKinney] Barely working direct HDFS-Parquet reads
06ddd06 [Wes McKinney] Slight refactoring of read table to be able to also handle classes wrapping C++ file interfaces
c7a913e [Wes McKinney] Provide a means to expose abstract native file handles
e6724de [Wes McKinney] Implement alternate ctor to construct parquet::FileReader from an arrow::io::RandomAccessFile
Uwe L. Korn