The tarballs produced by `setup.py sdist` have always been broken. This creates a tarball that can be installed assuming that the Arrow/Parquet C++ libraries are installed already on the target system.
I wanted to see if I could bundle the `cpp/cmake_modules` files without using a symlink but I couldn't figure it out; it might require subclassing the sdist cmdclass to make this work. I'm not sure having the symlink is so bad
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#1188 from wesm/ARROW-1535 and squashes the following commits:
dcc2e47d [Wes McKinney] Fix RAT excludes given changes to cmake files
310af1b7 [Wes McKinney] Move all cmake files to cpp/cmake_modules, fix up sdist tarball so it can be built/installed
I don't have access to a Windows machine at the moment to try this out locally. I will try from wesm/arrow-dist on Appveyor in the meantime
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#1127 from wesm/ARROW-1606 and squashes the following commits:
9edd85d [Wes McKinney] Bundle .lib files on Windows in CMakeLists.txt
e8f6c73 [Wes McKinney] Copy all suffixes for libraries on Windows to get .dll and .lib files
Now the test suite does not fail if you build the Plasma libraries but forget to pass `--with-plasma` to the Python build
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#903 from wesm/ARROW-1289 and squashes the following commits:
0e9ce78b [Wes McKinney] Add PYARROW_BUILD_PLASMA CMake option, make plasma build work like parquet build
This PR introduces a Cython API to Plasma, a FindPlasma.cmake to make it easier to integrate Plasma with CMake projects and sets up packaging with pyarrow.
Author: Philipp Moritz <pcmoritz@gmail.com>
Author: Robert Nishihara <robertnishihara@gmail.com>
Closes#797 from pcmoritz/plasma-cython and squashes the following commits:
d8319fc [Philipp Moritz] get for of PlasmaClient.connect
d14ab87 [Philipp Moritz] get rid of MutableBuffer
08f24a5 [Philipp Moritz] fix typos and move FixedSizeBufferOutputStream
e33443d [Philipp Moritz] fix setup.py develop for plasma
5f7b779 [Philipp Moritz] changes needed to make Ray work with Plasma in Arrow
b9e2dee [Philipp Moritz] fix windows build
3e4a84d [Philipp Moritz] fix segfault
0bea267 [Philipp Moritz] debug
23fe5f5 [Philipp Moritz] make plasma store binary part of the pyarrow package for tests
b863d13 [Philipp Moritz] fix
997de1e [Philipp Moritz] fix
47dc739 [Philipp Moritz] fixes
47033e7 [Philipp Moritz] switch to pytest
ed84c53 [Philipp Moritz] partial fixes
9bc5c15 [Philipp Moritz] implement wait and fetch for the client
45f338f [Philipp Moritz] test plasma on macOS
8b53618 [Philipp Moritz] fix
54f595e [Philipp Moritz] try fixing python 2 tests
2c6d652 [Philipp Moritz] convert docs to numpy format
3270628 [Philipp Moritz] try to get documentation up
44d1a55 [Philipp Moritz] cleanups and release GIL
a9f6502 [Philipp Moritz] more fixes
1ff88e7 [Philipp Moritz] fix travix ci
348f9bf [Philipp Moritz] fixes
4ae1a27 [Philipp Moritz] fix
fd80203 [Philipp Moritz] Plasma Python extension packaging: It compiles!
3b69973 [Robert Nishihara] Fixed minor python linting.
c9f6bcf [Robert Nishihara] Fix indentation and line lengths in plasma.pyx.
67b0951 [Robert Nishihara] Fix long lines in plasma/test/test.py.
e26527c [Robert Nishihara] Convert plasma test.py from 2 space indentation to 4 space indentation.
acc71d2 [Philipp Moritz] add round trip test for dataframes
2b7f949 [Philipp Moritz] implement mutable arrow python buffers
c06f1b5 [Philipp Moritz] fix test
1d7928f [Philipp Moritz] add arrow roundtrip test
6371e2e [Philipp Moritz] fix tests
3021d59 [Philipp Moritz] make ObjectID pickleable
dd5a7d8 [Philipp Moritz] fix tests
777e9c7 [Philipp Moritz] introduce plasma namespace
a4a9628 [Philipp Moritz] fix c++ tests
924888b [Philipp Moritz] update
f970df3 [Philipp Moritz] reduce logging
2ff2480 [Philipp Moritz] workaround for python visibility
d4934a9 [Philipp Moritz] update
cba92c1 [Philipp Moritz] setup.py for plasma
066d0ea [Philipp Moritz] test
1aea320 [Philipp Moritz] run plasma tests
3c4de52 [Philipp Moritz] use cmake to build the cython extension
bf39297 [Philipp Moritz] build and install pyarrow for plasma tests
5bf722a [Philipp Moritz] fix plasma path
1c5434c [Philipp Moritz] fix formatting
187cc24 [Philipp Moritz] add travis tests
c3d462d [Philipp Moritz] remove Python C extension
d9261b4 [Philipp Moritz] add documentation and license
db2d09a [Philipp Moritz] get all python tests in place
78d08ac [Philipp Moritz] make eviction work in Cython
18e0ac4 [Philipp Moritz] get tests
bc681ca [Philipp Moritz] port some python tests
f8e05f2 [Philipp Moritz] implement plasma.get in the cython client
d590c8a [Philipp Moritz] update
5178ee7 [Philipp Moritz] update
9044a01 [Philipp Moritz] initial plasma cython client commit
Change-Id: Ib03392431851773df3b59b2c9a4d9a7bd672d2cb
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#761 from xhochy/ARROW-1123 and squashes the following commits:
2c75f56d [Uwe L. Korn] Use shared jemalloc always if available
2205586f [Uwe L. Korn] Run parquet tests in manylinux1 build
97f77d08 [Uwe L. Korn] Add pthread to static dependencies
7d01e9eb [Uwe L. Korn] Cpplint
bf478f1c [Uwe L. Korn] Fix allocator
4fbc7bac [Uwe L. Korn] Correct small allocations
c3bacc05 [Uwe L. Korn] Revert "Also link static libs to librt"
44f0cfc5 [Uwe L. Korn] Also link static libs to librt
bac694df [Uwe L. Korn] Also link static libs to librt
35212bc0 [Uwe L. Korn] Remove obsolete import check
4b714a14 [Uwe L. Korn] Don't force optional that is no longer there
e004150c [Uwe L. Korn] Only search the for pthread library, not the headers
e65d0d1d [Uwe L. Korn] ARROW-1123: Make jemalloc the default allocator
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#720 from wesm/ARROW-1078 and squashes the following commits:
4bcf580e [Wes McKinney] Remove move_lib for parquet_arrow
9bd6c365 [Wes McKinney] Remove PARQUET_ARROW CMake flag in build fileS
47b38cd6 [Wes McKinney] Clean up after libparquet/libparquet_arrow consolidation
Author: Max Risuhin <risuhin.max@gmail.com>
Closes#707 from MaxRis/ARROW-1034 and squashes the following commits:
796d1554 [Max Risuhin] ARROW-1034: [PYTHON] Resolve wheel build issues on Windows
Miscellaneous fixes to build with ``--with-parquet`` and pass the test suite. We still have a bunch of compiler warnings; not blocking for 0.4.0, but after PARQUET-991 is resolved we should fix the MSVC compiler warnings in pyarrow
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#700 from wesm/ARROW-1029 and squashes the following commits:
b651169c [Wes McKinney] Fix post rebase issue
0f438488 [Wes McKinney] Fixes for building pyarrow with Parquet support on MSVC. Add to appveyor build
This mainly uses the same logic we already use for arrow-cpp
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#698 from xhochy/parquet-abi-version-bundling and squashes the following commits:
4aa17f82 [Uwe L. Korn] ARROW-1030: Python: Account for library versioning in parquet-cpp
…odules
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#660 from xhochy/ARROW-813 and squashes the following commits:
47459ba [Uwe L. Korn] Call git later
5b3b4a4 [Uwe L. Korn] Symlink fixes
70ca0d8 [Uwe L. Korn] ARROW-813: [Python] setup.py sdist must also bundle dependent cmake modules
Change-Id: Ic7fb227342782dfed5885f8fc5e73418fd31d504
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#563 from xhochy/ARROW-852 and squashes the following commits:
9630352 [Uwe L. Korn] Remove ARROW_HOME
5fc43ce [Uwe L. Korn] Always search for libs
This will avoid build failures due to a stale system-level Arrow install
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#558 from wesm/ARROW-845 and squashes the following commits:
4f89207 [Wes McKinney] Sync changes from PARQUET-955; explicit ARROW_HOME will override pkgconfig
This leaves c_glib in a broken state, since I'd rather let @kou fix up things the way he prefers (I started refactoring the automake files and realized I would probably just make a mess).
I'm going to submit a patch to make parquet-cpp work on top of this.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Author: Kouhei Sutou <kou@clear-code.com>
Closes#516 from wesm/ARROW-795 and squashes the following commits:
7221e66 [Kouhei Sutou] Update arrow-glib after codebase consolidation
207749c [Wes McKinney] Consolidate arrow/arrow_io/arrow_ipc into a single shared and static library
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#471 from xhochy/fix-manylinux1-build and squashes the following commits:
5259118 [Uwe L. Korn] Instead of copying, symlink to correct ABI versions
cffcca3 [Uwe L. Korn] Don't hardcode ABI versions
98f79af [Uwe L. Korn] Set PKG_CONFIG_PATH
b97bcf4 [Uwe L. Korn] Read SOVERSION from pkg-config
a83dbc8 [Uwe L. Korn] Build arrow-cpp for each Python version
This will enable third parties to link to `libarrow_python`.
For now, the pyarrow build system continues to use CMake -- for the purpose of resolving the thirdparty toolchain we may or may not want to go completely to distutils, but we can sort that out later.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#440 from wesm/ARROW-341 and squashes the following commits:
193bc51 [Wes McKinney] Ensure that '-undefined dynamic_lookup' is passed when linking shared library on OS X
a93496b [Wes McKinney] Add missing backslash
7620f50 [Wes McKinney] Fix cpplint issues
0617c69 [Wes McKinney] Fix LD_LIBRARY_PATH, ARROW_HOME
090c78c [Wes McKinney] Build Arrow library stack specific to active Python version
10e4626 [Wes McKinney] Get Python test suite passing again
cfb7f44 [Wes McKinney] Remove print statement
c1e63dc [Wes McKinney] Scrubbing python/CMakeLists.txt
b80b153 [Wes McKinney] Cleanup, build pandas-test within main test suite
7ef1f81 [Wes McKinney] Start moving python/src/pyarrow tp cpp/src/arrow/python
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#443 from wesm/ARROW-684 and squashes the following commits:
c18ca81 [Wes McKinney] More helpful error message if libparquet_arrow not built
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#315 from xhochy/ARROW-457 and squashes the following commits:
dc5abdb [Uwe L. Korn] Use aligned deallocator
20c8505 [Uwe L. Korn] ARROW-458: Python: Expose jemalloc MemoryPool
2962bd8 [Uwe L. Korn] ARROW-457: Python: Better control over memory pool
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#291 from wesm/ARROW-461 and squashes the following commits:
b3efe96 [Wes McKinney] Fix cpp unit test, code review comments
285f863 [Wes McKinney] Accept list input in Array.from_pandas
16aa9d6 [Wes McKinney] Add Categorical conversion for single array or column. Required moving code around a little bit in pandas.cc
3d409e8 [Wes McKinney] First round of DataFrame-level dictionary to Categorical conversion
0b20c38 [Wes McKinney] Draft Python wrapper classes for DictionaryType, DictionaryArray. Avoid segfault when conversion to UTF8 fails. Starting on CategoricalBlock implementation
This reintroduces `setuptools_scm` versioning for git clones and sdists/wheels. git-archives are handled by a separate chunk of code that will infer the version from the `pom.xml`
As in the Maven world, always the next to-be-released version is specified and Python PEP 440 development versions are based on the previous release, it used the most minimal pre-release version specified there.
I would suggest to keep the conda package versioning as it is currently, i.e. manually set it to 0.1.0postX.
Also: I would rather not parse the Maven XML but that is the most simple way currently to ensure that the versioning system is in a state where we can still make releases with the Maven release plugin.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#248 from xhochy/ARROW-430 and squashes the following commits:
39753f8 [Uwe L. Korn] Infer version from java/pom.xml
05c44ea [Uwe L. Korn] Get rid of setuptools_scm_git_archive
14b8136 [Uwe L. Korn] Revert "ARROW-429: Revert ARROW-379 until git-archive issues are resolved"
Change-Id: I4f6d291e63b2518af47c2a81049aa24a38c92821
This also adds some IO scaffolding for interacting with `arrow::Buffer` objects from Python and assorted additions to help with testing.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#164 from wesm/ARROW-312 and squashes the following commits:
7df3e5f [Wes McKinney] Set BUILD_WITH_INSTALL_RPATH on arrow_ipc
be8cee0 [Wes McKinney] Link Cython modules to libarrow* libraries
5716601 [Wes McKinney] Fix accidental deletion
77fb03b [Wes McKinney] Add / test Buffer wrapper. Test that we can write an arrow file to a wrapped buffer. Resize buffer in BufferOutputStream on close
316537d [Wes McKinney] Get ready to wrap Arrow buffers in a Python object
4822d32 [Wes McKinney] Implement RecordBatch::Equals, compare in Python ipc file writes
a931e49 [Wes McKinney] Permit buffers (write padding) in a non-multiple of 64 in an IPC context, to allow zero-copy writing of NumPy arrays
2c49cd4 [Wes McKinney] Some debugging
ca1562b [Wes McKinney] Draft implementations of Arrow file read/write from Python
Author: Christopher C. Aycock <christopher.aycock@twosigma.com>
Closes#155 from chrisaycock/ARROW-313 and squashes the following commits:
e47cc01 [Christopher C. Aycock] ARROW-313: Build on any version of XCode
This patch depends on PARQUET-728 (to run the full test suite, including pyarrow Parquet tests)
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#145 from wesm/ARROW-296 and squashes the following commits:
d67b4f9 [Wes McKinney] Refactor to link against parquet_arrow, fix up cmake files
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#132 from wesm/ARROW-284 and squashes the following commits:
e3410cf [Wes McKinney] Install miniconda in $HOME to avoid long prefix issues in conda-build 2.0
9fd94f5 [Wes McKinney] Do not run death test when valgrind is enabled. Gracefully skip pyarrow.parquet when ARROW_PARQUET=off
ccf56f8 [Wes McKinney] Disable arrow_parquet module in Travis CI
- Switch Travis CI back to Ubuntu trusty (old Boost in precise has issues with
C++11)
- Adapt SFrame libhdfs shim for arrow
- Create C++ public API within arrow:io to libhdfs
- Implement and test many functions in libhdfs
- Start Cython wrapper interface to arrow_io. Begin Python file-like interface,
unit tests
- Add thirdparty hdfs.h so builds are possible without a local Hadoop distro
(e.g. in Travis CI).
Change-Id: I4a46e50f6c1c22787baa3749d8a542216341e630
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#95 from xhochy/arrow-223 and squashes the following commits:
4fdf1e7 [Uwe L. Korn] ARROW-223: Do not link against libpython
Change-Id: I1238a48aaf94ab175b367551f74c335c6455d78a
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#83 from xhochy/arrow-203 and squashes the following commits:
405f85d [Uwe L. Korn] Remove FindParquet duplication
38d786c [Uwe L. Korn] Make code more readable by using using
ec07768 [Uwe L. Korn] Set LD_LIBRARY_PATH in python build
8d90d3f [Uwe L. Korn] Do not set LD_LIBRARY_PATH in python build
000e1e3 [Uwe L. Korn] Use unique_ptr and shared_ptr from Cython
8f6010a [Uwe L. Korn] Linter fixes
0514d01 [Uwe L. Korn] Handle exceptions on RowGroupWriter::Close better
77bd21a [Uwe L. Korn] Add pandas roundtrip to tests
f583b61 [Uwe L. Korn] Fix rpath for libarrow_parquet
00c1461 [Uwe L. Korn] Also ensure correct OSX compiler flags in PyArrow
4a80116 [Uwe L. Korn] Handle Python3 strings correctly
066c08a [Uwe L. Korn] Add missing functions to smart pointers
5706db2 [Uwe L. Korn] Use length and offset instead of slicing
443de8b [Uwe L. Korn] Add miniconda to the LD_LIBRARY_PATH
2dffc14 [Uwe L. Korn] Fix min mistake, use equals instead of ==
2006e70 [Uwe L. Korn] Rewrite test py.test style
9520c39 [Uwe L. Korn] Use PARQUET from miniconda path
cd3b9a9 [Uwe L. Korn] Also search for Parquet in PyArrow
6a41d23 [Uwe L. Korn] Re-use conda installation from C++
81f501e [Uwe L. Korn] No need to install conda in travis_script_python anymore
b505feb [Uwe L. Korn] Install parquet-cpp via conda
5d4929a [Uwe L. Korn] Add test-util.h
9b06e41 [Uwe L. Korn] Make tests templated
be6415c [Uwe L. Korn] Incorportate review comments
0fbed3f [Uwe L. Korn] Remove obsolete parquet files
081db5f [Uwe L. Korn] Limit and document chunk_size
7192cfb [Uwe L. Korn] Add const to slicing parameters
0463995 [Uwe L. Korn] ARROW-203: Python: Basic filename based Parquet read/write
Author: Dan Robinson <danrobinson010@gmail.com>
Closes#54 from danrobinson/ARROW-93 and squashes the following commits:
ddff5b0 [Dan Robinson] ARROW-93: Fix builds when using XCode 7.3
Current CMake FindPythonLibs also uses this option instead of NO_DEFAULT_PATH.
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#43 from xhochy/arrow-65 and squashes the following commits:
10eb9e0 [Uwe L. Korn] ARROW-65: Be less restrictive on PYTHON_LIBRARY search paths
Author: Wes McKinney <wesm@apache.org>
Closes#22 from wesm/ARROW-42 and squashes the following commits:
3b056a1 [Wes McKinney] Modularize Travis CI build and add Python build script. Remove parquet.pyx from Cython build for now, suppress -Wunused-variable in Cython compilation. Add missing formatting.py file
This patch provides no actual functionality; it only builds an empty Cython extension that links to libarrow.so. I will hook this into Travis CI at some later time.
I have adapted a limited amount of BSD (2- or 3-clause) or Apache 2.0 3rd-party code (particularly the cmake/Cython integration) to bootstrap this Python package / build setup in accordance with http://www.apache.org/legal/resolved.html. I have noted the relevant copyright holders and licenses in `python/LICENSE.txt`. In particular, I expect to continue to refactor and reuse occasional utility code from pandas (https://github.com/pydata/pandas) as practical.
Since a significant amount of "glue code" will need to be written to marshal between Arrow data and pure Python / NumPy / pandas objects, to get started I've adopted the approach used by libdynd/dynd-python -- a C++ "glue library" that is then called from Cython to provide a Python user interface. This will allow us to build shims as necessary to abstract away complications that leak through (for example: enabling C++ code with no knowledge of Python to invoke Python functions). Let's see how this goes: there are other options, like Boost::Python, but Cython + shim code is a more lightweight and flexible solution for the moment.
Author: Wes McKinney <wesm@apache.org>
Closes#17 from wesm/ARROW-7 and squashes the following commits:
be059a2 [Wes McKinney] Nest arrow::py namespace
3ad3143 [Wes McKinney] Add preliminary Python development toolchain
Wes McKinney