### Rationale for this change
`shields.io` for pyarrow is broken for some reasons for last few days specifically for several packages with failing a request timeout:
<img width="425" height="116" alt="Screenshot 2025-12-08 at 7 09 02 AM" src="https://github.com/user-attachments/assets/cca495d5-e34f-491e-95d0-b8595c306e60" />
Instead of relying on the thirdparty project, we should better use the official ones provided by Anaconda in https://anaconda.org/conda-forge/pyarrow/badges
### What changes are included in this PR?
Changes the URI for Conda badge to use the official one provided in Anaconda (https://anaconda.org/conda-forge/pyarrow/badges).
### Are these changes tested?
Yes, manually tested via my browser.
### Are there any user-facing changes?
No.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Update an outdated link.
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
Authored-by: TennyZhuang <zty0826@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The structure of the Python build and development documentation has become unclear due to multiple incremental changes over time. This PR aims to improve clarity and organization.
### What changes are included in this PR?
- Introduces tabs to improve navigation.
- Splits the existing documentation into two focused pages: Building and Developing PyArrow.
### Are these changes tested?
Yes. With docs preview build.
### Are there any user-facing changes?
No. Only docs changes.
* GitHub Issue: #20125
Authored-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
This is a temporary documentation fix for ARROW-6015. Please keep the JIRA issue open
Closes#5353 from wesm/ARROW-6015 and squashes the following commits:
d513f466b <Wes McKinney> Add note to python/README.md about installing Visual C++ Redistributable
Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>
This was pretty much a huge pain but addresses accumulated documentation debt after the conda compiler migration and the CMake refactor. I suggest we not stress too much over small details on this and do more work to improve these docs in follow up PRs. I did the best I could under the circumstances and need to move on to other things now
I think the overall organization of the Sphinx project for developers is much improved, take a look (I will post a link to a published version for review)
JIRAs addressed by this PR and other things I did
* Update cpp/thirdparty/README.md given CMake refactor (this was totally out of date). This now directs users to the Sphinx C++ developer guide
* ARROW-4339: Move cpp/README.md to Sphinx documentation (and clean it up a lot!!)
* ARROW-4425: Move Contributing Guidelines from Confluence to Sphinx, update top level README
* ARROW-4232: Remove references to pre-gcc5 ABI issues
* ARROW-4165: Move Windows C++ developer guide to Sphinx (from cpp/apidoc/Windows.md)
* ARROW-4547: Update Python development instructions re: producing CUDA-enabled pyarrow
* ARROW-4326 / ARROW-3096: Update Python build instructions re: January 2019 compiler migration
Author: Wes McKinney <wesm+git@apache.org>
Closes#3942 from wesm/developer-docs-0.13 and squashes the following commits:
a3c3dd5de <Wes McKinney> Add some Boost info, misc cleaning
2ccc3de18 <Wes McKinney> Remove index.md altogether
66da97e7f <Wes McKinney> Remove unused text from cpp/apidoc/index.md
504bc134e <Wes McKinney> restore 'what's in the arrow libraries' section
8d1f33e19 <Wes McKinney> Finish initial documentation revamp for 0.13, stopping here
84dd680a2 <Wes McKinney> Some docs reorg, begin rewriting cpp/README.md into docs/source/developers/cpp.rst
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#3249 from xhochy/ARROW-3932 and squashes the following commits:
8e969c1b <Uwe L. Korn> Link to Sphinx documentation for benchmarks
06c3b8d0 <Uwe L. Korn> ARROW-3932: Include Benchmarks.md in Sphinx docs
Author: Benjamin Kietzman <bengilgit@gmail.com>
Closes#3185 from bkietz/ARROW-4039-update-development-link and squashes the following commits:
a18596e7 <Benjamin Kietzman> Update README.md
Author: Kengo Seki <sekikn@apache.org>
Closes#3127 from sekikn/ARROW-3961 and squashes the following commits:
87d03e4a <Kengo Seki> ARROW-3961: Fix wrong path in the pyarrow README
Author: Uwe L. Korn <uwelk@xhochy.com>
Author: Korn, Uwe <Uwe.Korn@blue-yonder.com>
Author: Wes McKinney <wesm+git@apache.org>
Closes#2856 from xhochy/doc-merge and squashes the following commits:
5b687ffa7 <Wes McKinney> Add simple README for the format/ directory
071d16ad7 <Uwe L. Korn> Move format specifications back to /format/
337088bfd <Uwe L. Korn> Review comments
fbe99c96c <Uwe L. Korn> Add more C++ docs
78a5eaf4b <Uwe L. Korn> Fix Python docs build
0b4dd3341 <Uwe L. Korn> Rename doc to docs
918e76244 <Uwe L. Korn> Convert format docs to reST
7aeff65ef <Uwe L. Korn> Add doc generation to docker-compose
185cba8fb <Uwe L. Korn> Add pre-commit check for RAT
671d24466 <Uwe L. Korn> Fix references to format documents
bdd824c87 <Uwe L. Korn> Move doc to top-level
985d42806 <Uwe L. Korn> Move Sphinx doc to top-level directory
f7d5e92d9 <Uwe L. Korn> Build C++ API docs
7850db8a6 <Uwe L. Korn> Add breathe as a requirement
d4cf54259 <Uwe L. Korn> Fix linter issues
fd75660e6 <Korn, Uwe> Fix Sphinx build for sphinx>=1.8
9be6fbe6f <Korn, Uwe> Merge C++ and Python documentation
Author: Wes McKinney <wesm+git@apache.org>
Closes#2983 from wesm/python-readme-unit-testing and squashes the following commits:
fbdd23cc <Wes McKinney> Fix typos. Be more explicit about what is an in place build
b6e304f6 <Wes McKinney> Document how to run the Python unit tests in python/README.md
This is a partial C++ implementation of the Flight RPC system initially proposed by Jacques in ARROW-249. As in Java, I had to dig into gRPC and Protocol Buffers internals to ensure that
* On write, memory is only copied once into the outgoing gRPC buffer
* On read, no memory is copied
The way that I tricked gRPC into circumventing the built-in protobuf serde paths might look a bit hacky, but after digging around in the library a bunch I've convinced myself that it's the best and perhaps only way to accomplish this. Luckily, the message that's being serialized/deserialized is pretty opaque to the rest of the gRPC system, and it's controlled by the `SerializationTraits<T>` class. So you can take a gRPC stream reader and make it create any kind of type you want, even if the input data is a protocol buffer.
Some things that won't be addressed in this patch, as scope is too large:
* gRPC build toolchain issues (this is rather complex, I will create follow-up issues)
* Security / encryption, and authentication issues. I have only implemented an insecure server
* Integration with Travis CI
* Python bindings
API is preliminary and I expect to be the subject of iteration to make general and fast over the next several months.
Author: Wes McKinney <wesm+git@apache.org>
Closes#2547 from wesm/flight-cpp-prototype and squashes the following commits:
64bcdea43 <Wes McKinney> Initial Arrow Flight C++ implementation
system. Add unit test label granularity options, ability to add component group
targets like 'make parquet' that build libraries and tests
Change-Id: I250fd10da3a9505952115b3cec18cd7cb5589bdb
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#1630 from wesm/ARROW-2175 and squashes the following commits:
b6bdd3d2 [Wes McKinney] ARROW_HOME needed to override PARQUET_BUILD_TOOLCHAIN
119b290b [Wes McKinney] Test the behavior of a Python-only change
49e8706f [Wes McKinney] Just use PARQUET_BUILD_TOOLCHAIN [force travis] [skip appveyor]
cdd9929f [Wes McKinney] Always build Arrow libraries when either C++/Python affected. Add missing ARROW_HOME env var to Parquet build CI script. [force travis] [skip appveyor]
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#900 from wesm/fix-license-headers and squashes the following commits:
73941bf2 [Wes McKinney] Fix many license headers to use proper ASF one
Also migrates DEVELOPMENT.md to the Sphinx docs
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#584 from wesm/ARROW-862 and squashes the following commits:
50049dd [Wes McKinney] Revise python/README.md. Move DEVELOPMENT.md to Sphinx docs. Other cleaning
2187c1c [Wes McKinney] Migrate DEVELOPMENT.md to sphinx docs
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#550 from wesm/ARROW-830 and squashes the following commits:
c1ca9fb [Wes McKinney] Expose jemalloc memory pool and other memory pool functions in public pyarrow API
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#445 from wesm/ARROW-716 and squashes the following commits:
2608d2b [Wes McKinney] Update README after moving libpyarrow to main C++ source tree
Added note to use LD_LIBRARY_PATH env var to add $ARROW_HOME/lib path so PyArrow can locate Arrow-Cpp shared libs.
Author: Bryan Cutler <cutlerb@gmail.com>
Closes#199 from BryanCutler/pyarrow-README-note-LD_LIBRARY_PATH-ARROW-368 and squashes the following commits:
15861c4 [Bryan Cutler] Added note for LD_LIBRARY_PATH in Python README
Added a couple ways to do this, either via the `--with-parquet` command line option (preferred) or by passing through an option to CMake
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#194 from wesm/ARROW-323 and squashes the following commits:
07c05cc [Wes McKinney] Update readme to illustrate proper use of with build_ext
3bd9a8d [Wes McKinney] Add --with-parquet option to setup.py
374e254 [Wes McKinney] Add to README about building the parquet extension
cab55cb [Wes McKinney] Opt in to building the pyarrow.parquet extension, do not silently fail
This also resolves ARROW-230.
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#187 from wesm/ARROW-348 and squashes the following commits:
3cdaeaf [Wes McKinney] Cast build_type to lowercase in case env variable is uppercase
74bfa71 [Wes McKinney] Pull default build type from environment variable
d0b3154 [Wes McKinney] Tweak readme
6017948 [Wes McKinney] Add built-type command line option to setup.py, build extensions in release type subdirectory to avoid conflicts with setuptools
Author: Uwe L. Korn <uwelk@xhochy.com>
Closes#175 from xhochy/ARROW-97 and squashes the following commits:
2ec3e11 [Uwe L. Korn] Add license headers
d838e81 [Uwe L. Korn] ARROW-97: API documentation via sphinx-apidoc
Author: Julien Le Dem <julien@dremio.com>
Closes#159 from julienledem/fix_licenses and squashes the following commits:
0c97810 [Julien Le Dem] fix NOTICE
1489289 [Julien Le Dem] more licenses
0eb2aeb [Julien Le Dem] more licenses
9ac1159 [Julien Le Dem] more licenses
eafa0e1 [Julien Le Dem] more licenses
30b0fa1 [Julien Le Dem] more licenses
bcfc75f [Julien Le Dem] add ci
51db31b [Julien Le Dem] ARROW-321: fix arrow licenses
Also removes a redundant LICENSE.txt from cpp/
Author: Wes McKinney <wes.mckinney@twosigma.com>
Closes#157 from wesm/ARROW-318 and squashes the following commits:
9e802f2 [Wes McKinney] Update python/README.md. Remove redundant LICENSE.txt from cpp/
This patch provides no actual functionality; it only builds an empty Cython extension that links to libarrow.so. I will hook this into Travis CI at some later time.
I have adapted a limited amount of BSD (2- or 3-clause) or Apache 2.0 3rd-party code (particularly the cmake/Cython integration) to bootstrap this Python package / build setup in accordance with http://www.apache.org/legal/resolved.html. I have noted the relevant copyright holders and licenses in `python/LICENSE.txt`. In particular, I expect to continue to refactor and reuse occasional utility code from pandas (https://github.com/pydata/pandas) as practical.
Since a significant amount of "glue code" will need to be written to marshal between Arrow data and pure Python / NumPy / pandas objects, to get started I've adopted the approach used by libdynd/dynd-python -- a C++ "glue library" that is then called from Cython to provide a Python user interface. This will allow us to build shims as necessary to abstract away complications that leak through (for example: enabling C++ code with no knowledge of Python to invoke Python functions). Let's see how this goes: there are other options, like Boost::Python, but Cython + shim code is a more lightweight and flexible solution for the moment.
Author: Wes McKinney <wesm@apache.org>
Closes#17 from wesm/ARROW-7 and squashes the following commits:
be059a2 [Wes McKinney] Nest arrow::py namespace
3ad3143 [Wes McKinney] Add preliminary Python development toolchain
Hyukjin Kwon