### Rationale for this change
Now that we have a general security model for the Arrow specs, add a security model for Arrow C++ specifically, meant to describe appropriate API usage.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
* GitHub Issue: #49274
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
Parameterizing the comparison of `metadata` and `arrow::Schema` via `arrow::EqualOptions`.
### What changes are included in this PR?
* Added two attributes to `arrow::EqualOptions` for parameterizing the comparison of `metadata` and `schema`.
### Are these changes tested?
Yes, I ran the relevant unit tests.
### Are there any user-facing changes?
* Added `use_schema()` to `arrow::EqualOptions`.
* Added `use_metadata()` to `arrow::EqualOptions`.
* Added a new overload for `arrow::RecordBatch::Equals` that accepts only another `arrow::RecordBatch` and an `arrow::EqualOptions` instance.
* GitHub Issue: #46835
Lead-authored-by: Arash Andishgar <arashandishgar1@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The docs don't build cleanly for me locally and some warnings and errors that are consequential are being ignored. Some of these issues ended up being as severe as entirely missing tables.
### What changes are included in this PR?
- Fixes for each issue I ran into. Each change is a commit with whatever extra detail I needed included in the commit body.
- This also includes an auto-update of our Doxyfile which silences a number of deprecation warnings. I took a look at the diff and it seemed pretty safe and the docs look good to me locally. We can do a preview in CI first though.
### Are these changes tested?
Yes, each change was tested locally.
### Are there any user-facing changes?
No.
* GitHub Issue: #46520
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
The [Array](https://arrow.apache.org/docs/cpp/api/array.html) section of the API Doc
does not explain `FlatArray`, `PrimitiveArray`, and `RunEndEncodedArray` .
### What changes are included in this PR?
Add `FlatArray`, `PrimitiveArray`, and `RunEndEncodedArray` API docs
### Are these changes tested?
No. (Document only)
### Are there any user-facing changes?
No.
* GitHub Issue: #46537
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
These functions are generally useful and stable so it would be a good idea to clearly include them in the public API. I'm starting here with just the basic ones in order to make the PR small. See https://github.com/apache/arrow/issues/45908 for more information.
### What changes are included in this PR?
- Moves `ArrayFromJSON`, `ChunkedArrayFromJSON`, `DictArrayFromJSON`, `ScalarFromJSON`, `DictScalarFromJSON` from `arrow::ipc::internal` namespace to `arrow::json` so it's clearer they're part of the public API and that they're more useful than just for IPC
- Renames each of the above from `{Array,...}FromJSON` to `{Array,...}FromJSONString` to avoid confusion between these helpers and the main JSON(L) reader
- Renames `arrow/util/json_simple.{h,cc}` to `arrow/json/from_string.{h,cc}` both because of the namespace jump but also because the filename is more clear.
- Adds User Guide and adds a listing in the API docs for moved functions
### Are these changes tested?
Yes.
### Are there any user-facing changes?
This expands the scope of our public API but does not break any existing public APIs though it's possible users
* GitHub Issue: #45908
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Co-authored-by: Enrico Minack <github@enrico.minack.dev>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
There are no Extension Array/Type documentation in the [API Reference page](https://arrow.apache.org/docs/cpp/api.html).
### What changes are included in this PR?
Add Extension Array / Type API documents.
### Are these changes tested?
No. (Add document only)
### Are there any user-facing changes?
No.
* GitHub Issue: #46318
Authored-by: Hiroyuki Sato <hiroysato@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
https://arrow.apache.org/docs/cpp/api/array.html should have `arrow::ArrayStatistics`.
### What changes are included in this PR?
Add missing `\class` and `doxygenclass`.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #44710
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
Since ChunkLocation is a typedef/alias, it needs to use the `doxygentypedef` Sphinx directive rather than `doxygenstruct` in order for it to show up in the generated HTML. Otherwise we get an error.
### What changes are included in this PR?
Just docs:
- Replace `doxygenstruct` directive with `doxygentypedef`
- Remove `:members:` directive as it's not needed
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes, though just docs.
* GitHub Issue: #44506
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
Adopting https://github.com/apache/arrow/pull/40226.
The creation and return of a shared_ptr does result in some performance overhead, that makes a difference for a performance-sensitive application.
If someone could use ChunkResolver to learn the indices, they could then instead access the data directly.
### What changes are included in this PR?
- [X] Updates to documentation (thanks to @ SChakravorti21 )
- [X] Moving `ChunkResolver` to public API, and updating all references to it in the code
### Are these changes tested?
There seemed to be comprehensive tests already: https://github.com/apache/arrow/blob/main/cpp/src/arrow/chunked_array_test.cc#L324 If an edgecase is missing, I'd be happy to add it.
### Are there any user-facing changes?
`ChunkResolver` and `TypedChunkLocation` are now in the public API.
* GitHub Issue: #34535
Lead-authored-by: Anja Kefala <anja.kefala@gmail.com>
Co-authored-by: anjakefala <anja.kefala@gmail.com>
Co-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: SChakravorti21 <schakravorti@bloomberg.net>
Co-authored-by: SChakravorti21<schakravorti@bloomberg.net>
Co-authored-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
Update documentation, protobufs, and class documentation to remove experimental tags from Flight and Flight SQL documentation.
### Rationale for this change
Flight SQL has been used by multiple databases now and has been voted as stable per the
mailing list discussion: [https://lists.apache.org/thread/qoshg8mln3t2ovr90o1yklz4yrpv503h](url)
### What changes are included in this PR?
Update protobuf, class comments, and user documentation to remove references to Flight and Flight SQL being experimental. This change excludes the UCX transport and the session option messages
### Are these changes tested?
No, documentation only.
### Are there any user-facing changes?
User documentation.
* GitHub Issue: #39204
Authored-by: James Duong <james.duong@improving.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
See https://github.com/apache/arrow/issues/41410.
### What changes are included in this PR?
Just changes to filesystem.rst.
### Are these changes tested?
Yes, locally.
### Are there any user-facing changes?
These are those changes.
* GitHub Issue: #41410
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### What changes are included in this PR?
This adds developer tooling to the repo for linting the docs by adding the sphinx-lint tool to archery and our pre-commit hooks. In both locations, only two rules are enabled at the moment (Discussed in https://github.com/apache/arrow/pull/40006): `trailing-whitespace` and `missing-final-newline`.
This PR also fixes the individual issues covered by the new tooling.
### Are these changes tested?
Yes, though manually. I tested this works by running `archery lint --docs` and `pre-commit` without and without changes that should get caught by the rules. It works as expected.
### Are there any user-facing changes?
Yes,
1. Developers that use pre-commit hooks will see a change in behavior when they modify docs
2. Developers using archery will see a new --docs option in `archery lint`
3. Developers working on the docs may see CI failures related to the new checks
* Closes: #39990
* GitHub Issue: #39990
Authored-by: Bryce Mecum <petridish@gmail.com>
Signed-off-by: Bryce Mecum <petridish@gmail.com>
### Rationale for this change
Each filesystem implementation carries unique and potentially heavy dependencies, so it'd be useful to build them separately. Furthermore, one typically doesn't need all of them at the same time and building separate modules would allow them to be dynamically loaded as necessary. Finally, defining this interface allows custom filesystem implementations to be supported seamlessly.
### What changes are included in this PR?
An initial sketch of a registry, with documentation as if the registry were complete to illustrate intended usage.
### Are these changes tested?
A toy module is added and a single unit test too.
### Are there any user-facing changes?
Users would be able to add their own filesystem implementations to the registry
* Closes: #38309
Authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>
### Rationale for this change
These factory functions are generally useful and available, so documenting them helps external users find them without having to search the source code.
### What changes are included in this PR?
This PR adds the array factories in arrow/array/util.h into a doxygen group for array factories and adds that group to the Sphinx C++ API documentation.
### Are these changes tested?
I built the docs locally to verify.
### Are there any user-facing changes?
Nothing to the API, only docs.
* GitHub Issue: #40318
Authored-by: Vyas Ramasubramani <vyasr@nvidia.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>
### Rationale for this change
It's useful when troubleshooting issues with Arrow's S3 filesystem implementation to raise the log level. Currently, this can only be done in C++ and Python, but not from R. In addition, the log level can only be set during S3 initialization and not directly so the user has to introduce explicit S3 initialization code to turn on logging and must make sure this code is called before S3 initialization.
While discussing exposing control of log level to R, we realized that allowing the log level to be controlled by environment variable may be more intuitive and useful and would just be a good addition for C++, Python, and R.
### What changes are included in this PR?
- A new environment variable `AWS_S3_LOG_LEVEL` with documentation for controlling S3 log level
- Updated documentation for C++, Python, and R
- A new `InitializeS3()` as a quality-of-life thing for C++ users. Feel free to ask me to remove this.
No changes are needed directly for Python and R because these implementation uses the internal implicit initializer `EnsureS3Initialized` rather than the explicit form, `InitializeS3`. And it's the behavior of the `EnsureS3Initialized` routine that's changed here.
### Are these changes tested?
Yes. I added a unit test for the new `GetS3LogLevelFromEnvOrDefault` and tested from Python and R manually. I didn't add a test to make sure the underlying `AwsInstance` gets set up correctly because it looked like it would require a refactor and didn't seem worth it.
### Are there any user-facing changes?
Yes. A new way to turn on logging for S3 and matching docs in C++, Python, and R.
* Closes: #35260
Lead-authored-by: Bryce Mecum <petridish@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Nic Crane <thisisnic@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change
The documentation for Acero was incomplete. This PR refactors the existing documentation and adds several entirely new sections to form a complete design document for Acero.
### What changes are included in this PR?
Some existing documentation is cleaned up. Acero documentation is moved into its own folder and broken into several pages.
### Are these changes tested?
The documentation is built as part of the CI but I wouldn't say it is fully tested.
### Are there any user-facing changes?
There are not code changes (other than the removal of two legacy methods) but there are many user-facing documentation changes.
* Closes: #32335
Lead-authored-by: Weston Pace <weston.pace@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
This PR adds corrections to the source docs to remove some of the warnings when building the documentation.
* Closes: #35438
Authored-by: Alenka Frim <frim.alenka@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
This PR includes a documentation update to visibility of `exec.h` components.
Authored-by: Vibhatha Abeykoon <vibhatha@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
This reworks the Flight protocol documentation and the C++ Flight documentation, and adds a brief Python documentation page. It also fixes some warnings and errors I found.
Closes#12815 from lidavidm/arrow-16065
Lead-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
This improves the protocol docs and adds C++ API documentation. Java already has Javadocs so this PR doesn't touch anything there.
Closes#12616 from lidavidm/arrow-14698
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
## Summary of Changes
* Adds `VisitArrayInline` and `VisitScalarInline` to the public API
* Adds API documentation for inline visit functions, type traits, and type trait predicates.
* Adds type traits user guide to datatypes.rst
* Moves visitor user guide to datatypes.rst (this section was added after the 7.0.0, so changing it before the 8.0.0 release shouldn't break any links)
Closes#12717 from wjones127/ARROW-15975-inline-visitors
Lead-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.
For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of `strftime()` and friends.
Closes#12536 from wjones127/ARROW-13168-timezone-database
Authored-by: Will Jones <willjones127@gmail.com>
Signed-off-by: Jonathan Keane <jkeane@gmail.com>
Defines interfaces to add new network transports to Arrow Flight, and rewrite the gRPC implementation in terms of these interfaces. The bulk of the new APIs are in `transport.h`. Where possible, code has been factored out so that transports have to implement as little as possible.
Closes#12465 from lidavidm/arrow-15282
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Yibo Cai <yibo.cai@arm.com>
This PR's objective is to create a detailed documentation on using ExecPlan and available options. The task is divided into two sub-components. The first component is to work on a set of examples showing how to use these options with sample data.
## Examples
- [x] Source-Sink
- [x] Scan-Sink
- [x] Scan-Filter-Sink
- [x] Scan-Project-Sink
- [x] Source-Aggregate-Sink
- [x] Scan-ConsumingSink
- [x] Scan-OrderBySink
- [x] Scan-HashJoin
- [x] Scan-SelectSink
- [x] Scan-Filter-Write
- [x] Scan-Union-Sink
## Documentation
- [x] General Description
- [x] Option Description with code snippets
## Notes
Some of the util functions were referred from the `test_utils`.
There is a potential duplication of code due to that. Need
feedback on best way to handle this.
Closes#12033 from vibhatha/arrow-15091
Lead-authored-by: Vibhatha Abeykoon <vibhatha@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
Add some classes or functions that were missing.
Some parts of the public API, such as pretty-printing, are still uncovered and should be handled in individual JIRAs.
Closes#12273 from pitrou/ARROW-15435-cpp-api-docs-coverage
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This fixes a number of warnings emitted when building the Sphinx docs.
Closes#12242 from pitrou/ARROW-15433-doc-warnings
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also ensure that all concrete datatype and array classes are documented.
Closes#11201 from pitrou/ARROW-14056-doc-array-data
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
This is mostly based on the Python documentation. This also adds a new C++ example to accompany the documentation.
Closes#9810 from lidavidm/arrow-11677
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
This doesn't fix all warnings (some are due to limitations of Numpydoc, autodoc and Sphinx).
Closes#9848 from pitrou/ARROW-12089-sphinx-warnings
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also other minor doc improvements, and add a test.
Closes#9784 from pitrou/ARROW-12067-doc-wrap-scalar
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also add some tables of contents here and there for long nested documents.
Closes#8914 from pitrou/ARROW-10887-doc-ipc-api
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
The goal is to have a standardized ABI to communicate streams of homogeneous arrays or record batches (for example for database result sets).
The trickiest part is error reporting. This proposal tries to strike a compromise between simplicity (an integer error code mapping to errno values) and expressivity (an optional description string for application-specific and context-specific details).
Closes#8052 from pitrou/ARROW-9761-c-array-stream
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Followup from PR #7755: a new doc file wasn't checked in, and Sphinx only emits a warning.
Closes#7760 from pitrou/ARROW-9390-missing-file
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
Also fix glaring bugs in arithmetic kernels
(signed overflow detection was broken).
Closes#7695 from pitrou/ARROW-8989-doc-compute-functions
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
Use Result-returning AllocateBuffer() version in example.
Also improve cross-referencing in some places.
Closes#7731 from pitrou/ARROW-9428-buffer-allocation-doc
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Wes McKinney <wesm@apache.org>
If we use FileInfo instead of FileStats, we can use singular form
"info" and plural form "infos" as variable names instead of "stats"
and "stats_vector". It will help writing readable code.
Closes#6514 from kou/cpp-file-system-stat and squashes the following commits:
de69164ea <Sutou Kouhei> Rename fs::FileStats to fs::FileInfo
Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Closes#6454 from pitrou/ARROW-7879-device-api-doc and squashes the following commits:
9c51a4cb2 <Antoine Pitrou> Improve docstring for Buffer::View
7f7302d40 <Antoine Pitrou> Some rewrites
9d34ea3f7 <Antoine Pitrou> More changes
690ee03dd <Antoine Pitrou> ARROW-7879: Add doc for the Device API
Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>
tadeja