mirror of
https://github.com/apache/arrow.git
synced 2026-10-29 00:56:31 +08:00
paultiq
120b0b760d
### Rationale for this change This PR moves crossbow builds to short-lived S3 credentials. It does this via GitHub's OIDC provider and an AWS ROLE_ARN with a trust relationship configured. This configuration is detailed here: - [Use IAM roles to connect GitHub Actions to actions in AWS](https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/) The benefits of short-lived / temporary credentials discussed further here: [AWS Well-architected Framework Best Practices SEC02-BP02 Use temporary credentials](https://docs.aws.amazon.com/wellarchitected/latest/framework/sec_identities_unique.html) sccache supports a variety of S3 creds: [S3 credentials](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials) ### What changes are included in this PR? Use [AssumeRoleWithWebIdentity](https://github.com/mozilla/sccache/blob/main/docs/S3.md#credentials) for sccache S3 credentials. By assuming the AWS_ROLE_ARN, the sccache session relies on credentials that expire\*, issued only to workflows in the trusted repository, and scoped to the sccache bucket. Specific tasks: - create a role in IAM with the bucket and object level permissions, - trust the github oidc-provider, - define AWS_ROLE_ARN and set id-token permission - grant sts:AssumeRoleWithWebIdentity for the specific repositories or organization. - pass AWS_SESSION_TOKEN - add a `aws-actions/configure-aws-credentials@ v6` step to the workflow. Separately, someone will need to add the role and trust to AWS: - Procedure is in https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/ - Set a "Maximum session duration" to maximum runtime of any job... maybe 4 hours? 8 hours? - Permissions: Role needs BucketLevel and ObjectLevels permissions to the bucket. \* I can share an example if needed ### Are these changes tested? Yes, locally and on a fork. ### Are there any user-facing changes? No * GitHub Issue: #50899 Authored-by: paul@iqmo.com <paul@iqmo.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
89 lines
2.9 KiB
YAML
89 lines
2.9 KiB
YAML
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
{% import 'macros.jinja' as macros with context %}
|
|
|
|
{{ macros.github_header() }}
|
|
|
|
permissions:
|
|
contents: read
|
|
# For OIDC token for sccache
|
|
id-token: write
|
|
|
|
jobs:
|
|
test:
|
|
name: |
|
|
Docker Test {{ flags|default("") }} {{ image }} {{ command|default("") }}
|
|
runs-on: ubuntu-latest
|
|
{{ macros.github_set_env(env) }}
|
|
timeout-minutes: {{ timeout|default(60) }}
|
|
steps:
|
|
{{ macros.github_checkout_arrow(fetch_depth=fetch_depth|default(1))|indent }}
|
|
{{ macros.github_free_space()|indent }}
|
|
{{ macros.github_install_archery()|indent }}
|
|
|
|
{{ macros.github_configure_aws_credentials()|indent }}
|
|
- name: Execute Docker Build
|
|
shell: bash
|
|
env:
|
|
{{ macros.github_set_sccache_envvars()|indent(8) }}
|
|
run: |
|
|
# GH-40558: reduce ASLR to avoid TSAN crashing
|
|
sudo sysctl -w vm.mmap_rnd_bits=28
|
|
source arrow/ci/scripts/util_enable_core_dumps.sh
|
|
archery docker run \
|
|
-e SETUPTOOLS_SCM_PRETEND_VERSION="{{ arrow.no_rc_version }}" \
|
|
{{ flags|default("") }} \
|
|
{{ image }} \
|
|
{{ command|default("") }}
|
|
|
|
- name: Dump R install logs
|
|
if: always()
|
|
run: |
|
|
log=arrow/r/check/arrow.Rcheck/00install.out
|
|
if [ -f ${log} ]; then
|
|
cat ${log}
|
|
fi
|
|
- name: Dump R test logs
|
|
if: always()
|
|
run: |
|
|
for log in arrow/r/check/arrow.Rcheck/tests/testthat.Rout*; do
|
|
if [ -f ${log} ]; then
|
|
echo ${log}
|
|
cat ${log}
|
|
fi
|
|
done
|
|
- name: Save the R test output
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: test-output
|
|
path: |
|
|
arrow/r/tests/
|
|
arrow/r/arrow.Rcheck/
|
|
!arrow/r/arrow.Rcheck/00_pkg_src/
|
|
!arrow/r/arrow.Rcheck/arrow/
|
|
if-no-files-found: ignore
|
|
|
|
{% if arrow.is_default_branch() %}
|
|
{{ macros.github_login_dockerhub()|indent }}
|
|
- name: Push Docker Image
|
|
if: {{ push|default("true") }}
|
|
shell: bash
|
|
run: archery docker push {{ image }}
|
|
{% endif %}
|