mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2026-10-28 16:18:23 +08:00
Brandon Pereira
18277b71c9
The HyperDX in-app onboarding mirrors the SDK guides under
docs/clickstack/ingesting-data/sdks/ from this single source of truth, so it
parses them by convention. This standardizes those conventions and adds a
scoped validator so a drifting page cannot silently break the app's parsing.
- Add a `<ClickStackIntegrates signals={[...]} />` component (new snippet at
docs/snippets/components/ClickStackIntegrates/) that renders the
"This guide integrates" indicator as ClickHouse-yellow pills for the
supported signals. The `signals` prop (logs/metrics/traces) is the single
source of truth the app parses; it replaces the drifting hand-written blocks
(HTML tables and ad-hoc bullet lists across the pages).
- Standardize replaceable env-var placeholders: `<YOUR_INGESTION_API_KEY>`
(was also bare, `***`-wrapped, and `<YOUR_INGESTION_KEY>`),
`<YOUR_OTEL_COLLECTOR_HTTP_ENDPOINT>` for the OTLP endpoint (was
localhost:4318 / your-otel-collector:4318, and fixes the
`<YOU_OTEL_COLLECTOR_HTTP_ENDPOINT>` typo), and `<MY_SERVICE_NAME>` for the
service name (was `<NAME_OF_YOUR_APP_OR_SERVICE>` / `<YOUR_APP_NAME>`).
- Fix a stray newline in the index keywords.
- Add ci/jobs/scripts/docs/sdk_docs_check.py enforcing all of the above: the
ClickStackIntegrates component + valid signals, an allowlist of `<UPPER_SNAKE>`
placeholders (so a new typo/variant is caught generically), and exact
deployment tab titles (Managed ClickStack / ClickStack Open Source). It is
wired into the existing "Docs check (Mintlify)" job and gated to run only when
an SDK page, the component, or the validator changes.
Closes: https://linear.app/clickhouse/issue/HDX-5262/standardize-clickstack-sdk-docs-for-in-app-onboarding
348 lines
8.1 KiB
Plaintext
348 lines
8.1 KiB
Plaintext
---
|
|
slug: /use-cases/observability/clickstack/sdks/aws_lambda
|
|
description: 'AWS Lambda for ClickStack - The ClickHouse Observability Stack'
|
|
title: 'AWS Lambda'
|
|
doc_type: 'guide'
|
|
keywords: ['ClickStack', 'observability', 'aws-lambda', 'lambda-layers']
|
|
---
|
|
|
|
import { ClickStackIntegrates } from '/snippets/components/ClickStackIntegrates/ClickStackIntegrates.jsx'
|
|
|
|
<ClickStackIntegrates signals={['logs', 'metrics', 'traces']} />
|
|
|
|
## Installing the OpenTelemetry Lambda layers {#installing-the-otel-lambda-layers}
|
|
|
|
The OpenTelemetry project provides separate lambda layers to:
|
|
|
|
1. Automatically instrument your Lambda function code with OpenTelemetry auto-instrumentation.
|
|
2. Forward the collected logs, metrics, and traces to ClickStack.
|
|
|
|
### Adding the language-specific auto-instrumentation layer {#adding-language-specific-auto-instrumentation}
|
|
|
|
The language-specific auto-instrumentation lambda layers automatically instrument your Lambda function code with OpenTelemetry auto-instrumentation package for your specific language.
|
|
|
|
Each language and region has its own layer ARN.
|
|
|
|
If your Lambda is already instrumented with an OpenTelemetry SDK, you can skip this step.
|
|
|
|
**To get started**:
|
|
|
|
1. In the Layers section click "Add a layer"
|
|
2. Select specify an ARN and choose the correct ARN based on language, ensure you replace the `<region>` with your region (ex. `us-east-2`):
|
|
|
|
<Tabs>
|
|
<Tab title="Javascript">
|
|
|
|
```shell
|
|
arn:aws:lambda:<region>:184161586896:layer:opentelemetry-nodejs-0_7_0:1
|
|
```
|
|
|
|
</Tab>
|
|
<Tab title="Python">
|
|
|
|
```shell copy
|
|
arn:aws:lambda:<region>:184161586896:layer:opentelemetry-python-0_7_0:1
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="Java">
|
|
|
|
```shell copy
|
|
arn:aws:lambda:<region>:184161586896:layer:opentelemetry-javaagent-0_6_0:1
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="Ruby">
|
|
|
|
```shell copy
|
|
arn:aws:lambda:<region>:184161586896:layer:opentelemetry-ruby-0_1_0:1
|
|
```
|
|
|
|
</Tab>
|
|
|
|
</Tabs>
|
|
|
|
_The latest releases of the layers can be found in the [OpenTelemetry Lambda Layers GitHub repository](https://github.com/open-telemetry/opentelemetry-lambda/releases)._
|
|
|
|
3. Configure the following environment variables in your Lambda function under "Configuration" > "Environment variables".
|
|
|
|
<Tabs>
|
|
<Tab title="Javascript">
|
|
|
|
```shell
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
|
|
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
|
|
OTEL_PROPAGATORS=tracecontext
|
|
OTEL_TRACES_SAMPLER=always_on
|
|
```
|
|
|
|
</Tab>
|
|
<Tab title="Python">
|
|
|
|
```shell
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
|
|
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
|
|
OTEL_PROPAGATORS=tracecontext
|
|
OTEL_TRACES_SAMPLER=always_on
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="Java">
|
|
|
|
```shell
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
|
|
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
|
|
OTEL_PROPAGATORS=tracecontext
|
|
OTEL_TRACES_SAMPLER=always_on
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="Ruby">
|
|
|
|
```shell
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
|
|
AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
|
|
OTEL_PROPAGATORS=tracecontext
|
|
OTEL_TRACES_SAMPLER=always_on
|
|
```
|
|
|
|
</Tab>
|
|
|
|
</Tabs>
|
|
|
|
### Installing the OpenTelemetry collector Lambda layer {#installing-the-otel-collector-layer}
|
|
|
|
The collector Lambda layer allows you to forward logs, metrics, and traces from your Lambda function to ClickStack without impacting response times due to exporter latency.
|
|
|
|
**To install the collector layer**:
|
|
|
|
1. In the Layers section click "Add a layer"
|
|
2. Select specify an ARN and choose the correct ARN based on architecture, ensure you replace the `<region>` with your region (ex. `us-east-2`):
|
|
|
|
<Tabs>
|
|
|
|
<Tab title="x86_64">
|
|
|
|
```shell
|
|
arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-amd64-0_8_0:1
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="arm64">
|
|
|
|
```shell
|
|
arn:aws:lambda:<region>:184161586896:layer:opentelemetry-collector-arm64-0_8_0:1
|
|
```
|
|
|
|
</Tab>
|
|
|
|
</Tabs>
|
|
|
|
3. Add the following `collector.yaml` file to your project to configure the collector to send to ClickStack:
|
|
|
|
<Tabs>
|
|
<Tab title="Managed ClickStack">
|
|
|
|
```yaml
|
|
# collector.yaml
|
|
receivers:
|
|
otlp:
|
|
protocols:
|
|
grpc:
|
|
endpoint: 'localhost:4317'
|
|
http:
|
|
endpoint: 'localhost:4318'
|
|
|
|
processors:
|
|
batch:
|
|
decouple:
|
|
|
|
exporters:
|
|
otlphttp:
|
|
endpoint: "<YOUR_OTEL_COLLECTOR_HTTP_ENDPOINT>"
|
|
compression: gzip
|
|
|
|
service:
|
|
pipelines:
|
|
traces:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp]
|
|
metrics:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp]
|
|
logs:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp]
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="ClickStack Open Source">
|
|
|
|
```yaml
|
|
# collector.yaml
|
|
receivers:
|
|
otlp:
|
|
protocols:
|
|
grpc:
|
|
endpoint: 'localhost:4317'
|
|
http:
|
|
endpoint: 'localhost:4318'
|
|
|
|
processors:
|
|
batch:
|
|
decouple:
|
|
|
|
exporters:
|
|
otlphttp:
|
|
endpoint: "<YOUR_OTEL_COLLECTOR_HTTP_ENDPOINT>"
|
|
headers:
|
|
authorization: <YOUR_INGESTION_API_KEY>
|
|
compression: gzip
|
|
|
|
service:
|
|
pipelines:
|
|
traces:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp]
|
|
metrics:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp]
|
|
logs:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp]
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
4. Add the following environment variable:
|
|
|
|
```shell
|
|
OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/collector.yaml
|
|
```
|
|
|
|
## Checking the installation {#checking-the-installation}
|
|
|
|
After deploying the layers, you should now see traces automatically
|
|
collected from your Lambda function in HyperDX. The `decouple` and `batching`
|
|
processor may introduce a delay in telemetry collection, so traces may be
|
|
delayed in showing up. To emit custom logs or metrics, you'll need to instrument your code your language-specific
|
|
OpenTelemetry SDKs.
|
|
|
|
## Troubleshooting {#troubleshoting}
|
|
|
|
### Custom instrumentation not sending {#custom-instrumentation-not-sending}
|
|
|
|
If you're not seeing your manually defined traces or other telemetry, you may
|
|
be using an incompatible version of the OpenTelemetry API package. Ensure your
|
|
OpenTelemetry API package is at least the same or lower version than the
|
|
version included in the AWS lambda.
|
|
|
|
### Enabling SDK debug logs {#enabling-sdk-debug-logs}
|
|
|
|
Set the `OTEL_LOG_LEVEL` environment variable to `DEBUG` to enable debug logs from
|
|
the OpenTelemetry SDK. This will help ensure that the auto-instrumentation layer
|
|
is correctly instrumenting your application.
|
|
|
|
### Enabling collector debug logs {#enabling-collector-debug-logs}
|
|
|
|
To debug collector issues, you can enable debug logs by modifying your collector
|
|
configuration file to add the `logging` exporter and setting the telemetry
|
|
log level to `debug` to enable more verbose logging from the collector lambda layer.
|
|
|
|
<Tabs>
|
|
<Tab title="Managed ClickStack">
|
|
|
|
```yaml
|
|
# collector.yaml
|
|
receivers:
|
|
otlp:
|
|
protocols:
|
|
grpc:
|
|
endpoint: 'localhost:4317'
|
|
http:
|
|
endpoint: 'localhost:4318'
|
|
|
|
exporters:
|
|
logging:
|
|
verbosity: detailed
|
|
otlphttp:
|
|
endpoint: "<YOUR_OTEL_COLLECTOR_HTTP_ENDPOINT>"
|
|
compression: gzip
|
|
|
|
service:
|
|
telemetry:
|
|
logs:
|
|
level: "debug"
|
|
pipelines:
|
|
traces:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp, logging]
|
|
metrics:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp, logging]
|
|
logs:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp, logging]
|
|
```
|
|
|
|
</Tab>
|
|
|
|
<Tab title="ClickStack Open Source">
|
|
|
|
```yaml
|
|
# collector.yaml
|
|
receivers:
|
|
otlp:
|
|
protocols:
|
|
grpc:
|
|
endpoint: 'localhost:4317'
|
|
http:
|
|
endpoint: 'localhost:4318'
|
|
|
|
exporters:
|
|
logging:
|
|
verbosity: detailed
|
|
otlphttp:
|
|
endpoint: "<YOUR_OTEL_COLLECTOR_HTTP_ENDPOINT>"
|
|
headers:
|
|
authorization: <YOUR_INGESTION_API_KEY>
|
|
compression: gzip
|
|
|
|
service:
|
|
telemetry:
|
|
logs:
|
|
level: "debug"
|
|
pipelines:
|
|
traces:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp, logging]
|
|
metrics:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp, logging]
|
|
logs:
|
|
receivers: [otlp]
|
|
processors: [batch, decouple]
|
|
exporters: [otlphttp, logging]
|
|
```
|
|
|
|
</Tab>
|
|
</Tabs>
|