mirror of
https://github.com/dapr/dapr.git
synced 2026-10-28 08:35:25 +08:00
MirelandClaude Opus 5
d31c4a642b
* Add Workflow and Scheduler Grafana dashboards, extend system services The three shipped dashboards have not had a functional change since #7121 in 2023, and two subsystems have no coverage at all. Workflow: 25 workflow metrics are registered and none were charted. Adds a dashboard covering execution and activity rates by status, execution and scheduling latency percentiles by workflow name, client operations by operation and status, and the two payload size ratio instruments with their 0.95 graceful-stall threshold. The overview row includes activity executions per workflow execution, which reads near 1 when work runs inside the orchestrator instead of being fanned out to activities. Scheduler: 15 scheduler metrics and the embedded etcd collectors had no coverage. Adds a dashboard for job lifecycle including undelivered jobs, trigger latency, trigger concurrency, placement dissemination, and etcd health: size against quota, fragmentation, WAL fsync and backend commit latency, leader changes, compaction and defragmentation. The etcd histogram panels need --etcd-metrics=extensive; the default basic level omits them. Addresses #9341, which specified these panels. System services: adds placement and raft leadership status, hosts against actor hosts, injection rate by outcome, injection failures by reason, and operator service reconciliation. All new panels use timeseries rather than the deprecated graph type, so they add nothing to #9183 and do not overlap #10266, which changes panel types on the existing files only. Metric names were checked against the registered view names, and the etcd series against a live Scheduler metrics endpoint on 1.18.4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> * Fix panels found wrong when run against a live cluster Deployed the three dashboards against Dapr 1.18.4 with a workflow application driving real traffic, and ran every panel query against Prometheus. Three corrections: Workflow payload size ratio panels queried the metric as a gauge with max(). Both payload size ratio instruments are histograms, so the queries returned nothing. Changed to histogram_quantile over the _bucket series at p99, which is also the useful reading given the 0.95 stall threshold. Scheduler gauges used sum() across replicas. Every Scheduler replica reports the same cluster-wide value, so sum multiplies by the replica count, and multiplies again for every Prometheus job scraping the same target. Sidecars connected read 144 on a cluster with four sidecars. Changed sidecars connected, placement streams, incapable sidecars and the concurrency gauges to max. Noted on the sidecars connected panel that the metric counts Scheduler connections rather than distinct sidecars; it read 12 against four daprd containers. Also noted on the placement hosts panel that dapr_placement_actor_runtimes_total was not emitted on this cluster despite actor hosts being connected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> * Add reminder, timer and lifecycle panels to the actor dashboard The actor dashboard charted one actor metric, dapr_runtime_actor_pending_actor_calls. Its other panels are HTTP latency filtered by an actor path, and the placement and lifecycle actor metrics sit on the sidecar dashboard. Five actor metrics were charted nowhere, all of them reminders and timers. Adds two rows, leaving every existing panel untouched: Reminders and timers reminders fired by actor type and success timers fired by actor type and success timers dropped by actor type active timers by actor type Placement and lifecycle pending actor calls by actor type actor rebalancing deactivations, including failures Reminders matter more than the count of them suggests: Dapr Workflow runs on actor reminders, so the internal dapr.internal.<ns>.<app>.workflow and .activity types show up here and this is where a workflow that never started becomes visible. Timers dropped records a timer that fired for an actor no longer hosted on that instance, so the work did not run, which makes it the clearest signal of actors moving faster than their timers can complete. Four actor metrics are registered and documented but have no recorder in 1.18: dapr_runtime_actor_status_report_total, _status_report_fail_total, _table_operation_recv_total and dapr_runtime_actor_reminders. They are not charted here. The existing panels for three of them on the actor and sidecar dashboards are left as they are. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> * Remove sidecar panels charting metrics that no longer record Three panels in the Actor row of the sidecar dashboard chart metrics that nothing has recorded since 1.17, so they have rendered empty for everyone on 1.17 and later: Status report status dapr_runtime_actor_status_report_total dapr_runtime_actor_status_report_fail_total Report status error reasons dapr_runtime_actor_status_report_fail_total Placement table operation dapr_runtime_actor_table_operation_recv_total The recorder methods are still registered but have no call sites; the placement reporting path was rewritten in 1.17 and the instrumentation was not carried across. Removed rather than annotated, so the shipped dashboards work against current releases. The two remaining panels in the row, Actor Deactivation and Actor table rebalancing, both chart metrics that still record and are reflowed to fill the row. Also replaces a description on the system services placement hosts panel that speculated about when dapr_placement_actor_runtimes_total appears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> * Fix the actor dashboard actor type variable The ACTOR TYPE variable is populated from label_values(dapr_runtime_actor_pending_actor_calls, actor_type). That metric only records while a call is queued behind the per-actor lock, so on an idle or uncontended cluster it has no series at all and the variable resolves to nothing. Every panel on the dashboard filters on the variable, so the whole page reads No data even when actors are running. Sources it from label_values(actor_type) instead, which enumerates the label across every metric that carries it, and adds an All option so the dashboard is populated on open rather than showing a single type. Also drops the Pending actor calls by type panel added earlier in this branch, which charts the same metric as the existing Pending locks Per Actor panel, and widens the two remaining panels in that row. Adds the Workflow and Scheduler dashboards to the README. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> * Remove panels charting metrics that do not exist Verified every dapr_* metric referenced by the five dashboards against the views registered in v1.18.4. Seven panels chart names that are not registered anywhere, so they can never draw: Actor dashboard Timers dropped -- dapr_runtime_actor_timers_dropped_total. The actor package registers eleven views and this is not one of them. Scheduler dashboard Placement streams connected, Placement leader, Incapable sidecars and the whole Placement dissemination row -- dapr_scheduler_placement_streams_connected, _placement_leader, _placement_incapable_sidecars, _placement_disseminations_total, _placement_table_updates_total and _placement_dissemination_latency. The scheduler registers no placement views. The placement service publishes its own dapr_placement_* metrics and the system services dashboard already charts all four of them, so there is nothing to re-point these at. The rest of the scheduler panels are kept: concurrency_inflight, concurrency_pending, concurrency_throttled_total, jobs_deleted_total, jobs_bulk_deleted_total, jobs_created_failed_total and sidecar_errors_total are all registered in 1.18.4. They read No data only until something records against them, because a view is not exported before its first record. Also sets noValue to 0 on the four stat panels that count events over a window, so a quiet cluster shows 0 rather than No data. Rows reflowed to close the gaps the removals left. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> --------- Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Monitoring Dashboard
This includes dashboard templates to monitor Dapr system services and sidecars. For more detailed information, please read Dapr Observability.
Grafana
You can set up Prometheus and Grafana and import the templates to your Grafana dashboard to monitor Dapr.
-
- Shows Dapr system service status - dapr-operator, dapr-sidecar-injector, dapr-sentry, and dapr-placement
-
- Shows Dapr sidecar status - sidecar health/resources, throughput/latency of HTTP and gRPC, Actor, mTLS, etc.
-
- Shows Dapr sidecar status - actor invocation throughput/latency, timer/reminder triggers, and turn-based concurrency.
-
- Shows workflow and activity execution counts, latency and failures, client operations, and payload sizes against the configured limits.
-
- Shows Scheduler health, job lifecycle and trigger latency, connected sidecars, placement dissemination, and the embedded etcd.