mirror of
https://github.com/google/adk-go.git
synced 2026-10-28 16:44:38 +08:00
Haiyuan Cao
b7049b87a9
The BigQuery Agent Analytics plugin still wrote the wrong terminal row in the cases listed in #1885. Each is now reported correctly, with an end-to-end test that goes through `runner.Run`: - **Rejected streamed chunks & partial-event response accounting**: A streamed chunk that an agent's `AfterModelCallback` rejects no longer marks the model call as streaming or its agent as responded. `afterModel` only times the first token now; `onEvent` calls `recordStreamed` when ADK emits the partial event, after every callback accepted the chunk (and the `NewBigQueryAgentAnalyticsPlugin*` registration-order comments note that an earlier plugin replacing partial events in `OnEventCallback` hides streaming detection). Every partial event keeps a caller that stops reading on it from being reported as failed, with or without content, while only a partial event with content or a typed output counts as a response of its author (`spanRecord.hasResponse`). - **Lineage-scoped partial-event tracking across workflows & parallel agents**: `lastSessionEventPartial`, `lastEventPartial`, and `partialLineage` track partial session events and the lineage of their authors. Work that starts under that author or once the author has ended (such as the next sibling in a sequential agent) clears `lastEventPartial` — and in `popUnclosedAgentSpans`, once `lastEventPartial` is cleared, an open ancestor in `partialLineage` is excused as consumer-stop teardown (`stoppedOn`) only if a workflow node or `parallelagent` branch under it was canceled by stop (`hasCanceledByStopUnderLocked`) — so a sub-agent that fails after a custom root streamed, a synchronous custom root that fails after draining a successful child following an earlier contentless partial event, and a sibling that fails after a progress event are all reported. Because ADK workflows (`workflow/scheduler.go`) treat partial events as fire-and-forget and `parallelagent` closes its acknowledgement before breaking when `yield` returns `false`, descendant work can resume during consumer-stop teardown before context cancellation arrives; while `lastSessionEventPartial` is set and the outer invocation context is live, activations or calls whose cancellation with cause `context.Canceled` originated at a runner root, workflow node, or `parallelagent` branch (`spanRecord.canceledByStop` via `spanRecord.parent` and `spanRecord.stopPoint`) and activations in `partialLineage` with an active partial or stop-canceled descendant are treated as consumer-stop teardown rather than failures, whereas work whose context an agent, callback, or tool canceled itself (such as a custom agent canceling a sub-agent's context on a timeout while the caller reads on) still fails the run. Conversely, when a branch of a `parallelagent` fails, its `errgroup` cancels every branch context — including the failing branch's own context — with the failing branch's error as `context.Cause(ctx)` (so a branch whose `Run` yields an error leaves its own span open with that cause even with one branch or after siblings have finished, while a failure after the branch's span closed, such as a branch `AfterAgentCallback` error, still cancels any running sibling with that cause): collateral `context.Canceled` model errors on canceled siblings (`recordedError.collateral` via `collateralFailureCause`) are set aside while preserving their cause, `contextFailureCause` ignores bare `context.DeadlineExceeded` from finished `NodeConfig.Timeout` shortcut spans (while still treating a wrapped `context.DeadlineExceeded` cause as a failure cause), `retireAgentLocked` retains retired sub-agent causes on their nearest live ancestor (`spanRecord.retiredCauses`, tagged with `swallowableBySelf` only when a custom agent drained an inherited failure without its own context failing and cleared via `clearRetiredCauseFor` when an ancestor custom agent recovers in `Run` or `AfterAgentCallback` by emitting a final response), and `openFailureCauseLocked` overrides `lastEventPartial` so `INVOCATION_ERROR` and any `AGENT_ERROR` rows selected by the open-call / unresponded scan carry that `errgroup` cause. - **Internally recovered failures & concurrent same-agent activations**: Failures that ADK handles itself no longer fail the run. Each agent activation is keyed by its per-activation `*agent.InvocationContext` pointer (`activationKey(ctx)` on `spanRecord.activation`), and `pushAgentSpan` / `popAgentSpan` / `beforeModel` / `afterModel` / `onModelError` / `beforeTool` / `afterTool` / `onToolError` match the exact activation rather than the newest `(name, branch, path)` entry. A workflow node that `RetryConfig` runs again on the same node path under the same live workflow parent activation (`parentOutsideLocked`, with `rec.parent.isWorkflow` or an all-synthetic unit-test activation/parent pair, and `!rec.concurrentPeer`) purges its failed attempt when the next attempt starts only if both activations are attempts of the node and something shows that the earlier one ended (`laterAttempt` in `pushAgentSpan`). An activation is an attempt only if the stack trace of its `BeforeAgentCallback` shows a goroutine that runs nothing but the attempt (`spanRecord.attempt`, which `attemptSiteOfTrace` reads from the `runtime.Stack` trace that `currentGoroutine` takes): from the bottom up, the function that the go statement started (`workflow.runNode`, or `ParallelWorker.runWorker` with its `runWrappedOnce`), the `AgentNode`'s `Run`, frames of ADK's own packages that adapt the agent to the node (as `llmagent.RunLLMAgentAsNode` does), exactly one agent `Run`, and `runBeforeAgentCallbacks`. So the agents that a function node, a custom agent, a tool or the runner runs are never attempts, and neither is an agent that runs on the goroutine of an `iter.Pull2` iterator, one whose trace is incomplete or elided, or a run nested inside the loop over the events of an earlier run on the same goroutine, whose iterator stays suspended in a `yield` below it (`TestAttemptSiteOfTrace`; `TestAttemptSitesMatchADK` checks the goroutines of the ADK release under test). Two kinds of evidence count: (a) both activations are attempts of a `workflow.ParallelWorker` item on the current goroutine, as when `RetryConfig` runs the item's node again on the worker's goroutine: the stack shows no other agent run on it, so the earlier attempt's run has returned and its `InvocationContext` can be read (`settleEndedInSubtreeLocked`); or (b) both are node attempts that the workflow agent's scheduler started on goroutines of their own (`spanRecord.origin`, read from the `created by` line of `runtime.Stack` and matched against `(*scheduler).startNode` by `goroutineOrigin.startedBy`), and a node completion event reported that the earlier one's node run ended with a failed or canceled outcome (`spanRecord.nodeRunEnded`, set by `recordNodeCompletion`, which `onEvent` calls for the events that `nodeCompletion` recognizes). A node can write the same metadata on an event of its own while it still runs, so (b) shows neither that the earlier attempt's run is over nor that it failed: no `InvocationContext` is read on its strength, and the earlier attempt is purged only for a failure recorded for it, such as an error or a failure cause, not for having no response yet, as it may still answer, nor for a model or tool call that it or a run under it has open, as the run may still be making that call (`failuresLocked` with `openCalls == false` counts an open call only for an activation whose run is known to be over, or, once a recorded failure decided the purge, one that the report shows ended with the attempt; `TestRunner_ReportedNodeOutcomeBeforeDelayedResponse_EmitsNoAgentError`, `TestRunner_ReportedNodeOutcomeBeforeDelayedResponse_CallInFlight_EmitsNoAgentError` with a declared helper's model call or tool call in flight, `TestFailuresLocked_OpenCallFailsWithoutOpenCallsOnlyARunThatIsOver`, `TestConcurrencyRace_ReportedNodeOutcomeDoesNotEndRunningRun`). Such an attempt whose open call does fail is purged when the workflow agent ends (`endRestartedNodeRuns`). Once a recorded failure decides the purge, an activation under the earlier attempt whose run is not known to be over counts as unresponded only if each agent between it and the attempt waits for the run of the one below it (`reportedOverLocked`). An earlier attempt that failed without a recorded failure, such as a custom agent whose `Run` yields an error, stays open and is purged when the workflow agent ends, as without node completion events (see below; `TestRunner_ConcurrentWorkflowRetries_OrdersPurgedAgentErrorBeforeNextStarting/silent_failure`). A completion is attributed only when that is unambiguous: exactly one open activation of the named workflow agent counts exactly one unreported run of the node on that path and branch (`spanRecord.nodeRuns`), and exactly one open activation that its scheduler started as the agent of a node attempt matches (`isNodeAgentOf`, which requires `attemptNode`; `TestTraceManager_RecordNodeCompletionAttributesOnlyUnambiguousRuns`). None of a different activation, a different cancellation scope, the parent's type, the node path, a shared goroutine or the goroutine's creator tells a later attempt apart from the runs that one node attempt makes: a `workflow.FunctionNode` that runs the same agent twice at once, with or without a `WithAgentCancel` context for each run (`TestConcurrencyRace_FunctionNodeConcurrentCachedChildIsNotRetry`, `TestConcurrencyRace_FunctionNodeConcurrentChildDistinctCancelScopes`), one run inside its loop over the events of another on its own goroutine (`TestRunner_FunctionNodeNestedChildRuns_EmitNoAgentError`, `TestConcurrencyRace_FunctionNodeResumesChildAfterNestedRun`), or a run again after it recovered from the agent's error without `RetryConfig`, whose failure stays visible (`TestRunner_FunctionNodeRerunsChildWithoutRetryConfig_KeepsFailure`, also for a function node named like its agent and for one that runs as a `workflow.ParallelWorker` item); nor from a second run of a node that two join nodes route to, which the scheduler starts while the first run still goes on (`TestConcurrencyRace_RoutedNodeStartedAgainWhileRunning`). Such runs are concurrent peers that are neither read nor purged when the other starts. Node completion events exist only in ADK after v2.6.0 (#1572, not yet in a release). Without them, or when a completion was not attributed, a failed earlier attempt that the scheduler started stays open as a concurrent peer when the next one starts, is marked as started again (`spanRecord.startedAgain`, only when both are node attempts that the same scheduler started, on different goroutines), and is purged when the workflow agent ends (`endRestartedNodeRuns`, which `afterAgent` calls before the workflow agent's `AGENT_COMPLETED` row): the scheduler waits for every node run that it started before the workflow agent's `Run` returns, and ADK runs the workflow agent's `AfterAgentCallback` only after that. A new run of a node is parented without the open activations of the same agent, branch and node path that the same scheduler started on other goroutines (`parentForRunLocked`), so the retry of a composite node whose failed attempt left a sub-agent open on the node's branch and path still runs right under the workflow agent (`TestRunner_RetriedCompositeNodeSkipsCachedChildAgentError`). Concurrent overlapping runs of the same agent under a custom parent or invocation root (`concurrentPeer`, registered before completion inference and including when a sub-node path is promoted to the activation's span path) keep separate activations and reconcile response credits across live and already-popped peers (`reconcileConcurrentResponseOnPopLocked`, `selectResponseOwnerLocked`, `retiredPeerCount`, and `unrespondedRetired`; a new model call and a recorded error reset both `awaitingModelEvent` and `awaitingToolEvent`, a model call that ends without a response resets `awaitingModelEvent`, and a new tool call resets only `awaitingModelEvent`, because ADK runs the tool calls of one model response concurrently by default and their merged function-response event still needs the credits of the calls that already ended) without masking a peer's failure, clearing a live peer's `unrecoveredErrors` on a retired peer's post-`AfterAgentCallback` trailer or after a prior tool turn, clearing the error of a later activation that overlapped no peer on a retired peer's late trailer (`retiredPeerCount`; `TestTraceManager_StaleRetiredPeerTrailerDoesNotClearLaterActivationError`), or duplicating `AGENT_ERROR` rows on an activation that already emitted `AGENT_COMPLETED`, and re-entering the same node path under a new parent activation keeps the yielded failure. An agent that runs as a tool (such as a `single_turn` sub-agent) is retired or purged once its owner has no tool call left open: failed activations are returned directly by the purging operation (`pushAgentSpan` or `popToolSpan` via `purgeActivationLocked` / `endToolRunsLocked`) so `logPurgedAgents` writes each callback's prior-attempt `AGENT_ERROR` rows (`error_message` and `latency_ms`) before its next `AGENT_STARTING` row (or before `TOOL_COMPLETED` / `TOOL_ERROR` when a tool-run helper ends with its tool call) without cross-goroutine reordering or failing the invocation. Only an activation that started on the goroutine of a tool call that returned, after the call started, is known to have finished with it (`markToolRunsDoneLocked` sets `spanRecord.runDone` when `popToolSpan` ends the call, so each of two parallel tool calls whose custom agent fails without an event still gets its `AGENT_ERROR` row: `TestRunner_ParallelToolRunsOfSilentlyFailingAgents_EmitAgentErrors`); an activation that a tool started on another goroutine and did not wait for is not known to be over when the tool returns, so its `InvocationContext` is not read: it is retired if it owns no open call and has no failure recorded (`TestConcurrencyRace_FunctionToolReturnsBeforeUnjoinedCachedChild` for an ordinary function tool, `TestConcurrencyRace_LiveToolReturnsBeforeUnjoinedCachedChild` for a Live tool), and otherwise stays open, with its calls, until its own callbacks, the end of a run that waits for it, or the end of the invocation end it, instead of being purged as failed when the tool returns (`TestRunner_ToolReturnsBeforeHelperModelCallEnds_KeepsHelperOpen`, whose helper still has a model call in flight). Non-failed activations — including one answered by `BeforeAgentCallback`, which marks the activation's own `InvocationContext` as ended (`spanRecord.responded`; see the next bullet for when that state is read) even when an enclosing `workflow.ParallelWorker` or custom parent consumes its event before `OnEventCallback` sees it, and one whose run is not known to be over, which `failuresLocked` does not count as unresponded without `spanRecord.runDone`, such as a cached sub-agent that a custom agent run by a tool started on a goroutine that it did not wait for (`TestRunner_ToolRunCustomParentWithUnjoinedCachedChild_EmitsNoAgentError`) — are excluded from `failuresLocked` and retire quietly via `retireAgentLocked` without emitting a bogus `AGENT_ERROR` row. - **When the ended state of an `InvocationContext` is read**: An activation's `InvocationContext` is marked as ended (`EndInvocation`) on the goroutine that runs the activation, without synchronization: by ADK when a `BeforeAgentCallback` answers for it, before the activation yields its first event, or by a custom agent that calls `EndInvocation` while it runs. `spanRecord.responded` therefore never calls `InvocationContext.Ended`, and neither `OnEventCallback` (`clearLastPartial`, `completeResponse`, `popTaskAgentSpan`) nor starting a sibling sub-agent (`pushAgentSpan`) infers completion from event delivery or branch/path ordering. `spanRecord.settleEndedLocked` calls `InvocationContext.Ended` only for an activation whose run something has shown to be over, and records a true result in `spanRecord.contextEnded` (a false result is checked again at a later such point). The evidence that it accepts is: the activation's own `AfterAgentCallback` (`popAgentSpan`); the end of a tool call in `popToolSpan`, for an activation that started on the call's goroutine after the call started (`markToolRunsDoneLocked`, which `endToolRunsLocked` relies on, including for live-run tools owned by the invocation root); a later attempt of its `workflow.ParallelWorker` item on the goroutine on which it ran (case (a) of `laterAttempt`, see above); the end of the workflow agent whose scheduler started the activation as a node attempt and then started that node again (`endRestartedNodeRuns`); and, for an activation under one of those, a chain of parents that each wait for the run of the activation below them (`settleEndedInSubtreeLocked` with `canSettleInSubtreeLocked` and `spanRecord.joins`), or a start on the goroutine of the callback that records one of those ends after the activation or tool call inside whose run that callback runs (the `since` bound, which admits every earlier start only when the callback's stack shows no other agent run: a run that started there later ran inside that activation or call and has returned by the time the callback runs, whereas one that started earlier may still be suspended in a `yield` below the callback; `TestTraceManager_RunsOnThisGoroutineAreOverOnlyAfterTheirFrame`). That an activation ran on the current goroutine shows nothing by itself, and neither does a node completion event, which takes part only in the purge of case (b). Retiring a span on an event is separate from this synchronized completion: a successful `finish_task` function response (`popTaskAgentSpan`, which `onEvent` calls) and the new activation of an agent that delegated to a completed task (`pushAgentSpan`, for an activation with `spanRecord.taskDelegated`) retire the span with `done == false`, reading no `InvocationContext` and marking no run as over, because a custom agent can forward the event of a sub-agent that it runs on another goroutine and wait for that goroutine only later, while the sub-agent still ends its `InvocationContext` (`TestConcurrencyRace_ForwardedFinishTaskEventDoesNotEndRunningChild`). The response ends the activation on the event's node path whose `finish_task` call it answers, which `afterTool` records with that activation's own context (`noteTaskFinished`, `spanRecord.finishedTasks`, `finishedTaskLocked`), so a concurrent run of the same task agent on the same branch and node path, such as a second `llmagent.RunLLMAgentAsNode` run in `ModeTask`, is not ended in its place (`TestConcurrencyRace_FinishTaskEndsTheRunThatMadeTheCall`), and neither is a run on another node path whose model gave its call the same ID, as a function call ID is scoped to the activation that made the call and providers reuse IDs such as `call_0` (`TestRunner_FinishTaskWithReusedCallIDEndsTheRunOnTheEventPath`); a response whose call the plugin saw complete in no run on the event's node path, or in more than one, falls back to `findAgentForContext`, as on `main` (`TestFinishedTaskLocked_RequiresOneRunOnTheEventPath`). The callers of `retireAgentLocked` and `settleEndedInSubtreeLocked` that pass `done == true`, after synchronizing with the end of the run, are `popAgentSpan` (directly and through `reconcileConcurrentResponseOnPopLocked`), `endToolRunsLocked` for a run on the returning tool call's goroutine, case (a) of `laterAttempt` in `pushAgentSpan`, and `endRestartedNodeRuns`; those that pass `done == false` are `popTaskAgentSpan`, the task-delegation retirement in `pushAgentSpan`, and `endToolRunsLocked` for a run that a tool started on another goroutine. When an earlier plugin's `AfterToolCallback` answers a tool call, this plugin's `popToolSpan` does not run for it: the function response event ends the call in `completeResponse`, which removes the matched tool spans and lets `settleFinishedLocked` settle activations already known to be over, but marks no run as finished and does not call `endToolRunsLocked`, because receiving the event shows that the call answered, not that a run that it made returned. `joins` holds only for ADK's built-in agents that yield their sub-agents' errors (`propagatesChildError`), and only in three cases: the child ran on the parent's goroutine, as the sub-agents of a `SequentialAgent` or a `LoopAgent` do; the parent is a workflow agent whose scheduler started the child's goroutine (`(*scheduler).startNode`), because the scheduler waits for every node run that it started before the workflow agent's `Run` returns; or the parent is a `ParallelAgent` that started the child's goroutine (`errgroup.(*Group).Go`) for a declared sub-agent on the branch that it assigns to that sub-agent, and the parent's own `AfterAgentCallback` ran (`spanRecord.completed`) or, only where such a report decides which runs under a purged attempt ended (`reportedOverLocked`), a node completion event reported the parent's node run as ended (`nodeRunEnded`; a `ParallelAgent` waits for its branches when it stops early only since ADK v2.1.0, #1186). It does not hold for a custom agent or a function node, which can return while a run that it started on another goroutine still goes on (`TestConcurrencyRace_FunctionNodeReturnsBeforeUnjoinedCachedChild`), nor for a goroutine that a sub-agent started and left running under a sequential or parallel agent (`TestConcurrencyRace_SequentialAgentUnjoinedLateChild`, `TestConcurrencyRace_ParallelAgentUnjoinedLateChild`). Such a run is left unsettled, with its `InvocationContext` unread, until its own `AfterAgentCallback` or one of the other points shows that it is over; when its custom parent ends cleanly with no open launcher or recorded failure on it, `retireAgentLocked` and `retireDrainedDescendantsLocked` retire it with its parent without reading it, and `endToolRunsLocked` retires an activation that a tool started on another goroutine the same way (`retireAgentLocked` with `done == false`) if it owns no open call and has no failure recorded, and otherwise leaves it open. The goroutine ID, the goroutine's creator and the attempt site come from the header, the first `created by F in goroutine M` line and the frames of `runtime.Stack(buf, false)` (`currentGoroutine`, `parseGoroutineTrace`, `attemptSiteOfTrace`). `TestGoroutineCreatorsMatchADK` checks the two creator names, and `TestAttemptSitesMatchADK` the attempt sites, against the ADK release under test: if a release started nodes or branches from other functions, or ran a node's agent through other frames, those tests would fail, and the plugin would treat those runs as not waited for, or not as attempts, and never read them early (retried workflow nodes may then report spurious failures, and a silently failing parallel branch may go unreported). These rules encode how ADK's built-in agents and workflow scheduler start and wait for goroutines; the race-detector tests named here cover those topologies, not every possible arrangement of agents. Because a tool, a custom agent, or a workflow agent can run a sub-agent on another branch or descendant branch (`agent.InvocationContextDelta.Branch`, `workflow.WithOverrideBranch`), and a custom or workflow agent without `SubAgents` declared can run concurrent sub-agents where `parentFor` places a later child under an earlier open sibling rather than under the enclosing launcher, `launcherOpenLocked` (ordered by `spanRecord.seq`, with the parallel-branch sibling exemption restricted to declared children of a real `ParallelAgent` via `isParallel`) holds settlement back while an earlier tool call, cross-branch non-LLM agent, or enclosing non-LLM agent above an undeclared sibling link (on equal or descendant branches/paths) is still open: the activation is recorded as finished (`spanRecord.runDone`) when an enclosing parent that waits for it ends and is settled once the open launcher ends (`settleFinishedLocked`), with `subtreeLocked` and `retireAgentLocked` walking `spanRecord.parent` across already-retired intermediate spans so the descendant retires cleanly when its enclosing launcher ends. A run that ends while such a tool call or enclosing parent still runs (or has not finished joining) a sub-agent that a `BeforeAgentCallback` answered, as when a live run is closed during the call, is therefore conservatively reported with `AGENT_ERROR` for the sub-agent and `INVOCATION_ERROR` (whereas `main` ended that live-tool scenario with `INVOCATION_COMPLETED` because the live root's event marked the tool's sub-agent as responded; in a live run, where the agent that the runner runs has no span, its own events no longer mark a sub-agent that one of its tools already runs on its branch and node path as responded in `completeResponse`). - **Silent custom child failures**: A sub-agent whose completion was settled (`runDone`) or that still has an open launcher (`launcherOpenLocked`) and neither produced a response nor ended its own `InvocationContext`, as ADK does when a `BeforeAgentCallback` answers for it (`spanRecord.responded`), stays open when its parent ends (`retireAgentLocked`), so a custom child of a sequential root that fails without any event is reported when the caller reads to the end (while a `BeforeAgentCallback`-answered sub-agent retires quietly with its composite parent even when an enclosing `workflow.ParallelWorker` or custom parent drops its event, and with the converse trade-off that a custom parent stopping an unresponded sub-agent on the same goroutine before it produced content or ended its own `InvocationContext` without emitting a settling final event while the ancestor spans between it and the sub-agent remain open — reading an intermediate sub-agent such as a `sequentialagent` to its end closes that intermediate span and leaves its unresponded child open beyond `completeResponse`'s open-span `parentSpanID` walk — or recovering from a sub-agent's silent error through a fallback sub-agent also leaves the sub-agent open). - **Mid-stream stop vs independent model or sub-agent cancellation**: A `context.Canceled` error in `unrecoveredErrors` is suppressed for consumer-stop teardown only when `lastSessionEventPartial` is set, the outer invocation context is still live, and the error callback's context was canceled with cause `context.Canceled` at a workflow or `parallelagent` stop point (`recordedError.canceled` via `canceledByStop()`), while stop-point cancellations carrying another branch's error (`recordedError.collateral`) defer to that branch cause instead of blaming the canceled sibling for `context canceled`. A model call that cancels its own private request context, or whose context an agent canceled itself, keeps `recordedError.canceled == false` and `recordedError.collateral == false` (wrapping any custom cause) and still fails the invocation. ## What changes for existing users - Runs where an `AfterModelCallback` rejects a streamed chunk or where a custom child of a sequential agent fails silently without emitting events now emit `AGENT_ERROR` and `INVOCATION_ERROR` instead of `INVOCATION_COMPLETED` (for standalone `plugin/agentanalytics` consumers outside the repo workspace, silent custom-child detection under `sequentialagent` and `loopagent` requires `google.golang.org/adk/v2 >= v2.6.0`, which includes #1684 (`9daa958c`) so `sequentialagent` and `loopagent` stop iterating on a sub-agent error instead of continuing and running the failed sub-agent's `AfterAgentCallback`). Conversely, a run in which a custom parent agent stops reading a sub-agent on the same goroutine before the sub-agent produced content or ended its own `InvocationContext` (without emitting a settling final event while the ancestor spans between it and the sub-agent remain open; reading an intermediate sub-agent such as a `sequentialagent` to its end after its child failed silently retires the intermediate span and leaves the child open beyond `completeResponse`'s open-span walk), swallows a sub-agent's silent failure and recovers through a fallback sub-agent, or handles a `parallelagent` error in `Run` after the `parallelagent` already retired and yielded its branch error when a branch failed before responding or a canceled sibling recorded a collateral `context.Canceled` error (whereas recovering in an ancestor's `AfterAgentCallback`, or recovering in a custom parent's `Run` by draining the `parallelagent` to its end and emitting a final response when every branch of the `parallelagent` had already responded and no sibling recorded a collateral `context.Canceled` error, clears swallowable `retiredCauses` and completes with `INVOCATION_COMPLETED`, as tested in `TestFailedParallelAgent_RecoveredByRootAfterAgentCallbackReportsCompleted`, `TestFailedParallelAgent_RecoveredByAfterAgentCallbackWithConcurrentUndeclaredSibling`, and `TestRunner_CustomAgentHandlesParallelFailure_RecoveryModes`) now also emits `INVOCATION_ERROR` (and `AGENT_ERROR` when an unresponded activation remains open) instead of `INVOCATION_COMPLETED`. - Runs where a caller stops consuming on a partial event — whether it carries content/output or is contentless — while a `parallelagent` has a sibling model or sub-agent call in flight (or while a streaming tool-run helper is running) now emit `INVOCATION_COMPLETED` without spurious `AGENT_ERROR` rows instead of `INVOCATION_ERROR` when teardown cancels the in-flight work (`context.Canceled` with `context.Cause(ctx) == context.Canceled`; for standalone `plugin/agentanalytics` consumers outside the repo workspace, `parallelagent` early-stop branch cancellation and synchronous teardown require `google.golang.org/adk/v2 >= v2.1.0`, which includes #1186 (`7cf1fecd`)). On `main`, stopping mid-stream under a `parallelagent` with an in-flight sibling call ended `INVOCATION_ERROR` even when the partial event had content. A parallel branch whose `Run` yields an error — including when it is the only branch, when sibling branches have already finished, or when it had already emitted a response before returning its error — or whose post-span failure (such as an `AfterAgentCallback` error) cancels a still-running sibling now emits `INVOCATION_ERROR` with the `errgroup`'s branch error both when the caller drains and when the caller stops at the error, under both non-LLM and LLM roots, instead of blaming a canceled streaming sibling for `"context canceled"` or completing with `INVOCATION_COMPLETED` when no `OnModelErrorCallback` ran. - Runs where ADK internally recovers a failure (`RetryConfig` node retry under the same live parent activation or an agent-as-tool error returned to the calling coordinator, including when a concurrent helper's handled failure is purged while the caller stops on a partial event) now emit `INVOCATION_COMPLETED` instead of `INVOCATION_ERROR` (a function node that runs an agent again after the agent's error is not such a retry and keeps `INVOCATION_ERROR`), whereas re-entering a workflow node path under a new workflow activation after a yielded node error preserves `INVOCATION_ERROR`; the failing lineage of the failed attempt — or the retried node's own agent when every branch of the failed attempt had already responded so no single failing branch can be identified — emits its own `AGENT_ERROR` row (`error_message` and `latency_ms`, timestamped at its last callback and returned directly to the purging callback: with ADK node completion events and a failure recorded for the failed attempt (such as a model error) it is appended before the next attempt's `AGENT_STARTING` row even under concurrent retries; for an attempt that failed without a recorded failure, and without node completion events, it is appended when the workflow agent ends, before that agent's `AGENT_COMPLETED` row and after the later attempts' rows, and for an agent-as-tool call it is appended before `TOOL_COMPLETED` / `TOOL_ERROR` when the call ends), while non-failing activations of the purged attempt (including `BeforeAgentCallback`-answered steps whose events an enclosing `workflow.ParallelWorker` or custom parent consumed) end without a closing row. - A custom agent that emits a contentless partial event (`Partial: true` with no `Content` or typed output) and immediately returns an error before starting further work is no longer distinguishable from a caller stopping on that partial event, so it ends with `INVOCATION_COMPLETED` (matching the existing boundary on `main` for partial events with content), whereas if it starts and drains a descendant sub-agent after the contentless partial event and then returns an error, starting the descendant clears `lastEventPartial` and no workflow or `parallelagent` stop-point cancellation occurs under the synchronous parent (`hasCanceledByStopUnderLocked` in `popUnclosedAgentSpans`), so `AGENT_ERROR` and `INVOCATION_ERROR` are still reported (`TestFailedAgent_RootFailsAfterContentlessPartialAndCompletedChild`). - **Before/after change for `BeforeAgentCallback`-answered sub-agents whose event is dropped while a tool call or enclosing parent is still open**: A run that ends while a tool call still runs a sub-agent that a `BeforeAgentCallback` answered, as when a live run is closed during the call (`TestTraceManager_LiveToolCachedChildStopRace`), **now emits `AGENT_ERROR` for the sub-agent and `INVOCATION_ERROR` instead of `INVOCATION_COMPLETED`** (on `main`, the live root agent's own event marked the tool's sub-agent on the same branch and node path as responded in `completeResponse`, masking any unresponded tool child; `completeResponse` no longer marks a sub-agent that an open tool call of the root agent runs, because the callbacks cannot tell a cached tool child apart from one that failed silently while the tool is still open, and reading `Ended()` before the tool returns would race with the tool's goroutine). Likewise, because `Ended()` is now read only at the points listed above (and never from `OnEventCallback` on a contentless event or when a sibling starts), a run where the caller stops reading after an enclosing parent dropped a `BeforeAgentCallback`-answered sub-agent's event but before that parent ends — or while a tool call, a custom or workflow agent on another branch, or an enclosing custom/workflow agent above an undeclared concurrent sibling that started before the sub-agent (on equal or descendant branches/paths) is still open — emits `AGENT_ERROR` for the sub-agent and `INVOCATION_ERROR` (and such a sub-agent below a retried workflow node attempt or an agent that runs as a tool is not purged as failed when the attempt or tool run ends: it is settled later once the open launcher finishes if `BeforeAgentCallback` answered it, and reported at the end of the run otherwise). A tool of a live run that `functiontool.NewStreaming` makes runs without tool callbacks, so a sub-agent that it runs on another branch is not recognized. ## Known limits - **ADK releases without node completion events** (every release up to v2.6.0, including this module's minimum v2.0.0; #1572 is not in a release yet): the failed attempt of a retried workflow node is purged when the workflow agent ends (`endRestartedNodeRuns`), so its `AGENT_ERROR` rows come after the rows of the attempts that followed it. If the workflow agent does not reach its `AfterAgentCallback`, the attempt is still open when the run ends and is reported then like any other failed activation. A `ParallelAgent` node's failed attempt is not known to have waited for its branches, so a branch of it that failed silently is not settled, and the purged attempt's `AGENT_ERROR` row names the `ParallelAgent` instead of that branch (`TestRunner_RetriedParallelNodeWithSilentBranchFailure_EmitsBranchAgentError` runs only with #1572). - **Silent failures of a retried node attempt**: with node completion events, an attempt that failed without a failure recorded for it, such as a custom agent whose `Run` yields an error, is purged when the workflow agent ends rather than when the next attempt starts, because the report alone shows neither that the attempt is over nor that it failed: a node can write the same metadata while it runs and still answer later. Its `AGENT_ERROR` row then comes after the rows of the attempts that followed it. - **Completion attribution**: a node completion is used only when it is unambiguous. A node run that ends with an output gets no completion event, so once a node has completed with an output, later runs of it on the same path and branch are not attributed; neither are runs of a node that the scheduler started again while it ran. Those cases fall back to the purge at the end of the workflow agent. - **Recovery in a function node**: the agents that a function node runs are never node attempts, so a function node that runs an agent again after the agent's error keeps the earlier activation's failure (`AGENT_ERROR` and `INVOCATION_ERROR`), both when its own code recovers and when `RetryConfig` runs the function node again (on the node, or on the `workflow.ParallelWorker` that runs it for an item): the callbacks do not show which failures the function's code handled. - **Reported purges read no `InvocationContext`**: in a purge that a node completion event and a recorded failure decide (case (b)), an activation under the earlier attempt that a `BeforeAgentCallback` answered counts as responded only if `OnEventCallback` saw its event or its run was already known to be over; otherwise the purge writes an `AGENT_ERROR` row for it, without failing the invocation. - **Runs that nothing waits for**: an activation that a custom agent, a function node, a `workflow.ParallelWorker` item or a nested workflow node without an agent span started on a goroutine of its own, including the goroutine of an `iter.Pull2` iterator that its caller does not read to the end, is not known to be over when an enclosing built-in agent ends. It stays unsettled, with its `InvocationContext` unread, and fails only by a recorded error, a failure cause, or an open call, so a silent failure of such a run is not reported. - **Goroutine creators and attempt sites**: `(*scheduler).startNode` and `errgroup.(*Group).Go`, and the frames that `attemptSiteOfTrace` classifies (`workflow.runNode`, `ParallelWorker.runWorker` and `runWrappedOnce`, `(*AgentNode).Run`, the agent's `Run` and `runBeforeAgentCallbacks`), are taken from the current ADK and `golang.org/x/sync` code. `TestGoroutineCreatorsMatchADK` and `TestAttemptSitesMatchADK` fail if a release changes them; the plugin would then treat the affected runs as not waited for, or not as attempts, and never read them early: retried workflow nodes may then report spurious failures, and a silently failing parallel branch may go unreported. ## Alignment with adk-python Checked against `google/adk-python` `src/google/adk/plugins/bigquery_agent_analytics_plugin.py` (pinned at `25d191710985`): - Python ADK provides explicit `on_agent_error_callback` (`lines 10827-10925` with `on_run_error_callback`: `on_agent_error_callback` at `10827-10869`, `on_run_error_callback` at `10872-10925`) hooks that receive unhandled agent and runner exceptions directly, alongside `after_agent_callback` (`lines 10243-10263`) and `after_run_callback` (`lines 10102-10145`). - ADK Go's `plugin.Config` has no `OnAgentErrorCallback` or `OnRunErrorCallback` (`AfterRunCallback` receives no `error`), and ADK Go yields events over Go iterators (`iter.Seq2[*session.Event, error]`) where callers can `break` mid-stream while workflow nodes and `parallelagent` branches run concurrently in child goroutines. `traceManager` therefore infers terminal agent and invocation status at `AfterRun` from recorded callback errors, open activations, and partial-event / callback-context cancellation state. ## Testing - `go test -mod=readonly -shuffle=on -count=1 ./...` without `-race`, and `go test -race -mod=readonly -shuffle=on -count=1 ./...` (with multi-module `go.work`, `0` skipped tests) - `GOWORK=off go test -race -shuffle=on -count=1 ./...` (standalone module against pinned `google.golang.org/adk/v2 v2.0.0`, skipping the tests/subtests that require `#1186` in `v2.1.0+`, `#1267` in `v2.4.0+`, `#1684` in `v2.6.0+`, or the workflow node completion events of `#1572`, which no release includes yet: `TestRunner_PurgedRetryAttemptTimestampPrecedesNextAttempt`, `TestRunner_ConcurrentWorkflowRetries_OrdersPurgedAgentErrorBeforeNextStarting` and `TestRunner_RetriedParallelNodeWithSilentBranchFailure_EmitsBranchAgentError`) - `go test -race -count=1` 20 times in both modes, each run in a separate process, on `TestConcurrencyRace_*` (including `TestConcurrencyRace_ForwardedFinishTaskEventDoesNotEndRunningChild`, `TestConcurrencyRace_FinishTaskEndsTheRunThatMadeTheCall` and `TestConcurrencyRace_ReportedNodeOutcomeDoesNotEndRunningRun`), `TestFailedParallelAgent_*`, `TestFailedAgent_RootFailsAfterContentlessPartialAndCompletedChild`, `TestTraceManager_ContentlessEventCancelAndSiblingFailureEndInvocationRace`, `TestTraceManager_ConcurrentChildrenSameBranchCachedEndInvocationRace`, the goroutine and node-completion helpers (`TestParseGoroutineTrace`, `TestCurrentGoroutine`, `TestGoroutineCreatorsMatchADK`, `TestAttemptSiteOfTrace`, `TestAttemptSitesMatchADK`, `TestTraceManager_RunsOnThisGoroutineAreOverOnlyAfterTheirFrame`, `TestNodeCompletion`, `TestTraceManager_RecordNodeCompletionAttributesOnlyUnambiguousRuns`), the function-node re-entry tests (`TestRunner_FunctionNodeNestedChildRuns_EmitNoAgentError`, `TestRunner_FunctionNodeRerunsChildWithoutRetryConfig_KeepsFailure`), the `RetryConfig`, reported-outcome and tool-run tests (`TestRunner_ConcurrentWorkflowRetries_OrdersPurgedAgentErrorBeforeNextStarting`, `TestRunner_ReportedNodeOutcomeBeforeDelayedResponse_EmitsNoAgentError`, `TestRunner_ReportedNodeOutcomeBeforeDelayedResponse_CallInFlight_EmitsNoAgentError`, `TestRunner_WorkflowRetryParallelRespondedBranches_PurgesFailedAttemptAndCompletes`, `TestRunner_ParallelWorkerPipelineWithCachedStep_EmitsInvocationCompleted`, `TestRunner_PurgedRetryAttemptTimestampPrecedesNextAttempt`, `TestRunner_RetriedCompositeNodeSkipsCachedChildAgentError`, `TestTraceManager_NodeReentryPurgesOnlyFailedAttempt`, `TestTraceManager_PushAgentSpanReturnsOnlyOwnNodePurgedAttempt`, `TestTraceManager_RunnerConcurrentWorkflowRunNodeSameAgent`, `TestTraceManager_CrossBranchDynamicNodeRunsCachedChild`, `TestRunner_InternallyRecoveredErrors_SingleTurnToolAndWorkflowRetry_EmitsCompletedTerminalRows`, `TestRunner_SingleTurnToolAnsweredByBeforeAgentCallback_EmitsNoAgentError`, `TestAfterTool_FailedToolRunEndsWithAgentError`, `TestOnToolError_FailedToolRunEndsWithAgentError`, `TestRunner_ToolRunCustomParentWithUnjoinedCachedChild_EmitsNoAgentError`, `TestRunner_ToolReturnsBeforeHelperModelCallEnds_KeepsHelperOpen`, `TestRunner_ParallelToolRunsOfSilentlyFailingAgents_EmitAgentErrors`, `TestRunner_RetriedParallelNodeWithSilentBranchFailure_EmitsBranchAgentError`), the task-mode tests (`TestRunner_TaskModeDelegation_EmitsAgentCompletedAndRetiresFirstCoordinatorActivation`, `TestRunner_FinishTaskWithReusedCallIDEndsTheRunOnTheEventPath`, `TestFinishedTaskLocked_RequiresOneRunOnTheEventPath`), the open-call rule of `failuresLocked` (`TestFailuresLocked_OpenCallFailsWithoutOpenCallsOnlyARunThatIsOver`), and the retired-peer trailer tests (`TestTraceManager_RetiredPeerTrailerDoesNotClearOpenFailingPeerError`, `TestTraceManager_StaleRetiredPeerTrailerDoesNotClearLaterActivationError`) - `golangci-lint run ./plugin/agentanalytics/...` Fixes #1885