Operations & Events

The cross-cutting mechanics that keep the pipeline running day to day under Temporal (the workflow engine): the daily spend ceiling, the (off-by-default) discovery schedule, where every stage persists, how domain events reach the web app over Server-Sent Events (SSE), how read-model projections recover after a crash, and how the pipeline behaves when things fail.

Read this if you are setting a budget, scheduling discovery, tracing where a stage's data lands, or working out how the system recovers from a failure.

Spend Ceilings

A global daily USD ceiling and per-lane daily observed-token thresholds backstop LLM use. The check_spend_budget activity (llm.py) is the preflight in every spendful workflow, and the same authority runs immediately before each actual provider attempt, including retries and Apply subprocess launch. The global daily_budget_usd defaults to $25, and 0 means unlimited. Each lane_token_limits value is a non-negative integer; omitted or 0 is unlimited. If the UTC day's global estimated spend or the requested lane's normalized input-plus-output total already meets its ceiling, the authority raises BudgetExceededError (budget_exceeded, non-retryable), failing the run before that attempt. Exhausting one lane does not block peers while the global ceiling allows them.

Provider observations are written to the llm_spend UPSERT ledger, keyed by UTC day and product lane. Global totals are sums across lane rows. Input and output are the normalized provider totals; cached input and reasoning output subsets are not added again. Usage is persisted before downstream parsing, schema, or result-envelope validation, and one observation callback is counted once. Distinct retry attempts that reach a provider each count. Providers that expose no usage do not create token usage. Cost estimation uses per-model-family rates (estimate_llm_cost_usd in llm.py); the rates are coarse family buckets, and models without a listed family fall back to a generic rate, so the ledger is an estimate, not billing truth. The ceiling is a preflight gate per workflow, not a mid-call interrupt. Because there is no proven maximum token cost for an in-flight call, an admitted call can overshoot a lane threshold. JobCtrl does not reserve tokens or claim a strict concurrent cap; the next provider attempt is denied after the observation is recorded.

Supervised contact research (ContactResearchWorkflow, Contact & Outreach) is a spendful workflow and reuses this same preflight — the check_spend_budget activity + the dailyBudgetUsd ledger — before its LLM candidate extraction. There is no second spend table or preflight.

Automatic Preparation Recovery

The worker checks saved preparation state at startup and on its 15-second heartbeat. pipeline/automatic_preparation.py resumes pending or retryable failed Enrich, Score, Tailor, and Cover work without running Discover or Apply. Discovery, imports and unscoped/manual batch workflows retain a global dispatch gate. A single-job JobPreparationWorkflow or one-stage automatic JobPipelineWorkflow may overlap recovery for other jobs: its first immutable history event must identify the exact run, local cohort, permitted stage and compatible workspace. Every member stays excluded even between activities. Missing or unavailable inventory/history prevents dispatch.

Under the SQLite write lock, every queued/running stage must belong to a verified execution or an unchanged frozen reservation. Unknown or conflicting ownership prevents dispatch. A frozen reservation already executing under its verified cohort keeps its rows while unrelated work can proceed. An unstarted reservation retains its exact cohort and deterministic workflow ID; a recorded run must be described before settlement, and missing owner history cannot justify a replacement execution. Enrich's activity/run/attempt lease token and Score/Tailor/Cover's run owner are checked using their respective formats.

Each recovery workflow handles one stage for at most 25 local jobs. Automatic batches share the saved pipeline_internal_concurrency budget (default 1, bounded 1–16): each executing batch accounts for the smaller of its immutable worker count and unfinished cohort membership. The next dispatch receives only the remaining parallelism, rather than multiplying the setting for each batch. Existing execution input is never rewritten. Worker activity slots and source politeness remain separate limits. One new cohort is dispatched per heartbeat. Heartbeats retain only the generated prepare-auto-local-<stage>-<24 hex> workflow ID and Temporal run ID. Pipelines uses the literal activity type to show its stage without assigning the batch to an individual job or inventing Discovery membership.

Canonical queue predicates determine description, score, policy, threshold, and approved material prerequisites. A later heartbeat selects the next eligible stage. Recovery excludes explicit cancellation, blocked/non-retryable and exhausted states, deleted jobs, and closed/incompatible postings. Existing scores and approved materials are reused. Attempts are preserved, capped by the smaller of the stage limit and five, and delayed by min(1800, 60 * 2^attempts) seconds. Both dispatch and the workflow enforce the normal spend preflight.

Automatic Tailor/Cover also respect a canonical manual Apply decline: skipped, no error code, and non-retryable. This differs from a MIN_SCORE policy skip; legacy fields and retained reason metadata do not decide the outcome. New selection, activity filtering and the locked per-job claim share this veto. Before reusing a frozen reservation, the coordinator releases only declined, unstarted queued automatic material ownership to pending, recording its former reservation without consuming an attempt. Surviving members retain the original frozen cohort/workflow identity; an all-declined cohort cannot block unrelated work. The Apply outcome and accepted materials remain intact. Resetting Apply can restore normal automatic eligibility. Explicit foreground generation and Enrich/Score are unchanged; cooperative Cancel stops already-admitted work.

pipeline/public_fetch_recovery.py is a narrow exception for a still-current non-retryable DETAIL_UNSAFE_URL: the canonical stage, enrichment attempt, and latest StageFailed event must agree. It accepts typed dns_non_public evidence or exact recognized legacy DNS/transport error shapes; arbitrary error text cannot enable recovery. The worker checks at most five candidates per pass and five times per failure, starting after one minute with exponential backoff. Checks only resolve the posting and the recorded failed request; they do not fetch pages. A ten-second check deadline leaves unresolved conditions blocked. Both destinations must validate as public, then a write transaction rechecks the complete candidate snapshot and version before returning the stage to pending. Existing attempt counts and cooldowns remain effective. Private literals, cancellations, new owners, stale/superseded attempts, deleted/closed jobs, and exhausted budgets are excluded. EnrichmentFetchRechecked records the cause, host, results, count, and next check; the immutable original failure remains. A successful recheck emits StageReset with reason public_fetch_condition_resolved, after which normal guarded dispatch owns the retry.

Owned scoring persists the requirement-fit report alongside its score evidence. After a rescore restores a missing report, dependency reconciliation releases only the retryable REQUIREMENT_FIT_MISSING Tailor block when the report matches the latest score, posting analysis, and score profile version and has requirement items. It preserves attempt limits and cancellation, requires no current approved resume, and lets normal eligibility and cooldown checks admit subsequent work.

A SQLite transaction reserves the exact cohort in queued stage metadata and records StageQueued with its prior state and attempt count. The frozen cohort and deterministic workflow ID survive a worker restart or lost start acknowledgement. Temporal uses USE_EXISTING and REJECT_DUPLICATE to prevent a second execution of that reservation. Selected activities recheck ownership, deletion, cancellation, prerequisites, and attempt limits before processing. Automatic runs use one Temporal activity attempt; durable job state and the heartbeat own later retries. After a started activity times out, the exact Temporal history and a charged attempt from that cohort allow unconsumed reservations to return to pending without spending or resetting their attempts. This also releases the former PREPARATION_RECOVERY_STOPPED outcome when the same proof exists. Current eligibility and cooldowns still apply. Cancellation requests, termination, missing history, and batches without a consumed attempt retain their stopped/block outcome, preventing a preflight retry loop. History copied into a reset descendant cannot establish a timeout in that new run; the recorded original run and the scheduled activity owner must both match.

Before checking for busy stage rows, the worker describes each interrupted automatic activity's exact Temporal execution. A closed owner restores its committed result or records one bounded failed attempt; canceled work stays canceled. Missing history leaves ownership intact. Enrichment also advances its execution lease so a disconnected predecessor cannot commit late results. Scoring and material generation fence both admission and persistence against cancellation or changed ownership, without holding a database lock during provider calls. Global Score sweeps use the same admission and publication fences, including queued automatic reservations that have no activity owner yet. A profile continuation cannot replace a live recovery owner. Selected batches report a typed ownership loss as a skipped job and continue independent peers. Global failed-job summaries and safe determination errors still fail their activity; previously accepted peer results remain saved.

Stopping discovery's streaming enrichment consumer for its terminal pass is an internal handoff, not a user cancellation. The consumer releases unfinished work to that pass. Historical incorrectly canceled rows are repaired only when their exact successful Discover execution claimed terminal enrichment after the stop; any recorded workflow cancellation request or terminal cancellation lease prevents repair.

Standing Auto-Apply Loop

Auto apply is a settings-reconciled continuous Apply workflow, not a hidden UI shortcut. The auto_apply setting defaults to false. When it is true, the Python worker ensures one deterministic workflow id, apply-auto-local, is running on the worker task queue with ApplyWorkflowInput(continuous=True, auto_apply_loop=True). When it is false, the worker cancels that deterministic workflow if it exists. The reconciler runs at worker startup and from the worker heartbeat loop, so turning the setting on or off takes effect without restarting the worker.

The loop intentionally reuses the existing continuous poll mode in ApplyWorkflow and apply.launcher.worker_loop:

• apply_approval_required is re-read by the apply activity at claim time. With the default true value, the loop claims only jobs already approved in Apply Review and parks unapproved jobs as awaiting approval. With false, it may claim eligible prepared jobs without that wait, but it still cannot perform final browser submission. Browser forms stop for manual completion; the owned Gmail sender still requires an exact recipient/attachment approval.
• min_fit_score and apply_concurrency are re-read by auto-apply activities, so threshold and fan-out changes affect later polls without recreating the workflow.
• The spend ceiling still runs as the workflow preflight. A depleted budget fails the standing loop before browser apply work starts, instead of retrying into a spend storm. The reconciler treats that failed projection as a budget halt and does not restart the loop while today's spend remains at or above the configured ceiling.
• The existing apply boundaries remain the source of truth: dry-run paths never submit, model-driven browser sessions are transport-locked, submit-intent tracking protects owned email sends, and CAPTCHA or third-party challenge handling fails closed.

Because the loop is a normal workflow run, it is projected into workflow_run_projections and appears in the Runs / Operations surface as Standing apply loop. Automation is therefore visible and cancelable from the same place as manually-started runs.

Discovery Schedule

Scheduled discovery is off by default. A single Temporal Schedule, jobctrl-discovery-local, can run DiscoverWorkflow on a cron expression, but it is reconciled from settings only at worker startup (_reconcile_discovery_schedule in cli.py, before worker.run()):

• It reads load_discovery_schedule_settings() → (enabled, cron).
• If disabled, it deletes the schedule handle (idempotent) and returns.
• If enabled, it creates (or updates) the schedule to start DiscoverWorkflow (discover-local) on the given cron, with ScheduleOverlapPolicy.SKIP so a slow run never overlaps the next tick.

The gotcha: because reconciliation happens once at startup, toggling the schedule setting has no effect until the worker is restarted. Turning the schedule on or off, or changing its cron, requires bouncing the worker.

Supervised contact research has no schedule at all — ContactResearchWorkflow runs only when the user explicitly starts a run (there is no automation, and no public source is auto-fetched by default; see the source-access policy in the domain model).

Discovery Run Progress

The Runs view progress bar is driven by the progress payload (_discovery_progress_payload in pipeline/runner.py) that discovery activities emit onto the discovery_runs aggregate. The denominator is fixed at plan time: progress_total = len(families) + 2 — one step per source family plus a terminal enrichment and a terminal preparation step. The counter is monotonic: family source activities advance it 0 … N-1, then the terminal reconcile enrichment (N) and preparation (N+1 → N+2, i.e. 100%) finalize it.

Under score-as-you-discover streaming (R9 Phase 1) the execution-scoped live enrichment activity and per-family preparation backstops are progress-silent: they pass progress_total=0, which suppresses progress emission (emit_progress = progress_total > 0), so the bar advances only on the family + terminal spine and can never oscillate or shrink. Scores still appear incrementally in the Jobs view because that path is independent of the progress bar: JobScored events → projection builders → GET /v1/events/stream. Under R9 Phase 2 (per-job handoff) those scores arrive at per-job granularity — a job is scored the moment it is individually enriched — while the outer progress spine stays unchanged. No new discovery_runs progress columns are added, so both projection builders stay in parity.

Broad-board progress uses search units as its source-level unit. Its newJobs and existingJobs values come from durable acceptance receipts, filteredJobs comes from hashed filtered-result receipts, and recoveredUnits counts units reclaimed by a newer Temporal activity attempt. The TypeScript read model preserves that optional field and Pipelines renders it as N resumed. JobStreaming 0.0.5 ProviderProgress is a separate immutable value inside the source progress: provider, phase, unit, completed/total units, raw items seen, emitted jobs, and tri-state continuation. The worker stamps the exact Discover workflow/run identity onto the progress event, and Operations accepts only a matching execution before rendering the latest traversal facts. Unknown provider totals stay null and do not become a synthetic percentage or provider-derived ETA. A separate source-family ETA may use recent completed whole-family durations when live worker inventory and the Temporal queue bound the source lane. Raw provider output, cursors, resume dictionaries, checkpoints, owner tokens, and error messages are not projected into this broad progress payload.

Pipeline Operations Snapshot

The legacy Runs progress bar answers one narrow question: how far the source family plus terminal-reconciliation spine has advanced. It is not whole-pipeline progress. GET /v1/pipeline/operations is the separate operational read model for answering which execution owns work, what remains in each scope, what the worker can currently execute, and whether an ETA can be defended.

Immutable execution lineage

At the beginning of DiscoverWorkflow.run, the workflow reads its Temporal identity once and creates an immutable DiscoveryExecutionRef:

(tenant_id, discover_workflow_id, discover_run_id)

The run ID is required because the deterministic workflow ID can be started again. Source-run IDs are observation details, not execution identity. Every linked job is persisted in discovery_execution_jobs, keyed by that exact execution plus stable JobId, with one of two cohorts:

• observed_this_run — a source in this execution observed the job;
• existing_backlog — the execution deliberately swept pre-existing eligible work. A later observation promotes this one row to observed_this_run; it cannot create a second membership or demote a current one.

Planning is explicit. pending and failed plan states do not mean that a job needs no work. planned records the immutable required-step list and child preparation workflow ID; not_eligible records a bounded safe reason. The terminal preparation_fanout step closes membership for operational phase and overall-ETA purposes.

Scoped work, not one denominator

The response keeps three scopes distinct:

Scope	Ownership
current_execution	This run's observed-job cohort and execution-owned orchestration.
execution_sweep	This run's explicit pre-existing-backlog cohort. Only separately attributable per-job queues are counted.
global_outside_execution	Canonical per-job backlog outside both selected-execution cohorts.

Source-family progress and reconciliation are also separate. The former counts planned family crawls; the latter counts enrichment passes and preparation fan-out. Source completion therefore never implies that downstream scoring, materials, or PDF work is complete.

Provider traversal progress is observational evidence inside the active broad- board family, not a third completion authority. It can explain that a provider has completed pages and whether continuation is known. It cannot bound shared- pool contention or establish remaining pages when totalUnits is null. The source-family estimator therefore never derives a remaining-page count from that payload; it can instead price the known planned-family backlog from recent whole-family duration samples when runtime evidence bounds the source lane.

The API prefers an active or draining Discover execution and otherwise shows the latest terminal execution. The phase vocabulary is discovering, draining, completed, completed_with_issues, failed, and canceled. Both execution cohorts participate in the delivered phase calculation: unresolved observed jobs or swept backlog can keep an otherwise successful workflow in draining. While a successful, error-free source workflow is closed and shared work has not been confirmed idle, the UI reports Source collection complete, counts remaining jobs across both execution cohorts, and points to running and waiting preparation stages. Closing source collection does not imply that active preparation needs another Discover run. An unavailable runtime inventory remains explicit; the notice does not claim the workers are idle.

Operator stop and failure recovery

The Pipelines workspace exposes cancellation only when the selected Discover execution has workflowStatus=in_progress and phase discovering or draining. The action cancels the deterministic Discover workflow ID, then invalidates the workflow-run, dashboard, and pipeline-operations queries so the stop request cannot leave the live workspace stale. A closed workflow whose durable phase still says draining is not cancelable.

A failed execution is never treated as proof that the runtime is idle. The UI uses activeItemsTotal as a three-state recovery gate: a positive value reports remaining work, null reports that the inventory is unavailable, and exactly zero permits Set up a new Discover run. That action only selects and focuses the Discover launch controls; it never dispatches work implicitly. The same fresh zero-active-slot guard disables direct Discover submission for failed, completed-with-issues, closed-draining, and incomplete-history replacement runs, so selecting the action tab cannot bypass recovery safety.

reconciled_not_found is a provisional verdict: the worker could not find the exact recorded Temporal run in the history store it could currently reach. The reconciler keeps probing that exact run. When the authoritative history is available again, a marked recovery event automatically restores the run to in_progress or replaces the provisional verdict with its real closed outcome; the false terminal remains visible in the audit history. Starting a replacement run is appropriate only when the exact execution is genuinely absent and the runtime inventory confirms that no work remains. Workflow IDs, Temporal run IDs, and the raw reason code stay in a collapsed technical disclosure.

A candidate-profile edit and its audit event commit in one SQLite transaction, but saving a profile never schedules Score, Tailor or Cover. API startup leaves historical profile-continuation events intact without dispatching them. Already admitted work retains its frozen profile version. Explicit score refresh uses the source-bound profile impact check; material generation remains an explicit action.

Other condition-driven preparation keeps its existing ownership. A selected live-profile extension connection maps the resolved discovery_browser_extension_unavailable condition to canonical enrich rows blocked with ENRICH_ROBOTS_DISALLOWED, selects only LinkedIn jobs carrying that typed condition (including the former authenticated_linkedin_browser_unavailable value and source-identity fallback for legacy rows), and dispatches those JobIds through Enrich, Score, Tailor, and Cover. Unrelated robots blocks and ordinary pending Enrich rows are not swept into the recovery. The extension continuation uses one stable workflow identity for the resolved condition even while individual rows leave the blocked cohort; a concurrent trigger attaches to the active execution, while a later resolution episode may reuse the identity after it closes. These continuations use the normal idempotent workflow path and do not rerun Discover. A low-confidence posting keeps Tailor explicitly blocked by Enrich rather than leaving it as unexplained pending. Description trust and application- target readiness are independent: a missing external application URL alone cannot create this blocker. For legacy rows where it did, Enrich appends a corrected immutable snapshot and resets that exact condition-blocked Tailor row to pending before any optional authenticated URL lookup. LinkedIn on-site apply is persisted as a terminal target-readiness outcome rather than retried as a generic unresolved URL.

Score, Tailor, and Cover stage rows record the Temporal run ID that owns every running write. Rescore/retailor rows also preserve the score version or accepted-material generation that existed before that run. An activity retry reconciles its owned rows before selector-based work: a result committed after the baseline is accepted without another model call, while an uncommitted row becomes explicitly retryable. If the stage activity exhausts, that same owner-scoped reconciliation is a mandatory workflow step; the workflow cannot publish a terminal outcome while its rows still claim that owner. This is not a second scheduler or a polling reaper—Temporal durably delivers one idempotent reconciliation decision.

Unscoped Tailor freezes its eligible tenant cohort, profile snapshot and model policy once, then uses the same per-JobId lifecycle as selected Tailor. A job's StageStarted is recorded when a worker dispatches that job, not when the batch is selected. The bounded material executor stops admitting later jobs after cancellation; in-flight work must still pass the canonical cancellation and activity-owner checks before material or terminal-state writes. Selected runs keep partial results and approved IDs for Cover, while the unscoped adapter retains aggregate failure and durable-exhaustion escalation.

Two durable progress authorities

Canonical job_stage_states remains the source of truth for per-job enrich, score, tailor, and cover work. Orchestration gaps use the four PipelineStep* lifecycle events, folded into pipeline_step_projections for source_planning, source_family, enrichment_pass, preparation_fanout, existing_backlog_sweep, and pdf_render.

The fold is keyed by execution, step kind, and bounded item key. A higher attempt replaces an older attempt; a late lower attempt is ignored; within one attempt, the first terminal result wins. Details contain only allowlisted codes and counts, and failures persist bounded error codes rather than exception text. The Python and TypeScript projection builders fold the same events and share the normal operations-projection watermark/rebuild path.

A current-execution membership with work_plan_state='pending' and no enrich stage row is an expected pre-dispatch state and is projected as waiting. Likewise, while the Discover workflow is starting or in progress, a planned source family whose bounded batch has not been scheduled yet is waiting. Once that workflow is terminal, a planned family with no durable lifecycle is unknown because it is then a genuine projection gap. An unrecorded planned PDF is waiting while a recorded prerequisite in its required-step list is pending, queued or running, provided its preparation owner is not already terminal. A recorded PDF lifecycle always takes precedence. A missing PDF is skipped only when a run-owned PreparationWorkflowOutcomeObserved receipt proves that the exact successful preparation execution stopped at an earlier required step. The worker derives that bounded receipt from the exact Temporal run's start input and completed result, binding tenant, job, idempotency key, required plan and Discover run. The API joins its recorded owner/run and indexed event identity; current unowned MIN_SCORE rows or successful preparation alone cannot supply this proof. The same classification resolves cohort completion and removes PDF demand from remaining paths and swept-backlog contention. It does not change the immutable plan, create a PDF lifecycle or claim that a document was generated.

A missing PDF is succeeded when the exact successful preparation run's literal completed result records PDF completion. This includes a cover letter disabled after admission: the result may skip Cover while completing all other required steps. No receipt can override an actual queued, running or failed PDF lifecycle.

Discover can associate an already-started manual or reused preparation with its execution sweep. That owner's immutable input retains its actual origin. When the origin differs from the observing Discover run, a version-2 outcome receipt records the explicit membership association and original origin separately. The worker checks the exact job, tenant, owner/run, frozen plan and workspace identity against the start input and completed history, then rechecks the current membership/cohort and successful owner under its write transaction. Every new preparation producer captures the current workspace identity when its caller does not supply an explicit identity. Existing unscoped histories that also lack a workspace binding remain unknown; a recorded job row alone cannot establish that missing authority. Receipts are idempotent per association, cohort and frozen plan; an observation for one Discover run cannot suppress a different association. The API validates the recorded association, current owner/run, plan and indexed identity before using its result. Missing history, foreign workspaces, replaced owners or changed plans remain unknown. This observation appends an audit receipt only; it never rewrites workflow input or fabricates a PDF lifecycle or accepted artifact.

A successful preparation result that completed Score and stopped at Tailor may also resolve a recorded eligibility hold. The API requires the exact owner/run receipt, the current successful Score owner and accepted score version, no unresolved score staleness, and matching non-retryable SCORE_ELIGIBILITY_BLOCKED gate provenance for every planned material stage. It reads the saved typed eligibility verdict and exact blocker list; it does not judge the posting again. Canonical Tailor/Cover counts remain blocked, while cohort completion, issue classification, ETA demand, remaining paths and swept-backlog contention treat that proven policy outcome as finished. A newer score or owner, reset stage, foreign/missing receipt, generic setup block or retryable block remains unresolved. An actually recorded PDF lifecycle still wins. No canonical stage, score, accepted file or frozen plan is rewritten.

A closed failed/canceled preparation owner plus a recorded non-retryable, exhausted or canceled prerequisite also proves that its unexecuted PDF cannot start. That PDF is blocked or canceled, and the cohort finishes with issues rather than staying in draining. This never claims a render succeeded. Queued/running successor work, retryable failures and an actual PDF lifecycle retain their own authority.

Otherwise, without the exact successful outcome receipt, missing prerequisites, a terminal owner alone, or no unfinished required prerequisite leave the missing PDF unknown. Other missing stage state after work-plan resolution remains unknown and actionable; a genuinely missing or failed PDF is never hidden by unrelated work. The producer-lifetime live enrichment activity is runtime telemetry, not a durable enrichment_pass projection: it ends by intentional cancellation and the lifecycle schema has no canceled terminal state. Per-job stage rows record its durable work; terminal reconciliation owns the persisted enrichment-pass boundary.

This internal consumer stop is not a user cancellation: unfinished Enrich rows remain reclaimable by the terminal pass. Only a canceled DiscoverWorkflow requests exact workflow/run-owned cohort cancellation. Released pending rows retain their ownership metadata until reclaimed, so cancellation cleanup can still find them without touching a successor's work.

Durable projection recovery and runtime-only visibility

projectionCoverage is the selected execution's durable recovery checkpoint; it does not restate telemetry freshness. Its states are:

• ready: the worker decoded one exact Temporal workflow/run history, refreshed its projections, and verified equality of the expected and persisted membership-key and pipeline-step-key sets. For native runs, later canonical writes carrying that same execution identity advance the verified key-set proof atomically while the history-event watermark remains the last audited history boundary;
• recovering: startup or heartbeat reconciliation is rebuilding those exact sets;
• retrying: the latest safe reconciliation attempt refused an ambiguous input or met a transient dependency failure and will retry automatically; and
• incomplete: an immutable legacy history ended before it could identify every originally targeted job. JobCtrl preserves every exact recovered member and step, records the bounded terminal reason, and does not retry or invent the missing lineage.

The checkpoint is stored in discovery_execution_recoveries, keyed by tenant, Discover workflow ID, and Temporal run ID. It records the decoder version, native/reconstructed mode, history-event watermark, expected and persisted counts, a digest of both canonical key sets, and a bounded error code. A ready row is valid only while its counts and digest still match the current durable rows. Row counts alone, workflow terminal state, and runtime telemetry can never promote an execution to ready.

Once a native checkpoint is ready, linking a new execution member and folding a new pipeline-step key update that checkpoint's counts and digest in the same SQLite transaction as the canonical write. Both the Python and TypeScript projection owners apply the identical digest algorithm. A failed proof update therefore rolls back the native key write instead of exposing a one-row-ahead recovering snapshot. Reconstructed checkpoints never use this path; their expected sets remain exclusively owned by history reconciliation. A heartbeat also keeps an already-verified checkpoint visible while it reads history and publishes recovering only after it has evidence that repair is needed.

Final recovery publication reserves the SQLite writer while folding projections, reading the exact key sets and saving their counts/digest. A native producer may have extended the execution while its marker was recovering; those canonical keys join the final proof. Expected historical keys must still exist. No history or network read runs inside this final transaction. Reconstructed proofs retain strict expected-set equality.

The worker runs reconciliation before accepting activities and again on every heartbeat. Legacy activity history is decoded into queued, running, completed, and failed step events. Exact source observations and preparation-workflow roots restore execution membership and work plans. Existing partial rows are merged idempotently, so a process kill during repair resumes without duplicate events or jobs. Native activities retain ownership of their normal events; mixed legacy/native histories merge the two key sets without replaying native work. Decoder v3 also reads the exact native activity attempt that Temporal proves terminal. If the activity process stopped before recording its final durable event, reconciliation appends that missing terminal event and refreshes the projection. A terminal workflow therefore cannot leave one of its native steps queued or running. Ambiguous provenance is refused rather than guessed.

Reconstructed decoder v2 does not treat workflow_run_projections as causal evidence because repeated generic workflow-start events can legitimately fold a full preparation input down to a job-only summary. For each succeeded legacy preparation-fanout attempt, it selects the exact job-only JobPreparationWorkflow starts inside that activity's recorded time interval and requires their distinct workflow count to equal the fanout's declared target count. It unions overlapping targets from repeated passes, then resolves each exact workflow/run against its append-only full WorkflowStarted summary and validates the job URL, required steps, and prep-{idempotencyKey} identity. Any count, run, identity, or plan conflict keeps the checkpoint in automatic retry; a successful repair records the bounded reason code legacy_history_recovery and verifies the complete membership and step-key digest before publishing ready.

Temporal retry attempts remain distinct causal attempts. A successful retry can be recovered from its exact start and completion events even when legacy history has no separate queue event for that attempt; the decoder leaves queuedAt unknown rather than fabricating a timestamp. If a fanout is terminally failed with no retry remaining, the decoder publishes incomplete from the exact partial dispatch set and failed-step evidence instead of retrying forever.

activeStageCounts is a separate aggregation of allowlisted activity types from fresh worker heartbeats. It is null when that runtime inventory is stale or unavailable. These counts expose what the shared worker pool is executing now; they never fabricate selected-execution membership, pipeline-step rows, or a durable stage denominator.

While coverage is recovering or retrying, the Pipelines surface keeps worker, slot, queue, and active-activity facts visible, labels selected-run history as being restored, and hides selected-run counts, percentages, and ETAs. Recovery is an automatic transition state, not an accepted tracking mode. For selected scopes, no_work is valid only after coverage is ready, execution membership is closed, and projected demand is exactly zero.

incomplete is a terminal audit state, not a degraded operating mode. Pipelines shows the exact partial evidence and bounded reason, withholds claims about the unknown remainder, and offers a new Discover setup only when fresh runtime inventory proves there is no active work.

Runtime capacity and conservative ETA

Worker heartbeats add runtime facts that cannot be reconstructed from durable events: exact activity slots in use, configured capacity, bounded allowlisted active-work detail, completed-activity duration summaries, and approximate Temporal task-queue observations. The operations reader derives the expected app directory from the configured database path, filters heartbeats to that resolved database/app-dir identity, selects the task queue named by the newest matching heartbeat, and aggregates fresh schema-valid workers from that queue. Active slots count every activity; active-item detail is a separate oldest-first list capped at 20 with an exact allowlisted total and truncation marker. This global runtime inventory remains available when no Discover execution is selected, including manual URL intake and preparation activity. URL imports appear as job_url_import / job-url-import runtime items with the label Import job URL. Intake does not invent a pipeline stage or selected Discover membership. Fresh empty inventory reports zero; unavailable or stale inventory reports unknown. A fresh preparation activity can receive its job title and company from an exact same-tenant JobPreparationWorkflow projection: the workflow ID and Temporal run ID must match the runtime detail, and its text jobId must join a canonical Job. This bounded, read-only join works without a selected Discover; it reuses the opaque runtime display key and never exposes workflow inputs. Selected-membership enrichment also requires the exact preparation run. Missing, malformed, foreign-tenant or stale-run bindings remain unresolved. These job labels do not establish Discover membership, change selected-run counts or coverage, or contribute new ETA evidence. Their absence does not erase worker activity.

This boundary is deliberately privacy-safe. The interceptor never reads activity arguments. It retains only allowlisted activity kinds, validated safe workflow/run references, and non-reversible local opaque identifiers. URLs, descriptions, profiles, prompts, provider responses, artifact paths, payloads, credentials, and exception text cannot enter heartbeat detail. Task-queue pollers, backlog count/age, and add/dispatch rates are separately labeled as approximate infrastructure observations, never as job counts.

pipeline-eta-v1 uses successful canonical stage or pipeline-step durations (operational-attempt metrics are fallback evidence only), a 14-day window, at most 50 recent samples, and a minimum of five samples for every remaining stage. The overall numeric range is withheld while membership is open; per-stage and source-family ranges can price their already-known scoped backlog before closure. A source-family range uses the currently observed source concurrency rather than claiming every configured slot. Dormant global rows outside the selected execution are not live contention until their canonical state is queued or running, or activity inventory or the Temporal queue observes them. Selected-run sweep work reserves one slot per remaining preparation workflow rather than one per sequential stage; the source estimate is withheld when that demand exceeds spare capacity. Unbounded retry demand, queued work, and truncated inventory remain unbounded. Every estimate still withholds a number when telemetry is stale, a worker is unavailable, work is budget-blocked, or shared contention is unbounded. Non-numeric calibrating, paused, stale, and unavailable states are part of the contract. The web surface presents a paused estimate as No ETA plus the bounded reason so it cannot be mistaken for the Temporal workflow's lifecycle state. In particular, no_dispatch means the estimator has not observed recent dispatch activity; it is not a resumable workflow command. An available low/high range is a current projection estimate, not a completion promise.

Persistence Map

The Python worker writes to a single local SQLite database. Tables group by context; the append-only job_events log plus the projection tables are the read-model spine.

Group	Representative tables
Discovery	jobs, source observations, source_registry_entries, source locator / manual-capture / review queue, quarantine, discovery_runs, discovery_execution_jobs, discovery_settings, plus target search overlaid from candidate_profiles
Enrichment	enrichment fields / rows on jobs, posting content snapshots
Scoring	job_scores, scoring_policies, job_score_staleness, employer analysis
Materials	materials sets / tailored resumes, cover letters, rendered PDFs, tailoring_policies
Apply	apply stage state + apply lifecycle in job_events (see note)
Contact & Outreach	contacts, contact_attributes, contact_research_tasks (+ source_attempts_json), contact_candidates; projected into contact_projections + contact_research_task_projections
Orchestration / read model	job_events (append-only), operational_attempt_metrics, job_stage_states, pipeline_step_projections, workflow_run_projections, job_list_projections, job_detail_projections, dashboard_projections, artifact projections, apply_run_projections
Spend	llm_spend
Runtime	worker_runtime_heartbeats with runtime identity, exact slot totals, bounded safe activity detail, duration summaries, and task-queue observation

Note: the legacy apply_runs / apply_run_events tables were dropped at boot. Apply lifecycle now lives entirely in job_events and is projected into apply_run_projections.

Operational metrics are append-only rows written at pipeline boundaries (stage, source id, source role, adapter, attempt kind, outcome, counts, durations, error_class, error_message, run_id, job_url when known) rather than inferred from labels — so discovery_runs.status='failed' no longer has to carry unrelated failure causes.

Domain Events, Projections, and SSE

Retry-and-run preserves the same audit boundary: the API verifies worker readiness before resetting a stage or appending the reset event. A 503 worker_runtime_unavailable leaves the durable failed/blocked stage and its events unchanged. The reset happens first only after dispatch prerequisites are satisfied; a reset-only command remains a deliberate local transition. Explicit Enrich reset uses the same Python application service from HTTP and CLI. Its atomic reset and durable event invalidate the previous execution fence without discarding accepted content. See Enrichment Retry Ownership for transactions, bulk cohorts, attempt counters and downstream preservation.

The authoritative TypeScript runtime registry lives in packages/domain-types/src/events/. The frontend's every-event-has-handler test requires an invalidation handler for every member, and test_domain_event_parity.py requires the Python registry to match the same ordered tuple exactly. Both projection builders fold the same camelCase payloads, including the Workflow* and PipelineStep* lifecycle events.

Three catalog corrections, because the old doc drifted:

• There is no CoverLetterFailed event. Cover success is CoverLetterGenerated; cover failure surfaces as StageFailed + WorkflowFailed.
• StageQueued is not a typed domain event. It is not in the canonical registry. The TS bulk routes tag reset/queued rows with a StageQueued marker string (source: "bulk_retry_failed" / "bulk_run_pending_preparation"), but it is not folded like a domain event.
• DiscoveryRunProgress is not a domain event. It is the heartbeat progress payload persisted onto the discovery_runs aggregate; the typed discovery-run events are DiscoveryRunStarted / Completed / Failed.
• Pipeline-step lifecycle is typed and execution-scoped.PipelineStepQueued, PipelineStepStarted, PipelineStepCompleted, and PipelineStepFailed carry the exact Discover workflow/run identity plus bounded step kind, item key, codes, counts, and timing; raw activity payloads and exception text are excluded.

The read path is projection-backed, and there are two projection builders: the Python ProjectionBuilder (in the worker, bus-subscribed and also refreshed explicitly by finalize/reconciler) and the TypeScript refreshProjections (in the API). Both rebuild the same projection tables from the same events.

job_list_projections.current_stage is a product-stage field: builders write only discover or apply there (the full internal stage list stays in job_detail_projections.stages_json), with the cover→apply advance described earlier. Note that GET /v1/events/stream is a 250 ms poller over new job_events rows, not a push stream — which is why a stage can complete a beat before the UI card visibly changes: durable facts are recorded first, then projections refresh and the next SSE tick invalidates the query cache. The SSE contract is specified in Local TypeScript API.

Projection Recovery

Read-model projections are rebuildable state; two recovery paths plus a defensive dashboard read keep them truthful on existing databases:

• Both initialization paths — the Python worker and the TypeScript API — migrate legacy workflow_run_projections schemas before use, so older local databases cannot fail projection upserts mid-workflow.
• When workflow events were already watermarked but the Python-owned workflow-run projection table is missing rows, the projection is rebuilt from the event log instead of staying silently empty.
• Dashboard pipeline progress consults terminal workflow-run state, so a stale StageStarted row can no longer present a dead workflow as running.
• pipeline_step_projections uses the shared operations watermark and can be rebuilt from the append-only lifecycle events; projection refresh is attempt-aware and idempotent in both languages.

Failure Behavior Summary

• Transient failures retry; preconditions fail fast. Retryable errors retry up to each activity's attempt cap; configuration/authentication/ missing_input/budget_exceeded never retry.
• Discovery isolates sources. One failed source family yields a partial result; the workflow fails only if a family fails after retries, and it fails with the source error, not a swallowed one.
• Broad-board retries isolate search units. JobStreaming query/location/board units persist accepted results before acknowledgement, fence old activity attempts, and resume only unfinished work. Cancellation terminalizes units; incompatible checkpoints fail rather than reset silently.
• Preparation isolates jobs. A failed step fails only that job's workflow and resumes at the failed step; other jobs are unaffected.
• Apply fails safe. At-most-once + one live attempt + the CDP dry-run guard mean a failed or canceled apply never double-submits; cancellation is cooperative and terminalizes as WorkflowCanceled.
• Nothing stays "running" forever. Finalize records the terminal outcome on every normal/cancel path; the describe-based reconciler backstops killed workers, timeouts, and dev-server history loss. Short lifecycle writes use bounded local capacity independent of the remote model-work pool, so completed intake can acknowledge its result while scoring remains busy. Exact execution/event keys make those writes safe to repeat; versioned replay retains the old remote lifecycle history. Spend preflight and owner-scoped preparation cancellation share that same two-slot local control pool with independent patch markers. Their SQLite work runs off the worker event loop, each attempt is limited to 30 seconds, and scheduling plus retries is limited to three minutes. Old remote control histories retain their original commands. This removes dependence on remote model-slot occupancy; it does not shorten model calls or bypass spend, cancellation ownership, or accepted-result preservation checks. Per-job Score cancellation uses a separate replay marker, waits for the activity's bounded cancellation cleanup, and settles only its exact owned stage. Direct Score activities carry a cooperative stop event and fence both score publication and terminal stage writes against cancellation or a newer owner. Lost ownership skips the predecessor instead of retrying it. Startup recovery also checks ordinary preparation owners against their exact Temporal execution, canonical job and selected step; unavailable history leaves the row untouched. Already committed results survive cancellation. Direct admission and cached-score reuse check current ownership under the writer lock before changing a stage. Recovery mutates only the verified job for a single-job preparation and requires an explicit known terminal status; an unknown status never releases ownership. To revoke queued work, cancel its exact workflow and observe terminalization before resetting the stage. A reset-only action changes local stage state without canceling Temporal work.

Recurring Saved Posting Checks

Startup and heartbeat/reconnect admit bounded availability sweeps independently of Discover. All visible saved jobs are due immediately before the first check, then active evidence is due after 24 hours and confirmed unavailable evidence after seven days. Unknown acquisition retries after five minutes with exponential backoff capped at 24 hours; bounded Retry-After contributes a shared host cooldown. An explicit request can bypass evidence backoff after the one-minute job limit, but cannot bypass quotas, cooldowns or safety checks. Local refusals preserve evidence clocks and backoff; only explicit commands own coalesced deferral feedback. Preparation/ review jobs are preferred, then oldest due with stable JobId ordering. Sweeps read quota capacity before each claim and stop when saturated; they stop starting new checks within a 20-minute budget below the 30-minute activity timeout. Activity cancellation propagates to the acquisition and supervised browser cleanup. Stopped workers leave clocks intact and the UI honestly reports overdue evidence.

Standalone check failures preserve accepted content/materials, approvals, application attempts and outcomes. Reopening updates availability without another Discover run. Confirmed unavailable evidence blocks preparation/Apply. Unknown evidence permits preparation and independently authorized rehearsals or bound human-reviewed Apply, with an exact approval from the preceding 15 minutes, while unattended Apply retains the positive active-evidence gate. Claim, run and owned intent share that supervision rule; a refused candidate cannot starve the remaining queue. Reads never acquire. Candidate posting URL and the applicable availability policy are rechecked under the short queued/manual-to-running writer claim for Score, Tailor and Cover, then released before provider work. Reviewed email intent carries the original posting URL and checks it with freshness and run ownership under the intent writer boundary; replacement-posting evidence cannot authorize the original intent. See the Enrichment user policy.