Runtime & Processes
JobCtrl runs as four long-lived local processes, plus a jobctrl rpc subprocess the TypeScript API spawns on demand. This page walks each runtime boundary — what it owns, what it must never do, and how the pieces talk.
For a focused explanation of the workflow-engine choice, read Temporal Workflows In A Desktop App.
Read this if you need to know which process is responsible for a behavior, or where a change belongs.
| Process | What it is | Owns |
|---|---|---|
| The web app | React/Vite single-page app (apps/web) | user interaction and view state |
| The TypeScript API | Fastify server (apps/api), loopback-bound | typed read models and local product endpoints; spawns the jobctrl rpc subprocess |
| The Python worker | jobctrl worker, a Temporal worker | executes workflows and activities (discovery, scoring, tailoring, apply, …) |
| Temporal dev server | the workflow engine (gRPC :7233, Web UI :8233) | durable workflow execution and history |
jobctrl rpc subprocess | spawned by the TypeScript API on the first JSON-RPC call and reused | JSON-RPC dispatch for complex commands |
You start the first four; the API spawns and reuses the jobctrl rpc subprocess itself.
Agent execution
The Python worker and RPC process use a process-owned agent-runtime-kit service for provider generation. Its persistent event loop, bounded idle cache, fresh request conversations and cancellation/shutdown rules are defined in Agent Runtime Ownership. Pipeline admission and Temporal activity capacity remain owned by their existing layers.
Apply browser ownership
The Apply browser lifecycle records the exact Popen handle and original PID when it launches an isolated browser. Worker cleanup, launch failure, cancellation, capability revocation and Python exit use those records. They never discover termination targets from a CDP port or process name. An occupied CDP port fails before profile setup with guidance to choose another port or stop its owner explicitly. Worker IDs already being acquired or owned cannot replace a live record.
On POSIX, start_new_session creates the browser's own session and group. Cleanup verifies that the original child still leads that session and group, and holds CPython's Popen reaping lock across verification and signaling. The unreaped child prevents PID/group reuse during that operation; a reaped leader or changed identity cannot authorize a group signal. On Windows, cleanup terminates the original browser kernel handle; it never invokes a PID-based tree-kill tool. That root exit cannot certify descendant exit, so Windows Apply cleanup reports unavailable descendant verification and retains ownership. Native Windows Apply remains unqualified; this does not change the separate Windows credential adapter.
Root exit alone does not prove POSIX session exit. After waiting for the root, cleanup uses a read-only kernel process census (libproc on macOS and /proc on Linux) to observe the old session, including members in other groups. For an installed Apply child, the authenticated native scope is checked around its birth. A read-only scan can then use the scope's kernel task-start counter as its fork fence: task-start and exit counts must remain unchanged throughout the scan, every active task in the resource coalition must be accounted for, and no live session member may be observed. An uninspectable owned task keeps the proof unresolved. Other scopes may start and stop without invalidating it. If the binding or counter is unavailable, PID enumeration is fenced before and after inspection and retirement requires two complete identical global PID/generation/membership inventories with no live session members. macOS kernel unique IDs/PID versions and Linux kernel start ticks bind process generations; macOS also verifies identity around separate session/status reads. A fork followed by a session handoff, PID reuse, changed membership or unavailable/incomplete evidence keeps cleanup unresolved; a leader already exited before cleanup never starts new signal authority. Zombies count as exited. The observation does not recover authority over descendants that escape the launch session.
Scoped browser and agent cleanup attempt each selected worker and preserve failed ownership records while reporting incomplete cleanup. Worker-specific verification and adapter termination use per-worker guards, so one waiting handle does not serialize an unrelated worker. Publication of a tracked birth uses the same worker guard and checks its exact run and epoch; a standalone child whose publication and cleanup both fail remains separately retryable. The agent reserves its worker's MCP configuration before writing and releases that reservation after its own final unlink. A remaining file reservation alone does not imply a live or unverified process. The original child receives TERM first and has a bounded grace to drain. Only a still-owned, unreaped launch group is eligible for KILL after that grace, with its original PID pinned even if the leader exits during the wait. A leader already exited before cleanup cannot authorize a new signal. The agent adapter verifies cleanup after a naturally exited root before removing its record. A launcher cleanup pass shares a deadline across selected handles and ownership-lock waits. A full-run interrupt attempts available workers before waiting on contended worker guards. It fences concurrent birth attempts until the pass finishes, then permits later births if the run remains active. Handles registered during a pass remain owned and prevent that pass from declaring completion. Stream closure runs outside the run lock and remains tracked if a reader holds the pipe open. Expiry or an unavailable lock retains custody; neither extends signal authority to a reaped or newly discovered PID. The exit handler logs failures and does not sweep ports or adopt leftovers. These rules complement the native instance shutdown boundary described below.
Source and bundled runtime boundaries
The table above describes the source-development topology. The bundled production boundary packages the same application without carrying the checkout or asking the user for a language/runtime toolchain:
- the Vite output is static and is served by the loopback Fastify API, so an installed runtime has no Vite process;
- the API is compiled for the embedded Node runtime and its native SQLite module is pinned for that exact Node ABI;
- the worker and CLI run under an embedded CPython with only the core runtime closure; fixed relative system-site entries expose the separately owned worker and Python Playwright component roots;
- Temporal, one Python Playwright Chromium headless-shell revision, and Playwright MCP are addressed through absolute payload paths and never through user
PATH,npx,uv, or a package registry at execution time; and - PDF page previews are rendered by PDF.js in the web client. The installed runtime has no Poppler component or
pdftoppmroute.
The authenticated protocol-2 installer owns the forward upgrade from an original protocol-1/schema-14 installation. It verifies the retained predecessor as historical evidence and the target for execution under the transition and selection locks. It uses normal instance cleanup before the paired backup, never executes the predecessor, and continues an interrupted upgrade forward. Before selector replacement, recovery requires rerunning the trusted new installer because the immutable old binary cannot decode the new journal. After replacement, ordinary jobctrl update uses the new controller. There is no separate bridge catalog or reboot precondition. Actual signed/notarized release qualification remains separate from source merge; see the release contract.
JOBCTRL_RUNTIME_MODE=bundled makes this boundary fail closed. The launcher must supply an absolute JOBCTRL_PAYLOAD_DIR; the API realpaths its Python, web, and Chromium roots and rejects paths outside that payload. Python isolated mode ignores ambient PYTHONHOME, PYTHONPATH, user-site, and virtualenv state. Bundled dotenv discovery is limited to the JobCtrl-owned state file and never searches the current directory or a checkout.
Provider pack directories identify the exact signed wheel closure as well as the SDK version. A changed closure installs alongside the previous directory; activation validates its schema-2 installation identity against the signed payload. Compatible legacy schema-1 version directories retain the same full wheel and tree verification. See Agent Runtime Ownership.
Provider runtimes that JobCtrl cannot redistribute are a separate mutable boundary under the JobCtrl state directory. The signed payload owns the exact provider-pack lock; installation accepts only that lock, verifies every wheel's official HTTPS source, size, and SHA-256, rejects unsafe or overlapping wheel members, and retains the locked wheels. Every activation revalidates those wheels, deterministically re-extracts their expected tree, and compares it with live site-packages, so mutable activation metadata cannot authorize code. Pack paths are appended after the core runtime; core overlap is rejected and provider-to-provider overlap requires signed-identical wheel records. In bundled mode Claude accepts API/cloud authentication only, launches with --bare, and does not probe or reuse consumer Claude Code OAuth or Keychain credentials. Codex uses persisted CLI authentication in the stable $JOBCTRL_DIR/codex_home; raw OpenAI keys are accepted only as stdin enrollment input to codex login --with-api-key, never as a direct runtime credential. When isolated auth is absent, setup and generation retain one-time reuse of a regular Codex CLI auth.json; the explicit Codex provider verify action uses the same copy-once behavior before checking the isolated login. Existing JobCtrl-owned auth is not overwritten, and the normal Codex home is unchanged. The generated workspace also has an explicit private project root, so SDK configuration discovery cannot inherit unrelated settings from its physical ancestors. Its initialization and validation are defined in Agent Runtime Ownership. Only auth.json is reused; user configuration, instructions, skills, sessions and logs are never copied. Generation threads are ephemeral. Live native clients in one process share a private temporary sqlite_home for the same isolated credential home, avoiding repeated imports of retained provider history into separate databases. Each client has its own temporary log_dir. The last client removes the shared SQLite store, including SQLite logs, only after every client has closed; startup failures and cancellation release the same leases. An OS lease protects active stores from reclamation. A later startup can reclaim an abandoned store only when its private ownership record matches the same credential home and its OS lease is available. Unknown or redirected paths are preserved. Existing files in the stable credential home are preserved. JobCtrl retains its own provider-error, usage and provenance records. SDK processes clear ambient credential values and pin refresh/revocation URLs to OpenAI's default HTTPS endpoints. These URL overrides cannot be blank: Codex consumes an empty value literally and cannot build a refresh request. The auth file stays outside codex_home/workspace/; the permissions profile denies root reads and grants prompt-driven reads only to that workspace subtree, minimal runtime paths, and the one canonical Codex executable required to start the app server. It never grants the executable's parent directory, its site-packages tree, or sibling files. Structured Claude analysis/voice calls expose no built-in tools. Codex analysis disables its shell, denies approvals, and gives any command child an empty inherited environment, so provider API keys authenticate the SDK transport without becoming model-readable tool input.
Bundled doctor validates the payload-owned executable Playwright MCP wrapper instead of requiring system npx. Nested Python setup/doctor and MCP processes run with isolated mode plus bytecode suppression (-I -B) so they cannot write __pycache__ into the launcher-verified payload.
The installed payload uses a native jobctrl supervisor. It verifies the manifest envelope/tree before dispatch, then starts fixed loopback Temporal (gRPC 7233, Web UI 8233) → worker → API (8766) and waits for Temporal and API worker-heartbeat health. Each canonical JOBCTRL_DIR has one locked runtime registry. Records bind its instance, payload, manifest, PID/start identities, process groups and execution scope.
Runtime starts and Python CLI commands execute inside uniquely labeled, ephemeral per-user launchd jobs. The launcher records the kernel resource and jetsam coalition IDs in a private instance receipt before authorizing work. These IDs remain attached to descendants through fork, new sessions, changed environments, platform executables and parent exit. Shutdown enumerates those kernel memberships and rechecks each PID/start identity before signaling. Names, command substrings, environment tags and numeric process groups do not replace that durable ownership proof. Receipts contain ownership metadata, never the full environment or credentials; command configuration and standard descriptors travel over a private Unix connection. Jobs do not install automatic startup. The boot-session identity only prevents stale coalition-ID reuse after a natural OS boot; it imposes no reboot prerequisite.
stop requests the verified supervisor's ordered clean shutdown first. API and worker handlers close HTTP/RPC/provider work and drain admitted blocking activity threads. Foreground cancellation reaches the command root first; it does not signal its cooperative children concurrently. After the configured grace, verified leftovers receive TERM, then KILL after their timeout, followed by an actual zero-member proof. The kernel's atomic resource-coalition task count proves zero without requiring other applications' PID inventories to remain stable. A currently executing control command and its exact live transport are excluded only after their scope and process identities are verified. If the counter is unavailable, the existing complete inventory proof remains. Detached descendants remain identifiable even when all original parents and groups are gone. Other instances stay outside that ownership boundary. When a normal command finishes with no remaining tasks, its original controller can attest its own exact receipt after joining the helper. Stop may use that proof if the kernel later reports the same-boot coalition reaped. The controller authenticated that scope at birth; an absent ID from a later receipt read alone is not completion proof. A Stop may also remember an exact scope it directly observed while draining, then use kernel reaping as zero proof for that same scope and boot. It persists exact per-scope completions before the stopped registry checkpoint, so interruption leaves the same evidence available to recovery. Successful cleanup permits normal restart.
Stop's control lock also serializes new execution admission. After cleanup, the launcher durably publishes the stopped proof before retiring exact owned launchd labels and receipts. Interrupted finalization can be repeated without losing ownership evidence or blocking restart behind a deleted receipt. Missing, changed or unfamiliar active ownership metadata produces an explicit cleanup failure and preserves the registry and database pair. Legacy registries that never recorded complete ownership remain ownership-unverified; an empty old process group cannot certify their historical descendants. Do not delete that registry or restore old databases to bypass the refusal. Manual recovery must establish the missing boundary; reboot is last-resort troubleshooting for an unrecoverable cleanup failure, not normal stop, backup, update or restart.
Python CLI execution retains the shared release-selection lock across exec. Backup, update, rollback and uninstall take exclusive selection, and direct backup always invokes the same complete stop gate even without a live recorded leader. The API/worker drain and SQLite locks alone do not establish process ownership. The QA matrix requires detached platform and writer cases, other-instance preservation, ordered cancellation, actual process interruption, paired readback and restart.
The distribution smoke runs native controllers outside Seatbelt so launchd can establish the jobs, while the exact internal QA selector applies the compiled loopback-only sandbox to the authenticated helper and all payload descendants. Actual native Python dispatch proves loopback access and non-loopback EPERM. Direct extracted Python/Node/Temporal/MCP smoke subprocesses keep their existing sandbox. Signed/notarized artifact and clean-machine qualification remain release/first-use gates, separately from synthetic source proof.
Homebrew caches only prefix-owned release resources. Its build sandbox verifies strict Developer ID certificate-chain requirements for the launcher, installer and managed headless shell without contacting Apple's live syspolicy service. Before a cached release's installer can execute, the normal-user bootstrap requires strict notarization checks on both bootstrap executables. The installer then checks every payload executable's signature and notarization, and assesses outer application bundles through Gatekeeper, before selecting the payload. This preserves the pre-activation trust boundary without changing Homebrew's sandbox permissions or treating ad-hoc signatures as Developer ID signatures.
The P5 lifecycle store is user-owned (JOBCTRL_RUNTIME_HOME), not a package manager Cellar. active.json atomically records the selected payload, the compatible immutable selector build, and the current acquisition adapter; immutable release receipts never contain acquisition ownership. A durable transition journal records selector handoff, database-pair backup, pending and finalized channel policy, and promotion. Acquisition, update, rollback, backup, retention, and uninstall serialize through transition then selection locking; selector resolution holds a shared selection lock through supervisor readiness. Before a candidate is promoted, native cleanup must prove that instance-owned processes and descendants exited, including verified leftovers after the clean shutdown timeout. Both JOBCTRL_DIR/jobctrl.db and JOBCTRL_DIR/temporal.db receive WAL-aware, hash-verified online backups from the stopped pair. While a journal is unfinished, ordinary commands remain blocked except cleanup and forward-update recovery. Internal start --no-open and its detached supervisor may proceed only from the exact payload authorized by the matching durable journal, executable transition state and immutable manifest.
For signed protocol-2 targets, the journal binds source generation/schema and candidate identity and records original and recovery backup IDs. Each retry backs up the current live pair. Exact-v14 activation is additive and preserves the application database inode. Readiness failure or interruption keeps live data and the journal. Before selector replacement, rerun the trusted new installer; after replacement, jobctrl stop followed by jobctrl update continues forward. Pure native cleanup authenticates immutable evidence without requiring execution permission, so a revoked candidate's writers can still be stopped. Update and startup retain current policy checks. Revoked predecessors may remain authenticated historical evidence but cannot execute. A revoked candidate, changed source selection, regressed schema or unknown legacy handoff refuses without restoring stale data. Cross-protocol rollback is refused. Same-protocol explicit rollback retains its separate paired restore contract and must preserve the live pair on a target-start failure. The signed launcher/update-protocol.json declaration distinguishes controllers that support journal-bound selector startup from historical selectors that would wait on their parent's selection lock. A new installer owns staged activation; it does not delegate migration control back to the older installed executable. For an already interrupted update, a new formula can stage its verified recovery controller without replacing the old/candidate journal or activating new data. A signed forward update requires an initialized predecessor pair. If it has never created either database, initialize and stop the original version before retrying the new installer; the forward controller does not execute it. A partial pair, unsafe file or orphaned SQLite sidecar is refused before the update changes release selection. The macOS source controller shares the installed workspace selection and instance locks and uses an authenticated per-user process scope for foreground and detached source fleets. For an owned schema-14 pair, source startup stops the owned fleet, creates a WAL-aware paired backup, builds and binds an exact-v15 candidate, and activates it in place. A source journal fences installed writers until recovery finishes; retries take a fresh stopped-pair backup instead of restoring old data. A locally trusted checkout does not impersonate a signed release. Unverified older source processes are not migration authority.
The current runtime admits only exact schema v15. A stopped v6 database retains the Temporal quiescence proof and private v7/v8/v9 intermediates; exact v7 and v8 start at their next intermediate. Exact v9 transfers application URL authority to enrichment and retains historical lookup aliases before removing the legacy job column. Exact v10 then moves historical global llm_spend rows into the explicit legacy lane before v11 activation. The legacy lane contributes to global USD totals but is never a runtime write target. Exact v11 adds generation bindings through private v12, then v13 withdraws heuristic classifications and adds persisted determinations. Exact v14 removes the redundant search-preference approval binding and retains historical raw intake captures. New source results ingest without a separate intake model gate. Exact v15 adds durable job lifecycle revisions for reviewed permanent deletion. Historical employer-analysis generations remain stored, while current readers select only the v4 determination contract. Neither Python nor the TypeScript API runs against an intermediate schema. Successful or failed prepublication builds clean up their owned candidates and sidecars; ambiguous activation preserves the live pair, journal and evidence for recovery. Policy finalization follows readiness and a durable paired backup. Forward recovery authenticates the target under current channel policy and never executes a revoked predecessor. Published builds have passed signing, notarization, authenticated publication, and clean-machine acquisition gates before this lifecycle boundary can promote them.
Frontend
The web app under apps/web owns user interaction:
- dashboard summary
- jobs list and job detail
- artifacts list
- profile/style editor shell
- filtering, sorting, pagination, and selected-detail workspace state
- UI action buttons
The frontend uses @jobctrl/api-client for API transport and @jobctrl/contracts for shared schemas and DTOs. It should not know shell command syntax.
The frontend follows its own DDD + hexagonal target documented in the Frontend section — three-layer state separation, nine bounded contexts that mirror the backend 1:1, view-vs-context dichotomy, hexagonal frontend ports, Server-Sent Events (SSE) realtime via the invalidation router, and a projection-typed Operations read-side. The summary below cross-links to those pages; the Frontend section is the canonical detail.
Stack
| Concern | Choice | Target ref |
|---|---|---|
| Bundler / dev server | Vite (SPA today; TanStack Start named-not-built for SSR) | §4.1, §9.1 |
| UI library | React 19 | §4.7 |
| Styling | Tailwind CSS 4 with design tokens in tokens.css; darkMode: ["selector", "[data-theme='dark']"] | §4.8 |
| Component primitives | shadcn Rhea/Base preset over Base UI; copied + owned wrappers in shared/ui/ | §4.7 |
| Router | TanStack Router (file-based via @tanstack/router-vite-plugin) with route-level Zod search-param schemas | §4.3 |
| Server state | TanStack Query v5 with per-context query-key factories, tenant-first keys, central registry in contexts/operations/queryKeys.ts | §4.1, §4.4.1 |
| Tables | Shared filterable data grid (shared/ui/filterable-data-grid.tsx); column models (DataGridColumn<T>[]) live with the consuming view; cell renderers are imported from contexts; @tanstack/react-table supplies selection/sorting types only | §3.10, §11 |
| Forms | TanStack Form + Zod safeParse | §4.6 |
| Client state | Zustand (shared/stores/) — UI prefs, toast queue, command palette, profile-import wizard draft (persist middleware where durability matters) | §4.9, §4.10 |
| Test runner | Vitest + React Testing Library + MSW for unit / hook / component | §10.2, §10.3 |
| End-to-end | Playwright against a seeded local TypeScript API + SQLite fixture | §10.4 |
| Component-driven dev | Storybook with addon-msw and addon-a11y (critical+serious axe violations fail CI) | §10.5, §10.7 |
| Type-level tests | Vitest typecheck mode via vitest.types.config.ts; *.test-d.ts files live under apps/web/test/types/; invoked as pnpm --filter @jobctrl/web test-d | §10.6 |
Three Layers of State
Every piece of state lives in exactly one layer (Frontend §2.1):
| Layer | Owner | What lives here |
|---|---|---|
| Server state | TanStack Query cache | API-derived projections, profile, settings, dashboard summary — anything fetched from apps/api. |
| URL state | TanStack Router (typed search params via Zod) | Anything bookmarkable: view, filters, sort, page, page size, selected entity, detail-workspace/inspector state. |
| Client state | Zustand (with persist where appropriate) + React context | Theme, density, tenant context, transient UI like toast queue, ephemeral form drafts that do not survive navigation. |
No server data in useState; no filter / pagination / sort / detail-route state in useState; no durable user preferences in component-local state; one source of truth per fact; components consume state through hooks (never raw stores or the QueryClient directly).
Frontend Bounded Contexts
apps/web/src/contexts/<name>/ mirrors the backend's nine bounded contexts 1:1 (Frontend §3, §11):
| Frontend folder | Owns | Backend mirror |
|---|---|---|
discovery/ | useDeleteJobMutation, useDeleteJobsBulkMutation, useRestoreJobMutation, useRestoreJobsBulkMutation, useHideJobsBulkMutation, useUnhideJobsBulkMutation, usePermanentlyDeleteJobsBulkMutation; future useImportJobMutation. | Job Discovery |
enrichment/ | JobEnriched / EnrichmentFailed invalidation handlers; future useEnrichmentRetryMutation. The enrichment aggregate is internal to Discovery's detail queue drain. | Job Enrichment |
profile/ | useProfileQuery, useUpdateProfileMutation, useImportResumeMutation, settings + credentials hooks, profile-import wizard store, profile editor + resume preview components. | Candidate Profile |
scoring/ | <ScoreBadge>, <ScoreBreakdown>; future useCorrectScoreMutation. | Scoring |
materials/ | useGenerateMaterialsMutation, useOpenArtifactMutation, generate / open buttons. | Materials Generation |
apply/ | useApplyJobMutation, useDryRunApplyMutation, useCancelApplyMutation, <ApplyButton>, <DryRunButton>, <ApplyRunBadge>, <ApplyRunTimeline>, <ApplyHistory>. | Apply Automation |
pipeline/ | useRunPipelineStagesMutation, useRetryStageMutation, useCancelStageMutation, useMarkAppliedMutation, useMarkSkippedMutation, <StageTriggerPanel>, <StageBadge>, <StageTimeline>, <JobActions>. | Pipeline Orchestration |
operations/ | Projection/runtime read hooks (useDashboardSummaryQuery, useJobsListQuery, useJobDetailQuery, useArtifactsListQuery, useArtifactDetailQuery, useApplyRunsListQuery, useApplyRunQuery, usePipelineOperationsQuery); query-key registry; SSE subscription; invalidation router. | Operations / Read-Side |
outreach/ | Contact records with provenance: useContactsListQuery / useContactDetailQuery, create / update / delete / import-contact mutations, contact provenance + role components, the Contacts view + Jobs-detail panel, and contact event handlers. | Contact & Outreach |
The view folders under views/ (views/dashboard/, views/jobs/, views/artifacts/, views/apply-review/, views/pipelines/, views/runs/, views/discovery/, views/outreach/ (Contacts), and views/debug/) are composers, not contexts (Frontend §3.10). They import hooks from contexts/operations/ and components / mutations from aggregate contexts; they own layout and view-local ephemeral UI (e.g., bulk-selection sets) and nothing else. View → context dependency is one-way; views never depend on other views.
Hexagonal Frontend Ports
Components and feature hooks depend only on ports; concrete adapters bind to the ports in shared/providers/PortsProvider.tsx (Frontend §6):
| Port | Local-mode adapter | Hosted-mode adapter (named, not built) |
|---|---|---|
ApiClientPort | JobCtrlApiClient from @jobctrl/api-client | Transport with hosted authentication when implemented. |
EventStreamPort | SseEventStreamAdapter (new EventSource(...)) | WebSocketEventStreamAdapter if SSE proves limiting. |
StoragePort | LocalStorageAdapter | IndexedDbAdapter when client-side cache exceeds 5 MB. |
SessionPort | LocalSessionAdapter (returns LOCAL_TENANT) | JwtSessionAdapter (Auth0 / Cognito). |
ClipboardPort | NavigatorClipboardAdapter | Same adapter. |
OpenInOsPort | OpenArtifactAdapter (POSTs to /v1/artifacts/:id/open) | Disabled in hosted mode; UI surfaces a presigned-URL download instead. |
TelemetryPort | ConsoleTelemetryAdapter (no-op) | OpenTelemetryWebAdapter → OTLP collector. |
FeatureFlagPort | StaticFeatureFlagAdapter (always default) | Backend-served via apiClient.featureFlags(); cached in Query. |
The "frontend driving ports" (use cases) are the per-context hooks themselves (useApplyJobMutation, useDeleteJobMutation, …) — React conventions are the de-facto driving-port representation; no UseCase interface is formalised (Frontend §6.7).
Provider Stack
The provider stack as wired in apps/web/src/main.tsx (top-down):
Providers within each layer keep their source order from main.tsx. EventStreamProvider lives in contexts/operations/providers/ because the Operations context owns the SSE subscription and the invalidation-router dispatch (Frontend §3.9, §7.3); every other provider lives in shared/providers/.
Realtime — SSE → Invalidation Router → Cache
The invalidation router is the integration contract between the backend's DomainEvent taxonomy and the frontend cache — a pure function tested in isolation. Every backend event has a handler; the Record<DomainEvent["eventType"], InvalidationHandler> typing makes a missing handler a TypeScript compile error, and the every-event-has-handler.test.ts parity test catches obvious empty-stub implementations (Frontend §7.4).
Test Pyramid
Frontend §10. Vitest + React Testing Library + MSW for unit / hook / component tests; Playwright for end-to-end critical flows; Storybook with the a11y addon for component-driven development. Two parity tests guard the cross-language seams:
every-event-has-handler.test.ts— everyDomainEvent["eventType"]has a registered invalidation handler.every-stage-state-has-badge.test.tsx— everySTAGE_STATE_KINDSvalue has a<StageBadge>arm.
Detailed coverage and the a11y bar live in docs/local-reliability-qa.md.
TypeScript API
The TypeScript API under apps/api owns typed JSON read models and local product endpoints. It is intentionally bound to loopback by default because it exposes local job, profile, and artifact metadata.
Current responsibilities:
- health endpoint
- dashboard summary endpoint
- jobs list/detail endpoints
- immediate public job-URL import (via JSON-RPC
job_url_import, which startsJobUrlImportWorkflow, awaits either a canonical job or Manual Capture, and hands each newly usable import to an idempotent rootJobPreparationWorkflow; Apply remains separate) - artifacts list/detail endpoints
- artifact open endpoint with known-path validation
- profile/settings read and write endpoints
- sanitized provider-model catalog reads and preferred-model validation
- resume PDF import draft endpoint (via JSON-RPC
profile_import, which startsProfileImportWorkflow) - manual-capture import endpoints (via JSON-RPC
manual_capture_import, which startsManualCaptureImportWorkflowand awaits its persisted result) - structured job action endpoints for retry, material generation, dry-run apply, cancel, mark-applied, mark-skipped
- current-policy preparation maintenance endpoints for per-job/bulk rescore and per-job/bulk re-tailor
- global/batch pipeline stage actions via
POST /v1/pipeline/actions/run-stage - the current operations snapshot via
GET /v1/pipeline/operations, combining exact Discover execution lineage, durable stage/step projections, and fresh privacy-safe runtime telemetry - pagination, filtering, and global sorting
- read-model projection refresh on every request
Simple state-transition writes (resetJobStage, retryFailedJobs, markJobApplied, markJobSkipped, cancelJobAction, correctScore, soft delete/restore, hide/unhide, permanent delete, and settings writes) execute inline in the TS process against shared @jobctrl/domain-types value objects; the full cancel action additionally fires cancel_run over JSON-RPC to signal the Temporal workflow. Complex commands and provider reads travel through SubprocessJsonRpcAdapter to the long-lived jobctrl rpc subprocess. Method-specific calls use the shared contract boundary described in Contracts, Types & API Boundaries: parameters validate before transport and results validate before consumption. Profile saves commit their audit event without admitting preparation. Startup leaves historical profile-continuation events intact without reattachment. Already admitted legacy workflows retain their existing result and replay contracts. Explicit refresh uses the Scoring profile-impact boundary. Workflow-mode handlers return a workflow spec that the RPC server starts on Temporal (run_stage, rescore_job, rescore_jobs_not_on_current_scoring_policy, tailor_job, retailor_job, retailor_current_policy, refresh_compensation, apply, profile_import, job_url_import, manual_capture_import, generate_interview_prep, run_contact_research). Synchronous handlers include analyze_job (inline three-SDK employer analysis), generate_outreach_draft, provider_status, provider_models, provider_verify, and cancel_run (cooperative Temporal cancellation). The per-job maintenance methods rescore_job, tailor_job, and retailor_job start JobPreparationWorkflow runs directly. Workflow-mode dispatch returns {runId, workflowId, firstExecutionRunId}; callers can pass awaitResult to block on the workflow result (profile, job-URL, and manual-capture imports use this).
Worker-backed action routes are gated by worker readiness: GET /v1/health reports the worker heartbeat (healthy / missing / stale after 45 s / mismatched app dir or database) plus LLM spend health (ok / over_budget against the configured dailyBudgetUsd), and mutation routes return 503 worker_runtime_unavailable until a healthy heartbeat exists. For retry-stage with runAfter: true, this gate executes before the inline stage reset. A failed readiness check therefore appends no reset event and preserves the prior stage row, attempts, blockers, and error evidence. A plain runAfter: false reset remains an explicit local state transition. The operations route applies a stricter capacity read: it derives the expected app directory from the configured database path, filters heartbeats to that resolved database/app-dir identity, selects the Temporal task queue named by the newest matching heartbeat, and aggregates fresh schema-valid workers from that queue. Unavailable/stale/invalid states remain explicit. Runtime inventory is diagnostic only and does not authorize a mutation.
Browser capability listing follows the same authority separation. The worker may detect supported system installations, but serializes only bounded candidate IDs and labels; executable paths never cross the JSON-RPC or HTTP read boundary and detection performs no adoption. Enablement accepts exactly one detected ID or one write-only path. A detected ID is resolved afresh in the worker and fails closed when stale, so read-time detection cannot become an implicit or cached grant.
Request hardening beyond the loopback bind: a Host-header allowlist rejects non-loopback hosts with 403 forbidden_host (DNS-rebinding defense), and mutating requests require a first-party local web Origin/Referer or a local capability token on a request without browser origin/fetch metadata; arbitrary loopback web origins and no-token headerless callers are rejected with 403 cross_site_request. Browser-extension routes are additive: authenticated /v1/extension/* routes still require a loopback Host and a local capability token, then allow a trusted chrome-extension:// origin through route-scoped CORS and mutation-origin checks. The browser-extension capture route seeds manual_capture_queue with extension provenance and then delegates to the same worker-backed manual-capture importer used by the web app, so discovery dedupe, snapshots, quarantine, and projections remain owned by the existing Job Discovery pipeline. Both routes start ManualCaptureImportWorkflow with a bounded SHA-256 workflow id derived from tenant plus queue-item identity. If an activity commits before Temporal observes its completion, the retry validates the persisted URL, content hash, capture metadata, and provenance before reconstructing the same result; a different replay fails non-retryably instead of returning an unrelated prior import. Deterministic browser-extension autofill reads a separate sanitized profile DTO from the Candidate Profile read path; it does not expose profile passwords, resume content, generated artifacts, or apply submission authority.
The extension background worker persists a capture in its separate local queue before the first send. Short storage mutations are serialized; network sends run outside the queue lock, and a completed-import or explicit-dismissal acknowledgement removes only its matching queued ID. A request deadline covers the response body; a timeout or nonterminal response leaves the item queued. Flushes using different pairing tokens run in order, with each retry taking a fresh queue snapshot after the previous flush settles. Pending captures survive pairing, reload, and age until an acknowledgement or explicit clear. The 25-item capacity rejects new captures rather than evicting existing ones. Legacy rows without a capture ID receive one durably before their next send. A response lost before that legacy ID existed remains historically ambiguous; later retries reuse the new ID.
Discovery adds an installation-identity layer above the token: an explicit token save binds one extension-local UUID in a mode-0600 app-dir file, and every heartbeat, lease-control, and completion request must present it. The status read model exposes only the last eight characters. A different Chrome profile holding the token receives 409 unless the user explicitly replaces the selection there. Four extension executors match the broker's four-task admission cap. Active lease-control polls keep readiness fresh and propagate worker cancellation; execution owns an independent hard timeout and tab cleanup when a task renders a page. The API validates task DNS at creation and lease. HTTP/API tasks execute in the extension service worker with redirects disabled; rendered-page tasks install tab-scoped exact-origin DNR rules before navigating. LinkedIn job-page tasks move their known owned blank tab into an unfocused window so it can render as the active tab without taking focus. Other sources keep inactive tabs. Cancellation cleans only task-owned tabs and rules, including resources whose creation or installation completes after the abort.
Acquisition transport is independent of Temporal execution ownership. One one-second loopback availability probe at each source/adapter/site-batch setup prefers the extension only for a literal connected: true; offline, malformed, or unavailable status selects the existing guarded HTTP or anonymous Playwright path. Cancellation and programming errors propagate. After selection, remote site, DNS, and access failures do not select a second transport. The next setup may choose again after a disconnect. API launch/retry gates and Pipelines controls require worker readiness, not extension readiness.
Provider Credential Boundary
Provider credential storage crosses the TypeScript/Python process boundary; it is not a runtime secret read performed by the API:
- The web Settings form uses
PATCH /v1/credentials/batchto replace one provider configuration. The fixed allowlist covers the Anthropic/Gemini keys, Claude/Google cloud activation flags and non-secret identifiers, and a legacyOPENAI_API_KEYdeletion path. AWS, Google, and Azure credential files stay in their vendor stores. A batch either applies completely or restores its pre-change native credential state; a recovery failure is explicit and sanitized.GET /v1/credentialsand post-mutation responses return presence only; eachconfiguredstate istrue,false, ornull. An inspection failure isconfigured=nullwithunavailableReason=inspection_failed, not an absent credential. An operational credential-store failure is sanitized as503 credential_store_unavailable. Secret values used internally for rollback are never returned, logged, persisted in SQLite, or passed to Python by the API. GET /v1/providers/statusasks the long-lived JSON-RPC process for sanitized Codex/Claude/Google configuration and readiness. It is read-only and never copies ambient Codex auth.POST /v1/providers/codex/verifyis the explicit Settings action that invokes the same copy-once behavior used by setup and generation, then runscodex login statuswithout generating model output. It never overwrites isolated auth or changes the normal Codex home. Because Python environment/native-credential loading is process-start scoped, Settings combines fresh presence with the last runtime status and requires a JobCtrl restart before new values become ready.GET /v1/providers/modelsuses the same JSON-RPC boundary. It is read-only and never copies ambient Codex auth. Ready Codex uses the isolated JobCtrl-owned Codex App Server SDK's livemodel/listwithout shelling out or returning account data. Ready Google uses the authenticatedgoogle-genailive model list for Gemini-key or Vertex/ADC configuration. Ready Claude reads themodelscatalog returned by the same Claude Agent SDK runtime initialization used for its configured API or cloud route. All three usesource=live; unready providers return no models.- A
preferredModelssettings patch is validated against that current ready catalog beforewriteSettingsConfigmerges the canonicalpreferred_modelsobject intoconfig.json. Clears do not require readiness. Persistence stores provider/model IDs only and remains separate from credentials. - Provider-consuming Python CLI, RPC, and worker startup paths call
config.load_env(). After env files are loaded, it considers the same fixed secret allowlist and performs a bounded native-store lookup only for a missing or empty value. The API writer and Python reader share the same native target mapping. A successful value is copied only into that process's environment; a non-empty inherited environment value always wins. - Native credential resolution is cached for the life of the Python process. There is no hot reload, so a long-lived worker or RPC subprocess must restart after a Settings edit. macOS uses Keychain, Windows uses Credential Manager, and Linux uses Secret Service. Store failures remain sanitized diagnostics, not proof that an entry is absent. No adapter persists a plaintext fallback.
- Explicit legacy
.envmigration is separate from runtime startup and Settings reads. It moves only allowlisted secrets after verified native readback; inherited environment values and vendor-managed stores are outside its scope. See Configuration for the operator procedure.
Configuration Resolution And Activation
Launch controls are resolved at their owning boundary rather than through one global configuration object. SQLite owns every durable value composed on the Discovery page, including target search, runtime, scheduling, and Apply gates. config.json owns non-secret values under Settings, including cross-process controls, provider configuration, model IDs, AI execution policy, browser adoption metadata, and apply limits. The native OS credential store owns actual secrets, while the copied browser profile, extension token, and selected installation ID remain protected separate artifacts. The copied-profile artifact is not part of integrated Discovery, including its anonymous fallback. A connected extension authorizes use of the selected live Chrome profile; the API holds its execution-bound task leases and response bodies in process memory only. Standalone copied-profile compatibility retains its existing separate capability consent.
Normal settings resolve from the saved owner and then the built-in default; explicit per-workflow model input may override a saved provider preference. API responses expose source and activation timing so the frontend can distinguish live, next-poll/run/workflow, and restart-required changes. The health heartbeat is the source of truth for active worker activity slots; config.json holds the desired value until restart. Browser capability mutations and extension-token rotation are live, while Python native-store consumers load secrets at process start.
Python Automation Engine
Python owns automation execution:
- discovery
- job detail enrichment
- Discovery preparation workflow fan-out
- scoring
- resume tailoring
- cover letters
- PDF generation
- profile import from resume PDF
- compensation refresh
- apply automation
The worker package lives under workers/automation. Each bounded context owns its aggregate, repository (in infrastructure/<context>/), and ports (in domain/ports/). The CLI is the human-facing driving adapter; the JSON-RPC server (jobctrl rpc) is the API-facing driving adapter.
New LlmAdapter instances resolve a model without changing the established ready-provider order (Claude, Codex, Google): explicit non-default workflow model, then preferred_models[selected_ready_provider] from config.json, then the selected provider's SDK default. A saved ID cannot select another provider. Each get_llm_adapter() acquisition compares that effective (provider, model) selection under the singleton lock, reusing the process-stable selected provider instead of repeating SDK/readiness probes. A changed saved preference can replace the singleton for newly started work; provider auth and readiness changes require restart/reset. Previously returned adapters remain untouched for in-flight work. Employer-analysis wiring passes its leg-backed provider separately from model selection, so explicit and saved-model precedence is preserved while the actual synthesizer provider always has a draft leg.
Broad-Board Provider Boundary
JobStreaming 0.0.5 is the provider boundary for Indeed, LinkedIn, Glassdoor, and ZipRecruiter. The infrastructure gateway translates JobCtrl-owned immutable search specifications into provider requests and projects typed provider events back into the existing discovery ingestion shape. Provider types do not enter the Discovery domain model.
The responsibility split is deliberate:
- JobCtrl owns the exact Discover execution, one immutable query/location/board unit per provider stream, title/location admission, durable accepted counts, run-wide limits, lifecycle state, Temporal-attempt fencing, checkpoint storage, selective detail decisions for content identity, and product-visible recovery.
- JobStreaming owns the board adapter and transport, cursor/resume state, stable job keys, typed error classification, cancellation-aware waits, and explicit event acknowledgement. Its optional targeted-detail operation fetches one already-known listing without changing search checkpoints.
The consumer stores an accepted posting and its idempotent unit receipt before acknowledging the event. A newer activity attempt increments the unit lease epoch and fences the older process. Hard interruption therefore leaves a running unit available for Temporal reclaim; cooperative cancellation marks the execution's unfinished units canceled and terminal.
Crawl Politeness Gateway (R10)
Every outbound discovery/enrichment fetch — the urllib client (infrastructure/network/http_client.py), the JobStreaming invocation boundary, and every Playwright navigation — routes through one process-shared choke point in infrastructure/network/:
PolitenessGateway.checkis a side-effect-free pre-fetch verdict (browser callers peek beforepage.goto);guardadditionally consumes one run-budget unit and holds the per-host rate/concurrency slot for the fetch's duration.HostRateLimiteris a process-lifetime singleton (aBoundedSemaphore+ monotonic min-interval per host) soThreadPoolExecutorfan-out cannot bypass pacing. A serverRetry-Afteris honored but clamped at the sink, so a hostile header cannot freeze a pooled worker.resolve_request_headers()snapshots SQLite'scrawl_user_agent.use_browser_headersfor each gateway. Missing/false usesresolve_honest_user_agent()to retain the saved product/version/contact UA. True usesua-generator's latest bundled desktop Chrome identity and matching client hints, generated once per gateway. Controlled HTTP and anonymous Playwright callers carry this set; caller-specific Accept/Content-Type and encoding headers remain unchanged. Connected Chrome retains its own headers; JobStreaming's anonymous integration accepts only the selected UA.- Robots-deny / rate-limit / budget-exhaustion are recorded as first-class outcomes in
operational_attempt_metrics(is_scrape_failure=0,is_operational_failure=0) viarecord_politeness_outcome, and projected to the source-quality read model the discovery UI reads (SourceHealthCard/SourcePolitenessBadges). Documented (undocumented-API) boards fetched by JobStreaming are policed only at the invocation boundary (budget + pacing), since that library owns its internal transport.
The plan and phase-by-phase surface inventory live in docs/plans/implemented/2026-07-05-crawl-politeness-plan.md; the ADR is in docs/decisions.md.
Apply Launcher Ownership
In the installed macOS runtime, the activity invokes the fixed native __apply-execution command before executable readiness or other external preflight. It creates a fresh existing launchd execution scope with durable purpose: "apply"; a mere environment marker cannot authorize the worker. The fixed jobctrl.apply.execution module runs the launcher there. Normal completion, error, cancellation and controller loss drain only that scope, including detached descendants, before publishing exact native completion. After joining the helper, the controller can retire its exact same-boot receipt using that durable completion; a missing or mismatched proof cannot substitute for a zero census. The parent consumes counts only after controller exit and proof verification. A known inspection permission denial is carried as a bounded error category and reported only against the exact retained Apply receipt. It is diagnostic only and never proves exit or authorizes cleanup; other failures stay unclassified. Ordinary same-job retries therefore need no whole-app stop. Other scopes remain active. This uses the existing native execution machinery and shared activity capacity, not another scheduler or containment service.
Apply launchers admit distinct jobs independently. Each invocation owns its cancellation event, cleanup monitor, dashboard resource set and fresh resource identities. Every subsequent attempt receives another exclusive work/profile namespace. Chrome's private endpoint file binds its OS-assigned port and browser WebSocket identity, and both MCP transports receive that exact socket; a display worker number never selects an earlier run's resources. Startup does not recover all running jobs.
Private apply-runs/<launcherId>/custody.json manifests bind the physical workspace, canonical job attempts and authenticated native execution receipt. A process birth intent is fsynced before Popen. A crash before or after spawn publication leaves unknown custody; there is no missing-receipt waiver or inherited-descriptor/bootstrap protocol. Synchronous no-created-child spawn failure can abort only that birth intent. Ordinary exact-handle/session cleanup closes in-memory resources but does not certify full-tree retirement: descendants may have escaped the recorded session. Different-job peers continue independently.
Recovery of the selected old installed launcher requires its exact native execution's completion attestation at instances/<instanceId>/completed-executions/<scopeId>.json. Native scope stop persists the full-receipt/digest-bound proof before removing the original execution receipt. It survives restart and covers escaped descendants. Missing, changed or linked proofs preserve native custody.
Source-checkout Apply uses best-effort descendant cleanup with no Go/helper requirement. It joins workers, attempts and waits for its exact tracked children, and allows an ordinarily finished job to retry after an accepted canonical terminal outcome. It records full_tree_verified: false; detached-descendant completion is not proven. Each source owner holds a private per-run lifetime lease through its blocking thread and cleanup. A normal source retry or an ended-owner crash recovery can settle only that exact logical claim; it never claims escaped-tree verification. An active/abandoned thread keeps its lease, known cleanup failures are durable, and recorded submit intent stays protected even after a generic stage reset. Lifetime files are never created, unlinked or replaced as recovery proof. Fresh namespaces and RunId writer fences preserve successors and accepted data.
After launcher workers have joined and tracked cleanup succeeds, source claims can publish best-effort ApplyClaimSettled only when the exact accepted event matches the canonical terminal stage. Installed claims with born resources wait for the native completion; a no-birth logical claim can settle directly. A saga event whose canonical result write rolled back does not settle ownership. Submit-intent ambiguity remains parked. Installed born-process custody uses the actual native completion instead.
ApplyClaimAcquired establishes the canonical attempt owner separately from replayable saga start events. Automatic results, repository events, determinations, artifacts, lock-release notices, integration publications and final submit checks compare the exact tenant/job/RunId under the SQLite writer. Late old callbacks cannot overwrite a successor. Same-job admission checks only that job's retained owner and preserves submit-intent ambiguity. Each worker proves its prior scoped cleanup before its next claim; failure preserves its files and stops further claims while peers settle already-owned work. See Apply launcher admission.
Workflow Orchestration (Local Temporal)
A local Temporal dev server (temporal server start-dev --db-filename "$JOBCTRL_TEMPORAL_DB") is the workflow engine for the Python worker. The dev launcher uses $JOBCTRL_DIR/temporal.db for a new workspace, matching the installed launcher. A sole existing source history under $JOBCTRL_DIR/temporal/temporal.db remains in place; conflicting histories or orphaned companions refuse startup. The updated installed launcher also uses the sole legacy source path after source ownership has been verified. The JobCtrl event/projection database and Temporal history are one runtime identity, so both remain stable when the same local app is relaunched from another source worktree. An isolated development stack uses a distinct JOBCTRL_DIR, rather than combining the shared jobctrl.db with a worktree-local history store. The infrastructure split lives under workers/automation/src/jobctrl/infrastructure/temporal/:
client.py—get_temporal_client()connects toTEMPORAL_ADDRESS(defaultlocalhost:7233) andTEMPORAL_NAMESPACE(defaultdefault).worker.py—build_worker(client, *, workflows, activities)returns atemporalio.worker.Workerbound toJOBCTRL_TASK_QUEUE. The worker uses aSandboxedWorkflowRunnerwithwith_passthrough_modules("jobctrl")so workflow code can construct activity-input dataclasses at the workflow boundary (the sandbox proxy mechanism otherwise refuses to instantiate frozen dataclasses imported throughimports_passed_through()). Activity Server-dispatched activity execution is bounded by theworker_activity_slotsvalue saved inconfig.json(default4) and two worker-ownedThreadPoolExecutor(max_workers = slots + 2)pools: one for Temporal's synchronous activities and one for blocking stage helpers. Cancellation retires the affected blocking-pool generation before its grace wait, so an immediate retry uses fresh bounded capacity without affecting Temporal's synchronous pool. The worker heartbeat records the activity slots and configured executor width, exact server-dispatched slot use, bounded active detail, completed-activity duration summaries, and an approximate task-queue observation forGET /v1/healthandGET /v1/pipeline/operations(see Concurrency & Fan-out).run_in_activity.py— shared helper for running synchronous domain work from async Temporal activities while heartbeating. Cancellation sets a cooperativethreading.Event, retires that blocking executor generation before waiting up to the activity's cancel deadline, and records anabandoned_threadoperational metric if the old thread ignores cancellation.task_queues.py— singleJOBCTRL_TASK_QUEUE = "jobctrl-default".registry.py— single source of truth forWORKFLOWSandACTIVITIES. The CLI imports both lists and passes them tobuild_worker; new workflows / activities are added by appending here.
The user-facing discover stage starts DiscoverWorkflow directly. That workflow plans source families, runs one source-family activity per planned family, drains detail enrichment in one activity, then fans out per-job JobPreparationWorkflow runs as independent root workflows (batches of 25, USE_EXISTING) — deliberately not children, so finishing discovery cannot terminate in-flight preparation. Other internal preparation stages (enrich, score, tailor, cover) still ship as Temporal Activities under the owning bounded context's package — e.g. jobctrl/scoring/activities.py, jobctrl/materials/activities.py. Activities are thin adapters: they defer heavy imports inside the activity body and forward to the relevant domain function. The product-facing stage order is narrower: discover -> apply.
DiscoverWorkflow.run() captures the immutable (tenant, workflow ID, Temporal run ID) execution identity before planning and carries it through source, reconciliation, preparation, and PDF work. The discovery_execution_jobs table owns the run's observed_this_run and existing_backlog memberships and explicit work-plan decisions. The four PipelineStep* events project attempt-aware orchestration lifecycle for source planning/families, reconciliation/fan-out, backlog sweep, and PDF. Per-job job_stage_states remains authoritative for enrich/score/tailor/cover.
The activity telemetry interceptor never reads activity arguments. It counts every active slot, but retains details only for allowlisted activity kinds, including saved-posting availability checks that can occupy shared capacity after a Discover run closes. Unsafe identifiers become non-reversible local opaque hashes and only validated safe workflow/run references remain readable. URLs, job/profile content, prompts, provider output, artifact paths, payloads, credentials, and exception text are excluded. Telemetry failure is caught so it cannot change business activity execution.
Pipeline activities translate Python exceptions into typed Temporal ApplicationErrors via domain/errors.py. Retry policies use the type value as the durable error code:
| Error type | Code | Retryable |
|---|---|---|
ConfigurationError | configuration | no |
AuthenticationError | authentication | no |
MissingInputError | missing_input | no |
TransientNetworkError | transient_network | yes |
BrowserTransientError | browser_transient | yes |
LlmTransientError | llm_transient | yes |
MaterialRejectedError | material_rejected | no |
SourceUnavailableError | source_unavailable | yes |
BudgetExceededError | budget_exceeded | no |
Workflow and activity retry policies are stage-specific:
| Unit | Attempts | Initial interval | Maximum interval | Non-retryable codes |
|---|---|---|---|---|
DiscoverWorkflow source-family activities | 3 | 5s | 60s | configuration, authentication, missing_input, budget_exceeded |
DiscoverWorkflow enrichment activity | 3 | 5s | 60s | configuration, authentication, missing_input, budget_exceeded |
enrich | 3 | 5s | 60s | configuration, authentication, missing_input, budget_exceeded |
score | 3 | 5s | 60s | configuration, authentication, missing_input, budget_exceeded |
tailor | 3 | 10s | 120s | configuration, authentication, missing_input, budget_exceeded |
cover | 3 | 10s | 120s | configuration, authentication, missing_input, budget_exceeded |
ApplyWorkflow | 1 live / 2 dry-run | 1s | 60s | none set — apply safety comes from the at-most-once claim and submit-intent parking, not error-type filtering |
ManualCaptureImportWorkflow | 2 | 2s | 10s | not_found, capture_replay_mismatch, invalid_capture_input, RuntimeIdentityMismatch |
JobUrlImportWorkflow | 2 | 2s | 10s | invalid_url, RuntimeIdentityMismatch |
ProfileImportWorkflow | 2 | 5s | 60s | configuration, authentication, missing_input, budget_exceeded |
CompensationRefreshWorkflow | 2 | 5s | 60s | configuration, authentication, missing_input, budget_exceeded |
JobPreparationWorkflow reuses the score, tailor, and cover policies above for its per-job steps; its pdf step uses the cover policy (3 attempts, 10s → 120s). The check_spend_budget preflight activity runs with a single attempt so a budget stop is immediate.
The runner still records StageStarted, StageCompleted, StageFailed, operational metrics, and OTel spans through _run_stage_observed; the change is that whole-stage failures propagate into Temporal instead of being converted to normal {"status": "error: ..."} results. Per-item failures inside a batch remain per-item facts when the owning context already records them that way.
Production workflows live alongside the activities:
JobPipelineWorkflow(jobctrl/pipeline/workflow.py) — drives the configured stage list serially in batch mode against eligible jobs in the local DB. Stage eligibility is owned by the underlying runner viastate.set_stage_state, not by the workflow. Passing"discover"delegates to childDiscoverWorkflow; passing"apply"delegates to childApplyWorkflow.DiscoverWorkflow(jobctrl/discovery/workflow.py) — deterministic tenant workflow with iddiscover-{tenantId}. It plans JobStreaming-backed broad-board, canonical ATS, Workday, and Smart Extract source-family activities, preserves the legacy source ordering for limit/budget semantics, emits real activity heartbeats fromDiscoveryRunProgress, then runs discovery enrichment and starts preparation root workflows in batches of 25. Source-family failures are attributed to concrete source ids for source-quality quarantine and fail the workflow after the remaining planned source families complete. Every job-source acquisition owned by this workflow—including JobStreaming provider sessions, ATS/Workday requests, Smart Extract renders and detail pages— prefers the paired, connected extension and otherwise selects guarded HTTP or anonymous Playwright before fetching. The execution reference, source checkpoints, leases, fences, and cohorts remain intact on both paths. Every extension task retains broker authorization; anonymous fallback cannot adopt a system browser or open a copied profile. Anonymous JobStreaming binds Requests sessions at the provider transport hook, including recreated/detail sessions and providers normally using tls-client. Each send/redirect validates public URL/DNS; direct sockets use the shared numeric-address pinning helper. Proxy routing fails closed because the worker cannot pin proxy-side target resolution. Provider request/response semantics remain supported; native TLS fingerprinting is not used on this path.ApplyWorkflow(jobctrl/apply/workflow.py) — single-activity, per-job workflow with live retry capped at one attempt and dry-run retry capped at two attempts.apply_activityre-raises transient failures so the retry policy fires;LookupErroris wrapped in a non-retryableApplicationErrorso operator errors fail fast. Continuous apply runs are bounded to batches of 25 and continue-as-new rather than growing one workflow forever.JobPreparationWorkflow(jobctrl/preparation/workflow.py) — durable per-job workflow that runs the requested subset ofscore,tailor,cover, andpdfin canonical order. Each step is an idempotent activity; already-complete steps returnalready_done, and Temporal resumes at the failed step after a worker interruption.ProfileImportWorkflow(jobctrl/profile/workflow.py) — starts profile PDF import through the same workflow visibility/finalize path as other heavy work, then calls the existing profile-import activity.ManualCaptureImportWorkflow(jobctrl/discovery/manual_capture_workflow.py) — runs the canonical Discovery manual-capture importer on the long-lived worker and returns only persisted queue/provenance fields, allowing exact recovery after a commit-before-ack retry.JobUrlImportWorkflow(jobctrl/discovery/job_url_import_workflow.py) — validates and fetches one user-supplied public posting URL, deterministically ingests readable posting evidence, and otherwise opens the existing Manual Capture path without creating a placeholder Job.CompensationRefreshWorkflow(jobctrl/infrastructure/compensation/workflow.py) — wraps the extracted compensation refresh core so posted facts and market estimates no longer run inside the JSON-RPC request thread.
One non-pipeline workflow is also registered: DurabilityProbeWorkflow (jobctrl/infrastructure/temporal/durability_probe.py) is a diagnostic self-test whose only in-flight state is a durable workflow.sleep timer — no network, no LLM, no browser, and never any apply. It is inert until explicitly started and exists so an operator can prove durable-execution recovery (TR-008 / CL-050) hermetically; scripts/reliability-demo.sh drives it.
The pipeline package (jobctrl/pipeline/) is split into runner.py (stage-core functions and _run_stage_observed) and workflow.py (the Temporal batch orchestrator). The deleted in-process run_pipeline engine is not re-exported; every CLI, API, and local-action entry point starts a workflow.
Preparation cancellation during check_spend_budget records a canceled outcome, including when Temporal wraps cancellation in an ActivityError. No preparation step runs after that canceled preflight; real budget failures retain their failure code. URL-import cancellation likewise preserves its canceled execution and event instead of returning a normal failed-import workflow result.
All workflows that can spend LLM tokens run check_spend_budget with an explicit product lane before their heavy activity. Provider and retry boundaries repeat the same check. Usage is recorded in llm_spend by UTC day and lane. dailyBudgetUsd remains the global ceiling, while laneTokenLimits controls observed input-plus-output tokens for each lane; 0 means unlimited. At or above either applicable threshold the next attempt raises non-retryable budget_exceeded; finalize still records the workflow outcome.
jobctrl worker is the long-lived process that runs the worker loop. At startup it reconciles the local discovery Temporal Schedule: scheduling_enabled=false deletes any existing jobctrl-discovery-local schedule, while scheduling_enabled=true creates or updates a cron schedule with ScheduleOverlapPolicy.SKIP that starts DiscoverWorkflow. The default is off, so fresh installs do not run background discovery. Live workflow state — running workflows, history, signals, retries — is visible at http://127.0.0.1:8233 in the Temporal Web UI.
Loop Closure — Visibility, Finalize, Reconciler
Workflow execution is made durable and visible in the read-model without a TypeScript Temporal SDK and without trigger-coupled reapers:
Workflow*event family (6 types) —WorkflowStarted,WorkflowCompleted,WorkflowFailed,WorkflowCanceled,WorkflowTimedOut,WorkflowTerminated— mirrored 1:1 across the Python and TS event registries and the web invalidation router. Each carriesworkflowId,workflowType, an input summary, and a terminal status within the 12-stateWORKFLOW_RUN_STATUSEScontract.PipelineStep*event family (4 types) —PipelineStepQueued,PipelineStepStarted,PipelineStepCompleted, andPipelineStepFailedidentify one bounded orchestration step by exact Discover workflow/run pair, step kind, item key, and attempt. They fold intopipeline_step_projectionsunder the same operations watermark.- Finalize activities (
infrastructure/temporal/finalize.py) — each workflow emits aWorkflowStartedmarker at the top ofrunand records exactly one terminal event on exit (WorkflowCompletedon success,WorkflowFailedon a stage/exception failure,WorkflowCanceledon cooperative cancellation) viarecord_workflow_started/record_workflow_outcome. Those activities reuserecord_job_event+ a projection refresh; workflow bodies stay deterministic (all SQLite/clock IO is inside the activities). New histories execute these short writes as local activities in two bounded SDK slots, independently of long-running remote activity capacity. Each attempt retains a 30-second deadline and five-attempt retry limit; the local schedule has a three-minute total bound. A version marker preserves existing remote-activity histories. Local writes do not heartbeat or consume reported remote slots. Events deduplicate on tenant, workflow ID, Temporal run ID and event type, so a repeated committed write cannot duplicate canonical history. - Describe-based reconciler —
_reconcile_workflow_runsruns in the worker heartbeat loop (15s). It pins each lookup to the exact recorded Temporal run ID. CLOSED executions map to their matching terminal event and RUNNING executions stay open. NOT_FOUND creates a provisionalreconciled_not_foundterminal so the UI does not show a run as live while its authority is unavailable, but that row remains in the reconciliation set. If the exact run reappears after reconnecting to the authoritative history store, a marked compensatingWorkflowStartedevent reopens it (or immediately precedes its actual terminal outcome). The false terminal remains in the audit stream. This is what lets akill -9'd or restarted worker heal itself without confusing a later execution that reuses the deterministic workflow ID. - Preparation recovery — startup and heartbeat also reconcile eligible saved enrichment, score, and material backlogs. Bounded, durable reservations start one preparation stage at a time; no fresh discovery is required. The controller preserves retry budgets and stop decisions and never starts Apply. Operations & Events owns dispatch, cooldown, and cancellation semantics.
- Dispatch-time open row — the default starter writes a
WorkflowStartedevent immediately after a workflow start returns from Temporal. The in-workflow start marker remains as a duplicate-safe upsert, but a workflow killed or canceled before its first activity is now visible in/runsand can be terminalized by the reconciler. - Deterministic workflow IDs —
WorkflowStartSpeccarriesid_conflict_policy/id_reuse_policy; the default starter passesUSE_EXISTING+ALLOW_DUPLICATE, so a double-start of a deterministic id returns the running handle instead of duplicating execution.discoverderivesdiscover-{tenantId},applyderives a stableapply-{tenantId}-{jobKey}id for single-job applies, and the pipeline orchestrator keepsrun-{uuid}.
The read side is workflow_run_projections (Python-sole-writer, folded from the Workflow* events under the shared operations_projections watermark, mirrored read-only in apps/api/src/projections.ts) — the unified list source for all workflow types. See docs/local-ts-api.md for the GET /v1/workflow-runs and GET /v1/workflow-runs/:runId read model.
The complementary current-operations read side selects a workflow projection, joins its exact run-scoped membership and pipeline-step rows, then overlays fresh worker/task-queue telemetry at request time. Because heartbeat state is ephemeral and no point-in-time lineage reconstruction is implemented, GET /v1/pipeline/operations is explicitly a current snapshot rather than a historical execution API.
Activity Storage Lifetimes
Broad-board and explicit-URL intake commit completed canonical identity and intake events before optional posted-compensation inference. Model calls run without retaining that intake writer. Fact persistence then reserves the writer, rechecks the source text and any discovery lease, and preserves the previous accepted fact if the source changed during inference. Provider retries therefore cannot hold the database writer for the duration of an intake pay analysis.
Activity cleanup closes thread-owned connections and rolls back unfinished writes before executor threads are reused. The Profile repository cache follows the live connection identity as well as its thread and reset generation; a later activity rebuilds a repository whose previous connection was closed. Keeping a pool thread alive cannot extend a repository's database lifetime.
Saved Posting Availability Runtime
The local worker admits SavedPostingAvailabilityWorkflow at startup and on heartbeat/reconnect, at most once a minute per workspace. This is independent of Discover and includes saved jobs before, during, or after enrichment. Each sweep selects at most 25 due, visible saved jobs, preferring preparation/review and then oldest due. Hidden/deleted jobs, running Apply and terminal application outcomes are excluded. Sleeping or offline runtimes append no observations; Jobs reads calculate overdue from retained timestamps.
enrichment/availability.py owns acquisition and observation without invoking Discovery hygiene or policy deletion. Its durable event-ledger leases serialize checks per job and reserve each actual request host; independent jobs can overlap, with foreground capacity reserved in the hourly acquisition quota; acquisition runs after short writer transactions commit. Each check caps cumulative posting-acquisition time at 120 seconds, including rendered capture and browser cleanup. Independent page-model interpretation pauses only that network clock; its actual monotonic duration is credited back, capped at the original job lease's deadline. After interpretation, cancellation and the owner/URL/deletion fence run before fallback can start. Browser supervision reserves the final three seconds for cancellation and reaping. Its admission and pacing waits in the shared politeness gateway also receive the remaining deadline; other fetch surfaces retain their existing wait policy. Public requests run in an owned spawned transport process with a 20-second total DNS/header/body deadline; timeout kills and reaps it before releasing the host reservation. The aggregate page/fallback deadline never exceeds the original five-minute lease; model work does not renew ownership or authorize a stale result. Preparation uses a separate local availability guard. Score, résumé and cover entry points reuse an exact-URL observation attempted within six hours, including unknown evidence, only while the current candidate remains usable. Otherwise they run the existing fenced acquisition. Cache hits write no observation or retry clocks and never declare unknown evidence active. Publication still rechecks the canonical candidate under the stage writer. Apply retains its independent active success and reviewed-uncertainty freshness gates.
The explicit API/RPC/CLI command uses the same activity and runtime identity guard. Availability uses exact public ATS APIs, guarded HTTP, and guarded anonymous Playwright. It does not select paired-extension capture: the extension's Discovery ownership and resource authorization remain separate. Authenticated/challenge/access-limited pages stay unknown. Ordinary Discovery/Enrich transport selection is unchanged. Anonymous availability renders run in an owned supervised process. Browser resource reads return over IPC to the acquisition owner, which retains durable actual-host reservations, pacing and response lineage. The total acquisition deadline covers launch, page navigation, every capture RPC and cleanup. If the renderer or close stalls, the supervisor terminates the owned driver and detached Chromium process groups and reaps its child before releasing the lease.
Anonymous availability guards the complete browser context before creating a page, so first popup requests and frame resources pass through the same actual host reservations and URL checks. Service workers, WebSockets, WebTransport, WebRTC and dedicated/shared workers are blocked. Service-worker registration is blocked on both the instance and native prototype; an attempted unsupported channel or a failed/redirected required document, data, script, style or frame request keeps the result unknown. Non-read data requests are aborted without sending their writes and keep availability unknown; their error views cannot prove closure. Passive assets may fail without invalidating status; their local admission refusals do not become employer evidence. Guards remain installed until the browser has closed. Ordinary enrichment transport selection is unchanged.
Automatic sweeps read remaining quota before claiming jobs and stop at exhaustion. Their 20-minute work budget leaves margin below the 30-minute activity limit. Activity cancellation signals the current acquisition, cancels/reaps its browser process and prevents admission of later jobs.
Required-Bullet Coaching RPC
profile_required_bullet_suggestions is a synchronous, read-only worker RPC behind the opt-in profile HTTP route. It verifies expected runtime/database, canonical selected sources and saved version, admits profile-lane spend, calls the configured LlmPort once, validates findings and rechecks the saved version. It has no semantic fallback. The API also checks the current version after the call. See the complete API contract.