Skip to content

Commit 54d95ee

Browse files
authored
feat: AI prompt management dashboard and enhanced span inspectors (#3244)
- Full prompt management UI: list, detail, override, and version management for AI prompts defined with `prompts.define()` - Rich AI span inspectors for all AI SDK operations with token usage, messages, and prompt context - Real-time generation tracking with live polling and filtering ## Prompt management Define prompts in your code with `prompts.define()`, then manage versions and overrides from the dashboard without redeploying: ```typescript import { task, prompts } from "@trigger.dev/sdk"; import { generateText } from "ai"; import { openai } from "@ai-sdk/openai"; import { z } from "zod"; const supportPrompt = prompts.define({ id: "customer-support", model: "gpt-4o", variables: z.object({ customerName: z.string(), plan: z.string(), issue: z.string(), }), content: `You are a support agent for Acme SaaS. Customer: {{customerName}} ({{plan}} plan) Issue: {{issue}} Respond with empathy and precision.`, }); export const supportTask = task({ id: "handle-support", run: async (payload) => { const resolved = await supportPrompt.resolve({ customerName: payload.name, plan: payload.plan, issue: payload.issue, }); const result = await generateText({ model: openai(resolved.model ?? "gpt-4o"), system: resolved.text, prompt: payload.issue, ...resolved.toAISDKTelemetry(), }); return { response: result.text }; }, }); ``` The prompts list page shows each prompt with its current version, model, override status, and a usage sparkline over the last 24 hours. From the prompt detail page you can: - **Create overrides** to change the prompt template or model without redeploying. Overrides take priority over the deployed version when `prompt.resolve()` is called. - **Promote** any code-deployed version to be the current version - **Browse generations** across all versions with infinite scroll and live polling for new results - **Filter** by version, model, operation type, and provider - **View metrics** (total generations, avg tokens, avg cost, latency) broken down by version ## AI span inspectors Every AI SDK operation now gets a custom inspector in the run trace view: - **`ai.generateText` / `ai.streamText`** — Shows model, token usage, cost, the full message thread (system prompt, user message, assistant response), and linked prompt details - **`ai.generateObject` / `ai.streamObject`** — Same as above plus the JSON schema and structured output - **`ai.toolCall`** — Shows tool name, call ID, and input arguments - **`ai.embed`** — Shows model and the text being embedded For generation spans linked to a prompt, a "Prompt" tab shows the prompt metadata, the input variables passed to `resolve()`, and the template content from the prompt version. All AI span inspectors include a compact timestamp and duration header. ## Other improvements - Resizable panel sizes now persist across page refreshes (patched `@window-splitter/state` to fix snapshot restoration) - Run page panels also persist their sizes - Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and chat messages - Added Operations and Providers filters to the AI metrics dashboard ## Screenshots <img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 14 17@2x" src="https://github.com/user-attachments/assets/f3e59989-a2fa-4990-a9d0-3cacda431868" /> <img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15 37@2x" src="https://github.com/user-attachments/assets/2f2d02df-2d2b-44fb-ac6f-9153f6a6c387" /> <img width="3680" height="2392" alt="CleanShot 2026-03-21 at 10 15 54@2x" src="https://github.com/user-attachments/assets/baa161e0-ef91-4fa4-a55f-986b71cccdf0" />
1 parent 35298ac commit 54d95ee

File tree

92 files changed

+8284
-409
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

92 files changed

+8284
-409
lines changed

.changeset/ai-prompt-management.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@trigger.dev/sdk": patch
3+
---
4+
5+
Define and manage AI prompts with `prompts.define()`. Create typesafe prompt templates with variables, resolve them at runtime, and manage versions and overrides from the dashboard without redeploying.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
area: webapp
3+
type: feature
4+
---
5+
6+
AI prompt management dashboard and enhanced span inspectors.
7+
8+
**Prompt management:**
9+
- Prompts list page with version status, model, override indicators, and 24h usage sparklines
10+
- Prompt detail page with template viewer, variable preview, version history timeline, and override editor
11+
- Create, edit, and remove overrides to change prompt content or model without redeploying
12+
- Promote any code-deployed version to current
13+
- Generations tab with infinite scroll, live polling, and inline span inspector
14+
- Per-prompt metrics: total generations, avg tokens, avg cost, latency, with version-level breakdowns
15+
16+
**AI span inspectors:**
17+
- Custom inspectors for `ai.generateText`, `ai.streamText`, `ai.generateObject`, `ai.streamObject` parent spans
18+
- `ai.toolCall` inspector showing tool name, call ID, and input arguments
19+
- `ai.embed` inspector showing model, provider, and input text
20+
- Prompt tab on AI spans linking to prompt version with template and input variables
21+
- Compact timestamp and duration header on all AI span inspectors
22+
23+
**AI metrics dashboard:**
24+
- Operations, Providers, and Prompts filters on the AI Metrics dashboard
25+
- Cost by prompt widget
26+
- "AI" section in the sidebar with Prompts and AI Metrics links
27+
28+
**Other improvements:**
29+
- Resizable panel sizes now persist across page refreshes
30+
- Fixed `<div>` inside `<p>` DOM nesting warnings in span titles and chat messages

CLAUDE.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,25 @@ pnpm run dev --filter webapp # Run webapp (http://localhost:3030)
1818
pnpm run dev --filter trigger.dev --filter "@trigger.dev/*" # Watch CLI and packages
1919
```
2020

21+
### Verifying Changes
22+
23+
The verification command depends on where the change lives:
24+
25+
- **Apps and internal packages** (`apps/*`, `internal-packages/*`): Use `typecheck`. **Never use `build`** for these — building proves almost nothing about correctness.
26+
- **Public packages** (`packages/*`): Use `build`.
27+
28+
```bash
29+
# Apps and internal packages — use typecheck
30+
pnpm run typecheck --filter webapp # ~1-2 minutes
31+
pnpm run typecheck --filter @internal/run-engine
32+
33+
# Public packages — use build
34+
pnpm run build --filter @trigger.dev/sdk
35+
pnpm run build --filter @trigger.dev/core
36+
```
37+
38+
Only run typecheck/build after major changes (new files, significant refactors, schema changes). For small edits, trust the types and let CI catch issues.
39+
2140
## Testing
2241

2342
We use vitest exclusively. **Never mock anything** - use testcontainers instead.

apps/webapp/CLAUDE.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,48 @@
22

33
Remix 2.1.0 app serving as the main API, dashboard, and orchestration engine. Uses an Express server (`server.ts`).
44

5+
## Verifying Changes
6+
7+
**Never run `pnpm run build --filter webapp` to verify changes.** Building proves almost nothing about correctness. The webapp is an app, not a public package — use typecheck from the repo root:
8+
9+
```bash
10+
pnpm run typecheck --filter webapp # ~1-2 minutes
11+
```
12+
13+
Only run typecheck after major changes (new files, significant refactors, schema changes). For small edits, trust the types and let CI catch issues.
14+
15+
Note: Public packages (`packages/*`) use `build` instead. See the root CLAUDE.md for details.
16+
17+
## Testing Dashboard Changes with Chrome DevTools MCP
18+
19+
Use the `chrome-devtools` MCP server to visually verify local dashboard changes. The webapp must be running (`pnpm run dev --filter webapp` from repo root).
20+
21+
### Login
22+
23+
```
24+
1. mcp__chrome-devtools__new_page(url: "http://localhost:3030")
25+
→ Redirects to /login
26+
2. mcp__chrome-devtools__click the "Continue with Email" link
27+
3. mcp__chrome-devtools__fill the email field with "local@trigger.dev"
28+
4. mcp__chrome-devtools__click "Send a magic link"
29+
→ Auto-logs in and redirects to the dashboard (no email verification needed locally)
30+
```
31+
32+
### Navigating and Verifying
33+
34+
- **take_snapshot**: Get an a11y tree of the page (text content, element UIDs for interaction). Prefer this over screenshots for understanding page structure.
35+
- **take_screenshot**: Capture what the page looks like visually. Use to verify styling, layout, and visual changes.
36+
- **navigate_page**: Go to specific URLs, e.g. `http://localhost:3030/orgs/references-bc08/projects/hello-world-SiWs/env/dev/runs`
37+
- **click / fill**: Interact with elements using UIDs from `take_snapshot`.
38+
- **evaluate_script**: Run JS in the browser console for debugging.
39+
- **list_console_messages**: Check for console errors after navigating.
40+
41+
### Tips
42+
43+
- Snapshots can be very large on complex pages (200K+ chars). Use `take_screenshot` first to orient, then `take_snapshot` only when you need element UIDs to interact.
44+
- The local seeded user email is `local@trigger.dev`.
45+
- Dashboard URL pattern: `http://localhost:3030/orgs/{orgSlug}/projects/{projectSlug}/env/{envSlug}/{section}`
46+
547
## Key File Locations
648

749
- **Trigger API**: `app/routes/api.v1.tasks.$taskId.trigger.ts`

apps/webapp/app/components/BlankStatePanels.tsx

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,13 @@ import {
44
BookOpenIcon,
55
ChatBubbleLeftRightIcon,
66
ClockIcon,
7+
DocumentTextIcon,
78
PlusIcon,
89
QuestionMarkCircleIcon,
910
RectangleGroupIcon,
1011
RectangleStackIcon,
1112
ServerStackIcon,
13+
SparklesIcon,
1214
Squares2X2Icon,
1315
} from "@heroicons/react/20/solid";
1416
import { useLocation } from "react-use";
@@ -686,3 +688,55 @@ function DeploymentOnboardingSteps() {
686688
</PackageManagerProvider>
687689
);
688690
}
691+
692+
export function PromptsNone() {
693+
return (
694+
<InfoPanel
695+
title="Define your first prompt"
696+
icon={SparklesIcon}
697+
iconClassName="text-purple-500"
698+
panelClassName="max-w-lg"
699+
accessory={
700+
<LinkButton to={docsPath("prompt-management")} variant="docs/small" LeadingIcon={BookOpenIcon}>
701+
Prompt docs
702+
</LinkButton>
703+
}
704+
>
705+
<Paragraph spacing variant="small">
706+
Managed prompts let you define AI prompts in code with typesafe variables, then edit and
707+
version them from the dashboard without redeploying.
708+
</Paragraph>
709+
<Paragraph spacing variant="small">
710+
Add a prompt to your project using <InlineCode variant="small">prompts.define()</InlineCode>:
711+
</Paragraph>
712+
<div className="rounded border border-grid-dimmed bg-charcoal-900 p-3">
713+
<pre className="text-xs leading-relaxed text-text-dimmed">
714+
<span className="text-purple-400">import</span>
715+
{" { prompts } "}
716+
<span className="text-purple-400">from</span>
717+
{' "@trigger.dev/sdk";\n'}
718+
<span className="text-purple-400">import</span>
719+
{" { z } "}
720+
<span className="text-purple-400">from</span>
721+
{' "zod";\n\n'}
722+
<span className="text-purple-400">export const</span>
723+
{" myPrompt = "}
724+
<span className="text-blue-400">prompts.define</span>
725+
{"({\n"}
726+
{" id: "}
727+
<span className="text-green-400">"my-prompt"</span>
728+
{",\n"}
729+
{" variables: z.object({\n"}
730+
{" name: z.string(),\n"}
731+
{" }),\n"}
732+
{" content: "}
733+
<span className="text-green-400">{"`Hello {{name}}!`"}</span>
734+
{",\n"});</pre>
735+
</div>
736+
<Paragraph variant="small" className="mt-2">
737+
Deploy your project and your prompts will appear here with version history and a live
738+
editor.
739+
</Paragraph>
740+
</InfoPanel>
741+
);
742+
}

apps/webapp/app/components/code/TSQLResultsTable.tsx

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -460,10 +460,12 @@ function CellValueWrapper({
460460
value,
461461
column,
462462
prettyFormatting,
463+
row,
463464
}: {
464465
value: unknown;
465466
column: OutputColumnMetadata;
466467
prettyFormatting: boolean;
468+
row?: Record<string, unknown>;
467469
}) {
468470
const [hovered, setHovered] = useState(false);
469471

@@ -478,6 +480,7 @@ function CellValueWrapper({
478480
column={column}
479481
prettyFormatting={prettyFormatting}
480482
hovered={hovered}
483+
row={row}
481484
/>
482485
</span>
483486
);
@@ -491,11 +494,13 @@ function CellValue({
491494
column,
492495
prettyFormatting = true,
493496
hovered = false,
497+
row,
494498
}: {
495499
value: unknown;
496500
column: OutputColumnMetadata;
497501
prettyFormatting?: boolean;
498502
hovered?: boolean;
503+
row?: Record<string, unknown>;
499504
}) {
500505
// Plain text mode - render everything as monospace text with truncation
501506
if (!prettyFormatting) {
@@ -562,12 +567,20 @@ function CellValue({
562567
switch (column.customRenderType) {
563568
case "runId": {
564569
if (typeof value === "string") {
570+
const spanId = row?.["span_id"];
571+
const runPath = v3RunPathFromFriendlyId(value);
572+
const href = typeof spanId === "string" && spanId
573+
? `${runPath}?span=${spanId}`
574+
: runPath;
575+
const tooltip = typeof spanId === "string" && spanId
576+
? "Jump to span"
577+
: "Jump to run";
565578
return (
566579
<SimpleTooltip
567-
content="Jump to run"
580+
content={tooltip}
568581
disableHoverableContent
569582
hidden={!hovered}
570-
button={<TextLink to={v3RunPathFromFriendlyId(value)}>{value}</TextLink>}
583+
button={<TextLink to={href}>{value}</TextLink>}
571584
/>
572585
);
573586
}
@@ -1010,13 +1023,16 @@ export const TSQLResultsTable = memo(function TSQLResultsTable({
10101023
prettyFormatting = true,
10111024
sorting: defaultSorting = [],
10121025
showHeaderOnEmpty = false,
1026+
hiddenColumns,
10131027
}: {
10141028
rows: Record<string, unknown>[];
10151029
columns: OutputColumnMetadata[];
10161030
prettyFormatting?: boolean;
10171031
sorting?: SortingState;
10181032
/** When true, show column headers + "No results" on empty data. When false, show a blank state icon. */
10191033
showHeaderOnEmpty?: boolean;
1034+
/** Column names to hide from display but keep in row data (useful for linking) */
1035+
hiddenColumns?: string[];
10201036
}) {
10211037
const tableContainerRef = useRef<HTMLDivElement>(null);
10221038

@@ -1030,9 +1046,13 @@ export const TSQLResultsTable = memo(function TSQLResultsTable({
10301046

10311047
// Create TanStack Table column definitions from OutputColumnMetadata
10321048
// Calculate column widths based on content
1049+
const visibleColumns = useMemo(
1050+
() => hiddenColumns?.length ? columns.filter((col) => !hiddenColumns.includes(col.name)) : columns,
1051+
[columns, hiddenColumns]
1052+
);
10331053
const columnDefs = useMemo<ColumnDef<RowData, unknown>[]>(
10341054
() =>
1035-
columns.map((col) => ({
1055+
visibleColumns.map((col) => ({
10361056
id: col.name,
10371057
accessorKey: col.name,
10381058
header: () => col.name,
@@ -1041,6 +1061,7 @@ export const TSQLResultsTable = memo(function TSQLResultsTable({
10411061
value={info.getValue()}
10421062
column={col}
10431063
prettyFormatting={prettyFormatting}
1064+
row={info.row.original}
10441065
/>
10451066
),
10461067
meta: {
@@ -1050,7 +1071,7 @@ export const TSQLResultsTable = memo(function TSQLResultsTable({
10501071
size: calculateColumnWidth(col.name, rows, col),
10511072
filterFn: fuzzyFilter,
10521073
})),
1053-
[columns, rows, prettyFormatting]
1074+
[visibleColumns, rows, prettyFormatting]
10541075
);
10551076

10561077
// Initialize TanStack Table

0 commit comments

Comments
 (0)