{"id":"phoenix-observability","title":"Phoenix: Making the Research Loop Observable","subtitle":"Why we wired Phoenix, the Cloudflare edge limit we proved, and how to get more value from it.","category":"changelog","tags":["phoenix","observability","evals","featured"],"source":"articles/docs/phoenix-observability.md","lang":"en","words":579,"readMinutes":3,"toc":[{"depth":2,"text":"Why","id":"why"},{"depth":2,"text":"What we did","id":"what-we-did"},{"depth":2,"text":"What we proved (the hard part)","id":"what-we-proved-the-hard-part"},{"depth":2,"text":"Results","id":"results"},{"depth":2,"text":"Suggested next steps — getting more value from this tool","id":"suggested-next-steps-getting-more-value-from-this-tool"}],"html":"

Phoenix: Making the Research Loop Observable

Featured. This is the story of wiring Arize Phoenix\ninto the Lupine research loop — why we did it, what broke, what we proved, and how to\nget more value out of it next. It is the first instrument that lets us ask whether the\nloop is actually getting better, instead of assuming it is.

\n

Why

The Lupine research loop runs hypotheses through an LLM-driven pipeline: harvest →\ncomprehend → conjecture → evaluate → close. For months it ran blind. We could see\nthat it produced output; we could not see whether the output was improving, which\nmodel actually served a request, or where latency and cost went. A self-improving\nsystem you cannot measure is just a system that changes.

\n

Observability is the precondition for the entire north-star plan: the hypothesis\nlifecycle is the unit of optimization, Phoenix evals are the fitness function, and the\nEvolver is the actuator. Without the fitness function wired in, the Evolver has nothing\nto climb.

\n

What we did

    \n
  1. Diagnosed the two disconnected halves. Phoenix configuration was split: GitHub\nrepo secrets fed the hourly evaluation workflow, while wrangler secrets fed the\nCloudflare Worker's trace exporter. Neither was set by deploy automation, so\n0 / 300 spans reached Phoenix Cloud. We set both halves.
    2. \n
  2. Consolidated to one LLM path. glim-think had two LLM code paths; the gateway\none was dead (it produced none of the 300 spans). We deleted it and kept a single\nAI-SDK-native path so telemetry reflects reality.
    3. \n
  3. Wired OpenInference telemetry, golden datasets and a functional eval runner via\nthe Phoenix REST API, a per-model performance scorecard reading live-path\nattribution (ai.telemetry.functionId), and finally closed the eval→routing\nloop so model selection is driven by measured performance.
    4. \n
\n

What we proved (the hard part)

The Worker still showed zero spans even with valid secrets. We proved a hard\ninfrastructure limit: a Cloudflare Worker cannot export OTLP directly to Phoenix\nCloud. The Cloudflare edge black-holes the Worker→external subrequest and returns a\nfabricated 200 — curl from outside gets the real server: uvicorn, the Worker\ngets a fake server: cloudflare. The Phoenix key was valid the entire time. The fix is\nnot a config tweak; it requires a GCP egress relay (mirroring deploy-otlp-relay.yml).\nNaming the limit precisely is itself a result — it stopped weeks of key-rotation\nguesswork.

\n

Results

\n

Suggested next steps — getting more value from this tool

Phoenix is wired; the leverage now is in using it:

\n
    \n
  1. Stand up the GCP OTLP relay so Worker spans actually land — until then the\nrichest traces (the live research loop) are still dark.
    2. \n
  2. Make the hypothesis lifecycle a first-class trace. One trace per hypothesis,\nspanning harvest→close, so a reader can replay why a conjecture moved. This is the\nsubstrate the Evolver optimizes against.
    3. \n
  3. Promote scientific-throughput evals (did a round produce a literature-anchored,\nnon-trivial, falsifiable claim?) to the loop's fitness function — not just latency\nand cost.
    4. \n
  4. Feed the scorecard into the Evolver's model-selection actuation, keeping\nstructural change PR-gated and prompt/rubric/criteria change autonomous.
    5. \n
  5. Regression-gate every Evolver change against the golden datasets so the loop can\nonly ratchet forward.
    6. \n
\n

The goal is not dashboards. It is a measurable fitness function on the hypothesis\nlifecycle, so "the loop is improving" becomes a claim we can check instead of hope.

\n"}