Evaluation
What has been checked so far, and what has not. The first section was recorded from live Qloo and Ticketmaster runs on Oct 21 2026. Live runs are paused for judging; the sheets on this site are what a live run produced. The product eval after it ran on sample data: a stand-in for Qloo and Ticketmaster with made-up places and numbers.
Live runs (Oct 21 2026)
Two real Ticketmaster bills at The Anthem, Washington DC, an independent room (I.M.P.), run on live Qloo and Ticketmaster calls and recorded for the hosted site, walkup-eight.vercel.app. A fresh run makes about 30 Qloo calls. The hosted home page opens on these two sheets.
A fixed rule from the spec picked the two bills: in one independent room, the first Americana, country or folk bill in the window (here Dec 20 2026 to Feb 28 2027), and the bill in that window whose genre is most distant from it. On The Anthem's Ticketmaster calendar, Rhiannon Giddens on Feb 28 was the only Americana, country or folk bill in that window, so the rule picked her show. FLO's R&B show on Feb 17 was the most distant genre. No bill was skipped.
Same room, two crowds
The two sheets share none of their top 5 areas and none of their suggested partners. The table names the area where each bill's crowd stands out most from the room's usual crowd.
| Bill | Stands out most in | Lift | Partners |
|---|---|---|---|
| Rhiannon GiddensSun Feb 28 2027 | H Street NE | about 2.2x | 16 |
| FLOWed Feb 17 2027 | U Street / Shaw | about 2.8x | 17 |
Lift is against the room's usual crowd, the acts The Anthem usually hosts.
Shows: in one room, the bill changes where the street team goes and which places it calls. Does not show: that either list sells tickets.
Popularity check
For each area, the same Qloo place search without the bill as the signal, which Qloo returns in popularity order, with the same list length as the partners picked there. The count is the partners that are not in that popularity list for their area.
| Bill | Not in the popularity list |
|---|---|
| Rhiannon Giddens | 10 of 16 |
| FLO | 15 of 17 |
| Both | 25 of 33 |
Shows: most partners are places a popularity list of the same areas would not name. Does not show: ticket sales. It compares two lists, not sales, and not whether the partners are better.
With Qloo and without, by name
Each live sheet shows one area's two lists side by side: the places most popular there for any crowd, and Walkup's picks there for this show's fans. The area is the one where the most picks are missing from the popularity list.
| Bill | Area | Of the area's most popular places |
|---|---|---|
| Rhiannon Giddens | H Street NE, DC | 2 of the 7 most popular places made the list |
| FLO | Laurel, MD | none of the 8 most popular places made the list |
| Greensky Bluegrass | Oak Park, IL | 2 of the 8 most popular places made the list |
| The Roots | South Shore, IL | 1 of the 8 most popular places made the list |
A second room: The Salt Shed, Chicago
The same rule and window at an independent room in another city: the indoor Shed (3,600), owned and run by 16" On Center, the Chicago group behind the Empty Bottle and Thalia Hall, and a member of the Chicago Independent Venue League. The first listing in the window is a two-night pass for Greensky Bluegrass, a ticket package rather than a show, so bill A is their Dec 30 show. The Roots on Jan 15, hip-hop, is the most distant genre. The two sheets share none of their top 5 areas and none of their partners (19 and 22): Greensky Bluegrass's crowd stands out in the suburbs (Schaumburg, Highwood, Oak Park), The Roots' on the South Side (Bronzeville, South Shore) and in Midlothian. Neither signal is thin. 29 of the 41 partners are not in the popularity list for their area.
| Bill | Stands out most in | Lift | Not in the popularity list |
|---|---|---|---|
| Greensky BluegrassWed Dec 30 2026 | Schaumburg, IL | about 2.6x | 12 of 19 |
| The RootsFri Jan 15 2027 | Bronzeville, IL | about 2.4x | 17 of 22 |
Shows: the same rule runs in a second city, and a thin signal is said plainly instead of filled in. Does not show: that the result holds in every room, or that either list sells tickets.
A change note
“All ages, no bars” on FLO's sheet. Claude (Haiku 5.5 through OpenRouter) read it as two tool calls, set_all_ages and exclude_partner_categories, and the plan ran again with those settings. An earlier run on Oct 20, with Opus 5.5, made the same two calls.
| Tool calls | 2 |
|---|---|
| Partners removed | 5 |
| New Qloo calls | 0 |
| Answered from cache | 30 |
Shows: a plain note became settings, and the re-run cost no Qloo calls. Does not show: that every note is read correctly.
A partial map
FLO's map is marked partial: only 159 areas came back, so the search widened to 60 km. Some suburban areas still carry national chains when no local place fits; they are listed after local places and labelled.
Shows: the sheet says when its map is thin. Does not show: which areas a fuller map would pick.
Is lift just foot traffic?
A separate check, run on Oct 20 2026 at another room: The Truth, Nashville, with its real Ticketmaster roster as the room's usual crowd. If lift ranked areas in the same order as raw heat (where people are), or two bills got the same top areas, the map would add nothing. Passes.
| Measure | Value | Passes if |
|---|---|---|
| Rank agreement between lift and raw heat, per bill (1 means the same order) | 0.20 and 0.01 | below 0.8 |
| Overlap between the two bills' top tenth of areas (1 means the same areas) | 0.47 | below 0.5 |
Shows: lift is not just foot traffic for that pair of bills, at one room. Does not show: that it holds in other rooms.
API notes we handled
Things the live Qloo responses do that the docs don't spell out, and what the code does with each.
- Explainability shares can sum to less than 1 (0.936 seen in the recorded responses). The sheet uses them only to name the act behind a place, never as a percentage.
- A demographics call can leave out an act Qloo has no age data for. The age card is marked partial and names the act.
- An exact /search hit can be an entry Qloo has no taste data for (Jason Isbell, where “Jason Isbell and The 400 Unit” works), and heatmaps drop such ids quietly. A signal check runs before planning, swaps in the next usable entry and says so. Found on Oct 15 2026.
Shows: response shapes the code expects and labels. Does not show: anything about the picks.
Product eval on sample data
20 of 20 cases pass. Last run Oct 21 2026.
Each case runs one plan end to end and checks the sheet it produces: every card has a status and says why when it is not ok, every receipt is in that run's ledger, the policy rules hold, and the sheet fits one page. It checks that Walkup behaves as described. It says nothing about whether Qloo's picks are better than another list, and nothing about ticket sales.
What the cases check
Every check below passes on sample data unless it says otherwise.
- Each of the four sample sheets renders.
- No claim that cites a source is dropped by the checks.
- Every partner is open on the street-team day or marked call first.
- Every footnote points to a request made in that same run.
- A fresh run stays within its Qloo call budget, and every fallback request shows on its section.
- Every step has a status and a fallback, and a section that isn't ok says why.
- Policy: aggregate tastes only, blocked tags never in copy or stored data, alcohol rules only from Ticketmaster, partners suggested and not affiliated, and no words about how a show is selling.
- Two bills in the same room get different areas and partner lists (at most 5 of the top 8 in common).
- CI re-runs these cases on every push, and on the main branch commits the score file.
Cases
Every case passes.
- Sample sheet: Hollis Creek String Band at The Truth
- Sample sheet: Neon Orchard at The Truth
- Sample sheet: Dorian Pike at Hampton Coliseum
- Sample sheet: Ada Pell at Super Funny Comedy Club: thin signal, each fallback labelled
- Pick needed: pause on an ambiguous name, resume with the operator's pick
- Empty compare falls back to comparing two tag lists
- Thin heatmap widens the radius to 60 km
- Lift follows general popularity: partners are picked near the venue
- All-ages alcohol rule: from Ticketmaster only, enforced on fan-facing copy
- Denied tags and words never reach copy or stored data
- Places marked closed are dropped and counted
- A place with unknown hours is kept and marked call ahead
- Month quota guard: fresh calls stop, presets come from the store or cache
- HTTP 429 then success: retried, logged, same sheet
- Declared partners: catalog places ranked, the rest kept in your order
- Partner diversity caps hold, even on a bar-heavy category set
- Budget split: shares sum to 1 over the numbered areas, labelled a heuristic
- Swap the bill: same room, different crowd (top-8 overlap at most 5 of 8)
- House heatmap fails: fans of the genre stand in
- Narration falls back to the fixed template (no model, model error, off-schema draft)
Run it yourself with npx tsx evals/product/run.ts. The full report, check by check, is in evals/product/out/score.md.
What nothing here shows
- That a sheet sells tickets. No test here measures sales.
- Where buyers live. The map shows where this bill's fans stand out from the room's usual crowd.
- That a partner will take part. Partners are suggested, not affiliated.
- How live Qloo data behaves beyond the live sheets and the roster check above and the Fan-Bar Test below. The sample data was written from the documentation, not from real responses.
What we tried and dropped
- Public tour grosses as an outcome: nearly every reported date is close to sold out, so they cannot score a tool for filling seats.
- Ticketmaster "sold out" as a live outcome: the public status field has no sold-out value and no inventory, and a public board of slow shows would hurt artists.
- Minting promo codes: the box office creates them. We export suggested codes.
- Emailing partners: places carry phone numbers and websites, not email. We never scrape addresses.
- Radio format from demographics: a proxy inference, so the promoter picks the format.
See also the system card.
The Fan-Bar Test: no better than popularity
Run Oct 21 2026 on live Qloo calls; plan sealed Oct 13 before any data.
Football supporters' clubs list their official bar in each US city. We gave Qloo the club as the taste signal (Arsenal, Liverpool, the US men's team) and asked for the top 50 bars and pubs within 25 km, then asked again with no signal, which Qloo returns in popularity order. The test, its numbers and its pass mark were written down and hashed before any call.
- In 325 club-and-city pairs across 225 US cities, the club's signal put its official bar in the top 50 for 25% of pairs (95% CI 21% to 30%); popularity alone did for 27% (22% to 32%). Mean reciprocal rank 0.024 against 0.030: a difference of −0.006 (95% CI −0.018 to 0.006), one-sided p = 0.89. The test fails.
- In cities where two clubs both list a bar, the club's own bar ranked above its rival's in 95 of 182 trials: 52% (95% CI 45% to 60%), no better than a coin flip.
- Not run: the arm with a language model searching Google Places (the project's Google billing is closed) and the model-only arm. Closed bars could not be checked. These changes were logged and pushed before the first call.
What it means: a football club's taste signal did not find that club's bar better than popularity. Walkup's signal is a show's acts, not a club, but this test gives no evidence that its picks beat popularity, so every partner table says “Ranked by Qloo affinity; our test did not show this beats popularity.” Full results, coverage and deviations.