# BidBrain project status, part 1: Phase 2 decisions and the first readers (June 2026)

Moved unchanged from CLAUDE.md on 2026-09-18. The standing rules stay in CLAUDE.md, which lists every part of this log.

**Version markers, added by Mark 2026-08-21, extended the same day.** Everything in this log from here down to the "v2.1" marker below was built up to about 3am on 2026-08-20, call that v2. Everything from the v2.1 marker down to the "v2.2" marker is v2.1, 2026-08-20 evening through the v2.1 GitHub release. Everything from the v2.2 marker onward is v2.2. See CHANGELOG.md for the short version of each, and the repo's GitHub Releases page for the tags themselves.

- Current phase: Phase 2 in progress. Both platform readers work live: Motorway and Carwow stock is read off the logged in screens, auction only, and fed through the brain. Still to do in Phase 2: Glass's and Cazana valuation lookups so cars get a real recommended bid, wiring the readers into the brain and the cockpit page as one daily run, the detail page read for owners and exact engine where a banned petrol variant exists, and the private web address via a Cloudflare quick tunnel.
- Phase 2 decisions:
  - Browser automation: Playwright with its own saved Chrome profile, so logins stay alive for the unattended daily run. Installed into the project.
  - Private web address: Cloudflare quick tunnel to start. Unguessable address, no account or domain needed, link changes on restart. Swap to a stable named tunnel later.
  - Glass's and Cazana: Steven has confirmed both are content with automated reads through his own login. The lookups will be built.
  - Readers are built against the live logged in pages, never against guessed HTML. Site by site, simplest first.
  - Motorway: logged in, saved profile working. Dealer stock list is https://pro.motorway.co.uk/vehicles. The reader is bidbrain/readers/motorway.py, proven against a real captured page: it reads reg, make, model, derivative, year, mileage, fuel, distance, grade, reserve, photo and listing link from each card. CAP Clean, owners, service history, VAT and exact engine are not on the list card and need each car's own page, still to build. Tested live on 35 real cars, gate culled 12, 23 passed the list level checks.
  - Carwow: dealer stock list is https://dealers.carwow.co.uk (it redirects to a marketing page until logged in). The reader is bidbrain/readers/carwow.py, proven against a real captured page: it reads reg, make, model, derivative, year, mileage, fuel, service history, reserve, CAP value, grade, distance, photo and listing link from each card. Each card carries data-listing-state, and the reader keeps only auction states, enforcing the auction only rule. Tested on 30 real auction cars, all fields read.
  - Carwow login persists in the saved profile, confirmed by a fresh browser reading the live auction stock end to end. The real stock URL is https://dealers.carwow.co.uk/dealers/listings/filtered/stock and the login form is https://dealers.carwow.co.uk/dealers/login. Gotcha: the site root https://dealers.carwow.co.uk/ always redirects to a marketing page even when logged in, so never test login state against the root, use the stock URL.
  - login.py now opens straight to each site's login form, writes data/inspect/<site>_status.txt and a live screenshot <site>_live.png each second so the session can see the window, auto captures when it reaches the logged in dealer area (host correct and not the login path), snapshots all tabs, and saves a session state file via storage_state as a safety net. It also auto clicks a Log in link if a dealer URL bounces to marketing.
  - Both readers run live and headless via browser.open_reader_context, which uses a saved session state file if present, else the persistent profile. Motorway and Carwow both proven reading live.
  - Glass's: login at https://uk.glass.co.uk/auth/login, ticking "Don't ask for 30 days" keeps the session alive for the daily run. Login persists, saved via storage_state to data/glass_state.json. Lookup is by registration plus mileage. On a valuation result page (uk.glass.co.uk/valuations/<uuid>) the value the brain wants is GLASS'S RETAIL, which reads cleanly from the element id avBasicRetailPrice, attribute data-price (a plain integer, for example 9085). The page also shows Glass's Trade and a separate Live Retail Price, do not confuse those with Glass's Retail. The /valuations page is the history of past valuations with a VRM filter column and a Bulk valuation option that may suit valuing the whole daily shortlist at once. Bulk valuation is a file upload batch job (Add bulk valuation, upload a file of regs, it processes then you export), table columns USER, CREATE TIME, NAME, STATUS, VEHICLES, PROCESSED, ISSUES, FILE NAME. Not yet confirmed whether the export returns Glass's Retail rather than the live price, would need a small test job in Steven's account, get his ok first. Plan: build single reg at a time first since the value location is known, treat bulk as a later speed up. The whole Cazana side is not started yet.
  - Glass's lookup BUILT and proven live. bidbrain/readers/glass.py: lookup(playwright, reg, mileage) enters the reg in #plateNumberInput and mileage in #mileageInput on uk.glass.co.uk, clicks Go, handles the /identification/valuations chooser (clicks the single matched ag-grid row, and if more than one genuine edition is offered it returns None to hold the car back, never guessing), opens the result, clicks Show more, and reads Glass's Retail from #avBasicRetailPrice data-price. Tested: PN68ZWX returns 9085 as expected, a fresh reg VF17OML returns 9032. Readers use the saved glass_state.json session, which persists. Note: login.py opens the persistent profile which can show logged out for Glass's even though the storage_state session is good, so always run readers via open_reader_context, and if Glass's ever asks to log in again, run python3 login.py glass and log in ticking Don't ask for 30 days.
  - Repeat appearance flag: data layer built and tested. db.py has a sightings table (reg, platform, sale_date, unique per car per platform per day), record_sightings(cars, platform, sale_date) to log each daily read, repeat_summary(reg, sale_date, 30) to count previous distinct sale dates per platform inside the 30 day window plus same day cross platform, and repeat_flag_lines(summary) for the card wording. Verified against Steven's Mini example (3 previous Motorway, 2 previous Carwow, same day both caught, old sightings drop out). Still to wire: call record_sightings in the daily run and show repeat_flag_lines on each card.
  - Cazana lookup BUILT and proven live. Cazana is now branded Percayso, login at https://trade.percayso-vehicle-intelligence.co.uk/. bidbrain/readers/cazana.py: lookup(playwright, reg, mileage) fills input[name=value] with the reg and input[name=mileage], presses Enter, lands on /companion/search/<uuid>, and reads the headline Retail value. The page has no stable id, so it finds the Retail block by the fact it uniquely contains the text Retail franchise, then reads the right hand amount. It deliberately takes the headline Retail, not the franchise, independent or supermarket breakdowns, and not Trade. Tested: PN68ZWX returns 11251, fresh reg VF17OML returns 9688. Login saved to cazana_state.json, persists.
  - End to end pricing proven on a real car: Skoda PN68ZWX, Glass's 9085 and Cazana 11251 live, governing value 10448 (Glass's, over 10k), recommended max bid 7448. Both valuation sources and the brain work on live data.
  - Daily run WIRED and proven on live data. daily_run.py reads Motorway and Carwow live, records every car to sightings for the repeat tracker, applies the list level gate, opens each Motorway survivor's page to read owners and engine via motorway.enrich_from_detail (which parses __NEXT_DATA__, wait_for_selector must use state=attached for the script tag, that was a bug), runs the full brain, looks up Glass's and Cazana for gate passers, prices them, adds repeat flags, and writes cockpit.html. Test run (python3 daily_run.py --test 3) valued the 3 cheapest of 19 survivors: Nissan Qashqai reserve 6043 max bid 7082, Ford Focus reserve 5806 max bid 5681, Peugeot 308 held because Glass's would not read (held not guessed, correct). Full run python3 daily_run.py values all survivors.
  - Fixes after Steven's review of the first daily page:
    - Wrong bid links: each Motorway card is wrapped by its own anchor like <a id=vehicle_card_ID href="/vehicles/ID"> sitting just before the card div, and the old split paired a card with the next card's href. motorway._card_blocks now splits on that wrapping anchor and takes the href from it. Verified reg to link against the live DOM.
    - CAP missing: CAP is not on the Motorway list card, it is in the detail page __NEXT_DATA__ as a price entry with priceSource CAP. enrich_from_detail now reads it. Verified per car (it is normal for two different cars to share a CAP value).
    - Auction only locked into the source URL: motorway stock_url is now .../vehicles?listType=auction.
  - Motorway saved search DONE. Steven has a saved search titled "Normal search" that encodes his brief. Applying it gives this URL, now baked into motorway stock_url: /vehicles?ageFrom=2&ageTo=10&displayPriceTo=11000&maxDistance=210&mileageTo=90000&numericGrade=1&numericGrade=2&numericGrade=3&previousKeepersCountTo=4&sellerType=private&listType=auction. So the daily read starts pre filtered to his brief (age, price, distance, mileage, grade 1 to 3, owners up to 4, auction, private sellers). If he edits the saved search, re capture this URL by applying "Normal search" once and copying the address bar. With this, 34 of 36 read cars pass the list gate instead of 21 of 36.
  - Rejected cars are hidden on the daily page. render_page now takes show_rejected, daily_run passes show_rejected=False. Held back cars still show (they pass the rules but could not be priced). Rejected cars are still saved to the database.
  - Known item: the read currently takes the first page of the stock list (about 36 cards). If the saved search returns more than one page, later pages are not yet read. Add pagination if the daily list is being truncated.
  - Limitation seen live: a car with no registration cannot be valued by reg lookup, so it is held. Cars without a plate also carry no repeat flag.
  - Carwow pricing DONE. carwow.enrich_from_detail(page, car) opens each Carwow car's own page and reads Former keepers (owners) and VAT qualifying as label then value lines. daily_run now reads Carwow, records sightings, list gates, enriches candidates, and prices them pooled with Motorway. Proven: a run read 29 Carwow cars, 3 passed the list checks and were enriched, both platforms assessed together.
  - Remaining in Phase 2:
    - Carwow brief filter DONE. Steven has a Carwow saved filter, id 1741. Applying it gives ?saved_filter_id=1741, now baked into the carwow stock_url. With it, 20 of 29 read Carwow cars fit the brief (was 3 unfiltered) and Carwow cars now price and pool with Motorway. If he edits the saved filter, re capture the saved_filter_id by applying it and reading the address bar.
    - Pagination DONE, both platforms.
      - Motorway: the stock page has no page param and does not infinite scroll, but it has a Download CSV which is the whole filtered list in one file. motorway.read_export(playwright) clicks Download, picks "Filtered vehicles", captures the CSV and parses every row. The CSV carries Buying type (Live sale means auction), VRM, Make, Model, Year, Mileage, Number of owners, Service history, Exterior grade, VIN, Fuel, Engine size, Reserve price, CAP clean value, Location and the vehicle link. This replaces card scraping AND the per car detail reads (owners, engine, CAP all in the CSV). Proven reading the full filtered list (about 245 to 286 cars). The CSV gives Location not exact miles, but the saved search enforces the 210 mile limit, so the brain trusts the filter for distance: assess(car, today, assume_distance_ok=True) and gate_failures skip the distance check, set by daily_run for Motorway cars via _assess. Engine for the ban check is built as litres from Engine size plus the Model text (which carries TSI, TFSI, EcoBoost, PureTech, DCI etc.) plus Fuel. The brain still re checks mileage, reserve, age, grade, owners from the CSV and applies the bans. The old read_live and enrich_from_detail remain but are no longer used by the daily run.
      - Carwow: paginates with &page=N. carwow.read_live now loops pages, accumulating auction cars and stopping when a page shows no new listings. Proven against the filtered list (about 136 vehicles across about 5 pages).
    - Carwow listing state varies by time of day. Each card has data-listing-state. The next day auction stock reads as waiting_for_auction (after 5pm and overnight). Between sales the same cars show as second_chance_quotes, which are not the auction, so the reader correctly returns zero auction cars then. This is why the read must run after 5pm. parse_listing keeps only states containing "auction".
    - The Cloudflare quick tunnel for the private web address.
    - Optionally investigate why some Glass's lookups return nothing (likely a multi edition identification case, currently held which is safe).
  - Cockpit redesign, decided by Steven. Card hierarchy, most prominent first: governing value (hero, biggest), then max bid (green, with an over or under reserve pill), then CAP Clean, then platform. Platform is a coloured pill on the photo top left (Motorway blue, Carwow purple). Condition grade is a coloured circle on the photo bottom right, grade 1 green, 2 lighter green, 3 amber, 4 and 5 reds. The "governing value less the 3000 spread" line was removed. Reserve, mileage, year, owners, distance are small chips, and the Glass's and Cazana breakdown is tucked in an expandable Valuation detail. Lives in bidbrain/render.py.
  - daily_run now caches each run to data/last_run.json and supports python3 daily_run.py --render-only to rebuild cockpit.html instantly from the last run, so the design can be tuned without re valuing.
  - Cockpit tweaks after Steven's review: hero label is "Retails for" (was Governing value). Grade 2 circle is yellow with a dark number. Cards reserve two lines for the name (header h2 min-height) and the bid button is pinned to the bottom (margin-top auto) so buttons line up across a row. Real platform logos sit in the top left pill: Motorway black wordmark on yellow, Carwow cyan flower and wordmark on white. Logos are embedded as data URIs read from assets/mw_logo.datauri and assets/cw_logo.datauri by render.py (sources assets/mw_src.png and assets/cw_src.png, cropped with sips). The Carwow logo was captured from Steven's clipboard via osascript as «class PNGf» since pasted images are not files.
  - Diesel rule, decided by Steven: a car that is plainly a diesel with no banned diesel variant passes without needing the exact engine code. Only chase the exact engine on the detail page where the model also has a banned petrol variant. Keeps real diesels in and the uncertainty rule focused where it matters.