Independent · Agentic data pipeline · 2026
The catalog nobody
would sell me.
Fiddle is only as good as the parts data behind it, and the aftermarket has no clean API you can go and buy. So Orb-Weaver is the pipeline I built to go and get it. It tracks 471 brands, has crawled 310 of them, and holds 245,471 parts, with every brand it couldn't reach recorded against a cause rather than a shrug. Scraping is the boring bit. Which leaves the actual question, and the one I spent the time on: what is this thing allowed to conclude without me.
01 · The problem
You can rent almost anything except this
Fiddle answers one question over and over: what part is this, does it fit my car, and who else is running it. Every one of those answers needs a catalog underneath it, with part numbers, prices, fitment and photos. I assumed I'd pay somebody for that. The aftermarket doesn't work that way. The tidy commercial feeds cover OEM replacement parts, the aftermarket ones cover a handful of the largest distributors, and none of them carry the Japanese and boutique brands that people build cars out of.
So the data had to be harvested, brand by brand, off four hundred and some websites that share no format, no platform and no opinion about being read by software. That's a scraping problem for about a day. After that it becomes a bookkeeping problem, and then a judgement problem, which is the part I found genuinely interesting.
The short version
- Fiddle needed a catalog. Nobody sells one, so the crawler became a project of its own.
- Four ways in, cheapest first: Shopify feed, sitemap and JSON-LD, a platform harvester, and only then a hand-written template.
- 310 brands crawled, 245,471 parts. The 153 it can't reach each carry a named, evidenced blocker.
- It runs unattended and grades its own output. It isn't allowed to delete a row it flagged.
- The dashboard is where I queue a brand and read what came back. No terminal, no session.
The Orb-Weaver dashboard: 245,471 parts indexed across 362 brands, searchable by name, code, brand, make or model, with the harvested product photo on every row.
The counts across the top are the four questions I was opening a terminal to answer, so they stopped being a report and became the header.
02 · The approach
Four ways in, cheapest first
System diagram · OW-01
Orb-Weaver pipeline
How roughly 400 independent storefronts with no shared format became one queryable index. A cheap probe reads each site first and decides the rung. A browser only ever spins up when the evidence says it must.
-
R1
Shopify feed Cheapest
Structured JSON, no browser at all.
-
R2
Sitemap + JSON-LD
The structured data the site already publishes.
-
R3
Platform harvester
BigCommerce, WooCommerce, Yahoo: written once, reused across stores.
-
R4
Bespoke template Last resort
Hand-written selectors. Expensive, non-scalable, last resort.
Three rules that carry the system
- Resumable beats fast: every step writes its outcome to the registry before the next begins.
- Verify from scratch: validators re-derive from source; a system that grades its own homework will pass.
- Say what you are: identifies itself, honors robots.txt, records why a site can't be harvested.
471
brands tracked
310
crawled successfully
245,471
parts indexed
153 brands deferred, every one with an evidenced cause in the registry.
Acts alone
- Probe decisions
- Feed harvesting
- Quality scoring
- Fitment computation
- Index rebuilds
Stops & asks
- Deleting a row
- Blockers without evidence
- Writing a template
- Seeding failed manifests
The pipeline is allowed to reach a wrong conclusion. It isn't allowed to act on one.
03 · The engine decision
The probe answers a question I used to guess at
Before anything crawls, a probe fetches the raw HTML with no browser at all and reports what it finds: the platform, whether product markers are already present in the served markup, and which feeds the site exposes. The engine choice falls out of that. Cheerio only gets returned when at least two product markers are sitting in the HTML that was served, which means the decision is evidence rather than a hunch about how modern the site looks.
That sounds like a small thing. It's the difference between spinning up a headless browser 400 times and spinning one up 90 times, and browsers are the expensive part of any crawl in both time and blocking risk.
04 · The surprise
Prestige runs backwards
05 · The dashboard
Where the pipeline is legible
The crawler could have stayed a set of command-line scripts, and for about a month it did. The dashboard exists because I got tired of opening a terminal to answer questions I should be able to see: what do we have, what is missing, and what happened to the brand I queued last night.
13,284 exhaust parts across the whole index. Fitment is a first-class column, because a row without it's invisible to anyone browsing with a car selected.
An empty fitment cell is shown as an em dash rather than hidden. It's a gap in the data, and the dashboard’s job is to make gaps easy to find.
06 · The state
Every brand ends with a recorded outcome
07 · Bookkeeping
A blocker needs evidence, not a feeling
153 brands can't be crawled, and it would be easy to record all of them as failures and move on. Instead each one carries a cause: dealer-only, bot-defense, robots-disallow, pdf-only, no-catalog, no-attributes, region-locked, needs-template. Setting a blocker sets the status automatically, so the two can never drift apart and disagree with each other.
The rule that made this worth doing is that a blocker only goes on the record when there's evidence for it. A plain fetch coming back empty proves the site needs a browser. It proves nothing whatsoever about whether the site has a catalog. I got that wrong early, wrote off a stack of brands as having no attributes, and had to go back and re-open all of them. That mistake is why 87 brands now sit under needs-template, which is an honest description of work remaining rather than a verdict on the site.
08 · Autonomy
What it decides alone, and where it stops
09 · Quality
It grades its own work and isn't allowed to act on the grade
After every successful harvest a quality pass runs on its own and judges the products rather than the counts. It measures the merchandise share, the automotive-vocabulary share and the navigation-text share, then attaches a verdict to the brand. A big number isn't evidence of anything: one store handed back 1,416 real parts and 3,756 pieces of apparel, and only a pass that reads the rows would ever notice.
One ordering detail took two attempts to get right. Merchandise has to be checked before automotive vocabulary, because a t-shirt called “BMW M3 Track Day Tee” contains a make and a model and sails straight through a vocabulary test. Getting that backwards is how twenty-three Formula E merchandise rows nearly shipped as Nismo parts.
When the verdict comes back as review, nothing is deleted. The samples go in front of me. An automated system that removes rows it finds suspicious is a system you can't audit, and the whole point of the registry is that I can go back and see why anything is the way it is.
10 · Queue
Adding a brand without opening a session
The Add Brand form is the whole operator interface. Type a name and a catalog URL, and a detached worker picks it up, tries the ladder from the top, harvests, runs the quality pass, computes fitment coverage, rebuilds the index and writes the result back. I read what happened in the morning.
Behind, the crawl queue: method used, parts harvested, quality verdict, and whether the brand made it to fiddle.diy, with sites the generic pipeline can't read flagged needs-template rather than reported as failures. In front, the brand registry: tier, status, part and category counts, catalog URL, when it was last checked, and the note explaining how it went.
“Nothing to harvest” and “needs-template” are two different outcomes, and collapsing them into one status would hide the only queue rows worth my attention. And the notes column earns its width: six months from now the question is never what the status is, it's why somebody set it.
11 · Iteration
Five things the running system disagreed with me about
None of these came out of planning. Each one is a thing I believed until the pipeline produced something wrong in front of me.
-
01
Named images by product code → content-addressed them
TriedSaving each photo as <product-code>.jpg, which reads perfectly well in a folder.
WallSites reuse one product code across several fitment rows, so two genuinely different parts could land on the same filename and overwrite each other.
PivotHash the part’s own image URL and name the file after that. Two parts now share a file only when they share a source image. It happened to have been harmless on the first brand (I checked), but that was luck, and luck isn't a guarantee I can ship on.
-
02
“Scroll until the count stops growing” → scroll until it holds and the page is at the bottom
TriedScrolling a lazy listing until the row count stopped changing, then reading the table.
WallThe page rendered in bursts with flat spells in between, so the loop declared victory during a pause. It returned 207 of 430 products and reported success.
PivotRequire the count to hold across eight consecutive polls and the viewport to have reached the bottom. A silent undercount is the worst failure mode in a crawler, because nothing about the output looks wrong.
-
03
Keyed on column position → keyed on the cell’s own label
TriedReading the attribute table by column index, the obvious way.
WallOne re-order on the site and every record silently shifts a column.
PivotRead the mobile-only label span each cell carries. The site can rearrange its own table and the extraction still lands in the right fields.
-
04
Fitment as a later retrofit → fitment carried end to end
TriedHarvesting parts first and working out what they fit afterwards.
WallA part with no fitment is invisible to every Fiddle user browsing with a car selected. The first 250 brands needed a re-feed to fix it.
PivotKeep the raw store tags and long descriptions through the whole harvest (chassis codes live in there), and report a coverage split on every brand, flagging anything over 25% dark. No brand since has needed the retrofit.
-
05
Harvesting from a category page → harvesting from the sitemap
TriedPointing a harvester at a store’s category listing, which looked like the tidy way to scope it.
WallOne brand capped at 28 parts out of a 489-product catalog, and captured the whole product card as the price. It didn't error. It returned a confident, tiny, wrong answer.
PivotEnumerate from the sitemap and filter on the vendor field. And when a run comes back suspiciously small, treat that as a bug rather than as a small catalog.
12 · Design position
Three rules the pipeline is built on
-
01
Resumable beats fast
Every step writes its outcome back to the registry before it moves on, and each category crawls in its own process. A run that dies at brand 200 picks up at 201 instead of starting over. Nothing in this project is time-critical, but plenty of it is failure-prone, and those want opposite optimizations.
-
02
Say what you are
The crawler identifies itself, honors robots.txt, and respects a published crawl-delay even when that means a 30-second gap between requests. When a site pushed back with a 429 I recorded that its published delay wasn't its real budget, and slowed down rather than reaching for a disguise. This data feeds a product I put my name on.
-
03
Verify from scratch, never from the claim
The validator re-derives every part ID and image hash from the source data rather than reading back what the crawl wrote. Checking a system against its own output only ever confirms that it was consistent, which isn't the same as being correct.
13 · The stack
Everything it runs on, and what each piece is for
I didn't pick anything here because it was interesting. Each one earned its place by being the cheapest thing that solved a problem I hit.
| Tool | What it does here | Why this one |
|---|---|---|
| Node.js | The pipeline, the harvesters, the dashboard server | Same runtime as Fiddle, so the seed contract is shared code rather than a translation |
| Crawlee | Queue, retries, concurrency, session and proxy handling | Well-worn answers to the boring half of crawling |
| Playwright | Renders JS listings and drives click-through detail popups | Only used when the probe proves a browser is genuinely needed |
| Cheerio | Parses server-rendered HTML with no browser at all | Roughly an order of magnitude cheaper, and it can't be fingerprinted |
| Python | Registry build, brand reconciliation, worklist and status reporting | Pandas over a 410-row spreadsheet is a five-minute job |
| Claude Code | Pair-programming across all 77 scripts | I make the calls about what the system may conclude; it does the volume |
| Shopify Product JSON | /products.json?limit=250, rung one of the ladder | Complete and structured, and one feed can cover several brands at once |
| schema.org JSON-LD | Product markup lifted off sitemap-enumerated pages | Rung two, and still no browser |
| BigCommerce Stencil | Bespoke harvester for undeclared paginated sitemaps | The generic path 404s on these; microdata, no JSON-LD |
| WooCommerce Store API | Harvests Woo storefronts | Another whole platform for the price of one script |
| XML sitemaps | Enumerates the catalog instead of walking category pages | Category pages are how one brand capped at 28 parts out of 489 |
| robots.txt | Read and obeyed, crawl-delay included | This data feeds a product with my name on it |
| SHA-256 | Content-addresses every image and stamps the downloaded bytes | The only way two parts can never inherit each other’s photo |
| node:http | The dashboard, with zero added dependencies | A local operator UI shouldn't drag in a framework |
14 · Built with Claude
Seventy-seven scripts, and the design work underneath them
Orb-Weaver is roughly 77 scripts, and I didn't type most of them. I worked through it with Claude Code the same way I build Fiddle: I make the calls about what the system is allowed to conclude, and Claude does the volume. What I spent my time on was the contract: what a blocker means, what has to be true before a part can be seeded, which verdicts are allowed to trigger an action and which have to wait for me.
That's the part that transfers. Everything in here is a smaller version of the argument I have at work most weeks: how sure does a thing have to be before it acts, what does it owe you when it's unsure, and what does it record so that somebody can go back later and work out why. Those answers are a great deal easier to test when it's your own data and your own product on the other end of a bad one.
15 · Where it stands
Where it stands
- In service. 471 brands tracked, 310 crawled, 245,471 parts indexed, and it's the catalog behind Fiddle.
- Adding a brand is a form, not a session: queue it, the worker runs the ladder, grades the result and refreshes the index on its own.
- The 153 brands it can't reach each carry an evidenced cause, so the backlog is a work list rather than a mystery.
- Named the whole thing Orb-Weaver because that's the spider that rebuilds its web every day, which turned out to be more accurate than I meant it to be.
- Next: working the needs-template backlog, and pushing harder on the brands sitting behind bot defenses that I have no intention of disguising my way past.