“Billions of part numbers” is a counting problem: an open component database should build only the identity layer

The most expensive part of this project is not storing billions of records, it is making every one of them verifiable by somebody else. SiliconExpert says so plainly on its About page: a 1-billion-plus part database handcrafted by more than 500 electrical, software and data engineers, with API data updated daily. At public cloud rates, a few hundred gigabytes cost a few dollars a month — four to five orders of magnitude below several hundred people for a year. So the right move is not to clone a "free SiliconExpert", but to build the one layer it is least likely to open and that most needs public governance: a unique identity for brand plus part number, and the evidence chain behind that judgement.

Figures checked 2026-11-08. Direct access to the vendor site returns Akamai Access Denied; this dossier uses indexed snapshots of public pages (About / FAQ / Pricing / the Chinese "Design Right" page), official Chinese copy, public reseller material and third-party 2026 buying guides. Every estimate is labelled as such.

Working codename CUPI (Component URL & Provenance Index), not a formal product name. Opened 2026-11-08.

1. Correct three figures first, or the budget is wrong from page one

Two of the three assumptions in the brief need a different denominator. Only claims that can be traced to a source are listed below, with marketing numbers kept separate from hard metrics.

So do not start by sizing servers. The real question is how much human effort it takes to declare one brand-plus-part-number unique — and still have somebody else able to re-verify that call three years later.

2. "1 billion+" is not a billion part numbers

Unpack the billion: one chip can occupy separate records for tape-and-reel, tray or tube packing, lead-free and RoHS variants, different distributor SKUs, datasheet revisions, and pre- and post-discontinuation status. Passives are worse — a chip-resistor series expands across resistance × tolerance × voltage × temperature coefficient × package × packing, a Cartesian product in its own right.

Compare against verifiable anchors. DigiKey states "Over 18 million listed parts, with over a million available for immediate shipment"; Mouser is around 6.8 million products across 1,200+ lines (third-party 2026 guide); SiliconExpert claims 1 billion+. A catalog distributor already lists in the tens of millions, so a commercial intelligence database two orders of magnitude larger is best explained as indexed records — duplicated source entries, packaging variants, historical revisions and parametric combinations — not a billion distinct orderable entities.

This means the instinct that "the volume must be large" points in the right direction but at the wrong object. What is large is not the set of unique entities but the combinatorial space of aliases and variants. Establishing identity means folding that space back into one stable ID — an entity-resolution problem, not a capacity problem.

The conclusion is firm: make "how many rows we store" the goal and you fall into a hole you can never fill; make it "can we decide whether two strings denote the same orderable entity" and the scale becomes manageable, and the capability is hard to copy.

3. Benchmark: three market layers, and the expensive one is not search

Grouping Chinese and international platforms by how they make money reveals more about pricing than a feature list does.

On pricing: the SiliconExpert Pricing page and FAQ offer only a "customized quote". Third-party 2026 buying guides suggest USD 5,000-9,000 per seat per year, other reports give USD 10k-25k for a team tier and roughly USD 100k a year for API access, with Z2Data at USD 12k-35k and Supplyframe DSI at USD 7.5k-50k. A spread that wide says this is not commodity pricing. "Tens of thousands a year" is consistent with an entry seat or single-module deal via a local reseller, but it is not an official price.

The important observation: none of the three layers charges by record count. Transaction earns on fulfilment, discovery on call volume, decision on whether you dare use the data to sign off an obsolescence decision. The subscription buys "a source to point at when something goes wrong", not "a result I can look up".

4. What the annual fee actually buys

Three sentences in the official FAQ can be read as a requirements document: "over 1 billion components, including cross references, lifecycle, parametric, obsolescence forecasts, regulation compliance and inventory data"; "We centralize and normalize that data"; and "With over 500 data engineers ... Supporting source documentation is always provided".

The third sentence is the decisive one — it exposes what the product really is. The deliverable is not a parameter table, it is the evidence behind every assertion. In a compliance workflow, a wrong RoHS status does not put liability on the vendor server; it lands on the engineer who signed. That is why one click has to lead back to the manufacturer declaration, its revision and its date. The field list on the Chinese site says the same: ECCN, REACH, SCIP, China RoHS, conflict minerals, material declarations, PCNs, JEDEC documents — nearly every item needs supporting paperwork.

Then there is freshness. The Pricing page says API data is "updated daily". Component data decays fast: PCNs, EOL and last-time-buy notices, part-number suffix changes and brand migration after acquisitions all happen weekly. A database that cannot be kept current is a historical archive on the day it ships.

The cost structure follows: crawling and parsing is one-off engineering; normalisation and disambiguation, provenance maintenance and compliance classification all carry permanent headcount, and each needs a named human behind it. The first can be automated. The other three are what those 500-plus engineers do every day.

One more thing an open project cannot obtain: SiliconExpert is an Arrow Electronics subsidiary and can draw on authorised-distribution transaction data to train end-of-life forecasts. That data is out of reach for an open project, and pretending otherwise is dishonest. Listing the feature in a comparison table without stating that data prerequisite is the most common self-deception in plans like this one.

5. Turn the cost into arithmetic you can check

Everything below is an order-of-magnitude estimate, not a quote. Unit prices use common public object-storage tiers; headcount is roughly costed at the officially stated 500+ engineers, excluding benefits, management and sales. The point is only to compare magnitudes.

If somebody uses "billions of records" as the budget justification, the direction is backwards. What pushes the price into five figures a year is that somebody must decide, every day, whether two part numbers are the same thing — and file the evidence for that decision. Storage is the cheapest line in the whole calculation.

Looked at the other way, what an open project can save is precisely the largest line: do not employ hundreds of people to clean full parametrics. Make only the identity layer thin and verifiable, let manufacturer notices, distributor catalogs and community contributions each supply evidence, and let rules rather than headcount do the merging.

6. Legal boundary: decide what must not be copied

Facts and expression have to be separated. A part-number string, a nominal resistance or a package dimension is normally not protected by copyright; datasheet PDFs, the arrangement of parametric tables and their wording are. In the EU, a database may also enjoy a 15-year sui generis database right (Directive 96/9/EC) for a substantial investment in obtaining, verifying or presenting its contents, which blocks extraction and re-utilisation of substantial parts — and a component catalog plainly qualifies as a substantial investment.

Articles 3 and 4 of Directive (EU) 2019/790 introduce text-and-data-mining exceptions, but Article 4 lets rightsholders reserve rights in machine-readable form (robots.txt, TDM reservation protocols), and the research exception has eligibility conditions. A commercial open project cannot assume it qualifies. When you see a robots.txt, stop; do not route around it.

The architectural consequence is firm: do not import somebody else cleaned result set. Draw only from a rights-cleared factual layer — public manufacturer part pages, PCN / EOL notices, standards-body documents, and data explicitly released under CC0, ODbL or open API terms. Store only "assertion + source URL + capture time + content hash + licence" per record. Do not reproduce the text or mirror the PDF.

Publish your own database under ODbL (OpenStreetMap moved from Creative Commons to ODbL in 2012 precisely to address database rights), license the code under Apache-2.0 or MIT, and consider CC0 for the pure factual layer; where content rights are mixed, mark them per record as ODbL recommends. This must be settled before the first line of collection code, or the licence chain becomes unrecoverable later and a single takedown request sends the whole project back to zero.

7. How far the open-source side has already got

This direction is not empty; what is missing is quite specific. The table lists only projects whose implementation details can be inspected.

Put the four together and the gap is obvious: there is no public identifier owned by no platform, referenceable across catalogs, versioned, evidenced and subject to community arbitration. Everyone rebuilds the same primary key inside their own system, and nobody is willing to claim that "these two part numbers are the same thing" on a basis anyone can re-check.

8. Conclusion: build only the identity layer

Codename CUPI, defined in one sentence: store no full parametrics, only "who said, when, and on what evidence, that these two strings denote the same orderable entity".

This is division of labour, not competition. The open layer owns the public good of identifiers, aliases, brand lineage and change events; commercial platforms keep charging for years-to-end-of-life forecasts, compliance documentation and risk scores. The more trustworthy the open layer, the lower the normalisation cost for commercial databases — the two are not mutually exclusive.

This also explains why "too expensive" does not lead straight to "build a free one". What the customer cannot afford is judgement and liability, and judgement needs permanent people and institutions. Open source can replace the fact that one company decides identity unilaterally; it cannot replace having somebody sign their name.

Verifiability is the only moat. Every assertion can be replayed by anyone from its source URL and hash, disputes run through public pull requests and an arbitration record, and mistakes can be rolled back while history stays immutable. Commercial databases deliver conclusions; the open layer delivers the ability to check them — the same problem this site keeps running into when writing about components.

9. Minimal data model, ID design and phasing

Keep seven field groups only: canonical ID, brand namespace, canonical MPN, alias set, lifecycle events (PCN / EOL / last-time-buy, with date and notice link), source evidence (URL + timestamp + hash + licence), and confidence plus arbitration record. No stock, no prices, no compliance verdicts, no hosted datasheets.

The ID borrows from purl (Package URL), shaped like cupi:brand/mpn@rev?pkg=...&grade=...#variant. The namespace absorbs brand acquisitions — legacy part numbers map into the new namespace while keeping their original ID — and qualifiers absorb packing and grade suffixes so variants do not explode into main-table rows. The spec must pin down case, hyphens, leading zeros and suffix handling: MPN disambiguation usually fails on exactly those characters, not on parameter comparison.

Governance borrows the Git workflow from nawotech/parts: JSONL as source, SQLite build, pull-request review, a maintainer per category. A new merge candidate needs two independent sources; conflicts go to an arbitration queue rather than being merged automatically. That is the same discipline as project one — emit candidates, never merge automatically — because a false merge is irreversible.

Phasing: weeks 0-12 cover one category only (passives first, or one MCU family), 10,000 entities, three sources, plus a public validator and CI. Phase two adds brand lineage and PCN / EOL event subscriptions. Only phase three discusses community governance and a read-only API. No phase is accepted on record count; acceptance uses dispute-resolution rate and evidence replayability.

10. An explicit list of refusals and three stop lines

• No full parametric database, no distributor prices or stock, no years-to-end-of-life prediction — those three are exactly where commercial databases earn their fee and carry the heaviest liability.
• No marketing as a "free SiliconExpert replacement". If the name says open database but not identity layer, it will be compared on record count from day one and die on record count.
• No records from unclear or uncleared sources. Ten thousand verifiable records are worth more than a hundred million rows of unknown provenance.
• No promise of engineering fitness for use. The documentation homepage must state that this layer serves identification and provenance, does not replace the manufacturer datasheet, and constitutes neither alternative-part qualification nor a compliance verdict.

Stop lines: first, if the mean arbitration time across 10,000 entities exceeds 14 days within the first 12 weeks, the governance model does not work. Second, if an upstream source publishes a machine-readable rights reservation or sends a formal request to stop collection, remove that source immediately instead of routing around it. Third, if no third-party tool adopts the ID — not even a single KiCad plugin — the layer has no public-good value; stop.

Summary

Back to the opening question: is billions of records expensive? Measured in bytes, not at all. Measured as "every row can be re-verified", it is expensive enough that a company employs more than 500 engineers — which is why a subscription costs tens of thousands a year.

What open source can change is not that price but that structure. Turn identity resolution from one company private asset into a public layer with sources, rollback and arbitration, and leave the high-value judgements that need a signature on the commercial side. That way you neither pretend to clone SiliconExpert for free, nor fail to open the one layer that most needs opening.

One sentence to close: do not build a bigger database, build a ledger anyone can audit.

Back to Idea Vault · Project one PrivateGraph BP · Project two One-click distribution risk analysis; related practice: component sourcing and matchmaking.

Postscript 2026-11-08: upstream sources are turning access into a priced commodity (Pay Per Crawl evolving into Pay Per Use and x402, which can charge for pages, datasets, APIs and MCP tools), while Content Signals remains a request rather than a licence. The original refusal list — no distributor pricing or stock, only original facts with clear rights — holds more strongly than ever. The first order should be a manual BOM disambiguation report; see the 5W1H re-sequencing behind the access walls.