← Analysis
Why we don't know what we know
Nearly every fact needed to answer a real question about American life is already sitting in a public government database somewhere. The problem was never access. It's that no two agencies built their data to fit together, so knowing something in principle and knowing it in practice have quietly become two different things.

Start with a concrete case, not the general claim. The Federal Emergency Management Agency publishes a National Risk Index that scores hurricane, flood, and other hazard exposure for every county in the country, using a five-digit code -- two digits for the state, three for the county -- called a FIPS code. The National Weather Service publishes real-time hurricane and tropical storm event data, including property damage estimates, for the same country. Put those two datasets side by side to ask a simple question -- which counties took the worst hurricane damage since 2005 -- and the join silently fails. The Weather Service doesn't report hurricanes by county at all. It reports them by forecast zone, a different geographic unit built around how weather actually moves rather than where county lines happen to sit, and zones carry their own three-digit codes that look like county codes but aren't.[1] A machine matching on code alone would silently stitch damage figures to the wrong places. It would run, return an answer, and be wrong in a way nothing on the screen would ever reveal.

This is not one agency's oversight. It's the normal condition of the data. FIPS codes themselves trace back to 1965, a NIST-issued standard the federal government formally retired in 2008 -- and kept using anyway, because too much downstream infrastructure already depended on it, official replacement or not.[2] The Census Bureau's own metropolitan-area definitions are a third, separate geography: built from whole counties, but grouped by commuting and economic ties rather than administrative lines, redrawn on its own schedule by the Office of Management and Budget, and routinely crossing state boundaries in the process.[3] Three real, official, actively maintained ways to describe the exact same American land, none of them designed with the other two in mind. The National Weather Service does maintain an official file translating its zones back to counties -- proof the mismatch is well known inside government, and proof nobody made translating it anyone's job by default.[1]

Zoom out and the pattern isn't a handful of edge cases -- it's structural, and it's been flagged by the government's own watchdog. The Government Accountability Office's geospatial-data review found no effective coordination mechanism across the National Spatial Data Infrastructure, producing duplicated and incompatible geographic data efforts across federal, state, and local government.[4] A more recent GAO review of federal award and payment data found the same root cause in a different domain: no agency has actually been designated to enforce data-interoperability standards across government, so adoption stays voluntary, and different agencies land on different definitions and validation rules for what should be the same underlying facts.[5] This isn't a story about incompetence. It's a story about incentives: every agency built its data collection to serve its own mission first, and cross-agency compatibility was never anyone's job to own.

The failure mode that matters most is the quiet one, not the loud one. A dataset that's simply missing announces itself -- the question goes unanswered, and everyone knows it. A dataset that joins to the wrong geography does something worse: it produces a confident, specific, wrong answer that looks exactly like a right one. The same government data source can carry this trap inside a single agency's own historical archive -- IPUMS NHGIS, the standard source for pre-1970s county-level census data, encodes its state and county identifiers with an extra trailing digit that isn't standard FIPS at all; a real county's genuine FIPS code of 01001 shows up in the historical file as state-code 010 and county-code 0010, and has to be divided by ten to recover the code every other modern dataset expects.[6] Nothing about that file announces the difference. It just quietly produces wrong joins for anyone who assumes "the county code" means the same thing everywhere.

What this means, stated plainly: knowing and having access to the facts needed to know are not the same thing. The hurricane damage data is public. The hazard-risk data is public. The historical settlement data is public. Nothing about any of it is locked behind a paywall or classified. And still, the honest answer to "which Gulf Coast counties have taken the worst hurricane losses since Katrina, and how does that compare to their underlying hazard exposure" is not sitting anywhere, fully formed, waiting to be read. Someone has to notice that the codes don't mean the same thing across the two files, build the translation by hand, and verify it against a known case before trusting a single row of the output. That translation work is invisible from the outside -- it leaves no trace in the final report, which is exactly why the fragmentation underneath it goes unnoticed by everyone except whoever had to do it.

The honest name for this gap is the one in the title. Not "we don't have the data" -- the data exists, is public, and is often extremely rich. Not "we haven't looked" -- plenty of people have looked at each individual dataset. The actual gap is between data that exists and knowledge that's usable, and the bridge between them is unglamorous, mostly invisible, integration work that nobody's mission statement assigns to anybody. Until that work gets done, for a real question, the answer stays technically public and functionally unknown -- which is a strange, quiet kind of not-knowing, since every fact required to know it was sitting in plain sight the whole time.

Three real geographies, one American county FIPS county code: 2-digit state + 3-digit county, NIST standard since 1965, formally retired 2008, still what nearly everyone actually uses.

NWS forecast zone: a separate geography built around weather patterns, not administrative lines -- coastal and mountain counties often split across multiple zones, with a real but easy-to-miss official crosswalk back to counties.

Census MSA/CBSA: a third geography built from whole counties but grouped by commuting/economic ties, redrawn periodically by OMB, routinely crossing state lines.

The trap: codes from different systems can look identical in shape (three digits, five digits) while meaning something completely different -- IPUMS NHGIS's historical county codes need to be divided by ten to become standard FIPS, with nothing in the file itself announcing that.

The real gap: not access to data, but the unglamorous, invisible work of making incompatible public datasets actually fit together -- work no single agency's mission ever assigned to anyone.
Sources
  1. National Weather Service, Zone/County reference and County-Public Forecast Zones Correlation files, NOAA
  2. Wikipedia, "FIPS county code", on the 1965 NIST origin and 2008 formal withdrawal
  3. U.S. Census Bureau, "About Metropolitan and Micropolitan Statistical Areas"; Congressional Research Service, "Core Based Statistical Areas"
  4. U.S. Government Accountability Office, GAO-15-193, "Geospatial Data: Progress Needed on Identifying Expenditures, Building and Utilizing a Data Infrastructure"
  5. U.S. Government Accountability Office, GAO-26-107466, "Federal Data: Congressional Action Needed to Improve Interoperability of Award and Payment Eligibility Data", June 2026
  6. Not a published study -- a real, documented finding from this site's own civic-data ingest engineering: building a NOAA hurricane-impact pipeline against the site's existing FEMA county-level hazard data surfaced the NWS zone-vs-county mismatch directly, and cross-referencing IPUMS NHGIS's historical county encoding (used elsewhere in this same data layer) confirmed the same class of trap recurring inside a single archive.