248 Cities and Counting: How We Built the Largest Permit Database in the US

When we launched the first version of Permitlify in early 2024, our "permit database" was a Python script that ran on a $5 DigitalOcean droplet and scraped 12 city permit portals in the Dallas-Fort Worth area every morning. It took about 40 minutes to run, frequently timed out, and had a tendency to silently fail when a city updated their portal layout.

Today, Permitlify monitors 248+ city and county permit portals across the United States, processes 30,000+ permits daily, and delivers normalized, enriched, AI-scored lead data to contractors' dashboards before 6 AM local time — every single day. Here's how we got there.

The fragmentation problem

The fundamental challenge of building a permit database isn't technical — it's political and jurisdictional. The United States has no national building permit system. Permits are issued at the local government level — sometimes by cities, sometimes by counties, sometimes by both, and sometimes by special districts or townships. Each jurisdiction picks its own software, its own data format, and its own policies about public access.

The most common permit platforms are Accela (used by 800+ municipalities), EnerGov, Socrata, OpenGov, and ArcGIS. Each of these has multiple versions, each with different API structures, rate limits, and authentication requirements. Then there are hundreds of custom-built systems — everything from polished React apps to Microsoft Access databases published as HTML tables.

Building the scraper fleet

We built what we internally call the "scraper fleet" — a modular system of data collectors, each tailored to a specific permit platform or portal. Each scraper handles:

• Authentication (cookies, API keys, session tokens, or no auth)
• Pagination (some portals return 25 records per page, some 1,000)
• Rate limiting (government servers are sometimes slow; we're respectful of their infrastructure)
• Change detection (detecting layout changes and alerting our engineering team before data quality degrades)
• Normalization (converting each portal's unique field names into our standard permit schema)

Each scraper runs on a schedule calibrated to the portal's update frequency. Most city portals update once daily, typically overnight. Some larger metros update in near real-time. A few update only weekly — we scrape those on their publication schedule and clearly label the data freshness in the dashboard.

Contact enrichment

Raw permit records contain property addresses and applicant names. They don't contain phone numbers or emails. Contact enrichment is the step that turns a property address into a callable lead.

We cross-reference each permit address against multiple public data sources: county property records (available in most states as public downloads), voter registration data (also frequently public), and licensed contact databases. We run a multi-step matching process that confirms the homeowner name matches the permit applicant before appending contact details. Our current enrichment match rate is 78% across all permits — meaning roughly 4 in 5 permits we process have at least one verified contact method.

What's next: 500 cities by year end

We add new cities every month based on subscriber demand and engineering capacity. Cities and metros currently in our pipeline include: Phoenix, Las Vegas, Denver, Seattle, Portland, Minneapolis, St. Louis, Pittsburgh, Cleveland, Cincinnati, and hundreds of smaller metros across the South and Midwest.

If your city isn't covered yet, you can request it directly from your Permitlify dashboard. We prioritize new city coverage based on the number of subscriber requests — so the more contractors requesting a market, the faster it gets added.