ryanwold.net - Civic Tech, Bitcoin, and Open Government Innovation

Show

An entry

Get it down. Make it good. Share it.

Title

Text

Work Completed:

Scrape data from c.org
C.org has a page for each of 50 states.
Each state page has an A-Z index that leads to a "State Alpha" page.  For example, California cities that begin with the letter A.
Each "State Alpha" page lists all the cities, counties, or jurisdictions that begin with a letter.  The data is structured with a link to the city, followed by its County/Borough/Parish.  For example: Vacaville «- Solano County.
Each city or county also has a detail page which holds the valuable information that we're looking for, including Address, and Public Officials.
I have run into an anti-scraping mechanism on c.org that blocks repeated, unthrottled page requests.  This has been avoided by adding a 2 second + 4 second random delay between pages.

To do this, the webpages must be viewed and saved to the local disk (for processing later) - this data is considered "cached".
Additionally, it is necessary to continually update and ensure accurate data. This can be done by a) identifying official "source" data that is often maintained at local jurisdictions, such as Counties, b) crowd-sourcing updates and verifications of data.

The overall workflow of this effort is as follows:
C.org page -» stored as local page -» scraped data to .yaml -» import data into database

Work to be done:
For every city and county, search:
Board, City Council, City Manager, Elected Positions, District Attorney, Sheriff, Auditor-Controller, Commissions, Committees, Voting Records, Board Minutes, Agendas, Members, Budget, Finances, Financials, GASB? Search for RSS feeds, Excel files, and other tools. Facebook, Twitter.

Status

Series