considering running this as a rake task. google spreadsheet to batch upload/upsert Positions

spreadsheet ----> .json ------> into a datastore url (scraper wiki)

When it comes to gathering data for the creation of a comprehensive database of public officials at all jurisdictions in the United States, there is a lot to consider.

It is integral to have a well-defined and executable strategy to ensure the integrity, timeliness, and completeness of data. There are 2 primary considerations in this data management strategy:

Structure Database (1) Table (10+) Fields (1..) Curated Tables (TODO: discuss what is available by API and/pr download) Agencies Governing Bodies Positions Sourced Tables (TODO LINK ALL SOURCES) Area Codes Cities Counties Shapes FIPS States Zip Codes (Jurisidictions - States, Counties, Cities, Towns) Curation Gathering Scraping Look for a list of positions at every level Look for a list of governing bodies at every level Scrape for feeds Calendar Notices Others Crowdsourcing Allow a user to create Agency for a Jurisdiction create Governing Body for an Agency Verifiying Agencies Governing Bodies Admin Data integrity Ensure at least one agency for every Jurisdiction Monitor how many Jurisdiction’s Agencies have a URL In addition to #2, How many do not? There is normally a Governing Body for each jurisdiction. Each Governing Body typically has more than 1 member (2..). Often members of bodies participate formally or informally in subgroups. Ensure Positions are properly linked to Governing Bodies Ensure Governing Bodies are properly linked to Agencies Seek completeness for Elected Positions as well Scan websites for recent updates Scan websites for accessibility (W3C, Section 508)