"Gather"

These are helper methods I've used to scrape and compile information from the web. This code will need to be cleaned thoroughly and documented to ensure all data is properly and clearly sourced.

How to QA data? How to verify disparate datasets?

Scrape Sources

Check Links

.json/.yml verify incoming data with validations and callbacks.

web -> .json -> table batch load and show a status report for each record with count (red/green status & filter)

ability to schedule pulls and show job logs

all wrapped in a web interface

Validate common formats and version field changes in archive_tables.

people (vcard) events news_rss calendars twitter facebook

extract and transform data into open datatypes (reference each schema)

Syncing to public calendars (of various formats and time periods)

Idea: allow users to embellish (this?) to closely document the structure. Make it highly interactive and enjoyable to use. This relies on user functionality (perhaps like FourSquare) +1 point, +10 points.

How do we determine % complete? What is 100%?

Sources Election

Data Officials Positions Structure (Ontology)

Scope Maps Officials Structure

Validate websites with Section 508

NACO shows 3143 counties Wiki shows 3140 counties

automate the maintenance of data seek completeness in all datasets identify authoraitve source references for each table (script the import from public sources!!!!) load data as additions and/or updates, log and make the results useful

for each record in a table, look for a missing piece of data (ie; lat/lng) - show the itme on the screen, in an actionable manner.

automate google search query to a popup - then select from the results

automate google searches for each county, state, city, town, position (and type of position)

TODO: write a wikipedia importer

TODO: define things that must stay in sync (counties to cities, jurisdictions, zip, shape files, election data, address, phones, urls) - How will this be handled?

Create holder records where necessary to enable input.

add newspaper to RSS events link newspapers to calendar

List of Gov Data Employees (Job Titles, Contact Information) Event Calendar (Public Meetings, Education Opportunities, Other) p293 of OpenGov - Correspondence Logs An index of information holdings financials, PO’s, contracts FOIA Requests RFP’s Non digital archived information + classified info Businesses & license data taxes demographic/population environment crime parking

where are source legislative documents? create stubs for wikipedia where they do not exist. link otherwise.

*Idea: allow a user to claim a city - specifically an official, government-employed official. how would this look? send an email to them, allow them to respond with the contact data. Enable them to update once a year. Query them after the times when data may change.
*Idea: how to ensure data is updated in a timely manner? how to ensure it is accurate?

Curation tool: Allow feedback, suggestions, and link submissions. Who will handle?

INCLUDE PUBLIC MEETING INFO

Entities to Track

Do source URL's exist for these agencies?

· Federal

· State

· County

· City

Sub-Entities to Track

These publicly-funded organizations typically operate at the city and/or county levels, and provide employment and services on the public's behalf.

The table to track this should look like a huge matrix. What exactly does the table look like?

· Sheriff

· Probation

· Health and Social Services

· Auditor

· Controller

· District Attorney

· Mayor

· Board of Education

· Superintendent

· Treasurer

· Secretary

· Admin

· Marketing

· Human Resources

· Information Technology

· Family Support

· Mental Health

· Registrar of Voters

· Tax Collector

· City Council

· County Boards

· Senators

· Fire Chief

· Police Chief

Services to Track

· Local and Public Services

· Business Licenses

· Cemetery

· Child Support Services

· Datasets

· DMV - Motor Vehciles

· Fire

· Jury Duty/Court Houses

· Libraries

· Medical

· Opportunities to Participate

· Parks

· Police

· Post Offices

· Prison/Corrections

· Public Art

· Public Sales

· Requests for Proposal

· Research

· Schools

· Senior Citizen Services

· Sports/Recreation

· Taxes (Property Taxes)

· Transportation/Roads

· Veteran Support Services

· Welfare

Utilities

· Electricity/Power

· Phone

· Sewer

· Television Cable

· Trash/Garbage/Waste

· Water

Data to Track

· Financials

· Budgets

· Employee Benefits

· Employee Counts

· Geographic Information

· List of Organizations

· Salary

· Legislation

· Demographic Data

· Taxes

· Custom Reports

· “Exposes”

Additional Opportunities for Exploration, Development, and Growth

scrape data from the web and store it in a database (if it is not otherwise readily accessible). Perform many scrape to populate lots of data, forms.

· get data from multiple sources

· types

· rss feed

· xml

· csv

· other?

· url

· http://www.123.com/filename.csv

· define the fields, ensure the data is valid and 'stamp' it with an update date. this would require admin permissions.

· be able to define and interface with those sources

· data360.org?

· add additional user content, but manage it independently of the "sourced" data

o about the "sourced data"

o identify the:

§ url

§ format, or other processing steps

§ what data is it? Who does it belongs to?

· THE PEOPLE!

· political offices

o sheriff

o Auditor-controller

o District Attorney

o Board of supervisors

o Treasurer

o Etc.

§ public buildings & services - tie these to budgets

§ public decisions - who makes them, and what is the status on any given upcoming decision

§ budget information

§ employee count

§ salary schedule

§ organization charts

§ other public documents

§ tax rates

§ crime statistics

§ health statistics

§ census

post RFP's online

Tenure's positionid personid startdate enddate

help Public Agencies comply (agency tracker (green checkmarks if meetings are posted in time - the system would need to identify a meeting date, and identify the agenda posting date (and verify it by storing a copy locally)