Spring 2010


Spring 201029 Mar 2010 11:04 am

Day 17 notes are available.

Spring 201025 Mar 2010 08:18 am

See Day 16 notes

Spring 201015 Mar 2010 11:31 am

Day 15 Notes are available.

Spring 201010 Mar 2010 12:28 pm

See slides.

Spring 201008 Mar 2010 12:23 pm

Day 13 notes

Spring 201003 Mar 2010 08:21 am

Day 12 notes are available.

Spring 201002 Mar 2010 10:32 am

I published notes for Day 11 yesterday.

Project Proposals and Spring 201025 Feb 2010 11:33 am

What problem is your project aimed at solving? Alternatively, why would someone want to use what you are making in your project?

The first immediate goal of the American Recovery and Reinvestment Act of 2009 is to “create new jobs as well as save existing ones.”

While Emily’s initial project idea was to compare jobs created and saved vs. unemployment, some initial research proved that the exact number of jobs created per grant or contract is difficult to determine with the level of regularity and reliability needed for creating a machine-readable, automated data mashup.  A good example is the problem of bulk grants.  The Regents of the University of California have been earmarked to receive a grant of $716 million dollars

which will save some 38,923.98 jobs, but the distribution of those jobs within the state of California is not clearly delineated, and furthermore, the number of jobs is a combination of full-time employees, part-time employees, sub-contractors, and vendor hours, again unspecified by county placement.

Due to these problems–the lack of consistency and specificity needed to achieve this initial project goal, we will be focusing on a slightly different question within the manageable scope of Alameda County, California:  How has the ARRA affected (or not affected) unemployment in Alameda County, California?  If the ARRA has been “effective,” we would hope to see the level of employment decreasing or remaining steady over time as funds are dispersed.  Of course it is difficult if not impossible to make any causal conclusions by comparing this data as opposed to examining job-for-job data (and taking into account other indices of the recovering economy), which I have already mentioned is an intractable problem until more granular data can be reliable extracted.

Is your project doable given the constraints of time, our starting knowledge, etc?

Yes, we think so.  We’ve narrowed the geographic scope from the entire U.S. to Alameda County and will only be needing two sources of data:

  1. Month-by-month unemployment data in Alameda County from January 2009 to the present
  2. The start date of all ARRA grants/contracts dispersed in Alameda County to date

The visualization portion will be tricky, but I am confident that we have the resourcefulness necessary to complete this part of the project.

What interface are you imagining? Is it a web, desktop, or mobile application? What platform are you running?

We want to build a web visualization which can be viewed with the Firefox web browser.

What data or services are planning to bring together?

  1. Unemployment data from the Bureau of Labor Statistics (http://www.bls.gov/lau/#tables)
  2. ARRA Project award start dates and amounts from http://www.recovery.ca.gov/HTML/RecoveryImpact/map.shtml?county=Alameda
  3. ClearMaps or Tableau or Google Visualization API

What’s your plan for getting the data? Often the data you might want might not be easily available.

Downloading the latest unemployment and ARRA files available and filtering for Alameda county.

Do the APIs you plan to use actually support the functionality that you need in your application?

No, we are not using any APIs, except for maybe the Google Visualization API, but that is not a data source.

What programming language do you plan to use?

Either Python or PHP, depending upon which one is easier to use with the data.

Action Plan:

  1. Download and examine unemployment and ARRA data files.
  2. Filter the files by Alameda County.
  3. Find zip codes for Alameda County and extract relevant ARRA data from master file.
  4. Investigate best mapping/visualization tool, hopefully one which will allow us to make monthly step-wise comparisons in unemployment, new project contract start dates, and the percentage of ARRA money spent out of the total currently slated for Alameda County.
  5. Integrate the data sets with the visualization utility.
  6. Celebrate.

Risk Areas/Mitigation Plan:

Right now our biggest risk is scrubbing the ARRA data.  We imagine that not every award will have listed its zip code, zip codes may span more than just Alameda County, or that field may be blank.

Our best mitigation plan might actually be to expand the scope of the project to include all of California, since the award’s state is a more reliable data point.

Spring 201024 Feb 2010 12:36 pm

Day 10 Notes.

Project Proposals and Spring 201024 Feb 2010 12:32 am

– What problem is your project aimed at solving? Alternatively, why would someone want to use what you are making in your project?
My project is not necessarily aimed at solving a problem, but more an interesting tool that might turn out to be useful. I am making this tool so that it can be quick and easy for a user to check how the counties or districts of California compare on different issues (environmental, gay rights, tax reform, etc.), the party they registered for, their age, and possibly more factors (based on how the population voted, registered, or Census data). In addition to quenching a users interest I could imagine a scenario where a user was thinking of moving and wanted to know what areas were fairly well aligned with their political views to help in making the decision. Another scenario would be to quickly find areas in the state that could use more attention during campaign season based off how populations have voted in the past.
– Is your project doable given the constraints of time, our starting knowledge?
This project is doable but likely in a limited fashion. I will take a scaling approach and start with just California split into counties. I’ve still had trouble finding good sources of election result data sets but I have found fairly detailed registration data. This gives me slightly different data to map with but a good amount to work with and build with. If I have to I can just had input state wide vote results to work with the existing data. I’m also sure that my technical ability can keep up with the needs of this project.
– What interface are you imagining? Is it a web, desktop, or mobile application? What platform are you running?
This would be a web based application. A user will be able to select the parameters of the color coded map and then the map will be generated (picture below is edited from a pdf downloaded fromhttp://swdb.berkeley.edu/maps.html to resemble a zoomed in view on a map that was created to measure democrats registered versus republicans registered). If I can get all that running smoothly I would like to add the ability to change options on the fly and have the map recolor itself but that might be beyond my abilities for now.

– What data or services are planning to bring together? Be specific.
I will be bringing together voting, registration, census and other data from different cities, counties and states to be combined to create color coded comparison (temperature) maps.
– What’s your plan for getting the data?
These are files that I can find very separately (pretty much as subdivided as possible) but am still hoping to find in one place. I will also be introducing registration data gained fromhttp://swdb.berkeley.edu/. This registration data will really widen the parameters to create maps based off of. I am currently investigating census data as well as http://www.datamasher.org/ (a recent Sunlight Foundation Apps for America 2 winner) for further data sets.
– Do the APIs you plan to use actually support the functionality that you need in your application? Show how it does so.
This is where most of the complexity of this project will likely come in. I would love it if I could just write a script to grab the information I need, but it looks like I will most likely need to grab the data myself and possibly even parse it myself.
– What programming language do you plan to use?
PHP. I think server side is where I’ll be keeping all my data so that is where I’ll be doing all my computing.
– Break down the project into steps. You can end up changing the steps later, but I want to make sure you have a clear conception on what the steps are.
1) Write script to take data from http://swdb.berkeley.edu/ and return the number hex color code that each district should be colored.
2) Write a script to generate the map based off the resulting colors.
3) Create user interface to select from voter registration data.
4) Find more relevant data to use.
5) Add scripts to generate correct numbers from new data files.
6) Add data to parameters available to users.
7) Repeat from 4 with more data or more states.
– Identify areas of “high risk,” areas that you are uncertain about and/or things that might undermine the entire project. Write about how you are planning to deal with these potential problem areas.
1) I’ll likely have to move to Python if I want to allow for on the fly changing of map parameters.
2) Difficulty in finding the data sets I started out looking for has caused me to start using other data. This data is still interesting but not quite what my original intention was, I would really like to get voting data but hand entry will just prove to time consuming, so if it comes to that I will likely only feature a few options in that category.  This might be solved by allowing users to update data as they see fit, but i would then have to provide an interface to allow user input.

Next Page »