Search This Blog

Saturday, August 15, 2015

Obtaining NYC Pedestrian safety data for Analysis

In this post, I wanted to obtain New York City Safety data for subsequent analysis using R and Hadoop. Even though the data may not be as substantial, I wanted to see what insights we could derive in the process. The first post is focused on downloading the relevant data from NYC websites.

The overall process has the following steps

1. Download the data for pedestrian safety and related data
2. Download the data for other datasets that may be relevant to analyze different trends.

Here is the script that I created to download the data

cd ../data

wget -nc http://www.nyc.gov/html/dot/downloads/misc/injury_yearly_shapefile.zip


wget -nc http://www.nyc.gov/html/dot/downloads/misc/injury_all_monthly_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/fatality_yearly_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/fatality_all_monthly_shapefile.zip

wget -nc http://www.nyc.gov/html/dot/downloads/misc/arterial_slow_zones_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/leading_pedestrian_intervals_signals_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/neighborhood_slow_zones_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/safe_streets_for_seniors_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/speed_bumps_2014_to_2015_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/street_improvement_projects_2013_to_2015_intersections_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/street_improvement_projects_2013_to_2015_corridors_shapefile.zip

wget -nc http://www.nyc.gov/html/dot/downloads/misc/hands_on_safety_demos_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/outreach_schools_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/senior_centers_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/street_team_flyers_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/taxi_and_car_service_trainings_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/town_hall_locations_shapefile.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/workshops_shapefile.zip

wget -nc http://www.nyc.gov/html/dot/downloads/misc/summary_city_council_districts.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/summary_community_districts.zip
wget -nc http://www.nyc.gov/html/dot/downloads/misc/summary_police_precincts.zip

# subway entrances
wget -nc "https://data.cityofnewyork.us/api/geospatial/drex-xx56?method=export&format=Shapefile" -O subway_entrances.zip

#bicycle parking
wget -nc "https://data.cityofnewyork.us/download/qpbf-g2yx/SHAPEFILE"  -O bicycle_parking.zip

#street network changes
wget -nc "https://data.cityofnewyork.us/download/bymk-vktx/XML"  -O street_network_changes.xml

#safe route to schools
wget -nc "https://data.cityofnewyork.us/api/views/pc34-d3sx/rows.csv?accessType=DOWNLOAD"  -O safe_route_to_schools.csv

#through truck routes
wget -nc "https://data.cityofnewyork.us/download/qf28-yqqv/ZIP"  -O through_truck_routes.zip

#local truck routes
wget -nc "https://data.cityofnewyork.us/download/wha9-m3tq/ZIP"  -O local_truck_routes.zip

#realtime speed data
wget -nc "https://data.cityofnewyork.us/download/xsat-x5sa/TXT"  -O realtime_speed_data.txt

#street pavement rating
wget -nc "https://data.cityofnewyork.us/api/geospatial/2cav-chmn?method=export&format=Shapefile"  -O street_pavement_rating.zip

#state assembly districts
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nyad_15b.zip" -O stat_assmbly_dst.zip

#U.S. Congressional Districts
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nyadwi_15b.zip" -O us_cngrssnl_dstrcts.zip

#State Senate Districts
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nycgwi_15b.zip" -O st_snt_dstrcts.zip

#Municipal Court Districts
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nymcwi_15b.zip" -O mncpl_crt_dstrcts.zip

#City Council Districts
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nyccwi_15b.zip" -O cty_cncl_dstrcts.zip

#Election Districts 
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nyedwi_15b.zip" -O elctn_dstrcts.zip

#Borough Boundaries
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nybbwi_15b.zip" -O brgh_bndrs.zip

#Community Districts
wget -nc "http://www.nyc.gov/html/dcp/download/bytes/nycd_15b.zip" -O cmmnty_dstrcts.zip

rm *.zip.*

ls .

mkdir shpfiles

unzip -nj injury_yearly_shapefile.zip -d ./shpfiles
unzip -nj injury_all_monthly_shapefile.zip -d ./shpfiles
unzip -nj fatality_yearly_shapefile.zip -d ./shpfiles
unzip -nj fatality_all_monthly_shapefile.zip -d ./shpfiles
unzip -nj arterial_slow_zones_shapefile.zip -d ./shpfiles
unzip -nj leading_pedestrian_intervals_signals_shapefile.zip -d ./shpfiles
unzip -nj neighborhood_slow_zones_shapefile.zip -d ./shpfiles
unzip -nj safe_streets_for_seniors_shapefile.zip -d ./shpfiles
unzip -nj speed_bumps_2014_to_2015_shapefile.zip -d ./shpfiles
unzip -nj street_improvement_projects_2013_to_2015_intersections_shapefile.zip -d ./shpfiles
unzip -nj street_improvement_projects_2013_to_2015_corridors_shapefile.zip -d ./shpfiles
unzip -nj hands_on_safety_demos_shapefile.zip -d ./shpfiles
unzip -nj outreach_schools_shapefile.zip -d ./shpfiles
unzip -nj senior_centers_shapefile.zip -d ./shpfiles
unzip -nj street_team_flyers_shapefile.zip -d ./shpfiles
unzip -nj taxi_and_car_service_trainings_shapefile.zip -d ./shpfiles
unzip -nj town_hall_locations_shapefile.zip -d ./shpfiles
unzip -nj workshops_shapefile.zip -d ./shpfiles
unzip -nj summary_city_council_districts.zip -d ./shpfiles
unzip -nj summary_community_districts.zip -d ./shpfiles
unzip -nj summary_police_precincts.zip -d ./shpfiles

unzip -nj subway_entrances.zip -d ./shpfiles
unzip -nj bicycle_parking.zip -d ./shpfiles
unzip -nj through_truck_routes.zip -d ./shpfiles
unzip -nj local_truck_routes.zip -d ./shpfiles
unzip -nj street_pavement_rating.zip -d ./shpfiles 

unzip -nj stat_assmbly_dst.zip -d ./shpfiles 
unzip -nj us_cngrssnl_dstrcts.zip -d ./shpfiles 
unzip -nj st_snt_dstrcts.zip -d ./shpfiles 
unzip -nj mncpl_crt_dstrcts.zip -d ./shpfiles 
unzip -nj cty_cncl_dstrcts.zip -d ./shpfiles 
unzip -nj cty_cncl_dstrcts.zip -d ./shpfiles 
unzip -nj elctn_dstrcts.zip -d ./shpfiles 
unzip -nj brgh_bndrs.zip -d ./shpfiles 
unzip -nj cmmnty_dstrcts.zip -d ./shpfiles 

Here is my script in the editor




To execute, just save the file as download_NYC_data.sh in my scripts folder

chmod a+x *.sh
./download_NYC_data.sh

If the file is already downloaded and/or unarchived, the script does not do anything..

Here is the screenshot from my terminal



Just executed the script to download all data and unzip it on my local machine

No comments: