Aug 18, 2022 · Archived
Extracting data from a variety of Databases/APIs
Showcase of data extraction
From earlier work — part of my archive of foundational machine-learning & data-science projects.
Extracting data from databases with APIs
In this post I'll show how to get data from a variety of APIs through Python and how the data can be used to construct pandas dataframes in Python which can then be readily used for subsequent data analysis and visualization. Those last two steps will be the topic of future posts though. APIs enable software components to communicate with each other via a set of definitions and protocols. APIs are a great way to get data from a variety of sources to do research and/or teach yourself things.
Modules Used
APIs sometimes have their own libraries/modules that can facilitate data extraction. In the simplest cases, all you need is an API key and a URL. To get an API key simply go to whatever website you want to get data from and see if they have API keys available. The URL is simply where we are querying from. APIs are generally constructed using JSON files. Finally, tabulating our extracted data can make it easier to get a cursory understanding of it. Hence, the 3 libraries/modules that you will generally need are:- requests
- json
- pandas
Some places have their own libraries/modules specific to their own data. I'll show some examples of how to use them. A great place to find and practice data extraction from APIs is https://rapidapi.com/hub. RapidAPI has a collection of over 7500 APIs that is constantly growing.
In [13]:
#General modules for API extraction
import requests
import json
import pandas as pd
from glom import glom #Can help with extracting deeply nested objects from JSON files
import pprint as pp #Prettifies json output
Getting nutritional information from Nutritionix API
I'll start by going through the process of getting nutritional data from the Nutritionix API. The Nutritionix database can be found here: https://www.nutritionix.com/database. For Nutritionix we need three things to make a query:- API key
- APP ID
- UPC
The API Key and the App ID can be obtained from Nutritionix after you sign up for a develop account. The UPC is a Universal Product Code that identifies a particular product. Here I'll be using a UPC of 706173006046 which corresponds to Natural Value Organic Beans Garbanzo. You can get UPC codes from a variety of sites. I used this site for mine
Once you have those 3 values, the next thing you need is a URL that you can GET a request from. For Nutritionix the URL has the following base structure:
https://api.nutritionix.com/v1_1/item?upc=[UPC]&appId=[APPID]&appKey=[APIKEY]
I like setting up my URLs as formatted strings since it allows me to readily change my query, app_id, and/or api_key if/as needed
In [2]:
#Nutritionix
nix_apikey = '0cfcb1e3abae28de2ec81c3fccb6c7ea'
upc = '706173006046'
nix_appid = 'addc9626'
url = f'https://api.nutritionix.com/v1_1/item?upc={upc}&appId={nix_appid}&appKey={nix_apikey}'
Now that we have those elements, we can start requesting data using requests
In [3]:
r = requests.get(url)
r
Out[3]:
<Response [200]>
If your r value turns out to be 400 or 404 then chances are that the UPC code you entered is invalid. If this doesn't work you may want to try the EAN-13 code instead. Next, let's unpack this data using JSON.
In [4]:
json_dict = r.json()
json_dict
Out[4]:
{'old_api_id': None,
'item_id': '59841cec3695799351031983',
'item_name': 'Garbanzo Beans, Organic',
'leg_loc_id': None,
'brand_id': '51db37b1176fe9790a8984af',
'brand_name': 'Natural Value',
'item_description': None,
'updated_at': '2017-08-04T07:06:33.000Z',
'nf_ingredient_statement': None,
'nf_water_grams': None,
'nf_calories': 130,
'nf_calories_from_fat': 10,
'nf_total_fat': 1,
'nf_saturated_fat': 0,
'nf_trans_fatty_acid': 0,
'nf_polyunsaturated_fat': None,
'nf_monounsaturated_fat': None,
'nf_cholesterol': 0,
'nf_sodium': 90,
'nf_total_carbohydrate': 19,
'nf_dietary_fiber': 4,
'nf_sugars': 1,
'nf_protein': 5,
'nf_vitamin_a_dv': 0,
'nf_vitamin_c_dv': 2,
'nf_calcium_dv': 2,
'nf_iron_dv': 8,
'nf_refuse_pct': None,
'nf_servings_per_container': 3.5,
'nf_serving_size_qty': 0.5,
'nf_serving_size_unit': 'cup',
'nf_serving_weight_grams': 130,
'allergen_contains_milk': None,
'allergen_contains_eggs': None,
'allergen_contains_fish': None,
'allergen_contains_shellfish': None,
'allergen_contains_tree_nuts': None,
'allergen_contains_peanuts': None,
'allergen_contains_wheat': None,
'allergen_contains_soybeans': None,
'allergen_contains_gluten': None,
'usda_fields': None}
Let's take a look at some of the keys we have access to for this product
In [5]:
json_dict.keys()
Out[5]:
dict_keys(['old_api_id', 'item_id', 'item_name', 'leg_loc_id', 'brand_id', 'brand_name', 'item_description', 'updated_at', 'nf_ingredient_statement', 'nf_water_grams', 'nf_calories', 'nf_calories_from_fat', 'nf_total_fat', 'nf_saturated_fat', 'nf_trans_fatty_acid', 'nf_polyunsaturated_fat', 'nf_monounsaturated_fat', 'nf_cholesterol', 'nf_sodium', 'nf_total_carbohydrate', 'nf_dietary_fiber', 'nf_sugars', 'nf_protein', 'nf_vitamin_a_dv', 'nf_vitamin_c_dv', 'nf_calcium_dv', 'nf_iron_dv', 'nf_refuse_pct', 'nf_servings_per_container', 'nf_serving_size_qty', 'nf_serving_size_unit', 'nf_serving_weight_grams', 'allergen_contains_milk', 'allergen_contains_eggs', 'allergen_contains_fish', 'allergen_contains_shellfish', 'allergen_contains_tree_nuts', 'allergen_contains_peanuts', 'allergen_contains_wheat', 'allergen_contains_soybeans', 'allergen_contains_gluten', 'usda_fields'])
Let's access some info from this dictionary
In [6]:
print(json_dict['brand_name'],'\n',
json_dict['nf_calories'],'\n',
json_dict['nf_sodium'],'\n',
json_dict['nf_sugars'],'\n',
json_dict['nf_iron_dv'])
Natural Value 130 90 1 8
Cool! We can get all sorts of info for a variety of foods this way. There is a wrapper for the Nutritionix database that can facilitate operations to get queries in a much more streamlined fasion. We can use it as follows:
In [3]:
from nutritionix import Nutritionix #Nutritionix database for nutrition info
nix = Nutritionix(app_id = nix_appid, api_key = nix_apikey)
results = nix.search('Garbanzo').json()
results
Out[3]:
{'total_hits': 514,
'max_score': 13.36457,
'hits': [{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '513fceb675b8dbbc210021c8',
'_score': 13.36457,
'fields': {'item_id': '513fceb675b8dbbc210021c8',
'item_name': 'Garbanzo Beans - 1 cup',
'brand_name': 'USDA',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '60a7bb52805e4a2d62e7842e',
'_score': 4.7935605,
'fields': {'item_id': '60a7bb52805e4a2d62e7842e',
'item_name': 'Garbanzo',
'brand_name': 'SCN Sales',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '550d8dfb1abac49e69394465',
'_score': 4.62599,
'fields': {'item_id': '550d8dfb1abac49e69394465',
'item_name': 'Garbanzo',
'brand_name': 'The Fresh Market',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '55dc291973184b617abd9eaf',
'_score': 4.580312,
'fields': {'item_id': '55dc291973184b617abd9eaf',
'item_name': 'Garbanzo',
'brand_name': 'Sadaf',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '59dc70293b737c9301da9545',
'_score': 4.580312,
'fields': {'item_id': '59dc70293b737c9301da9545',
'item_name': 'Garbanzo',
'brand_name': 'Montjardin',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '529e7f31f9655f6d350088bd',
'_score': 4.36608,
'fields': {'item_id': '529e7f31f9655f6d350088bd',
'item_name': 'Garbanzo Seasoned Chips',
'brand_name': 'Garbanzo',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '529e7f31ea63d49335008954',
'_score': 4.037829,
'fields': {'item_id': '529e7f31ea63d49335008954',
'item_name': 'Tahini Sauce, Entree Salad',
'brand_name': 'Garbanzo',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '529e7f31ea63d49335008959',
'_score': 4.037829,
'fields': {'item_id': '529e7f31ea63d49335008959',
'item_name': 'White Milk, 1%',
'brand_name': 'Garbanzo',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '529e7f31f9655f6d3500889e',
'_score': 4.037829,
'fields': {'item_id': '529e7f31f9655f6d3500889e',
'item_name': 'Chicken Kabob',
'brand_name': 'Garbanzo',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '23cb91820f531c9059752f15',
'_score': 4.037829,
'fields': {'item_id': '23cb91820f531c9059752f15',
'item_name': 'Rice, Pita',
'brand_name': 'Garbanzo',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}}]}
We can also have it return just the first item as follows:
In [8]:
nix.search('Garbanzo', results="0:1").json()
Out[8]:
{'total_hits': 513,
'max_score': 13.48859,
'hits': [{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'item',
'_id': '513fceb675b8dbbc210021c8',
'_score': 13.48859,
'fields': {'item_id': '513fceb675b8dbbc210021c8',
'item_name': 'Garbanzo Beans - 1 cup',
'brand_name': 'USDA',
'nf_serving_size_qty': 1,
'nf_serving_size_unit': 'serving'}}]}
We can also search by brands by using the following notation
In [9]:
nix.brand().search(query="chipotle").json()
Out[9]:
{'total': 3,
'max_score': 11.98567,
'hits': [{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'brand',
'_id': '513fbc1283aa2dc80c000002',
'_score': 11.98567,
'fields': {'name': 'Chipotle',
'website': 'http://www.chipotle.com',
'type': 1,
'_id': '513fbc1283aa2dc80c000002'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'brand',
'_id': '561c00871485de5737d086dd',
'_score': 7.4692764,
'fields': {'name': 'Chicken Chipotle',
'website': None,
'type': 2,
'_id': '561c00871485de5737d086dd'}},
{'_index': 'f762ef22-e660-434f-9071-a10ea6691c27',
'_type': 'brand',
'_id': '54c751ece1a9773d18e42bd8',
'_score': 7.3735495,
'fields': {'name': "Smokin' Chipotle",
'website': None,
'type': 2,
'_id': '54c751ece1a9773d18e42bd8'}}]}
Let's now load this json data into a pandas dataframe. Since the json data we got is in a nested list format, we need to use the json_normalize function.
In [24]:
results = nix.search('Garbanzo, Apple').json()
print(type(results['hits']))
df = pd.json_normalize(results)
df
<class 'list'>
Out[24]:
| total_hits | max_score | hits | |
|---|---|---|---|
| 0 | 18317 | 2.651146 | [{'_index': 'f762ef22-e660-434f-9071-a10ea6691... |
This is not quite what we want though. However, we can include the record_path parameter as part of our input and set it to the hits column to fix this.
In [30]:
results = nix.search('Garbanzo').json()
df = pd.json_normalize(results, record_path = ['hits'])
df.head()
Out[30]:
| _index | _type | _id | _score | fields.item_id | fields.item_name | fields.brand_name | fields.nf_serving_size_qty | fields.nf_serving_size_unit | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 513fceb675b8dbbc210021c8 | 13.503290 | 513fceb675b8dbbc210021c8 | Garbanzo Beans - 1 cup | USDA | 1 | serving |
| 1 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 60a7bb52805e4a2d62e7842e | 4.844969 | 60a7bb52805e4a2d62e7842e | Garbanzo | SCN Sales | 1 | serving |
| 2 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 550d8dfb1abac49e69394465 | 4.642818 | 550d8dfb1abac49e69394465 | Garbanzo | The Fresh Market | 1 | serving |
| 3 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 55dc291973184b617abd9eaf | 4.625291 | 55dc291973184b617abd9eaf | Garbanzo | Sadaf | 1 | serving |
| 4 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 59dc70293b737c9301da9545 | 4.625291 | 59dc70293b737c9301da9545 | Garbanzo | Montjardin | 1 | serving |
Here's an example of we can process multiple queries programatically
In [33]:
# Make a food list
food = ['waffle', 'apple', 'cottage', 'cheese', 'muffin', 'soup', 'coffee', 'butter', 'ham',
'syrup', 'tortilla','salt', 'pepper', 'butternut', 'walnut', 'cinnamon', 'kale', 'spinach',
'banana', 'bean', 'turkey', 'jam','carrot', 'cracker', 'rice', 'soy', 'tempura', 'sushi', 'orange',
'pasta', 'chicken', 'beef', 'fish', 'lamb','pork', 'broccoli', 'lettuce', 'tomato', 'garlic',
'bacon', 'salad', 'vanilla', 'raspberry','cake', 'beer','green', 'onion', 'vegetable', 'chocolate',
'sugar','chip', 'cookie', 'milk', 'cashew', 'vinegar', 'artichoke','cheddar', 'cauliflower',
'pinto','egg', 'peanut', 'cocoa', 'avocado', 'pomegranate','clementine', 'grape']
#Initiate list to hold generated dataframes
df_list = []
for elem in food:
results = nix.search(elem).json()
df = pd.json_normalize(results, record_path = ['hits'])
df_list.append(df)
final_df = pd.concat(df_list, ignore_index = True)
In [35]:
print(len(food))
final_df
66
Out[35]:
| _index | _type | _id | _score | fields.item_id | fields.item_name | fields.brand_name | fields.nf_serving_size_qty | fields.nf_serving_size_unit | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 513fceb775b8dbbc2100294d | 13.242171 | 513fceb775b8dbbc2100294d | Waffle - 1 waffle, round (7" dia) | USDA | 1 | serving |
| 1 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 5900a1ef7924e49b03571ec5 | 12.982630 | 5900a1ef7924e49b03571ec5 | Waffle - 1 waffle, round (8" dia) | USDA | 1 | serving |
| 2 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 513fceb775b8dbbc2100294e | 12.897008 | 513fceb775b8dbbc2100294e | Waffle - 1 oz | USDA | 1 | serving |
| 3 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 5755d4a47faba81b4d8c87ee | 4.631118 | 5755d4a47faba81b4d8c87ee | Waffle | Kingsmill | 1 | serving |
| 4 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 5701fc820af88e5e7a66f7a1 | 4.509888 | 5701fc820af88e5e7a66f7a1 | Waffle | Giant Eagle | 1 | serving |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 655 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 513fceb575b8dbbc21001052 | 11.694013 | 513fceb575b8dbbc21001052 | Grapes - 1 cup | USDA | 1 | serving |
| 656 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 5d76608c34d5374445dc6fb2 | 11.637632 | 5d76608c34d5374445dc6fb2 | Grapes, raw - 1 oz | USDA | 1 | serving |
| 657 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 5589f586c1aa12d129a28017 | 4.571935 | 5589f586c1aa12d129a28017 | Grape | Funnbar | 1 | serving |
| 658 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 6141fb50615135000990dd85 | 4.571935 | 6141fb50615135000990dd85 | Grape | Torani | 1 | serving |
| 659 | f762ef22-e660-434f-9071-a10ea6691c27 | item | 5bd2bc6fa8cc29736deb742f | 4.158502 | 5bd2bc6fa8cc29736deb742f | Grape | Pollapo | 1 | serving |
660 rows × 9 columns
Awesome! For some reason though I'm only getting at most 10 results from this process even though the query says that there are more... I'll have to investigate that further but it could probably be resolved by submitting queries within a loop to get all the values
Extracting country information from World Bank Data
The World Bank has data that you can find here. All you need is a URL and a properly structured search query using indicators to add to your URL.Once you have that, you can use the same requests that we used before. You can find examples of different API Call Structures for this database here
The world_bank_data module has all the data we need so all it requires is for us to call it. You can get more info here https://pypi.org/project/world-bank-data/. The nice thing about this module is that it places your request into a dataframe so there's no need to do additional coding to get our data ready.
We can get information from a variety of topics like Agriculture, Education, Poverty, and Climate Change as shown below
In [5]:
import world_bank_data as wb #This is the database for the World Bank
wb.get_topics()
Out[5]:
| value | sourceNote | |
|---|---|---|
| id | ||
| 1 | Agriculture & Rural Development | For the 70 percent of the world's poor who liv... |
| 2 | Aid Effectiveness | Aid effectiveness is the impact that aid has i... |
| 3 | Economy & Growth | Economic growth is central to economic develop... |
| 4 | Education | Education is one of the most powerful instrume... |
| 5 | Energy & Mining | The world economy needs ever-increasing amount... |
| 6 | Environment | Natural and man-made environmental resources –... |
| 7 | Financial Sector | An economy's financial markets are critical to... |
| 8 | Health | Improving health is central to the Millennium ... |
| 9 | Infrastructure | Infrastructure helps determine the success of ... |
| 10 | Social Protection & Labor | The supply of labor available in an economy in... |
| 11 | Poverty | For countries with an active poverty monitorin... |
| 12 | Private Sector | Private markets drive economic growth, tapping... |
| 13 | Public Sector | Effective governments improve people's standar... |
| 14 | Science & Technology | Technological innovation, often fueled by gove... |
| 15 | Social Development | Data here cover child labor, gender issues, re... |
| 16 | Urban Development | Cities can be tremendously efficient. It is ea... |
| 17 | Gender | Gender equality is a core development objectiv... |
| 18 | Millenium development goals | |
| 19 | Climate Change | Climate change is expected to hit developing c... |
| 20 | External Debt | Debt statistics provide a detailed picture of ... |
| 21 | Trade | Trade is a key means to fight poverty and achi... |
The WBDB is obtained from a variety of sources. We can get information about what those sources are as shown below.
In [4]:
wb.get_sources()
Out[4]:
| lastupdated | name | code | description | url | dataavailability | metadataavailability | concepts | |
|---|---|---|---|---|---|---|---|---|
| id | ||||||||
| 1 | 2021-08-18 | Doing Business | DBS | Y | Y | 3 | ||
| 2 | 2022-07-20 | World Development Indicators | WDI | Y | Y | 3 | ||
| 3 | 2021-09-27 | Worldwide Governance Indicators | WGI | Y | Y | 3 | ||
| 5 | 2016-03-21 | Subnational Malnutrition Database | SNM | Y | Y | 3 | ||
| 6 | 2022-01-14 | International Debt Statistics | IDS | Y | Y | 4 | ||
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 84 | 2022-07-19 | Education Policy | EDP | Y | Y | 3 | ||
| 85 | 2022-07-06 | PEFA_2021_SNG | SNG | Y | N | 4 | ||
| 86 | 2021-09-24 | Global Jobs Indicators Database (JOIN) | JON | Y | Y | 3 | ||
| 87 | 2022-06-10 | Country Climate and Development Report (CCDR) | CCD | Y | Y | 3 | ||
| 88 | 2022-07-05 | Food Prices for Nutrition | FPN | Y | Y | 4 |
68 rows × 8 columns
We can also get a variety of information about countries
In [32]:
countries = wb.get_countries()
countries
Out[32]:
| iso2Code | name | region | adminregion | incomeLevel | lendingType | capitalCity | longitude | latitude | |
|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||
| ABW | AW | Aruba | Latin America & Caribbean | High income | Not classified | Oranjestad | -70.0167 | 12.5167 | |
| AFE | ZH | Africa Eastern and Southern | Aggregates | Aggregates | Aggregates | NaN | NaN | ||
| AFG | AF | Afghanistan | South Asia | South Asia | Low income | IDA | Kabul | 69.1761 | 34.5228 |
| AFR | A9 | Africa | Aggregates | Aggregates | Aggregates | NaN | NaN | ||
| AFW | ZI | Africa Western and Central | Aggregates | Aggregates | Aggregates | NaN | NaN | ||
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| XZN | A5 | Sub-Saharan Africa excluding South Africa and ... | Aggregates | Aggregates | Aggregates | NaN | NaN | ||
| YEM | YE | Yemen, Rep. | Middle East & North Africa | Middle East & North Africa (excluding high inc... | Low income | IDA | Sana'a | 44.2075 | 15.3520 |
| ZAF | ZA | South Africa | Sub-Saharan Africa | Sub-Saharan Africa (excluding high income) | Upper middle income | IBRD | Pretoria | 28.1871 | -25.7460 |
| ZMB | ZM | Zambia | Sub-Saharan Africa | Sub-Saharan Africa (excluding high income) | Low income | IDA | Lusaka | 28.2937 | -15.3982 |
| ZWE | ZW | Zimbabwe | Sub-Saharan Africa | Sub-Saharan Africa (excluding high income) | Lower middle income | Blend | Harare | 31.0672 | -17.8312 |
299 rows × 9 columns
There's a ton of information we can get from this database via indicators. You can get the list of indicators as shown below.
In [68]:
indicators = wb.get_indicators()
indicators
Out[68]:
| name | unit | source | sourceNote | sourceOrganization | topics | |
|---|---|---|---|---|---|---|
| id | ||||||
| 1.0.HCount.1.90usd | Poverty Headcount ($1.90 a day) | LAC Equity Lab | The poverty headcount index measures the propo... | LAC Equity Lab tabulations of SEDLAC (CEDLAS a... | Poverty | |
| 1.0.HCount.2.5usd | Poverty Headcount ($2.50 a day) | LAC Equity Lab | The poverty headcount index measures the propo... | LAC Equity Lab tabulations of SEDLAC (CEDLAS a... | Poverty | |
| 1.0.HCount.Mid10to50 | Middle Class ($10-50 a day) Headcount | LAC Equity Lab | The poverty headcount index measures the propo... | LAC Equity Lab tabulations of SEDLAC (CEDLAS a... | Poverty | |
| 1.0.HCount.Ofcl | Official Moderate Poverty Rate-National | LAC Equity Lab | The poverty headcount index measures the propo... | LAC Equity Lab tabulations of data from Nation... | Poverty | |
| 1.0.HCount.Poor4uds | Poverty Headcount ($4 a day) | LAC Equity Lab | The poverty headcount index measures the propo... | LAC Equity Lab tabulations of SEDLAC (CEDLAS a... | Poverty | |
| ... | ... | ... | ... | ... | ... | ... |
| WP15163_4.3 | Mobile account, female (% age 15+) | WDI Database Archives | Mobile account denotes the percentage of respo... | Demirguc-Kunt et al., 2015, Global Financial I... | Financial Sector ,Gender | |
| WP15163_4.8 | Mobile account, income, poorest 40% (% ages 15+) | WDI Database Archives | Mobile account denotes the percentage of respo... | Demirguc-Kunt et al., 2015, Global Financial I... | Financial Sector | |
| WP15163_4.9 | Mobile account, income, richest 60% (% ages 15+) | WDI Database Archives | Mobile account denotes the percentage of respo... | Demirguc-Kunt et al., 2015, Global Financial I... | Financial Sector | |
| wpremia_F | Wage Premia for Females (the ratio of skilled ... | Gender Disaggregated Labor Database (GDLD) | ||||
| wpremia_M | Wage Premia for Males (the ratio of skilled m... | Gender Disaggregated Labor Database (GDLD) |
20256 rows × 6 columns
We can get a list of topics that start with a certain letter using the following code:
In [69]:
new_df = indicators.loc[indicators['topics'].str[0].isin(['S']), 'topics']
print(type(new_df))
new_df
<class 'pandas.core.series.Series'>
Out[69]:
id
4.0.nini.15a18 Social Protection & Labor
4.0.nini.15a24 Social Protection & Labor
4.0.nini.19a24 Social Protection & Labor
4.0.stud.15a18 Social Protection & Labor
4.0.stud.15a24 Social Protection & Labor
...
SL.WAG.0714.FE.ZS Social Protection & Labor
SL.WAG.0714.MA.ZS Social Protection & Labor
SL.WAG.0714.ZS Social Protection & Labor
SP.POP.SCIE.RD.P6 Science & Technology
SP.POP.TECH.RD.P6 Science & Technology
Name: topics, Length: 2141, dtype: object
The method above returns a 'pandas.core.series.Series' type object. We can place it into a dataframe again using the to_frame method
In [94]:
final_df = new_df.to_frame()
final_df
Out[94]:
| topics | |
|---|---|
| id | |
| 4.0.nini.15a18 | Social Protection & Labor |
| 4.0.nini.15a24 | Social Protection & Labor |
| 4.0.nini.19a24 | Social Protection & Labor |
| 4.0.stud.15a18 | Social Protection & Labor |
| 4.0.stud.15a24 | Social Protection & Labor |
| ... | ... |
| SL.WAG.0714.FE.ZS | Social Protection & Labor |
| SL.WAG.0714.MA.ZS | Social Protection & Labor |
| SL.WAG.0714.ZS | Social Protection & Labor |
| SP.POP.SCIE.RD.P6 | Science & Technology |
| SP.POP.TECH.RD.P6 | Science & Technology |
2141 rows × 1 columns
We can also search for a particular topic via the following code where I'm looking for indicators associated with Climate Change.
In [92]:
cc = wb.get_indicators(topic=19)
cc.head()
Out[92]:
| name | unit | source | sourceNote | sourceOrganization | topics | |
|---|---|---|---|---|---|---|
| id | ||||||
| AG.LND.AGRI.K2 | Agricultural land (sq. km) | World Development Indicators | Agricultural land refers to the share of land ... | Food and Agriculture Organization, electronic ... | Agriculture & Rural Development ,Climate Change | |
| AG.LND.AGRI.ZS | Agricultural land (% of land area) | World Development Indicators | Agricultural land refers to the share of land ... | Food and Agriculture Organization, electronic ... | Agriculture & Rural Development ,Climate Chan... | |
| AG.LND.ARBL.ZS | Arable land (% of land area) | World Development Indicators | Arable land includes land defined by the FAO a... | Food and Agriculture Organization, electronic ... | Agriculture & Rural Development ,Climate Chan... | |
| AG.LND.EL5M.RU.K2 | Rural land area where elevation is below 5 met... | World Development Indicators | Rural land area below 5m is the total rural la... | Center for International Earth Science Informa... | Agriculture & Rural Development ,Climate Chan... | |
| AG.LND.EL5M.RU.ZS | Rural land area where elevation is below 5 met... | World Development Indicators | Rural land area below 5m is the percentage of ... | Center for International Earth Science Informa... | Agriculture & Rural Development ,Climate Chan... |
To get information about population we can enter the following indicator 'SP.POP.TOTL'
In [86]:
# Population dataset, by the World Bank (most recent value)
population = wb.get_series('SP.POP.TOTL')
population
Out[86]:
Country Series Year
Africa Eastern and Southern Population, total 1960 130836765.0
1961 134159786.0
1962 137614644.0
1963 141202036.0
1964 144920186.0
...
Zimbabwe Population, total 2017 14236599.0
2018 14438812.0
2019 14645473.0
2020 14862927.0
2021 15092171.0
Name: SP.POP.TOTL, Length: 16492, dtype: float64
There's a whole lot more information we can extract and I'll play with this database more in a future post. You can of course always use the standard requests method, just make sure you have a URL and something to search.
Getting music data from Spotify
Spotify has a module called spotipy that allows you to extract all sorts of data about the music in their platform. First thing you'll need is to make a developer account with Spotify and get an API ID and a Client Secret key. You can get started with that here: https://developer.spotify.com/.Once you have that setup, you'll want to also install the spotipy API which you can do using pip install spotipy.
Then we'll import spotipy and set our credentials so that we can start querying the API endpoints as follows:
In [13]:
import spotipy
from spotipy.oauth2 import SpotifyClientCredentials
cid = '4a560a9a37c54f5ca18aff5b645c1736'
secret = '0d5fa5169ea34753ae2c70a4b751db51'
s_uri = 'http://localhost/'
client_credentials_manager = SpotifyClientCredentials(client_id=cid, client_secret=secret)
sp = spotipy.Spotify(client_credentials_manager = client_credentials_manager)
Now we should be ready to start retrieving data from Spotify! There's a variety of ways to extract info, I'll cover a few here but you can also find the docs here: https://spotipy.readthedocs.io/en/master/#getting-started
Let's start by finding album info from a specific artist. Here I'll use The Smashing Pumpkins. To do this all we need is the URI for them in Spotify. If you are on Windows, you can get the URI by:
- Going to the artist spotify page
- Clicking the 3 dots next to the 'FOLLOW/FOLLOWING' button
- Click 'Share'
- When you click it initially, it'll say 'Copy link to artist'. To change this, press the
CTRLkey in your keyboard. The text will now say 'Copy Spotify URI' - Copy that code
For The Smashing Pumpkins, the Spotify URI link is: spotify:artist:40Yq4vzPs9VNUrIBG5Jr2i
Once you have that you can run the following code:
In [18]:
tsp_uri = 'spotify:artist:40Yq4vzPs9VNUrIBG5Jr2i'
results = sp.artist_albums(tsp_uri, album_type='album')
albums = results['items']
while results['next']:
results = sp.next(results)
albums.extend(results['items'])
for album in albums:
print(album['name'])
CYR SHINY AND OH SO BRIGHT, VOL. 1 / LP: NO PAST. NO FUTURE. NO SUN. SHINY AND OH SO BRIGHT, VOL. 1 / LP: NO PAST. NO FUTURE. NO SUN. SHINY AND OH SO BRIGHT, VOL. 1 / LP: NO PAST. NO FUTURE. NO SUN. Monuments to an Elegy Monuments to an Elegy Monuments To An Elegy Oceania Oceania Oceania Oceania Oceania Oceania Machina / The Machines Of God Adore (Super Deluxe) Mellon Collie And The Infinite Sadness (Deluxe Edition) Pisces Iscariot (Deluxe) Siamese Dream (2011 - Remaster) Siamese Dream (Deluxe Edition) Gish (Deluxe Edition)
We can also get album covers for the top 5 tracks using the following method:
In [39]:
import urllib.request
import cv2
import numpy as np
import matplotlib.pyplot as plt
results = sp.artist_top_tracks(tsp_uri)
fig, axs = plt.subplots(1, 5, figsize = (20,20))
i = 0
for track in results['tracks'][:5]:
print('track : ' + track['name'])
print('cover art: ' + track['album']['images'][0]['url'])
#This is just to display the album covers in the output
req = urllib.request.urlopen(track['album']['images'][0]['url'])
arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
img = cv2.imdecode(arr, -1) # 'Load it as it is'
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
axs[i].imshow(img)
axs[i].title.set_text(track['name'])
i += 1
print()
plt.show()
track : 1979 - Remastered 2012 cover art: https://i.scdn.co/image/ab67616d0000b273431ac6e6f393acf475730ec6 track : Bullet With Butterfly Wings - Remastered 2012 cover art: https://i.scdn.co/image/ab67616d0000b273431ac6e6f393acf475730ec6 track : Zero - Remastered 2012 cover art: https://i.scdn.co/image/ab67616d0000b273431ac6e6f393acf475730ec6 track : Tonight, Tonight - Remastered 2012 cover art: https://i.scdn.co/image/ab67616d0000b273431ac6e6f393acf475730ec6 track : Today - 2011 Remaster cover art: https://i.scdn.co/image/ab67616d0000b2735274788f34fc7656d2856dfd
Here I'll show how you can query for multiple artists and find their track names, popularity score, and track IDs. This example is for 1000 artists. sp.search only returns 50 results at most per query, so in order to get more results, the query needs to be enclosed inside a loop.
In [41]:
artist_name = []
track_name = []
popularity = []
track_id = []
for i in range(0,1000,50):
track_results = sp.search(q = 'year:2021', type = 'track', limit = 50, offset = i)
for i, t in enumerate(track_results['tracks']['items']):
artist_name.append(t['artists'][0]['name'])
track_name.append(t['name'])
track_id.append(t['id'])
popularity.append(t['popularity'])
For easier viewing, I'll place the results of our query into a dataframe. The dataframe will have four columns, corresponding to each of the things we queried for
In [48]:
track_dataframe = pd.DataFrame({'artist_name' : artist_name,
'track_name' : track_name,
'track_id' : track_id,
'popularity' : popularity})
track_dataframe
Out[48]:
| artist_name | track_name | track_id | popularity | |
|---|---|---|---|---|
| 0 | Nicky Youre | Sunroof | 4h4QlmocP3IuwYEj2j14p8 | 91 |
| 1 | Dj Biffen | Merry Christmas and Happy New Year 2021 - Radi... | 7h7yPyss4MIBVPXVeF83vg | 18 |
| 2 | Morgan Wallen | Wasted On You | 3cBsEDNhFI9E82vPj3kvi3 | 82 |
| 3 | Elvis Costello & The Attractions | Pump It Up - 2021 Remaster | 3oyc1mIdCBGaU55wX7otqM | 61 |
| 4 | Drake | Knife Talk (with 21 Savage ft. Project Pat) | 2BcMwX1MPV6ZHP4tUT9uq6 | 84 |
| ... | ... | ... | ... | ... |
| 995 | Armin van Buuren | For All Time | 17lMWEUwogW3qRTZipD12p | 12 |
| 996 | Andy Grammer | Lease On Life | 4ufkuONjQMNR2fyXu1bO9w | 69 |
| 997 | Tinlicker | Children | 4BdZyQpNHXlvK4co5qV3So | 18 |
| 998 | Florida Georgia Line | Long Live | 0YfRAYTNTNszGg8VARrSIH | 66 |
| 999 | Funcc. | i still remember toronto | 5jNWFTEa6WYX04E5XRgSVt | 6 |
1000 rows × 4 columns
Cool! Now we can start doing more stuff with the data. I'll do a project with this in a future post.
Getting US Census Data via the Census API
The US Census data has a plethora of information on US related things. You can find the full list of available datasets here: https://api.census.gov/data.html. Here's the documentation for the wrapper https://jtleider.github.io/censusdata/To start we need to determine which tables contain the data we are interested in. We do this by using the search function which takes the following form:
censusdata.variable_info.search(src, year, field, criterion, tabletype='detail')
- src (str) – Census data source
- year (int) – Year of data.
- field (str) – Field in which to search.
- criterion (str or function) – Search criterion.
- tabletype (str, optional) – Type of table from which variables are drawn
This function returns a list of tuples that have names, concepts and labels that match our search criterion.
In this example I'll show how to find the status of science and education in Hudson County, NJ which contains Jersey City.
I'll start by making a query for science on the year 2019 using the AC5 database. You'll have to play with the search ranges a bit to find data that seems interesting/relevant to you.
In [270]:
from census import Census
import censusdata
from us import states
#Show first 10 results of search
censusdata.search('acs5', 2019, 'label', 'science')[128:140]
Out[270]:
[('B24022_053E',
'SEX BY OCCUPATION AND MEDIAN EARNINGS IN THE PAST 12 MONTHS (IN 2019 INFLATION-ADJUSTED DOLLARS) FOR THE FULL-TIME, YEAR-ROUND CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Female:!!Management, business, science, and arts occupations:!!Healthcare practitioners and technical occupations:!!Health diagnosing and treating practitioners and other technical occupations'),
('B24022_054E',
'SEX BY OCCUPATION AND MEDIAN EARNINGS IN THE PAST 12 MONTHS (IN 2019 INFLATION-ADJUSTED DOLLARS) FOR THE FULL-TIME, YEAR-ROUND CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Female:!!Management, business, science, and arts occupations:!!Healthcare practitioners and technical occupations:!!Health technologists and technicians'),
('B24114_028E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Natural sciences managers'),
('B24114_081E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Other mathematical science occupations'),
('B24114_125E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Agricultural and food science technicians'),
('B24114_128E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Environmental science and geoscience technicians'),
('B24114_130E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Social science research assistants'),
('B24114_131E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Other life, physical, and social science technicians'),
('B24115_028E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED MALE POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Natural sciences managers'),
('B24115_081E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED MALE POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Other mathematical science occupations'),
('B24115_125E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED MALE POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Agricultural and food science technicians'),
('B24115_128E',
'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED MALE POPULATION 16 YEARS AND OVER',
'Estimate!!Total:!!Environmental science and geoscience technicians')]
The result labeled as 'B24114_028E' contains 'DETAILED OCCUPATION FOR THE CIVILIAN EMPLOYED POPULATION 16 YEARS AND OVER' which seems like it may be relevant, so I'll save that value for use later.
Then, I'll make a query for education on the year 2019 using the AC5 database. Once again, we'll have to sift through the results to see if we can find something of interest/relevancy to us.
In [262]:
#Show first 10 results of search
censusdata.search('acs5', 2019, 'concept', 'education')[367:380]
Out[262]:
[('B15003_002E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!No schooling completed'),
('B15003_003E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!Nursery school'),
('B15003_004E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!Kindergarten'),
('B15003_005E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!1st grade'),
('B15003_006E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!2nd grade'),
('B15003_007E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!3rd grade'),
('B15003_008E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!4th grade'),
('B15003_009E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!5th grade'),
('B15003_010E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!6th grade'),
('B15003_011E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!7th grade'),
('B15003_012E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!8th grade'),
('B15003_013E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!9th grade'),
('B15003_014E',
'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER',
'Estimate!!Total:!!10th grade')]
The entry labeled under 'B15003_025E' has information on the 'EDUCATIONAL ATTAINMENT FOR THE POPULATION 25 YEARS AND OVER'. This may have some stuff that is of use to us.
Next, let's look at what data each of those reports contain.
In [256]:
censusdata.printtable(censusdata.censustable('acs5', 2019, 'B24114'))
Variable | Table | Label | Type ------------------------------------------------------------------------------------------------------------------- B24114_001E | DETAILED OCCUPATION FOR THE CI | !! Estimate Total: | int B24114_002E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chief executives | int B24114_003E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: General and operations managers | int B24114_004E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Legislators | int B24114_005E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Advertising and promotions manager | int B24114_006E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Marketing managers | int B24114_007E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sales managers | int B24114_008E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Public relations and fundraising m | int B24114_009E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Administrative services managers | int B24114_010E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Facilities managers | int B24114_011E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer and information systems m | int B24114_012E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Financial managers | int B24114_013E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Compensation and benefits managers | int B24114_014E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Human resources managers | int B24114_015E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Training and development managers | int B24114_016E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Industrial production managers | int B24114_017E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Purchasing managers | int B24114_018E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Transportation, storage, and distr | int B24114_019E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Farmers, ranchers, and other agric | int B24114_020E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Construction managers | int B24114_021E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Education and childcare administra | int B24114_022E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Architectural and engineering mana | int B24114_023E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food service managers | int B24114_024E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Funeral home managers | int B24114_025E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Entertainment and recreation manag | int B24114_026E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Lodging managers | int B24114_027E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Medical and health services manage | int B24114_028E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Natural sciences managers | int B24114_029E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Postmasters and mail superintenden | int B24114_030E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Property, real estate, and communi | int B24114_031E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Social and community service manag | int B24114_032E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Emergency management directors | int B24114_033E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Personal service managers, all oth | int B24114_034E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Managers, all other | int B24114_035E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Agents and business managers of ar | int B24114_036E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Buyers and purchasing agents, farm | int B24114_037E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Wholesale and retail buyers, excep | int B24114_038E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Purchasing agents, except wholesal | int B24114_039E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Claims adjusters, appraisers, exam | int B24114_040E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Compliance officers | int B24114_041E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cost estimators | int B24114_042E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Human resources workers | int B24114_043E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Compensation, benefits, and job an | int B24114_044E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Training and development specialis | int B24114_045E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Logisticians | int B24114_046E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Project management specialists | int B24114_047E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Management analysts | int B24114_048E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Meeting, convention, and event pla | int B24114_049E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fundraisers | int B24114_050E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Market research analysts and marke | int B24114_051E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Business operations specialists, a | int B24114_052E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Accountants and auditors | int B24114_053E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Property appraisers and assessors | int B24114_054E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Budget analysts | int B24114_055E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Credit analysts | int B24114_056E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Financial and investment analysts | int B24114_057E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Personal financial advisors | int B24114_058E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Insurance underwriters | int B24114_059E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Financial examiners | int B24114_060E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Credit counselors and loan officer | int B24114_061E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tax examiners and collectors, and | int B24114_062E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tax preparers | int B24114_063E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other financial specialists | int B24114_064E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer and information research | int B24114_065E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer systems analysts | int B24114_066E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Information security analysts | int B24114_067E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer programmers | int B24114_068E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Software developers | int B24114_069E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Software quality assurance analyst | int B24114_070E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Web developers | int B24114_071E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Web and digital interface designer | int B24114_072E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer support specialists | int B24114_073E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Database administrators and archit | int B24114_074E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Network and computer systems admin | int B24114_075E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer network architects | int B24114_076E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer occupations, all other | int B24114_077E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Actuaries | int B24114_078E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Mathematicians | int B24114_079E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Operations research analysts | int B24114_080E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Statisticians | int B24114_081E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other mathematical science occupat | int B24114_082E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Architects, except landscape and n | int B24114_083E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Landscape architects | int B24114_084E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Surveyors, cartographers, and phot | int B24114_085E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Aerospace engineers | int B24114_086E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Agricultural engineers | int B24114_087E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bioengineers and biomedical engine | int B24114_088E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chemical engineers | int B24114_089E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Civil engineers | int B24114_090E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer hardware engineers | int B24114_091E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electrical and electronics enginee | int B24114_092E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Environmental engineers | int B24114_093E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Industrial engineers, including he | int B24114_094E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Marine engineers and naval archite | int B24114_095E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Materials engineers | int B24114_096E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Mechanical engineers | int B24114_097E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Mining and geological engineers, i | int B24114_098E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nuclear engineers | int B24114_099E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Petroleum engineers | int B24114_100E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Engineers, all other | int B24114_101E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Architectural and civil drafters | int B24114_102E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other drafters | int B24114_103E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electrical and electronic engineer | int B24114_104E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other engineering technologists an | int B24114_105E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Surveying and mapping technicians | int B24114_106E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Agricultural and food scientists | int B24114_107E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Biological scientists | int B24114_108E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Conservation scientists and forest | int B24114_109E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Medical scientists | int B24114_110E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Life scientists, all other | int B24114_111E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Astronomers and physicists | int B24114_112E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Atmospheric and space scientists | int B24114_113E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chemists and materials scientists | int B24114_114E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Environmental scientists and speci | int B24114_115E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Geoscientists and hydrologists, ex | int B24114_116E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Physical scientists, all other | int B24114_117E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Economists | int B24114_118E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Survey researchers | int B24114_119E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Clinical and counseling psychologi | int B24114_120E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: School psychologists | int B24114_121E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other psychologists | int B24114_122E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sociologists | int B24114_123E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Urban and regional planners | int B24114_124E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Miscellaneous social scientists an | int B24114_125E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Agricultural and food science tech | int B24114_126E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Biological technicians | int B24114_127E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chemical technicians | int B24114_128E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Environmental science and geoscien | int B24114_129E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nuclear technicians | int B24114_130E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Social science research assistants | int B24114_131E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other life, physical, and social s | int B24114_132E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Occupational health and safety spe | int B24114_133E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Substance abuse and behavioral dis | int B24114_134E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Educational, guidance, and career | int B24114_135E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Marriage and family therapists | int B24114_136E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Mental health counselors | int B24114_137E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Rehabilitation counselors | int B24114_138E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Counselors, all other | int B24114_139E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Child, family, and school social w | int B24114_140E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Healthcare social workers | int B24114_141E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Mental health and substance abuse | int B24114_142E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Social workers, all other | int B24114_143E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Probation officers and correctiona | int B24114_144E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Social and human service assistant | int B24114_145E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other community and social service | int B24114_146E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Clergy | int B24114_147E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Directors, religious activities an | int B24114_148E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Religious workers, all other | int B24114_149E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Lawyers | int B24114_150E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Judicial law clerks | int B24114_151E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Judges, magistrates, and other jud | int B24114_152E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Paralegals and legal assistants | int B24114_153E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Title examiners, abstractors, and | int B24114_154E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Legal support workers, all other | int B24114_155E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Postsecondary teachers | int B24114_156E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Preschool and kindergarten teacher | int B24114_157E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Elementary and middle school teach | int B24114_158E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Secondary school teachers | int B24114_159E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Special education teachers | int B24114_160E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tutors | int B24114_161E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other teachers and instructors | int B24114_162E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Archivists, curators, and museum t | int B24114_163E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Librarians and media collections s | int B24114_164E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Library technicians | int B24114_165E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Teaching assistants | int B24114_166E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other educational instruction and | int B24114_167E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Artists and related workers | int B24114_168E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Commercial and industrial designer | int B24114_169E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fashion designers | int B24114_170E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Floral designers | int B24114_171E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Graphic designers | int B24114_172E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Interior designers | int B24114_173E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Merchandise displayers and window | int B24114_174E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other designers | int B24114_175E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Actors | int B24114_176E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Producers and directors | int B24114_177E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Athletes and sports competitors | int B24114_178E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Coaches and scouts | int B24114_179E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Umpires, referees, and other sport | int B24114_180E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dancers and choreographers | int B24114_181E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Music directors and composers | int B24114_182E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Musicians and singers | int B24114_183E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Disc jockeys, except radio | int B24114_184E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Entertainers and performers, sport | int B24114_185E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Broadcast announcers and radio dis | int B24114_186E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: News analysts, reporters, and jour | int B24114_187E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Public relations specialists | int B24114_188E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Editors | int B24114_189E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Technical writers | int B24114_190E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Writers and authors | int B24114_191E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Interpreters and translators | int B24114_192E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Court reporters and simultaneous c | int B24114_193E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Media and communication workers, a | int B24114_194E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Broadcast, sound, and lighting tec | int B24114_195E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Photographers | int B24114_196E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Television, video, and film camera | int B24114_197E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Media and communication equipment | int B24114_198E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chiropractors | int B24114_199E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dentists | int B24114_200E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dietitians and nutritionists | int B24114_201E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Optometrists | int B24114_202E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pharmacists | int B24114_203E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Emergency medicine physicians | int B24114_204E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Radiologists | int B24114_205E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other physicians | int B24114_206E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Surgeons | int B24114_207E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Physician assistants | int B24114_208E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Podiatrists | int B24114_209E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Audiologists | int B24114_210E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Occupational therapists | int B24114_211E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Physical therapists | int B24114_212E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Radiation therapists | int B24114_213E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Recreational therapists | int B24114_214E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Respiratory therapists | int B24114_215E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Speech-language pathologists | int B24114_216E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Exercise physiologists | int B24114_217E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Therapists, all other | int B24114_218E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Veterinarians | int B24114_219E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Registered nurses | int B24114_220E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nurse anesthetists | int B24114_221E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nurse midwives | int B24114_222E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nurse practitioners | int B24114_223E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Acupuncturists | int B24114_224E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Healthcare diagnosing or treating | int B24114_225E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Clinical laboratory technologists | int B24114_226E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dental hygienists | int B24114_227E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cardiovascular technologists and t | int B24114_228E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Diagnostic medical sonographers | int B24114_229E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Radiologic technologists and techn | int B24114_230E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Magnetic resonance imaging technol | int B24114_231E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nuclear medicine technologists and | int B24114_232E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Emergency medical technicians | int B24114_233E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Paramedics | int B24114_234E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pharmacy technicians | int B24114_235E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Psychiatric technicians | int B24114_236E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Surgical technologists | int B24114_237E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Veterinary technologists and techn | int B24114_238E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dietetic technicians and ophthalmi | int B24114_239E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Licensed practical and licensed vo | int B24114_240E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Medical records specialists | int B24114_241E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Opticians, dispensing | int B24114_242E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Miscellaneous health technologists | int B24114_243E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other healthcare practitioners and | int B24114_244E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Home health aides | int B24114_245E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Personal care aides | int B24114_246E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Nursing assistants | int B24114_247E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Orderlies and psychiatric aides | int B24114_248E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Occupational therapy assistants an | int B24114_249E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Physical therapist assistants and | int B24114_250E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Massage therapists | int B24114_251E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dental assistants | int B24114_252E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Medical assistants | int B24114_253E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Medical transcriptionists | int B24114_254E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pharmacy aides | int B24114_255E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Veterinary assistants and laborato | int B24114_256E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Phlebotomists | int B24114_257E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other healthcare support workers | int B24114_258E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of correcti | int B24114_259E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of police a | int B24114_260E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of firefigh | int B24114_261E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of security | int B24114_262E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of protecti | int B24114_263E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Firefighters | int B24114_264E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fire inspectors | int B24114_265E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bailiffs | int B24114_266E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Correctional officers and jailers | int B24114_267E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Detectives and criminal investigat | int B24114_268E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fish and game wardens | int B24114_269E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Parking enforcement workers | int B24114_270E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Police officers | int B24114_271E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Animal control workers | int B24114_272E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Private detectives and investigato | int B24114_273E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Security guards and gambling surve | int B24114_274E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Crossing guards and flaggers | int B24114_275E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Transportation security screeners | int B24114_276E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: School bus monitors | int B24114_277E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other protective service workers | int B24114_278E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chefs and head cooks | int B24114_279E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of food pre | int B24114_280E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cooks | int B24114_281E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food preparation workers | int B24114_282E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bartenders | int B24114_283E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fast food and counter workers | int B24114_284E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Waiters and waitresses | int B24114_285E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food servers, nonrestaurant | int B24114_286E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dining room and cafeteria attendan | int B24114_287E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dishwashers | int B24114_288E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Hosts and hostesses, restaurant, l | int B24114_289E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food preparation and serving relat | int B24114_290E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of housekee | int B24114_291E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of landscap | int B24114_292E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Janitors and building cleaners | int B24114_293E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Maids and housekeeping cleaners | int B24114_294E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pest control workers | int B24114_295E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Landscaping and groundskeeping wor | int B24114_296E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tree trimmers and pruners | int B24114_297E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other grounds maintenance workers | int B24114_298E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Supervisors of personal care and s | int B24114_299E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Animal trainers | int B24114_300E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Animal caretakers | int B24114_301E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Gambling services workers | int B24114_302E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Ushers, lobby attendants, and tick | int B24114_303E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other entertainment attendants and | int B24114_304E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Embalmers, crematory operators and | int B24114_305E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Morticians, undertakers, and funer | int B24114_306E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Barbers | int B24114_307E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Hairdressers, hairstylists, and co | int B24114_308E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Manicurists and pedicurists | int B24114_309E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Skincare specialists | int B24114_310E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other personal appearance workers | int B24114_311E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Baggage porters, bellhops, and con | int B24114_312E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tour and travel guides | int B24114_313E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Childcare workers | int B24114_314E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Exercise trainers and group fitnes | int B24114_315E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Recreation workers | int B24114_316E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Residential advisors | int B24114_317E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Personal care and service workers, | int B24114_318E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of retail s | int B24114_319E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of non-reta | int B24114_320E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cashiers | int B24114_321E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Counter and rental clerks | int B24114_322E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Parts salespersons | int B24114_323E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Retail salespersons | int B24114_324E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Advertising sales agents | int B24114_325E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Insurance sales agents | int B24114_326E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Securities, commodities, and finan | int B24114_327E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Travel agents | int B24114_328E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sales representatives of services, | int B24114_329E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sales representatives, wholesale a | int B24114_330E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Models, demonstrators, and product | int B24114_331E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Real estate brokers and sales agen | int B24114_332E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sales engineers | int B24114_333E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Telemarketers | int B24114_334E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Door-to-door sales workers, news a | int B24114_335E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sales and related workers, all oth | int B24114_336E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of office a | int B24114_337E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Switchboard operators, including a | int B24114_338E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Telephone operators | int B24114_339E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Communications equipment operators | int B24114_340E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bill and account collectors | int B24114_341E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Billing and posting clerks | int B24114_342E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bookkeeping, accounting, and audit | int B24114_343E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Gambling cage workers | int B24114_344E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Payroll and timekeeping clerks | int B24114_345E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Procurement clerks | int B24114_346E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tellers | int B24114_347E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Financial clerks, all other | int B24114_348E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Brokerage clerks | int B24114_349E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Correspondence clerks | int B24114_350E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Court, municipal, and license cler | int B24114_351E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Credit authorizers, checkers, and | int B24114_352E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Customer service representatives | int B24114_353E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Eligibility interviewers, governme | int B24114_354E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: File clerks | int B24114_355E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Hotel, motel, and resort desk cler | int B24114_356E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Interviewers, except eligibility a | int B24114_357E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Library assistants, clerical | int B24114_358E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Loan interviewers and clerks | int B24114_359E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: New accounts clerks | int B24114_360E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Order clerks | int B24114_361E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Human resources assistants, except | int B24114_362E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Receptionists and information cler | int B24114_363E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Reservation and transportation tic | int B24114_364E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Information and record clerks, all | int B24114_365E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cargo and freight agents | int B24114_366E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Couriers and messengers | int B24114_367E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Public safety telecommunicators | int B24114_368E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dispatchers, except police, fire, | int B24114_369E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Meter readers, utilities | int B24114_370E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Postal service clerks | int B24114_371E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Postal service mail carriers | int B24114_372E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Postal service mail sorters, proce | int B24114_373E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Production, planning, and expediti | int B24114_374E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Shipping, receiving, and inventory | int B24114_375E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Weighers, measurers, checkers, and | int B24114_376E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Executive secretaries and executiv | int B24114_377E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Legal secretaries and administrati | int B24114_378E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Medical secretaries and administra | int B24114_379E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Secretaries and administrative ass | int B24114_380E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Data entry keyers | int B24114_381E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Word processors and typists | int B24114_382E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Desktop publishers | int B24114_383E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Insurance claims and policy proces | int B24114_384E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Mail clerks and mail machine opera | int B24114_385E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Office clerks, general | int B24114_386E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Office machine operators, except c | int B24114_387E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Proofreaders and copy markers | int B24114_388E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Statistical assistants | int B24114_389E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Office and administrative support | int B24114_390E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of farming, | int B24114_391E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Agricultural inspectors | int B24114_392E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Animal breeders | int B24114_393E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Graders and sorters, agricultural | int B24114_394E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Miscellaneous agricultural workers | int B24114_395E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fishing and hunting workers | int B24114_396E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Forest and conservation workers | int B24114_397E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Logging workers | int B24114_398E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of construc | int B24114_399E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Boilermakers | int B24114_400E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Brickmasons, blockmasons, and ston | int B24114_401E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Carpenters | int B24114_402E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Carpet, floor, and tile installers | int B24114_403E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cement masons, concrete finishers, | int B24114_404E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Construction laborers | int B24114_405E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Construction equipment operators | int B24114_406E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Drywall installers, ceiling tile i | int B24114_407E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electricians | int B24114_408E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Glaziers | int B24114_409E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Insulation workers | int B24114_410E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Painters and paperhangers | int B24114_411E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pipelayers | int B24114_412E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Plumbers, pipefitters, and steamfi | int B24114_413E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Plasterers and stucco masons | int B24114_414E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Reinforcing iron and rebar workers | int B24114_415E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Roofers | int B24114_416E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sheet metal workers | int B24114_417E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Structural iron and steel workers | int B24114_418E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Solar photovoltaic installers | int B24114_419E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Helpers, construction trades | int B24114_420E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Construction and building inspecto | int B24114_421E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Elevator and escalator installers | int B24114_422E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Fence erectors | int B24114_423E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Hazardous materials removal worker | int B24114_424E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Highway maintenance workers | int B24114_425E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Rail-track laying and maintenance | int B24114_426E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Septic tank servicers and sewer pi | int B24114_427E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Miscellaneous construction and rel | int B24114_428E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Derrick, rotary drill, and service | int B24114_429E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Excavating and loading machine and | int B24114_430E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Earth drillers, except oil and gas | int B24114_431E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Explosives workers, ordnance handl | int B24114_432E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Underground mining machine operato | int B24114_433E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Roustabouts, oil and gas | int B24114_434E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other extraction workers | int B24114_435E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of mechanic | int B24114_436E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer, automated teller, and of | int B24114_437E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Radio and telecommunications equip | int B24114_438E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Avionics technicians | int B24114_439E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electric motor, power tool, and re | int B24114_440E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electrical and electronics install | int B24114_441E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electrical and electronics repaire | int B24114_442E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electronic equipment installers an | int B24114_443E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Audiovisual equipment installers a | int B24114_444E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Security and fire alarm systems in | int B24114_445E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Aircraft mechanics and service tec | int B24114_446E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Automotive body and related repair | int B24114_447E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Automotive glass installers and re | int B24114_448E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Automotive service technicians and | int B24114_449E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bus and truck mechanics and diesel | int B24114_450E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Heavy vehicle and mobile equipment | int B24114_451E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Small engine mechanics | int B24114_452E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Miscellaneous vehicle and mobile e | int B24114_453E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Control and valve installers and r | int B24114_454E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Heating, air conditioning, and ref | int B24114_455E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Home appliance repairers | int B24114_456E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Industrial and refractory machiner | int B24114_457E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Maintenance and repair workers, ge | int B24114_458E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Maintenance workers, machinery | int B24114_459E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Millwrights | int B24114_460E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electrical power-line installers a | int B24114_461E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Telecommunications line installers | int B24114_462E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Precision instrument and equipment | int B24114_463E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Wind turbine service technicians | int B24114_464E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Coin, vending, and amusement machi | int B24114_465E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Commercial divers | int B24114_466E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Locksmiths and safe repairers | int B24114_467E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Manufactured building and mobile h | int B24114_468E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Riggers | int B24114_469E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Helpers--installation, maintenance | int B24114_470E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other installation, maintenance, a | int B24114_471E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: First-line supervisors of producti | int B24114_472E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Aircraft structure, surfaces, rigg | int B24114_473E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Electrical, electronics, and elect | int B24114_474E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Engine and other machine assembler | int B24114_475E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Structural metal fabricators and f | int B24114_476E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other assemblers and fabricators | int B24114_477E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bakers | int B24114_478E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Butchers and other meat, poultry, | int B24114_479E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food and tobacco roasting, baking, | int B24114_480E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food batchmakers | int B24114_481E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food cooking machine operators and | int B24114_482E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Food processing workers, all other | int B24114_483E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Computer numerically controlled to | int B24114_484E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Forming machine setters, operators | int B24114_485E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cutting, punching, and press machi | int B24114_486E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Grinding, lapping, polishing, and | int B24114_487E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other machine tool setters, operat | int B24114_488E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Machinists | int B24114_489E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Metal furnace operators, tenders, | int B24114_490E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Model makers and patternmakers, me | int B24114_491E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Molders and molding machine setter | int B24114_492E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tool and die makers | int B24114_493E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Welding, soldering, and brazing wo | int B24114_494E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other metal workers and plastic wo | int B24114_495E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Prepress technicians and workers | int B24114_496E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Printing press operators | int B24114_497E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Print binding and finishing worker | int B24114_498E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Laundry and dry-cleaning workers | int B24114_499E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pressers, textile, garment, and re | int B24114_500E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sewing machine operators | int B24114_501E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Shoe and leather workers | int B24114_502E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tailors, dressmakers, and sewers | int B24114_503E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Textile machine setters, operators | int B24114_504E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Upholsterers | int B24114_505E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other textile, apparel, and furnis | int B24114_506E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cabinetmakers and bench carpenters | int B24114_507E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Furniture finishers | int B24114_508E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sawing machine setters, operators, | int B24114_509E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Woodworking machine setters, opera | int B24114_510E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other woodworkers | int B24114_511E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Power plant operators, distributor | int B24114_512E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Stationary engineers and boiler op | int B24114_513E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Water and wastewater treatment pla | int B24114_514E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Miscellaneous plant and system ope | int B24114_515E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Chemical processing machine setter | int B24114_516E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Crushing, grinding, polishing, mix | int B24114_517E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cutting workers | int B24114_518E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Extruding, forming, pressing, and | int B24114_519E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Furnace, kiln, oven, drier, and ke | int B24114_520E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Inspectors, testers, sorters, samp | int B24114_521E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Jewelers and precious stone and me | int B24114_522E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Dental and ophthalmic laboratory t | int B24114_523E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Packaging and filling machine oper | int B24114_524E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Painting workers | int B24114_525E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Photographic process workers and p | int B24114_526E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Adhesive bonding machine operators | int B24114_527E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Etchers and engravers | int B24114_528E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Molders, shapers, and casters, exc | int B24114_529E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Paper goods machine setters, opera | int B24114_530E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Tire builders | int B24114_531E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Helpers--production workers | int B24114_532E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other production equipment operato | int B24114_533E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other production workers | int B24114_534E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Supervisors of transportation and | int B24114_535E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Aircraft pilots and flight enginee | int B24114_536E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Air traffic controllers and airfie | int B24114_537E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Flight attendants | int B24114_538E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Ambulance drivers and attendants, | int B24114_539E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bus drivers, school | int B24114_540E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Bus drivers, transit and intercity | int B24114_541E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Driver/sales workers and truck dri | int B24114_542E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Shuttle drivers and chauffeurs | int B24114_543E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Taxi drivers | int B24114_544E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Motor vehicle operators, all other | int B24114_545E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Locomotive engineers and operators | int B24114_546E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Railroad conductors and yardmaster | int B24114_547E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other rail transportation workers | int B24114_548E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Sailors and marine oilers | int B24114_549E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Ship and boat captains and operato | int B24114_550E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Ship engineers | int B24114_551E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Parking attendants | int B24114_552E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Transportation service attendants | int B24114_553E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Transportation inspectors | int B24114_554E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Passenger attendants | int B24114_555E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other transportation workers | int B24114_556E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Crane and tower operators | int B24114_557E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Conveyor, dredge, and hoist and wi | int B24114_558E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Industrial truck and tractor opera | int B24114_559E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Cleaners of vehicles and equipment | int B24114_560E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Laborers and freight, stock, and m | int B24114_561E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Machine feeders and offbearers | int B24114_562E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Packers and packagers, hand | int B24114_563E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Stockers and order fillers | int B24114_564E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Pumping station operators | int B24114_565E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Refuse and recyclable material col | int B24114_566E | DETAILED OCCUPATION FOR THE CI | !! !! Estimate Total: Other material moving workers | int -------------------------------------------------------------------------------------------------------------------
In [66]:
censusdata.printtable(censusdata.censustable('acs5', 2019, 'B15003'))
Variable | Table | Label | Type ------------------------------------------------------------------------------------------------------------------- B15003_001E | EDUCATIONAL ATTAINMENT FOR THE | !! Estimate Total: | int B15003_002E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: No schooling completed | int B15003_003E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Nursery school | int B15003_004E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Kindergarten | int B15003_005E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 1st grade | int B15003_006E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 2nd grade | int B15003_007E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 3rd grade | int B15003_008E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 4th grade | int B15003_009E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 5th grade | int B15003_010E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 6th grade | int B15003_011E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 7th grade | int B15003_012E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 8th grade | int B15003_013E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 9th grade | int B15003_014E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 10th grade | int B15003_015E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 11th grade | int B15003_016E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: 12th grade, no diploma | int B15003_017E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Regular high school diploma | int B15003_018E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: GED or alternative credential | int B15003_019E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Some college, less than 1 year | int B15003_020E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Some college, 1 or more years, no | int B15003_021E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Associate's degree | int B15003_022E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Bachelor's degree | int B15003_023E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Master's degree | int B15003_024E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Professional school degree | int B15003_025E | EDUCATIONAL ATTAINMENT FOR THE | !! !! Estimate Total: Doctorate degree | int -------------------------------------------------------------------------------------------------------------------
These two reports may have something of use. The first report contains information about a variety of occupations held by civilians while the second report has information about the education levels of civilians.
Now that we have the variables we want to explore (i.e. position held and education level) we can start getting more geographic specific info. To do that we use censusdata.censusgeo.censusgeo(geo, name='') which needs two inputs:
- geo - Tuple of 2-tuples of the form (geographic component, identifier). Geographic component can be a state or a county. Identifier is a numeric code or wildcard (*) that identifies the geographic component
- name (str, optional) – Name of geography (e.g., ‘Alabama’).
We'll also need the censusdata.download.geographies(within, src, year, key=None, endpt='') function to download that data. Let's find out what the code is for New Jersey by looking up the state data using the wildcard for names.
In [67]:
censusdata.geographies(censusdata.censusgeo([('state', '*')]), 'acs5', 2019)
Out[67]:
{'Alabama': censusgeo((('state', '01'),)),
'Alaska': censusgeo((('state', '02'),)),
'Arizona': censusgeo((('state', '04'),)),
'Arkansas': censusgeo((('state', '05'),)),
'California': censusgeo((('state', '06'),)),
'Colorado': censusgeo((('state', '08'),)),
'Delaware': censusgeo((('state', '10'),)),
'District of Columbia': censusgeo((('state', '11'),)),
'Connecticut': censusgeo((('state', '09'),)),
'Florida': censusgeo((('state', '12'),)),
'Georgia': censusgeo((('state', '13'),)),
'Idaho': censusgeo((('state', '16'),)),
'Hawaii': censusgeo((('state', '15'),)),
'Illinois': censusgeo((('state', '17'),)),
'Indiana': censusgeo((('state', '18'),)),
'Iowa': censusgeo((('state', '19'),)),
'Kansas': censusgeo((('state', '20'),)),
'Kentucky': censusgeo((('state', '21'),)),
'Louisiana': censusgeo((('state', '22'),)),
'Maine': censusgeo((('state', '23'),)),
'Maryland': censusgeo((('state', '24'),)),
'Massachusetts': censusgeo((('state', '25'),)),
'Michigan': censusgeo((('state', '26'),)),
'Minnesota': censusgeo((('state', '27'),)),
'Mississippi': censusgeo((('state', '28'),)),
'Missouri': censusgeo((('state', '29'),)),
'Montana': censusgeo((('state', '30'),)),
'Nebraska': censusgeo((('state', '31'),)),
'Nevada': censusgeo((('state', '32'),)),
'New Hampshire': censusgeo((('state', '33'),)),
'New Jersey': censusgeo((('state', '34'),)),
'New Mexico': censusgeo((('state', '35'),)),
'New York': censusgeo((('state', '36'),)),
'North Carolina': censusgeo((('state', '37'),)),
'North Dakota': censusgeo((('state', '38'),)),
'Ohio': censusgeo((('state', '39'),)),
'Oklahoma': censusgeo((('state', '40'),)),
'Oregon': censusgeo((('state', '41'),)),
'Pennsylvania': censusgeo((('state', '42'),)),
'Rhode Island': censusgeo((('state', '44'),)),
'South Carolina': censusgeo((('state', '45'),)),
'South Dakota': censusgeo((('state', '46'),)),
'Tennessee': censusgeo((('state', '47'),)),
'Texas': censusgeo((('state', '48'),)),
'Vermont': censusgeo((('state', '50'),)),
'Utah': censusgeo((('state', '49'),)),
'Virginia': censusgeo((('state', '51'),)),
'Washington': censusgeo((('state', '53'),)),
'West Virginia': censusgeo((('state', '54'),)),
'Wisconsin': censusgeo((('state', '55'),)),
'Wyoming': censusgeo((('state', '56'),)),
'Puerto Rico': censusgeo((('state', '72'),))}
We find that entry for is 'New Jersey': censusgeo((('state', '34'),)), which means the code we need to enter on censusgeo is ('state', '34').
Next, let's find Hudson County. We'll use a similar process as before but include, 'county' as part of our search.
In [68]:
censusdata.geographies(censusdata.censusgeo([('state', '34'), ('county', '*')]), 'acs5', 2019)
Out[68]:
{'Monmouth County, New Jersey': censusgeo((('state', '34'), ('county', '025'))),
'Sussex County, New Jersey': censusgeo((('state', '34'), ('county', '037'))),
'Essex County, New Jersey': censusgeo((('state', '34'), ('county', '013'))),
'Gloucester County, New Jersey': censusgeo((('state', '34'), ('county', '015'))),
'Passaic County, New Jersey': censusgeo((('state', '34'), ('county', '031'))),
'Bergen County, New Jersey': censusgeo((('state', '34'), ('county', '003'))),
'Warren County, New Jersey': censusgeo((('state', '34'), ('county', '041'))),
'Burlington County, New Jersey': censusgeo((('state', '34'), ('county', '005'))),
'Mercer County, New Jersey': censusgeo((('state', '34'), ('county', '021'))),
'Cumberland County, New Jersey': censusgeo((('state', '34'), ('county', '011'))),
'Somerset County, New Jersey': censusgeo((('state', '34'), ('county', '035'))),
'Ocean County, New Jersey': censusgeo((('state', '34'), ('county', '029'))),
'Atlantic County, New Jersey': censusgeo((('state', '34'), ('county', '001'))),
'Middlesex County, New Jersey': censusgeo((('state', '34'), ('county', '023'))),
'Salem County, New Jersey': censusgeo((('state', '34'), ('county', '033'))),
'Cape May County, New Jersey': censusgeo((('state', '34'), ('county', '009'))),
'Camden County, New Jersey': censusgeo((('state', '34'), ('county', '007'))),
'Union County, New Jersey': censusgeo((('state', '34'), ('county', '039'))),
'Hudson County, New Jersey': censusgeo((('state', '34'), ('county', '017'))),
'Hunterdon County, New Jersey': censusgeo((('state', '34'), ('county', '019'))),
'Morris County, New Jersey': censusgeo((('state', '34'), ('county', '027')))}
Nice! Here are all the counties of New Jersey and now we can get the information specific to Hudson County 'Hudson County, New Jersey': censusgeo((('state', '34'), ('county', '017')))' by adding ('county', '017') to our search.
Now that we have the state and county identifiers, we can extract the specific data blocks that we want from each dataset.
I'll extract the following 5 blocks from the B15003 report: B15003_017E, B15003_021E, B15003_022E, B15003_023E, and B15003_025E which correspond to education levels of regular high school diploma, associates degree, bachelors degree, masters degree and doctorate degree.
I'll extract the following 5 blocks from the B24114 report: B24114_017E, B24114_021E, B24114_022E, B24114_023E, and B24114_025E which correspond to employment positions of Chief executives , legislators, information security analysts, library technicians, and firefighters.
In [71]:
cookbg = censusdata.download('acs5', 2019,
censusdata.censusgeo([('state', '34'), ('county', '017'), ('block group', '*')]),
['B15003_017E', 'B15003_021E', 'B15003_022E', 'B15003_023E', 'B15003_025E',
'B24114_002E', 'B24114_004E', 'B24114_066E', 'B24114_164E', 'B24114_263E'])
cookbg.head()
Out[71]:
| B15003_017E | B15003_021E | B15003_022E | B15003_023E | B15003_025E | B24114_002E | B24114_004E | B24114_066E | B24114_164E | B24114_263E | |
|---|---|---|---|---|---|---|---|---|---|---|
| Block Group 3, Census Tract 199, Hudson County, New Jersey: Summary level: 150, state:34> county:017> tract:019900> block group:3 | 179 | 33 | 95 | 76 | 15 | None | None | None | None | None |
| Block Group 2, Census Tract 198, Hudson County, New Jersey: Summary level: 150, state:34> county:017> tract:019800> block group:2 | 554 | 113 | 579 | 242 | 61 | None | None | None | None | None |
| Block Group 1, Census Tract 198, Hudson County, New Jersey: Summary level: 150, state:34> county:017> tract:019800> block group:1 | 461 | 126 | 846 | 538 | 0 | None | None | None | None | None |
| Block Group 1, Census Tract 158.02, Hudson County, New Jersey: Summary level: 150, state:34> county:017> tract:015802> block group:1 | 721 | 115 | 617 | 127 | 0 | None | None | None | None | None |
| Block Group 3, Census Tract 158.02, Hudson County, New Jersey: Summary level: 150, state:34> county:017> tract:015802> block group:3 | 581 | 65 | 95 | 72 | 0 | None | None | None | None | None |
Great! Now we have this data for further processing which I'll probably use in a future post.
Getting COVID Data from The Covid Tracking Project
Getting COVID data is quite easy from the COVID Tracking project. All we need is a URL which ww can get fromt their website: https://covidtracking.com/data/api
For instance, if we want the historic US values we can do the following query which will give us data from Janury 13,2020 until March 7,2021:
In [88]:
url = 'https://api.covidtracking.com/v1/us/daily.json'
r = requests.get(url)
daily_US = r.json()
In [77]:
url = 'https://api.covidtracking.com/v1/us/current.json'
r = requests.get(url)
current_US = r.json()
current_US
Out[77]:
[{'date': 20210307,
'states': 56,
'positive': 28756489,
'negative': 74582825,
'pending': 11808,
'hospitalizedCurrently': 40199,
'hospitalizedCumulative': 776361,
'inIcuCurrently': 8134,
'inIcuCumulative': 45475,
'onVentilatorCurrently': 2802,
'onVentilatorCumulative': 4281,
'dateChecked': '2021-03-07T24:00:00Z',
'death': 515151,
'hospitalized': 776361,
'totalTestResults': 363825123,
'lastModified': '2021-03-07T24:00:00Z',
'recovered': None,
'total': 0,
'posNeg': 0,
'deathIncrease': 842,
'hospitalizedIncrease': 726,
'negativeIncrease': 131835,
'positiveIncrease': 41835,
'totalTestResultsIncrease': 1170059,
'hash': 'a80d0063822e251249fd9a44730c49cb23defd83'}]
We can get data for a specific state by writing a query of the following format:
https://api.covidtracking.com/v1/states/{state}/info.json
In [87]:
state = 'ny'
url = f'https://api.covidtracking.com/v1/states/{state}/daily.json'
r = requests.get(url)
nj_data = r.json()
We can also get COVID data for all states using the following URL: https://api.covidtracking.com/v1/states/daily.json.
Placing the result into a pandas dataframe only requires us to use theDataFrame function on our query.
In [86]:
df = pd.DataFrame(nj_data)
df
Out[86]:
| date | state | positive | probableCases | negative | pending | totalTestResultsSource | totalTestResults | hospitalizedCurrently | hospitalizedCumulative | ... | dataQualityGrade | deathIncrease | hospitalizedIncrease | hash | commercialScore | negativeRegularScore | negativeScore | positiveScore | score | grade | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 20210307 | NY | 1681169 | None | None | NaN | totalTestEncountersViral | 39695100 | 4789.0 | None | ... | None | 59 | 0 | 8b1ad6376deb03844d4e5e3f615197da1b3a214e | 0 | 0 | 0 | 0 | 0 | |
| 1 | 20210306 | NY | 1674380 | None | None | NaN | totalTestEncountersViral | 39467332 | 4954.0 | None | ... | None | 79 | 0 | c7ae8ab7600d0db452429e5692968b185e90d1de | 0 | 0 | 0 | 0 | 0 | |
| 2 | 20210305 | NY | 1666733 | None | None | NaN | totalTestEncountersViral | 39194200 | 5034.0 | None | ... | None | 95 | 0 | 99bf166f33908a152c9badb3464d1899821323a5 | 0 | 0 | 0 | 0 | 0 | |
| 3 | 20210304 | NY | 1657777 | None | None | NaN | totalTestEncountersViral | 38897265 | 5177.0 | None | ... | None | 61 | 0 | 39d0cec8753202639bdd32abdd228cc6c891d862 | 0 | 0 | 0 | 0 | 0 | |
| 4 | 20210303 | NY | 1650184 | None | None | NaN | totalTestEncountersViral | 38627176 | 5323.0 | None | ... | None | 75 | 0 | 91cc3a4e0e57073b8997de61de8a6e385cf66599 | 0 | 0 | 0 | 0 | 0 | |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 366 | 20200306 | NY | 25 | None | None | 236.0 | totalTestEncountersViral | 122 | NaN | None | ... | None | 0 | 0 | 9e99ccc04e6d33397cc412ee1b0fbecddbaeecd8 | 0 | 0 | 0 | 0 | 0 | |
| 367 | 20200305 | NY | 3 | None | None | 24.0 | totalTestEncountersViral | 30 | NaN | None | ... | None | 0 | 0 | 04e82e80db72556a73999ead493c993e90b8969e | 0 | 0 | 0 | 0 | 0 | |
| 368 | 20200304 | NY | 1 | None | None | 24.0 | totalTestEncountersViral | 10 | NaN | None | ... | None | 0 | 0 | 3feb3bdb879194f81d11c8bbf994d8e3ea2a5252 | 0 | 0 | 0 | 0 | 0 | |
| 369 | 20200303 | NY | 1 | None | None | NaN | totalTestEncountersViral | 1 | NaN | None | ... | None | 0 | 0 | 9e3daed71a696c0c3fa30150ee6423a0a94de8f5 | 0 | 0 | 0 | 0 | 0 | |
| 370 | 20200302 | NY | 0 | None | None | NaN | totalTestEncountersViral | 0 | NaN | None | ... | None | 0 | 0 | b348e3592c0b30cb944b37ddbc88044ceeeadfc9 | 0 | 0 | 0 | 0 | 0 |
371 rows × 56 columns
I'm sure there's plenty of interesting stuff to analyze here for a future project and generate interesting time-series visualizations.
Getting Space Data from NASA
We can get information about Mars Rovers, Asteroids, Astronomy Picture of the Day, Exoplanet Archive, and Earth Polychromatic Imaging Camera (EPIC) amongst many other things from their API. You can register for an API quite easily here: https://api.nasa.gov/
In [249]:
api_key = 'uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'
I'm going to start by retrieving the Astronomy Picture of the Day (APOD). I'll write a little function that will take a date string in YYYY-MM-DD format, the NASA api_key, and string bool to determine whether we are extracting the HD version of the APOD. This function will output information about the APOD and display the image.
In [29]:
import urllib.request
import cv2
import numpy as np
import matplotlib.pyplot as plt
def fetchAPOD(api_key, date, hd):
URL_APOD = "https://api.nasa.gov/planetary/apod"
params = {
'api_key':api_key,
'date':date,
'hd': hd
}
response = requests.get(URL_APOD,params=params).json()
pp.pprint(response)
path2apod = response['url']
#This is just to display the album covers in the output
fig = plt.figure(figsize=(10,10))
req = urllib.request.urlopen(path2apod)
arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
img = cv2.imdecode(arr, -1) # 'Load it as it is'
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img)
plt.show()
fetchAPOD(api_key, '2005-09-12', 'True')
{'copyright': 'Gemini Obs.',
'date': '2005-09-12',
'explanation': 'Is this one galaxy or two? The jumble of stars, gas, and '
'dust that is NGC 520 is now thought to incorporate the '
'remains of two separate galaxies. A combination of '
'observations and simulations indicate the NGC 520 is actually '
'the collision of two disk galaxies. Interesting features of '
'NGC 520 include an unfamiliar looking tail of stars at the '
'image bottom and a perhaps more familiar looking band of dust '
'running diagonally across the image center. A similar '
'looking collision might be expected were our disk Milky Way '
'Galaxy to collide with our large galactic neighbor Andromeda '
'(M31). The collision that defines NGC 520 started about 300 '
'million years ago and continues today. Although the speeds '
'of stars are fast, the distances are so vast that the '
'interacting pair will surely not change its shape noticeably '
'during our lifetimes. NGC 520, at visual magnitude 12, has '
'been noted to be one of the brightest interacting galaxies on '
'the sky, after interacting pairs of galaxies known as the '
'Antennae. NGC 520 was imaged above in spectacular fashion by '
'the Gemini Observatory in Hawaii, USA. Also known as Arp 157, '
'NGC 520 lies about 100 million light years distant, spans '
'about 100 thousand light years, and can be seen with a small '
'telescope toward the constellation of the Fish (Pisces).',
'hdurl': 'https://apod.nasa.gov/apod/image/0509/ngc520_gemini_big.jpg',
'media_type': 'image',
'service_version': 'v1',
'title': 'The Colliding Galaxies of NGC 520',
'url': 'https://apod.nasa.gov/apod/image/0509/ngc520_gemini.jpg'}
I think I'll write a little project for fun to randomly generate an APOD sometime soon :]
Next, I'll look into the Asteroid Near Earth Objects (ANEO) API. This API only allows data to be collected over a maximum span of 7 days. However, we could readily extract more data by enclosing our queries in a for loop, similar to how it was done with Spotify.
In [250]:
def fetchANEOWS(api_key,start_date,end_date):
URL_NeoFeed = "https://api.nasa.gov/neo/rest/v1/feed"
params = {
'api_key':api_key,
'start_date':start_date,
'end_date':end_date
}
response = requests.get(URL_NeoFeed,params=params).json()
pp.pprint(response)
return response
aneo = fetchANEOWS(api_key,'2022-08-16','2022-08-17');
{'element_count': 27,
'links': {'next': 'http://www.neowsapp.com/rest/v1/feed?start_date=2022-08-17&end_date=2022-08-18&detailed=false&api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH',
'prev': 'http://www.neowsapp.com/rest/v1/feed?start_date=2022-08-15&end_date=2022-08-16&detailed=false&api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH',
'self': 'http://www.neowsapp.com/rest/v1/feed?start_date=2022-08-16&end_date=2022-08-17&detailed=false&api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'near_earth_objects': {'2022-08-16': [{'absolute_magnitude_h': 18.05,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'19:05',
'epoch_date_close_approach': 1660676700000,
'miss_distance': {'astronomical': '0.2238852542',
'kilometers': '33492757.152728554',
'lunar': '87.0913638838',
'miles': '20811434.2556758852'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '63921.2642544193',
'kilometers_per_second': '17.7559067373',
'miles_per_hour': '39718.1877807529'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 4786.5767834811,
'estimated_diameter_min': 2140.6222134772},
'kilometers': {'estimated_diameter_max': 1.4589485569,
'estimated_diameter_min': 0.6524616298},
'meters': {'estimated_diameter_max': 1458.9485569187,
'estimated_diameter_min': 652.4616297891},
'miles': {'estimated_diameter_max': 0.9065483238,
'estimated_diameter_min': 0.4054207354}},
'id': '2159928',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/2159928?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '159928 (2005 CV69)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2159928',
'neo_reference_id': '2159928'},
{'absolute_magnitude_h': 20.29,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'01:51',
'epoch_date_close_approach': 1660614660000,
'miss_distance': {'astronomical': '0.3948717663',
'kilometers': '59071975.161617781',
'lunar': '153.6051170907',
'miles': '36705623.3030601378'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '70023.6554030608',
'kilometers_per_second': '19.4510153897',
'miles_per_hour': '43509.9763253074'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1706.1807197036,
'estimated_diameter_min': 763.0272142313},
'kilometers': {'estimated_diameter_max': 0.5200438667,
'estimated_diameter_min': 0.2325706875},
'meters': {'estimated_diameter_max': 520.0438667242,
'estimated_diameter_min': 232.5706874555},
'miles': {'estimated_diameter_max': 0.3231401775,
'estimated_diameter_min': 0.1445126806}},
'id': '2445974',
'is_potentially_hazardous_asteroid': True,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/2445974?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '445974 (2013 BJ18)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2445974',
'neo_reference_id': '2445974'},
{'absolute_magnitude_h': 20.2,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'12:39',
'epoch_date_close_approach': 1660653540000,
'miss_distance': {'astronomical': '0.109943137',
'kilometers': '16447259.11631819',
'lunar': '42.767880293',
'miles': '10219852.914003422'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '47839.62239667',
'kilometers_per_second': '13.2887839991',
'miles_per_hour': '29725.6809275309'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1778.3819018419,
'estimated_diameter_min': 795.3165644948},
'kilometers': {'estimated_diameter_max': 0.5420507863,
'estimated_diameter_min': 0.2424124811},
'meters': {'estimated_diameter_max': 542.0507863358,
'estimated_diameter_min': 242.4124811008},
'miles': {'estimated_diameter_max': 0.3368146392,
'estimated_diameter_min': 0.1506280858}},
'id': '3290881',
'is_potentially_hazardous_asteroid': True,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3290881?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2005 SQ)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3290881',
'neo_reference_id': '3290881'},
{'absolute_magnitude_h': 22.6,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'14:43',
'epoch_date_close_approach': 1660660980000,
'miss_distance': {'astronomical': '0.3344201149',
'kilometers': '50028536.874195263',
'lunar': '130.0894246961',
'miles': '31086291.3231421094'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '58490.6495788559',
'kilometers_per_second': '16.2474026608',
'miles_per_hour': '36343.8150119285'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 588.8775935812,
'estimated_diameter_min': 263.3540659348},
'kilometers': {'estimated_diameter_max': 0.1794898848,
'estimated_diameter_min': 0.0802703167},
'meters': {'estimated_diameter_max': 179.4898847799,
'estimated_diameter_min': 80.2703167283},
'miles': {'estimated_diameter_max': 0.1115298092,
'estimated_diameter_min': 0.049877647}},
'id': '3560817',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3560817?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2011 FS29)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3560817',
'neo_reference_id': '3560817'},
{'absolute_magnitude_h': 21.3,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'14:24',
'epoch_date_close_approach': 1660659840000,
'miss_distance': {'astronomical': '0.2994940718',
'kilometers': '44803675.218907066',
'lunar': '116.5031939302',
'miles': '27839712.8363908708'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '52802.968399447',
'kilometers_per_second': '14.6674912221',
'miles_per_hour': '32809.7111146453'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1071.581062656,
'estimated_diameter_min': 479.2256199},
'kilometers': {'estimated_diameter_max': 0.3266178974,
'estimated_diameter_min': 0.1460679643},
'meters': {'estimated_diameter_max': 326.6178974458,
'estimated_diameter_min': 146.0679642714},
'miles': {'estimated_diameter_max': 0.2029508896,
'estimated_diameter_min': 0.090762397}},
'id': '3723955',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3723955?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2015 NK3)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3723955',
'neo_reference_id': '3723955'},
{'absolute_magnitude_h': 21.3,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'06:33',
'epoch_date_close_approach': 1660631580000,
'miss_distance': {'astronomical': '0.2891952171',
'kilometers': '43262988.492347577',
'lunar': '112.4969394519',
'miles': '26882374.4968754826'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '5567.3396910941',
'kilometers_per_second': '1.5464832475',
'miles_per_hour': '3459.3283763912'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1071.581062656,
'estimated_diameter_min': 479.2256199},
'kilometers': {'estimated_diameter_max': 0.3266178974,
'estimated_diameter_min': 0.1460679643},
'meters': {'estimated_diameter_max': 326.6178974458,
'estimated_diameter_min': 146.0679642714},
'miles': {'estimated_diameter_max': 0.2029508896,
'estimated_diameter_min': 0.090762397}},
'id': '3778647',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3778647?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2017 OR18)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3778647',
'neo_reference_id': '3778647'},
{'absolute_magnitude_h': 21.1,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'23:23',
'epoch_date_close_approach': 1660692180000,
'miss_distance': {'astronomical': '0.2136063756',
'kilometers': '31955058.808179972',
'lunar': '83.0928801084',
'miles': '19855952.8106367336'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '96210.2103546877',
'kilometers_per_second': '26.7250584319',
'miles_per_hour': '59781.2832062225'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1174.9652706022,
'estimated_diameter_min': 525.4604432536},
'kilometers': {'estimated_diameter_max': 0.358129403,
'estimated_diameter_min': 0.160160338},
'meters': {'estimated_diameter_max': 358.1294030194,
'estimated_diameter_min': 160.1603379786},
'miles': {'estimated_diameter_max': 0.2225312253,
'estimated_diameter_min': 0.0995189894}},
'id': '3799743',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3799743?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2018 DT1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3799743',
'neo_reference_id': '3799743'},
{'absolute_magnitude_h': 24.2,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'23:27',
'epoch_date_close_approach': 1660692420000,
'miss_distance': {'astronomical': '0.2441899855',
'kilometers': '36530301.706130885',
'lunar': '94.9899043595',
'miles': '22698876.919280813'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '24767.4056852611',
'kilometers_per_second': '6.8798349126',
'miles_per_hour': '15389.5027159335'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 281.8545369825,
'estimated_diameter_min': 126.0491808919},
'kilometers': {'estimated_diameter_max': 0.0859092601,
'estimated_diameter_min': 0.0384197891},
'meters': {'estimated_diameter_max': 85.9092601232,
'estimated_diameter_min': 38.4197891064},
'miles': {'estimated_diameter_max': 0.0533815229,
'estimated_diameter_min': 0.0238729428}},
'id': '3826623',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3826623?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2018 PV7)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3826623',
'neo_reference_id': '3826623'},
{'absolute_magnitude_h': 22.61,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'12:10',
'epoch_date_close_approach': 1660651800000,
'miss_distance': {'astronomical': '0.2582663589',
'kilometers': '38636097.184095543',
'lunar': '100.4656136121',
'miles': '24007357.5542347734'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '28395.3301493837',
'kilometers_per_second': '7.8875917082',
'miles_per_hour': '17643.7538919879'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 586.1719468077,
'estimated_diameter_min': 262.1440639131},
'kilometers': {'estimated_diameter_max': 0.1786652037,
'estimated_diameter_min': 0.0799015081},
'meters': {'estimated_diameter_max': 178.6652036697,
'estimated_diameter_min': 79.9015081239},
'miles': {'estimated_diameter_max': 0.1110173763,
'estimated_diameter_min': 0.04964848}},
'id': '3843359',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3843359?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2019 OH1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3843359',
'neo_reference_id': '3843359'},
{'absolute_magnitude_h': 20.8,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'17:56',
'epoch_date_close_approach': 1660672560000,
'miss_distance': {'astronomical': '0.4750418782',
'kilometers': '71065253.139519434',
'lunar': '184.7912906198',
'miles': '44157900.6718348292'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '21721.6268160229',
'kilometers_per_second': '6.0337852267',
'miles_per_hour': '13496.9741735449'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1349.040630575,
'estimated_diameter_min': 603.309310875},
'kilometers': {'estimated_diameter_max': 0.411187571,
'estimated_diameter_min': 0.1838886721},
'meters': {'estimated_diameter_max': 411.1875710413,
'estimated_diameter_min': 183.8886720703},
'miles': {'estimated_diameter_max': 0.2555000322,
'estimated_diameter_min': 0.1142630881}},
'id': '54050972',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54050972?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2020 OC7)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54050972',
'neo_reference_id': '54050972'},
{'absolute_magnitude_h': 20.6,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'10:48',
'epoch_date_close_approach': 1660646880000,
'miss_distance': {'astronomical': '0.2304836679',
'kilometers': '34479865.787627373',
'lunar': '89.6581468131',
'miles': '21424795.1194808274'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '91559.042679109',
'kilometers_per_second': '25.4330674109',
'miles_per_hour': '56891.2284913608'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1479.1936371367,
'estimated_diameter_min': 661.5155049046},
'kilometers': {'estimated_diameter_max': 0.4508582062,
'estimated_diameter_min': 0.2016299194},
'meters': {'estimated_diameter_max': 450.8582061718,
'estimated_diameter_min': 201.6299194428},
'miles': {'estimated_diameter_max': 0.2801502144,
'estimated_diameter_min': 0.1252869847}},
'id': '54075323',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54075323?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2020 TN8)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54075323',
'neo_reference_id': '54075323'},
{'absolute_magnitude_h': 29.6,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'04:07',
'epoch_date_close_approach': 1660622820000,
'miss_distance': {'astronomical': '0.1876322222',
'kilometers': '28069380.784486714',
'lunar': '72.9889344358',
'miles': '17441504.4461720932'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '21111.7259858685',
'kilometers_per_second': '5.8643683294',
'miles_per_hour': '13118.0055160526'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 23.4436392583,
'estimated_diameter_min': 10.4843142043},
'kilometers': {'estimated_diameter_max': 0.007145621,
'estimated_diameter_min': 0.0031956189},
'meters': {'estimated_diameter_max': 7.1456210173,
'estimated_diameter_min': 3.1956188672},
'miles': {'estimated_diameter_max': 0.0044400817,
'estimated_diameter_min': 0.0019856649}},
'id': '54106530',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54106530?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2021 BP1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54106530',
'neo_reference_id': '54106530'},
{'absolute_magnitude_h': 23.92,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'18:06',
'epoch_date_close_approach': 1660673160000,
'miss_distance': {'astronomical': '0.4361941952',
'kilometers': '65253722.508284224',
'lunar': '169.6795419328',
'miles': '40546782.9873373312'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '67129.4739183728',
'kilometers_per_second': '18.6470760884',
'miles_per_hour': '41711.6445022245'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 320.6454119044,
'estimated_diameter_min': 143.3969875384},
'kilometers': {'estimated_diameter_max': 0.0977327184,
'estimated_diameter_min': 0.0437074004},
'meters': {'estimated_diameter_max': 97.732718421,
'estimated_diameter_min': 43.7074004031},
'miles': {'estimated_diameter_max': 0.060728277,
'estimated_diameter_min': 0.0271585111}},
'id': '54247854',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54247854?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2016 FL60)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54247854',
'neo_reference_id': '54247854'},
{'absolute_magnitude_h': 26.32,
'close_approach_data': [{'close_approach_date': '2022-08-16',
'close_approach_date_full': '2022-Aug-16 '
'15:03',
'epoch_date_close_approach': 1660662180000,
'miss_distance': {'astronomical': '0.4007251199',
'kilometers': '59947624.392534613',
'lunar': '155.8820716411',
'miles': '37249726.5047511394'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '28389.979134549',
'kilometers_per_second': '7.8861053152',
'miles_per_hour': '17640.4289794646'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 106.1756748422,
'estimated_diameter_min': 47.4832053008},
'kilometers': {'estimated_diameter_max': 0.0323623447,
'estimated_diameter_min': 0.0144728805},
'meters': {'estimated_diameter_max': 32.3623446563,
'estimated_diameter_min': 14.4728805126},
'miles': {'estimated_diameter_max': 0.0201090225,
'estimated_diameter_min': 0.0089930282}},
'id': '54248345',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54248345?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2022 DN1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54248345',
'neo_reference_id': '54248345'}],
'2022-08-17': [{'absolute_magnitude_h': 17.53,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'23:06',
'epoch_date_close_approach': 1660777560000,
'miss_distance': {'astronomical': '0.4753897322',
'kilometers': '71117291.356990414',
'lunar': '184.9266058258',
'miles': '44190235.7207751532'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '77829.6287147987',
'kilometers_per_second': '21.6193413097',
'miles_per_hour': '48360.3045755925'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 6081.7005136835,
'estimated_diameter_min': 2719.8191534783},
'kilometers': {'estimated_diameter_max': 1.8537022573,
'estimated_diameter_min': 0.8290008515},
'meters': {'estimated_diameter_max': 1853.7022572523,
'estimated_diameter_min': 829.0008514522},
'miles': {'estimated_diameter_max': 1.1518368253,
'estimated_diameter_min': 0.5151170881}},
'id': '2285638',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/2285638?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '285638 (2000 SO10)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2285638',
'neo_reference_id': '2285638'},
{'absolute_magnitude_h': 24.7,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'07:33',
'epoch_date_close_approach': 1660721580000,
'miss_distance': {'astronomical': '0.2738866136',
'kilometers': '40972854.016073032',
'lunar': '106.5418926904',
'miles': '25459350.9197985616'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '46204.4968798295',
'kilometers_per_second': '12.8345824666',
'miles_per_hour': '28709.6775195807'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 223.8850168104,
'estimated_diameter_min': 100.1244233463},
'kilometers': {'estimated_diameter_max': 0.0682401509,
'estimated_diameter_min': 0.0305179233},
'meters': {'estimated_diameter_max': 68.2401509401,
'estimated_diameter_min': 30.5179232594},
'miles': {'estimated_diameter_max': 0.0424024508,
'estimated_diameter_min': 0.0189629525}},
'id': '3330687',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3330687?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2006 HU50)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3330687',
'neo_reference_id': '3330687'},
{'absolute_magnitude_h': 21.4,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'08:46',
'epoch_date_close_approach': 1660725960000,
'miss_distance': {'astronomical': '0.390291961',
'kilometers': '58386846.04372307',
'lunar': '151.823572829',
'miles': '36279903.810075566'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '45520.6099457225',
'kilometers_per_second': '12.6446138738',
'miles_per_hour': '28284.7367743296'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 1023.3519701574,
'estimated_diameter_min': 457.6569140361},
'kilometers': {'estimated_diameter_max': 0.3119176705,
'estimated_diameter_min': 0.1394938229},
'meters': {'estimated_diameter_max': 311.9176705226,
'estimated_diameter_min': 139.4938229344},
'miles': {'estimated_diameter_max': 0.1938165949,
'estimated_diameter_min': 0.0866774163}},
'id': '3532394',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3532394?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2010 MP1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3532394',
'neo_reference_id': '3532394'},
{'absolute_magnitude_h': 25.0,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'06:15',
'epoch_date_close_approach': 1660716900000,
'miss_distance': {'astronomical': '0.4146458219',
'kilometers': '62030131.760639353',
'lunar': '161.2972247191',
'miles': '38543736.5792469514'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '45400.7893213213',
'kilometers_per_second': '12.611330367',
'miles_per_hour': '28210.2848980177'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 194.9956979785,
'estimated_diameter_min': 87.2047272},
'kilometers': {'estimated_diameter_max': 0.0594346868,
'estimated_diameter_min': 0.02658},
'meters': {'estimated_diameter_max': 59.4346868419,
'estimated_diameter_min': 26.58},
'miles': {'estimated_diameter_max': 0.0369309908,
'estimated_diameter_min': 0.0165160412}},
'id': '3690061',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3690061?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2014 SH144)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3690061',
'neo_reference_id': '3690061'},
{'absolute_magnitude_h': 23.7,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'10:34',
'epoch_date_close_approach': 1660732440000,
'miss_distance': {'astronomical': '0.0529267402',
'kilometers': '7917727.599963374',
'lunar': '20.5885019378',
'miles': '4919847.7942436012'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '39097.8162363992',
'kilometers_per_second': '10.8605045101',
'miles_per_hour': '24293.8625386671'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 354.8338390368,
'estimated_diameter_min': 158.6865169607},
'kilometers': {'estimated_diameter_max': 0.1081533507,
'estimated_diameter_min': 0.0483676488},
'meters': {'estimated_diameter_max': 108.1533506775,
'estimated_diameter_min': 48.3676488219},
'miles': {'estimated_diameter_max': 0.0672033557,
'estimated_diameter_min': 0.0300542543}},
'id': '3714001',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3714001?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2015 FN35)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3714001',
'neo_reference_id': '3714001'},
{'absolute_magnitude_h': 25.9,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'06:31',
'epoch_date_close_approach': 1660717860000,
'miss_distance': {'astronomical': '0.3409988372',
'kilometers': '51012699.717596764',
'lunar': '132.6485476708',
'miles': '31697821.7569887832'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '58920.2233387679',
'kilometers_per_second': '16.3667287052',
'miles_per_hour': '36610.7354406916'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 128.8323800441,
'estimated_diameter_min': 57.6155918963},
'kilometers': {'estimated_diameter_max': 0.0392681082,
'estimated_diameter_min': 0.0175612318},
'meters': {'estimated_diameter_max': 39.2681081809,
'estimated_diameter_min': 17.561231848},
'miles': {'estimated_diameter_max': 0.0244000636,
'estimated_diameter_min': 0.0109120402}},
'id': '3744834',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3744834?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2016 DW1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3744834',
'neo_reference_id': '3744834'},
{'absolute_magnitude_h': 23.1,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'05:18',
'epoch_date_close_approach': 1660713480000,
'miss_distance': {'astronomical': '0.4633320661',
'kilometers': '69313490.191259207',
'lunar': '180.2361737129',
'miles': '43069405.6499707766'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '55029.9047739469',
'kilometers_per_second': '15.2860846594',
'miles_per_hour': '34193.4427746783'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 467.7620993781,
'estimated_diameter_min': 209.1895703015},
'kilometers': {'estimated_diameter_max': 0.1425738833,
'estimated_diameter_min': 0.063760979},
'meters': {'estimated_diameter_max': 142.5738833281,
'estimated_diameter_min': 63.7609789875},
'miles': {'estimated_diameter_max': 0.0885912765,
'estimated_diameter_min': 0.0396192233}},
'id': '3797908',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3797908?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2018 BK5)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3797908',
'neo_reference_id': '3797908'},
{'absolute_magnitude_h': 24.9,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'04:58',
'epoch_date_close_approach': 1660712280000,
'miss_distance': {'astronomical': '0.3892544769',
'kilometers': '58231640.632204203',
'lunar': '151.4199915141',
'miles': '36183463.6393478814'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '23690.2095709795',
'kilometers_per_second': '6.5806137697',
'miles_per_hour': '14720.1749414789'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 204.1855621004,
'estimated_diameter_min': 91.3145593761},
'kilometers': {'estimated_diameter_max': 0.0622357573,
'estimated_diameter_min': 0.0278326768},
'meters': {'estimated_diameter_max': 62.2357573367,
'estimated_diameter_min': 27.8326768072},
'miles': {'estimated_diameter_max': 0.0386714948,
'estimated_diameter_min': 0.0172944182}},
'id': '3825100',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/3825100?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2018 LC1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3825100',
'neo_reference_id': '3825100'},
{'absolute_magnitude_h': 23.9,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'06:25',
'epoch_date_close_approach': 1660717500000,
'miss_distance': {'astronomical': '0.1938562821',
'kilometers': '29000486.888279127',
'lunar': '75.4100937369',
'miles': '18020066.9507688726'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '26034.1585677636',
'kilometers_per_second': '7.2317107133',
'miles_per_hour': '16176.6136945085'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 323.6123073718,
'estimated_diameter_min': 144.7238235278},
'kilometers': {'estimated_diameter_max': 0.0986370281,
'estimated_diameter_min': 0.04411182},
'meters': {'estimated_diameter_max': 98.6370281305,
'estimated_diameter_min': 44.1118199997},
'miles': {'estimated_diameter_max': 0.0612901888,
'estimated_diameter_min': 0.0274098057}},
'id': '54048888',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54048888?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2020 OK1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54048888',
'neo_reference_id': '54048888'},
{'absolute_magnitude_h': 22.0,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'15:49',
'epoch_date_close_approach': 1660751340000,
'miss_distance': {'astronomical': '0.3826928775',
'kilometers': '57250039.338170925',
'lunar': '148.8675293475',
'miles': '35573524.878471765'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '79434.0716872817',
'kilometers_per_second': '22.0650199131',
'miles_per_hour': '49357.243043689'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 776.2918559233,
'estimated_diameter_min': 347.1682720448},
'kilometers': {'estimated_diameter_max': 0.2366137501,
'estimated_diameter_min': 0.1058168859},
'meters': {'estimated_diameter_max': 236.6137501138,
'estimated_diameter_min': 105.8168859331},
'miles': {'estimated_diameter_max': 0.1470249225,
'estimated_diameter_min': 0.0657515442}},
'id': '54053722',
'is_potentially_hazardous_asteroid': True,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54053722?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2020 RC)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54053722',
'neo_reference_id': '54053722'},
{'absolute_magnitude_h': 26.24,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'22:34',
'epoch_date_close_approach': 1660775640000,
'miss_distance': {'astronomical': '0.0924481569',
'kilometers': '13830047.357665803',
'lunar': '35.9623330341',
'miles': '8593592.9378539614'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '24885.6751694751',
'kilometers_per_second': '6.9126875471',
'miles_per_hour': '15462.9907740552'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 110.1602797174,
'estimated_diameter_min': 49.2651747737},
'kilometers': {'estimated_diameter_max': 0.0335768522,
'estimated_diameter_min': 0.0150160248},
'meters': {'estimated_diameter_max': 33.5768521834,
'estimated_diameter_min': 15.0160247905},
'miles': {'estimated_diameter_max': 0.0208636822,
'estimated_diameter_min': 0.0093305223}},
'id': '54123272',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54123272?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2021 DC1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54123272',
'neo_reference_id': '54123272'},
{'absolute_magnitude_h': 28.12,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'23:36',
'epoch_date_close_approach': 1660779360000,
'miss_distance': {'astronomical': '0.4323786926',
'kilometers': '64682931.446344762',
'lunar': '168.1953114214',
'miles': '40192109.8678592356'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '19930.7553682022',
'kilometers_per_second': '5.5363209356',
'miles_per_hour': '12384.1963008699'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 46.3473630674,
'estimated_diameter_min': 20.7271708793},
'kilometers': {'estimated_diameter_max': 0.0141266758,
'estimated_diameter_min': 0.0063176415},
'meters': {'estimated_diameter_max': 14.1266758109,
'estimated_diameter_min': 6.3176414818},
'miles': {'estimated_diameter_max': 0.0087779067,
'estimated_diameter_min': 0.0039255992}},
'id': '54265632',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54265632?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2022 FR1)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265632',
'neo_reference_id': '54265632'},
{'absolute_magnitude_h': 26.32,
'close_approach_data': [{'close_approach_date': '2022-08-17',
'close_approach_date_full': '2022-Aug-17 '
'14:35',
'epoch_date_close_approach': 1660746900000,
'miss_distance': {'astronomical': '0.1988044094',
'kilometers': '29740716.192847978',
'lunar': '77.3349152566',
'miles': '18480024.1121309764'},
'orbiting_body': 'Earth',
'relative_velocity': {'kilometers_per_hour': '24154.531639754',
'kilometers_per_second': '6.7095921222',
'miles_per_hour': '15008.6866180461'}}],
'estimated_diameter': {'feet': {'estimated_diameter_max': 106.1756748422,
'estimated_diameter_min': 47.4832053008},
'kilometers': {'estimated_diameter_max': 0.0323623447,
'estimated_diameter_min': 0.0144728805},
'meters': {'estimated_diameter_max': 32.3623446563,
'estimated_diameter_min': 14.4728805126},
'miles': {'estimated_diameter_max': 0.0201090225,
'estimated_diameter_min': 0.0089930282}},
'id': '54265877',
'is_potentially_hazardous_asteroid': False,
'is_sentry_object': False,
'links': {'self': 'http://www.neowsapp.com/rest/v1/neo/54265877?api_key=uIuYlzCW70zB1F9oRcBUIIzhpM9m4V2neOwhn9iH'},
'name': '(2022 FM2)',
'nasa_jpl_url': 'http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265877',
'neo_reference_id': '54265877'}]}}
Yikes! This is a pretty complicated looking nested json! The data we want is contained in the 'near_earth_objects' key. This key then maps to the dates we queried data for. I tried doing a simple flattening as well as normalizing the json data in order to put the data into a pandas dataframe but wasn't successful. However, the following function can be used to recursively look through our desired level.
In [65]:
def flatten_json(nested_json: dict, exclude: list=['']) -> dict:
"""
Flatten a list of nested dicts.
"""
out = dict()
def flatten(x: (list, dict, str), name: str='', exclude=exclude):
if type(x) is dict:
for a in x:
if a not in exclude:
flatten(x[a], f'{name}{a}_')
elif type(x) is list:
i = 0
for a in x:
flatten(a, f'{name}{i}_')
i += 1
else:
out[name[:-1]] = x
flatten(nested_json)
return out
The level we want is here
In [247]:
aneo['near_earth_objects']['2022-08-17'];
Then we can make our data frame by applying the function above in order to obtain the result below :]
In [74]:
df = pd.DataFrame([flatten_json(x) for x in aneo['near_earth_objects']['2022-08-17']])
df
Out[74]:
| links_self | id | neo_reference_id | name | nasa_jpl_url | absolute_magnitude_h | estimated_diameter_kilometers_estimated_diameter_min | estimated_diameter_kilometers_estimated_diameter_max | estimated_diameter_meters_estimated_diameter_min | estimated_diameter_meters_estimated_diameter_max | ... | close_approach_data_0_epoch_date_close_approach | close_approach_data_0_relative_velocity_kilometers_per_second | close_approach_data_0_relative_velocity_kilometers_per_hour | close_approach_data_0_relative_velocity_miles_per_hour | close_approach_data_0_miss_distance_astronomical | close_approach_data_0_miss_distance_lunar | close_approach_data_0_miss_distance_kilometers | close_approach_data_0_miss_distance_miles | close_approach_data_0_orbiting_body | is_sentry_object | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | http://www.neowsapp.com/rest/v1/neo/2285638?ap... | 2285638 | 2285638 | 285638 (2000 SO10) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2285638 | 17.53 | 0.829001 | 1.853702 | 829.000851 | 1853.702257 | ... | 1660777560000 | 21.6193413097 | 77829.6287147987 | 48360.3045755925 | 0.4753897322 | 184.9266058258 | 71117291.356990414 | 44190235.7207751532 | Earth | False |
| 1 | http://www.neowsapp.com/rest/v1/neo/3330687?ap... | 3330687 | 3330687 | (2006 HU50) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3330687 | 24.70 | 0.030518 | 0.068240 | 30.517923 | 68.240151 | ... | 1660721580000 | 12.8345824666 | 46204.4968798295 | 28709.6775195807 | 0.2738866136 | 106.5418926904 | 40972854.016073032 | 25459350.9197985616 | Earth | False |
| 2 | http://www.neowsapp.com/rest/v1/neo/3532394?ap... | 3532394 | 3532394 | (2010 MP1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3532394 | 21.40 | 0.139494 | 0.311918 | 139.493823 | 311.917671 | ... | 1660725960000 | 12.6446138738 | 45520.6099457225 | 28284.7367743296 | 0.390291961 | 151.823572829 | 58386846.04372307 | 36279903.810075566 | Earth | False |
| 3 | http://www.neowsapp.com/rest/v1/neo/3690061?ap... | 3690061 | 3690061 | (2014 SH144) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3690061 | 25.00 | 0.026580 | 0.059435 | 26.580000 | 59.434687 | ... | 1660716900000 | 12.611330367 | 45400.7893213213 | 28210.2848980177 | 0.4146458219 | 161.2972247191 | 62030131.760639353 | 38543736.5792469514 | Earth | False |
| 4 | http://www.neowsapp.com/rest/v1/neo/3714001?ap... | 3714001 | 3714001 | (2015 FN35) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3714001 | 23.70 | 0.048368 | 0.108153 | 48.367649 | 108.153351 | ... | 1660732440000 | 10.8605045101 | 39097.8162363992 | 24293.8625386671 | 0.0529267402 | 20.5885019378 | 7917727.599963374 | 4919847.7942436012 | Earth | False |
| 5 | http://www.neowsapp.com/rest/v1/neo/3744834?ap... | 3744834 | 3744834 | (2016 DW1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3744834 | 25.90 | 0.017561 | 0.039268 | 17.561232 | 39.268108 | ... | 1660717860000 | 16.3667287052 | 58920.2233387679 | 36610.7354406916 | 0.3409988372 | 132.6485476708 | 51012699.717596764 | 31697821.7569887832 | Earth | False |
| 6 | http://www.neowsapp.com/rest/v1/neo/3797908?ap... | 3797908 | 3797908 | (2018 BK5) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3797908 | 23.10 | 0.063761 | 0.142574 | 63.760979 | 142.573883 | ... | 1660713480000 | 15.2860846594 | 55029.9047739469 | 34193.4427746783 | 0.4633320661 | 180.2361737129 | 69313490.191259207 | 43069405.6499707766 | Earth | False |
| 7 | http://www.neowsapp.com/rest/v1/neo/3825100?ap... | 3825100 | 3825100 | (2018 LC1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3825100 | 24.90 | 0.027833 | 0.062236 | 27.832677 | 62.235757 | ... | 1660712280000 | 6.5806137697 | 23690.2095709795 | 14720.1749414789 | 0.3892544769 | 151.4199915141 | 58231640.632204203 | 36183463.6393478814 | Earth | False |
| 8 | http://www.neowsapp.com/rest/v1/neo/54048888?a... | 54048888 | 54048888 | (2020 OK1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54048888 | 23.90 | 0.044112 | 0.098637 | 44.111820 | 98.637028 | ... | 1660717500000 | 7.2317107133 | 26034.1585677636 | 16176.6136945085 | 0.1938562821 | 75.4100937369 | 29000486.888279127 | 18020066.9507688726 | Earth | False |
| 9 | http://www.neowsapp.com/rest/v1/neo/54053722?a... | 54053722 | 54053722 | (2020 RC) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54053722 | 22.00 | 0.105817 | 0.236614 | 105.816886 | 236.613750 | ... | 1660751340000 | 22.0650199131 | 79434.0716872817 | 49357.243043689 | 0.3826928775 | 148.8675293475 | 57250039.338170925 | 35573524.878471765 | Earth | False |
| 10 | http://www.neowsapp.com/rest/v1/neo/54123272?a... | 54123272 | 54123272 | (2021 DC1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54123272 | 26.24 | 0.015016 | 0.033577 | 15.016025 | 33.576852 | ... | 1660775640000 | 6.9126875471 | 24885.6751694751 | 15462.9907740552 | 0.0924481569 | 35.9623330341 | 13830047.357665803 | 8593592.9378539614 | Earth | False |
| 11 | http://www.neowsapp.com/rest/v1/neo/54265632?a... | 54265632 | 54265632 | (2022 FR1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265632 | 28.12 | 0.006318 | 0.014127 | 6.317641 | 14.126676 | ... | 1660779360000 | 5.5363209356 | 19930.7553682022 | 12384.1963008699 | 0.4323786926 | 168.1953114214 | 64682931.446344762 | 40192109.8678592356 | Earth | False |
| 12 | http://www.neowsapp.com/rest/v1/neo/54265877?a... | 54265877 | 54265877 | (2022 FM2) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265877 | 26.32 | 0.014473 | 0.032362 | 14.472881 | 32.362345 | ... | 1660746900000 | 6.7095921222 | 24154.531639754 | 15008.6866180461 | 0.1988044094 | 77.3349152566 | 29740716.192847978 | 18480024.1121309764 | Earth | False |
13 rows × 27 columns
As I did this it hit me... why don't I simply loop over the json starting at this level? So I loaded the json file starting at that level and... voila!
In [76]:
aneo_df = pd.DataFrame(aneo['near_earth_objects']['2022-08-17'])
aneo_df
Out[76]:
| links | id | neo_reference_id | name | nasa_jpl_url | absolute_magnitude_h | estimated_diameter | is_potentially_hazardous_asteroid | close_approach_data | is_sentry_object | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 2285638 | 2285638 | 285638 (2000 SO10) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2285638 | 17.53 | {'kilometers': {'estimated_diameter_min': 0.82... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 1 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3330687 | 3330687 | (2006 HU50) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3330687 | 24.70 | {'kilometers': {'estimated_diameter_min': 0.03... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 2 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3532394 | 3532394 | (2010 MP1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3532394 | 21.40 | {'kilometers': {'estimated_diameter_min': 0.13... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 3 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3690061 | 3690061 | (2014 SH144) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3690061 | 25.00 | {'kilometers': {'estimated_diameter_min': 0.02... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 4 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3714001 | 3714001 | (2015 FN35) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3714001 | 23.70 | {'kilometers': {'estimated_diameter_min': 0.04... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 5 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3744834 | 3744834 | (2016 DW1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3744834 | 25.90 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 6 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3797908 | 3797908 | (2018 BK5) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3797908 | 23.10 | {'kilometers': {'estimated_diameter_min': 0.06... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 7 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3825100 | 3825100 | (2018 LC1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3825100 | 24.90 | {'kilometers': {'estimated_diameter_min': 0.02... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 8 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54048888 | 54048888 | (2020 OK1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54048888 | 23.90 | {'kilometers': {'estimated_diameter_min': 0.04... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 9 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54053722 | 54053722 | (2020 RC) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54053722 | 22.00 | {'kilometers': {'estimated_diameter_min': 0.10... | True | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 10 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54123272 | 54123272 | (2021 DC1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54123272 | 26.24 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 11 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54265632 | 54265632 | (2022 FR1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265632 | 28.12 | {'kilometers': {'estimated_diameter_min': 0.00... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 12 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54265877 | 54265877 | (2022 FM2) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265877 | 26.32 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
The data there is for just one day. But now we can easily loop through the days and concatenate our dataframes! I only did a query for two days so my date list is as follows
In [80]:
date_list = ['2022-08-17','2022-08-16']
tot_aneo_list = []
for date in date_list:
aneo_df = pd.DataFrame(aneo['near_earth_objects'][date])
tot_aneo_list.append(aneo_df)
aneo_tot_df = pd.concat(tot_aneo_list, ignore_index = True)
aneo_tot_df
Out[80]:
| links | id | neo_reference_id | name | nasa_jpl_url | absolute_magnitude_h | estimated_diameter | is_potentially_hazardous_asteroid | close_approach_data | is_sentry_object | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 2285638 | 2285638 | 285638 (2000 SO10) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2285638 | 17.53 | {'kilometers': {'estimated_diameter_min': 0.82... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 1 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3330687 | 3330687 | (2006 HU50) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3330687 | 24.70 | {'kilometers': {'estimated_diameter_min': 0.03... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 2 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3532394 | 3532394 | (2010 MP1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3532394 | 21.40 | {'kilometers': {'estimated_diameter_min': 0.13... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 3 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3690061 | 3690061 | (2014 SH144) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3690061 | 25.00 | {'kilometers': {'estimated_diameter_min': 0.02... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 4 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3714001 | 3714001 | (2015 FN35) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3714001 | 23.70 | {'kilometers': {'estimated_diameter_min': 0.04... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 5 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3744834 | 3744834 | (2016 DW1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3744834 | 25.90 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 6 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3797908 | 3797908 | (2018 BK5) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3797908 | 23.10 | {'kilometers': {'estimated_diameter_min': 0.06... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 7 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3825100 | 3825100 | (2018 LC1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3825100 | 24.90 | {'kilometers': {'estimated_diameter_min': 0.02... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 8 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54048888 | 54048888 | (2020 OK1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54048888 | 23.90 | {'kilometers': {'estimated_diameter_min': 0.04... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 9 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54053722 | 54053722 | (2020 RC) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54053722 | 22.00 | {'kilometers': {'estimated_diameter_min': 0.10... | True | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 10 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54123272 | 54123272 | (2021 DC1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54123272 | 26.24 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 11 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54265632 | 54265632 | (2022 FR1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265632 | 28.12 | {'kilometers': {'estimated_diameter_min': 0.00... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 12 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54265877 | 54265877 | (2022 FM2) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54265877 | 26.32 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-17', 'close_... | False |
| 13 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 2159928 | 2159928 | 159928 (2005 CV69) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2159928 | 18.05 | {'kilometers': {'estimated_diameter_min': 0.65... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 14 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 2445974 | 2445974 | 445974 (2013 BJ18) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=2445974 | 20.29 | {'kilometers': {'estimated_diameter_min': 0.23... | True | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 15 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3290881 | 3290881 | (2005 SQ) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3290881 | 20.20 | {'kilometers': {'estimated_diameter_min': 0.24... | True | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 16 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3560817 | 3560817 | (2011 FS29) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3560817 | 22.60 | {'kilometers': {'estimated_diameter_min': 0.08... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 17 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3723955 | 3723955 | (2015 NK3) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3723955 | 21.30 | {'kilometers': {'estimated_diameter_min': 0.14... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 18 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3778647 | 3778647 | (2017 OR18) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3778647 | 21.30 | {'kilometers': {'estimated_diameter_min': 0.14... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 19 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3799743 | 3799743 | (2018 DT1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3799743 | 21.10 | {'kilometers': {'estimated_diameter_min': 0.16... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 20 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3826623 | 3826623 | (2018 PV7) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3826623 | 24.20 | {'kilometers': {'estimated_diameter_min': 0.03... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 21 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 3843359 | 3843359 | (2019 OH1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3843359 | 22.61 | {'kilometers': {'estimated_diameter_min': 0.07... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 22 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54050972 | 54050972 | (2020 OC7) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54050972 | 20.80 | {'kilometers': {'estimated_diameter_min': 0.18... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 23 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54075323 | 54075323 | (2020 TN8) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54075323 | 20.60 | {'kilometers': {'estimated_diameter_min': 0.20... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 24 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54106530 | 54106530 | (2021 BP1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54106530 | 29.60 | {'kilometers': {'estimated_diameter_min': 0.00... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 25 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54247854 | 54247854 | (2016 FL60) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54247854 | 23.92 | {'kilometers': {'estimated_diameter_min': 0.04... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
| 26 | {'self': 'http://www.neowsapp.com/rest/v1/neo/... | 54248345 | 54248345 | (2022 DN1) | http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=54248345 | 26.32 | {'kilometers': {'estimated_diameter_min': 0.01... | False | [{'close_approach_date': '2022-08-16', 'close_... | False |
Sweet! Now we have this data ready to do more analysis on which I will do something with in a future post :]
Getting Weather Data from National Weather Service
The National Weather Service (NWS) API gives us access to forecasts, alerts, and observations, and other weather data. You can get more information about the data you can collect here: https://weather-gov.github.io/api/general-faqsTo get a forecast reading, we need to 3 things:
- NWS Office name which we can get from here: https://www.weather.gov/srh/nwsoffices
- latitude of location we want a forecast for
- longitude of location we want a forecast for
The base URL for a forecase reading looks as follows:
https://api.weather.gov/gridpoints/{office}/{grid X},{grid Y}/forecast
For this example, I'll show how to get the weather forecast for New York City. The latitude and longitude of NYC are 40.7128° N, 74.0060° W.
In [246]:
grid_X = '40.730610'
grid_Y = '-73.935242'
url = f'https://api.weather.gov/points/{grid_X},{grid_Y}'
rmd = requests.get(url)
rmd = rmd.json()
#rmd
Now that we have the metadata for NYC we can get information about the forecast using the URL provided in the json. So let's do that.
In [245]:
url = 'https://api.weather.gov/gridpoints/OKX/35,35/forecast'
nyc_forecast_request = requests.get(url)
nyc_forecast_request
nyc_forecast = nyc_forecast_request.json()
#nyc_forecast
What do you know... another nested json dataset. No problem though! The data we want is within at the periods level so we just have to set that for our DataFrame entry and now we have our weekly forecast placed nicely in a pandas dataframe! :]
In [174]:
nyc_fc_df = pd.json_normalize(nyc_forecast['properties']['periods'])
nyc_fc_df
Out[174]:
| number | name | startTime | endTime | isDaytime | temperature | temperatureUnit | temperatureTrend | windSpeed | windDirection | icon | shortForecast | detailedForecast | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | Tonight | 2022-08-18T22:00:00-04:00 | 2022-08-19T06:00:00-04:00 | False | 73 | F | None | 3 to 7 mph | W | https://api.weather.gov/icons/land/night/few?s... | Mostly Clear | Mostly clear, with a low around 73. West wind ... |
| 1 | 2 | Friday | 2022-08-19T06:00:00-04:00 | 2022-08-19T18:00:00-04:00 | True | 89 | F | None | 5 to 12 mph | SW | https://api.weather.gov/icons/land/day/sct?siz... | Mostly Sunny | Mostly sunny, with a high near 89. Southwest w... |
| 2 | 3 | Friday Night | 2022-08-19T18:00:00-04:00 | 2022-08-20T06:00:00-04:00 | False | 74 | F | None | 5 to 12 mph | SW | https://api.weather.gov/icons/land/night/sct?s... | Partly Cloudy | Partly cloudy, with a low around 74. Southwest... |
| 3 | 4 | Saturday | 2022-08-20T06:00:00-04:00 | 2022-08-20T18:00:00-04:00 | True | 86 | F | falling | 2 to 13 mph | S | https://api.weather.gov/icons/land/day/bkn?siz... | Partly Sunny | Partly sunny. High near 86, with temperatures ... |
| 4 | 5 | Saturday Night | 2022-08-20T18:00:00-04:00 | 2022-08-21T06:00:00-04:00 | False | 71 | F | rising | 3 to 12 mph | SE | https://api.weather.gov/icons/land/night/sct?s... | Partly Cloudy | Partly cloudy. Low around 71, with temperature... |
| 5 | 6 | Sunday | 2022-08-21T06:00:00-04:00 | 2022-08-21T18:00:00-04:00 | True | 85 | F | falling | 3 to 12 mph | SE | https://api.weather.gov/icons/land/day/bkn/rai... | Partly Sunny then Slight Chance Rain Showers | A slight chance of rain showers after 2pm. Par... |
| 6 | 7 | Sunday Night | 2022-08-21T18:00:00-04:00 | 2022-08-22T06:00:00-04:00 | False | 72 | F | None | 3 to 10 mph | SE | https://api.weather.gov/icons/land/night/rain_... | Chance Rain Showers | A chance of rain showers. Mostly cloudy, with ... |
| 7 | 8 | Monday | 2022-08-22T06:00:00-04:00 | 2022-08-22T18:00:00-04:00 | True | 82 | F | None | 3 to 8 mph | SE | https://api.weather.gov/icons/land/day/rain_sh... | Showers And Thunderstorms Likely | A chance of rain showers before 2pm, then show... |
| 8 | 9 | Monday Night | 2022-08-22T18:00:00-04:00 | 2022-08-23T06:00:00-04:00 | False | 71 | F | None | 2 to 7 mph | SW | https://api.weather.gov/icons/land/night/tsra_... | Showers And Thunderstorms Likely | Showers and thunderstorms likely. Mostly cloud... |
| 9 | 10 | Tuesday | 2022-08-23T06:00:00-04:00 | 2022-08-23T18:00:00-04:00 | True | 84 | F | None | 2 to 9 mph | W | https://api.weather.gov/icons/land/day/rain_sh... | Chance Rain Showers | A chance of rain showers before 2pm, then a ch... |
| 10 | 11 | Tuesday Night | 2022-08-23T18:00:00-04:00 | 2022-08-24T06:00:00-04:00 | False | 71 | F | None | 3 to 8 mph | SW | https://api.weather.gov/icons/land/night/tsra_... | Chance Showers And Thunderstorms then Slight C... | A chance of showers and thunderstorms before 8... |
| 11 | 12 | Wednesday | 2022-08-24T06:00:00-04:00 | 2022-08-24T18:00:00-04:00 | True | 87 | F | None | 5 to 9 mph | W | https://api.weather.gov/icons/land/day/sct/rai... | Mostly Sunny then Slight Chance Rain Showers | A slight chance of rain showers after 2pm. Mos... |
| 12 | 13 | Wednesday Night | 2022-08-24T18:00:00-04:00 | 2022-08-25T06:00:00-04:00 | False | 72 | F | None | 3 to 9 mph | SW | https://api.weather.gov/icons/land/night/rain_... | Slight Chance Rain Showers then Partly Cloudy | A slight chance of rain showers before 8pm. Pa... |
| 13 | 14 | Thursday | 2022-08-25T06:00:00-04:00 | 2022-08-25T18:00:00-04:00 | True | 88 | F | None | 3 to 10 mph | W | https://api.weather.gov/icons/land/day/sct?siz... | Mostly Sunny | Mostly sunny, with a high near 88. |
Getting Earthquake Data from USGS
In this section I'll show how to get data from the United States Geological Survey (USGS) API. From the USGS we can get earthquake information. You can find the documentation for the API here: https://earthquake.usgs.gov/fdsnws/event/1/?ref=springboardThe URL has the following basic structure:
https://earthquake.usgs.gov/fdsnws/event/1/[METHOD[?PARAMETERS]]
We'll be requesting the METHOD to be format=geojason in order to return a json file. There are options to get your data as a csv, xml, and plain text too though.
The parameters can return help refine our query by providing information regarding the time, desired location and many others for earthquake data.
Let's start by seeing what kind of information we can extract. I'm not displaying the ouput because it's really long.
In [244]:
url = 'https://earthquake.usgs.gov/fdsnws/event/1/application.json'
r = requests.get(url)
r = r.json()
Interesting! There's stuff about volcanic events, nuclear explosions, rock slides and even meteorites! Let's find out how many requests we can make
In [183]:
url = 'https://earthquake.usgs.gov/fdsnws/event/1/count?format=geojson'
r = requests.get(url)
r = r.json()
r
Out[183]:
{'count': 8868, 'maxAllowed': 20000}
In [242]:
start ='2014-01-01'
end = '2014-01-02'
url = f'https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime={start}&endtime={end}'
r = requests.get(url)
r = r.json()
We have data! Let's get this into a dataframe!
In [243]:
earthquake_df = pd.json_normalize(r['features'])
earthquake_df
Out[243]:
| type | id | properties.mag | properties.place | properties.time | properties.updated | properties.tz | properties.url | properties.detail | properties.felt | ... | properties.types | properties.nst | properties.dmin | properties.rms | properties.gap | properties.magType | properties.type | properties.title | geometry.type | geometry.coordinates | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Feature | ci11408890 | 1.29 | 10km SSW of Idyllwild, CA | 1388620296020 | 1457728844428 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,cap,focal-mechanism,nearby-cities,origin,phas... | 39.0 | 0.067290 | 0.09 | 51.0 | ml | earthquake | M 1.3 - 10km SSW of Idyllwild, CA | Point | [-116.7776667, 33.6633333, 11.008] |
| 1 | Feature | ak01421ig3u | 1.10 | Central Alaska | 1388620046501 | 1558392330681 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,associate,origin,phase-data, | NaN | NaN | 0.57 | NaN | ml | earthquake | M 1.1 - Central Alaska | Point | [-151.6459, 63.102, 14.1] |
| 2 | Feature | ak01421i2zj | 1.20 | 7 km SSW of Big Lake, Alaska | 1388619956476 | 1558392330249 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,associate,cap,origin,phase-data, | NaN | NaN | 0.47 | NaN | ml | earthquake | M 1.2 - 7 km SSW of Big Lake, Alaska | Point | [-150.0165, 61.4581, 44.6] |
| 3 | Feature | ak01421heui | 1.40 | 32 km N of Petersville, Alaska | 1388619763623 | 1558392329762 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,associate,origin,phase-data, | NaN | NaN | 0.35 | NaN | ml | earthquake | M 1.4 - 32 km N of Petersville, Alaska | Point | [-150.8276, 62.7884, 87.6] |
| 4 | Feature | usc000mnnn | 4.00 | 28 km WNW of Coquimbo, Chile | 1388619735000 | 1394151954000 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,origin,phase-data, | NaN | NaN | 3.09 | NaN | ml | earthquake | M 4.0 - 28 km WNW of Coquimbo, Chile | Point | [-71.621, -29.888, 40] |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 321 | Feature | ak0141nol21 | 2.00 | 69 km ENE of Pedro Bay, Alaska | 1388535883771 | 1558392307098 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,associate,origin,phase-data, | NaN | NaN | 0.77 | NaN | ml | earthquake | M 2.0 - 69 km ENE of Pedro Bay, Alaska | Point | [-153.0185, 60.0872, 105.8] |
| 322 | Feature | nc72133841 | 0.57 | 10 km W of Cobb, California | 1388535010970 | 1486062277926 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,nearby-cities,origin,phase-data,scitech-link, | 30.0 | 0.007207 | 0.04 | 65.0 | md | earthquake | M 0.6 - 10 km W of Cobb, California | Point | [-122.8435, 38.8216667, 2.14] |
| 323 | Feature | uw60664271 | 2.20 | 1 km NW of Sweet Home, Oregon | 1388534755060 | 1469215315190 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,cap,origin,phase-data, | 4.0 | 0.115700 | 0.07 | 179.0 | md | explosion | M 2.2 Explosion - 1 km NW of Sweet Home, Oregon | Point | [-122.7498333, 44.408, -0.348] |
| 324 | Feature | usc000lv5j | 4.50 | 163 km W of Neiafu, Tonga | 1388534728830 | 1394151953000 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,cap,origin,phase-data, | NaN | 5.337000 | 0.60 | 41.0 | mb | earthquake | M 4.5 - 163 km W of Neiafu, Tonga | Point | [-175.5282, -18.4965, 217.9] |
| 325 | Feature | usc000lv5e | 5.10 | 76 km NNW of Davila, Philippines | 1388534476610 | 1394151953000 | None | https://earthquake.usgs.gov/earthquakes/eventp... | https://earthquake.usgs.gov/fdsnws/event/1/que... | NaN | ... | ,cap,origin,phase-data, | NaN | 3.794000 | 0.85 | 29.0 | mb | earthquake | M 5.1 - 76 km NNW of Davila, Philippines | Point | [120.2389, 19.0868, 10.07] |
326 rows × 30 columns
Awesome! More data to play with! Below is an example of how to run a query that has more specifications and that template could be used for even further refinments by adding more sections to the parameters string.
In [199]:
def earthquake(f):
parameters = {
"format": "geojson",
"starttime": "2020- 01-01",
"endtime": "2020-12-31",
"alertlevel": "orange"
}
data = requests.get(f, params = parameters)
data = json.loads(data.text)
return data
f = r"https://earthquake.usgs.gov/fdsnws/event/1/query?"
a = earthquake(f)
Getting News from The New York Times
This API gives you access to an enormous amount of articles from the New York Times. Some of the data goes back all the way to 1851!Here I'll show how to get different types of information like archives and movie reviews. You can get detailed information about the API usage here: https://developer.nytimes.com/apis
Let's start by getting the archive for the month of June and the year 2020. I won't print the ouput of this particular query because it's really long...
In [241]:
year = '2020'
month = '6'
api_key = 'jhkMYAlxMsPPYrLatKFLwF6JbQLhME9V'
url = f'https://api.nytimes.com/svc/archive/v1/{year}/{month}.json?api-key={api_key}'
r = requests.get(url)
r = r.json()
We have a succesful request. Now let;s put it into a dataframe, which can be readily done using the json nesting method I've been using.
In [204]:
nyt_df = pd.DataFrame(r['response']['docs'])
nyt_df
Out[204]:
| abstract | web_url | snippet | lead_paragraph | source | multimedia | headline | keywords | pub_date | document_type | news_desk | section_name | byline | type_of_material | _id | word_count | uri | print_section | print_page | subsection_name | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | So far, 47 police vehicles have been damaged. ... | https://www.nytimes.com/2020/05/31/nyregion/po... | So far, 47 police vehicles have been damaged. ... | On Friday night in the Fort Greene section of ... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Symbol of N.Y.C. Unrest: A Burning P... | [{'name': 'subject', 'value': 'Demonstrations,... | 2020-06-01T00:07:56+0000 | article | Metro | New York | {'original': 'By Azi Paybarah and Nikita Stewa... | News | nyt://article/3022efc6-c3a7-538e-9c33-9396be9a... | 808 | nyt://article/3022efc6-c3a7-538e-9c33-9396be9a... | NaN | NaN | NaN |
| 1 | The death of George Floyd at the hands of the ... | https://www.nytimes.com/video/us/1000000071654... | The death of George Floyd at the hands of the ... | The death of George Floyd at the hands of the ... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': '‘We’re Sick and Tired’: Voices From ... | [{'name': 'persons', 'value': 'Floyd, George (... | 2020-06-01T00:08:23+0000 | multimedia | U.S. | {'original': 'By Katie G. Nelson, Mike Shum, S... | Video | nyt://video/d426a47f-812b-5fee-b0e0-75733d2e8543 | 0 | nyt://video/d426a47f-812b-5fee-b0e0-75733d2e8543 | NaN | NaN | NaN | |
| 2 | The truck can be seen speeding toward proteste... | https://www.nytimes.com/video/us/1000000071663... | The truck can be seen speeding toward proteste... | The truck can be seen speeding toward proteste... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Video Appears to Show a Tanker Truck... | [{'name': 'subject', 'value': 'Demonstrations,... | 2020-06-01T00:22:27+0000 | multimedia | U.S. | {'original': 'By The New York Times', 'person'... | Video | nyt://video/2c42036b-8ef6-5ba5-8375-a85cb7deda40 | 0 | nyt://video/2c42036b-8ef6-5ba5-8375-a85cb7deda40 | NaN | NaN | NaN | |
| 3 | In communities across the country, congregatio... | https://www.nytimes.com/2020/05/31/us/churches... | In communities across the country, congregatio... | SACRAMENTO — With their most vulnerable member... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Amid Riots and a Pandemic, Church At... | [{'name': 'subject', 'value': 'Churches (Build... | 2020-06-01T00:28:09+0000 | article | National | U.S. | {'original': 'By Shawn Hubler and Rick Rojas',... | News | nyt://article/eeb31381-63db-5bf0-8e56-c92b1bc2... | 1524 | nyt://article/eeb31381-63db-5bf0-8e56-c92b1bc2... | A | 14 | NaN |
| 4 | Videos showed officers using batons, tear gas,... | https://www.nytimes.com/2020/05/31/us/police-t... | Videos showed officers using batons, tear gas,... | [Follow our live updates on the George Floyd p... | The New York Times | [] | {'main': 'Facing Protests Over Use of Force, P... | [{'name': 'subject', 'value': 'George Floyd Pr... | 2020-06-01T00:45:13+0000 | article | National | U.S. | {'original': 'By Shaila Dewan and Mike Baker',... | News | nyt://article/25642b46-e875-56a1-b7fd-1be7c34a... | 1539 | nyt://article/25642b46-e875-56a1-b7fd-1be7c34a... | A | 1 | NaN |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 4487 | https://www.nytimes.com/interactive/2020/06/30... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Oklahoma Primary Election Results: F... | [{'name': 'subject', 'value': 'Elections', 'ra... | 2020-06-30T23:50:20+0000 | multimedia | U.S. | U.S. | {'original': '', 'person': [], 'organization':... | nyt://interactive/d954de2f-d0d3-5b03-a828-cb21... | 0 | nyt://interactive/d954de2f-d0d3-5b03-a828-cb21... | NaN | NaN | Elections | ||||
| 4488 | https://www.nytimes.com/interactive/2020/06/30... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Colorado Primary Election Results: T... | [{'name': 'subject', 'value': 'Elections', 'ra... | 2020-06-30T23:50:20+0000 | multimedia | U.S. | U.S. | {'original': '', 'person': [], 'organization':... | nyt://interactive/f486d28b-555d-52ec-b069-fe5e... | 0 | nyt://interactive/f486d28b-555d-52ec-b069-fe5e... | NaN | NaN | Elections | ||||
| 4489 | https://www.nytimes.com/interactive/2020/06/30... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Oklahoma U.S. Senate Primary Electio... | [{'name': 'subject', 'value': 'Elections', 'ra... | 2020-06-30T23:50:20+0000 | multimedia | U.S. | U.S. | {'original': '', 'person': [], 'organization':... | nyt://interactive/f501e742-e85e-52b6-bc00-ca8c... | 0 | nyt://interactive/f501e742-e85e-52b6-bc00-ca8c... | NaN | NaN | Elections | ||||
| 4490 | https://www.nytimes.com/interactive/2020/06/30... | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'Oklahoma Primary Election Results: F... | [{'name': 'subject', 'value': 'Elections', 'ra... | 2020-06-30T23:50:20+0000 | multimedia | U.S. | U.S. | {'original': '', 'person': [], 'organization':... | nyt://interactive/fb8ae641-5cb6-552e-9be4-d73a... | 0 | nyt://interactive/fb8ae641-5cb6-552e-9be4-d73a... | NaN | NaN | Elections | ||||
| 4491 | This file was published in error | https://www.nytimes.com/2020/06/30/theater/thi... | This file was published in error | This file was published in error | The New York Times | [{'rank': 0, 'subtype': 'xlarge', 'caption': N... | {'main': 'This file was published in error', '... | [{'name': 'subject', 'value': 'Theater', 'rank... | 2020-06-30T23:54:48+0000 | article | Culture | Theater | {'original': '', 'person': [], 'organization':... | News | nyt://article/7fc5a07f-d62c-523f-a1f1-8152bf02... | 6 | nyt://article/7fc5a07f-d62c-523f-a1f1-8152bf02... | NaN | NaN | NaN |
4492 rows × 20 columns
There is plenty of data cleanup that I can do on this dataset but that will be the subject of a different post.
Next, I'll search for movie reviews on The Matrix Trilogy that were posted from March 1999 until December 2003. The Matrix Trilogy movies were released on March 1999, May 2003, and December 2003.
In [218]:
api_key = 'jhkMYAlxMsPPYrLatKFLwF6JbQLhME9V'
movie_title = 'matrix'
start_date = '1999-03-01'
end_date = '2003-12-31'
base_url = 'https://api.nytimes.com/svc/movies/v2/reviews/search.json?query='
search_url = base_url + f'{movie_title}&api-key={api_key}&opening-date={start_date}:{end_date}'
print(search_url)
r = requests.get(search_url)
r = r.json()
r
https://api.nytimes.com/svc/movies/v2/reviews/search.json?query=matrix&api-key=jhkMYAlxMsPPYrLatKFLwF6JbQLhME9V&opening-date=1999-03-01:2003-12-31
Out[218]:
{'status': 'OK',
'copyright': 'Copyright (c) 2022 The New York Times Company. All Rights Reserved.',
'has_more': False,
'num_results': 3,
'results': [{'display_title': 'The Matrix Revolutions',
'mpaa_rating': 'R',
'critics_pick': 0,
'byline': 'A. O. SCOTT',
'headline': 'The Game Concludes With Light and Noise',
'summary_short': '"Everything that has a beginning has an end." Yes, thank God — or Neo, or the One, or the Source, or the Oracle, or the Architect, or whoever it was who set this metaphysical action trilogy in motion. "Reloaded," the second installment, was overstuffed and ungainly; this concluding chapter feels padded, with a long, grinding battle sequence in the middle and a whole lot of mumbo-jumbo at the end. Keanu Reeves, perhaps worried that he was showing too much range, has purged himself of all expression apart from a worried frown and a sorrowful grimace, which is mirrored in the face of Carrie-Anne Moss, who returns as Trinity. The salient distinction of the human remnants in Zion, apart from their earth-toned natural-fiber fashions, has been the capacity for love, a word much spoken in this movie. But their ranks, in spite of the presence of fine actors like Laurence Fishburne, Harry Lennix and Harold Perrineau, become more robotic with every passing scene. The sole exception is Jada Pinkett Smith as the daredevil pilot Niobe, who brings a touch of bad-girl B-movie attitude into the humorless subterranean gloom. It is not altogether worthless: the Wachowski Brothers still know how to stage an action sequence, and there are some scenes that have a quiet, mysterious beauty. If they would only let go of their grandiose theological pretensions, they might once again be interesting filmmakers. — A. O. Scott',
'publication_date': '2003-11-05',
'opening_date': '2003-11-05',
'date_updated': '2017-11-02 04:18:03',
'link': {'type': 'article',
'url': 'https://www.nytimes.com/2003/11/05/movies/film-review-the-game-concludes-with-light-and-noise.html',
'suggested_link_text': 'Read the New York Times Review of The Matrix Revolutions'},
'multimedia': None},
{'display_title': 'The Matrix Reloaded',
'mpaa_rating': 'R',
'critics_pick': 0,
'byline': 'Elvis Mitchell',
'headline': 'An Idealized World and a Troubled Hero',
'summary_short': "The mastery of filmmaking hasn't abated in "Reloaded" — the Wachowskis are as suited to the task of guiding this sequel to their surprise 1999 hit as their unsure hero, Neo (Keanu Reeves), is to saving the world. Though the staging of the action sequences is as viciously elegant as you've been primed to expect, there's a dispiriting more-of-the-same aspect to the picture. The mentor Morpheus (Laurence Fishburne) and the virtual villain Agent Smith (Hugo Weaving) also return, and a whole new group of characters surface as Neo, his lover, Trinity (Carrie-Anne Moss), and the crew continue to fight the oppressive forces threatening to eliminate humanity and the hidden city of Zion. As the middle film of a trilogy, "Reloaded" seeks to increase the emotional stakes — which it doesn't quite accomplish — while leaving enough of a cliffhanger that audiences will be bedeviled enough to flock back into the tents for the last portion. The directors pay their audiences the unlikely compliment of taking them seriously. Unfortunately, they take themselves too seriously. However, "Reloaded" has one of the most excitingly subversive and radical points of view ever seen in a major motion picture — a postmodern purview that accords philosophical ideals from people of color equal weight. This includes the warmhearted spotlight given to Gloria Foster, who returns as Oracle, the source of all knowledge for Neo. — Elvis Mitchell",
'publication_date': '2003-05-14',
'opening_date': '2003-05-15',
'date_updated': '2017-11-02 04:18:02',
'link': {'type': 'article',
'url': 'https://www.nytimes.com/2003/05/14/movies/film-review-an-idealized-world-and-a-troubled-hero.html',
'suggested_link_text': 'Read the New York Times Review of The Matrix Reloaded'},
'multimedia': None},
{'display_title': 'The Matrix',
'mpaa_rating': 'R',
'critics_pick': 0,
'byline': 'Janet Maslin',
'headline': 'Matrix, The (Movie) ',
'summary_short': 'Hacker as action messiah. Enough visual bravado to sustain steady suspense. Cult classic, virtually speaking.',
'publication_date': '1999-03-31',
'opening_date': '1999-03-31',
'date_updated': '2017-11-02 04:17:57',
'link': {'type': 'article',
'url': 'https://www.nytimes.com/1999/03/31/movies/film-review-the-reality-is-all-virtual-and-densely-complicated.html',
'suggested_link_text': 'Read the New York Times Review of The Matrix'},
'multimedia': None}]}
In [222]:
matrix_reviews_df = pd.DataFrame(r['results'])
matrix_reviews_df
Out[222]:
| display_title | mpaa_rating | critics_pick | byline | headline | summary_short | publication_date | opening_date | date_updated | link | multimedia | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | The Matrix Revolutions | R | 0 | A. O. SCOTT | The Game Concludes With Light and Noise | "Everything that has a beginning has an e... | 2003-11-05 | 2003-11-05 | 2017-11-02 04:18:03 | {'type': 'article', 'url': 'https://www.nytime... | None |
| 1 | The Matrix Reloaded | R | 0 | Elvis Mitchell | An Idealized World and a Troubled Hero | The mastery of filmmaking hasn't abated in &qu... | 2003-05-14 | 2003-05-15 | 2017-11-02 04:18:02 | {'type': 'article', 'url': 'https://www.nytime... | None |
| 2 | The Matrix | R | 0 | Janet Maslin | Matrix, The (Movie) | Hacker as action messiah. Enough visual bravad... | 1999-03-31 | 1999-03-31 | 2017-11-02 04:17:57 | {'type': 'article', 'url': 'https://www.nytime... | None |
Sweet! I could start generating a database on movie reviews to do things like sentiment analysis for a future project :]
Getting Memes from ImgFlip
So anyone that knows me even a little knows that I LOVE MEMES. So it is only natural that at some point or another I'd be interested in making my own meme routine. imgFlip was a nice API that allows you to get memes in a pretty easy way. You can find info on the ImgFlip API here: https://imgflip.com/api.
Let's start by requesting a json with the 100 meme templates that were most captioned over the last 30 days.
In [84]:
url = "https://api.imgflip.com/get_memes"
memes = requests.get(url)
meme_dict = memes.json()
meme_dict
Out[84]:
{'success': True,
'data': {'memes': [{'id': '181913649',
'name': 'Drake Hotline Bling',
'url': 'https://i.imgflip.com/30b1gx.jpg',
'width': 1200,
'height': 1200,
'box_count': 2},
{'id': '87743020',
'name': 'Two Buttons',
'url': 'https://i.imgflip.com/1g8my4.jpg',
'width': 600,
'height': 908,
'box_count': 3},
{'id': '112126428',
'name': 'Distracted Boyfriend',
'url': 'https://i.imgflip.com/1ur9b0.jpg',
'width': 1200,
'height': 800,
'box_count': 3},
{'id': '131087935',
'name': 'Running Away Balloon',
'url': 'https://i.imgflip.com/261o3j.jpg',
'width': 761,
'height': 1024,
'box_count': 5},
{'id': '129242436',
'name': 'Change My Mind',
'url': 'https://i.imgflip.com/24y43o.jpg',
'width': 482,
'height': 361,
'box_count': 2},
{'id': '217743513',
'name': 'UNO Draw 25 Cards',
'url': 'https://i.imgflip.com/3lmzyx.jpg',
'width': 500,
'height': 494,
'box_count': 2},
{'id': '124822590',
'name': 'Left Exit 12 Off Ramp',
'url': 'https://i.imgflip.com/22bdq6.jpg',
'width': 804,
'height': 767,
'box_count': 3},
{'id': '247375501',
'name': 'Buff Doge vs. Cheems',
'url': 'https://i.imgflip.com/43a45p.png',
'width': 937,
'height': 720,
'box_count': 4},
{'id': '222403160',
'name': 'Bernie I Am Once Again Asking For Your Support',
'url': 'https://i.imgflip.com/3oevdk.jpg',
'width': 750,
'height': 750,
'box_count': 2},
{'id': '131940431',
'name': "Gru's Plan",
'url': 'https://i.imgflip.com/26jxvz.jpg',
'width': 700,
'height': 449,
'box_count': 4},
{'id': '4087833',
'name': 'Waiting Skeleton',
'url': 'https://i.imgflip.com/2fm6x.jpg',
'width': 298,
'height': 403,
'box_count': 2},
{'id': '135256802',
'name': 'Epic Handshake',
'url': 'https://i.imgflip.com/28j0te.jpg',
'width': 900,
'height': 645,
'box_count': 3},
{'id': '80707627',
'name': 'Sad Pablo Escobar',
'url': 'https://i.imgflip.com/1c1uej.jpg',
'width': 720,
'height': 709,
'box_count': 3},
{'id': '438680',
'name': 'Batman Slapping Robin',
'url': 'https://i.imgflip.com/9ehk.jpg',
'width': 400,
'height': 387,
'box_count': 2},
{'id': '93895088',
'name': 'Expanding Brain',
'url': 'https://i.imgflip.com/1jwhww.jpg',
'width': 857,
'height': 1202,
'box_count': 4},
{'id': '252600902',
'name': 'Always Has Been',
'url': 'https://i.imgflip.com/46e43q.png',
'width': 960,
'height': 540,
'box_count': 2},
{'id': '188390779',
'name': 'Woman Yelling At Cat',
'url': 'https://i.imgflip.com/345v97.jpg',
'width': 680,
'height': 438,
'box_count': 2},
{'id': '97984',
'name': 'Disaster Girl',
'url': 'https://i.imgflip.com/23ls.jpg',
'width': 500,
'height': 375,
'box_count': 2},
{'id': '178591752',
'name': 'Tuxedo Winnie The Pooh',
'url': 'https://i.imgflip.com/2ybua0.png',
'width': 800,
'height': 582,
'box_count': 2},
{'id': '110163934',
'name': "I Bet He's Thinking About Other Women",
'url': 'https://i.imgflip.com/1tl71a.jpg',
'width': 1654,
'height': 930,
'box_count': 2},
{'id': '1035805',
'name': 'Boardroom Meeting Suggestion',
'url': 'https://i.imgflip.com/m78d.jpg',
'width': 500,
'height': 649,
'box_count': 4},
{'id': '102156234',
'name': 'Mocking Spongebob',
'url': 'https://i.imgflip.com/1otk96.jpg',
'width': 502,
'height': 353,
'box_count': 2},
{'id': '91538330',
'name': 'X, X Everywhere',
'url': 'https://i.imgflip.com/1ihzfe.jpg',
'width': 2118,
'height': 1440,
'box_count': 2},
{'id': '79132341',
'name': 'Bike Fall',
'url': 'https://i.imgflip.com/1b42wl.jpg',
'width': 500,
'height': 680,
'box_count': 3},
{'id': '119139145',
'name': 'Blank Nut Button',
'url': 'https://i.imgflip.com/1yxkcp.jpg',
'width': 600,
'height': 446,
'box_count': 2},
{'id': '101470',
'name': 'Ancient Aliens',
'url': 'https://i.imgflip.com/26am.jpg',
'width': 500,
'height': 437,
'box_count': 2},
{'id': '61579',
'name': 'One Does Not Simply',
'url': 'https://i.imgflip.com/1bij.jpg',
'width': 568,
'height': 335,
'box_count': 2},
{'id': '195515965',
'name': 'Clown Applying Makeup',
'url': 'https://i.imgflip.com/38el31.jpg',
'width': 750,
'height': 798,
'box_count': 4},
{'id': '27813981',
'name': 'Hide the Pain Harold',
'url': 'https://i.imgflip.com/gk5el.jpg',
'width': 480,
'height': 601,
'box_count': 2},
{'id': '216951317',
'name': 'Guy Holding Cardboard Sign',
'url': 'https://i.imgflip.com/3l60ph.jpg',
'width': 700,
'height': 702,
'box_count': 2},
{'id': '180190441',
'name': "They're The Same Picture",
'url': 'https://i.imgflip.com/2za3u1.jpg',
'width': 1363,
'height': 1524,
'box_count': 3},
{'id': '148909805',
'name': 'Monkey Puppet',
'url': 'https://i.imgflip.com/2gnnjh.jpg',
'width': 923,
'height': 768,
'box_count': 2},
{'id': '100777631',
'name': 'Is This A Pigeon',
'url': 'https://i.imgflip.com/1o00in.jpg',
'width': 1587,
'height': 1425,
'box_count': 3},
{'id': '161865971',
'name': 'Marked Safe From',
'url': 'https://i.imgflip.com/2odckz.jpg',
'width': 618,
'height': 499,
'box_count': 2},
{'id': '226297822',
'name': 'Panik Kalm Panik',
'url': 'https://i.imgflip.com/3qqcim.png',
'width': 640,
'height': 881,
'box_count': 3},
{'id': '114585149',
'name': 'Inhaling Seagull',
'url': 'https://i.imgflip.com/1w7ygt.jpg',
'width': 1269,
'height': 2825,
'box_count': 4},
{'id': '55311130',
'name': 'This Is Fine',
'url': 'https://i.imgflip.com/wxica.jpg',
'width': 580,
'height': 282,
'box_count': 2},
{'id': '124055727',
'name': "Y'all Got Any More Of That",
'url': 'https://i.imgflip.com/21uy0f.jpg',
'width': 600,
'height': 471,
'box_count': 2},
{'id': '3218037',
'name': "This Is Where I'd Put My Trophy If I Had One",
'url': 'https://i.imgflip.com/1wz1x.jpg',
'width': 300,
'height': 418,
'box_count': 2},
{'id': '89370399',
'name': 'Roll Safe Think About It',
'url': 'https://i.imgflip.com/1h7in3.jpg',
'width': 702,
'height': 395,
'box_count': 2},
{'id': '134797956',
'name': 'American Chopper Argument',
'url': 'https://i.imgflip.com/2896ro.jpg',
'width': 640,
'height': 1800,
'box_count': 5},
{'id': '123999232',
'name': 'The Scroll Of Truth',
'url': 'https://i.imgflip.com/21tqf4.jpg',
'width': 1280,
'height': 1236,
'box_count': 2},
{'id': '135678846',
'name': 'Who Killed Hannibal',
'url': 'https://i.imgflip.com/28s2gu.jpg',
'width': 1280,
'height': 1440,
'box_count': 3},
{'id': '99683372',
'name': 'Sleeping Shaq',
'url': 'https://i.imgflip.com/1nck6k.jpg',
'width': 640,
'height': 631,
'box_count': 2},
{'id': '28251713',
'name': 'Oprah You Get A',
'url': 'https://i.imgflip.com/gtj5t.jpg',
'width': 620,
'height': 465,
'box_count': 2},
{'id': '21735',
'name': 'The Rock Driving',
'url': 'https://i.imgflip.com/grr.jpg',
'width': 568,
'height': 700,
'box_count': 2},
{'id': '155067746',
'name': 'Surprised Pikachu',
'url': 'https://i.imgflip.com/2kbn1e.jpg',
'width': 1893,
'height': 1893,
'box_count': 3},
{'id': '259237855',
'name': 'Laughing Leo',
'url': 'https://i.imgflip.com/4acd7j.png',
'width': 470,
'height': 470,
'box_count': 2},
{'id': '6235864',
'name': 'Finding Neverland',
'url': 'https://i.imgflip.com/3pnmg.jpg',
'width': 423,
'height': 600,
'box_count': 3},
{'id': '61556',
'name': 'Grandma Finds The Internet',
'url': 'https://i.imgflip.com/1bhw.jpg',
'width': 640,
'height': 480,
'box_count': 2},
{'id': '132769734',
'name': 'Hard To Swallow Pills',
'url': 'https://i.imgflip.com/271ps6.jpg',
'width': 680,
'height': 979,
'box_count': 2},
{'id': '61520',
'name': 'Futurama Fry',
'url': 'https://i.imgflip.com/1bgw.jpg',
'width': 552,
'height': 414,
'box_count': 2},
{'id': '84341851',
'name': 'Evil Kermit',
'url': 'https://i.imgflip.com/1e7ql7.jpg',
'width': 700,
'height': 325,
'box_count': 2},
{'id': '175540452',
'name': 'Unsettled Tom',
'url': 'https://i.imgflip.com/2wifvo.jpg',
'width': 680,
'height': 550,
'box_count': 2},
{'id': '101288',
'name': 'Third World Skeptical Kid',
'url': 'https://i.imgflip.com/265k.jpg',
'width': 426,
'height': 426,
'box_count': 2},
{'id': '5496396',
'name': 'Leonardo Dicaprio Cheers',
'url': 'https://i.imgflip.com/39t1o.jpg',
'width': 600,
'height': 400,
'box_count': 2},
{'id': '196652226',
'name': 'Spongebob Ight Imma Head Out',
'url': 'https://i.imgflip.com/392xtu.jpg',
'width': 822,
'height': 960,
'box_count': 2},
{'id': '91545132',
'name': 'Trump Bill Signing',
'url': 'https://i.imgflip.com/1ii4oc.jpg',
'width': 1866,
'height': 1529,
'box_count': 2},
{'id': '8072285',
'name': 'Doge',
'url': 'https://i.imgflip.com/4t0m5.jpg',
'width': 620,
'height': 620,
'box_count': 5},
{'id': '61544',
'name': 'Success Kid',
'url': 'https://i.imgflip.com/1bhk.jpg',
'width': 500,
'height': 500,
'box_count': 2},
{'id': '61532',
'name': 'The Most Interesting Man In The World',
'url': 'https://i.imgflip.com/1bh8.jpg',
'width': 550,
'height': 690,
'box_count': 2},
{'id': '563423',
'name': 'That Would Be Great',
'url': 'https://i.imgflip.com/c2qn.jpg',
'width': 526,
'height': 440,
'box_count': 2},
{'id': '14371066',
'name': 'Star Wars Yoda',
'url': 'https://i.imgflip.com/8k0sa.jpg',
'width': 620,
'height': 714,
'box_count': 2},
{'id': '29617627',
'name': 'Look At Me',
'url': 'https://i.imgflip.com/hmt3v.jpg',
'width': 300,
'height': 300,
'box_count': 2},
{'id': '61546',
'name': 'Brace Yourselves X is Coming',
'url': 'https://i.imgflip.com/1bhm.jpg',
'width': 622,
'height': 477,
'box_count': 2},
{'id': '101716',
'name': 'Yo Dawg Heard You',
'url': 'https://i.imgflip.com/26hg.jpg',
'width': 500,
'height': 323,
'box_count': 2},
{'id': '163573',
'name': 'Imagination Spongebob',
'url': 'https://i.imgflip.com/3i7p.jpg',
'width': 500,
'height': 366,
'box_count': 2},
{'id': '460541',
'name': 'Jack Sparrow Being Chased',
'url': 'https://i.imgflip.com/9vct.jpg',
'width': 500,
'height': 375,
'box_count': 2},
{'id': '61539',
'name': 'First World Problems',
'url': 'https://i.imgflip.com/1bhf.jpg',
'width': 552,
'height': 367,
'box_count': 2},
{'id': '1367068',
'name': 'I Should Buy A Boat Cat',
'url': 'https://i.imgflip.com/tau4.jpg',
'width': 500,
'height': 368,
'box_count': 2},
{'id': '24557067',
'name': 'Afraid To Ask Andy',
'url': 'https://i.imgflip.com/emccr.jpg',
'width': 620,
'height': 608,
'box_count': 2},
{'id': '6531067',
'name': 'See Nobody Cares',
'url': 'https://i.imgflip.com/3vzej.jpg',
'width': 620,
'height': 676,
'box_count': 2},
{'id': '285870',
'name': 'Squidward',
'url': 'https://i.imgflip.com/64ku.jpg',
'width': 500,
'height': 750,
'box_count': 2},
{'id': '61580',
'name': 'Too Damn High',
'url': 'https://i.imgflip.com/1bik.jpg',
'width': 420,
'height': 316,
'box_count': 2},
{'id': '142921050',
'name': 'Car Salesman Slaps Roof Of Car',
'url': 'https://i.imgflip.com/2d3al6.jpg',
'width': 800,
'height': 450,
'box_count': 2},
{'id': '29562797',
'name': "I'm The Captain Now",
'url': 'https://i.imgflip.com/hlmst.jpg',
'width': 478,
'height': 350,
'box_count': 2},
{'id': '61585',
'name': 'Bad Luck Brian',
'url': 'https://i.imgflip.com/1bip.jpg',
'width': 475,
'height': 562,
'box_count': 2},
{'id': '21604248',
'name': 'Mugatu So Hot Right Now',
'url': 'https://i.imgflip.com/cv1y0.jpg',
'width': 620,
'height': 497,
'box_count': 2},
{'id': '101511',
'name': "Don't You Squidward",
'url': 'https://i.imgflip.com/26br.jpg',
'width': 500,
'height': 333,
'box_count': 2},
{'id': '61533',
'name': 'X All The Y',
'url': 'https://i.imgflip.com/1bh9.jpg',
'width': 500,
'height': 355,
'box_count': 2},
{'id': '922147',
'name': 'Laughing Men In Suits',
'url': 'https://i.imgflip.com/jrj7.jpg',
'width': 500,
'height': 333,
'box_count': 2},
{'id': '8279814',
'name': 'Cute Cat',
'url': 'https://i.imgflip.com/4xgqu.jpg',
'width': 480,
'height': 532,
'box_count': 2},
{'id': '56225174',
'name': 'Be Like Bill',
'url': 'https://i.imgflip.com/xh3me.jpg',
'width': 913,
'height': 907,
'box_count': 4},
{'id': '7183956',
'name': 'Oprah You Get A Car Everybody Gets A Car',
'url': 'https://i.imgflip.com/49z6c.jpg',
'width': 620,
'height': 1004,
'box_count': 4},
{'id': '101910402',
'name': 'Who Would Win?',
'url': 'https://i.imgflip.com/1ooaki.jpg',
'width': 802,
'height': 500,
'box_count': 2},
{'id': '183518946',
'name': 'Blank Transparent Square',
'url': 'https://i.imgflip.com/319g4i.png',
'width': 1000,
'height': 1000,
'box_count': 2},
{'id': '61527',
'name': 'Y U No',
'url': 'https://i.imgflip.com/1bh3.jpg',
'width': 500,
'height': 500,
'box_count': 2},
{'id': '1464444',
'name': 'Happy Star Congratulations',
'url': 'https://i.imgflip.com/vdz0.jpg',
'width': 450,
'height': 292,
'box_count': 4},
{'id': '89655',
'name': 'Uncle Sam',
'url': 'https://i.imgflip.com/1x6f.jpg',
'width': 620,
'height': 833,
'box_count': 2},
{'id': '176908',
'name': 'Shut Up And Take My Money Fry',
'url': 'https://i.imgflip.com/3si4.jpg',
'width': 500,
'height': 281,
'box_count': 2},
{'id': '1202623',
'name': 'Keep Calm And Carry On Red',
'url': 'https://i.imgflip.com/pry7.jpg',
'width': 500,
'height': 704,
'box_count': 2},
{'id': '101287',
'name': 'Third World Success Kid',
'url': 'https://i.imgflip.com/265j.jpg',
'width': 500,
'height': 500,
'box_count': 2},
{'id': '16464531',
'name': "But That's None Of My Business",
'url': 'https://i.imgflip.com/9sw43.jpg',
'width': 600,
'height': 600,
'box_count': 2},
{'id': '71428573',
'name': 'Say it Again, Dexter',
'url': 'https://i.imgflip.com/16iyn1.jpg',
'width': 698,
'height': 900,
'box_count': 2},
{'id': '61516',
'name': 'Philosoraptor',
'url': 'https://i.imgflip.com/1bgs.jpg',
'width': 500,
'height': 500,
'box_count': 2},
{'id': '371382',
'name': 'Simba Shadowy Place',
'url': 'https://i.imgflip.com/7yk6.jpg',
'width': 363,
'height': 720,
'box_count': 2},
{'id': '28034788',
'name': 'Marvel Civil War 1',
'url': 'https://i.imgflip.com/govs4.jpg',
'width': 423,
'height': 734,
'box_count': 2},
{'id': '405658',
'name': 'Grumpy Cat',
'url': 'https://i.imgflip.com/8p0a.jpg',
'width': 500,
'height': 617,
'box_count': 2},
{'id': '61581',
'name': 'Put It Somewhere Else Patrick',
'url': 'https://i.imgflip.com/1bil.jpg',
'width': 343,
'height': 604,
'box_count': 2},
{'id': '109765',
'name': "I'll Just Wait Here",
'url': 'https://i.imgflip.com/2cp1.jpg',
'width': 491,
'height': 550,
'box_count': 2}]}}
Another, nested json. No worries though, we know how to deal with them. Let's put these results into a dataframe
In [89]:
meme_df = pd.DataFrame(meme_dict['data']['memes'])
meme_df
Out[89]:
| id | name | url | width | height | box_count | |
|---|---|---|---|---|---|---|
| 0 | 181913649 | Drake Hotline Bling | https://i.imgflip.com/30b1gx.jpg | 1200 | 1200 | 2 |
| 1 | 87743020 | Two Buttons | https://i.imgflip.com/1g8my4.jpg | 600 | 908 | 3 |
| 2 | 112126428 | Distracted Boyfriend | https://i.imgflip.com/1ur9b0.jpg | 1200 | 800 | 3 |
| 3 | 131087935 | Running Away Balloon | https://i.imgflip.com/261o3j.jpg | 761 | 1024 | 5 |
| 4 | 129242436 | Change My Mind | https://i.imgflip.com/24y43o.jpg | 482 | 361 | 2 |
| ... | ... | ... | ... | ... | ... | ... |
| 95 | 371382 | Simba Shadowy Place | https://i.imgflip.com/7yk6.jpg | 363 | 720 | 2 |
| 96 | 28034788 | Marvel Civil War 1 | https://i.imgflip.com/govs4.jpg | 423 | 734 | 2 |
| 97 | 405658 | Grumpy Cat | https://i.imgflip.com/8p0a.jpg | 500 | 617 | 2 |
| 98 | 61581 | Put It Somewhere Else Patrick | https://i.imgflip.com/1bil.jpg | 343 | 604 | 2 |
| 99 | 109765 | I'll Just Wait Here | https://i.imgflip.com/2cp1.jpg | 491 | 550 | 2 |
100 rows × 6 columns
Cool beans! Now we can do stuff with them. I'll take the url column and then convert it into a list. Then, I'll display the first 10 memes in the dataframe:
In [126]:
from PIL import Image #I'm using PIL because the urlopen method I was using with OpenCV
#was giving me a HTTP 403 error
meme_url = meme_df['url'].tolist()
fig, axs = plt.subplots(1, 10, figsize=(30,5))
i = 0
for meme in meme_url:
im = Image.open(requests.get(meme, stream=True).raw)
axs[i].imshow(im)
i+=1
if i > 9:
break
plt.show()
Cool! Now we can download meme templates at our hearts content! I have some ideas for projects with this :]
Getting Real Estate Information from Zillow
In [ ]:
Getting Tweet Information from Twitter
In [ ]:
Getting Post Information from Reddit
You can get a variety of information from the massive post database that's Reddit. You can look at their docs in more detail here: https://praw.readthedocs.io/en/stable/index.html. You will need to create an API with Reddit in order to get an API ID, a Client SECRET, and also have a Reddit username. You should also install thepraw wrapper function for the Reddit API to facilitate operations.
In [228]:
import praw #This is to access Reddit data
cid = 'PGDDWn6zsy1RBZ6E3ExWsA'
secret = 'AmtIRyYfqGF4ak430RIyU83GUK5ymg'
username = 'anisotropy55'
user_agent = "anisotropy55"
password = "YY7MJnCMgKsZcxU"
#This is for read only operations
reddit = praw.Reddit(client_id = cid,client_secret = secret,user_agent = username,)
print(reddit.read_only)
True
Let's start by getting the top 10 posts in r/programming
In [227]:
for submission in reddit.subreddit("programming").hot(limit=10):
print(submission.title)
We need young programmers; we need old programmers After two years of self-studying programming, I made an AI-powered essay writing tool/website that helps you organize ideas and craft better arguments Coming Soon: Reddit Developer Platform I have complicated feelings about TDD Agile Projects Have Become Waterfall Projects With Sprints Removing HTTP/2 Server Push from Chrome This IBM security article will implement a simple authentication mechanism to demonstrate how developers can write their own Java based authentication mechanism. Browser extension Surfingkeys, which implements vim-style shortcuts in browsers, appears to be bundling a search hijacker. This does not appear in source Julia 1.8 Teams, Zoom and Meeting Overload Are Slowing Down Development and Disrupting Developers
If we navigate to r/programming then we'll see that our 10 produced posts match what is actually posted. Great!
Now I'll see if I can make something a bit more involved by making an Authorized Reddit instance. This would enable me to use my code in whatever I, as a human could.
In [229]:
reddit = praw.Reddit(client_id = cid, client_secret = secret,
password = password, user_agent = user_agent, username = username,)
print(reddit.read_only)
False
Let's get a Subreddit. Let's say r/memes. For r/memes we'll retrieve the display name and the title. The greyed out comment is to get the subreddit description.
In [232]:
# assume you have a praw.Reddit instance bound to variable `reddit`
subreddit = reddit.subreddit("memes")
print(subreddit.display_name)
print(subreddit.title)
#print(subreddit.description)
memes /r/Memes the original since 2008
Now that we have a subreddit instance, let's get some post information within that subreddit! I'll place the search results into lists that can later go into a pandas dataframe by zipping them together :]
In [240]:
titles = []
scores = []
ids = []
urls = []
# assume you have a Subreddit instance bound to variable `subreddit`
for submission in subreddit.hot(limit=10):
#print(submission.title)
#print(submission.score)
#print(submission.id)
#print(submission.url)
titles.append(submission.title)
scores.append(submission.score)
ids.append(submission.id)
urls.append(submission.url)
col_names =['title','score','id','url']
posts = list(zip(titles, scores, ids, urls))
reddit_df = pd.DataFrame(posts, columns = col_names)
reddit_df
Out[240]:
| title | score | id | url | |
|---|---|---|---|---|
| 0 | Gifs have been enabled in the comments for thi... | 478 | wfd0ji | https://www.reddit.com/r/memes/comments/wfd0ji... |
| 1 | Pure anarchy | 18127 | wrstv0 | https://i.redd.it/us9coxh00ji91.jpg |
| 2 | Makes Sense | 4270 | wrx6yz | https://i.redd.it/0968es6ovji91.jpg |
| 3 | I'm the chosen one | 8817 | wru9r9 | https://i.redd.it/a0uxvoycaji91.gif |
| 4 | The urban legend of Wanksy | 9069 | wrt1f5 | https://i.redd.it/4f343t4k1ji91.jpg |
| 5 | double the d's | 979 | ws3ew6 | https://i.redd.it/274x770jbli91.jpg |
| 6 | Spanking then monkey amiright | 2424 | wrwrwo | https://i.redd.it/nmiidr3hsji91.gif |
| 7 | You gotta fight! For your right! To POTTY! - [... | 1635 | wryro5 | https://i.redd.it/o4z5zqn08ki91.jpg |
| 8 | That doesn't narrow it down... | 11375 | wro4c3 | https://i.redd.it/80223ljx1ii91.png |
| 9 | truly remarkable | 2098 | wrvpzn | https://i.redd.it/249qs5prkji91.jpg |
We can get more information like instances of specific Redditors, Comments and Submissions. The data should be able to be handled in a similar way as above so it's a simple matter of formatting the query correctly. I'll see what project I can carry out using this.
Conclusions
Hopefully this post was helpful to you in case you are stuck navigating through the world of APIs. Getting data and cleaning it up so that it is ready to work to do analysis on is perhaps not the most exciting part of doing data science but I definitely improved my programming skills (especially in terms of working with json files and pandas dataframes) as a result of trying to process all these different databases.I included a variety of ways to do things here ranging from performing simple, single queries to parsing data from deeply nested json files to performing multiple queries to compile sufficiently large datasets.
Happy data gathering!