Assignment 4 - Hypothesis Testing

This assignment requires more individual learning than previous assignments - you are encouraged to check out the pandas documentation to find functions or methods you might not have used yet, or ask questions on Stack Overflow and tag them as pandas and python related. And of course, the discussion forums are open for interaction with your peers and the course staff.

Definitions:

• A quarter is a specific three month period, Q1 is January through March, Q2 is April through June, Q3 is July through September, Q4 is October through December.
• A recession is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.
• A recession bottom is the quarter within a recession which had the lowest GDP.
• A university town is a city which has a high percentage of university students compared to the total population of the city.

Hypothesis: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the mean price of houses in university towns the quarter before the recession starts compared to the recession bottom. (price_ratio=quarter_before_recession/recession_bottom)

The following data files are available for this assignment:

• From the Zillow research data site there is housing data for the United States. In particular the datafile for all homes at a city level, City_Zhvi_AllHomes.csv, has median home sale prices at a fine grained level.
• From the Wikipedia page on college towns is a list of university towns in the United States which has been copy and pasted into the file university_towns.txt.
• From Bureau of Economic Analysis, US Department of Commerce, the GDP over time of the United States in current dollars (use the chained value in 2009 dollars), in quarterly intervals, in the file gdplev.xls. For this assignment, only look at GDP data from the first quarter of 2000 onward.

Each function in this assignment below is worth 10%, with the exception of run_ttest(), which is worth 50%.

Q1：

def get_list_of_university_towns():
    '''Returns a DataFrame of towns and the states they are in from the 
    university_towns.txt list. The format of the DataFrame should be:
    DataFrame( [ ["Michigan", "Ann Arbor"], ["Michigan", "Yipsilanti"] ], 
    columns=["State", "RegionName"]  )
    
    The following cleaning needs to be done:

    1. For "State", removing characters from "[" to the end.
    2. For "RegionName", when applicable, removing every character from " (" to the end.
    3. Depending on how you read the data, you may need to remove newline character '\n'. '''
    
    df = pd.read_table('university_towns.txt', header=None, names=['data'])
    bool_df = df['data'].str.contains('[edit]', regex=False)
    university_array = []
    for index, value in bool_df.items():
        if value:
            state = df.loc[index].values[0]
        else:
            region = df.loc[index].values[0]
            university_array.append([state, region])
    university_df = pd.DataFrame(university_array, columns=['State', 'RegionName'])
    university_df['State'] = university_df['State'].str.replace('\[edit.*', '')
    university_df['RegionName'] = university_df['RegionName'].str.replace(' \(.*', '')
    
    # 坑。。又说让转换成缩写，提交答案却还是要全名的
    # 后记：是 Q5 用来做转换的。。
#     states_swap = {value:key for key, value in states.items()}
    
#     university_df['State'].replace(states_swap, inplace=True)
    
    return university_df

get_list_of_university_towns()

# A recession is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth
gdp_df = pd.read_excel('gdplev.xls', header=5, usecols=[3,4,5], names=['Quarter', 'GDP current', 'GDP chained 2009']).set_index('Quarter').dropna()

gdp_df            

# A recession is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.
# A recession bottom is the quarter within a recession which had the lowest GDP.
def cal_recession():
    result = {
   
        'start': [],
        'end': [],
        'bottom': []
    }
    
    # 下降计数
    down_counter = 2
    down_start_index = ''
    # 上涨计数
    up_counter = 0
    
    bottom = 99999999
    bottom_index = ''
    
    flag = False  # 进入recession的征兆
    
    last_value = gdp_df.iloc[0]['GDP chained 2009'] # 坑啊，要认真审题！ 题干中说明了要用 chained 2009 字段来计算
    print('初始值：{}'.format(last_value))
    
    for index, item in gdp_df.iterrows():
        value = item['GDP chained 2009']
        print('{} {}'.format(index, value))
        # 环比下降
        if value < last_value:
            print('-------- 开始下降 ！！！')
            if down_counter == 2: # 记录可能进入recession的起始时间
                down_start_index = index
            down_counter -= 1
            if value < bottom:
                bottom = value
                bottom_index = index
        else:
            down_counter = 2  # 当前值不小于上季度值，则重置下降标记
        # 连续两季度下降，进入recession的征兆
        if down_counter == 0:
            print