Survey of Income and Program Participation (SIPP)

analyze the survey of income and program participation (sipp) with r if the census bureau's budget was gutted and only one complex sample survey survived, pray it's the survey of income and program participation (sipp). it's giant. it's rich with variables. it's monthly. it follows households over three, four, now five year panels. the congressional budget office uses it for their health insurance simulation . analysts read that sipp has person-month files, get scurred, and retreat to inferior options. the american community survey may be the mount everest of survey data, but sipp is most certainly the amazon. questions swing wild and free through the jungle canopy i mean core data dictionary. legend has it that there are still species of topical module variables that scientists like you have yet to analyze. ponce de león would've loved it here. ponce. what a name. what a guy. the sipp 2008 panel data started from a sample of 105,663 individuals in 42,030 households. once the sample gets drawn, the census bureau surveys one-fourth of the respondents every four months, over f our or five years (panel durations vary). you absolutely must read and understand pdf pages 3, 4, and 5 of this document before starting any analysis (start at the header 'waves and rotation groups'). if you don't comprehend what's going on, try their survey design tutorial. since sipp collects information from respondents regarding every month over the duration of the panel, you'll need to be hyper-aware of whether you want your results to be point-in-time, annualized, or specific to some other period. the analysis scripts below provide examples of each. at every four-month interview point, every respondent answers every core question for the previous four months. after that, wave-specific addenda (called topical modules) get asked, but generally only regarding a single prior month. to repeat: core wave files contain four records per person, topical modules contain one. if you stacked every core wave, you would have one record per person per month for the duration o f the panel. mmmassive. ~100,000 respondents x 12 months x ~4 years. have an analysis plan before you start writing code so you extract exactly what you need, nothing more. better yet, modify something of mine. cool? this new github repository contains eight, you read me, eight scripts: 1996 panel - download and create database.R 2001 panel - download and create database.R 2004 panel - download and create database.R 2008 panel - download and create database.R since some variables are character strings in one file and integers in anoth er, initiate an r function to harmonize variable class inconsistencies in the sas importation scripts properly handle the parentheses seen in a few of the sas importation scripts, because the SAScii package currently does not create an rsqlite database, initiate a variant of the `read.SAScii` function that imports ascii data directly into a sql database (.db) download each microdata file - weights, topical modules, everything - then read 'em into sql 2008 panel - full year analysis examples.R< br /> define which waves and specific variables to pull into ram, based on the year chosen loop through each of twelve months, constructing a single-year temporary table inside the database read that twelve-month file into working memory, then save it for faster loading later if you like read the main and replicate weights columns into working memory too, merge everything construct a few annualized and demographic columns using all twelve months' worth of information construct a replicate-weighted complex sample design with a fay's adjustment factor of one-half, again save it for faster loading later, only if you're so inclined reproduce census-publish ed statistics, not precisely (due to topcoding described here on pdf page 19) 2008 panel - point-in-time analysis examples.R define which wave(s) and specific variables to pull into ram, based on the calendar month chosen read that interview point (srefmon)- or calendar month (rhcalmn)-based file into working memory read the topical module and replicate weights files into working memory too, merge it like you mean it construct a few new, exciting variables using both core and topical module questions construct a replicate-weighted complex sample design with a fay's adjustment factor of one-half reproduce census-published statistics, not exactly cuz the authors of this brief used the generalized variance formula (gvf) to calculate the margin of error - see pdf page 4 for more detail - the friendly statisticians at census recommend using the replicate weights whenever possible. oh hayy, now it is. 2008 panel - median value of household assets.R define which wave(s) and spe cific variables to pull into ram, based on the topical module chosen read the topical module and replicate weights files into working memory too, merge once again construct a replicate-weighted complex sample design with a fay's adjustment factor of one-half reproduce census-published s tatistics, not exactly due to topcoding (read more about topcoding by searching this and that user guide for, well, `topcoding`). huh. so topcoding affects asset statistics. replicate census poverty statistics.R load a single wave of data limit the table to the variables needed for an example analysis construct the complex sample survey object print statistics and standard errors matching the target replication table click here to view these eight scripts for more detail about the survey of income and program participation (sipp), visit: the reengineering of the survey of income and program participation, snore. the survey of income and program participation wikipedia entry notes: sipp is right in the middle of an ultra-long pan el, these scripts will update as new files are released. don't let the deprecated-looking homepage dissuade you. the survey of income and program participation is happening now, red-hot. everything you need is available, albeit somewhat hidden. there's a short introduction, the data release schedule, an official ftp site - with codebooks - advanced user notes, census publications based on sipp - don't miss the table packages - aww cool even questionnaires. the core variable codebook might not win any beauty pageants, but it'd be a wise use of time to slowly scroll through the first fifty variables. interview months take place after `srefmon == 4` and actual times of month and year can be determined with the `rhcalmn` + `rhcalyr` variables. perhaps more than any of the other data sets on this website, working with sipp will get more comfortable as you increase your ram. so long as you manipulate t hese files with sql commands inside the sqlite database (.db) that my automated-download scripts create, you'll process these data line-by-line and therefore be untethered from any computer hardware limits. but the moment a dbReadTable or dbGetQuery command pulls something into working memory, you'll begin gobbling up those precious four, eight, or sixteen gigabytes on your local computer. in practice, this simply requires that you to define the columns you need at the start, then limit what gets read-in to only those variables. you'll see it done in my scripts. if you don't copy that strategy -fair warning- you may hit allocation errors. maybe keep the performance tab of your windows task manager handy and take out the trash. confidential to sas, spss, stata, and sudaan users: watch this. time to transition to r. :D若美国人口普查局的预算大幅缩减，仅剩一项复杂的抽样调查得以留存，那必然是收入与项目参与调查（Survey of Income and Program Participation，SIPP）——请务必如此祈祷。该数据集体量庞大，变量极其丰富，采用月度追踪设计，对家庭进行为期3年、4年乃至如今5年的面板追踪。美国国会预算办公室会利用其开展健康保险模拟分析。不少研究者在得知SIPP包含个人-月层面文件时会望而却步，转而使用质量稍逊的数据集。美国社区调查（American Community Survey）或许堪称调查数据领域的珠穆朗玛峰，但SIPP无疑是这片数据丛林的亚马逊雨林。穿梭于林冠之间的各类问题（此处指核心数据字典）包罗万象，坊间传闻仍有诸多主题模块变量尚未被像您这样的科研人员深入分析。庞塞德莱昂（Ponce de León）定会钟情于此——这个名字何其绝妙，此人亦何其不凡。 SIPP 2008面板数据的初始样本涵盖42030个家庭中的105663名个体。样本抽取完成后，人口普查局每4个月对四分之一的受访者进行追踪调查，整个面板周期为4至5年（面板时长存在差异）。在开展任何分析前，您务必通读本文档的第3、4、5页（从“波次与轮换组”标题处开始阅读）。若对研究设计仍有困惑，可参考其官方调查设计教程。 由于SIPP会在面板周期内逐月收集受访者信息，您需要明确研究结果需对应时点快照、年度化数据还是其他特定周期的统计量，下文的分析脚本将提供各类场景的示例。每4个月的访谈节点中，每位受访者需回答覆盖此前4个月的全部核心问题；随后会开展针对特定波次的附加调查（即主题模块），但此类问题通常仅对应此前单个月份的数据。需再次强调：核心波次文件每人对应4条记录，而主题模块文件每人仅对应1条记录。若将所有核心波次文件合并，将得到面板周期内每人每月1条记录的完整数据集——规模极其庞大：约10万名受访者 × 12个月 × 约4年。 请务必在编写代码前制定清晰的分析方案，确保仅提取所需变量，不多取分毫。更优的选择是直接复用我提供的脚本，如何？ 本全新GitHub仓库包含8个脚本（请留意，共8个）： 1. 1996面板 - 下载与创建数据库.R 2. 2001面板 - 下载与创建数据库.R 3. 2004面板 - 下载与创建数据库.R 4. 2008面板 - 下载与创建数据库.R 由于部分变量在不同文件中分别为字符型与整型，需编写R函数以统一SAS导入脚本中的变量类型不一致问题；同时需妥善处理部分SAS导入脚本中的括号，因当前SAScii包无法直接生成RSQLite数据库，需对`read.SAScii`函数进行改造，实现将ASCII数据直接导入SQL数据库（.db文件）。下载所有微观数据文件——包括权重文件、主题模块文件等全部内容——并将其导入SQL数据库。 2008面板 - 全年度分析示例.R 根据选定的年份，定义需加载至内存的波次与特定变量；遍历12个月份，在数据库中构建单年度临时表；将该12个月的数据加载至工作内存，若有需要可保存以便后续快速加载；同时将主权重与复本权重列加载至内存，完成数据合并；利用全部12个月的数据构建若干年度化与人口统计学变量；构建采用Fay调整因子为0.5的复权复杂抽样设计对象，若您有需要可保存该对象以便后续快速加载；复刻人口普查局发布的统计数据（由于第19页PDF中提及的封顶截断（topcoding）处理，无法实现完全一致的结果）。 2008面板 - 时点快照分析示例.R 根据选定的日历月份，定义需加载至内存的波次与特定变量；将基于访谈节点（srefmon）或日历月份（rhcalmn）的文件加载至工作内存；同时加载主题模块与复本权重文件，完成数据合并；利用核心与主题模块的问题构建若干全新变量；构建采用Fay调整因子为0.5的复权复杂抽样设计对象；复刻人口普查局发布的统计数据，由于本简介的作者采用广义方差公式（GVF）计算边际误差（详见第4页PDF），结果无法完全一致——人口普查局的友好统计学家建议，应尽可能使用复本权重进行计算。 2008面板 - 家庭资产中位数分析.R 根据选定的主题模块，定义需加载至内存的波次与特定变量；同时加载主题模块与复本权重文件，完成数据合并；构建采用Fay调整因子为0.5的复权复杂抽样设计对象；复刻人口普查局发布的统计数据，由于封顶截断处理（可通过搜索本用户指南中的`topcoding`关键词了解更多），结果无法完全一致。看来封顶截断确实会对资产类统计量产生影响。 复现人口普查贫困统计量.R 加载单一波次的数据；将数据表限制为示例分析所需的变量；构建复杂抽样调查对象；输出与目标复现表格匹配的统计量与标准误。 如需了解更多关于收入与项目参与调查（SIPP）的细节，可点击此处查看这8个脚本。访问：https://www.census.gov/programs-surveys/sipp.html。《收入与项目参与调查的重构》一文略显枯燥，而收入与项目参与调查的维基百科条目提到：SIPP正处于超长期面板数据的更新周期中，随着新文件发布，本脚本库也将同步更新。请勿因页面外观略显过时就望而却步，收入与项目参与调查的相关工作目前仍在火热推进中。所有所需数据均已公开，只是查找起来稍显隐蔽：包括简短介绍、数据发布日程、官方FTP站点（附带代码本）、高级用户指南、基于SIPP的人口普查出版物——切勿错过表格包——哦，甚至还有调查问卷。核心变量代码本或许谈不上美观，但花时间慢慢浏览前50个变量无疑是明智之举。 访谈月份的判定需满足`srefmon == 4`，实际的访谈月份与年份可通过`rhcalmn`与`rhcalyr`变量获取。相较于本网站上的其他数据集，使用SIPP时，随着可用内存的增加，操作体验会显著提升。只要通过我编写的自动下载脚本创建的SQLite数据库（.db文件），并使用SQL命令对文件进行处理，您就可以逐行处理数据，从而不受本地计算机硬件性能的限制。但一旦使用`dbReadTable`或`dbGetQuery`命令将数据加载至工作内存，就会迅速消耗本地计算机宝贵的4GB、8GB乃至16GB内存。实际操作中，您只需在初始阶段明确所需的列，并仅将这些变量加载至内存即可——我的脚本中已经实现了这一策略。若您不遵循此方法——友情提示——可能会遇到内存分配错误。建议随时打开Windows任务管理器的性能选项卡，及时清理无用占用。 致SAS、SPSS、Stata与SUDAAN用户：请看这里。是时候转向R语言了。:D