Hunger Safety Net Programme Impact Evaluation 2012, Second Follow-up Round - Kenya

Abstract
---------------------------

The Hunger Safety Net Programme (HSNP) is a social protection project being conducted in the Arid and Semi-Arid Lands (ASALs) of northern Kenya. The pilot phase has now concluded and the HSNP is beginning to scale up under Phase 2. The ASALs are extremely food-insecure areas highly prone to drought, which have experienced recurrent food crises and food aid responses for decades. The HSNP is intended to reduce dependency on emergency food aid by sustainably strengthening livelihoods through cash transfers.

Oxford Policy Management was responsible for the monitoring and evaluation (M&E) of the programme under the pilot phase, with the intention of informing programme scale-up as well as the government’s social protection strategy more generally. The M&E involved a large-scale rigorous community-randomised controlled impact evaluation household survey, assessment of targeting performance of three alternative targeting mechanisms (Social Pension; Dependency Ratio; Community-based Targeting), qualitative research (interviews and focus group discussions) to assess targeting and impact issues less easily captured in the quantitative survey, and on-going operational and payments monitoring to ensure the smooth implementation of the programme. Findings were communicated to the HSNP Secretariat, Government of Kenya and the Department for International Development (DFID) on a regular basis to inform and advise on policy revisions and development. The M&E component used the data it produced to advise the design of HSNP Phase 2, including micro-simulations of different programme targeting scenarios and review of the phase 2 targeting approach which combines proxy means testing with community-based targeting.

The impact evaluation study compares the situation of HSNP and control households at the time of their selection into the programme (baseline), with their situation 12 months (year 1 follow-up) and 24 months later (year 2 follow-up). Over this 24-month period most of the HSNP households covered by the evaluation had received 11 or 12 bi-monthly transfers (initially KES 2,150, increased to KES 3,500 by the end of the evaluation period).

The baseline data collection was completed in November 2010, the first round of follow-up data collection finished in November 2011, and the final round of fieldwork - in November 2012.

Geographic coverage
---------------------------

Counties of Turkana, Marsabit, Wajir, and Mandera.

Analysis unit
---------------------------

- individuals,
- households,
- community.

Universe
---------------------------

All persons living within "secure" sub-locations across all counties at the time of sampling (2008; due to sporadic insecurity across the four counties, a small portion of sub-locations were deemed to be insecure when the sample was drawn and so excluded from the sample frame).

Kind of data
---------------------------

Sample survey data [ssd]

Sampling procedure
---------------------------

At follow-up 2, in addition to attrition, the sample size is further reduced because the follow-up 2 survey covered eight fewer sub-locations, 40 rather than 48. Overall 2,436 households were surveyed (at the baseline, 5,108 households were covered).

The evaluation sub-locations were selected from a sample frame of all secure sub-locations in each district. In each district 12 sub-locations were selected with PPS (Probability Proportional to Size) with implicit stratification by population density such that there is an even number of selected sub-locations per new district.

The evaluation sub-locations were sorted within districts by population density and paired up, with one of the pair being control and one being treatment.
The sampling strategy for the quantitative survey was designed in order to enable a comparison of the relative targeting performance of three different targeting mechanisms. These are:

- Community-based targeting (CBT): The community collectively selected households they consider most in need of transfers, up to a quota of 50% of all households in the community;
- Dependency ratio targeting (DR): Households were selected if individuals under 18 years old, over 55 years old, disabled or chronically ill made up more than a specified proportion of all household members;
- Social pension (SP): All individuals aged 55 or older were selected.

For both the treatment and control sub-locations there are an equal number of CBT, SP and DR sub-locations. Assignment of targeting mechanisms to sub-locations was done randomly across the same pairs that were defined to assign treatment and control status.
In all the evaluation sub-locations, the HSNP Admin component implemented the targeting process. In half the sub-locations the selected recipients started receiving the transfer as soon as they were enrolled on the programme - these are referred to as the treatment sub-locations. In the other half of the evaluation sub-locations, the selected recipients were not to receive the transfer for the first two years after enrolment - these are referred to as the control sub-locations.

The households in the treatment sub-locations that are selected for the programme are referred to as the treatment group. These households are beneficiaries of the programme. In control sub-locations the households that are selected for the programme are referred to as the control group. These households are also beneficiaries of the programme but only begin to receive payments two years after registration. The targeting process was identical in the treatment and control sub-locations.
The following population groups can thus be identified and sampled:
- Group A: Households in the treatment sub-locations selected for inclusion in the programme;
- Group B: Households in control sub-locations selected for inclusion in the programme but with delayed payments;
- Group C: Households in treatment sub-locations that were not selected for inclusion in the programme;
- Group D: Households in control sub-locations that were not selected for inclusion in the programme.

Because targeting was conducted in both treatment and control areas, households were sampled in the same way across treatment and control areas. Selected households (groups A and B) were sampled from HSNP administrative records. Sixty six beneficiary households were sampled using simple random sampling (SRS) in each sub-location (in two of sub-locations this was not possible due to insufficient numbers of beneficiaries in the programme records). In cases of household non-response replacements were randomly drawn from the remaining list of non-sampled households. This process was strictly controlled by the District Team Leaders.

Non-selected households (groups C and D) were sampled from household listings undertaken in a sample of three settlements within each sub-location. These settlements were randomly sampled. The settlement sample was stratified by settlement type, with one settlement of each type being sampled. Settlements were stratified into three different types:
1. Main settlement (the main settlement was defined as the main permanent settlement in the sub-location, often known as the sub-location centre and usually where the sub-location chief was based. As there was always one main settlement by definition, the main settlement was thereby always selected with certainty).
2. Permanent settlements (permanent settlement is defined as a collection of dwellings where at least some households are always resident, and/or there is at least one permanent structure).
3. Non-permanent settlements.

As concern community level data, community questionnaires were conducted in every community for which at least one household interview was attached. A community was defined as a settlement or a sub-section of a settlement if that settlement had been segmented due to its size. Due to missing data, a small proportion of households are not linked to any community data.

The above explanation is taken from "Kenya HSNP Monitoring and Evaluation Component: Impact Evaluation Final Report 2009 to 2012". For more details please refer to this report in Related Materials section.

Sampling deviation
---------------------------

The reduction in the number of sub-locations surveyed at follow-up 2 was the result of decisions made by the programme and its stakeholders, rather than a technical decision by the evaluation team. This reduction in sample size is unfortunate for a number of reasons. Firstly, it undermines the study design to the extent that the smaller sample size reduces the ability to detect impact with statistical significance. Secondly, it affects the balance of the sample, meaning that treatment and control populations are less balanced at baseline than they were with the original sample structure. Lastly, the sample was designed to be seasonally balanced across the whole calendar year, which is no longer the case as sub-locations that would have been surveyed in the latter and early part of the calendar were dropped. Another implication of the reduced sample at follow-up 2 is that the baseline estimates presented in this report differ from those presented in the baseline and follow-up 1 impact reports. This is because the estimates now relate to slightly different populations.

Mode of data collection
---------------------------

Computer Assisted Personal Interview [capi]

摘要
---------------------------
肯尼亚干旱和半干旱地区（ASALs）的《饥饿安全网计划》（HSNP）是一项社会保障项目。该试点阶段现已结束，HSNP 正在进入第二阶段的扩大规模阶段。ASALs 是极度粮食不安全区域，极容易遭受干旱，数十年来一直遭受反复的食物危机和食物援助响应。HSNP 的目的是通过可持续增强生计来减少对紧急食物援助的依赖。
牛津政策管理公司负责在试点阶段对计划进行监测和评估（M&E），旨在为计划的扩大以及政府的总体社会保障战略提供信息。M&E 包括一项大规模严谨的社区随机对照影响评估家庭调查，评估三种替代目标机制（社会养老金；依赖比率；基于社区的定位）的目标性能，以及定性研究（访谈和焦点小组讨论）以评估在定量调查中不易捕捉到的定位和影响问题，并持续进行运营和支付监控以确保计划的顺利实施。研究结果定期传达给 HSNP 秘书处、肯尼亚政府和国际发展部（DFID），以提供政策修订和发展的咨询。M&E 组件使用其产生的数据来咨询 HSNP 第二阶段的设计，包括不同项目定位场景的微观模拟以及审查第二阶段结合代理收入测试与基于社区的定位方法的定位方法。
影响评估研究比较了 HSNP 和对照组家庭在入选计划时的状况（基线），以及 12 个月（第一年后续）和 24 个月后（第二年后续）的状况。在 24 个月的评估期间内，大多数参与评估的 HSNP 家庭已经接受了 11 或 12 次双月转账（最初为 KES 2,150，到评估期结束时增加到 KES 3,500）。
基线数据收集于 2010 年 11 月完成，第一轮后续数据收集于 2011 年 11 月完成，最终一轮实地工作于 2012 年 11 月完成。
地理覆盖范围
---------------------------
图尔卡纳、马萨比特、瓦吉尔和曼德拉县。
分析单元
---------------------------
- 个人，
- 家庭，
- 社区。
总体
---------------------------
在抽样时（2008 年；由于四个县的不时安全状况，当抽取样本时，一小部分次区域被认为是不安全的，因此被排除在样本框架之外）生活在所有县“安全”次区域的所有人。
数据类型
---------------------------
样本调查数据 [ssd]
抽样程序
---------------------------
在第二次后续调查中，除了流失外，样本量进一步减少，因为第二次后续调查覆盖了八个较少的次区域，40 个而不是 48 个。总体而言，共调查了 2,436 个家庭（基线时覆盖了 5,108 个家庭）。
评估次区域是从每个区的所有安全次区域的样本框架中选择的。在每个区中，选择了 12 个次区域，使用 PPS（按规模成比例的概率抽样）进行抽样，通过人口密度进行隐式分层，以便每个新地区都有相同数量的选定次区域。
评估次区域在每个区内部按人口密度排序并配对，其中一对中的一方是对照组，另一方是处理组。
定量调查的抽样策略旨在能够比较三种不同目标机制的相对定位性能。这些是：
- 基于社区的定位（CBT）：社区集体选择他们认为最需要转账的家庭，最多可达社区所有家庭的 50% 配额；
- 依赖比率定位（DR）：如果 18 岁以下、55 岁以上、残疾或慢性病患者占所有家庭成员的超过指定比例的家庭将被选中；
- 社会养老金（SP）：所有 55 岁及以上的人都被选中。
对于处理组和对照组的次区域，CBT、SP 和 DR 次区域的数量相等。将目标机制分配给次区域的操作是在定义分配处理和控制状态的同一对中随机进行的。
在所有评估次区域中，HSNP 行政组件实施了目标过程。在半数次区域中，选定接收者一旦被纳入计划就开始接收转账——这些被称为处理次区域。在另一半评估次区域中，选定接收者在前两年注册后不会收到转账——这些被称为对照组次区域。
在处理次区域中选定纳入计划的称为处理组。这些家庭是项目的受益者。在对照组次区域中，选定纳入计划的称为对照组。这些家庭也是项目的受益者，但只有在注册两年后才开始收到付款。处理组和对照组的目标过程是相同的。
因此，可以识别和抽取以下人口群体：
- A 组：被选定纳入处理次区域的家庭；
- B 组：被选定纳入对照组次区域但付款延迟的家庭；
- C 组：未选定纳入处理次区域的处理次区域家庭；
- D 组：未选定纳入对照组次区域的对照组次区域家庭。
由于在处理区和对照组地区都进行了目标定位，因此在处理区和对照组地区都以相同的方式进行家庭抽样。选定的家庭（A 和 B 组）是从 HSNP 行政记录中抽取的。每个次区域抽取了 66 户受益家庭进行简单随机抽样（SRS）（在两个次区域中，由于项目记录中受益者数量不足，这不可能实现）。在家庭无响应的情况下，从剩余的非抽样家庭名单中随机抽取替代者。这一过程严格由区队长控制。
未选定的家庭（C 和 D 组）是从每个次区域三个定居点进行的家庭清单中抽取的。这些定居点是随机抽取的。定居点样本按定居点类型进行分层，每个类型的定居点都抽取一个。定居点分为三种不同类型：
1. 主要定居点（主要定居点被定义为次区域中的主要永久定居点，通常被称为次区域中心，通常也是次区域长驻的地方。由于定义上总是只有一个主要定居点，因此主要定居点总是被确定选中）。
2. 永久定居点（永久定居点是指至少有一些家庭总是居住的住宅集合，以及/或者至少有一个永久性结构）。
3. 非永久定居点。
就社区层面的数据而言，每个至少有一个家庭访谈附加的社区都进行了社区问卷调查。如果定居点因规模过大而分割，则定居点或定居点的一部分被视为一个社区。由于数据缺失，一小部分家庭没有与任何社区数据相关联。
上述解释摘自《肯尼亚 HSNP 监测和评估组件：影响评估最终报告 2009 年至 2012 年》。有关更多详细信息，请参阅相关材料部分的此报告。
抽样偏差
---------------------------
第二次后续调查中调查的次区域数量减少是由于项目和其利益相关者的决策，而不是评估团队的技术决策。这种样本量减少是不幸的，原因有很多。首先，它削弱了研究设计，因为较小的样本量减少了检测影响的能力。其次，它影响了样本的平衡，这意味着与原始样本结构相比，基线时的处理组和对照组人口更不平衡。最后，样本旨在在整个日历年度内季节平衡，但由于在日历年度后部和早期部分将被调查的次区域被排除，这不再成为可能。第二次后续调查中样本减少的另一个影响是，本报告中呈现的基线估计与基线和第一年后续影响报告中呈现的估计不同。这是因为估计现在与略微不同的总体相关。
数据收集方式
---------------------------
计算机辅助个人访谈 [capi]