Agribusiness Development 2008-2012 - Georgia

Abstract
---------------------------

The evaluation considers a range of outcomes of the ADA program, including production and profitability, investment and technology adoption, employment and wages, and access to credit and markets. Though it was originally designed as a rigorous impact evaluation that incorporated a randomized design, the evaluation was not able to undertake a rigorous statistical analysis of the program on these outcomes for a number of reasons, including the small overall size of the program, changes during implementation that compromised the original evaluation design, and the timing of the evaluation. Instead, the evaluation uses a mixed methods approach combining qualitative data with descriptive quantitative analysis to assess the impact of the project. Qualitative data collection included focus group discussions and in-depth interviews that collected detailed information from a total of 69 respondents. Respondents were recruited from among those who responded to the ADA survey and were grouped together by type of grantee (PP, VA/VCI, and FSC as separate groups) and by characteristics of interest based on responses to the ADA survey (those that reported an increase in income, those that didn't respond to income questions, those that closed their businesses, exporters, and machinery ring grantees).These interviews and focus groups were transcribed and analysed using the specialized software package NVivo to systematically categorize responses and identify commonalities. Themes of interest to the evaluation were identified and then coded in all of the transcriptions. Summaries of responses by code and respondent type were completed and interesting cases were highlighted, providing some concrete examples of project results and/or feedback that also served in helping interpret the quantitative data.

Geographic coverage
---------------------------

National

Analysis unit
---------------------------

Agricultural holdings

Universe
---------------------------

Applicants to the ADA program from all application rounds (9 in total) in Georgia.

Kind of data
---------------------------

Sample survey data [ssd]

Sampling procedure
---------------------------

(a) ROUND 1:
The frame for the survey is the list of all applicants. It was supplied by CNFA, the program implementer, along with the scores from the initial evaluation, various statuses assigned by CNFA, and various items of information taken from the applications. Each of the four applicant types were considered as separate strata, that is, primary producers (PPs), farm service centers (FSCs), value adders (VAs) and value chain enterprises (VCHs). For PPs, one comparison case was selected for each new treatment case. A propensity score matching (PSM) methodology was used to select the comparison cases, using binary logistic regression. The dependent variable was the event of being a treatment case. The independent variables, all available from data supplied by CNFA on the frame, were:

* the amount of matching contribution the applicant proposed to make
* the current turnover of the business when it made its application
* the number of employees of the business when it made its application
* whether the business was able to secure credit
* the year in which the business was established
* whether the business was located in a village or larger town
* the type of activity the business was proposing to be engaged in
* the round in which the applicant applied

For each PP treatment case, the comparison case with the closest PSM score was selected for inclusion in the survey sample, as long as it had not been selected for interview previously. For the other applicant types (FSCs, VAs and VCHs), stratified random sampling was used to select comparison cases. Because the populations were relatively small, two comparison cases were selected for each treatment case. Selection of comparison cases was to be made within the same strata in which the treatment cases occurred. The strata were defined in terms of the current turnover of the business when it made its application and the year in which the business was established. Type of activity was also used to define the strata for VAs and VCHs.

(b) ROUND 2:
The following sampling rules were applied:
1. Include all businesses that had been interviewed in Round 1 from ADA application waves 1 to 7.
a) Interviewees from ADA application waves 8 and 9 were excluded because those interviews had been conducted too recently to expect significant change to have taken place in the meantime.
b) Selections were made in terms of "businesses" rather than "applications" because some businesses had applied several times. Where a selected business had made multiple applications, the most recent application was nominally selected for inclusion in the survey, regardless of whether that application or an earlier one was the basis of interview in ADA application waves 1 to 7. The most recent one was chosen because it would have the most up-to-date contact information.
c) 199 applications were selected on this basis.

2. Include treatments from any ADA application wave that had not yet been interviewed in Round 1. Some of these were previously non-response and some appeared to have wrongly claimed to have been previously interviewed on the basis of another application. 29 applications were selected on this basis.

3. Include applicants that scored 70+ (passing score) in ADA application waves 1-7, that have not yet been interviewed, but that are not previous nonresponse. Most appear to have wrongly claimed to have been previously interviewed on the basis of another application. 8 applications were selected on this basis.

4. PPs and VAs were not fully enumerated in Round 1, and the process used to randomly select applicants with a score less than 70 has not enabled the probability of selection to be derived. Therefore, for Round 2, select a random sample of 100 PPs and 25 VAs applications, where (i) neither they nor any related application was interviewed in ADA application waves 8 or 9, and (ii) neither they nor any related application received a score of 70+. If the selected application has not already been selected under condition 1 above, include in the Round 2 Survey.
a) 78 PP applications were selected on this basis, that is, 22 of the 100 were already selected under condition 1 above.
b) 18 VA applications were initially selected on this basis, that is, 7 of the 25 were already selected under condition 1 above.

However, as there were only 20 eligible VAs to be chosen under this condition, all 20 were included and so the VAs became fully enumerated. In total there were 334 applications selected for inclusion in the survey. The frame and summary information about the selections are included in the external resource "Followup frame and selections.xlsx".

(c) ROUND 3:
The sample frame was created by NORC and included all cases that were part of the sample in Round 1 and all the cases that were part of the sample in Round 2. The sample comprised of treatment and control groups with three main types of businesses in each group. Overall, 600 face-to-face interviews were planned to be conducted for Round 3. This sample frame was then put through a re-listing exercise to update it since the list of business status and contact information included many incorrect telephone numbers and addresses, there was turnover in owners/managers of agribusinesses, and some had shut down. For the relisting exercise, ACT first tried calling the phone numbers, then conducted field visits to the listed addresses. If still unable to locate the business, ACT regional coordinators contacted local authorities/representatives. Upon contacting the business, updated information about the business status, location, and contact information was collected for use during the main data collection. This updated list was the sample used for data collection.

Sampling deviation
---------------------------

A. Round 1
It should be noted that the model for PPs was re-estimated many times and some comparison cases were selected on the basis of the PSM scores generated in each of those runs. First of all, it had to be re-estimated for each wave of the survey, as new applicants appeared in the frame and new treatment cases were chosen by CNFA. Secondly, many applicants did not have values for all the independent variables, and therefore the model was re-estimated a number of times with varying reduced sets of independent variables. In practice, the strata were defined with too much detail and comparison cases often could not be found in the same strata as treatment cases. Therefore, strata had to be combined. This was done in an ad hoc way, with the result that the probability of selection is not available and corresponding sampling weights cannot be calculated. By wave 4, it was also found that the pool of comparison cases was so small for FSCs and VCHs that all cases had to be included in the sample, that is, these categories are fully enumerated. This then applies to wave 5 also. Selection of comparison cases was on a quota basis, that is, there was substitution for non-responding selections and for selections that no longer existed as separate entities.
This occurred because some green-field proposals never commenced operations, because some businesses ceased operations, and because some businesses merged with or had always operated jointly with other applicants that had already been interviewed.

B. Round 2
During the course of the survey, two notable changes were made to the frame. First, it was discovered that one applicant had not been included in the CNFA Masterlist. This was a VCI applicant and it was therefore added to the survey. Second, it was discovered during interview (and subsequently confirmed) that applicant #318 should have been classified as a PP and not as a VA.

C. Round 3
None

Mode of data collection
---------------------------

Face-to-face [f2f]

Cleaning operations
---------------------------

Round 1 and Round 2:
Data editing took place at a number of stages throughout the processing, including:

a) Primary logical control of the questionnaire:
At the end of each week the interviewers submitted the completed questionnaire to the regional supervisors. In the presence of the interviewers the supervisors checked the accuracy of filling the questionnaires and logical skip patterns. Where mistakes were discovered, a question without a response or ambiguous information, they called the respondents and checked the data.

b) Secondary logical control of the questionnaire:
From the beginning of the field period, at the end of each week and in some cases after two weeks, the regional supervisors sent the questionnaires checked by them to Tbilisi. The secondary logical control of these questionnaires and coding was conducted in the IPM office.

Round 3:
Data editing took place at number of stages throughout the processing, including:

a) Primary revision of the completed questionnaires by regional coordinators
b) Comprehensive checking of the questionnaire by revision specialist
c) Coding of the questionnaire
d) Structural checking of SPSS data files

Response rate
---------------------------

a. Round 2:
In total, 335 applicants were selected for interview. Of these, 217 (65%) were interviewed face to face. A further 51 (15%) had ceased business or were merged with other businesses but were interviewed briefly by telephone. There were 33 noncontacts (including 2 businesses that are known to have been sold to new owners) and 32 refusals (including one ceased or merged business).

b. Round 3:
The response rate was calculated based on the number of complete cases out of the number of eligible cases in the sample. A case was considered complete using the American Association for Public Opinion Research (AAPOR) standard where 80% of the questions that were supposed to be answered have a valid answer (non-valid answers consisted of “don't know”, refusals, or interviewer errors that skipped the question), taking into consideration the skip patterns. The overall response rate was 68.3% (397 complete cases out of 593 eligible cases). The response rates for each group are as
follows:

Primary producer treatment: 75% (114 out of 151)
Primary producer comparison: 43% (155 out of 362)
Value Adders/Value Chain initiative treatment: 71% (34 out of 48)
Value Adders/Value Chain initiative comparison: 34% (56 out of 163)
Farm Service Center treatment: 71% (24 out of 34)
Farm Service Center comparison: 21% (14 out of 68)

Data appraisal
---------------------------

100% double data entry was completed with the exception of the first waves of Round 1. From the ADA application wave 8 (Round 1, wave 4 of data collection), based on DQR recommendations, the decision was made to do 100% double data entry. Detailed information is provided in the external resources.

摘要
---------------------------
本评估涵盖了ADA项目的一系列成果，包括生产与盈利、投资与技术采纳、就业与工资、以及信贷与市场获取等方面。尽管该评估最初设计为一种严格的影响评估，并纳入了随机设计，但由于多种原因，包括项目总体规模较小、实施过程中的变化损害了原始评估设计，以及评估的时间安排，评估未能对这些成果进行严格的统计分析。因此，评估采用了一种混合方法，结合定性与描述性定量分析，以评估项目的冲击效应。定性数据收集包括焦点小组讨论和深度访谈，共收集了69名受访者的详细信息。受访者来自对ADA调查作出回应的人群，并根据ADA调查的回答（报告收入增加的、未回答收入问题的、关闭其业务的、出口商和机械设备资助获得者）以及资助类型（PP、VA/VCI和FSC作为单独的群体）进行分组。访谈和焦点小组讨论被转录并使用专业软件包NVivo进行系统分类和分析，以识别回答的共性和主题。然后对所有转录资料进行编码，并根据编码和受访者类型完成回答摘要，并突出有趣案例，提供一些项目成果和/或反馈的具体示例，这些示例也有助于解释定量数据。
地理覆盖范围
---------------------------
全国
分析单元
---------------------------
农业经营
总体
---------------------------
来自格鲁吉亚所有申请轮次（共9轮）的ADA项目申请人。
数据类型
---------------------------
样本调查数据 [ssd]
抽样程序
---------------------------
（a）第一轮：
调查的框架是所有申请人的名单。该名单由项目实施机构CNFA提供，包括初步评估的得分、CNFA分配的各种状态以及从申请中提取的各种信息。四种申请人类型被视为单独的层，即初级生产者（PPs）、农场服务中心（FSCs）、增值者（VAs）和价值链企业（VCHs）。对于PPs，为每个新的处理案例选择一个比较案例。使用二元逻辑回归的倾向得分匹配（PSM）方法选择比较案例，因变量是成为处理案例的事件。独立变量，所有来自CNFA提供的框架数据，包括：
* 申请人提议的匹配贡献金额
* 申请时企业的当前营业额
* 申请时企业的员工人数
* 企业是否能够获得信贷
* 企业成立的年份
* 企业是否位于村庄或较大的城镇
* 企业提议参与的活动类型
* 申请人申请的轮次

对于每个PP处理案例，选择具有最接近PSM分数的比较案例作为调查样本的一部分，前提是它之前没有被选中进行访谈。对于其他申请人类型（FSCs、VAs和VCHs），使用分层随机抽样选择比较案例。由于人口相对较小，为每个处理案例选择两个比较案例。比较案例的选择应在处理案例发生的同一层中进行。层根据企业申请时的当前营业额和成立年份来定义。对于VAs和VCHs，活动类型也用于定义层。
（b）第二轮：
应用以下抽样规则：
1. 包含第一轮至第七轮ADA申请波次中所有接受过访谈的企业。
a) 排除第八轮和第九轮的ADA申请波次的访谈对象，因为这些访谈进行得太晚，无法期望在这段时间内发生显著的变化。
b) 选择基于“企业”而非“申请”，因为一些企业多次申请。如果选定的企业已多次申请，则选择最新的申请作为调查的代表性申请，无论该申请或早期的申请是否是ADA申请波次1至7的访谈基础。选择最新的一个是因为它将具有最新的联系信息。
c) 根据上述原则，选择了199个申请。
2. 包含在第一轮中尚未接受访谈的任何ADA申请波次的处理。
其中一些是以前未回应的，一些似乎错误地声称已根据另一申请接受过访谈。根据此原则，选择了29个申请。
3. 包含在第一轮至第七轮ADA申请波次中得分70分以上（及格分数）的申请人，尚未接受访谈，但不是以前的未回应者。大多数似乎错误地声称已根据另一申请接受过访谈。根据此原则，选择了8个申请。
4. PPs和VAs在第一轮中未进行全面统计，使用随机选择得分低于70分的申请人的过程没有使选择概率得以推导。因此，在第二轮中，选择100个PP申请和25个VA申请的随机样本，其中（i）他们或任何相关申请在第八轮或第九轮的ADA申请波次中未被访谈，以及（ii）他们或任何相关申请未得分70分以上。如果选定的申请已根据上述条件1被选中，则将其包括在第二轮调查中。
a) 根据上述原则，选择了78个PP申请，即100个中的22个在上述条件1中已被选中。
b) 根据上述原则，最初选择了18个VA申请，即25个中的7个在上述条件1中已被选中。
然而，由于只有20个VA符合此条件，因此所有20个都包括在内，因此VA已完全统计。总共选择了334个申请纳入调查。框架和选择摘要信息包含在外部资源“Followup frame and selections.xlsx”中。
（c）第三轮：
样本框架由NORC创建，包括第一轮和第二轮中所有样本案例。该样本包括处理组和对照组，每个组有三类主要企业。总体而言，计划在第三轮进行600次面对面访谈。然后对样本框架进行重新列表作业，以更新它，因为包含许多错误的电话号码和地址的业务状态和联系信息列表，农业企业的所有者/经理有所变动，一些已关闭。重新列表作业中，ACT首先尝试拨打电话号码，然后对列出的地址进行实地访问。如果仍然无法找到企业，ACT区域协调员联系当地当局/代表。联系企业后，收集有关企业状态、位置和联系信息的最新信息，以用于主要数据收集。此更新列表是数据收集的样本。
抽样偏差
---------------------------
A. 第一轮
应注意，PPs的模型被多次重新估计，并且一些比较案例是基于每次运行中生成的PSM分数选择的。首先，必须为调查的每一轮重新估计，因为框架中出现新的申请人，CNFA选择了新的处理案例。其次，许多申请人没有所有独立变量的值，因此模型多次使用不同的独立变量组合进行重新估计。在实践中，层被定义得过于详细，比较案例通常无法在处理案例所在的同一层中找到。因此，必须合并层。这是临时进行的，结果是选择概率不可用，无法计算相应的抽样权重。到第四轮时，还发现FSCs和VCHs的比较案例池如此之小，以至于必须将所有案例纳入样本，即这些类别已完全统计。这也适用于第五轮。比较案例的选择基于配额，即对未回应的选择和不再作为单独实体存在的选择进行替代。
这种情况发生是因为一些绿地提案从未开始运营，因为一些企业已停止运营，因为一些企业与其他已接受访谈的申请人合并或一直与已接受访谈的申请人联合运营。
B. 第二轮
在调查过程中，对框架进行了两项显著修改。首先，发现一位申请人没有被包括在CNFA主名单中。这是一位VCI申请人，因此将其添加到调查中。其次，在访谈过程中（随后得到证实）发现，申请人编号318应被归类为PP，而不是VA。
C. 第三轮
无
数据收集方式
---------------------------
面对面 [f2f]
数据清洗操作
---------------------------
第一轮和第二轮：
在数据处理过程中，在多个阶段进行了数据编辑，包括：
a) 问卷的初步逻辑控制：
在每周结束时，访谈员将完成的问卷提交给区域主管。在访谈员面前，主管检查问卷填写的准确性以及逻辑跳转模式。发现错误时，如果没有回答或信息含糊不清的问题，他们就会联系受访者并检查数据。

b) 问卷的次级逻辑控制：
从田野工作开始，在每周结束时，在某些情况下在两周后，区域主管将他们检查过的问卷发送到第比利斯。这些问卷的次级逻辑控制和编码在IPM办公室进行。

第三轮：
在数据处理过程中，在多个阶段进行了数据编辑，包括：
a) 区域协调员对完成的问卷进行初步修订
b) 修订专家对问卷进行全面检查
c) 问卷编码
d) SPSS数据文件的结构性检查
回应率
---------------------------
a. 第二轮：
总共选择了335名申请人进行访谈。其中，217人（65%）接受了面对面访谈。另外51人（15%）已停止业务或与其他企业合并，但通过电话进行了简短访谈。有33个无法联系（包括已知已售给新所有者的2家企业）和32个拒绝（包括1家已停止或合并的企业）。
b. 第三轮：
根据样本中合格案例数与合格案例总数的比例计算回应率。合格案例使用美国公共舆论研究协会（AAPOR）标准考虑，其中80%应回答的问题有有效答案（无效答案包括“不知道”、“拒绝”或访谈员错误跳过的问题），同时考虑跳转模式。总体回应率为68.3%（397个合格案例中的593个合格案例）。每个组的回应率如下：
初级生产者处理：75%（151个中的114个）
初级生产者比较：43%（362个中的155个）
增值者/价值链倡议处理：71%（48个中的34个）
增值者/价值链倡议比较：34%（163个中的56个）
农场服务中心处理：71%（34个中的24个）
农场服务中心比较：21%（68个中的14个）
数据评估
---------------------------
除了第一轮的第一波次外，完成了100%的双录入。从ADA申请波次8（第一轮，数据收集的第4波）开始，根据DQR的建议，决定进行100%的双录入。详细信息包含在外部资源中。