Meta-synthesis and R analysis of stakeholder engagement in food, energy, and water systems literature

We conducted a literature review for manuscripts including food, energy, and water systems, and stakeholder engagement. Each manuscript was analyzed and the following data was entered into an Excel workbook in numerical, narrative or yes/no format: year of publication, citation of manuscript, location where research was conducted, country of residence of authors, author(s) affiliation, funding agency, whether a solution to the issue addressed in the paper was proposed or implemented, whether the authors employed a statistical or computational model, scale of solution, type of solution, types of stakeholders involved, description of when stakeholders were involved, how stakeholders were involved, and how stakeholders were identified. The variables that were analyzed using R were coded to a .csv format. Those variables include: solutions proposed, solutions implemented, type of solution, whether a computational or statistical model was used, researcher field, stakeholder types, level of stakeholder engagement measured by three scales (Ghodsvali, IAP2, and a scale developed by the authors), geographic scale of the issue addressed in the study, location of study and residence of researchers. R was used to analyze the data. Full details of the R analysis are included in a pdf file uploaded with the Excel and csv data.

Methods
Literature search and selection

 We conducted two literature searches: an initial search in 2020 and a follow-up search ending in April 2023 to capture literature published since the initial search. We conducted the searches in two different online databases, ScienceDirect and WorldCat, to ensure a comprehensive identification of relevant literature. We were interested in examining papers related to the nexus of food, energy, water systems (FEWS) so we used the following search string: [(“food energy water” OR “food water energy” OR “water energy food” OR “water food energy” OR “energy water food” OR “energy food water”) AND (stakeholder OR engagement OR community OR nexus)]. In ScienceDirect, we searched for this Boolean expression in “Title, Abstract, and Author specified keywords,” and in WorldCat, we searched in “Keywords.” We eliminated duplicates in the two databases. We limited our search to the English language and to articles and book chapters published since 2015. We reviewed the titles, abstracts, and keywords of each paper to determine whether they were relevant to FEWS. If the abstract was unclear, we conducted a word search within the body of the articles to examine the context to the nexus. We identified 177 publications from our initial search and 540 publications in our follow up search, resulting in a total of 717 publications. We then manually screened the papers and removed those in which there was no indication that the paper included at least one term from both groups of terms, those that related to fewer than two FEW systems (food and water, food and energy, water and energy, or food and energy), and review articles. Our database totaled 483 papers for analysis. 

Literature assessment

We analyzed each article to characterize the existence and extent of stakeholder involvement. In the beginning of the literature assessment, a sample of 50 articles were assessed by two team members to ensure agreement on the concepts of interest, in particular the level of engagement and types of solutions that were proposed in the article as those were more subject to differences of opinion. If we were unable to resolve differences of opinion, our working group met monthly and we brought questions for group discussion and resolution. Once sufficient agreement was reached, those team members individually analyzed the remainder of the papers and brought additional questions to the working group. We identified the citation year, the affiliation of the authors (department within their organization), whether stakeholders were involved, the extent of stakeholder involvement, how stakeholders were identified, and when they were involved. We also determined whether a solution was proposed or implemented. The data was entered into Excel in narrative or numeric format, as appropriate. Of the 483 papers, 90 engaged stakeholders at some level, but six did not describe stakeholder engagement sufficiently to characterize the interactions resulting in 84 that were analyzed for stakeholder engagement.

We define ‘stakeholders’ as actors or organizations who have an interest in, have power or influence over, or are affected by the research or its implications. This may include individuals or groups, state or non-state actors, community members, or others. Although researchers and scientists can be stakeholders, for this research, they were not counted if they were the sole stakeholders, but were included if other stakeholders were involved in the research. We recorded all types of stakeholders described in each paper in an Excel database. The list was grouped into types through discussions by the working group. The typology developed included farmers, ranchers and landowners; governmental entities; coalitions, NGOs, and citizen groups; industry and local businesses; the general public; indigenous entities/tribal representatives; experts; youth; migrants; and academics.

We defined ‘stakeholder engagement’ as the inclusion of people or organizations in any part of the project or research. We identified whether stakeholders were engaged and categorized the type of stakeholder engagement in each publication. To categorize stakeholder engagement, we used two frameworks (Table 1). The first is adapted from a categorization of transdisciplinary research for the FEWS nexus and includes engagement at four levels: Nominal, instrumental, representative, and transformative. The second framework, titled the Spectrum of Public Participation, was developed by the International Association for Public Participation (IAP2) to define the public’s role in a public participation process. The IAP2 framework has five levels: Inform, consult, involve, collaborate, and empower. In both frameworks, the levels denote increasingly greater roles, influence, or authority given to stakeholders in the engagement process. If stakeholders were engaged at multiple levels in a project, we coded that engagement at the highest level described in the paper. 

The working group also developed its own scale with an emphasis on the interests of stakeholders and on finding solutions to problems at the community level. Those levels included: Data gathering; data gathering, but the study addresses a problem of importance to stakeholders; inform or educate stakeholders; help stakeholders view issues from different perspectives; engage stakeholders to plan for the future; help stakeholders identify solutions; help stakeholders envision how to put solutions into practice; and help stakeholders put solutions into practice. Each engagement was coded to the highest level described in the manuscript.

In addition to analyzing whether and to what extent stakeholders were involved in the research, we analyzed manuscripts to determine whether researchers described how they had identified their stakeholders and when they had engaged them. We entered that information into the database in narrative form. We created an additional column in which we entered Y (Yes) or N (No) in Excel to count the number of papers that explicitly included these descriptions.

We defined solutions and determined if they were proposed or implemented for each paper. Our working group developed the following definition and description of proposed solutions: A proposition to address or solve the problem identified in the manuscript. The manuscript must explicitly articulate the solution(s). Solutions should have intended outcome(s) to improve the previous state, must be implementable or achievable by the community experiencing the identified problem, must be related to the research conducted in that publication, and must inform decisions or behaviors in practice or be relevant to decision-making. Solutions must have sufficient specificity that they can be implemented by stakeholders without significant additional work. Solutions cannot be recommendations for further research or generic policy recommendations and cannot be hypothetical or theoretical. A list of additional research questions is not a solution, nor are ways in which research in FEWS networks can be improved if it only benefits researchers.

We defined and operationalized implementation as “included methods that sought to be applied in practice by being accessible to decision-makers, addressing relevant questions, and offering a systematic process to employ the method(s) in practice” and creating a system to employ methods in practice (p. 5). Solution implementation should follow a plan of action (e.g., process, workflow or framework with stages/phases) with details on the actions and resources needed to apply the solution. It can be monitored and evaluated over time. Providing the tools and methods to implement a solution to stakeholders, even if they choose not to implement it, is still considered implementation. For example, if a computer model is made available in a way that it is usable to the public, it is considered to be an implemented solution.

Author affiliation was determined based on the listing in the manuscript. Affiliations were categorized into: NGO, engineering, math, computer science, physical science, interdisciplinary, social science, economics, agriculture, and other. If the affiliation was not clear from the manuscript, for example, if the affiliation was a center, one author searched for the listed affiliation and examined the stated purpose of the center to determine the affiliation. There were a number of affiliations that were unavailable and those were not coded. Many of the descriptions were unclear on websites, especially those located in China. The authors acknowledge that this dataset is not robust and attempts to reproduce this data are likely to have different results.

The following data were coded and entered into a csv for analysis using R (Table 2).

Table 2: Variable Descriptions

Variable Name

Description
R Coding

Year
Year of Citation
year

Solution proposed
Was a solution proposed?
solution proposed

Solution implemented
Was a solution implemented?
solution implemented

Stakeholder types
Farmers, ranchers and landowners; governmental entities; coalitions, NGOs and citizen groups; industry and local businesses; the general public; experts; youth; indigenous entities/tribal representatives; migrants; academics.
Farmers, combined government, combined coalition, combined industry, public, experts, youth, tribal nations, migrants, university.

Ghodsvali stakeholder engagement scale (Table 1)
Nominal, Instrumental, Representative, Transformative
nominal, instrumental, representative, transformative

IAP2 stakeholder engagement scale (Table 1)
NA, Inform, Consult, Involve, Collaborate, Empower
data gathering, inform, consult, involve, collaborate, empower

Williams et al. developed stakeholder engagement scale (Conceptual framework 3)
Benefits researchers; problem of importance to stakeholders; informs stakeholders; different perspectives; plan for future, identify solutions; envision solutions in practice; put solutions into practice
researcher, data gathering, inform, perspectives, plan, identify, envision, implement

Statistical Analysis

Using the full sample size of the refined literature review (n = 483), several tests of independence (chi-squares test, Fishers exact test, Barnards unconditional test) were initially performed to determine if there was a significant association between whether a solution was proposed or not in comparison to whether stakeholder engagement was initiated. An initial generalized linear model (glm) was then constructed to evaluate the relationship of stakeholder engagement (all groups included), in comparison to whether a solution was proposed or not. To examine the diversity of stakeholder engagement and its impact on the likelihood of a proposed solution, a secondary glm was constructed which incorporated a diversity index, based on the number of stakeholder types involved in a particular research effort. The diversity index was calculated as follow:

Diversity Index = total number of stakeholder types involved per research effort

                         total potential stakeholder types (n=10)

This index was then calculated for each observation and used as part of our secondary glm, using a binomial distribution, with whether a solution was proposed or not as the response variable (Figure x). Levels of stakeholder engagement are described in Table 1.