WHEN THEORY MEETS PRACTICE: INSIGHTS FROM AN ONGOING RCT WITH SMALLHOLDER FARMERS IN INDIA

Randomized control trials, or RCTs, are touted as the gold standard for causal inference. Their seemingly simple design assumes away many uncertainties that might hinder both implementation, and the sacrosanct causal pathways. While this simplicity is true in theory, there could be many real-life situations in practice which RCTs fail to account for. Numerous methodological and implementation challenges could come up, hindering the exact application of theory on the ground. In this blog, we share our experience of some of these issues from an ongoing RCT with smallholder farmers in India, and discuss what the implementation process really looked like, before and after the launch, and the quiet lessons that shape research, beyond data and methods.

Cluster-RCT to understand adoption of agroecological practices in Mandla

We employ a cluster RCT method to understand the impact of different modes of information delivery on the adoption of agroecological practices among farmers in Mandla district, Madhya Pradesh- a state in central India. The RCT, implemented in partnership with PRADAN, covers 1200+ farmers from 67 villages, who are part of PRADAN managed women’s self-help groups.

The study aims to make two major contributions to literature. It provides causal evidence on drivers of agroecological practice adoption, compared to existing studies which are descriptive in nature. The study also contributes to research on agriculture extension policy. In India, effective agriculture extension is challenging due to a low extension personnel-to-farmer ratio, making individual face-to-face extension difficult. Our study explores how agriculture information delivered through group settings could be useful in driving adoption of new practices.

The participants are divided into three groups (Figure 1). Treatment group 1 receives participatory extension: technical training about agroecological practices (AE) and group discussions about farmers’ long-term vision for themselves and their farm, relevance of AE in meeting their vision, anticipated challenges and support needed for adoption of AE practices. Treatment group 2 receives traditional (non-participatory) extension covering only technical information about AE without any groupwork. The control group receives no group-based extension/information on AE except a brochure. 50% of villages in each participatory and traditional extension group are provided with kits of seeds and materials to adopt AE practices, free of cost.

The overarching research question is to understand whether providing information alone is beneficial for adoption of AE practices, which method of extension works better, and if there are complementarities when information constraints are relaxed in conjunction with resource constraints.

Planning phase: Calm before the storm

While the experiment design appears straightforward, it masks many hurdles one needs to cross to successfully implement an RCT, and it all begins with the planning phase.

For most RCTs, pre-launch planning is contingent on the availability and quality of existing information, and it is imperative that one refers to the most recent data for this purpose. Power and sample size calculation is the first step in an RCT/experimental study, which is made much easier if researchers have existing data on similar interventions. This is more likely if the intervention is not entirely novel. This however was not the case for us, since, to the best of our knowledge, casual studies that measure impacts of group-based extension on adoption of agroecology are rare. Therefore, our sample size calculations were based on strong assumptions that only gave a ballpark measure of how much power our experiment would have. While the baseline data helped us to reconfirm these numbers, the lesson is that wherever possible, we must always over-sample to avoid any sample size shortages later.

In the absence of baseline data, one could use nationally representative census datasets or, as in our case, the data provided by our implementing partner PRADAN, for randomization of villages and households into treatment groups. However, such data too might not have complete information about key covariates for every individual. We ultimately had to rely on rapid listing exercises to identify participants for this study, which was a time-consuming and costly process.

Despite these early hiccups, our most challenging curveball was yet to unfold. After baseline data was collected, we found that our treatment and control groups were not fully balanced on a set of pre-defined sample characteristics. Baseline imbalance suggests that randomization may not have performed as intended–rare but not statistically improbable. Since the intervention had not yet started, we `fixed’ this by rerandomizing our sample.

Field launch: Enter reality

Despite being familiar with the context, there could be numerous aspects which can impact or delay the launch of RCT interventions. This is especially true for the agriculture sector, where interventions must be timed to the crop calendar, and where there is high uncertainty given local weather conditions.

Attrition is a common feature of experiments. We faced significant challenges in reminding the participants about the training and motivating them to actually attend. Women farmers deal with multiple demands on their time–child and elderly care along with income generating work, and it was difficult for them to leave their homes and attend a day-long training away from their village. In some instances, the main participant sent another household member in her place such as the mother-in-law, daughter, sister-in-law, neighbor and even husband. This challenge intensified as we got closer to the end of the monsoon season, when most farmers were busy harvesting their crops and preparing land for the upcoming winter season. This also coincided with several religious festivals and holidays that delayed implementation. While providing compensation for attendance could motivate participants to attend training and offset lost wages, it is challenging for implementing NGOs and government stakeholders to sustain this when the intervention is scaled.

To circumvent this issue, we planned the training in a phased manner by first delivering the intervention to participants who were available and willing to attend the training at a centralized location and then holding training sessions for the remaining participants in their villages. This resulted in a more than anticipated increase in the total time taken for conducting the training but was much more preferable than waiting until everyone was available at the same time and place.

It is important to remember that at its core, the experiment involves real people. At the beginning of the intervention, we conducted training of trainers (ToT) over multiple days and visits with PRADAN staff, master trainers from the women collectives, and externally hired facilitators and note takers. While everyone was given the same training, there were variations in how the trainers conducted training, driven in part by their motivations and interest in this activity. This underscored for us the need for effective people management skills, which proved critical in building rapport with the field team and ultimately underpinned our capacity to adjust to emerging field challenges.

Pro tips for effective management

There is no doubt that RCTs can be expensive and time consuming, but a major component of this cost is human resources, a large portion of which remains unaccounted for. Not only do we need teams and partners who have similar interests in the project, we also need individuals who are highly motivated and have the skills and sensitivity needed to deal with vulnerable and marginalized groups such as women. Here are some pro tips from us on managing fieldwork:

• Hire field managers to obtain regular reports on the ground situation, especially when the principal investigator of the study cannot be in the field for extended periods.
• Devise strategies for real time monitoring of training such as obtaining regular data on participant attendance through online data collection platforms, hiring note takers to keep track of quality of training, and feedback forms to monitor participant experience.
• Hold an initial orientation meeting with all village-level Community Resource Persons (CRPs) and field staff — clearly explain the research objectives and treatment model so everyone understands the experiment before training begins.
• To improve horizontal and vertical communication, ensure regular catch-up meetings with implementation partners and all relevant field staff to resolve any issues and effective plan fieldwork.
• Document everything, from little instances in the field to major decisions taken or changes made in the study design, especially if the experiment or endline data collection is planned over an extended period. This is helpful to reflect later and understand the underlying mechanisms of impact.

 Reflections:

Our aim is not to discourage the reader from implementing RCTs, as there are few techniques that match the rigor of RCTs in generating causal evidence. Our objective is simply to encourage the practitioners to be mindful of the potential challenges and contingencies which can impact the study. Implementation of RCTs in social science and development sector studies could appear significantly different from that in conventional clinical trials. Staying connected with field realities is important to foresee practical constraints and unintended consequences that may not be visible in theory and ensure more realistic planning. One may have to adjust the study design, sampling strategies, and even analysis due to these realities. And in doing so, we are not making any compromises with the rigour of RCT but only making it sound and realistic.

References:

Jones, S. K., Bergamini, N., Beggi, F., Lesueur, D., Vinceti, B., Bailey, A., ... & Quintero, M. (2022). Research strategies to catalyze agroecological transitions in low-and middle-income countries. Sustainability Science, 17(6), 2557-2577.

Kerr, R. B., Nyantakyi-Frimpong, H., Dakishoni, L., Lupafya, E., Shumba, L., Luginaah, I., & Snapp, S. S. (2018). Knowledge politics in participatory climate change adaptation research on agroecology in Malawi. Renewable Agriculture and Food Systems, 33(3), 238-251.

Nicholls, C. I., & Altieri, M. A. (2018). Pathways for the amplification of agroecology. Agroecology and Sustainable Food Systems, 42 (10), 1170-1193.

Authors’ description:

Shweta Gupta is a Senior Research Analyst at International Food Policy Research Institute, New Delhi. Muzna Alvi is a Research Fellow at International Food Policy Research Institute, New Delhi. Sahith Acharya Goverdhanam is an Executive at Professional Assistance for Development Action (PRADAN), Narayanganj, Mandla, Madhya Pradesh.

This research is conducted as part of the Multifunctional Landscapes Science Program.

Header Picture Credits: Ashish Dwivedi, the picture shows participants during a training session.