The World Health Organization promotes the SAFE (Surgery, Antibiotics, Facial cleanliness, and Environmental improvements) strategy for trachoma control and prevention. The F&E components of the strategy focus on promotion of healthy hygiene and sanitation behaviors. In order to monitor F&E activities implemented across villages and schools in Malawi, Tanzania, and Uganda, an F&E Monitoring and Evaluation (FEME) framework was developed to track quarterly program outputs and to provide the basis for a pre and post evaluation of the activities. Results showed an increase in knowledge at the school and household levels, and in some cases, an increase in presence of hand/face washing stations. However, this did not always result in a change in trachoma prevention behaviors such as facial cleanliness or keeping compounds free of human feces. The results highlight that the F&E programs were effective in increasing awareness of trachoma prevention but not able to translate that knowledge into changes in behavior during the time between pre and post-surveys. This study also indicates the potential to improve the data collection and survey design and notes that the period of intervention was not long enough to measure significant changes.

Trachoma control programs promote facial cleanliness, use of latrines, and emphasizes the importance of access to water as means to reduce trachoma transmission. To address these areas, various country ministries and supporting non-governmental organizations (NGOs) support the implementation of tailored behavior change programs. The process and results of evaluating these types of programs in Malawi, Tanzania, and Uganda are described here. The goal of this manuscript is to provide trachoma program managers and supporting NGOs with insights, recommendations, and data collection tools that could be used to support their efforts to conduct monitoring and evaluation of their current or future trachoma prevention activities.

Trachoma and water, sanitation, and hygiene

Trachoma, the result of ocular infection with the bacterium Chlamydia trachomatis, is the leading infectious cause of blindness worldwide [1]. Over time, repeated infection can cause the inside of the eyelid to become scarred and the eyelid and eyelashes to turn inward. Left untreated, the painful condition of the in-turned eyelashes scrapping the cornea can cause the individual to become irreversibly visually impaired or blind. Children have been documented to have higher levels of trachoma infection compared to other age groups, with the prevalence of active trachoma greatest among preschool age children [2–4]. The progression of the disease to the advanced blinding stage occurs over time with the onset of visual impairment typically occurring in those aged 35 years and over [5].

Transmission of the bacterium Chlamydia trachomatis can occur when fingers and fomites such as clothes and towels come into contact with infected ocular and nasal secretions from one individual and then contact the eyes of another individual [6]. Flies have also been documented to transmit Chlamydia trachomatis when the fly feeds on the ocular and nasal secretions of an infected person and then lands on the eyes of another person [7]. These same flies are strongly attracted to odors produced by human feces and lay their eggs on exposed feces on the ground [8]. Trachoma frequently affects poor and marginalized populations that typically sleep in close quarters, experience overcrowding, and lack adequate access to water and sanitation [1,9,10]. It has been challenging to determine the relative importance of one route of transmission compared to others, and it is unlikely that any single route of transmission is responsible for all trachoma transmission [6,11]. Despite not knowing the extent that Water, Sanitation, and Hygiene (WASH) programs impact trachoma prevalence within communities, certain human behaviors have been demonstrated to reduce the risk of trachoma transmission. First, reducing exposed human feces, through methods such as using pit latrines, reduces fly breeding grounds [7,8]. Second, avoiding sharing cloth with infected ocular or nasal discharge reduces the transmission of the bacterium between people [11]. Third, keeping one’s face clean from ocular and nasal secretions reduces the attractiveness of the face to the flies that feed on the secretions for nutritional purposes while simultaneously transmitting the bacterium [7]. Lastly, increasing communities’ sufficient and reliable access to water helps increase the likelihood that residents will be able to keep their faces, clothes, and bedding clean thereby reducing risk [12]. It is because of the transmission dynamics and progression of the disease that the World Health Organization (WHO) promotes the SAFE strategy (Surgery, Antibiotics, Facial cleanliness, and Environmental improvements) for trachoma control and prevention [1]. The F&E components of the strategy are focused on promoting healthy hygiene and sanitation behaviors and have a significant overlap with WASH activities. Given that the F&E components are outcomes, there are diverse types of interventions that can be used to achieve these outcomes. The lack of standardization and consistency in how these interventions are designed and implemented present a challenge in comparing across settings and clearly demonstrating effectiveness.

Selection of F&E interventions

The trachoma control programs in Malawi, Tanzania, and Uganda received funding from The Queen Elizabeth Diamond Jubilee Trust (hereafter referred to as the Trust) for F&E activities, with additional funding provided to Tanzania from the United Kingdom’s Department For International Development (DFID) [13,14]. In order to guide decision making about how these funds should be spent the Ministry of Health in each country used a multi-step process. First, a situational analysis of WASH and trachoma programming within each country was conducted in order to understand the partners, resources, and existing WASH, trachoma, and F&E activities already taking place at the regional and district levels [15]. Second, the "All You Need for F&E" International Coalition for Trachoma Control (ICTC) toolkit was used to guide an F&E stakeholder workshop [16]. Representatives from ministries of water and sanitation, health, and education participated in this workshop alongside representatives from NGOs from the WASH, neglected tropical diseases (NTDs), and research sectors. Workshop participants identified a list of F&E activities and then participants narrowed down the number of activities based on available donor funding and what they felt should be prioritized and could be achievable in three years. Table 1 shows an overview of the different activities chosen for implementation by region within each country. A more detailed description of activities implemented within each country is provided in S1 Text . Ministries of health, water, and education, and supporting implementing partners implemented these interventions from July 2015 to March 2018 in Malawi, May 2016 to March 2018 in Tanzania, and May 2015 to March 2018 in Uganda. While F&E interventions targeted districts with a trachomatous inflammation-follicular (TF) prevalence over 5% (in children ages one to nine years), not all selected program activities were conducted in all villages within the targeted districts due to lack of funding and implementing partner availability.

Monitoring and evaluating F&E

An F&E Monitoring and Evaluation (FEME) framework was developed for each country to assist with conducting quarterly monitoring of F&E activities and to provide the basis for a pre and post evaluation of F&E activities. The indicators used within each country’s FEME included a combination of WASH and NTD indicators identified during a Delphi consultative process [17] and country specific indicators requested by the Ministry of Health. Country specific FEMEs are provided in supplemental information (S1, S2 and S3 Tables).

The FEME can be divided into two parts: 1) the logical framework that includes outcome and output indicators; and 2) program implementation activities. Theoretically implementation of activities leads to the achievement of the desired outputs and outcomes reflected by changes in their indicators. For example, conducting community meetings about trachoma (activity) should result in an increase in the percentage of people who have knowledge of hygiene practices in relation to trachoma prevention (output) which thereby contributes to an increase in the percentage of children with clean faces (outcome). Throughout the life of the Trust funded project within the three countries the F&E activities were reported on a quarterly basis. For measuring progress on achieving outcomes and outputs, each country conducted a pre and post-survey.

Objectives of the paper

The purpose of this paper is threefold: 1) present the methods and results of the pre and post surveys in each of the three countries; 2) discuss challenges and successes of the survey process and indicator measures selected; and 3) provide recommendations based on this experience for implementing F&E monitoring and evaluation mechanisms.

Ethics statement

Ethical approval was obtained for the pre and post-surveys in all three countries. In Malawi from the National Health Sciences Research Committee (Ref: #16/12/1700), in Tanzania from the National Institute for Medical Research (NIMR) National Health Ethics Review Committee (NatHREC) (Ref: NIMR/HQ/R.8a/Vol. IX/2405), and in Uganda from the Uganda National Council for Science and Technology (Ref: HS 2166). The study conducted in Uganda had additional ethical approval from Emory University (eIRB#: IRB00093647). All survey participants gave written informed consent prior to participating, head teachers provided written informed consent on behalf of school children who individually assented to take part. Assent was documented.

Evaluation units

F&E programmatic intervention regions were used as evaluation units (EUs) with sampling of villages from a sample frame of all intervention villages. Where the scale of interventions was small (Tanzania) multiple regions were grouped into a single EU. For regions where there was both school and community programming the EUs were considered separately (Table 2).

In all three countries, implementation of F&E activities began prior to the pre-surveys being conducted. Pre-surveys were conducted in 2017 and post-surveys conducted in 2018. Fig 1 provides a visual of the regions within each country where community and school activities took place and the location of the villages and schools that were selected for evaluation.

Sampling

The sample size calculations were performed in STATA using the "sampsi" function. Sample size was based on the indicator ‘percentage of children with a clean face’ assuming 50% at baseline and the desire to detect a policy relevant change of 10 percentage points. We used 80% power in all calculations, incorporated a design effect of 2 (villages) and 1.5 (schools) and an anticipated non-response rate of 15%.

Community sampling

Household data was collected using a population-based survey following a two-stage cluster sampling methodology. In each community EU all villages targeted for F&E interventions were listed along with their estimated population sizes. The village was the primary sampling unit and was selected using probability proportional to size. The same villages were surveyed at both pre and post-survey. The secondary sampling unit, the household, was randomly selected using a household listing approach. Households that did not have at least one adult (≥18 years in Tanzania and Uganda, ≥15 years in Malawi) who identified as the primary care giver and at least one child under nine years old were excluded from participating and replaced. Every effort was made to collect data from the selected households, which were visited three times before being replaced in cases of recurrent non-occupancy. Non-participating selected households were also replaced. Twelve villages in Lindi which had both school and community programming were excluded from the sample frames for this EU.

School sampling

In Uganda schools were randomly selected from a list of all intervention schools in the EU. In Malawi they were selected using probability proportionate to size. In Tanzania, all intervention schools were included in the survey. In all three countries, in each selected school, two classes were randomly selected from all classes targeted with the trachoma F&E interventions. Within each of these two classes, 21 students were randomly selected for questionnaires. The same schools were used in the pre and post-surveys.

Data collection

The surveys were designed and conducted through collaboration of partners in each country. Independent teams were recruited for the data collection who were blind to the interventions conducted and other indicators that would be generated from the data. Household questionnaires were conducted with the primary caregiver. All present household residents were directly observed for signs of ocular or nasal discharge [18]. As a proxy indicator for hand and face washing, data was collected on the presence and functionality of hand/face washing stations [19]. Direct observation of the availability of handwashing and toilet facilities was conducted. School questionnaires were conducted with head teachers along with observations of availability of toilet and handwashing facilities. Individual questionnaires were conducted with students along with observations of presence of ocular or nasal discharge on the face. Student hand and face washing behaviors were observed for a period of at least three hours in each school. In order to reduce bias we ensured that all observers/research assistants were from the respective countries, spoke the local language, and observed the events as discretely as possible. We also did not tell the schools exactly what we were observing (i.e. we did not say we were there to observe hand and face washing practices specifically). The extended observation time (3 hours) was selected in an effort to capture different opportunities in which children would wash their hands and face and also allowed the children to get used to the observer being present. There is always a risk of observer bias in the study, and we recognized this as a limitation and made as many efforts as possible to limit these biases. Household questionnaire guides were translated into the predominant local language. All questionnaires were pilot tested in the language in which it was to be conducted and revisions were made to increase clarity. School questionnaires with the head teacher were conducted in English while student ones were conducted in the local language. The English version of the questionnaires and observational data collection tools are provided in supporting information (S4, S5 and S6 Tables). Table 3 documents the different types of data collection methods used in schools and communities. With the exception of the student ‘hand/face washing behavior observation’, all observations were embedded components of the school or household questionnaire.

Data management and analysis

Data was collected electronically using a purpose-built Open Data Kit-based Android smartphone application- LINKS for pre-survey and CommCare for post-survey [20,21]. Data was downloaded, imported to STATA13 and weighted according to sampling design and analyzed for a difference between pre and post-surveys within EUs using chi squared statistical test. A p-value of <0.05 was used to attribute a statistical difference between pre and post.

School EU Results

Key results from each school EU within each country are provided in Table 4.

Only the EU in Tanzania showed a significant increase in the percentage of schools that had hand washing facilities with soap with 4.8% (Confidence Interval (CI): 0.6–30.7) at baseline and 35.0% (CI: 17.3–58.0) at post-survey (P = 0.043). In Uganda, there was significant increase in the percentage of schools that had at least one clean latrine for both boys and girls, with 27.8% (CI: 9.9–57.3) at baseline and 70.8% (CI: 39.1–90.2) at post-survey (P = 0.039).

There was minimal positive behavior change across the three countries. Only the two school EUs in Malawi showed a significant change in the percentage of school compounds free of human feces, with those in the Central region increasing from 19.6% (CI: 6.0–48.1) at baseline to 92.6% (72.6–98.3) at post-survey (P < 0.01) and those in the Southern region increasing from 7.5% (CI: 1.9–25.0) at baseline to 74.2% (CI: 28.6–95.4) at post-survey (P < 0.01). However, 100% of surveyed schools in Tanzania were free from human feces at baseline which was maintained through the post-survey. In Karamoja, Uganda, there was a decrease in the percentage of school children washing their faces when washing their hands during the school day with a baseline of 22.0% (CI: 12.2–36.2) decreasing to 13.0% (CI: 6.0–25.9) at post-survey (P = 0.048). There was no significant improvement in the percentage of students with a clean face between pre and post-surveys; however, the baseline percentage of children with clean face was already high across the three countries ranging from 90.8% (CI: 81.2–95.7) in Karamoja, Uganda, to 96.9% (CI: 92.6–98.7) in the Southern EU of Malawi. All but the Southern region of Malawi had a significant increase in the percentage of school children who knew at least one measure to prevent trachoma.

Community EU Results

Programmatic achievements

Recommendations

Based on the experience of selecting, implementing, monitoring, and evaluating F&E activities, there are several recommendations. First, the F&E activities chosen for implementation in these three countries were based on donor funding, implementing partner presence and perceptions of what was achievable in three years. Though these factors are important for program implementation, it is recommended that behavior change theory also be used to develop programs and M&E. This will not only help tailor programming to the specific needs of each community but also aid in the consistent and clear measurement directly linked to the activities. Second, ideally any future pre-surveys should occur before F&E activities are implemented if time and resources allow; however, in areas that have on-going F&E related activities, this might not be possible. Third, programs should determine what their output targets are before implementing F&E programs and when determining methods of measuring success. This should include not only change from pre and post-survey but also pre-identification of indicator levels which are considered optimal and which the maintenance of (as opposed to statistically significant increase) would be considered a successful outcome. Fourth, in order to evaluate whether activities are effective, it is important to know if they are being implemented as intended. Routine monitoring of activity implementation is critical and can provide insights and add depth to results from program evaluations. The FEME (S1, S2 and S3 Tables) provides a document that could be used as an example on which to improve. Fifth, there should be careful consideration of possible confounders of seasonality and time of day that data is collected, with efforts made to standardize the months and time of day data is collected. Sixth, unless multiple surveys are being conducted to measure longitudinal data, at least 24 months should pass between pre- and post-surveys allowing for more time for behavioral change practices to take root. Lastly, where possible, instead of conducting a standalone F&E survey, F&E data should be collected in connection with other planned programmatic, demographic, or health surveys in order to streamline human and financial resources.

S1 Text. Program Achievements.

https://doi.org/10.1371/journal.pntd.0009962.s001

(DOCX)

S1 Table. Uganda FEME.

https://doi.org/10.1371/journal.pntd.0009962.s002

(XLSX)

S2 Table. Malawi FEME.

https://doi.org/10.1371/journal.pntd.0009962.s003

(XLSX)

S3 Table. Tanzania FEME.

https://doi.org/10.1371/journal.pntd.0009962.s004

(XLS)

S4 Table. Household Survey.

https://doi.org/10.1371/journal.pntd.0009962.s005

(DOCX)

S5 Table. School Survey.

https://doi.org/10.1371/journal.pntd.0009962.s006

(DOCX)

S6 Table. Student Observation.

https://doi.org/10.1371/journal.pntd.0009962.s007

(DOCX)

S7 Table. Supervisor Forms.

https://doi.org/10.1371/journal.pntd.0009962.s008

(XLSX)