Binary Failure: Paul Byars

Binary Failure: Paul Byars

Since our most recent blog the research team has been primarily field based in some of the more remote regions of Tonkolili, in Sierra Leone. The team has been engaged in a simplified and more quantitative approach to understanding the widespread nature of the failure of water supply systems which we have encountered throughout this field research.

Using the information and trends that were uncovered in the initial research period we have moved on to the final stage of the research project. This final stage involves simplifying the surveys questions, in-line with our initial research findings. Doing so has allowed us to engage with more communities in a single day. This involves much more travelling in any given day and, particularly now that the rains have started, much more walking! In total almost 100 villages have been reached in Tonkolili in the final stage of the project.

The reason that we have altered the approach to the research was that the initial in-depth period of studying each community’s relationship to their water supplies indicated almost continuously reoccurring problems. More importantly it also highlighted the weaknesses in current efforts to understand the systems that were provided as part of the relief efforts following the civil war in Sierra Leone.

It was expressed in my previous blog that there are problems with simplifying the nature of failure to ‘working/not working’. I want to take the opportunity in this blog to explore this statement in more detail. The problem that we have found is that there are an increasing number of organisations quoting huge failure rates of water supply systems, particularly in Sub-Saharan Africa. However definitions of what is classified as failure are hard to come by. Therefore the true extent and nature of the problem remains unknown.

It appears reasonable, when seeking to understanding the current status of water supply systems, to show up at each village and (with the communities permission) attempt to see if any water comes from the pump. If there is none, mark it as non-functional, otherwise it is clearly functional. Add up the total and you have an indication of how many pumps are working at any given time. Unfortunately this oversimplification doesn’t really indicate very much at all.

Take village A, they have recently encountered a problem with their pump. However, with technicians at hand, and spare parts available, this ‘failure’ should only take a day or two to fix. Our initial stage of research indicated that if the problem encountered is fixable, and there are resources such as spare parts, tools and skilled personnel available, then fixing the problem could take less than a week. Therefore if every village that we encountered, which had a non-functioning well, was like village A then the ‘failure’ reported in many reports has been over-dramatised.

Village A

However, as a contradiction of village A, take village B. They were given a well in response to their water supply needs in 2004. However the well lining collapsed in the first week of being provided with the system. The failure has not only affected the lining, but also the well walls and pulley systems. The well itself has no water and unless it is completely re-dug, never will. Furthermore – as a permanent large hole in the middle of a village, it continues to pose a threat to the local children. There has, unfortunately, been more than one case study we have noted where a child has fallen into a well – suffering severe injury and in one case, death.  If every non-functioning well was like village B then the real magnitude of failure has been downplayed. It could be argued, at the very least, that the water supply systems provided are a complete waste of money and very possibly lethal to the local community.

 

Village B

There are further examples that indicate why a binary approach to failure is flawed. Take Village C- they were provided with a hand-pump well in late 2007. Ever since it was installed it has been fully functional (i.e. it has never lost the ability to pump water from that aquifer). However, shortly after installation part of the well wall collapsed. The collapse, due to poor construction quality, has badly affected the well formwork. To correct this problem this well would have to undergo major rehabilitation to be returned to an ideal state. As the well sits just below a flood plain it is highly likely that it will become contaminated with surface water when it rains. This asks the question: how should this well be marked on the ‘functional/non-functional’ part of the failure surveys? Though a simple method should be to mark this as failed, where is the lines drawn in this regard? Should the same be done when similar failure is noted on other areas such as the well cap, the well area, the apron, or the spillways? And how bad does the damage have to be to warrant being marked as failed? Hairline, finger-width or football sized cracks?

Village C

Ultimately Village C is the best indication that some wells exist in an intermediary stage of failure. If all villages we encountered are like Village C then any reporting on failure is exceptionally flawed if it doesn’t include a wide range of potential failure mechanisms. Furthermore, there is a whole range of factors that fall into this category – such as seasonal failure (e.g., no water during the dry season), poor water quality (or even rejection of the water due to its aesthetic properties – regardless of its safety), poor yields from the aquifer or pump, community rejection of the water extraction technology – the list goes on!

Finally there is the community of Village D. Their well is fully functional – there are no noted signs of weaknesses in any component of its structure. The preliminary water quality tests indicate that the well water is ideal. Though there are small areas for improvement, they are within the village’s capacity to address. However there is one major problem – the well was given to an individual – in this case the village chief. Surveys in the peripheries of the village clearly indicate that there are large parts of the community that are restricted from using the source; the village chief clearly uses the well as his personal property. The low usage explains its lack of technical issues. Though Village D is an extreme example, there is more than one case study which indicates that certain sections of communities are intentionally restricted from using the water from their improved source. However, if only technical specifications are relied upon, then this well is marked as a ‘success’, it will only appear as ‘functional’ in the reports. The implications are that this village, or the individuals affected, will be passed over for future interventions.

Village D

There are more situations that have arisen as part of our research than can be discussed here. Though the big question remains – which of the four village’s types discussed here are most common? The answer is that most villages exhibit wells that are a combination, though there are many clear cut cases of each example. What this project seeks to display is a better understanding of how the wells systems can be classified. By breaking down the components of a well – and asking questions with regards to the impact, risk and costs of failures, it is possible to understand, in more depth, what the real situation is. Similarly, understanding that failure is not restricted to only technical but also to social issues is a big step in fully evaluating the water supply systems that were provided as a response to the war in Sierra Leone.

Our research team has shown that it is possible to give a more accurate indication of the condition of the water supply systems in Sierra Leone. By creating a framework of questions that conform to a specific criteria; with respect to risk, impact and the communities ability to respond to their wells issues, and crucially without over simplifying the problem, it is possible to receive a fair indication of the water systems current condition. The implications for planning, developing policy and understanding where future innovation can be applied are immense.

Paul Byars

Identifying Socio-Technical Trends for Rural water supply schemes using Case-based Reasoning (CBR)

Identifying Socio-Technical Trends for Rural water supply schemes using Case-based Reasoning (CBR) 

- Jack Barrie -

This blog will introduce the fundamental concepts of Case-Based Reasoning (CBR) and why we have chosen to use this method to help identify complex trends in both the social and technical factors that lead to the failure of rural water supply schemes.

At first CBR appears to be quite complex, but actually it is effectively the same methodology that humans use to solve problems – by experience and logic. I will ease you gently into the world of CBR by explaining how I began to work with it and how it can be applied to our research here.

Background to CBR

I first encountered CBR when studying the problem of determining the significance of a wide range of factors that could influence the time in which it takes to complete a specific building task. These could include the number of people working on the project, level of access for deliveries of materials, level of skill required, volume of concrete required etc etc. Case-Based Reasoning was successfully applied to help determine the level of significance of each factor as well as being able to help project managers predict the time taken to complete future projects by comparing the factors (mentioned above) in the future project compared to previous projects of similar style.

When working in Cambodia with Angus McBride, analysing the wide ranging project styles and technologies used for supplying water to rural communities, it became apparent that CBR could be used similarly to the example I gave above. It could not only help to identify the significance of specific socio-technical community characteristics on water supply projects, but also help governments and NGO’s better predict the impact of future projects.

Therefore I co-developed a CBR model with Angus for my MEng Thesis. We designed the model to be used as such:

1. Locate new rural village that you plan to initiate a water project in
2. Collect specific social and technical characteristics of the village
3. Enter these details into the CBR and the model compares the characteristics to all the past projects
4. It produces the % likelihood of success for various different project methods and technologies based on the outcome of the most similar cases.

After rigorous testing this model was able to predict the outcome of previous water supply projects (pretending they were a new case) to the accuracy of 74%.

What is CBR?

Case-based reasoning (CBR) is a form of artificial intelligence which attempts to replicate human learning by using past experience to solve complex problems. It has been successfully applied to solve complex problems in a wide range of holistic fields including medicine, law and engineering.

There are generally four main components to CBR:

1.                   Retrieve and build a database of similar cases (water projects),

2.                   reuse the cases to solve a new problem,

3.                   revise the solution, and

4.                   retain experience to solve a future problem (learn)

Each case contains dozens of factors such as; the type of hand pump, number or years since installation, hand pump productivity, water quality and number of users, effectiveness of pump user committee.

Each case contains one evaluation indicator that identifies if the case in question has failed.

 A new case, for which a solution is sought, is tested by comparing its similarity with other ‘known’ cases from the systems case base to determine the possible outcome of the case.

The use of Genetic Algorithms in CBR

Case-based reasoning uses the process of genetic algorithms (GA) to quantitatively determine the significance and level of interdependencies of factors affecting the outcome of a process by processing hundreds of thousands of case evaluations, and therefore replicating the experience of the expert.

Therefore the main advantage of GA is that you can determine the significance of each factor through a complex iterative process. The second advantage is that the genetic algorithms can actually mutate (slightly alter) past cases to help better predict unique cases in the future.

The effectiveness of the GA's ability to ascertain the significance of each factor is determined by the size of the case base, or 'experience'. Therefore, as the number of cases (water projects) in the case base increases, the prediction accuracy of the GA increases. The advantage of GA is its ability to gain a much larger 'experience' than any single expert and the influence of bias is much reduced as the outcome is based purely on qualitative data. Furthermore it can assess all interdependencies between all factors included in the case-base.

Therefore if the user selects 100 generations for a case size of 20 factors, the model will analyse 2000 possible weighting scenarios per case. Hence, a case base of 151 cases will amount to the equivalent of 302,000 individual case evaluations.

How will it be applied to the aspect of post conflict water supply schemes in rural Sierra Leone?

I plan to develop the model to carry out similar functions to the Cambodia CBR model developed in my Masters Thesis. It should be able to suggest the significance of specific factors affecting the level of failure of a well. For instance, hypothetically, it may identify that the salty taste of the water is 3 times as more likely to influence the failure of the well compared to the location of the well within the village.

Furthermore, it also raises the possibility of being used by and to help stakeholders involved in installing wells in Sierra Leone to better plan for future problems by running scenarios of communities through the model.

Therefore to simplify, the model would work like this:

1.  A donor/implementer carries out a rapid widespread survey of the targeted villages.  Collecting a detailed well observation and community survey.

2.  Each individual well surveyed makes up an individual case in the case base built up of both the social and technological data collected. Therefore the database would consist of 1000’s of individual case evaluations (collected in the survey)

3.  The collection of these cases would then be analysed using Genetic Algorithms to determine the influence of each factor collected in the survey.

4.   The characteristics of any future villages to be supplied with a well can be input to the model and compared to the vast database of cases already collected to determine if (a) there are any specific factors that may cause problems (important factors determined in 3.) and (b) the likelihood of success of the project.

In summary, I believe CBR offers the opportunity to investigate in a more concise manner the complexity of water supply projects in developing countries and help us to learn from our mistakes made in the past. I hope that helps clarify what I am aiming to do with this research.

Jack

Systems Failure - Paul Byars

Our research team, from Edinburgh University, have been working since February on analysing failure in water supply systems in Sierra Leone. Before we arrived in Sierra Leone we were already aware that this country had significant problems. Various NGOs and research groups had already indicated that the failures of the water supply systems in Sierra Leone were as high as 65%. Providing water supplies, in particular handpump wells, is a typical response of many Non-Governmental Organisations (NGOs), when attempting to reduce widespread problems related to health. In Sierra Leone’s neighbouring country, Liberia, a recent report by ‘the Liberian WASH Consortium’ suggested that 85% of the wells were provided in the immediate aftermath of its civil war. In the account of Liberia, after only 8 year of peace, there is a reported 50% failure of their systems. Sierra Leone has had a longer period of peace, and with a larger population and more water supply interventions. Therefore it is possible to suggest that the rates of failure may also be higher than their neighbour.

In Sierra Leone the water supply interventions have ranged from providing hand-pump well systems, to pulley systems to boreholes. We are primarily interested in the water supplies provided to the rural communities in the aftermath of the war. More recent interventions, such as reservoirs and piped water systems (few as they are) have not been included in our research. We are also aware that much of the rural population has yet to be served with any system. Studying this, though a significant problem in itself, would not help us assess the weaknesses of the current water supply systems.

Unlike the research carried out by the Rural Water Supply Network, InterAide and more recently by the World Bank using FLOW technology, our research is not focussed on targeting as many villages as possible, as this would only result in a binary (working/not working) analysis of the systems. Instead our focus is on a much more detailed, in-depth and qualitative understanding of failure. Our work is intended to give a reliable case study on what may be happening on the ground, on more than a single level, of the socio-technical system.

Our methodology is split into several parts. We start by engaging directly with the community as a group, meeting with chiefs, elders, community leaders and community spoke-persons (and anyone else that wants to attend, all are welcome). From this group we can better understand the village formation, the interactions with the NGOs and local government, and also the type of systems that they have been provided with. From this meeting we receive basic information about the functionality of their systems, and how the private and public sectors respond to their needs. The community also provide us with information on their social infrastructure that is designed to support the wells; these include their user committees, their local leaders, their well technicians, the local government support they have received as well as the community’s access to spare parts and tools.

The community surveys are followed by household Knowledge Attitude and Practice (KAP) surveys. These address the household aspects of water supply. These questions are dramatically different from the community questions. The KAP surveys are focused on individual household water demands, usage, supply, sources, treatment and storage. The questions are asked in confidence and offer an excellent insight into how individual household respond to their water needs.

The final part of our assessment makes use of the teams technical (engineering) skills. In this part we carry out a full technical assessment of the communities improved water points. This covers all aspects of the well, including the type of well, who built them, how long they have been working, the condition of each individual part, the water quality etc. We accompany this by visiting any of the village’s unimproved sources. These sources are normally used in times of community water shortages caused by seasonal failure and systems malfunctioning.

Though this initial assessment gives a good indication of some of the problems, even spending a day is insufficient time to fully understanding the links between the communities and their sources of water. Our many survey questions have to be structured (or in some cases semi-structured) so that we can complete our research in a single day, and have the ability to analyse the data. This means that there are still unresolved questions from the communities regarding water. To provide more qualitative responses some of the villages are targeted for a second day of focus group discussions. In this we speak to the Water User Committees (those responsible for ensuring access and upkeep of the wells and tariff systems), the technicians (those who have been trained to fix the wells) and the community well users (a random selection of community people that drink the water from these wells regularly). Each of these groups is separated by gender, to indicate if there is a gender imbalance with the responses. These focus group discussions each take a day in themselves to complete.

So far a total of 32 communities have been assessed, over 120 individual wells, over 96 household surveys and 25 focus group discussions. The culmination and preliminary analysis of this data has already indicated severe problems with water supply systems. Although the failure rates corroborate with the rates suggested in previous reports, our current research has indicated that not all failure happens in the same way or for the same reasons. Additionally, not all failure offers the same degrees of risk, impact or cost of recovery. Over simplifying the type of failure of the wells by the classification of them being ‘functional’ or ‘non-functional’ either trivialises or overly dramatises the actual situation.

In the context of an emergency, such as in this case study of the immediate aftermath of the civil war in Sierra Leone, there will have been reasons why the quality of work on the water supplies will not have been up to the standard expected in international development. The availability of materials and resources, as well as the technical assistance that is locally available, would have all contributed to limit the final quality of the projects. Therefore, out of context, it is difficult to assess if these mistakes were avoidable.

This project is focussed on better understanding the type of systems which remain at the end of a post conflict scenario. There currently exists an inadequate methodology for understanding the condition of the entire water supplies systems. Furthermore it is difficult to assess what questions are important to ask, which will contribute to developing an understanding of the condition of the system as a whole.

Given the time intensive nature of our current practice, it would be unfeasible for NGOs to contribute this amount of resources, which we are currently using, towards understanding the problem. What we hope to achieve is to identify key areas and trends in the well failures, isolating them and then put them into a Case-Based Reasoning model. Case based reasoning offers the ability to quantitatively determine the significance of individual failure trends of wells.

The next blog will outline in layman’s terms the methodology behind case-based reasoning and the reasons it was selected for this research.

Focus Group Discussions - Alannah Delahunty

We have conducted a number of focus group discussions (FGDs) in several villages, previously surveyed. FGDs are held with well user committees, technicians and community members. Committee and community FGDs were mixed and single gender. Questions range from what is safe drinking water to what are the responsibilities of the water user committee and their interactions with the community. Participants are eager to answer questions and comment on various water and well activities in the community.

The two most striking themes that have come from the FGDs are apathy and lack of knowledge, both relating to water and wells. Both community members and committee members seem apathetic to pump wells no longer working and tasks not being completed, such as fencing and cleaning the well area. Lack of knowledge relates to scientific and health knowledge. Members from all FGDs had mixed responses to safe drinking water, but knowledge was minimal when it came to questions about contamination, transmission and germs.

Apathy and lack of knowledge are particularly dangerous when relating to community water sources because it leads to fetching water from contaminated sources. Even if the source is not contaminated, lack of knowledge about appropriate storage systems can lead to contamination.

The provision of water supply systems must be accompanied with proper education and awareness about water. Communities are apathetic because they lack knowledge about safe water and the capacity to provide safe water at the community level. Focus group discussions prove to be an effective mechanism in identifying knowledge breakdown and community practices that limit livelihood improvement, such as access to safe and clean drinking water.

When Systems Fail - Alannah Delahunty

When water supply systems fail such as wells or boreholes, an entire community can be effected. Water supply system failure is inevitable in areas where upkeep is rare. What is disconcerting about this is that when systems fail, they remain in a perpetual state of failure. This is problematic in most rural areas where the alternative water source is an often contaminated open well source, swamp, river or stream. It is worth noting that most of the failed water supply systems that have failed such as hand pump wells, have been put in after the war in Sierra Leone, after 2003. A number of different problems stem from water failure and alternative water sources.

Health problems such as diarrhoea, vomiting and skin conditions due to drinking, cooking and bathing in contaminated water are common. This often prevents children from going to school, limiting their access to learning and education. Adults who suffer from health problems are prohibited from working in agricultural or other labour roles and thus suffer from a loss of income. Loss of health, education and income can be hugely detrimental to a household and family as rural households often depend on the sum of its parts and the income derived from each part.

The Beginning: Household and Community Surveys

After roughly ten days of work on the household and community surveys, which included a few test runs, we have begun to carry them out in villages. When we arrive in the village, there are customary introductions with the chief and elders. This usually results in a gathering of most of the population of the village, apart from those doing agricultural or other labour. From there, we conduct our surveys.

As it is a rural area, the village conditions are generally the same, depending on the population and accessibility or distance to the road. A general rule of thumb is the farther a village is from the road, the less developed it is. However, sometimes you stumble upon a village that has used indigenous knowledge to develop its own systems of doing things.

Houses and hand pumps or pulley wells are concentrated around the access route into the village. Houses consist of mud or cement brick walls and sheet metal or thatch roofing. Women and elders usually sit outside their houses cooking or minding small children.

People have responded well to the surveys, which is a good sign for data analysis. As we are still in the early stages, we are not making any assumptions about responses, but rather continuing to collect as much data as possible.

Starting Out

The project team has arrived in Sierra Leone! We are funded by the small grants scheme of the Humanitarian Innovation Fund (HIF). We are very happy to have received the grant and are confident that it will contribute to substantial understanding of the water supply systems in Tonkolili district, that were provided by NGOs as part of the post-war relief efforts.

The HIF grant will fund all field research carried out to determine and develop an innovative methodology for analysing the impact of post-conflict water systems in this district.

The research is a shared collaboration between Edinburgh University and Concern Worldwide. The project team will be based in the Concern field office in Magburka, Tonkolili district.

Currently, we are in the preliminary stages of determining the villages where we will conduct our research. We are also finalising the household surveys which will be carried out over the next month. Addionally, we are on the hunt for bicycles to help us to travel around the villages.

From here, we will begin moving into the field, meeting local chiefs, making connections with various community members and conducting our first round of household surveys. More on all this to follow...