One of most important elements in monitoring and evaluation, and probably one of the most difficult, is the design of the data collection tools. The data needs change rapidly and significantly depending upon the size, breadth, and scope of the program. There are myriad sources of data – direct beneficiaries, implementing partners, field staff, government departments, etc. And the data is available in variety of forms such as narrative data, participation lists, survey forms, distribution records, etc. Designing a perfect monitoring and evaluation tool can become an insurmountable challenge. Here are 5 tips to help you design a tool which will help you collect granular, comprehensive, consistent, and reliable data.
Start with the End Result in mind – Analytical Reports and Dashboards
The data collected using various data collection tools is a means to an end. Raw data does not serve any purpose unless converted into meaningful information using aggregate or comparative reports and analytical tables and charts. A Chief of Party might want to see a comparison indicator achievements of different projects to see which project needs her attention, a Finance Manager might want to see the amount of budget spent so far to plan her cash flows, or a project manager might be interested in seeing trends in activity completion to identify the problems in the field.
M&E Online® has various dashboards that provide information in a usable format. The data collection tools need to capture necessary data which can be aggregated in Indicator Tracking Tables, Budget Burn-down Charts, and Implementation Status Dashboards. Few key ideas that need to be incorporated in the tools are as below:The data captured by the tools need to be “complete”, or in other words enough to calculate the aggregate reports. For example, if you want to show the total number of children attending school regularly on your dashboard, your survey tool should have space to ask how many days in a week do the children from the house attend school. Imagine if this information is not collected in any of the forms, and then having to calculate the total number of children going to school.
In the case of determining the number of children going to school regularly, you could just ask the parents during survey how many of their children go to school regularly, but this poses a problem of ambiguity. The parents may define “regularly” in a vastly different way, or to hide embarrassment just say all their children (and give that number) go to the school regularly. Instead if the tool asks, how many days a week does each child goes to the school, then if you define regularly as at least 75% attendance, then you can easily and more reliably calculate the aggregate from the information available.
It is always a good idea to check the consistency of data within the tool itself. Continuing the example of school children, you might ask in a different section, how many children work full time jobs. The math will then reveal if there is any inconsistency in the data, for example, if out of total 4 children in the house, 3 happen to attend school 5 days a week and then it is revealed that 3 children work full time jobs, the data is inconsistent unless one child has miraculously found a way to be at two places at a time.
When collecting data, it is advisable to collect data in as granular form as possible and economically viable. Granularity means ability to break down data in its component dis-aggregations. For example, in the school children example, the survey tool might ask how many girls in the house go to school 5 days a week and how many boys in the house go to school 5 days a week. In this case, you can calculate for a district or region or nation where the poll is conducted, the total number of girls going to school regularly, or the boys or the total number of children (girls + boys) going to school regularly. In case the tool only asks the number of children in the house going to school regularly then it would be impossible to split the regional or national aggregate data by boys and girls because that information is not available.
Think Simple and Easy
Most of the data collection tools are designed to be used by a vast number of people – e.g. the field staff, enumerators, surveyors, etc. The ability and budget to provide training to people to fill data collection tools might be limited. The data to be collected or the information required may be complicated in itself, but the data collection tool can be made simple for use. For example, you might have to collect the financial health data for small enterprises which receive loans and grants from your organization. Instead of asking for figures such as liquidity ratios or return on capital employed, simply ask the total debt, total assets, or total sales and cost of goods sold. In this way you save the respondents and data collectors from the burden of calculation. You can then use tools like M&E Online® which has Report Builder Tool which can easily calculate the complicated measures from simple data.
Another aspect of simplicity is the structure and length of the forms. Sometimes there is no way around collecting data about multitude of things from each participant. This can lead to an awfully long form which can be time consuming, confusing, and tedious. Either break down the form into multiple different forms to be filled discretely or at least structure the form into sections to allow the participants to gather their thoughts and data in a more coordinated and categorized way.
No one likes to do extra work. If your form is full of sections that need filling of same information again and again, your participants as well as enumerators are going to lose interest in the form soon enough. If you have already asked the dates of birth for all the children in the house in one section, why ask for the ages of children in another section. Or when your tool has a place for capturing the fertilizers used by a farmer in her field, why ask the farmer again which fertilizers does she buy? This are simple examples, but in reality, many data collection tools are full of repetitive questions requiring same information to be filled again in multiple sections. Monitoring and Evaluation Software MIS like M&E Online® have Analytical Report Engine which can easily auto-fill the information at various places instead of requiring the respondent to provide the same information again and again. The relational database model render redundant to capture basic data like names and personal information again and again.
Design for Roles
Who is going to fill the data collection template? Is it the participants themselves, the enumerators, specialist staff or consultants? This is a particularly important aspect to consider while designing the data collection tools, as it gives an idea about capability to answer certain questions reliably. For instance, in a survey of HIV patients to find the effectiveness of nutrition intervention, if you ask a participant to enter whether there blood sugar levels are consistent or if the spike and dip vastly in a cyclical manner, the participant may not be able to answer this question correctly unless she is in a habit of monitoring her won blood sugar levels. Instead, you might ask how they feel – i.e. if they feel energetic throughout the day or they have periods of high energy and fatigue throughout the day.
However, the participant would be in a much better position to answer such question. Similarly, in case you are designing a data collection for a doctor or a nurse regularly monitoring the patient, you may directly ask about the blood sugar levels as the respondents would be equipped with knowledge and information to answer those questions correctly.
This point is related to the reports and dashboards we have discussed earlier. When designing data collection tools, it is important to consider the formulae and rules of calculation to be applied while calculating the aggregate and analytical reports. What type and format of data might be required as an input for these calculations?
A very illustrative example of this consideration is percentages. You might collect data about the percentage of land for each farmer which is under organic farming, but that would not help you calculate the total percentage of farmland in the entire district under organic farming. Let’s consider Farmer A has 80% of his land under organic farming while Farmer B does organic farming in 40% of his land. Just adding these two together would give you 120%. And adding the data for all the farmers in a district might give you over thousand percent which is absurd as you cannot have organic farming in more land than available.
Therefore, a simple solution is to instead collect both the numerator and denominator – the total land area belonging to each farmer and the absolute land area under organic farming with each farmer. If you add these numbers for all farmers and divide the total land under organic farming in the district with the total land in the district and multiply by hundred, no you would have the total percentage of land in the district under organic farming. This formula is then easily replicated at province, regional or national levels easily. Thus, it is undeniably important to consider what data to collect to be able to calculate the aggregate values. This applies not just to percentages but to ratios, fractions, and other such measures.
Discrete v/s Cumulative
Another aspect of calculation and formula considerations is whether the data collected is discrete or cumulative. In a quarterly report or survey, you might have to choose between collecting the data about total number of patients treated in a hospital this quarter or total number of patients treated in the hospital till date. Both are equally useful as one can be calculated from the other. You can add up quarterly data in case of discrete figures to find the cumulative numbers or in case the numbers collected are cumulative, you can subtract Q2 data from the Q3 data to calculate the discrete achievement of Q3.
However you would have to maintain consistency in your approach, if you collect discrete data for one quarter and cumulative data for next, or if you collect discrete data for one hospital and cumulative data for next, then aggregating and analysing data would lead to inconsistent and incorrect information. And in case of cumulative data you have to also consider the starting period – if you collect cumulative data for number of patients treated till date from the opening day of the hospital for two hospitals, one of which might be in service for ten years and other might have opened a year ago, and then use this data to compare the effectiveness of the hospitals, you may be misguided. In this case, it would be better to collect cumulative number of patients from the start of the year for each year or operation, which would provide a fairer comparison.
Designing data collection tools proves to be daunting task even to the most experienced Monitoring and Evaluation specialists, given the myriad possibilities in which the tools can be designed. But paying attention to few simple details can not only make the task easy but also the tools more effective and reliable. Designing tools with consistency, completeness, granularity, succinctness, and relevance will have huge payoffs in the long run. M&E Online® can help in designing simple, easy to use tools and web-based forms. To know more, contact us by using the form here. To know more about the M&E and Grants Management product offerings of United Business Solutions click here.