IATI Consultations Archive

Results: Require unambiguous indicator reference

Problem: In the <indicator> element, there is currently no way to identify precisely which indicator is being reported on, which makes it impossible to aggregate data coming from disparate sources.

Proposed change: Support (and eventually require) at least one <reference> element under <indicator>.

Details:

One or more `<reference>` elements must be listed under the `<indicator>` element. Like the `<sector>` element, this has a `@vocabulary` attribute and a `@code` attribute. The `@vocabulary` attribute references a known indicator library, like the [U.S. Foreign Assistance Framework](http://www.state.gov/f/indicators/) or the [WHO Indicator Registry](http://www.who.int/gho/indicator_registry/en/). The `@code` attribute is a unique identifier within that vocabulary. As with the `<sector>` element, `vocabulary="999"` denotes a private vocabulary.

More than one `<reference>` element can be listed, in cases where the same indicator is referred to by multiple codes within different vocabularies (e.g. an internal code and a standard code).

In short, I believe the proposed modification would be beneficial but only in a slightly modified form. Firstly, reference to a standard indicator should be optional not required. This is because in many cases, good indicators must be tailored to their specific project. Secondly, the range of allowed choices of indicator libraries should be limited and needs to be decided carefully through a collective process. This is partly because use of certain types of library (such as unilateral donor-defined libraries) would risk running against the Paris Principles of country ownership, alignment and harmonization. It is also partly because measurement of results is problematic in detail, and aggregation of results can easily be misleading. Any scheme designed to enable aggregation of results should involve detailed yet authoritative standards (standard standards) in the process of measurement and aggregation.

I strongly support adding optional vocabulary and code attributes to the indicator element. The vocabulary codelist should be non-embedded to allow for its modification between upgrades. This is both an agile and a controlled procedure which should assuage Michael's fears concerning unilaterally defined libraries.

Thanks, Bill. Agree that these elements should be optional, so the original title of this post is misleading.

The intent of the indicator vocabulary codelist should be descriptive, not prescriptive. IATI is not in the business of passing judgment on the indicators that any given organization is using.

The point of IATI is to get information out to the world, not to take a political position on what constitutes good development practice and what doesn't. Different organizations take different approaches to measuring and consequently will have different kinds of indicators.

Specifically to Michael's point, if you omit donor-defined libraries, you make it impossible for major donors to be able to report on what they're accomplishing. We may not like donors' indicators, or we might even think that they shouldn't "impose" indicators (although in that case I wonder how we then expect them to be able to tell us what happened with the money we entrusted them with). But it's not IATI's remit to tell people what to measure or how. These indicators exist in the real world, and if people can't use IATI to report on them, then they'll be forced to find some other way, which would be a real shame.

The IATI Technical Team supports the principal of this proposal as a non-mandatory addition to version 2.02 only. Further work is needed to clarify the required changes to the schema and codelist status in order to implement this proposal, with further discussion encouraged in the meantime.

Talking about Results and aggregation can always be difficult. It looks like the main concerns with this have been addressed by making it optional, but some feedback from Bond members about the Results section came to me when having a read through:

- NGOs are nervous about using the Results section as it is a limiting way of describing their work. By describing their work in a particular way on a donor-led Standard, they are risking losing context and misrepresenting themselves to their funders and the public. We should be inviting feedback from them about the usefulness of the Results sections and the proposed indicators.

- I agree with Michael's point that vocabulary libraries should be sought and agreed upon through a collective process. For example, we could invite NGOs in a similar sectors to work together to define indicators. This would be a time-intensive exercise, but may be the only way to get valuable aggregated data.

- Through working with NGOs on IATI, I'm torn on whether having this as an option will have a benefit or not.

1) If the indicators are not relevant, they will not use them and there is unlikely to be increased use of the Results section.

or

2) Sector Codes are not an unproblematic example of optional vocabulary. In order to appear in the sector aggregation on the visualisation website like d-portal.org, orgs MUST use the 5 Digit DAC. This may also be the case for DfID's DevTracker? These are public representations of data that orgs like to appear upon, which encourages them to shoehorn their work into sector codes that may not fit exactly. The peer pressure element of standardised indicators - even when optional - should not be dismissed. Future tools that work with Results data should take this into account and allow integration of 'private' vocabularies.

I also wholeheartedly agree that this shouldn't be about standardization, or about pressuring people to use donor-mandated indicators. We will never agree on a standard set of indicators that apply to everyone. Even small organizations have trouble harmonizing their indicators internally. I think the best we can hope for is eventually being able to cross-walk indicators from various sources - just to *observe* (without judgment) that indicator ABC measures the same thing as indicator 1.2.3.

Donor-mandated indicators are a fact of life for many development actors, for the simple reason that donors want to know what's being done with their money. So, again, I think we should not put IATI in the position of passing judgment on indicators.

<rant>

The international development community is facing a crisis of credibility because of its inability to account for what's being done with funds. Consider Haiti, where everyone from USAID to the Red Cross is facing fierce criticism for being unable to explain what they did with all the money that was put in their hands.

So I find it baffling that we don't have a bigger sense of urgency about making results reporting possible using IATI. I'd frankly be very reluctant to give *my* money to an NGO that is "nervous" about being asked to quantify what they're accomplishing.

Certainly some things are easier to measure than others. But if we can't answer the simplest of questions like "how many houses did you build" or "how many households did you serve with clean water" or "how many children did you immunize" or "how many teachers did you train", the public is frankly justified in their skepticism that we're accomplishing anything at all.

It's all good and well to be transparent on how much money we're spending, but our transparency isn't very USEFUL to anyone in the real world if we're not also transparent about what we're achieving.

We are currently supporting more humanitarian organisations to publish, and results in some ways are easier to quantify at least at output level: number of people helped, shelters built, child-friendly spaces established etc. I think the idea of identifying indicators to a particular vocabulary is very useful.

But I think Dan brings up a very relevant cautionary tale (the preference towards the OECD DAC Sector codes, and a practical example of how the IATI standard is not neutral. There is currently a bias towards mimicing DAC reporting, which is not helpful to other entities publishing to IATI. I'd hate to see that bias carried through in the results element, which is potentially one of the most useful elements of the whole Standard.

I would propose separating the bigger debate about a better results standard from the practical proposal here to at least be able to unambiguously refer to an indicator IF you are reporting quantitative results.

Making it possible to use your own indicator list, and (in another proposal) add basic disaggregation, will make the standard a lot more valuable to develop applications for quantitative results where appropriate.

Guidance on how this can be used is crucial.

There is a tendency of donors (well, DFID and DGIS) to just require everything, and of reference tools to favour the OECD DAC elements, which results in e.g. DAC sector code "shoehorning". But I think it would be a missed opportunity not to improve the technical basis with a few simple steps now. It also opens the way for organisations to use their own data in more meaningful ways.

Add definitions:reference: 'A standardised means of identifying the indicator from a code in a recognised vocabulary. Multiple vocabularies may be specified, but each vocabulary may be specified only once for each indicator.'reference@vocabulary: ‘A code for a recognised vocabulary of indicators. The value for this field should appear in the IndicatorVocabulary codelist’reference@code: ‘A code for an indicator defined in the specified vocabulary specified.’reference@indicator-uri: ‘If the vocabulary is 99 (reporting organisation), the URI where this internal vocabulary is defined’

New codelist: Indicator Vocabulary

Addition of a new non-embedded codelist ‘IndicatorVocabulary’. Content of the list requires further consultation. Possible values include:

I think the addition of the @indicator-uri attribute is a great idea. It should be clarified whether this is a link to an indicator vocabulary, or a link to an individual indicator.

I'd advocate for making `reference` into an element, so that you can support multiple references. See this pull request https://github.com/HerbCaudill/IATI-Schemas/commit/ed0329cbdd47c01f8323c2713a1cab8eec335969 . The same indicator might be defined in multiple places (e.g. an implementing organization's framework, a donor's framework, and a host government framework). This is very, very common in practice. Allowing multiple references should go some of the way towards addressing concerns about donor-imposed frameworks.

A couple of other thoughts:

There are lots of vocabularies out there. At the very least, I'd suggest using a larger number than 99 to refer to a private vocabulary.

My preference would be to use alphanumeric codes for indicator vocabularies, because they're human-readable without a lookup (e.g. 'UN-MDG' instead of '5'). There's precedent for this in other codelists (e.g. Organization Registration Agency codes look like 'AF-CBR' and 'US-DOS').

Re @indicator-uri attribute, the indicator-uri links to a list, rather than an item, and this is consistent with other uses of uri attributes within elements that offer vocabulary specifications, and hopefully this is made clear in the attribute definitions:“@vocabulary: ‘A code for a recognised vocabulary of indicators. The value for this field should appear in the IndicatorVocabulary codelist’@code: ‘A code for an indicator from the list referenced in @vocabulary’@indicator-uri: ‘If the vocabulary is 99 (reporting organisation), the URI where this internal vocabulary is defined’”Re point 1, we think that the code ‘99’, now being uniform across other vocabularies and codelists used in the standard, should remain as a reserved code. This shouldn’t raise collision issues, as we are assigning these numeric codes, rather than them being derived from the codelist itself.Re point 2, regarding the use of alphanumeric codes for indicator vocabularies, we’ve made a commitment in version 2.01 to move away from meaningful/human readable codes for several reasons. First is to move towards language neutrality where possible. Second is to make code management easier: there have been cases where third party codes have changed names and descriptions - the semantics of the codes hadn’t changed, but sometimes the names and descriptions undergo minor revisions. In this case it makes sense to divorce the reference from the meaning.Finally, we should emphasise that organisation registration codes like ‘GB-COH’ and ‘NL-KVK’ are non-numerical because they are based on ISO codes (GB/NL), and because COH / KVK are region/language specific institutions (Companies House and Kamer van Koophandel respectively).

This proposal has now been incorporated into a version 2.02 code repositories (see the above GitHub issue links for the schema and codelists for technical details). The relevant pages on a development version of the 2.02 reference and documentation site is available to view here:

We welcome scrutiny on the implementation of this proposal and encourage the community to feedback and suggest solutions for any errors, omissions and problems. This should be done before Monday 7 December, when the process will commence to release version 2.02 as a live version of the IATI Standard. More information on the inspection phase is available here.