Handle with caution: ESG data is plagued with gaps and holes – and we must treat it as such

In times gone by, the main problem with data on companies’ impact on the planet and its people, on their internal culture and governance, was that we didn’t really have any. For the first stint of the 21st century, intervention from up top was kept to a minimum and self-regulation and voluntary disclosure – or else simply keeping a low profile – were the standard.

An outline is now developing, however, thanks to the introduction of environmental, social and governance (ESG) reporting requirements. In the UK, the so-called Climate-related Financial Disclosure Regulations currently apply to the largest businesses and financial institutions. They released their ‘non-financial accounts’ for the first time in April 2022, putting numbers on everything from carbon emissions to workplace diversity, and revealing whether they have targets for improvement.

The release of the International Sustainability Standards Board (ISSB) Standards by the IFRS Foundation last month, marks a significant milestone in the realm of sustainability-related disclosures and it is clear that soon the measures will stretch to smaller businesses.

Problem solved, you might think; compulsory disclosure means bolder lines and a full-colour picture within the next few years. But while mandating measurement certainly gives us more information to play with, we’re only halfway to the truth, because the quality of lots of this information is poor.

—

Why does this matter? The reality is that we’re in trouble if we allow this mass of ESG data to become ‘a castle built on sand’ and we fail to treat it as such.

The prime audience for ESG data is investors, who need increasingly to weigh up non-financial risk alongside financial risk. As demand and pressure to channel cash into businesses prioritising social and environmental good increases, they need something other than verbal assurances to go on. On top of this, data is used by businesses themselves; they can get a sense of how they’re doing and work out how to improve. Numbers and ratings are also increasingly relevant outside the corporate world – to a public and press that want clarity on social and environmental impact, and to the governments that need to know where we are to manage where we’re going.

“If you can’t measure how we’re doing planet-wide, even just within jurisdictions or within certain areas of the economy … we don’t know where we have to course correct, where it is that we have to renew pledges,” says Sherra Madera, chair of the Future of Sustainable Data Alliance (FoSDA).

ESG measurement is useful across all strands of our efforts to combat environmental and social injustice. However, if the tool isn’t working properly and we act as if it is, we’re buying into a worldwide lie.

—

Last year, FoSDA published a comprehensive report on ESG data gaps and holes. Madera herself provides a helpful framework for how we need to think about collecting ESG data on a granular level, if we are to avoid multiplying mistakes and ending up with a misleading overall picture.

Information, in order to be fit for purpose, be “comparable, coherent, comprehensive”. As far as possible, we should be avoiding “errors, extrapolation and estimation”.

“Errors are fairly simply explained,” says Madera. “That error can be from collection. It could even be from faulty sensors if you’re using sensors to input data. It could be user error; it could be poor calculation, etc”

With regards to estimation and extrapolation, “These are tools that, if they are transparent, can add more data to the overall dataset – but perhaps have to be really considered as not being completely raw data that’s being disclosed by an organisation.”

—

Adhering to the laws currently in place, from the TCFD in the UK to the SFDR in the EU, does not automatically guarantee data will be comparable, coherent and comprehensive. Nor does following international standards like the widely used GHG Protocol. At the moment, these guidelines are unprescriptive and open to interpretation.

A study published in June by King’s Business School, London and co-authored by Omnevue’s co-founder and chief science officer Dr Marc Lepere even stated that ‘There is uncertainty in the application of the GHG Protocol; which combined with other incentives means that GHG emissions can (and likely are) gamed’. The scholars ran a pilot study and found that across three companies, on average, the maximum estimate was between 4.6 and 6.7 times the minimum!

Flexibility is positive in the sense that it accounts for ESG measurement being a new, tricky and costly endeavour for businesses, but it also facilitates an environment where businesses – as it’s not unreasonable to assume they would be inclined to do – can feasibly underreport the bad and overreport the good. This is something that’s, arguably, being fostered rather than prevented by the host of third-party agencies queuing up to take ESG measurement off businesses’ hands.

It’s not unreasonable to assume an ESG report produced by an external agency would be more reliable than one produced by an inexperienced company. Often this is the case. But something else is going on – something that will significantly affect our overall sense of environmental and social progress.

There’s evidence to suggest that lots of these agencies are churning out poor-quality data, which can’t easily be compared to other data. They might be making individual businesses’ lives easier, but they are making the overall task harder.

—

At the start of this year, Omnevue commissioned King’s Business School to conduct some market research into how data is being collected, measured and analysed by third-party platforms. The final report looks at 17 agencies and suggests that user experience is frequently being prioritised over data accuracy.

How is accuracy being compromised? When companies calculate overall scores or ratings, they use methodologies. The report found issues with the reliability of certain businesses’ methodologies, as well as an overall reluctance (or inability) to communicate how the maths worked.

For example, on the reliability point: one of the most common ways to calculate emissions specifically is known as the spend-based method, where you multiply the economic value of goods or services by a factor, ie the average emissions per financial unit. It is also one of the least accurate, because it’s deeply reliant on estimation and extrapolation. The GHG Protocol therefore prohibits its use in calculating scope 1 (direct) and 2 (indirect) emissions.

The Omnevue report finds that 8 of the 17 businesses surveyed rely first and foremost on the spend-based method and, in addition, that they do this while saying they are fully aligned with the GHG protocol.

The platforms that do stick to the GHG protocol do so in ways that differ from one another, emphasising the flexibility within this framework. “One of the main things we realised when doing this research was that there was a big lack of uniformity or standardisation across all the tools,” says Bruno Bossano, the ESG consultant and masters student who led the Omnevue study.

For a ratings agency, having a distinct methodology is a USP of sorts; you can make yourself out to be more accurate than your competition. But in terms of the overall usefulness of ESG data, this competitive mindset presents a problem – all the more so when that desire for differentiation is coupled with a deliberate obscuring of what’s going on behind the scenes. And indeed, the report found that only 2 platforms openly list the details of the methodology they use.

We need to be able to rank companies and compare their progress and commitments. Comparability of data is crucial, yet it’s something that the ESG ratings industry is hardwired to get in the way of and that lawmakers are not yet doing a good enough job at ensuring.

The same goes with the datasets used within methodologies, continues Bossano. These are “one of the main sources of value, or how to differentiate [third-party platforms’ offerings].”

Because they’re an asset and thus kept private, there’s no way of validating these datasets – of proving they’re any good. Bossano continues: “There is no regulation or clear framework as to where they get those factors.

That’s I think where the conflict lies. There’s no way of knowing if these factors are going to be accurate enough. If I tell you that I’m a company that has 15 million data points, there is nothing behind that that can guarantee that those are accurate.”

––

If the ESG reporting they are working with is cloudy or flawed, we’ll see decision-makers – at all levels – improving the wrong things at the wrong pace, perhaps even ceasing to make improvements at all.

It’s possible to argue that the ESG measuring exercise is inherently impossible: a dangerous waste of time; a distraction from the real work of cutting emissions and improving social and governance conditions; a myth-making endeavour, where we buy into the belief that things are changing when they’re not.

It’s also possible to insist early difficulties should not detract from how integral data is to improvement, or deter businesses from measuring and trusting it. “It’s really important to make sure that what we don’t do is say that the data is the problem, when it comes to achieving sustainable goals,” says Madera. “Data is obviously the solution.”

“These data are nothing like as certain or precise as financial information,” says Richard Spencer, director of sustainability at the Institute of Chartered Accountants. “However, there is enough to get going, make a lot of mistakes and improve over time.”

One thing is for sure: we need to acknowledge how tricky it is to get high-quality information, and work with what we have rather than with what it’s possible to tell ourselves we have.

One number or observation, put to paper and therefore assuming an air of certainty, can conceal numerous uncrossed ts and undotted is. If you multiply this by the number of businesses feeding back ESG data, you get a majorly distorted overall picture. This can’t happen. The accuracy of this picture – and our ability to compare aspects within it – is more important than ESG measurement platforms’ top lines.

To see the full research report please contact info@omnevue.com