Dark data is destroying the planet: here’s how to stop it

The Coronavirus pandemic has, without doubt, changed the way that many businesses around the world operate as the ‘stay at home’ message turned millions of us into remote-workers. Businesses in the UAE have shifted swiftly in line with this, causing hundreds of companies to adapt their IT infrastructures to meet the needs of employees and customers in tune with the ‘new normal’.

With this shift, we have seen some unintentional consequences, from living our daily lives on video conferencing platforms to reducing the emissions generated by our daily commutes to the office.

As we gradually transition out of lockdown, this is the time for people, businesses and governments to consider what lessons we should learn from the experience, and what behaviours we want to take forward. Our impact on the environment should certainly be one of these – and now that we’ve made a start, there’s a clear opportunity to go further in other areas.

Analysis by Veritas estimates that 5.8 million tonnes of CO2 will be pumped into the atmosphere this year as a result of storing unnecessary ‘dark data’ – this translates to more emissions than 80 individual countries. Businesses don’t often realise that this data comprises half (52 per cent) of all data organisations store. Over the last few decades, it has emerged as one of the largest and most unappreciated blind spots in the fight against climate change.

Data hoarding can be misguided or well-intentioned, but it’s always bad for the environment. Fortunately, the organisations as well as the IT industry has a unique chance to get ahead of this challenge. A mixture of cultural change, education, leadership and a comprehensive data deletion strategy can make a big difference.

How data goes dark

As data becomes more siloed and fragmented, it gets increasingly harder to find and manage. Employees often struggle with an overabundance of data sources and tools, which is only compounded by a lack of strategy and backup solutions. Consequentially, companies have built up vast stores of data – often decades in the making – that they no longer fully understand.

Last year, we released the 3rd edition of our Middle East Databerg Report which revealed that UAE businesses surveyed were failing to manage their dark and ROT data. The study found that only 12 per cent of the data stored by organisations is reported to be clean – the rest being ROT (redundant, outdated, trivial information) and unclassified data.

This figure was not a complete surprise. As technology advances, old data becomes harder to read and slower to utilise. Soon enough, it becomes obsolete and less care is taken to properly manage it. Once it has fallen off the radar, we call it ‘dark data’. Even if dark data is no longer used by employees in an organisation, it’s still there, and dark data isn’t just bad for the environment – it’s bad for business. It only adds to data storage costs and can pose a dangerous cybersecurity risk if it’s not protected under your latest security policies.

The heavily regulated environments of many industries are partly responsible for creating a culture that is too cautious to delete anything. However, many IT and data teams are also too afraid to reduce their data banks out of the fear they might lose something precious in the process. Old data can be a valuable source of customer insight. As a result, databases are becoming larger, harder, more expensive and environmentally damaging to manage.

Deletion makes a difference

The most effective defence against dark data is preventative. To stop dark data from forming in the first place, companies must create data management strategies that accommodate recent data while cycling obsolete data out of the system. They also have to resist the temptation of a data hoarder approach. Instead, they should take advantage of new tools that can locate, classify and delete data across multiple environments.

Databases can become infested with dark data when employees lack strong guidelines. Staff will often forget to label data correctly or will decide to save an extra copy just to be safe. This is where managers should step in, training employees in the correct use of metadata and discouraging unnecessary copying. Data management standards should be agreed from the outset and enforced from the bottom to the top. This means everyone knows what the data types and formats are and where they should be saved at all times.

Companies should also be willing to adopt the latest technologies for increased efficiency and utility. A single, unified data management platform can make it easier for employees to discover the data they need faster. By bringing together and explaining a company’s data, employees can make better-informed decisions on what data to keep and what to delete, making it less likely that data will go dark.

Automation is another important part of good data management. Once you have visibility over your data estate, automation tools allow you to deploy decisions and policies across all your different data environments. Data is automatically classified on upload, reducing error and improving accuracy down the line. To reduce the build-up of dark data, data can also be expired after a set period, keeping volumes under control by streamlining the deletion process.

Defeating dark data once and for all will require a change from within an organisation – both operational and cultural. Deletion and data responsibility are vital parts of sustainability, but to do it effectively your employees need insight and confidence which comes with knowing what data you have. By encouraging data responsibility and implementing the latest data management tools, businesses can do their bit to cut data emissions.

Johnny Karam is the regional vice president, Emerging Region at Veritas Technologies