Autonomous data management: A golden future for data

Data maybe the new gold, but many businesses are realising that too much of a good thing can be very bad. Starved of data on which to make their decisions, businesses towards the end of the last century were hungry for actionable insight, placing a high value on any information that could give them business advantage. This kickstarted a boom for data collection, which drove a raft of benefits.

Shops that could previously only work out from their cash register receipts that ‘someone’ had bought ‘something’ for Dhs10, introduced data capture solutions that first told them exactly what had been bought, through barcoding, then told them who bought it, through loyalty cards, and then, linking customer data to online activity, were able to build customer profiles that showed not only what people bought but what they looked at too. And for how long. Then came a golden age of data insight, where the possibilities seemed endless. Retailers could better predict purchasing habits, helping to get the right stock on the shelves at the right time and reducing food waste. Doctors got more detailed medical records on which to make diagnoses.

Too much of a good thing
But many organisations are now finding that getting all they wanted – and more – can be a double-edged sword. Data is only powerful if it’s accurate. So, if data is missing, corrupted or unavailable, the whole system fails. Drowning in information, data-driven decision making starts to become problematic. Often the easiest option for a business is to focus on the data that they know to be high-value and then to ‘park’ everything else in a cloud storage vault to assess its importance and deal with later. Storing all of unused data comes at a cost – and not just a financial one. The servers that store this data, on a global scale, require huge amounts of electricity, which create enormous volumes of carbon pollution. We calculated that, in 2020 alone, the storage of dark data contributed 5.8 million tonnes of CO2 waste to the Earth’s atmosphere. That’s the same carbon footprint as 80 of the world’s countries put together. So how do we change this? Well, the plan of coming back to assess this ‘parked’ data only works if there’s a material change to circumstances. Either the company needs to stem the incoming flow of data, or it needs more resources to deal with it.

Teaming with technology
So, what of this dark data piling up so fast that you’d have to be superhuman to ever get through it? Well, the answer is, perhaps, less about one person with superpowers and more about a team with augmented skills. Where people are great at creativity and decision making, technology is great at processing a lot of information at speed. Harnessing artificial intelligence (AI) and machine learning, and using them to augment the skills of the existing IT team, is the route not just to retaining good data-driven decision making, but also reducing the environmental impact of data storage. This is called autonomous data management (ADM) and it relies on technology platforms learning data-management practices and independently applying them to new data sets. Applying these policies is historically a manual task. Doing this on a micro basis, item by item, is time consuming so, often, organisations take a more blanket approach to data management, implementing a policy that labels medium to large pools of data. This is how you get a build-up of unused – and probably unusable – data that sits forever on unaccessed servers. But, when autonomous data management takes over, AI can enable proactive decision making and policy application at a much more granular level.

Reducing the data load
From a sustainability perspective, this can help to radically reduce the volume of data stored and the pollution associated with it. Not only can businesses delete the data that they know for sure is not needed, they’re also able to reduce the storage space that they need by optimising the way that data is held. In a dark data environment, each of those files need to be kept separately because no one knows that they’re the same document. With ADM, technology is used to monitor files across the whole enterprise, indexing which data is the same, storing only unique data, and replacing duplicates with links to the original versions. This “deduplication” is especially useful in backup data, where ADM-driven solutions are sometimes able to reduce the amount of power required to store this data. From a business perspective, it means that data risks can be minimised – or eliminated – from networks. This deluge of data that businesses have not been able to address
is vulnerable.

Mark Nutt is the senior vice president – International Sales at Veritas Technologies

Read: Financial services companies may be vulnerable to ransomware for another two years