The concept of "open data", has went from strength to strength in recent years, with the UK government recently pledging £2.6M of funding and support for greater data sharing between departments.  "Open data" is data that can be freely used, re-used and redistributed by anyone - subject only to the requirement to attribute. Disaster can be the mother of innovation, and the pandemic has greatly accelerated the transformation towards a future where data flows freely between otherwise siloed aspects of organisations. The governments' Covid-19 response has been dependent on the intermingling of data between departments in healthcare, transport, and finance. In volatile times, more businesses in the private sector could similarly look towards the interplay between data from different sources (both internal and external) to inform and improve their decision making.

How an open and integrated approach to data helped fight the virus

Though data has played a key role in formulating the government's healthcare strategy, the role open data has played went far beyond what most would consider purely healthcare related initiatives. The sharing of data between the NHS, the government and large supermarket chains allowed for demographics vulnerable to Covid-19 to be served first for grocery deliveries, when the demand for grocery deliveries far exceeded the available supply.  Supermarkets could access the names of those customers who were over 65, or had pre-existing lung conditions, and could then bump them up the priority queue as a result. Though under normal circumstances their data would have remained siloed, the pandemic provided the impetus for these data sets to be combined. Transportation data has also played a key role in the government's Covid-19 response. Local authorities from all over the UK contributed data collected from traffic control, including video analysis, car park usage, and Wi-Fi data usage on the roads. This allowed governments to assess to what extent stay at home guidelines were being followed. This provided valuable insights about the effectiveness of the policy, and how it was being implemented, which could be cross referenced with infection data from the NHS to determine the benefits of limiting internal movement within the UK.

What businesses can gain from embracing "open data" (and online data)

UK businesses could take a leaf out of the government's book and look to break out of their comfort zones when it comes to the type of data they utilize to inform their decision making. This could mean looking beyond traditional sources of information such as analyst reports, towards the some of the wealth of freely available online data,  which can be used in conjunction with internal data or that of their partners to inform strategy. Retailers who are considering whether to build a new outlet in a particular area can use freely available statistics from online data resources like the National Office of Statistics to determine employment growth in the region, demographic trends (for example a town getting progressively older or younger on average), and even new vehicle registrations (to roughly gauge disposable income).  If this online data shows that the population of "Muddleshire" has high disposable income and is getting progressively older, and the data from your records shows that your products are a hit with older audiences, it could potentially be a good location for investment. It is insights like this that could be key to reigniting the UK retail industries' stagnation. As the pandemic can cause overnight, rapid changes to the employment ratio in industries like hospitality, online data collected from openly available public sources and sites such as Adzuna, Monster, Milkround and other job websites can actually provide superior, more up-to-the minute insights on the economy than traditional reports. This ultimately allows for better, more agile decision making by businesses.

How businesses can better collect online data at scale

The above being said, collecting the online data needed to generate these type of actionable data insights is more difficult than it may appear at first - but these difficulties can be easily overcome if you have the right tools and platforms. First, the sheer workload involved in large scale data collection may be too high for even a large team of humans to handle. Manually shifting through the internet to find the data you need can be an enormously time-consuming endeavour, so businesses will want to use a platform that allows for automation. Being blocked in the process of data collection is another factor that needs due consideration to avoid. Many websites immediately block activity that looks like it could be competitor related, to prevent any malicious activity.  However, these sites are available to any consumer wishing to view them. Using a residential IPs proxy network can allow you to openly and transparently view the internet and collect data without the fear of being blocked, as your IP address would appear to be the same as a typical consumer (who knowingly chose to contribute their IP address) .

Though events of 2020 have been almost as far from ideal as one could imagine, the way in which more and more governments (and private organisations) are embracing a more inclusive attitude to their data sources represents at least one positive outcome. The severe stresses of surviving in the unforgiving environment of 2020 is showing leaders that they need to break away from their traditional attitudes towards data and look towards every available source. This means taking full and total advantage of the rich possibilities afforded by real-time online data, in combination with internal or 3rd party sources, if they want to be able to make the best possible decisions when it really counts.