Big Data Challenges Explored by Industry Experts: Key Issues in Security, Privacy, and Quality

At the recent AI Summit in London, industry experts from prominent sectors such as energy, healthcare, and telecommunications gathered to discuss the immense potential and pressing challenges associated with big data utilization. The panelists shed light on how they are harnessing data to enhance customer experiences and drive informed decision-making, all while navigating concerns related to data security and privacy.

Elena González Garcia, a lead operations and maintenance engineer at Scottish Power, shared the hurdles her company faces when integrating data from third-party contractors. “While we receive information from contractors, I'm constrained by contracts that limit my ability to improve the quality and accessibility of this data,” Garcia explained. She advocated for the establishment of contracts that prioritize data transparency and access. “As the owner of the data pertaining to activities conducted on my assets, I should have straightforward access to this information without ongoing disputes with contractors,” she asserted.

The panel highlighted the necessity of developing a robust data platform that encourages experimentation while ensuring security. They discussed the value of utilizing sandboxes—isolated environments where teams can safely engage with data before it goes live. Jamaria Kong, managing director of TowerBrook Capital Partners, noted that each of the firms within its portfolio maintains separate environments for testing new capabilities. Similarly, Kamal Jain, a principal data engineering manager at BT, remarked on the testing sandboxes in place at his company, which allow staff to work with pertinent data.

Jain emphasized the importance of stringent quality checks on the data used in these sandbox environments. “It's vital to maintain a safe space for experimentation while ensuring a single source of truth in our production environment,” he stated, underscoring its significance for developing various AI applications.

Pavithra Rajendran, a senior data scientist at Great Ormond Street Hospital for Children, identified cybersecurity as a critical issue in managing sensitive patient information. “Given the complexity of cybersecurity needs in our sector, we found that our security requirements were too stringent to explore cloud options effectively,” Rajendran explained.

Data quality also emerged as a key topic during the discussion, with panelists stressing the necessity of cultivating a workforce equipped to understand and enhance data integrity. Garcia emphasized the human element in this process, stating, “It's not solely about having sophisticated algorithms; rather, it's essential to implement guidelines that ensure people who engage with data and create solutions are involved in the decision-making loop.”

Kong further noted that her firm is actively investigating the use of generative AI to improve data quality. Meanwhile, Jain indicated that BT has partnered with hyperscalers to conduct thorough data quality assessments. This multifaceted approach highlights the critical interplay between technology, people, and data in driving effective solutions within today's data-driven landscape.