2022 has seen a monumental shift as organisations mitigate the risks of legacy platforms and seek to capitalise off the myriad of new capabilities and possibilities modern platforms present. For tech leaders at the start of their optimisation journey, mastering their data platforms can be a daunting challenge. Those further along their journey, face complex data ecosystems often plagued by rising costs, dwindling performance levels and limited observability. The route to optimisation is not always clear.

It is, however, those little-known insights into how to maximise tooling, free up capacity and troubleshoot issues faster that can provide the key to these common challenges. Nimbus Ninety, in partnership with Unravel Data and Databricks, brought together technology and data experts to connect, share challenges and offer up actionable insights to master a modern data platform.

OPTIMISING A UNIFIED ANALYTICS DATA PLATFORM

The evening began with a wonderful presentation from James Fielder, Lead Data Engineer at Cox Automotive who reflected on his experience of optimising their data platform over the last four years. James highlighted how previous iterations of their data platform were limited by computing costs, lack of autoscaling and deployment and workflow difficulties. As such, James and his relatively small team set about reconfiguring their data platform so that it was fit for purpose.

The first step James and his team took was to focus on optimising their data platform by prioritising internal use cases; emphasising tooling for reporting, analytics and trends that link into wider business strategy. Secondly, they focused on architectural simplicity and prioritisation. This involved prioritising Databricks which benefits from seamless autoscaling, greater storage, manageable data workflows and powerful improvements, whilst also limiting their selection of tools. Thirdly, they enlisted the observability expertise of Unravel Data to gain a better understanding of the health of their data. Through these steps, James was able to produce a highly optimised data platform that met the needs of his organisation.

Throughout their optimisation journey, James and his team faced a number of key challenges. The first was recruiting data scientists with the necessary skills and experience to use their newly designed data platforms. The team also addressed governance, tool construction and new data source challenges. As we look forward, James is looking forward to adding powerful features and gaining further value out of Unravel’s insights.

ROUNDTABLE DISCUSSION

Following James’ presentation, members joined a roundtable discussion and took part in a number of polls, in order to share their experiences. The key takeaways from the discussion were:

• What’s the next step for you?

Those members at the start of their journey, 40% of attendees, highlighted the specific challenges facing them. Experts from retail and the airline industry were particularly concerned about the cost of optimisation given the costly migrations they have just been through. Meanwhile, experts from larger financial service, energy and pharmaceutical organisations were envious of these greenfield deployments, citing the ability to manage upscaling, start afresh and maintain simple tooling. With fully operational cloud platforms, members from these larger organisations, another 40% of attendees, were concerned with data governance, managing stakeholders and nurturing a productive data culture.

• Cost and Performance is Key

53% of attendees highlighted that the most important outcome of a modern data platform is cost and performance. In particular, attendees explored how performance is linked to stakeholder management and business case. In terms of stakeholder management, members highlighted how data platforms should be optimised with all employees, from top to bottom, in mind. Furthermore, members, in particular from financial services, saw performance in terms of business case; ultimately, how it is being used. Alternatively, 33% of attendees were more interested in monitoring and visibility. Members from larger organisations said that tracking use cases, data stewardship, is particularly important before choosing between tools.

• What are your biggest challenges

When asked about their biggest concerns for cloud migration, 46% of attendees answered user skill gaps when adopting new technologies. Meanwhile, other members were interested in maintaining business as usual (23%) and delivering MVP’s to production (31%). Another challenge that emerged in the discussion was managing scale; in particular, how this impacts on cost. Many members suggested that, as echoed in James’ presentation, that prioritisation of technologies and simplifying tooling would help solve a number of these common challenges.

We held this event in partnership with Unravel Data.