Spark, big data, waste, MySQL, database, data, data monitoring, data management

A global survey of 1,100 IT, data and business professionals from large enterprises published this week finds nearly half (48%) citing a lack of skills as the biggest challenge to providing high-quality data.

Conducted by 451 Research, a unit of S&P Global Market Intelligence on behalf of BMC, the survey also finds enterprises struggling with errors/data loss (45%), lack of automation (43%), human errors/mistakes (43%) and scalability (40%).

Overall, on average more than half (52%) of data ingested/landed into the organization consists of data types from legacy applications or platforms such as mainframes. Relational data sources make up an average of 31% of the data while emergent data types such as rich media format or vector data account for 17%, the survey finds.

AWS

However, the rate at which emergent data types are being added is likely to substantially increase as organizations invest more in generative artificial intelligence (AI). The survey finds 75% of survey respondents that work for organizations that have more than 5,000 employees (357) are investing in generative AI or large language models (LLMs), compared to 50% of respondents that work for organizations with less than 5,000 employees (743).

BMC CTO Ram Chakravarti said as AI continues to evolve, organizations are moving toward more federated approaches to processing and analyzing data where it resides rather than attempting to move all their data into a single central repository. The massive amount of data required by AI models makes it more practical to bring compute to the data, he noted. In fact, because data has gravity, the more data there is in one location the more likely it becomes the place where AI inference engines will be deployed.

Itโ€™s more a matter of bringing the right proverbial horse to the right course depending on whether a specific use case requires access to data residing at the edge, in a data center or in the cloud, Chakravarti noted.

The one thing that is clear is there will be more types of data than ever to manage as generative AI is applied more frequently to, for example, video. Hopefully, as data management platforms evolve, IT teams will be able to take advantage of AI tools and automation platforms to manage all that data at scale. Many organizations today are relying on data engineers to programmatically manage data. The challenge is data engineers are often hard to find and retain, so organizations will need to provide IT administrators with the means to manage all the data at scale.

For better or worse, AI is shining a spotlight on data management practices that are often uneven at best. The root cause of many of these issues stems from the simple fact that end users create data using a wide variety of applications that have historically made it challenging for IT teams to really understand what data is most critical to the business. With the rise of AI, however, the models being created are only as good as the data being used to train them, so now itโ€™s only a matter of time before most organizations revisit how their data is being managed.

Techstrong TV

Click full-screen to enable volume control
Watch latest episodes and shows

Networking Field Day

SHARE THIS STORY

RELATED STORIES