Data is the key to the future. It's (allegedly) the source of every tech company's decisions, and it's become an essential component not only of the software industry, but verticals around the world. The relationship between Big Data, AI, and analytics is well-known at this point, so let's look at where industry experts see these technologies going in 2020.
- Sara Faatz, Sr. Product Marketing Manager, Progress: AI and ML get top billing: The increased popularity of artificial intelligence (AI) and machine learning (ML) and its promise to automate everyday tasks means developers with these skills and expertise are in high demand. AI and ML are the brains of the smarter applications, and through these technologies the apps learn from patterns of behavior and are able to more intelligently respond or produce an action. There is so much that can be done with the data that is collected by modern organizations.
Since organizations are beginning to prioritize the implementation of AI and ML across the entire business, developers who understand how to build, implement and use AI and ML effectively – and understand that these are powerful tools – will be in high demand in 2020.
- Haoyuan Li, founder and CTO, Alluxio: Hadoop storage (HDFS) is dead. Hadoop compute (Spark) lives strong. There is a lot of talk about Hadoop being dead... but the Hadoop ecosystem also had many rising stars. These were the compute frameworks like Spark that extracted more value from data. Others like Presto have also been adopted into the broader compute ecosystem. So today’s Hadoop has been broken up. Hadoop storage (HDFS) is dead because of its complexity and cost and because compute fundamentally cannot scale elastically if it stays tied to HDFS. To glean immediate, real-time insights, users need immediate and elastic compute capacity that’s plenty available in the cloud. Data in HDFS will move to the most optimal and cost-efficient system be it cloud storage or on-prem object storage. HDFS will die but Hadoop compute will live on and live strong.
- Tomer Shiran, CEO and co-founder, Dremio: The rise of data microservices for bulk analytics. Traditional operational microservices have been designed and optimized for processing small numbers of records, primarily due to bandwidth constraints with existing protocols and transports. But now this long-standing bottleneck issue has been solved with the arrival of Apache Arrow Flight, which provides a high performance, massively parallel protocol for big data transfer across different applications and platforms. We predict that in 2020 Arrow Flight will unleash a new category of data microservices focused on bulk analytical operations with high volumes of records, and in turn, these data microservices will enable loosely coupled analytical architectures that can evolve much faster than traditional monolithic analytical architectures.