You’ve probably heard striking stats around data growth and how it continues to increase. Companies are generating more data today than ever before, with 64.2 zettabytes of data created in 2020, according to IDC (the estimated amount of data created is expected to increase by 23% annually). At the same time, IDC also says that only 38% of data is actually put to use.
How do CIOs solve this disparity? It’s not necessarily about “boiling the data ocean.” In fact, a proven path to immediately turning data into value is focusing on one important area – harnessing operational data to build real-time applications.
These applications use data in real time to delight customers, drive revenue, or both. When done well, individual use cases are able to stand alone initially, while also getting “better together” as additional data is collected and use cases deployed. Consider, for example, a retailer’s e-commerce site can serve up new, personalised product recommendations at the moment you put an item in your shopping cart. Now imagine adding in smart substitutions for products that are out of stock. The additional interaction and data will improve both together.
Building on Data
Creating these kinds of experiences and real-time actions requires a singular focus on taking advantage of the data that matters most, in the moment it matters most.
To succeed at this in practice, you must first embrace the fact that the velocity your data needs and the amount of latency allowed isn’t up for negotiation. “Real time” is what the customer expects it to be. Depending on the type of experience, that might mean milliseconds to seconds to (in some cases) even minutes – but if you can’t figure out how to meet that expectation, someone else likely will.
Table stakes are ensuring multiple elements work well together, from the databases that process and hold your operational data, through to the event streaming and messaging technology that routes that data where it needs to go.
For most enterprise companies, it also means looking at multi-cloud and geo-replication from the start to support the requirements of real time apps and the amount of data growth that will occur as they become successful. No-one wants their critical application – and their business – slowed down due to issues around scale. An ideal data stack should make it easier to keep data near to users around the world to reduce latency. It should also run concurrently across multiple providers so you are not tied to any one company’s cloud.
Beyond the technical concerns, you must consider the cost of that infrastructure over time too. You don’t want to be hampered when scaling up takes extra time, and you don’t want to spend extra over-provisioning on infrastructure or pre-paying for unused consumption.
Why Developers Matter
To make the right real time application infrastructure sing, you need happy and productive developers.
Your software developers want to use open source where they can. According to Boston Consulting Group, 99% of Fortune 500 companies use open source, 35% of all enterprise software is based on open source, while open source will power 85% of all public cloud workloads by 2024. Open source projects make it easier for developers to innovate, and they are used successfully in many of the largest applications in the world.
They will also want to use microservices architectures and cloud-native technologies. These design approaches make it easier and faster to build those applications. Using Kubernetes makes it easier to scale and automate those applications from a management perspective.
To help developers be more productive, tell them what to build, but then let them get on with their tasks. This will involve catering to the tools that they will prefer to use – commonly, this will include JAMstack, Spring, Go, Ruby, Scala, Javascript, Typescript and C#. Each of these languages and frameworks will need to interact with your data stack, so look at how to use APIs to integrate services together more efficiently.
Rather than running multiple databases to interact with specific services and tools, using API level connections instead lets you provide data services. Developers can get started faster and run more efficiently. Give them SDKs, tooling and documentation that are idiomatically correct.
Alongside the big infrastructure decisions, there is the day-to-day developer experience to think about as well. When you spend millions on cloud infrastructure and developer salaries, the last thing that you should do is leave those resources hanging due to poor compile and build times. Carefully considering developer experience is important because it makes developers more productive. It also improves staff retention – developers that have an easier experience are less likely to look for a new role.
Taken together, these are the new architectures and approaches that drive the applications that are part of our daily lives. We rely on these applications but we can’t forget that developers have to create and run them behind the scenes, with calluses on their hands from building them. This includes planning out how to use the data that these applications create, and how this can then power the business and provide value to customers.
By Bryan Kirschner, Vice President Strategy, DataStax.
In 2006, Kirschner was named the first Director of Open Source Strategy in the history of Microsoft. He founded the Apigee Institute in 2013. Following the company’s acquisition by Google, he led research and strategy for Google Cloud Developer Relations and DevOps Platform. He holds a Bachelor of Arts in Philosophy from Yale University.
