Breaking down silos with DataOps

I was watching a movie “Ford Vs Ferrari” over this weekend which depicts one of the most epic rivalries in the world of Automobile. The biopic film shows the quest of a car designer and driver cum engineering specialist who wants to build a world-class racing car for Ford Motors which is capable enough to beat Ferrari at Le Mans, a 24-hour race. To make this happen, Carroll Shelby (car designer) sensitizes Henry Ford-II about multiple bureaucratic red tapes at Ford Motors that they need to leap through to seek reduction in car’s feedback loop.

This reminds me of “Conway’s Law” which when applied to enterprises using various software systems implies – “Organizations are constrained to produce system designs which reflect its own communication style.” Conway’s law provides a particularly important hint towards addressing challenges due to complex data teams and their data pipelines in data analytics systems.

This brings the need of “DataOps” to the fore!

Much more than hype

DataOps is a methodology to automate and optimize challenges in data management to deliver data through its lifecycle. It is based on the similar collaborative culture of Agile and DevOps foundations to balance control and quality with continuous delivery of data insights.

The landscape of data and business intelligence technologies are changing by leaps and bounds. As enterprises try to maximize value from data over a period, they moved from relational databases (RDBMS) to data warehouses (DW) to address growing data volume challenges, then from data warehouse (DW) to data lake (DL) enabled by cloud to address scalability and reliability challenges. Recently some teams have been migrating from data lake (DL) to Delta Lake for turning data lake transactional and to avoid reprocessing.

The evolving architecture patterns and the increasing complexity of all the data V’s (volume, variety, veracity etc.) is impacting the performance and agility of data pipelines. Businesses need more agile, on-demand, quality data to serve newer customer demands and keep innovating continuously to stay relevant in the industry.

Even though DataOps sounds like yet another marketing jargon in heavily crowded list of “*Ops” terms used within software industry, it has its own significance and importance. As stated in Conway’s law, different data teams scattered across organizations in the form of traditional roles (data architects, data analysts, data engineers etc.) as well as newer roles (machine learning (ML) engineers, data scientists, product owners etc.) work in silos. These data stakeholders need to come together to deliver data products and services in an agile, efficient, and collaborative manner.

DataOps addresses this concern along with bringing agility and reducing waste in time-to-value cycle through automation, governance, and monitoring processes. It also enables cross-functional analytics where enterprises can collaborate, replicate, and integrate analytics across their business value chain.

The method to madness!

The common goal of any enterprise data strategy is to utilize data assets effectively to fulfil an organization’s vision. DataOps plays a pivotal role in operationalizing this strategy through the data lifecycle. A set of steps to help you design a holistic DataOps solution design is outlined below:

Assess where you stand:

To design a DataOps solution that guarantees adoption, a detailed study involving enterprise people, process and technology is required. An enterprise-wide survey outlining current maturity through questionnaires is a great beginning to this journey. Undertake a maturity assessment involving key stakeholders within the enterprise covering the following areas:

• Customer journeys and digital touchpoints
• Enterprise data culture
• DevOps lifecycle processes and tools
• Infrastructure and application readiness
• Orchestration platforms and monitoring frameworks
• Skillset availability and roles definition
• Culture and collaboration across teams and functions

Design for outcomes:

A well-designed DataOps solution should have the following capabilities. Ensure these capabilities are catered to in your DataOps solution design.

• Real-Time Data Management – Single view of data, changes captured in real-time to make data available faster
• Seamless Data Ingestion and Integration – Ingest data from any given source database, API, ERP, CRM etc.
• End-to-End Orchestration and Automation – Orchestration of data pipeline and automated data workflow from environment creation, data ingestion, data pipelines, testing to notifications for stakeholders
• 360-Degree Monitoring – Monitoring end-to-end data pipeline using techniques like SPC (statistical process control) to ensure quality code, data, and processes
• Staging Environments and Continuous Testing – Customized Sandbox workspaces for development, testing to higher environments which promotes reuse
• Elevated Security and Governance – Enabling self-service capability with a secure (metadata, storage, data access etc.) as well as governed (auth/permissions, audit, stewardship etc.) solution

Make the right tool choices:

Make tool choices based on your use case, enterprise goals for DataOps and the capabilities you have considered as part of your design. Some tool choice considerations are provided below.

• DataOps solutions can be implemented using COTS (commercial off-the-shelf) tools or can be custom-built. To become a mature DataOps enterprise, it is important to have a repository of components that can be reused.
• There are specialized COTS tools that provide DataOps capabilities only or provide a mix of data management and DataOps capabilities. Some examples of COTS DataOps tools include: DataKitchen, DataOps.live, Zaloni, Unravel and so on.
• There are also several open source or cloud-native tool options that you could combine to implement your DataOps solution. Ex: GitHub, Jenkins, Nifi, Airflow, Spark, Ansible and so on.

In Summary, DataOps also allows enterprises to get better insights into pipeline operations, deliver data faster, bring resilience to handle changes and deliver better business results. DataOps enables organizations to take a step towards excellence in data transformation efforts and helps accelerate their IT modernization journey. It also empowers organizations to embrace change, drive business value through analytics and gain a competitive advantage in the market.

Get started with Infocepts to accelerate your DataOps strategy and implementation across the business value chain.