Don’t let data latency ruin the value of Insights: Use RTDS to build high velocity data pipelines

It is a well-known fact, thanks to the widely popular Latency Curve by Dr. Hackathorn, that the longer the time-difference between a business event and action (preventive or corrective), the lesser chance of potential impact of the action. To put it simply, data latency destroys the value of business insights. As a result, new architecture paradigms, technologies and tools promising to reduce data latency keep arising.

Lately, data streaming has emerged as a preferred choice to address latency challenges. Numerous tools, both open source and commercial, are available and accomplish the same. However they suffer from following shortcomings:

• Restrictive end-points and data volume-based pricing models – Separate databases and schemas qualify as end-points rapidly adding to investments even for modest setups.
• Additional workload on source systems – Running queries (at short time intervals) that poll source systems for changes is common practice.
• Generic JDBC based target systems integration – Optimized tools for loading target systems such as Snowpipe for Snowflake are not leveraged.
• Rigid design limiting out-of-the-box suitability – Specifics such as PII protection and in-flight analytics are excluded for user implementation. This requires custom user implementation leading to longer turnaround times.
• High total cost of ownership (TCO) – End-point based licensing coupled with restrictions on data volumes leads to significant upfront capital expense. In addition, ongoing enterprise support by end-points contributes to high operational expenses. For example, a medium scale set-up with 10 endpoints could cost up to $1.6M USD in CAPEX and about $450K USD in OPEX.

The above challenges can prohibit most businesses from adopting commercial data streaming solutions. To address these concerns, Infocepts created its own proprietary data streaming solution – Infocepts Real Time Data Streamer (RTDS).

What is Real Time Data Streamer?

RTDS helps enterprise set-up high-performance real-time data pipeline with ease. It provides flexibility of a custom solution and the stability of a packaged solution at the same time. In addition, RTDS provides cost savings up to 4X compared to similar commercial tools, making it an affordable option for every size organization.

Case Study: US Fashion Retailer

A leading fashion retailer focused on selling clothes, shoes, watches, handbags, and other accessories leveraged Infocepts’ RTDS to set-up high-performance real-time data pipeline with ease. They were challenged with integrating all of their distribution center (DC) systems data, like their Warehouse Management (WMI) and Labor Management (LM) into a data warehouse for a high priority transformation project. Through this project the customer aimed to:

• Optimize omni-channel operations by seamlessly integrating their website, POS and warehouse data
• Implement digital walls enabling real-time insights into distribution center operations, promoting transparency and efficiency
• Optimize Free Trade Zone (FTZ) reporting to prevent non-compliance costs
• Integrate data from additional systems such as Labor Management (LM) to optimize distribution center operational planning

However, due to challenges like capacity restrictions plaguing source systems (IBM DB2 iSeries and DB2 LUW), data sensitivity (PII data) and velocity (sub-second latency) open source solutions could not help and leading packaged solutions proved to be very expensive thus not fitting the bill (literally).

At this point they turned to Infocepts. We helped develop and implement RTDS as a solution to help the customer see:

• Estimated efficiency gains in distribution center operations of over 1.2 M USD over the course of 3 years
• Significant savings in taxes and duties due to timely Free Trade Zone (FTZ) reporting
• Intangible benefits such as improved transparency of digital walls and better productivity through self-service analytics
• To top it all, lower TCO of RTDS lead to jaw-dropping cost savings of 1.4 M USD over 3 years

Summed up, Real Time Data Streamer did not just enable real-time it enabled real-value!

If you have identified data latency as issue to be fixed, trust Infocepts RTDS to take care of the rest. Schedule your demo with an Infocepts expert today and start saving money tomorrow!