Data & Analytics Platforms: A Tale of Two Paths – The Right Path is the One You Commit To

Cloud modernization and, more specifically, data analytics in the cloud, can be an exercise in difficult decisions and ultimately taking a stance. When navigating the landscape of data and analytics platforms and understanding which is right for your business, frame the conversation from a lens that abstracts technology from your intended use of the data. A clearly defined goal will ultimately be the key to reaching the crucial decision point. What is it that you want to understand about your business through the careful analysis of your data?

It can also be beneficial to draw from experience in comparing similar and yet ultimately inherently different data analytics platforms to assist. Here, we delve into some of the reasons why Effectual has chosen to take an opinion on competing analytics technologies and, in the interest of helping our customers achieve an outcome, does not try to “be everything to everyone.”

Step One: Choosing a data & analytics platform has nothing to do with technology

Harnessing the power of data gives enterprise organizations a competitive edge in the race for innovation; however, the most important step in choosing your data & analytics platform has nothing to do with technology. The benefits of a strong data strategy are broad – which places the onus on the teams, individuals, and data champions to clearly identify the purpose and intent of your approach to data. Success starts with gaining consensus across the organization around the specific needs that can be addressed through strategic analysis of your data.

While often overlooked, defining these criteria is an essential first step that should take place long before considering what data and analytics platform is right for your business. Once you understand the “why” for your data platform, you may find that you are storing terabytes of data that ultimately provides little value to the business. On the other hand, you may discover that your business is not capturing the data that will inform the insights your teams are looking for. This stresses the importance of understanding the desired outcome of any data and analytics project before considering which data technologies to invest in.

Step 2: Defining key considerations for your business

Once you have an agreed upon “why” for your data and analytics platform, you can start defining specific criteria that will help determine which data platform is best for your business.

At a minimum, Effectual recommends considering the following factors:

• Security: Evaluate each platform’s security features and compliance standards to ensure that your data is protected and meets the regulatory requirements of your business and industry.
  
• Types of Users: Consider the needs of different users at your company, such as analysts, executives, and data scientists. Each platform may cater to different user types and offer varying levels of accessibility and ease of use.
  
• Use Cases: Assess the platform’s ability to support your specific use cases, such as streaming data, real-time analytics, or machine learning.
  
• Team Skillsets: Review the expertise of your teams, including their proficiency in SQL, Spark, or other relevant technologies. This will help you determine which platform is most suitable for your organization.
  
• Revisit Business Needs: Ultimately, choosing a data platform should be based on the business needs you determined in the first step.

With these five factors identified, you are now equipped with a scorecard to ask the right questions and make an informed decision on your data platform. It is important to take time to understand each aspect as it relates to your business and identified goals. What works for another company may not work for your business. There may be some contention here – procurement teams will be sensitive to cost, developers will be sensitive to their specific development environments and data stores, and executives will want instant, easy-to-use dashboards. You may be asking some teams to change their daily routines – and we know, change is hard.

Step 3: Gaining an understanding of your enterprise data & analytics platform options

As mentioned during the introduction of our path, ultimately, you’ll reach a fork in the road that requires you to make a decision. No matter how good your roadmap has been or how much preparation you’ve made, there will be no “perfect” fit. To combat the potential “analysis paralysis” that stems from the natural fear of taking the wrong path, it is important to first state:

“There is no wrong path – there is only the path you commit to.”

Another way of stating the above is that you can’t expect your organization to be the “Jack of All Trades” AND “Master of All” – you need to commit to a path and proceed with the confidence that it meets your business objectives and therefore will always be the right path.

To illustrate this further, let’s take a look at Effectual’s decision to commit to one path over another. For the purposes of this post, we will consider two technologies that fulfill the term “Data Analytics Platform” in its most basic definition: Databricks and Microsoft Fabric.

Both solutions are backed by technology giants, and both are viable paths forward. So, after defining our business objectives and our desired outcome, why did Effectual commit to one path over the other? Let’s follow the same process outlined above to derive the conclusion.

The highest priority in the largest of enterprises and even more so in the heavily regulated and compliant environments – this was an important consideration as Effectual serves enterprise customers and this is where our deep experience shines through.

Microsoft Fabric was found to havepolicies and permissions that don’t carry across its components, leaving potentially dangerous gaps and additional admin work to copy over. This admin work introduces human error into already risk-averse environments. In contrast, Effectual observed that Databricks excels in enterprise governance and security with a robust suite of features encompassing network protections, authentication, access control, encryption, and compliance standards. Its cloud-native controls ensure core security and efficient audit capabilities. The platform’s policy and permission settings, like the compliance security profile, provide a consistently applied security approach. Furthermore, the introduction of Unity Catalog centralizes governance, offering precise control over data and AI assets through standard ANSI SQL, hierarchical securable objects, and enhanced audit capabilities.

Another security challenge was an absolute necessity for granular control. On evaluation, we found that Fabric users who were denied access to data in one engine could still reach that data through another engine. In contrast, Databricks faced this challenge with dedicated product development cycles to remediate as a priority – a hyperfocus on solving complex business challenges akin to Effectual’s own.

When we asked ourselves what security our clients need in the real world, focusing on the use case of enterprise-grade security as the number one factor allowed us to take the first confident step down the path we could ultimately commit to – more steps followed.

Types of Users, Use Cases, and Team Skillsets – are all key factors, but each presents its own challenges. Cloud modernization and Data Analytics at scale are already a minefield of complexity, and avoiding additional complexity is essential when dealing with transformations at scale. Effectual found that for highly complex customer environments Fabric introduced increased complexity for integration and management due to less flexible provisioning models carried over from Microsoft’s older subscription data warehouses. On the other hand, Databricks fit Effectual’s customers’ desire to reduce complexity and adopt a cloud-native integration, management, and provisioning model that came with a cloud-native economic model.

The final consideration that allowed Effectual to commit to a path that best suited our customers’ requirements of their transformation partner was the inclusion of Databricks in the Leaders Quadrant for Data Science and Machine Learning Platforms by Gartner. Effectual consulted with numerous customers as Machine Learning and Data platforms were first emerging on the market – a period of uncertainty and hesitancy combined with the heavy fear of “being left behind” – not dissimilar to how Generative AI is being discussed today. Ultimately Effectual needed the best solution to source data, build models and operationalize machine learning that could be installed and operated at scale within a real enterprise analytics function. Microsoft Fabric wasn’t launched until 2023 – let alone was it up for consideration in a Magic Quadrant at the inception of Machine Learning and Data Science. As Effectual specializes in the leader of the Strategic Cloud Platform services, Amazon Web Services, Databricks presence in the leaders quadrant where Fabric was yet to exist, still to this day makes Effectual’s commitment to our path one that we walk confidently.