The DataNeuron-NetApp partnership aims to streamline and accelerate the enterprise adoption of Generative AI. DataNeuron's LLM Studio presents a no-code platform for developing and implementing AI workflows, including RAG (structured/ unstructured), prompt generation with DSEAL, response generation with EVALS and LLM Judge, and fine tuning. NetApp complements this with enterprise-grade intelligent data infrastructure services like NetApp® Snapshot™ and FlexClone® technologies for efficient data management and versioning.
DataNeuron and NetApp have partnered to create a GenAI solution that addresses the complications of on-premises, multicloud, and edge deployments. The alliance aims to deliver a unified, secure, and compliant GenAI solution that lowers operational costs, accelerates time-to-value, and meets business needs for data protection, privacy, customization, and scalability.
DSEAL: Revolutionizing Dataset Curation for NetApp Volumes
NetApp customers often deal with large and diverse datasets stored across extensive NetApp volumes, making data curation slow, costly, and labor intensive. Traditional methods, such as manually selecting data, lead to inefficiencies, high processing costs, coverage gaps, and stagnant dataset quality. These challenges, ranging from error-prone manual overhead to inefficient full-volume scans that strain GPU and memory resources, require a more effective approach to handle massive data volumes and improve model performance.
DataNeuron’s DSEAL (Divisive Sampling & Ensemble Active Learning) optimizes dataset curation by automating and streamlining the process. DSEAL targets the most informative data segments, reducing preprocessing time and GPU usage. Through divisive sampling, it minimizes redundancy and ensures balanced topic coverage, reducing bias. An active learning loop iteratively refines data selection, continuously enhancing dataset quality.
Our experiments confirmed that DSEAL accelerates data processing, cuts manual effort by 95%, reduces bias, and improves decision making, positioning it as a powerful tool for AI-driven data curation.
As demonstrated in the example above, DSEAL ensures sampling across clusters by precisely capturing diverse data segments. It preserves boundary integrity and maintains topic diversity, even when operating with large, complex data volumes.
Testing: Cloud and On-Premises Synergy
DataNeuron and NetApp worked together to test their solutions in both on-premises and cloud environments, focusing on seamless integration, reliable performance, and enterprise-grade scalability.
Cloud Services
DataNeuron’s LLM Studio was integrated with Google Cloud NetApp Volumes on Google Cloud. Leveraging NetApp Volumes features like Snapshot and FlexClone, DataNeuron enabled efficient data Snapshot copies and versioning without additional storage overhead. The proof of concept (PoC) successfully validated all DataNeuron workflows integrated with NetApp Volumes via the NFSv3 protocol, leveraging the NetApp DataOps Toolkit and Google Cloud libraries. A field-validated design was also released, detailing the integration and demonstrating how DataNeuron and NetApp together enable secure, scalable, and customizable LLM deployments for the enterprise.
On Premises
During the on-premises PoC at the NetApp lab, DataNeuron’s LLM Studio was successfully deployed within a Kubernetes cluster and seamlessly integrated with NetApp ONTAP® data management software. The PoC validated key DataNeuron workflows, including Retrieval-Augmented Generation (RAG), data curation, and fine tuning, while also demonstrating the effectiveness of NetApp technologies such as Snapshot and FlexClone. These capabilities enabled efficient data management and space-optimized versioning across AI workflows, including project forking without duplication of data.
The POC also showcased scalable distributed inference with LLMs of up to 70B parameters. DSEAL-based data curation was tested as well, enabling the generation of a high-quality, diverse training dataset, particularly effective for fine-tuning LLMs on large, complex data volumes managed through a NetApp infrastructure.
DataNeuron AI Studio Powered by NetApp
Call to Action
Ready to accelerate your GenAI initiatives with secure, scalable infrastructure?
Explore DataNeuron LLM Studio: https://dataneuron.ai/
Discover NetApp AI Solutions: https://www.netapp.com/
Contact our experts:
Bharath Rao (DataNeuron): Bharath@dataneuron.ai
Larry Bunka (NetApp): Lawrence.Bunka@netapp.com
Embark on a journey where data infrastructure meets AI innovation and unlock secure, efficient, and scalable LLM workflows today!
NETAPP, the NETAPP logo, and the marks listed at https://www.netapp.com/TM are trademarks of NetApp, Inc. Other company and product names may be trademarks of their respective owners.
