• 75% reduction in LLM costs per jailbreak attempt
• 25% performance boost in attack success rate (ASR)
• ‍25 FTE hours saved per week

‍

Dynamo’s Vision for Safe AI Adoption

NomadicML empowers ML teams to continuously optimize compound AI systems, from demo to production. Our platform offers a unified, systematic approach for refining hyperparameters, models, prompts, and components, enabling teams to confidently meet their success metrics.

‍

Founded in 2022, Dynamo AI provides enterprises with privacy, security, and compliance solutions for responsible AI adoption. Their Fortune 1000 clients span finance, electronics, insurance, and automotive sectors. “Our vision is to empower every organization to harness AI’s transformative potential with confidence and control. ,” says Eric Lin, VP of Applied AI.

‍

Two of Dynamo’s flagship products include:

• DynamoEval: An AI evaluation suite that stress-tests models for risks like data leakage and prompt injections, generating audit-ready documentation for frameworks such as MITRE ATLAS, NIST RMF, and OWASP Top-10.
• DynamoGuard: A guardrails platform that enforces custom AI policies and offers real-time moderation to prevent misuse. It allows organizations to monitor and adjust AI usage with customizable policies, fine-tuning guardrails based on real-world use.

‍

Why Dynamo Partnered with Us: Boosting Efficiency in GenAI Safety 

When Dynamo AI approached us, their goal was clear: optimize the efficiency of developing and running their safety products. Particularly, Dynamo wanted to productionize the latest techniques across jailbreaking and hallucination detection, bridging their research and product engineering teams. However, they were held back by blind spots, time-consuming manual experimentation, and high LLM costs.

‍

‍Jailbreaking and Red-Teaming: DynamoEval’s jailbreaking product tests AI models for vulnerabilities by feeding them unethical queries (e.g., “How to build a bomb”). However, productionizing the jailbreaking models was costly and slow. They spent 25 FTE hours weekly on manual hyperparameter tuning, relying on tedious individual grid searches.

‍

“Our team needed to ensure our platform consistently delivers robust performance for hundreds of different risk enterprise compliance scenarios, and continuously does so for new risk vectors that come up every month.  We needed a more efficient way to scale,” said Eric Lin.

‍

Hallucination Detection: Hallucination detection tests proved more challenging to optimize. Existing tools like LangChain lacked the ability to conduct the iterative prompt exploration at scale needed to develop robust hallucination detection tests, slowing down validation.

‍

“We had multiple internal detection models that need to be benchmarked against multiple off-the-shelf LLMs, on different types of hallucination detections. Prompting these LLMs in different settings yielded varying results that were difficult to reconcile. Nomadic's assistance in prompt optimization facilitated the internal model validation process."
– Joon Kim, ML Research Scientist

‍

Custom Guardrails: DynamoGuard offers custom AI guardrails, but tailoring them to enterprise-specific contexts (e.g. tuning guardrail strictness) required efficient hyperparameter tuning and training loop adjustments.

‍

Dynamo’s Setup: Continuous Parameter Tuning

Dynamo AI partnered with NomadicML to optimize core safety systems while confidently productionizing the latest techniques for jailbreaking and hallucination detection, bridging the gap between their ML research and engineering teams.

‍

Auto-HPO (Hyperparameter Optimization)

‍

Nomadic Auto-HPO automates hyperparameter tuning based on enterprise-specific evaluation metrics. Users can select the best state-of-the-art parameter search technique for their needs, including cost-frugal Bayesian optimization methods, which converge on optimal configurations within budget. Moreover, Nomadic continuously optimizes hyperparameter settings to production data by scheduling regular tuning runs synced to the Nomadic Workspace. Prompts, inference-time parameters, or even model choice—Auto-HPO ensures continuous optimization of system components.

‍

Dynamo AI’s Use Case:
Dynamo AI used Auto-HPO to cut LLM costs by 75% and save 25 FTE hours per week. By scheduling regular HPO runs and using cost-efficient Nomadic tuners from FLAML, they reduced average LLM queries per jailbreak from 60 to only 18.

‍

Iterative Prompt Tuning

‍

Prompt playground-level tuning, via SDK. Nomadic’s unique iterative prompt tuning library helps teams refine their prompts through real-time feedback loops, unlocking both thorough exploration and quick convergence to the best-performing prompts. By integrating with DSPy and supporting custom iterators, Nomadic enables continuous adjustments to prompt templates.

‍

Dynamo AI’s Use Case:
DynamoGuard improved the accuracy of their hallucination detection offering, which relies on LLMs as judges to verify faithfulness of model outputs. Previous tools like LangChain couldn’t handle iterative exploration at scale, slowing down optimization. With NomadicML, Dynamo boosted hallucination detection accuracy by 13%, strengthening their model evaluations.

‍

Continuous Optimization at Scale

‍

The Nomadic platform enables real-time, observable experimentation. With SDK integration into the Workspace, teams can easily manage projects, API keys, and configurations for streamlined collaboration. Users can also automate routine tuning to ensure parameter settings stay optimized as data changes.

‍

Dynamo AI’s Use Case:
Dynamo AI’s team of 15+ ML researchers leveraged Nomadic’s SDK and Workspace to scale experimentation across various projects. By continuously running Nomadic on their red-teaming and hallucination-detection systems, Dynamo was able to quickly adopt newer, better-performing models and jailbreaking techniques as soon as they’re made available. 

‍

Testing Locally

‍

Testing in Production

‍

Deep Dive: DynamoEval’s Jailbreak Attacks

Let’s dive into how Nomadic optimized Dynamo’s jailbreaking / auto-red-teaming offering.

‍

DynamoEval is their automated evaluation suite to stress-test AI models for vulnerabilities such as data leakage, jailbreaking, and hallucinations. Prior to NomadicML, Dynamo’s hyperparameter tuning efforts for adversarial jailbreaking algorithms were limited to expensive grid searches run periodically to ensure state of the art attack success rates.

‍

NomadicML’s Auto-HPO revolutionized Dynamo’s approach through continuous tuning of the jailbreaking system.

‍

"Nomadic helped us mitigate scaling issues on one of DynamoEval's flagship attack algorithms. We now have the ability to quickly adapt to evolving definitions of safety for AI red-teaming, while leveraging the best and most-efficient HP setting.”
– Eliott Zemour, Senior ML Research Engineer

‍

With NomadicML’s Auto-HPO, DynamoEval was able to reduce LLM costs and improve jailbreaking success rates. Additionally, as their customer base grows, these cost savings will scale exponentially.

‍

• Optimal Configuration: NomadicML helped Dynamo AI confidently find the best hyperparameter configuration for their tree-based algorithms, and migrate onto newer, cheaper open-source models. This configuration reduced costs by 75% and was 10X cheaper than running the same tests on GPT-4, all while maintaining an 85% attack success rate (ASR)!
• Efficiency Gains: Auto-HPO minimized both cost and latency by reducing the number of queries needed per jailbreak attempt while maintaining high ASR.  Cutting average queries per jailbreak from 60 to 15 queries, DynamoEval jailbreaks both ran faster for customers and reduced cost per jailbreak by 5X.
• Active Tuning: Dynamo AI leveraged Nomadic’s SDK to routinely schedule experiments to discover optimal hyperparameters, ensuring their system could incorporate the latest jailbreaking techniques from research (e.g. from ICML) and stay ahead of evolving customer data.
• Increased performance boost: NomadicML’s Auto-HPO also identified an alternate configuration to the low-cost setting, which resulted in increased performance while keeping costs similar. This configurations performed significantly better than the baseline - a 25% boost in the Attack Success Rate from 70% to 85%.

‍

Dynamo also appreciated the hands-on expertise with compound AI system optimization that the Nomadic team brings.

‍

“Our experience working with Nomadic was exceptional. Their team's level of engagement was unparalleled, setting them apart from other hyperparameter optimization platforms. The Nomadic team consistently exceeded expectations by not only responding promptly to our feature requests but also automating processes we didn’t even think to ask for.”  
– Eric Lin, VP of Applied AI, Dynamo AI

‍

Learn More

To explore our open-source AI Safety solutions, check out our open-source library and corresponding red-teaming tutorial here. 

‍

Get Access to Nomadic

Install the Nomadic SDK to start running local experiments for free, following our Quickstart.

pip install nomadic

To sync results with the Nomadic Workspace, sign up for our Workspace private beta. You will need a Nomadic account and an associated API key. 

‍

What’s Next?

We are excited to deepen integrations into Dynamo’s AI safety stack to continuously improve performance, particularly in powerful emerging areas, including cutting-edge jailbreaking techniques (reinforcement-learning-based agents) and post-training guardrail foundation models.

To learn more about Dynamo AI’s privacy, security, and compliance solutions, visit dynamo.ai/solutions. Find us at info@nomadicml.com!