Azure Managed Lustre for HPC and AI workloads

Microsoft has announced the general availability of Azure Managed Lustre, a managed high-performance parallel file system designed for high performance computing (HPC) and AI workloads. Azure Managed Lustre brings the reliable Lustre file system as a first-party managed service on Azure, allowing long-time on-premises Lustre users to transition seamlessly to the cloud. By leveraging this service, users gain access to a complete HPC solution that includes compute and high-performance storage, all integrated on Azure. With Azure Managed Lustre, businesses can now harness the high throughput and performance capabilities of Lustre while enjoying a seamless, managed experience that lets them focus on core business objectives.

The Need for Azure Managed Lustre

Lustre, one of the most widely-used distributed parallel file systems in the HPC ecosystem, has traditionally been deployed on-premises to meet the scalable and high-throughput storage demands of HPC workloads. As an open-source solution, Lustre has built a robust community of developers and users. With the rise of generative AI and the growing need for high-throughput storage solutions to keep expensive GPU cores from idling, Lustre has seen renewed growth in adoption. Organizations across various industries that depend on high-performance computing are increasingly seeking an end-to-end infrastructure solution that integrates compute, storage, and networking. This infrastructure must be fast to deploy, simple to manage, and capable of scaling to meet evolving data needs.

Key Features of Azure Managed Lustre

• Managed PaaS Experience: Azure Managed Lustre offers a fully managed platform-as-a-service (PaaS) solution, providing a hassle-free deployment and management experience for Lustre users.
• Cost Efficiency with Pay-as-You-Go Pricing: The service offers flexible pricing models, allowing customers to manage costs based on their workload requirements.
• Scalable Throughput Options: Azure Managed Lustre provides various throughput levels at different price points, accommodating diverse workloads while ensuring high performance.
• Independent Cluster Deployment: Users can quickly deploy independent clusters with predictable performance, eliminating the noisy neighbor issues often encountered in on-premises Lustre setups.
• Rapid Deployment: Clusters can be set up within minutes, removing the need for extensive planning and ensuring compliance with data residency requirements.
• Region Flexibility: Azure Managed Lustre supports deployments across multiple Azure regions, providing compliance and data residency options tailored to user needs.
• Scalability: The service can scale up to hundreds of petabytes (PB) with the durability, availability, and cost advantages of tiering to Azure Blob storage.
• Integration with Azure Services: Azure Managed Lustre seamlessly integrates with existing Azure services like Azure HPC services, Azure Machine Learning, and Azure Kubernetes Service, providing a cohesive HPC and AI environment.
• Support for Existing Automation: It offers investment protection for customers migrating from on-premises Lustre deployments by supporting existing automation and platform development tools.

Performance Options Available

Azure Managed Lustre offers two persistent, durable SSD-based instances that vary by performance:

• AMLFS Standard: Delivers 125 MB/s per provisioned Tebibyte (TiB) of capacity.
• AMLFS Premium: Provides 250 MB/s per provisioned Tebibyte (TiB) of capacity.

Users can access the Lustre client packages for their preferred Linux distributions and kernel versions via packages.microsoft.com. Additionally, Azure provides support for HPC images prebuilt with Lustre client packages for Ubuntu versions 18.04, 20.04, 22.04, and Alma 8.7.

Availability and Regions

Azure Managed Lustre is now available for Cloudfleet Kubernetes Engine clusters in several regions. To check which regions currently support Azure Managed Lustre, users are advised to consult the products available by region page. For more information on pricing, visit the Azure Managed Lustre pricing page.