Empowering Enterprise IT with Azure CycleCloud for Cloud HPC and Big Compute Environments

Overview

Enterprises increasingly turn to cloud solutions to meet their computational demands in the ever-evolving landscape of High-Performance Computing (HPC) and Big Compute. Azure CycleCloud, a powerful orchestration tool provided by Microsoft Azure, plays a crucial role in simplifying the deployment, management, and optimization of HPC and Big Compute environments. This article aims to demystify Azure CycleCloud, providing a comprehensive introduction to its features, deployment, and configuration of Azure CycleCloud on Azure for enterprise IT.

Introduction

The automated configuration of Azure CycleCloud allows IT teams to prioritize delivering services to business users without being bogged down by manual setup tasks.

Azure CycleCloud does not include built-in job scheduling functionality and is not a scheduler; it functions as a platform allowing users to deploy their chosen scheduler within the Azure environment. While it supports commonly used schedulers such as PBSPro, Slurm, IBM LSF, Grid Engine, and HT Condor, many Azure CycleCloud users develop and implement their schedulers using the available autoscaling API.

Azure CycleCloud does not prescribe a specific cluster topology. Instead, it provides installation templates designed for the swift deployment of HPC systems in Azure. However, HPC operators can customize these templates according to their specific infrastructure needs. The Azure HPC community offers pre-configured templates tailored for various workloads and industries to streamline the customization process further.

Deployment and Configuration of Azure CycleCloud HPC System on Azure Infrastructure

The complete Azure CycleCloud HPC system can be deployed on Azure infrastructure. Azure CycleCloud is installed as an application server on a virtual machine (VM) in Azure, requiring outbound access to Azure Resource Provider APIs. Subsequently, Azure CycleCloud initiates and oversees the VMs that constitute the HPC systems. These typically include the HPC scheduler head node(s) compute nodes and may incorporate VM-based Network Attached Storage like an NFS server or BeeGFS cluster, login nodes, bastion hosts, and other essential components supporting the HPC infrastructure. The composition of the HPC system is exclusively defined through Azure CycleCloud templates. Moreover, Azure CycleCloud HPC environments have the flexibility to leverage other Platform as a Service (PaaS) offerings, such as Azure NetApp Files, Azure HPC Cache, and Azure Active Directory Domain Service.

Features of Azure CycleCloud

1. User Access: Azure CycleCloud offers built-in support for establishing local user accounts on each node within an HPC system. This allows centralized control over user access through a unified management plane, eliminating the need for deploying a separate directory service.
2. Monitoring: The Azure CycleCloud UI displays node-level metrics, facilitating the monitoring of system loads. These metrics can be integrated into reporting and alerting services for comprehensive performance tracking.
3. Logging: Azure CycleCloud provides a logging system that captures activities and events at the node and application server levels.
4. Portability: Unlike requiring a specific VM image or operating system, Azure CycleCloud supports major Windows and Linux operating systems on HPC nodes. Users also have the flexibility to create and utilize their custom VM images in their HPC systems.
5. Infrastructure as Code: With everything defined in templates and configuration scripts, HPC systems deployed through Azure CycleCloud are both repeatable and portable. This consistency in deployment extends across various environments such as Sandbox, Development, Test, and Production. Operators can replicate identical HPC systems for different business groups or teams, ensuring separation of accounting concerns.
6. Loosely Coupled or Tightly Coupled Workloads: Azure CycleCloud-designed HPC clusters cater not only to loosely coupled or embarrassingly parallel jobs where scale is paramount but also consider Azure’s Infiniband backbone. This design supports tightly coupled or MPI-based workloads, emphasizing node proximity and network latency. The scheduler integrations within Azure CycleCloud inherently support scale-out and tightly coupled concepts.

Conclusion

Azure CycleCloud emerges as a powerful solution for enterprise IT seeking to navigate the complexities of Cloud HPC and Big Compute Environments. By offering dynamic scaling, automated configuration, and versatile support for various schedulers, Azure CycleCloud empowers organizations to optimize resource utilization efficiently. Its flexibility, portability, and infrastructure-as-code approach provide a seamless deployment experience across diverse environments. With user-friendly features such as user access control, monitoring, logging, and support for loosely and tightly coupled workloads, Azure CycleCloud is a comprehensive tool for IT professionals.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.