DATA CENTER BDREN

Introduction

BdREN High Performance Computing (HPC) system has been implemented with the intent to provide Cloud services to all members of BdREN which are Higher Education Institutes (HEIs), Medical Colleges, Research Institutes and similar other organizations located in Bangladesh. A total of 34 Public universities and 1(one) Private Universities were targeted in the first phase which has lately been extended to additional 7(seven) Private Universities, 12(twelve) Medical Colleges and 10(ten) Research Institutes. The setup utilizes the Cloud services for Infrastructure as a Service (IaaS) and may be expanded to Software as a Services (SaaS) and Platform as a Service (PaaS) at a later stage once the applications will be there to place in the cloud.

The solution consists of 2 portions of work which are the HP Cloud System Matrix (CSM) implementation and HP Cloud Management solution. The CSM will supply the compute infrastructure and Cloud Management Service will provide the web portal, monitoring and automation services.

Server

Shared Disk Storage

The necessary shared disk storage for the virtualized server infrastructure will be provided and sufficient for the growth of Cloud demand.

The virtualized server infrastructure is a consumer of services provided by the disk storage infrastructure. To this end, it will provide a set of requests and requirements to the disk storage infrastructure.

It is assumed that the proposed disk storage will provide the virtualized server infrastructure with shared disk storage of required capacity, structure, performance attributes and will be provisioned in time for implementation of the virtualized server infrastructure.

Operational Backup and Recovery

The operational backup and recovery for the virtualized server infrastructure will be provided as an outcome but implemented by our partner CSL in this project. VMware will leverage the snapshot capabilities of the proposed shared disk storage system and will be tightly integrated with it. Backup and Recovery will be delivered in another document set and therefore the operational backup and recovery design is not cover in Cloud System Matrix.

Data Centre Network

It is assumed that the data centre Ethernet switches will be able to provide redundant 10 Gb links to the UGC and BUET data centers for use by the virtualized server infrastructure.

It is also assumed that the data center network infrastructure will support use of VLANs and that VLANs will be used. The network will be an Extended Network from UGC to BUET for Management network and Production

Network to ensure seamless failover of services in the event of disaster. This seamless failover is regard as no changes to the VMs services in term of IP address and IP segment. External DNS available to handle VM workload that need to publish to internet. Setup of such external DNS server is not part of the project.

Data Centre Hosting

The two data centres (UGC and BUET) have the following infrastructural and passive support facilities:

  • Data centre floor space
  • Power
  • HVAC (Heating, Ventilation and Air Conditioning)
  • Structured cabling
  • Connections to Network Switches
  • Patch panels or Fiber and network
  • WAN links

Solution Overview

Solution Objective

HP CloudSystem Matrix is an IaaS solution for private and hybrid cloud deployment models, built on proven HP Converged Infrastructure technologies, such as HP CloudSystem, Matrix Operating Environment and Cloud Service Automation for Matrix as well as SiteScope monitoring functionality.

HP CloudSystem Matrix is a complete solution implementing the HP Converged Infrastructure strategy. It is based on a shared services model, using pools of compute, storage, and network resources. Matrix integrates proven technologies to provide a complete platform upon which infrastructure services can be readily provisioned and adjusted to meet changing business demands.

HP CloudSystem Matrix makes use of industry-standard, modular components, but it is far more than simply a collection of components. In a single Converged Infrastructure platform, Matrix integrates technologies from across the HP enterprise portfolios that have been proven in the marketplace to provide value in demanding datacenter environments. Matrix also incorporates the offerings of leading application, management, network, and storage partners, so that the solution fits seamlessly into diverse customer environments and works with all traditional application workloads.

The Matrix management console, built on HP Matrix Operating Environment, combines automated provisioning, capacity planning, disaster recovery, and a self-service portal. The figure provides a solution diagram of HP CloudSystem Matrix:

MOE

The Matrix Operating Environment (OE) includes an integrated service designer, self-service infrastructure portal, and auto-provisioning capabilities. It includes the tools to manage and optimize resource pools, multi-tenancy, and a recovery management solution for Proliant. CSA for Matrix OE integrates basic application provisioning, monitoring, and patch management. Using included Cloud APIs, one can easily customize the operating environment to one’s specific requirements, enabling chargeback and billing integration, integration into approval processes, and other process automation tasks.

Cloud System

The goals of the BDREN CloudSystem Matrix implementation is to provide this foundational infrastructure capabilities for BDREN’s IaaS cloud computing services platform by enabling the aforementioned provisioning of elastic infrastructure services in the form of Virtual Machines, the optimization of these infrastructure resources, and the protection of the continuity of these services.

Solution Description Overview

Overview of the CSM in the Overall Cloud Solution

From the overall hardware deployed at UGC Data Center and at BUET Disaster Recovery site it can be represented by the diagram below and it can be concluded that the design itself can be articulated by final rack layout and layer by specific resources component.

Cloud Solution

Storage replication recovery has been configured between DC and DR that leverage on HP 3Par remote copy feature to replicate production data to disaster recovery sites. Although there is Data Protector Backup software that manages the backup data at production site, however the 3Par Remote Copy (RC) data in DR site will serve as secondary data backup if granular recovery is required. Matrix Recovery Manager is deployed along with RC as the disaster recovery solution for VMs for automated powering on of VMs when disaster will occur.

The infrastructure as a services perform the virtualization of computing hardware, storage, and network resources. The services within the infrastructure as a services category in the Supply layer are built on the foundation of VMware vSphere platform.

List of equipment used to build BdREN Cloud

HP Hardware Description Location Quantity
Blade Chassis Total 2 Chassis (1 Starter Kit + 1 Expansion Kit):
2xHP CloudSystem Matrix including 1x DL360pG8
32G for CMS server [Per Blade Chassis] 2x
VC Flex-10/10D Module(10x10Gbps SFP+ uplinks,
16x 10Gbps downlinks to Server NIC),
2x VC-FC 8Gb 24port(8x 2/4/8Gbps uplinks to
external SAN SW, 16x 8Gbps downlinks ports)
UGC DC 1
Blade Chassis (1 Unit): 1xHP Cloud System
Matrix [Per Blade Chassis] 2x VC Flex-10/10D
Module(10x10Gbps SFP+ uplinks, 16x 10Gbps
downlinks to Server NIC), 2x VC-FC 8Gb 24port
(8x 2/4/8Gbps uplinks to external SAN SW,
16x 8Gbps downlinks ports)
BUET DR 1
Blade Server Blade Server : BL460c Gen8 with 2x3.0GHz
Ten Core CPU/96GB RAM/2x600GB HDD/2x10Gb NIC
(Integrated)/2xDual Port 8Gb FC/HP
UGC DC 16
Blade Server : BL460c Gen8 with 2x3.0GHz
Ten Core CPU/96GB RAM/2x600GB HDD/2x10Gb NIC
(Integrated)/2xDual Port 8Gb FC/HP
BUET DR 8
SAN Switch HP StoreFabric SN6500B Fibre Channel Switch UGC DC 2
HP StoreFabric SN6500B Fibre Channel Switch BUET DC 2
Backup Server DL 380p G8/8x8GB memory/8x600GB SAS 10K/82Q
FC dual ports HBA
UGC DC 1
DL 380p G8/8x8GB memory/8x600GB SAS 10K/82Q
FC dual ports HBA
BUET DC 1
Tape Library HP MSL6480 Tape Library with 4xLTO-6 Drive and
160 Cartridge Slots
UGC DC 1
HP MSL6480 Tape Library with 4xLTO-6 Drive and
160 Cartridge Slots
BUET DR 1
Cloud Management Server Hardware for Cloud Management Software Server for
SA - DL380p Gen8 with 1xE5-2643 3.0GHz 4c CPU/16GB
RAM/2x600GB SAS/Red Hat Linux
UGC DC 1
Hardware for Cloud Management Software Server for
CSA/Sitescope/AM/OO/DB -DL380p Gen8 with 1xE5-2643
3.0GHz 4c CPU/16GB RAM/2x600GB SAS/MS Win 2008
5
Hardware for Cloud Management Software Server for
SA - DL380p Gen8 with 1xE5-2643 3.0GHz 4c CPU/16GB
RAM/2x600GB SAS/Red Hat Linux
BUET DR 1
Hardware for Cloud Management Software Server for
CSA/Sitescope/AM/OO/DB -DL380p Gen8 with 1xE5-2643
3.0GHz 4c CPU/16GB RAM/2x600GB SAS/MS Win 2008
5
Server Rack HP 11642 1075mm Shock Rack 42U, KVM Console Kit UGC DC 2
HP 642 1075mm Intelligent Rack 42U, KVM Console Kit 2
HP 11642 1075mm Shock Rack 42U, KVM Console Kit BUET DR 2
IP KVM HP 2x1Ex16 KVM IP Console UGC DC 4
BUET DR 2
Active Directory Server for AD-DL380p Gen8 with 1xE5-2643 3.0GHz 4c
CPU/16GB RAM/2x600GB SAS/MS Win 2012
UGC DC 1
Server for AD-DL380p Gen8 with 1xE5-2643 3.0GHz 4c
CPU/16GB RAM/2x600GB SAS/MS Win 2012
BUET DR 1

CloudSystem Infrastructure Overview

cloud infrastructure

UCG Data Center Hardware Solution

Network Infrastructure that enables the transmission of several network traffic such as production, management and backup between all other nodes within the solution.

AD, DHCP and DNS services are to be provided through the network infrastructure in the compute/server farm and are critical for CSM to provision and inventory logical servers. Logical servers are server identities that can be flexibly allocated virtual machines or physical blades.

  • Consist of two C7000 Blade Enclosures
  • 16 units of HP BL460C Gen8 blade servers each hosting VMware ESX server instances that allow for the virtualization of multiple guest OS VMs. These 16 blade servers are aggregated as a common virtualized computing pool as an ESX farm. Sizing estimates assumed requirement for the capacity of the infrastructure hosting minimum of 100 VMs as a start. It will be increased based on demand over a period of time.
  • Two Virtual Connect (VC) Flex10/10d enable a redundant wire-once server-edge virtualization of network connectivity for the CSM compute blades and VMs. The port connection has the capability of supporting 10Gb and 1Gb network with the correct type and version of SFP.
  • Two Virtual Connect (VC) VC-FC8Gb 24-Port enable a redundant wire-once for server-edge virtualization of SAN connectivity for the CSM compute blades and VMs.
  • Two HP SAN SN6500B 96/96 switches configured as a dual redundant fabric.
  • One unit of HP 3Par StoreServ 10800 array configured with one FC disk group with 200TB RAW capacity using 2TB NL SAS HDD and 20TB RAW Capacity using 600GB 15K FC HDD. The usable disk capacity is determined based on the RAID type configure in the disk pool.

Note: Virtual machine has been deployed instead of physical server with 3par Service Processor that allows for the monitoring and management for HP 3Par remote support to diagnose any potential events.

  • One HP DL380 Gen8 server configured as Data Protector Cell Manager server to manage the robotic controller host of HP MSL6480 tape library with 4 LTO 6 tape drives. It is proposed as SAN backup host for backing up VMs server and physical server for file system as well databases.
  • One HP DL360 Gen8 server, configured as the HP SIM CMS, hosts the core HP Insight Software management environment. The CloudSystem Matrix (MOE) solution will also be installed to integrate directly with HP SIM CMS. This allows Cloud management and System management to be managed in an integrated platform.

The MOE platform will provide infrastructure orchestration capabilities for IaaS and will integrate with HP CSA for Private/Public Cloud Self Service Infrastructure Portal Services.

Cloud Management servers comprising:

  • Six DL380 G8 servers will use for CSA, DB, OO, AM, SiS and SA software. All the servers will run Windows 2008/2012 OS except for SA which will be on RedHat 6.x or 7.x
  • One DL380p G8 servers for CSA enables and manages the delivery of application services. It includes user interfaces that allow infrastructure design, specifying what assets will be available, and service design, in which a service designer can add to and manage service catalogs.
  • One DL380p G8 servers for DB server will be running MSSQL server to use as a centralize database server for CSA, OO and others.
  • One DL380p G8 for OO to orchestrate communication between integrated products such as CSA to SA, AM and SiS and managed devices.
  • One DL380p G8 for SA server. SA deploys operating systems and policies to managed devices. It provides lifecycle server management and automated application deployment, and automates tasks such as provisioning, patching, configuration management, and compliance management.
  • One DL 380p G8 for SiS to provides agentless monitoring of infrastructure platforms and the key performance indicators (KPIs) of applications. KPIs include CPU, disk, memory usage, etc.
  • One DL380p G8 for AM is use to track the VMs asset and use for chargeback.
  • All these servers has been configured to be booted from SAN LUNs. The setup is to facilitate for disaster recovery solution using 3PAR remote copy and MOE Matrix Recovery Manager.
  • Sixteen (16) BL460 Gen8 servers running VMware ESXi 5.5 virtualization software. The hypervisor configured as cluster group to provide redundancy between ESX hosts. Redundancy may also be established from allocation of blade servers to different C7000 chassis.
  • One DL380p G8 running on VMware vCenter 5.5 server. The vCenter server has been serving as hypervisor management console and integrated with CSM HPIO component.
  • One DL380p G8 server running on Windows 2012 server has been deployed as Microsoft Active Directory server with the additional services such as DNS and DHCP. This is also being used for centralize authentication for servers and portal users and it also serves the 5 AD FSMO roles.

Data Center Utilisation

Windows VM (Memory Utilization)

Windows VM (Memory Utilization)

Windows VM (CPU Utilization)

Windows VM (CPU Utilization)

Linux VM (Memory Utilization)

Linux VM (Memory Utilization)

Linux VM (CPU Utilization)

Linux VM (CPU Utilization)