Expanse System Overview

Expanse Feature Specs
System Component Configuration
AMD EPYC 7742 Compute Nodes
Node count 728
Clock speed 2.25 GHz
Cores/node 128
DRAM/node 256 GB
NVMe/node 1 TB
NVIDIA V100 GPU Nodes
Node count 52
CPU cores/node 40
CPU Type 6248 Xeon
CPU DRAM/node 384 GB
CPU Clock Speed 2.5 GHz
GPUs/node 4
GPU Type V100 SMX2
Memory/GPU 32 GB
NVMe/node 1.6TB

In late 2020 SDSC will launch its newest National Science Foundation (NSF)-funded supercomputer, Expanse. At over twice the performance of Comet, Expanse supports SDSC's theme of 'Computing without Boundaries' with a data-centric architecture and state-of-the art GPUs for incorporating experimental facilities and edge computing.
One of the key innovations of Expanse is its ability to support composable systems, allowing researchers to create a virtual 'tool set' of resources, such as Kubernetes, for a specific project and then re-compose it as needed. Expanse will also feature direct scheduler-integration with the major cloud providers, leveraging high-speed networks to ease data movement to/from the cloud.
While the system is suited for modest-scale jobs as few as tens of cores to several hundred cores, Expanse will also handle high-throughput computing jobs via integration with the Open Science Grid, which can have tens of thousands of single-core jobs, and provides connectivity to commercial clouds via the job queuing system. A low-latency interconnect based on Mellanox High Data Rate (HDR) InfiniBand will support a fabric topology optimized for jobs of one to a few thousand cores that require medium-scale parallelism.
Expanse's standard compute nodes are each powered by two 64-core AMD EPYC 7742 processors and contain 256 GB of DDR4 memory, while each GPU node contains four NVIDIA V100s (32 GB SMX2), connected via NVLINK, and dual 20-core Intel Xeon 6248 CPUs. Expanse also has four 2 TB large memory nodes.
The entire system, to be integrated by Dell, is organized into 13 SDSC Scalable Compute Units (SSCUs), comprising 56 standard nodes and four GPU nodes, and connected with 100 GB/s HDR InfiniBand. Every Expanse node has access to a 12 PB Lustre parallel file system (provided by Aeon Computing) and a 7 PB Ceph Object Store system. The Expanse cluster will be managed using the Bright Computing HPC Cluster management system, and will use the SLURM workload manager for job scheduling.
Like Comet, Expanse will be a key resource within the NSF's Extreme Science and Engineering Discovery Environment (XSEDE), which comprises the most advanced collection of integrated digital resources and services in the world.

Coming Fall 2020

Research allocations supported by XSEDE

Submission Period open June 15 - July 15 for allocations starting October 1, 2020

Research Allocation Information

Submit Allocation Request

News

Training & Events

Architecture

The architecture of Expanse will feature next-generation Intel Central Processing Units (CPUs), NVIDIA Graphics Processing Units (GPUs), a Lustre HPC file system, and a Mellanox InfiniBand network. For more architecture details, please see:

Expanse Computing Without Boundaries [PDF]

Expanse System Specifications

The Expanse project is funded under NSF award number 1928224.