Titan V Deep Learning Benchmarks with TensorFlow

In this post, Lambda Labs benchmarks the Titan V's Deep Learning / Machine Learning performance and compares it to other commonly used GPUs. We use the Titan V to train ResNet-50, ResNet-152, Inception v3, Inception v4, VGG-16, AlexNet, and SSD300. We measure the # of images processed per second while training each network.

A few notes:

• We use TensorFlow 1.12 / CUDA 10.0.130 / cuDNN 7.4.1
• Single-GPU benchmarks were run on the Lambda's deep learning workstation
• Multi-GPU benchmarks were run on the Lambda's PCIe GPU Server
• V100 Benchmarks were run on Lambda's SXM3 Tesla V100 Server
• Tensor Cores were utilized on all GPUs that have them

Titan V - FP32 TensorFlow Performance (1 GPU)

For FP32 training of neural networks, the NVIDIA Titan V is...

• 42% faster than RTX 2080
• 41% faster than GTX 1080 Ti
• 26% faster than Titan XP
• 4% faster than RTX 2080 Ti
• 90% as fast as Titan RTX
• 75% as fast as Tesla V100 (32 GB)

as measured by the # images processed per second during training.

Titan V - FP16 TensorFlow Performance (1 GPU)

For FP16 training of neural networks, the NVIDIA Titan V is..

• 111% faster than GTX 1080 Ti
• 94% faster than Titan XP
• 70% faster than RTX 2080
• 23% faster than RTX 2080 Ti
• 87% as fast as Titan RTX
• 68% as fast as Tesla V100 (32 GB)

as measured by the # images processed per second during training.

FP32 Multi-GPU Scaling Performance (1, 2, 4, 8 GPUs)

For each GPU type (Titan V, RTX 2080 Ti, RTX 2080, etc.) we measured performance while training with 1, 2, 4, and 8 GPUs on each neural networks and then averaged the results. The chart below provides guidance as to how each GPU scales during multi-GPU training of neural networks in FP32. The chart can be read as follows:

• Using eight Titan Vs will be 5.18x faster than using a single Titan V
• Using eight Tesla V100s will be 9.68x faster than using a single Titan V
• Using eight Tesla V100s is 9.68 / 5.18 = 1.87x faster than using eight Titan Vs

Titan V - FP16 vs. FP32

FP16 can reduce training times and enable larger batch sizes/models without significantly impacting model accuracy. Compared with FP32, FP16 training on the Titan V is...

• 80% faster on ResNet-50
• 69% faster on ResNet-152
• 70% faster on Inception v3
• 51% faster on Inception v4
• 96% faster on VGG-16
• 78% faster on AlexNet
• 57% faster on SSD300

as measured by the # of images processed per second during training. This gives an average speed-up of +71.6%.

Caveat emptor: If you're new to machine learning or simply testing code, we recommend using FP32. Lowering precision to FP16 may interfere with convergence.

GPU Prices

• Titan V: $2,999.00
• RTX 2080 Ti: $1,199.00
• RTX 2080: $799.00
• Titan RTX: $2,499.00
• Tesla V100 (32 GB): ~$8,200.00
• GTX 1080 Ti: $699.00
• Titan Xp: $1,200.00

Methods

• For each model we ran 10 training experiments and measured # of images processed per second; we then averaged the results of the 10 experiments.
• For each GPU / neural network combination, we used the largest batch size that fit into memory. For example, on ResNet-50, the V100 used a batch size of 192; the RTX 2080 Ti use a batch size of 64.
• We used synthetic data, as opposed to real data, to minimize non-GPU related bottlenecks
• Multi-GPU training was performed using model-level parallelism

Hardware

• Single-GPU training: Lambda Quad - Deep Learning GPU Workstation. CPU: i9-7920X / RAM: 64 GB DDR4 2400 MHz.
• Multi-GPU training: Lambda Blade - Deep Learning GPU Server. CPU: Xeon E5-2650 v4 / RAM: 128 GB DDR4 2400 MHz ECC
• V100 Benchmarks: Lambda Hyperplane - Tesla V100 Server. CPU: Xeon Gold 6148 / RAM: 256 GB DDR4 2400 MHz ECC

Software

• Ubuntu 18.04 (Bionic)
• TensorFlow 1.12
• CUDA 10.0.130
• cuDNN 7.4.1

Run Our Benchmarks On Your Own Machine

Our benchmarking code is on github. We'd love it if you shared the results with us by emailing s@lambdalabs.com or tweeting @LambdaAPI.

Step #1: Clone Benchmark Repository

git clone https://github.com/lambdal/lambda-tensorflow-benchmark.git --recursive

Step #2: Run Benchmark

• Input a proper gpu_index (default 0) and num_iterations (default 10)

cd lambda-tensorflow-benchmark
./benchmark.sh gpu_index num_iterations

Step #3: Report Results

• Check the repo directory for folder <cpu>-<gpu>.logs (generated by benchmark.sh)
• Use the same num_iterations in benchmarking and reporting.

./report.sh <cpu>-<gpu>.logs num_iterations

Raw Benchmark Data

FP32: # Images Processed Per Sec During TensorFlow Training (1 GPU)

FP16: # Images Processed Per Sec During TensorFlow Training (1 GPU)