You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
140 lines
4.2 KiB
140 lines
4.2 KiB
--- |
|
title: Tensorflow中的GPU配置及内存分配 |
|
date: 2021-07-23 09:01:59 |
|
updated: 2021-11-06 13:54:49 |
|
tags: |
|
categories: |
|
keywords: |
|
description: |
|
top_img: |
|
comments: |
|
cover: |
|
toc: |
|
toc_number: |
|
toc_style_simple: |
|
copyright: ruoneo |
|
copyright_author: |
|
copyright_author_href: |
|
copyright_url: |
|
copyright_info: |
|
mathjax: |
|
katex: |
|
highlight_shrink: |
|
aside: |
|
--- |
|
|
|
## Nvidia-smi |
|
|
|
nvidia-sim简称NVSMI,提供监控GPU使用情况和更改GPU状态的功能。这个工具是N卡驱动附带的。smi(System management interface)。 |
|
|
|
更详细的介绍:<https://blog.csdn.net/C_chuxin/article/details/82993350> |
|
|
|
CMD中输入: |
|
|
|
```text |
|
nvidia-smi |
|
``` |
|
|
|
![图 2](../images/5bc51343e21394d4f82ce4a63916e4af7df2acf0be93a444928e300bd564a089.png) |
|
|
|
- GPU:本机中的GPU编号(有多块显卡的时候,从0开始编号)图上GPU的编号是:0 |
|
|
|
- Fan:风扇转速(0%-100%),N/A表示没有风扇 |
|
|
|
- Name:GPU类型,图上GPU的类型是:Tesla T4 |
|
|
|
- Temp:GPU的温度(GPU温度过高会导致GPU的频率下降) |
|
|
|
- Perf:GPU的性能状态,从P0(最大性能)到P12(最小性能),图上是:P0 |
|
|
|
- Persistence-M:持续模式的状态,持续模式虽然耗能大,但是在新的GPU应用启动时花费的时间更少,图上显示的是:off |
|
|
|
- Pwr:Usager/Cap:能耗表示,Usage:用了多少,Cap总共多少 |
|
|
|
- Bus-Id:GPU总线相关显示,domain:bus:device.function |
|
|
|
- Disp.A:Display Active ,表示GPU的显示是否初始化 |
|
|
|
- Memory-Usage:显存使用率 |
|
|
|
- Volatile GPU-Util:GPU使用率 |
|
|
|
- Uncorr. ECC:关于ECC的东西,是否开启错误检查和纠正技术,0/disabled,1/enabled |
|
|
|
- Compute M:计算模式,0/DEFAULT,1/EXCLUSIVE_PROCESS,2/PROHIBITED |
|
|
|
- Processes:显示每个进程占用的显存使用率、进程号、占用的哪个GPU |
|
|
|
隔几秒刷新一下显存状态:nvidia-smi -l 秒数 |
|
|
|
隔两秒刷新一下GPU的状态:nvidia-smi -l 2 |
|
|
|
## tensorflow的显卡使用方式 |
|
|
|
### 1、直接使用 |
|
|
|
这种方式会把当前机器上所有的显卡的剩余显存基本都占用,注意是机器上所有显卡的剩余显存。 |
|
|
|
```python |
|
with tf.compat.v1.Session() as sess: |
|
# 输入图片为256x256,2个分类 |
|
shape, classes = (224, 224, 3), 20 |
|
# 调用keras的ResNet50模型 |
|
model = keras.applications.resnet50.ResNet50(input_shape = shape, weights=None, classes=classes) |
|
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"]) |
|
|
|
# 训练模型 categorical_crossentropy sparse_categorical_crossentropy |
|
# training = model.fit(train_x, train_y, epochs=50, batch_size=10) |
|
model.fit(train_x,train_y,validation_data=(test_x, test_y), epochs=20, batch_size=6,verbose=2) |
|
# # 把训练好的模型保存到文件 |
|
model.save('resnet_model_dog_n_face.h5') |
|
``` |
|
|
|
### 2、分配比例使用 |
|
|
|
```python |
|
from tensorflow.compat.v1 import ConfigProto# tf 2.x的写法 |
|
config =ConfigProto() |
|
config.gpu_options.per_process_gpu_memory_fraction=0.6 |
|
with tf.compat.v1.Session(config=config) as sess: |
|
model = keras.applications.resnet50.ResNet50(input_shape = shape, weights=None, classes=classes) |
|
``` |
|
|
|
### 3. 动态申请使用 |
|
|
|
**这种方式是动态申请显存的,只会申请内存,不会释放内存。而且如果别人的程序把剩余显卡全部占了,就会报错。** |
|
|
|
```python |
|
config = tf.compat.v1.ConfigProto() |
|
config.gpu_options.allow_growth = True |
|
session = tf.compat.v1.InteractiveSession(config=config) |
|
with tf.compat.v1.Session(config=config) as sess: |
|
model |
|
``` |
|
|
|
### 4 指定GPU |
|
|
|
- 在有多块GPU的服务器上运行tensorflow的时候,如果使用python编程,则可指定GPU,代码如下: |
|
|
|
```python |
|
import os |
|
os.environ["CUDA_VISIBLE_DEVICES"] = "2" |
|
|
|
os.environ["CUDA_VISIBLE_DEVICES"]="-1" CPU模式 |
|
|
|
|
|
CUDA_VISIBLE_DEVICES=1 Only device 1 will be seen |
|
CUDA_VISIBLE_DEVICES=0,1 Devices 0 and 1 will be visible |
|
CUDA_VISIBLE_DEVICES="0,1" Same as above, quotation marks are optional |
|
CUDA_VISIBLE_DEVICES=0,2,3 Devices 0, 2, 3 will be visible; device 1 is masked |
|
``` |
|
|
|
- 另一种方案-给指定的Session分配单独的GPU |
|
|
|
```text |
|
用 tf.device('/gpu:1') 指定Session在第二块GPU上运行。 |
|
tensorflow中不同的GPU使用`/gpu:0`和`/gpu:1`区分, |
|
CPU不区分设备号,表示为`/cpu:0`。 |
|
``` |
|
|
|
> 未完待续...
|
|
|