[Adapt] Running GPU Programs on Blackhole
Luo Kangqi
luo.kangqi at qq.com
Fri Sep 1 11:57:26 CST 2017
Hi Adapters,
The GPU memory on Blackhole is relatively small (11GB in total).
If you are running a Tensorflow program, the code will consume ALL the GPU memory by default, which blocks the use from other Adapters.
In order to make the best use of our GPU, I suggest the Adapters to do the following steps:
1. When you define a Session in your Tensorflow program, try to use these codes:
gpu_options = tf.GPUOptions(allow_growth=True, per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options, intra_op_parallelism_threads=8))
# allow_growth=True: the program attempts to allocate only as much GPU memory based on runtime allocations
# per_process_gpu_memory_fraction: determines the fraction of the overall amount of memory that each visible GPU should be allocated.
If you believe that your program has to allocate lots of memory, just remove this parameter.
# intra_op_parallelism_threads: controls the number of CPU cores in the parallel computation (effective only in CPU mode)
# For detail information, please check it out: https://www.tensorflow.org/tutorials/using_gpu
2. Always check the memory consumption of your program via "nvidia-smi" command.
The figure below is the snapshot of the GPU status by executing "nvidia-smi".
We can track the status of our own program, if the memory consumption is unexpected (for example, PID 25982 consumes 9GB memory), then we may need to check the code (something wrong, or without proper configuration).
Fri Sep 1 11:46:52 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22 Driver Version: 381.22 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 0000:03:00.0 Off | N/A |
| 48% 67C P2 91W / 280W | 10470MiB / 11171MiB | 39% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 7775 C python 281MiB |
| 0 17440 C python 287MiB |
| 0 20360 C python 249MiB |
| 0 25982 C python 9641MiB |
+-----------------------------------------------------------------------------+
Thanks your reading this, hope all the programs working well in our machine!
Best,
Kangqi
------------------
Kangqi Luo, PhD Candidate
ADAPT Lab, Department of Computer Science
SEIEE 3-341, Shanghai Jiao Tong University,
No. 800 Dongchuan Road, Shanghai, China
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20170901/06fde70e/attachment.html>
More information about the Adapt
mailing list