[Adapt] Running GPU Programs on Blackhole

[Luo Kangqi]

Hi Adapters,


The GPU memory on Blackhole is relatively small (11GB in total).
If you are running a Tensorflow program, the code will consume ALL the GPU memory by default, which blocks the use from other Adapters.
In order to make the best use of our GPU, I suggest the Adapters to do the following steps:


1. When you define a Session in your Tensorflow program, try to use these codes:
    gpu_options = tf.GPUOptions(allow_growth=True, per_process_gpu_memory_fraction=0.333)
    sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options, intra_op_parallelism_threads=8))
    # allow_growth=True: the program attempts to allocate only as much GPU memory based on runtime allocations
    # per_process_gpu_memory_fraction: determines the fraction of the overall amount of memory that each visible GPU should be allocated.
                                                                  If you believe that your program has to allocate lots of memory, just remove this parameter.
    # intra_op_parallelism_threads: controls the number of CPU cores in the parallel computation (effective only in CPU mode)
    # For detail information, please check it out: https://www.tensorflow.org/tutorials/using_gpu

2. Always check the memory consumption of your program via "nvidia-smi" command.
    The figure below is the snapshot of the GPU status by executing "nvidia-smi".
    We can track the status of our own program, if the memory consumption is unexpected (for example, PID 25982 consumes 9GB memory), then we may need to check the code (something wrong, or without proper configuration).
Fri Sep  1 11:46:52 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 0000:03:00.0     Off |                  N/A |
| 48%   67C    P2    91W / 280W |  10470MiB / 11171MiB |     39%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      7775    C   python                                         281MiB |
|    0     17440    C   python                                         287MiB |
|    0     20360    C   python                                         249MiB |
|    0     25982    C   python                                        9641MiB |
+-----------------------------------------------------------------------------+

Thanks your reading this, hope all the programs working well in our machine!


Best,
Kangqi
------------------
Kangqi Luo, PhD Candidate
ADAPT Lab, Department of Computer Science
SEIEE 3-341, Shanghai Jiao Tong University,
No. 800 Dongchuan Road, Shanghai, China
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cs.sjtu.edu.cn/pipermail/adapt/attachments/20170901/06fde70e/attachment.html>