Nsryjdtyk

Lately, I have been working with Tensorflow's Object Detection API, I have trained the ssd_mobilenet_v2 using my own data and the model works as expected.

I wanted to improve the performance by using multiple CPU cores for certain ops. Currently, Tensorflow utilizes the workload by using a small fraction of each core available on my system (I do not know how this utilization happens in the backend).

I tried adding the parameters device_count={ "CPU": n_cpus }, inter_op_parallelism_threads=n_inter, intra_op_parallelism_threads=n_intra, in the tf.Sessions() method. My aim was to specify the number of cores to be used and the number of threads to be created in order to have a speedup, but this had no impact on the performance.

After that, I also tried to pin certain processes to certain cores by using the following method: with tf.device('/cpu:0'):.

Using with tf.device('/cpu:0'): with trivial operations such as tf.matmul proved to speedup the performance indeed. I could also see the CPU performance and utilization, ex. CPU:0 was being used at around 98-99% (this was what I was looking for).

However, when I tried the same approach tf.device() with model inference, the CPU utilization went back to the default setting, where each the workload is shared between all cores (single op being shared between cores).

I want to know if the inference part can run on multiple CPU cores in parallel. My interest is in executing different ops of the inference in parallel on different cores.
So far, I had no success in making this work, maybe my logic is flawed, I would appreciate some guidance :)

Additional info: I am using TensorFlow 1.11.0, CPU version