如何提升Tensorflow服务性能( 二 ) _Tensorflow

#!/usr/bin/env pythonfrom __future__ import print_functionimport argparseimport numpy as npimport timett = time.time()import cv2import tensorflow as tffrom grpc.beta import implementationsfrom tensorflow_serving.apis import predict_pb2from tensorflow_serving.apis import prediction_service_pb2parser = argparse.ArgumentParser(description='incetion grpc client flags.')parser.add_argument('--host', default='0.0.0.0', help='inception serving host')parser.add_argument('--port', default='9000', help='inception serving port')parser.add_argument('--image', default='', help='path to JPEG image file')FLAGS = parser.parse_args()def main():# create prediction service client stub channel = implementations.insecure_channel(FLAGS.host, int(FLAGS.port)) stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)# create request request = predict_pb2.PredictRequest() request.model_spec.name = 'resnet' request.model_spec.signature_name = 'serving_default'# read image into numpy array img = cv2.imread(FLAGS.image).astype(np.float32)# convert to tensor proto and make request # shape is in NHWC (num_samples x height x width x channels) format tensor = tf.contrib.util.make_tensor_proto(img, shape=[1]+list(img.shape)) request.inputs['input'].CopyFrom(tensor) resp = stub.Predict(request, 30.0)print('total time: {}s'.format(time.time() - tt)) if __name__ == '__main__': main()

文章插图

使用输入JPEG图像运行客户机的输出如下所示:
python tf_serving_client.py --image=images/pupper.jpg total time: 2.56152906418s
输出张量的预测结果为整数值和特征概率

文章插图

对于单个请求，这种预测延迟是不可接受的。然而，这并非完全出乎意料;服务于二进制文件的默认Tensorflow目标是针对最广泛的硬件范围，以涵盖大多数用例。您可能已经从标准的Tensorflow服务容器日志中注意到:
I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA这表示Tensorflow服务二进制文件在不兼容的CPU平台上运行，并未进行优化。
构建CPU优化服务二进制
根据Tensorflow文档，建议从源代码编译Tensorflow ，并在运行二进制文件的主机平台的CPU上使用所有可用的优化。Tensorflow构建选项公开了一些标志，以支持构建特定于平台的CPU指令集:

文章插图

在本例中，我们将使用1.13:
USER=$1 TAG=$2 TF_SERVING_VERSION_GIT_BRANCH="r1.13" git clone --branch="$TF_SERVING_VERSION_GIT_BRANCH" https://github.com/tensorflow/serving

文章插图

Tensorflow服务开发镜像使用Bazel作为构建工具。处理器特定CPU指令集的构建目标可以指定如下:
TF_SERVING_BUILD_OPTIONS="--copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-msse4.1 --copt=-msse4.2"如果memory是约束，则可以使用--local_resources=2048,.5,1.0 flag 限制内存密集型构建过程的消耗。
以开发镜像为基础构建服务镜像：
#!/bin/bashUSER=$1TAG=$2TF_SERVING_VERSION_GIT_BRANCH="r1.13"git clone --branch="${TF_SERVING_VERSION_GIT_BRANCH}" https://github.com/tensorflow/servingTF_SERVING_BUILD_OPTIONS="--copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-msse4.1 --copt=-msse4.2"cd serving &&docker build --pull -t $USER/tensorflow-serving-devel:$TAG--build-arg TF_SERVING_VERSION_GIT_BRANCH="${TF_SERVING_VERSION_GIT_BRANCH}"--build-arg TF_SERVING_BUILD_OPTIONS="${TF_SERVING_BUILD_OPTIONS}"-f tensorflow_serving/tools/docker/Dockerfile.devel .cd serving &&docker build -t $USER/tensorflow-serving:$TAG--build-arg TF_SERVING_BUILD_IMAGE=$USER/tensorflow-serving-devel:$TAG-f tensorflow_serving/tools/docker/Dockerfile .