深度學習框架Caffe在Mac上的安裝和測試

2020-06-16 17:38:54

深度學習框架介紹

先概括一下深度學習的幾大流行的框架：Pylearn2, Theano, Caffe, Torch, Cuda-covnet，Deeplarning4j等。

Theano是一個Python庫，也是一個強大的數學表示式編譯器。Pylearn2是在Theano基礎上建立的機器學習庫。使用者可以用數學表示式寫Pylearn2的外掛(新的model, algorithm等), Theano將這些表示式進行優化和穩定化，然後進行編譯。
Caffe是由Berkely Vision and Learning Center的賈楊清博士（畢業後在谷歌工作）主導開發的基於ConvNets和C++的深度學習庫。Caffe允許網路模型和優化方法都定義在組態檔中而不需要寫程式碼，可以很方便地在CPU和GPU之間切換。
Torch更偏向企業級應用，是用Lua寫的，Facebook AI實驗室和Google DeepMind團隊等都使用Torch。可以為機器學習演算法提供類似於Matlab的環境。Lua可以輕易地與C結合，任何C或者C++庫都可以成為Lua庫。OverFeat是用Torch7在ImageNet上訓練得到的特徵提取工具。
Cuda-convnet或者CuDNN是NVIDIA提供的基於GPU加速的深度學習工具，對主流的軟體包括Caffe,Torch和Theano都提供支援。
Deeplarning4j面向商業應用，是基於Java的機器學習框架。更多介紹可閱讀各自的網站或者閱讀這篇文章。

Caffe的安裝

Caffe的網站上提供了安裝說明。由於其依賴的庫比較多，通常安裝過程會出現許多問題，在不同的機器和作業系統上可能遇到不同的問題。安裝時可以根據網站上提供的說明步驟進行，遇到有問題時用Google搜尋一下基本都能找到。本文記錄了筆者在Mac上安裝遇到的問題和解決辦法。系統版本：OS X 10.9.5。

1，安裝Caffe的依賴庫

1.1 安裝CUDA。推薦7.0以上版本，6.*版本也可以。我安裝的是最新版CUDA 7.5。

1.2 安裝BLAS。這裡我使用了OpenBLAS。推薦使用brew安裝：brew install openblas

1.3 安裝Boost。

通過brew install boost預設安裝版本為1.60。但建議使用1.59。因為1.60編譯後可能會出現問題。

$ brew search boost
boost                                    homebrew/versions/boost-python159 ?    
boost-bcp                                homebrew/versions/boost149             
boost-build                              homebrew/versions/boost150             
boost-python                             homebrew/versions/boost155             
homebrew/science/boost-compute           homebrew/versions/boost159 ?           
Caskroom/cask/iboostup                   Caskroom/cask/turbo-boost-switcher     
Caskroom/cask/pivotalbooster
$ brew install –build-from-source homebrew/versions/boost159

安裝好後可以後在/usr/local/opt/boost159下看到該庫。Caffe中把某些依賴庫所在的資料夾名字限定為boost，可以將/usr/local/opt/boost159複製貼上產生備份，將備份改名為/usr/local/opt/boost。

1.4 安裝CuDNN。下載cuDNN v5.0版本。解壓後將include和bin資料夾中的內容分別複製到/usr/lcoal 下面的/include和/bin中。

1.5 使用brew install 分別安裝 protobuf, glog, gflags, hdf5, snappy, leveldb, szip, lmdb等。

如果使用python, protobuf安裝命令為

$ brew install --build-from-source --with-python -vd protobuf<code> </code><code></code>

1.6 （可選）OpenCV, 我使用2.4.6版本。

1.7 （可選）Python 版本：2.7。

需要安裝numpy。推薦使用Anaconda，裡面包含了一個python版本2.7.11並且包含了大多數所需要的庫，包括hdf5、numpy等。Anaconda預設安裝在$(HOME)/anaconda目錄下。

還需要安裝python-boost。與boost類似的方法，推薦1.59版本。

1.8 （可選）Matlab 版本 2015a

2，安裝Caffe

2.1 下載Caffe後在caffe-master資料夾下，以Makefile.config.example為模板根據第一步中的安裝情況，建立組態檔Makefile.config,內容如下：

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#    You should not set this flag if you will be reading LMDBs with any
#    possibility of simultaneous read and write
ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 2.4

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
CUSTOM_CXX := clang++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 
        -gencode arch=compute_20,code=sm_21 
        -gencode arch=compute_30,code=sm_30 
        -gencode arch=compute_35,code=sm_35 
        -gencode arch=compute_50,code=sm_50 
        -gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /usr/local/opt/openblas/include
# BLAS_LIB := /usr/local/opt/openblas/lib

# Homebrew puts openblas in a directory that is not on the standard search path
BLAS_INCLUDE := $(shell brew --prefix openblas)/include
BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
MATLAB_DIR := /Applications/MATLAB_R2015a.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
#PYTHON_INCLUDE := /usr/include/python2.7 
        /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda
PYTHON_INCLUDE := $(ANACONDA_HOME)/include 
          $(ANACONDA_HOME)/include/python2.7 
          $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include 

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m 
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
#PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; #print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
#PYTHON_LIB +=/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/lib
PYTHON_LIB +=$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/lib

# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/Cellar/boost159/1.59.0/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/cuda/lib /usr/local/Cellar/boost159/1.59.0/lib /usr/local/opt/boost-python159/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
INCLUDE_DIRS += $(shell brew --prefix)/include
LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

2.2，命令列進入caffe-master資料夾下，執行：

$ make all

出現問題：

PROTOC src/caffe/proto/caffe.proto
make: protoc: No such file or directory

解決辦法：需要用brew建立protobuf的連結。為此，執行

$ brew link protobuf

如果執行上述命令又出問題，比如brew的許可權問題：permission denied for /usr/local。需要設定一下許可權，更新一下brew, 為此，執行

$ sudo chown -R $USER:admin /usr/local
$ cd /usr/local
$ git reset --hard origin/master
$ brew update

上述問題可以得到解決。

2.3 上一步通過後，執行

$ make test

這一步沒問題。將build_release/lib下的所有檔案複製到/usr/local/lib

$ cp -a .build_release/lib/. /usr/local/lib/

再執行

$ make runtest

報錯：

.build_release/tools/caffe
dyld: Library not loaded: @rpath/libcudart.7.5.dylib
Referenced from: /Developer/caffe/.build_release/tools/caffe
Reason: image not found

為此需要設定一下環境變數DYLD_FALLBACK_LIBRARY_PATH

$ export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/lib:$(HOME)/<span style="color:black;">anaconda/lib</span>

再執行make runtest，一切順利。

2.4 如果使用python,再執行

<pre name="code" class="html">$ make pycaffe
$ make pytest

3，執行mnist的例子。

詳細步驟見：http://caffe.berkeleyvision.org/gathered/examples/mnist.html

3.1，下載mnist資料。在caffe-master目錄下執行

$ ./data/mnist/get_mnist.sh

3.2，建立訓練資料和測試資料，執行

$ ./examples/mnist/create_mnist.sh

出現以下錯誤，說convert_mnist_data.bin找不到：

Creating lmdb...
./examples/mnist/create_mnist.sh: line 16: build/examples/mnist/convert_mnist_data.bin: No such file or directory
./examples/mnist/create_mnist.sh: line 18: build/examples/mnist/convert_mnist_data.bin: No such file or directory
Done.

解決辦法：搜尋convert_mnist_data.bin發現該檔案位於./distribute/bin目錄下，因此在在./examples/mnist/create_mnist.sh檔案中將BUILD的值改為distribute/bin即可。

3.3，訓練和測試，執行：

$ ./examples/mnist/create_mnist.sh

如果出現和上面類似的錯誤，說caffe找不到 (caffe.bin位於./distribute/bin目錄下或者build/tools下)，檢查create_mnist.sh的內容，保證caffe.bin的路徑正確

./distribute/bin/caffe.bin train--solver=examples/mnist/lenet_solver.prototxt

然後就能看到執行結果了。

4, 在python中使用caffe的例子

詳見：http://nbviewer.jupyter.org/github/BVLC/caffe/blob/master/examples/00-classification.ipynb

該例子用Caffe中已經訓練好的模型（基於Alexnet的結構）對影象進行分類。並且可以顯示不同層中訓練得到的特徵。

本文永久更新連結地址：http://www.linuxidc.com/Linux/2016-06/132548.htm