Poor Man's Rekognition

Developed under Google Summer of Code 2019 by Sarfaraz Iraqui

Mentor: Johannes Von Lochter

Poor Man's Rekognition (PMR) aims to provide a free and open source alternative to Amazon Rekognition.

Amazon Rekognition is a fairly complex system.

During GSoC, the project focused on face detection and recognition only.

There are many libraries especially in Python that provide these functionality but they require a lot of expensive hardware for practical use.

This project is aimed to make everything run on CPU but still provide reasonable performance for a REST API.

The project is divided into three sub-projects:

  • Nodoface: A Nodejs C++ addon (binding) to C++ libraries that helped attain high performance on CPU for ML algorithms.

  • PMR-Core: Nodejs library (with TypeScript support) that combines Nodoface and ML libraries that are not available in C++.

For eg. there is no ArcFace model on C++ or Nodejs. This provides completely everything that is needed to build Amazon Rekognition like API.

  • PMR-Server: The REST API for PMR-Core. This provides endpoints similar to Amazon Rekognition. Any API call generates response which is more or less identical to Rekognition.

  1. Proposal - https://github.com/sziraqui/pmr-gsoc-tracker/blob/master/Proposal-PMR-CCExtractor.pdf

  2. Nodoface repository - https://github.com/sziraqui/nodoface

  3. PMR-Core repository - https://github.com/sziraqui/pmr-core

  4. PMR-Server repository - https://github.com/sziraqui/pmr-server

Only Nodoface requires a tough installation because it depends on some C++ libraries.

PMR-Core and Server require just one command for setup.

Nodoface depends on-

  • OpenFace >= 2.1.0

  • OpenCV >= 3.4.0

  • dlib >= 19.13

All above libs must be compiled as a shared library

  • node-addon-api >= 1.6.3

Step 1: Install OpenCV 3.4.x.

If opencv is already installed, ensure it is recognised by pkg-config by executing:

$ pkg-config --modversion opencv

Step 2: Install dlib 19.13 or greater as a shared library (default is static).

Check this answer by DavisKing for compiling it as shared library.

Ensure it is recognised by pkg-config:

$ pkg-config --modversion dlib

Step 3: Install OpenFace. The original repo can only compile as static library.

So use my OpenFace fork instead. I have modified CMakelists.txt to compile OpenFace as shared lib.


          1
          $ git clone [https://github.com/sziraqui/OpenFace.git](https://github.com/sziraqui/OpenFace.git) && cd OpenFace
        
          2
          $ git checkout dynamic-compile
        
          3
          $ ./download_models.sh
        
          4
          $ mkdir build && cd build
        
          5
          $ cmake -D CMAKE_BUILD_TYPE=RELEASE CMAKE_CXX_FLAGS="-std=c++11" -D CMAKE_EXE_LINKER_FLAGS="-std=c++11"
        
          6
          $ make -j2
        
          7
          $ sudo make install
not set

After this, Nodoface can be installed in any Node project with npm install nodoface

The server depends on PMR-Core and knn-classifier.

Step 1: Create a project folder. Let's name it "pmr".

mkdir pmr

Step 2: Clone dependent repositories in project folder.


          1
          cd pmr
        
          2
          git clone [https://github.com/sziraqui/nodoface](https://github.com/sziraqui/nodoface)
        
          3
          git clone [https://github.com/sziraqui/pmr-core](https://github.com/sziraqui/pmr-core)
        
          4
          git clone [https://github.com/sziraqui/pmr-server](https://github.com/sziraqui/pmr-server)
        
          5
          git clone [https://github.com/tensorflow/tfjs-models/tree/master/knn-classifier](https://github.com/tensorflow/tfjs-models/tree/master/knn-classifier)
not set

Step 3: Install packages in PMR-Core


          1
          cd pmr-core
        
          2
          npm install
        
          3
          cd ../
not set

Step 4: Setup KNN-classifier


          1
          cp -r tfjs-models/knn-classifier knn-classifier
        
          2
          cd knn-classifier
        
          3
          npm install
        
          4
          cd ../
not set

Step 5: Setup server


          1
          cd pmr-server
        
          2
          npm install
        
          3
          cd ../
not set

The project uses PostgreSQL database. Face embeddings and PMR-Server data is stored in a database. Follow below instructions to setup a local database for the PMR-Server.

Step 1: Create DB schema

Install PostgreSQL and create a schema as per this ER diagram

Step 2:

Download LFW dataset from http://vis-www.cs.umass.edu/lfw/lfw-deepfunneled.tgz

Extract it somewhere (say pmr/lfw-deepfunneled)

Step 3: Populate face embedding data in DB


          1
          cd pmr-server
        
          2
          ./bin/add_faces_to_db.ts pmr/lfw-deepfunneled LFW DEEPFUNNELED
not set

For more info on script usage, pass it --help flag with no arguments.

Any other dataset can be used as long as the directory structure is same as that if LFW.

Run start-server script to start the server on localhost


          1
          ./bin/start-server
not set

Nodoface can be used without PMR-Core and Server. See nodoface/examples directory and nodoface/tests directory to learn the usage.

PMR-Core can be used without the Server but depends on Nodoface. Look for files ending with *test.ts in pmr-core/src to learn usage. Scripts provided in pmr-server/bin directory are also good place to explore.

See the endpoints in pmr-server/src/api/vX/*.ts.

Request/Response syntax can be understood by going through pmr-server/src/amzn-dtypes and pmr-server/src/pmr-dtypes.

At the time of writing, there is no UI for this server. I use Postman to test the endpoints.

  1. POST /api/v0/detect-faces

Request body:

{ "Image": { "Bytes": "base64-string-of-image" } }

"Bytes" is the base64 string representation of input image. This is same as DataURL after removing the first intial characters "data:image/jpeg;base64," (for jpeg image). I use https://www.base64-image.de to get base64 string while testing.

  1. POST /api/v0/recognize-celebrities

    Request body: Same as that for detect-faces endpoint above.

  2. POST /api/v0/celebrity-in-video

Request body:

{ "Video": { "Url": "direct-downloadable-video-url" } }

The server follows a "Dispatch and Poll" method. Everytime a request is received, the server dispatches a "Job" and sends response immediately with the Job status. Sample response:


          1
          {
        
          2
          "JobId": "string",
        
          3
          "JobStatus": "PENDING",
        
          4
          "ResultUrl": "url"
        
          5
          }
not set

One can poll on the ResultUrl until JobStatus becomes "COMPLETED". When that happens, the response will include output such as BoundingBox, Celebrity name, etc depending on the request. Also the RequestUrl of a COMPLETED job will point to output file with visualisations drawn (bonding boxes, labels, etc on image/video). Read this blogto learn more.

  1. My GSoC notes - https://github.com/sziraqui/pmr-gsoc-tracker/tree/master/notes

  2. Blog (GSoC final evaluation) - https://link.medium.com/qlbGEnFdyZ

Slack - @sziraqui

Email/Hangouts - sarfarazghlm@gmail.com

LinkedIn - https://www.linkedin.com/in/sziraqui