Poor Man's Rekognition [CCExtractor's home page]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======  Poor Man's Rekognition ======

Developed under Google Summer of Code 2019 by [[https://www.linkedin.com/in/sziraqui|Sarfaraz Iraqui]]  

Mentor: Johannes Von Lochter

===== Introduction =====

Poor Man's Rekognition (PMR) aims to provide a free and open source alternative to Amazon Rekognition. 

Amazon Rekognition is a fairly complex system. 

During GSoC, the project focused on face detection and recognition only. 

There are many libraries especially in Python that provide these functionality but they require a lot of expensive hardware for practical use. 

This project is aimed to make everything run on CPU but still provide reasonable performance for a REST API.


===== Project Structure =====

The project is divided into three sub-projects:

- **Nodoface**: A Nodejs C++ addon (binding) to C++ libraries that helped attain high performance on CPU for ML algorithms.

- **PMR-Core**: Nodejs library (with TypeScript support) that combines Nodoface and ML libraries that are not available in C++.
 For eg. there is no ArcFace model on C++ or Nodejs. This provides completely everything that is needed to build Amazon Rekognition like API.

- **PMR-Server**: The REST API for PMR-Core. This provides endpoints similar to Amazon Rekognition. Any API call generates response which is more or less identical to Rekognition.

===== Project links =====

1. Proposal - https://github.com/sziraqui/pmr-gsoc-tracker/blob/master/Proposal-PMR-CCExtractor.pdf

2. Nodoface repository - https://github.com/sziraqui/nodoface

3. PMR-Core repository - https://github.com/sziraqui/pmr-core   

4. PMR-Server repository - https://github.com/sziraqui/pmr-server

===== Installation =====

Only Nodoface requires a tough installation because it depends on some C++ libraries.

PMR-Core and Server require just one command for setup.

== Part 1: Install dependencies ==
Nodoface depends on-

- OpenFace >= 2.1.0

- OpenCV >= 3.4.0

- dlib >= 19.13   

  All above libs must be compiled as a **shared library**
  
- node-addon-api >= 1.6.3

Step 1: [[https://docs.opencv.org/3.4.0/d7/d9f/tutorial_linux_install.html|Install]] OpenCV 3.4.x.    

If opencv is already installed, ensure it is recognised by pkg-config by executing:

'''$ pkg-config --modversion opencv'''

Step 2: [[http://dlib.net/compile.html|Install]] dlib 19.13 or greater as a shared library (default is static). 

Check [[https://stackoverflow.com/a/33997825/6699069|this]] answer by DavisKing for compiling it as shared library.  
   
Ensure it is recognised by pkg-config:

'''$ pkg-config --modversion dlib'''

Step 3: Install OpenFace. The original repo can only compile as static library. 

So use my OpenFace fork instead. I have modified CMakelists.txt to compile OpenFace as shared lib.


    $ git clone https://github.com/sziraqui/OpenFace.git && cd OpenFace

    $ git checkout dynamic-compile

    $ ./download_models.sh

    $ mkdir build && cd build

    $ cmake -D CMAKE_BUILD_TYPE=RELEASE CMAKE_CXX_FLAGS="-std=c++11" -D CMAKE_EXE_LINKER_FLAGS="-std=c++11" 

    $ make -j2

    $ sudo make install

After this, Nodoface can be installed in any Node project with `npm install nodoface`

== Part 2: Setup server ==

The server depends on PMR-Core and knn-classifier.

Step 1: Create a project folder. Let's name it "pmr".


'''mkdir pmr'''

Step 2: Clone dependent repositories in project folder.


    cd pmr
    git clone https://github.com/sziraqui/nodoface
    git clone https://github.com/sziraqui/pmr-core
    git clone https://github.com/sziraqui/pmr-server
    git clone https://github.com/tensorflow/tfjs-models/tree/master/knn-classifier


Step 3: Install packages in PMR-Core

    cd pmr-core
    npm install
    cd ../


Step 4: Setup KNN-classifier

    cp -r tfjs-models/knn-classifier knn-classifier
    cd knn-classifier
    npm install
    cd ../

Step 5: Setup server

     cd pmr-server
     npm install
     cd ../

== Part 3: Setup database ==

The project uses PostgreSQL database. Face embeddings and PMR-Server data is stored in a database. Follow below instructions to setup a local database for the PMR-Server.    

Step 1: Create DB schema    

Install PostgreSQL and create a schema as per this [[https://cdn-images-1.medium.com/max/800/1*n9cL9QNsDTspoFLWJlNHhw.png|ER diagram]]    

Step 2: 

Download LFW dataset from http://vis-www.cs.umass.edu/lfw/lfw-deepfunneled.tgz  
  
Extract it somewhere (say pmr/lfw-deepfunneled)    

Step 3: Populate face embedding data in DB

    cd pmr-server
    ./bin/add_faces_to_db.ts pmr/lfw-deepfunneled LFW DEEPFUNNELED

For more info on script usage, pass it '''--help''' flag with no arguments. 
  
Any other dataset can be used as long as the directory structure is same as that if LFW.

== Start server ==

Run start-server script to start the server on localhost

    ./bin/start-server

===== Usage docs =====

== Nodoface: ==

Nodoface can be used without PMR-Core and Server. See nodoface/examples directory and nodoface/tests directory to learn the usage. 

== PMR-Core: ==

PMR-Core can be used without the Server but depends on Nodoface. Look for files ending with *test.ts in pmr-core/src to learn usage. Scripts provided in pmr-server/bin directory are also good place to explore.

== PMR-Server: ==

See the endpoints in pmr-server/src/api/vX/*.ts. 

Request/Response syntax can be understood by going through pmr-server/src/amzn-dtypes and pmr-server/src/pmr-dtypes.

At the time of writing, there is no UI for this server. I use Postman to test the endpoints.

=== Making requests to server ===

1. POST ''/api/v0/detect-faces''  
   Request body:
   
    {
        "Image": {
            "Bytes": "base64-string-of-image"
        }
    }
    
    
"Bytes" is the base64 string representation of input image. This is same as DataURL after removing the first intial characters "data:image/jpeg;base64," (for jpeg image). I use https://www.base64-image.de to get base64 string while testing.


2. POST ''/api/v0/recognize-celebrities''   
    Request body:
    Same as that for `detect-faces` endpoint above.

3. POST ''/api/v0/celebrity-in-video''
   Request body:
    
    {
        "Video": {
            "Url": "direct-downloadable-video-url"
        }
    }
    
=== Handling response from server ===

The server follows a "Dispatch and Poll" method. Everytime a request is received, the server dispatches a "Job" and sends response immediately with the Job status.
Sample response:


     {
        "JobId": "string",
        "JobStatus": "PENDING",
        "ResultUrl": "url"
      }


One can poll on the ResultUrl until JobStatus becomes "COMPLETED". When that happens, the response will include output such as BoundingBox, Celebrity name, etc depending on the request. Also the RequestUrl of a COMPLETED job will point to output file with visualisations drawn (bonding boxes, labels, etc on image/video). Read this [blog](https://link.medium.com/qlbGEnFdyZ)to learn more.

===== Useful links =====

1. My GSoC notes - https://github.com/sziraqui/pmr-gsoc-tracker/tree/master/notes

2. Blog (GSoC final evaluation) - https://link.medium.com/qlbGEnFdyZ

===== Contact details =====
Slack - @sziraqui    

Email/Hangouts - sarfarazghlm[at]gmail[dot]com

LinkedIn - https://www.linkedin.com/in/sziraqui