Faiss - 部署安装和演示

1. 概述

Faiss is a library for efficient similarity search and clustering of dense vectors.

It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.
It also contains supporting code for evaluation and parameter tuning.

Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU.
It is developed by Facebook AI Research.

• github repository
  https://github.com/facebookresearch/faiss

2. Setup

2.1. 通过conda安装faiss，简单方便

• dockerfile已经ok，可以尝试下

ARG IMAGE
FROM ${IMAGE}

ARG FAISS_CPU_OR_GPU
ARG FAISS_VERSION

RUN apt-get update && \
    apt-get install -y curl bzip2  && \
    curl https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh > /tmp/conda.sh && \
    bash /tmp/conda.sh -b -p /opt/conda && \
    /opt/conda/bin/conda update -n base conda && \
    /opt/conda/bin/conda install -y python=3.6.9 && \
    /opt/conda/bin/conda install -y -c pytorch faiss-${FAISS_CPU_OR_GPU}=${FAISS_VERSION} && \
    apt-get remove -y --auto-remove curl bzip2 && \
    apt-get clean && \
    rm -fr /tmp/conda.sh

ENV PATH="/opt/conda/bin:${PATH}"

how to build a faiss-docker image

• references
  https://anaconda.org/pytorch/faiss-cpu
  https://anaconda.org/pytorch/faiss-gpu
  https://github.com/plippe/faiss-docker

2.2. 源码安装

• 测试没过，卡在cuda上，有机会再试试

RUN apt-get update -y && apt-get install -y libopenblas-dev python-numpy python-dev swig python-pip curl
ENV BLASLDFLAGS=/usr/lib/libopenblas.so.0 \
    PYTHON=python
RUN cd /opt \
    && git clone https://github.com/facebookresearch/faiss.git \
    && cd faiss  && git checkout v1.3.0 \
    && pip3 install matplotlib==2.2.3 python-config numpy -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com \
    && mv example_makefiles/makefile.inc.Linux ./makefile.inc \
    && ./configure --with-python=python3 \
    && make -j $(nproc) \
    && make install \
    && make -C gpu -j $(nproc) \
    && make -C gpu/test \
    && make -C python gpu \
    && make -C python build \
    && make -C python install

3. How to use

3.1. build a index

dimensions = 128
INDEX_KEY = "IDMap,Flat"
index = faiss.index_factory(dimensions, INDEX_KEY)

3.2. is GPU used

if USE_GPU:
    res = faiss.StandardGpuResources()
    index = faiss.index_cpu_to_gpu(res, 0, index)

3.3. add features to index with ids

index.add_with_ids(features, ids)

3.4. search

scores, neighbors = index.search(siftfeature, k=topN)

3.5. save index and reload

faiss.write_index(index, "large.index")
index1 = faiss.read_index("large.index")

4. More documents

• Faiss Practice
  https://waltyou.github.io/Faiss-In-Project-English/
  https://github.com/plippe/faiss-web-service
• Analysis Report
  https://waltyou.github.io/Faiss-Indexs/