Add binary format support with HNSW method in Faiss Engine

[heemin32]

Description

What this PR has

Add binary format support with HNSW method in Faiss engine

What this PR does not have

1. Search parameter support which is required for dynamic ef_search value in query, efficient filtering, and multi vector support. This will be handled by a separate PR.
2. Script scoring

Changes

1. Introduction of UNDEFINED space type. This is needed because the space type is parsed before we know about data type and we are going to have a different default space type per data_type; hamming for binary and l2 for all other. Therefore, I had to assign UNDEFINED space type when user does not provide any space type first. Then later inside mapper, I set proper default space type based on data type if the space type is UNDEFINED
2. Introduction of isBinary parameter in JNIService.free method. Because faiss::Index and faiss::BinaryIndex does not share common parent class, we cannot determine if it is faiss::Index or faiss::BinaryIndex using cast. Therefore, we need to know if it is binary index or not. The reason why we not having a separate freeBinaryIndex method is because it is single delete method in cpp side whereas, createIndex has separate method for binary index like createBinaryIndex.
3. Adding vector data type as parameter in IndexEntryContext. This data will be used when loading an index into memory to determine if it is binary index or not. For creation and query, we use vector data type value in fieldInfo to determine if it is binary index or not.
4. New VectorTransfer, KNNIterator, and byteQueryVector variable inside Query was needed as binary vector will have byte[] type when non binary vector will have float[] type. Refactoring will be needed to enhance maintainability which is captured in [FEATURE]Refactoring of code for better maintainability #1810

Testing

1. Create index

PUT /my-knn-index-1
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "knn_vector",
        "dimension": 8,
        "data_type": "binary",
        "method": {
          "name": "hnsw",
          "engine": "faiss"
        }
      }
    }
  }
}

5. Ingest documents

POST /_bulk?refresh=true
{ "index": { "_index": "my-knn-index-1", "_id": "1" } }
{"my_vector" : [9]}
{ "index": { "_index": "my-knn-index-1", "_id": "2" } }
{"my_vector" : [14]}
{ "index": { "_index": "my-knn-index-1", "_id": "3" } }
{"my_vector" : [-100]}

6. Query

POST /my-knn-index-1/_search
{
  "query": {
    "knn": {
      "my_vector": {
        "vector": [13],
        "k": 2
      }
    }
  }
}

Response

{
  "took": 81,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.5,
    "hits": [
      {
        "_index": "my-knn-index-1",
        "_id": "1",
        "_score": 0.5,
        "_source": {
          "my_vector": [
            9
          ]
        }
      },
      {
        "_index": "my-knn-index-1",
        "_id": "2",
        "_score": 0.33333334,
        "_source": {
          "my_vector": [
            14
          ]
        }
      }
    ]
  }
}

Issues Resolved

#1767

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[navneet1v]

This should be behind the VectorSerializer interfaces otherwise there is no common way to read the binary vectors/float vectors.

[navneet1v]

better java doc needed

[heemin32]

This is child class of VectorTransfer. Wouldn't the java doc in VectorTransfer be enough?

[navneet1v]

the documentation is incorrect, we are not transferring 1 vector here. We are transferring more than 1 vector.

[heemin32]

I meant to say this method is called for a single vector.
Yes. We buffer vectors and transfer more than 1 vector later internally but it is hidden from caller.

[navneet1v]

how we are ensuring that close method is always called?

[heemin32]

I thought it is caller's responsibility to call the close. Any idea to enforce it?

[heemin32]

Let me make it as closable class.

[heemin32]

I think I got your point. I think it should be refactored so that there is no dependency between methods.

[heemin32]

@navneet1v I captured the all comments related with code refactoring in #1810. Please update in the issue if I missed any. I will follow up after completing this feature implementation.
The reason that I want to push this code as quickly as possible without doing all the suggested refactoring is to give enough time for reviewing IVF implementation which has developed on top of this PR so that the complete binary format support can be released in 2.16.

[jmazanec15]

I think for a big PR like this, it would be helpful to give a high level summary of what you had to change in the implementation and why. For instance, explaining the different components changed, like why we are changing the free api to take a boolean. This will make review a lot easier and there will be less back and forth.

[navneet1v]

@navneet1v I captured the all comments related with code refactoring in #1810. Please update in the issue if I missed any. I will follow up after completing this feature implementation. The reason that I want to push this code as quickly as possible without doing all the suggested refactoring is to give enough time for reviewing IVF implementation which has developed on top of this PR so that the complete binary format support can be released in 2.16.

I think there are multiple refactoring suggested in the code and my worry is there are new features that we are working on including Disk based vector search and reducing memory foot print during index creation they all touch the same code path. Doing refactoring later will impact those features.

[heemin32]

@navneet1v I captured the all comments related with code refactoring in #1810. Please update in the issue if I missed any. I will follow up after completing this feature implementation. The reason that I want to push this code as quickly as possible without doing all the suggested refactoring is to give enough time for reviewing IVF implementation which has developed on top of this PR so that the complete binary format support can be released in 2.16.

I think there are multiple refactoring suggested in the code and my worry is there are new features that we are working on including Disk based vector search and reducing memory foot print during index creation they all touch the same code path. Doing refactoring later will impact those features.

We can discuss offline. My plan is working on refactoring after 2.16 release. Therefore, the delay won't be much between now and later.

[heemin32]

Ran ./gradlew :spotlessApply to fix build failure.