[RAG프로젝트] Vector DB -milvus

이전 qdrant로 vectorDB를 구성했다.

https://pleasestudy-alswldi.tistory.com/340

[RAG프로젝트] Vector DB -qdrant

 

이를 milvus로 구성해보자.

 

 

🐹 초기 setting 및 데이터 삽입

• Milvus Connection 설정
      - `connections.connect`로 Milvus 서버에 연결
      - 컬렉션 생성 시 Milvus 전용 `CollectionSchema` 사용
• 컬렉션 초기화
      - Milvus 컬렉션 생성은 `Collection` 객체를 사용하며, Qdrant와 달리 필드 스키마를 정의해야 한다
   -https://milvus.io/docs/ko/schema.md
•  데이터 삽입
      - qdrant에서는 벡터 삽입 시 자동으로 인덱싱하지만 milvus는 명시해야한다.
          - `field_name="embedding"`: 인덱스를 생성할 필드의 이름을 지정
          - `"embedding"` 필드에 인덱스를 생성
          - `index_params`: 인덱스 생성에 필요한 파라미터를 정의
              - `index_type="IVF_FLAT"`: 벡터를 효율적으로 검색하기 위한 인덱스 설정.
                  - `IVF_FLAT`는 벡터 공간을 여러 클러스터로 분할하여 검색 속도를 향상
              -`metric_type="COSINE"`: 벡터 간 유사도를 측정하기 위해 코사인 유사도를 사용
                  - **`params={"nlist": 128}`: 벡터 공간을 128개의 클러스터로 분할하도록 설정
                      - `nlist` 값이 클수록 검색 정확 /검색 속도 느려짐
      - Milvus는 `insert` 메서드를 사용하며, ID는 `INT64`여야 함.
      - JSON 및 PDF 데이터 모두 Milvus 방식으로 벡터 삽입
• 컬렉션 정보 및 벡터 개수 확인
      - Milvus의 `Collection.num_entities` 메서드로 벡터 개수를 확인

스키마 관리 | Milvus 문서화

 

 

 

 

🐹 issue

RPC error: [has_collection], <MilvusException: (code=1100, message=Invalid collection name: 20241218_milvus. the first character of a collection name must be an underscore or letter: invalid parameter)>, <Time:{'RPC start': '2024-12-18 11:42:50.390509', 'RPC error': '2024-12-18 11:42:50.391502'}>
Traceback (most recent call last):
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 124, in <module>
    indexer = MilvusIndexer()
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 17, in __init__
    self.collection = self._initialize_collection()
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 34, in _initialize_collection
    collection = Collection(name=self.collection_name, schema=schema)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 118, in __init__
    has = conn.has_collection(self._name, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 141, in handler
    raise e from e
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 137, in handler
    return func(*args, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 176, in handler
    return func(self, *args, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 116, in handler
    raise e from e
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 86, in handler
    return func(*args, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 355, in has_collection
    raise MilvusException(reply.status.code, reply.status.reason, reply.status.error_code)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1100, message=Invalid collection name: 20241218_milvus. the first character of a collection name must be an underscore or letter: invalid parameter)>

→ 콜렉션의 이름이 알파벳이나 _로 시작해야함

Traceback (most recent call last):
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 126, in <module>
    indexer.process_and_index_json_files(data_path)  # JSON 파일 인덱싱
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 59, in process_and_index_json_files
    self.collection.insert([entities["id"], entities["embedding"], entities["path"], entities["text"]])
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 516, in insert
    check_insert_schema(self.schema, data)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/orm/schema.py", line 774, in check_insert_schema
    _check_data_schema_cnt(tmp_fields, data)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/orm/schema.py", line 746, in _check_data_schema_cnt
    raise DataNotMatchException(message=message)
pymilvus.exceptions.DataNotMatchException: <DataNotMatchException: (code=1, message=The data doesn't match with schema fields, expect 3 list, got 4)>

—> 컬렉션 스키마에서 id필드를 만들고 있으나 데이터 삽입 시 새로 만들어서 발생하는 문제

Traceback (most recent call last):
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 124, in <module>
    indexer.process_and_index_json_files(data_path)  # JSON 파일 인덱싱
  File "/home/manager/llama-index-rag-system/data_pipeline/data_update_to_milvus.py", line 57, in process_and_index_json_files
    self.collection.insert([entities["embedding"], entities["path"], entities["text"]])
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 518, in insert
    return conn.batch_insert(
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 141, in handler
    raise e from e
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 137, in handler
    return func(*args, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 176, in handler
    return func(self, *args, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 116, in handler
    raise e from e
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/decorators.py", line 86, in handler
    return func(*args, **kwargs)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 584, in batch_insert
    raise err from err
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 567, in batch_insert
    request = self._prepare_batch_insert_request(
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 551, in _prepare_batch_insert_request
    else Prepare.batch_insert_param(collection_name, entities, partition_name, fields_info)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/prepare.py", line 735, in batch_insert_param
    return cls._parse_batch_request(request, entities, fields_info, location)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/prepare.py", line 711, in _parse_batch_request
    field_data = entity_helper.entity_to_field_data(
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/entity_helper.py", line 531, in entity_to_field_data
    entity_to_str_arr(entity, field_info, CHECK_STR_ARRAY)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/entity_helper.py", line 184, in entity_to_str_arr
    return convert_to_str_array(entity.get("values", []), field_info, check=check)
  File "/home/manager/llama-index-rag-system/venv/lib/python3.10/site-packages/pymilvus/client/entity_helper.py", line 176, in convert_to_str_array
    raise ParamError(
pymilvus.exceptions.ParamError: <ParamError: (code=1, message=invalid input of field (text), length of string exceeds max length. length: 3964, max length: 2000)>

—> 'text' 필드에 저장하려는 문자열의 길이가 최대 허용 길이를 초과
—> 2000에서 15000으로 변경으로 해결

Indexed 290 rows into Milvus collection 'milvus_20241218'.
Milvus update latency : 17.696376085281372
Collection Info:
<Collection>:
-------------
<name>: milvus_20241218
<description>: Embedding storage
<schema>: {'auto_id': True, 'description': 'Embedding storage', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'embedding', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 1024}}, {'name': 'path', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 50}}, {'name': 'text', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 15000}}], 'enable_dynamic_field': False}

Number of vectors in the collection: 0

 

—> insert된 이후에 vector개수가 0으로 찍히는 경우
—> flush() 삽입
—> https://milvus.io/api-reference/pymilvus/v2.2.x/Collection/flush().md

flush() - Milvus pymilvus sdk v2.2.x/Collection

 

 

 

🐹유사도 검색 테스트

 

from pymilvus import connections, utility
# Milvus에 연결 설정
connections.connect(alias="default", host="host", port="19530")

collections = utility.list_collections()
print("Existing collections:", collections)

collection = Collection("test_collection")
num_entities = collection.num_entities
print(f"Number of entities in collection '{collection.name}': {num_entities}")

# 임베딩 모델 초기화
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-m3")

# 텍스트를 임베딩
text = "테스트입니다."
embedding_vector = embed_model.get_text_embedding(text)

# Milvus에 연결 설정
connections.connect(alias="default", host="host", port="19530")

# 컬렉션 선택
collection = Collection("test_collection")

# 검색 파라미터 설정
search_params = {
    "metric_type": "COSINE",  # 거리 측정 방법 
    "params": {"nprobe": 10}  # 탐색할 클러스터 수
}

# 유사도 검색 실행
results = collection.search(
    data=[embedding_vector],  # 쿼리 벡터
    anns_field="embedding",  # 벡터 필드 이름
    param=search_params,
    limit=5,  # 반환할 결과 수
    output_fields=["path", "text"]  # 반환할 필드
)

for hits in results:
    print(hits)
    for hit in hits:
        print("Text:", hit.get('text'), "Path:", hit.get('path'))

⚠️error

AttributeError: 'Hits' object has no attribute 'entity

 Milvus의 검색 결과 형식은 Python API 버전에 따라 다르며, 최신 버전의 `pymilvus`에서는 `Hit` 객체가 `entity` 속성을 가지지 않는다. (검색 결과의 필드에 직접 접근 해야함)

 

🐹파이프라인 생성

⚠️error

https://github.com/run-llama/llama_index/discussions/14538

How to read a milvus index that already contains my data? and how to add data to it? · run-llama llama_index · Discussion #145

 

ValueError: Node content not found in metadata dict

→ Milvus 벡터 저장소에서 노드 메타데이터를 불러올 때 '_node_content' 필드가 없어서 발생
→ '_node_content' 필드가 메타데이터에 포함되어 있지 않아 LlamaIndex가 노드를 파싱하는 데 실패
→ update코드와 vectorstore 설정에 context_field 추가 

  # Milvus Vector Store 설정
        self.vector_store = MilvusVectorStore(
            uri="http://url",
            collection_name=self.valves.collection_name,
            dim=1024,  # 벡터 차원수 (BAAI/bge-m3 과 동일)
            overwrite=False,
            content_field="_node_content"
        )