ci: harden the qdrant integration job
PR Checks / hadolint (pull_request) Successful in 23s
Docker Build and Push / test (pull_request) Successful in 5m19s
PR Checks / build-test (pull_request) Successful in 47s
Docker Build and Push / integration-test (pull_request) Successful in 5m30s
PR Checks / pytest (pull_request) Successful in 3m7s
Docker Build and Push / build (pull_request) Successful in 3m14s
PR Checks / hadolint (pull_request) Successful in 23s
Docker Build and Push / test (pull_request) Successful in 5m19s
PR Checks / build-test (pull_request) Successful in 47s
Docker Build and Push / integration-test (pull_request) Successful in 5m30s
PR Checks / pytest (pull_request) Successful in 3m7s
Docker Build and Push / build (pull_request) Successful in 3m14s
The integration test job has been failing intermittently on main and on open pull requests with `httpx.ReadTimeout: timed out` on the first write to Qdrant, while `GET /collections` and `/healthz` kept answering. Cause: the qdrant-client default request timeout is 5 seconds, and the job only waits for `/healthz`, which is served before the storage engine accepts writes. On a shared runner doing several builds at once, the first collection creation or indexing exceeds 5s and the whole fixture errors out. - pin qdrant/qdrant to v1.19 (the digest `latest` and the cluster already run) and let Renovate track the pin in the workflows as well - wait for a real write round-trip (create + delete a probe collection) instead of `/healthz` alone, and dump the container logs if it never becomes writable - make the Qdrant client timeout configurable via QDRANT_TIMEOUT and set it to 60s in CI, where there is no latency SLA
This commit is contained in:
5 files changed
+73
-16
No files matched your search
+6
-2
@@ -36,6 +36,10 @@ COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "")
|
||||
EMBEDDING_MODEL_NAME = os.environ.get("EMBEDDING_MODEL_NAME", "BAAI/bge-small-en-v1.5")
|
||||
DEFAULT_LIMIT = int(os.environ.get("SEARCH_LIMIT", "50"))
|
||||
|
||||
# Per-request timeout (seconds) for Qdrant. The client default is 5s, which is
|
||||
# tight for a cold collection or a busy host: unset means keep that default.
|
||||
QDRANT_TIMEOUT = int(os.environ["QDRANT_TIMEOUT"]) if os.environ.get("QDRANT_TIMEOUT") else None
|
||||
|
||||
if not QDRANT_URL:
|
||||
raise ValueError("QDRANT_URL environment variable is required.")
|
||||
if not COLLECTION_NAME:
|
||||
@@ -60,8 +64,8 @@ def get_qdrant_client() -> QdrantClient:
|
||||
"""Returns a singleton Qdrant client."""
|
||||
global _qdrant_client
|
||||
if _qdrant_client is None:
|
||||
logger.info(f"Connecting to Qdrant at {QDRANT_URL}")
|
||||
_qdrant_client = QdrantClient(url=QDRANT_URL)
|
||||
logger.info(f"Connecting to Qdrant at {QDRANT_URL} (timeout={QDRANT_TIMEOUT})")
|
||||
_qdrant_client = QdrantClient(url=QDRANT_URL, timeout=QDRANT_TIMEOUT)
|
||||
return _qdrant_client
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user