ci: harden the qdrant integration job
PR Checks / hadolint (pull_request) Successful in 23s
Docker Build and Push / test (pull_request) Successful in 5m19s
PR Checks / build-test (pull_request) Successful in 47s
Docker Build and Push / integration-test (pull_request) Successful in 5m30s
PR Checks / pytest (pull_request) Successful in 3m7s
Docker Build and Push / build (pull_request) Successful in 3m14s

The integration test job has been failing intermittently on main and on
open pull requests with `httpx.ReadTimeout: timed out` on the first write
to Qdrant, while `GET /collections` and `/healthz` kept answering.

Cause: the qdrant-client default request timeout is 5 seconds, and the job
only waits for `/healthz`, which is served before the storage engine
accepts writes. On a shared runner doing several builds at once, the first
collection creation or indexing exceeds 5s and the whole fixture errors
out.

- pin qdrant/qdrant to v1.19 (the digest `latest` and the cluster already
  run) and let Renovate track the pin in the workflows as well
- wait for a real write round-trip (create + delete a probe collection)
  instead of `/healthz` alone, and dump the container logs if it never
  becomes writable
- make the Qdrant client timeout configurable via QDRANT_TIMEOUT and set
  it to 60s in CI, where there is no latency SLA
This commit is contained in:
Sagent committed 2026-10-17 21:24:18 +00:00
1 parent e883a791fb
commit 01d15c486e
5 files changed
+73 -16

No files matched your search

+30 -4
View File
@@ -32,15 +32,32 @@ jobs:
docker rm -f qdrant-${{ github.run_id }} 2>/dev/null || true
docker run -d --name qdrant-${{ github.run_id }} \
--network "container:$(hostname)" \
docker.io/qdrant/qdrant:latest
docker.io/qdrant/qdrant:v1.19
- name: Wait for Qdrant
run: |
for i in $(seq 1 30); do
curl -s http://localhost:6333/healthz && echo "QDRANT ready" && break
echo "Waiting for Qdrant... ($i/30)"
# /healthz answers before the storage engine accepts writes, and a cold
# container on a loaded host can take several seconds to create its
# first collection. Gate on a real write round-trip, not on /healthz.
ready=""
for i in $(seq 1 60); do
if curl -sf "http://localhost:6333/healthz" > /dev/null 2>&1 \
&& curl -sf -X PUT "http://localhost:6333/collections/ci_readiness_probe?timeout=60" \
-H 'Content-Type: application/json' \
-d '{"vectors":{"size":4,"distance":"Cosine"}}' > /dev/null 2>&1; then
curl -sf -X DELETE "http://localhost:6333/collections/ci_readiness_probe" > /dev/null 2>&1 || true
ready="yes"
echo "QDRANT ready: write path OK (attempt $i)"
break
fi
echo "Waiting for Qdrant... ($i/60)"
sleep 1
done
if [ -z "$ready" ]; then
echo "Qdrant write path never became ready"
docker logs --tail 100 qdrant-${{ github.run_id }} || true
exit 1
fi
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
@@ -53,6 +70,15 @@ jobs:
env:
QDRANT_URL: http://localhost:6333
COLLECTION_NAME: test_mcp_maildir
# The qdrant client defaults to a 5s request timeout, which a cold
# collection on a shared runner can exceed; this job has no SLA.
QDRANT_TIMEOUT: "60"
- name: Qdrant diagnostics
if: failure()
run: |
docker ps -a || true
docker logs --tail 200 qdrant-${{ github.run_id }} || true
- name: Cleanup Qdrant
if: always()