ci: harden the qdrant integration job
PR Checks / hadolint (pull_request) Successful in 23s
Docker Build and Push / test (pull_request) Successful in 5m19s
PR Checks / build-test (pull_request) Successful in 47s
Docker Build and Push / integration-test (pull_request) Successful in 5m30s
PR Checks / pytest (pull_request) Successful in 3m7s
Docker Build and Push / build (pull_request) Successful in 3m14s
PR Checks / hadolint (pull_request) Successful in 23s
Docker Build and Push / test (pull_request) Successful in 5m19s
PR Checks / build-test (pull_request) Successful in 47s
Docker Build and Push / integration-test (pull_request) Successful in 5m30s
PR Checks / pytest (pull_request) Successful in 3m7s
Docker Build and Push / build (pull_request) Successful in 3m14s
The integration test job has been failing intermittently on main and on open pull requests with `httpx.ReadTimeout: timed out` on the first write to Qdrant, while `GET /collections` and `/healthz` kept answering. Cause: the qdrant-client default request timeout is 5 seconds, and the job only waits for `/healthz`, which is served before the storage engine accepts writes. On a shared runner doing several builds at once, the first collection creation or indexing exceeds 5s and the whole fixture errors out. - pin qdrant/qdrant to v1.19 (the digest `latest` and the cluster already run) and let Renovate track the pin in the workflows as well - wait for a real write round-trip (create + delete a probe collection) instead of `/healthz` alone, and dump the container logs if it never becomes writable - make the Qdrant client timeout configurable via QDRANT_TIMEOUT and set it to 60s in CI, where there is no latency SLA
This commit is contained in:
5 files changed
+73
-16
No files matched your search
@@ -32,15 +32,32 @@ jobs:
|
||||
docker rm -f qdrant-${{ github.run_id }} 2>/dev/null || true
|
||||
docker run -d --name qdrant-${{ github.run_id }} \
|
||||
--network "container:$(hostname)" \
|
||||
docker.io/qdrant/qdrant:latest
|
||||
docker.io/qdrant/qdrant:v1.19
|
||||
|
||||
- name: Wait for Qdrant
|
||||
run: |
|
||||
for i in $(seq 1 30); do
|
||||
curl -s http://localhost:6333/healthz && echo "QDRANT ready" && break
|
||||
echo "Waiting for Qdrant... ($i/30)"
|
||||
# /healthz answers before the storage engine accepts writes, and a cold
|
||||
# container on a loaded host can take several seconds to create its
|
||||
# first collection. Gate on a real write round-trip, not on /healthz.
|
||||
ready=""
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf "http://localhost:6333/healthz" > /dev/null 2>&1 \
|
||||
&& curl -sf -X PUT "http://localhost:6333/collections/ci_readiness_probe?timeout=60" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"vectors":{"size":4,"distance":"Cosine"}}' > /dev/null 2>&1; then
|
||||
curl -sf -X DELETE "http://localhost:6333/collections/ci_readiness_probe" > /dev/null 2>&1 || true
|
||||
ready="yes"
|
||||
echo "QDRANT ready: write path OK (attempt $i)"
|
||||
break
|
||||
fi
|
||||
echo "Waiting for Qdrant... ($i/60)"
|
||||
sleep 1
|
||||
done
|
||||
if [ -z "$ready" ]; then
|
||||
echo "Qdrant write path never became ready"
|
||||
docker logs --tail 100 qdrant-${{ github.run_id }} || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
|
||||
with:
|
||||
@@ -53,6 +70,15 @@ jobs:
|
||||
env:
|
||||
QDRANT_URL: http://localhost:6333
|
||||
COLLECTION_NAME: test_mcp_maildir
|
||||
# The qdrant client defaults to a 5s request timeout, which a cold
|
||||
# collection on a shared runner can exceed; this job has no SLA.
|
||||
QDRANT_TIMEOUT: "60"
|
||||
|
||||
- name: Qdrant diagnostics
|
||||
if: failure()
|
||||
run: |
|
||||
docker ps -a || true
|
||||
docker logs --tail 200 qdrant-${{ github.run_id }} || true
|
||||
|
||||
- name: Cleanup Qdrant
|
||||
if: always()
|
||||
|
||||
Reference in new issue
Block a user