Beholder

README Configuration
internal

README

Beholder

Beholder is an internal QA metrics application. It collects data from YouTrack and Qase, stores normalized metrics in PostgreSQL, and exposes a FastAPI/Jinja/HTMX web UI for QA dashboards, reports, and collection job operations.

The previous Apps Script and utility implementation is preserved under legacy/ as archive-only reference material. Active application code lives under src/beholder/ and must not import from legacy/.

Application Flows

Overview

The Overview screen summarizes the latest available QA and incident data:

  • QA tickets on the latest collected day.
  • Total test cases and failed test cases for the current data month.
  • Incidents for the current data month and latest incident day.
  • QASE automation coverage and Pytest Collect counts when automation metrics exist.

The screen is read-only. It is meant as the first health check for whether collectors are producing useful data.

Daily QA

Daily QA shows the latest collected YouTrack QA activity. Rows combine regular QA tickets and incidents so QA can review what changed on the latest data day.

Data source:

  • youtrack_qa_issues
  • youtrack_incidents

Collection source:

  • beholder regular-issues-daily
  • beholder incidents-daily

QA Metrics

QA Metrics has several subscreens:

  • Issue Overview: per-project, per-day QA ticket totals, total cases, passed cases,

failed cases, success rate, and incidents.

  • Regular Issues: ticket-level regular QA metrics with QA assignee, developer, case

counts, failed count, and success rate.

  • Testing Time: ticket-level testing time shown as human-readable duration, for example

1h 30m.

  • KPI: monthly project KPI table combining regular QA tickets and incidents. It shows

total cases, passed, failed, incidents, defect density, defect leakage, defect detection percentage, pass rate, and cases per ticket.

All metric tables support filtering and sorting where useful for the data shape.

Incidents

Incidents has overview and list screens:

  • Overview: daily incident counts per project.
  • List: individual incident rows with project, issue ID, date, QA assignee, priority,

and status.

Incident collection stores newly created active incidents and refreshes existing incident state when maintenance jobs run.

Automation

Automation has two DB-backed metric streams:

  • QASE Overview stores QASE automation coverage per connected QASE project:

run timestamp, project code, total cases, automated cases, manual cases, and automation percentage.

  • Pytest Overview stores Pytest Collect metrics per connected QASE project. The app

finds the latest QASE run named exactly Collect Only Run, counts skipped results, and stores that as collected test cases.

Both screens include on-demand collection buttons. The same collectors are also available as CLI commands and Kubernetes CronJobs.

Reports

Reports are generated for all projects for a selected month. A generated report is stored in PostgreSQL with report period, creation date, filename, content type, and content. The Reports screen lists generated reports and provides a download button.

Jobs

The Jobs screen manages scheduled collection definitions and manual operational tasks:

  • Trigger or enable/disable scheduled jobs.
  • Run YouTrack backfills for regular issues, incidents, or both.
  • Update a single YouTrack issue or incident by issue ID.
  • Review recent job runs and failures.

Current scheduled job types:

  • regular_issues_daily
  • incidents_daily
  • qase_automation_metrics
  • pytest_collect_metrics

Configuration

Configuration controls project membership used by collectors:

  • Connected YouTrack Projects: YouTrack project aliases used by regular issue and

incident collectors.

  • Connected QASE Projects: QASE project codes used by QASE automation and Pytest

Collect collectors.

  • Project Mapping: display names for connected YouTrack projects.

Project lists are stored in the database. The text areas are locked by default; click Update, edit, then Submit and confirm to save changes.

Default connected QASE projects are:

MITGOID, TP, CT, GETUNIQ

Runtime Commands

beholder web --host 0.0.0.0 --port 8000
beholder regular-issues-daily
beholder incidents-daily
beholder qase-automation-metrics
beholder pytest-collect-metrics
beholder monthly-incidents
beholder import-artifacts --artifacts-dir artifacts/backfill_csv

import-artifacts is a local migration utility for historical CSV artifacts. It is not part of the normal production schedule.

Configuration Variables

Environment variables are used for infrastructure and credentials. Project membership is managed in the app UI after the database is initialized.

Required variables:

  • DATABASE_URL: SQLAlchemy PostgreSQL URL, for example

postgresql+psycopg://beholder:password@beholder-postgres:5432/beholder.

  • YT_BASE_URL: YouTrack base URL.
  • YT_TOKEN: YouTrack API token.
  • QASE_API_TOKEN: Qase API token.

Optional variables:

  • APP_NAME: defaults to Beholder.
  • QASE_API_BASE: defaults to https://api.qase.io/v1.
  • INTERNAL_ONLY: defaults to true.

Where to set variables:

  • Local development: shell environment or .env.
  • Local PostgreSQL: docker-compose.local.yml.
  • Kubernetes non-secret settings: k8s/configmap.yaml.
  • Kubernetes secrets: copy k8s/secret.example.yaml, fill real values, and apply it

as beholder-secrets.

PROJECT_ALIASES and QASE_PROJECT_CODES are not required for deployment. Use the Configuration screen to update connected projects.

Local Development

Start PostgreSQL:

docker compose -f docker-compose.local.yml up -d

Run migrations:

DATABASE_URL=postgresql+psycopg://beholder:beholder@localhost:55432/beholder \
  alembic upgrade head

Start the app:

DATABASE_URL=postgresql+psycopg://beholder:beholder@localhost:55432/beholder \
  beholder web --host 127.0.0.1 --port 8000

Open:

http://127.0.0.1:8000

Health checks:

/health/live
/health/ready

Kubernetes Deployment

The repository contains raw manifests under k8s/ for an empty in-cluster PostgreSQL database and the Beholder app. Replace image names and secret values before applying them to a real cluster.

1. Set local deployment variables.

export NS=beholder
export IMAGE=registry.example.com/beholder:VERSION
export DB_PASSWORD='change-me-strong-password'
export YT_BASE_URL='https://youtrack.example.com'
export YT_TOKEN='change-me'
export QASE_API_TOKEN='change-me'

2. Create the namespace.

kubectl create namespace "$NS"

3. Build and push the image.

docker build -t "$IMAGE" .
docker push "$IMAGE"

4. Update app image references.

  • Set the image in k8s/deployment.yaml, k8s/migration-job.yaml, and

k8s/cronjobs.yaml.

perl -pi -e "s|image: beholder:latest|image: $ENV{IMAGE}|g" \
  k8s/deployment.yaml \
  k8s/migration-job.yaml \
  k8s/cronjobs.yaml

5. Deploy empty PostgreSQL.

k8s/postgres.yaml creates:

  • beholder-postgres-secret
  • beholder-postgres-data PVC
  • beholder-postgres Deployment
  • beholder-postgres Service

Set the database password in the manifest before applying it:

cp k8s/postgres.yaml /tmp/beholder-postgres.yaml
perl -pi -e "s|POSTGRES_PASSWORD: change-me|POSTGRES_PASSWORD: $ENV{DB_PASSWORD}|g" \
  /tmp/beholder-postgres.yaml
kubectl apply -n "$NS" -f /tmp/beholder-postgres.yaml
kubectl rollout status -n "$NS" deployment/beholder-postgres

This creates an empty beholder database owned by the beholder user.

6. Apply Beholder non-secret config.

kubectl apply -n "$NS" -f k8s/configmap.yaml

7. Create Beholder app secrets.

Recommended direct command:

kubectl create secret generic beholder-secrets \
  -n "$NS" \
  --from-literal=DATABASE_URL="postgresql+psycopg://beholder:${DB_PASSWORD}@beholder-postgres:5432/beholder" \
  --from-literal=YT_BASE_URL="$YT_BASE_URL" \
  --from-literal=YT_TOKEN="$YT_TOKEN" \
  --from-literal=QASE_API_TOKEN="$QASE_API_TOKEN"

Alternatively, copy k8s/secret.example.yaml outside git, fill real values, and apply that file as beholder-secrets. Do not commit real tokens.

8. Run database migrations.

kubectl delete job -n "$NS" beholder-migrate --ignore-not-found
kubectl apply -n "$NS" -f k8s/migration-job.yaml
kubectl wait -n "$NS" --for=condition=complete job/beholder-migrate --timeout=180s

If migration fails, inspect logs:

kubectl logs -n "$NS" job/beholder-migrate

9. Deploy the web app and service.

kubectl apply -n "$NS" -f k8s/deployment.yaml
kubectl apply -n "$NS" -f k8s/service.yaml
kubectl rollout status -n "$NS" deployment/beholder-web

10. Verify the web app.

kubectl port-forward -n "$NS" service/beholder-web 8000:80

Open:

http://127.0.0.1:8000/health/live
http://127.0.0.1:8000/health/ready
http://127.0.0.1:8000

11. Configure projects in the UI.

Open:

http://127.0.0.1:8000/configuration/projects

Confirm or update:

  • Connected YouTrack Projects
  • Connected QASE Projects

Default connected QASE projects are seeded as:

MITGOID, TP, CT, GETUNIQ

12. Apply scheduled collectors.

kubectl apply -n "$NS" -f k8s/cronjobs.yaml
kubectl get cronjobs -n "$NS"

13. Optional manual smoke job.

Trigger one CronJob manually:

kubectl create job -n "$NS" \
  --from=cronjob/beholder-regular-issues-daily \
  beholder-regular-issues-smoke
kubectl logs -n "$NS" job/beholder-regular-issues-smoke

You can also trigger jobs from:

http://127.0.0.1:8000/jobs

14. Final cluster checks.

kubectl get pods -n "$NS"
kubectl get svc -n "$NS"
kubectl get cronjobs -n "$NS"
kubectl logs -n "$NS" deployment/beholder-web --tail=100

In the UI, check Overview, Configuration, Jobs, Runs, and the README button.

Deployment Notes

  • Run migrations before rolling out web pods that need a new schema.
  • CronJobs and on-demand Jobs share the same app image and environment variables.
  • The web app does not run an in-process scheduler; Kubernetes CronJobs and manual UI

triggers are responsible for collection execution.

  • QASE Pytest Collect expects Qase runs named exactly Collect Only Run.
  • Real secrets belong in cluster secret management, not in git.

Cleanup Policy

Generated files are not part of the application and can be deleted safely:

  • __pycache__/
  • .playwright-mcp/
  • build/
  • local artifacts/
  • *.pyc

The legacy/ directory is kept only as historical reference. It is not packaged by the Dockerfile, not imported by src/beholder, and not used by Kubernetes manifests.