Eidetic
← Selected achievements Employment history · 2019–2026

Engineering systems at Reward Gateway.

I led quality engineering and worked hands-on on AI evaluation, delivery infrastructure and incident analysis.

Role
Head of Quality Engineering
Team
Grew from 4 to 15 engineers
Organisation
Supported more than 200 people in product and engineering

Built the evidence to assess an AI assistant for release.

The assistant answered employee questions from each company’s content across web, iOS and Android. The existing system was traced from content publishing to the employee’s answer.

AI validation and continuous evaluationVersioned evaluation datasets contain reference answers, expected content and typical, edge and adversarial cases. The layered suite contains approximately 200 tests, including approximately 50 security tests. Security tests covered authentication and tenant isolation, prompt injection and guardrails, and personal data protection. Testing also covered data ingest, load and performance, and AWS token costs. The AI assistant is one box across web, iOS and Android; selected checks use model stubbing. Repeated runs are evaluated with bounded assertions, semantic assertions and classifier validation. Retrieval relevance and scope, answer grounding, citation accuracy and consistency, and reliability are distinct validation concerns. Test evidence feeds automated regression gates, then a go/no-go report with coverage and gaps for the panel's release decision. Failed checks feed failure analysis. Production feedback from Datadog monitoring also feeds failure analysis, which informs new cases in the versioned datasets. Connections show testing and evidence relationships, not internal architecture or complete coverage.Versioned datasetsReference answers · expected contentTypical · edge · adversarial cases≈200 tests totalUnit · integration · API · functionalData ingest · load & performanceAWS token costs≈50 security testsAuthentication · tenant isolationPrompt injection · guardrailsPersonal data protectionAI assistantWeb · iOS · AndroidModel stubbingSelected checksRepeated runsMulti-run samplingEvaluation methodsBounded assertionsSemantic assertionsClassifier validationRetrievalRelevance · scopeAnswer qualityGrounding · citation accuracy · consistencyReliabilityLatency · streaming · failuresRegression gatesAutomated checksGo/no-go reportCoverage & gapsPanel release decisionFailure analysisNew regression casesProduction feedbackDatadog monitoringAI validation and continuous evaluationVersioned evaluation datasets contain reference answers, expected content and typical, edge and adversarial cases. The layered suite contains approximately 200 tests, including approximately 50 security tests. Security tests covered authentication and tenant isolation, prompt injection and guardrails, and personal data protection. Testing also covered data ingest, load and performance, and AWS token costs. The AI assistant is one box across web, iOS and Android; selected checks use model stubbing. Repeated runs are evaluated with bounded assertions, semantic assertions and classifier validation. Retrieval relevance and scope, answer grounding, citation accuracy and consistency, and reliability are distinct validation concerns. Test evidence feeds automated regression gates, then a go/no-go report with coverage and gaps for the panel's release decision. Failed checks feed failure analysis. Production feedback from Datadog monitoring also feeds failure analysis, which informs new cases in the versioned datasets. Connections show testing and evidence relationships, not internal architecture or complete coverage.Versioned datasetsReference answers · expected contentTypical · edge · adversarial cases≈200 tests totalUnit · integration · API · functionalData ingest · load & performanceAWS token costs≈50 security testsAuthentication · tenant isolationPrompt injection · guardrailsPersonal data protectionAI assistantWeb · iOS · AndroidModel stubbingSelected checksRepeated runsMulti-run samplingEvaluation methodsBounded assertionsSemantic assertionsClassifier validationRetrievalRelevance · scopeAnswer qualityGrounding · citation accuracy · consistencyReliabilityLatency · streaming · failuresRegression gatesAutomated checksGo/no-go reportCoverage & gapsPanel release decisionFailure analysisNew regression casesProduction feedbackDatadog monitoring
Versioned cases, repeated evaluation and regression gates support release decisions. Failures and production feedback inform the next test cycle.

Evaluation and security

The suite comprised approximately 200 tests across unit, integration, API and functional layers. Testing also covered data ingest, load and performance, and AWS token costs.

Approximately 50 security tests covered:

  • Authentication and tenant isolation.
  • Prompt injection and guardrails. Adversarial testing included oversized prompts and mixed-language attempts to bypass guardrails.
  • Personal data protection.

Testing variable answers

Across runs, validation methods addressed:

  • Required and prohibited content.
  • Retrieval relevance, scope, answer grounding and citation accuracy.
  • Meaning and consistency through semantic assertions, classifier-based validation and multi-run sampling.
  • Reliability, including latency, streaming and failure handling.

Evidence for the release decision

  • Versioned evaluation datasets were created with reference answers and expected content across typical, edge and adversarial cases.
  • Automated regression gates checked changes; failure analysis and production monitoring in Datadog informed new regression cases.
  • Follow-up-question rate was tracked as a product signal.

I wrote the gap analysis and risk register for the AI assistant’s production go/no-go, and supplied the QA test report for the panel.

AI validation and continuous evaluation

A stubbed model boundary was used where live inference was unnecessary.

Using incident analysis to guide engineering work

A Python pipeline connected incident, documentation and monitoring systems to analyse hundreds of production incidents over ninety days.

The £227,000 annual business case estimated potential value from preventing failures, not realised savings.

Building delivery infrastructure

Roughly one thousand end-to-end tests ran across pull-request, release and nightly stages. The infrastructure remained operational through two changes of ownership, including the Edenred acquisition.

  • Feedback and reliability. Suites ran in parallel by product area, with repeatable environments built from fixtures and seed data. Reporting combined test results with application-error checks.
  • Reusable frameworks. Test tooling prepared data, triggered background work and checked resulting state and notifications. Backend engineers owned API tests that exercised the application through external HTTP requests.
  • Localisation. A reusable test layer resolved expected interface text from translation files and reported missing translations.
  • Capacity. Representative workloads, controlled environment scaling and order-fulfilment checks informed platform capacity testing.
  • Evidence. A library of more than seven thousand cases linked test evidence to released changes. Teams agreed what to test before implementation.
  • Engineering judgement. A pull-request control was removed when its results no longer provided trustworthy evidence.

Growing the quality engineering team

The function spanned London and Sofia.

  • Built quality metrics in Datadog to give engineering teams a shared view of delivery quality.
  • A technical career framework was designed.
  • Nine colleagues from outside engineering successfully completed an 18-week quality engineering intern programme covering manual testing, test management and automation. Recommendations for engineering roles followed.

Accessibility across web and mobile

Evaluation covered web, iOS and Android.

  • Automated checks covered more than 150 user journeys.
  • Jointly scoped the client-facing WCAG 2.2 AA accessibility audit and VPAT report, including the user journeys assessed, to support enterprise customers’ purchasing requirements.

AI testing and accessibility evaluations informed supplier recommendations, documenting limitations. I supplied Reward Gateway’s digital accessibility CAPEX and OPEX to Edenred Group for its 2025 Universal Registration Document, at the external auditors’ request.

Other AI work

Pull-request review

Built in 2024, a GenAI agent reviewed open pull requests for test code and method naming against codebase conventions.