Skip to content
View feiiiiii5's full-sized avatar
🏠
Working from home
🏠
Working from home

Block or report feiiiiii5

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
feiiiiii5/README.md

Hi, I'm Fei

I am an independent developer and open-source contributor focusing on AI evaluation, red-teaming, and agent tooling. I also study Cyber Science and Technology at Sun Yat-sen University.

My primary research and engineering interest is silent failures in evaluation pipelines—cases where an eval harness, LLM judge, or RL reward function hits an underlying fault but still returns a plausible, normal-looking score.

  • Open Source: 412 merged PRs across 87 repositories (as of Nov 2026), primarily across LLM evaluation, observability, and structured generation.
  • Maintainer / Reviewer: Area Reviewer for feedback functions and metrics in TruLens.
  • Active Ecosystems: PyRIT · Opik · TruLens · Inspect · Rhesis · OpenInference · Pandera

Selected Contributions

PR Impact & Mechanism
Inspect Evals #2132 Preserved individual attempt outcomes in HLE calibration so averaging repeated runs no longer masks high-confidence incorrect answers.
Opik #8152 Fixed G-Eval score field resolution and mapped multi-token scores (e.g., 10) to their correct log-probabilities.
PyRIT #3125 Enforced literal key comparisons for metadata filters on SQLite and Azure SQL, preventing dotted keys from matching or deleting unintended records.
xgrammar #834 Allowed valid short arrays in JSON Schema prefixItems without permitting subsequent items to skip positional validation.
Serena #2030 Made pending requests fail immediately upon language server write errors instead of hanging until timeout.

Note on workflow: I use coding agents in my development loop. I manually triage issues, define acceptance criteria, verify root causes, and rigorously audit and test all generated patches before submitting PRs and addressing maintainer review.


Research & Projects

Silent Failures & Reliability

  • rewardfail — Empirical study and prototype fix on how math-checker failures in RL reward code silently degrade into zero rewards or batch crashes depending on exception boundaries. (Code, experiments & non-peer-reviewed preprint)
  • failroute — Static analysis CLI for Python that flags exception handlers capable of masking runtime failures behind normal-looking return values. (Tooling & non-peer-reviewed preprint)

Systems & Tooling

  • QuantCore — Local-first A-share research and backtesting workbench powered by a C++20 execution engine and zero-copy pybind11 bindings.
  • releasefloor — Static inspector that verifies recorded runtime requirements (ELF/Mach-O headers, target architectures) across release archives without binary execution.

Stack & Contact

  • Languages & Core: Python, C++20, pybind11, SQL
  • AI & Eval Infra: Inspect, TruLens, Opik, PyRIT, OpenInference

Found a bug in one of my tools or want to talk about eval reliability? Feel free to open an issue in the respective repo or reach out on LinkedIn.

Pinned Loading

  1. Johanson-colab/Archimedes Johanson-colab/Archimedes Public

    A local-first research agent workspace for customizable research apps and integrated, traceable scientific workflows.

    JavaScript 24 3

  2. rewardfail rewardfail Public

    Math-Verify failure audit: unreviewed manuscript, code and evidence

    Python 1

  3. truera/trulens truera/trulens Public

    Evaluation and Tracking for LLM Experiments and AI Agents

    Python 3.7k 397

  4. comet-ml/opik comet-ml/opik Public

    Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

    Java 23.5k 2k

  5. microsoft/PyRIT microsoft/PyRIT Public

    Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

    Python 4.9k 1.1k

  6. failroute failroute Public

    Python tool that flags exception handlers for review when they may hide failures. CLI, regression tests and a public, unreviewed manuscript.

    Python 2