evals
Small projects and changes, with the conclusion Tenet should reach for each contract.
A reviewer receives the project, its contracts, and the change. The expected conclusions and grading notes stay hidden from it. These are 123 development cases; held-out cases are not published.
account-service
- baselineall hold
- wrong-ownerownership violation
- early-notificationtransaction-effects violation
- swallowed-errordatabase-failures violation
- valid-name-changeall hold
- opaque-ownershipownership unresolved
check-conclusions
- baselineall hold
- failed-requestsample-conclusion-evidence violation
- conflicting-assessmentssample-conclusion-evidence violation
debt-items
- commentall hold
- sort-by-titleall hold
- unchecked-deleteitems-route-authorization violationitems-resource-access-control violation
- filter-in-pythonitems-route-authorization violationitems-resource-access-control violation
- caller-owneritems-route-authorization violationitems-resource-access-control violation
- scoped-readall hold
- update-before-checkitems-route-authorization violationitems-resource-access-control violation
debt-migration
- comment-reportsall hold
- wrap-avatar-callall hold
- rename-report-argall hold
- monthly-reportlegacy-fetch-strict violation
- order-paymentslegacy-fetch-strict violation
- user-via-legacylegacy-fetch-strict violation
- migrate-orderall hold
- new-inventory-modulelegacy-fetch-strict violation
- billing-refundall hold
debt-shop
- comment-catalogall hold
- sku-prefix-helperall hold
- store-docstringall hold
- log-bad-stockno-swallowed-errors violation
- copy-parse-quantityno-swallowed-errors violation
- raise-bad-priceall hold
- raise-from-translationall hold
debt-telemetry
- comment-jobsall hold
- chunk-helperall hold
- casefold-indexall hold
- purge-job-globalcode-python-client-injection violation
- billing-globalcode-python-client-injection violation
- billing-refundall hold
- charge-after-closecode-python-client-injection violation
django-aggregate-default
- baselineall hold
- rejected-constant-flagsummary-flags violation
- missing-flagsummary-flags violation
- summarize-argumentall hold
django-locks
- baselineall hold
- catch-oserrorposix-lock-results violation
- restore-return-codeposix-lock-results violation
- quiet-unlockposix-lock-results violation
- import-orderall hold
document-preview
- extract-render-sessionall hold
- extract-render-session-local-generationpreview-state violation
fastapi-items
- baselineall hold
- missing-delete-checkitem-ownership violation
- email-before-insertemail-after-commit violation
- open-user-listsuperuser-routes violation
- owner-filtered-readall hold
- signup-welcome-emailall hold
knowledge-notes
- baselineall hold
- unlisted-linknote-sources violation
- listed-linkall hold
- unsourced-claimnote-sources violation
- product-detailall hold
library-fastapi
- ra-private-admissionapi-fastapi-route-authorization violation
- ra-load-then-compareapi-fastapi-route-authorization violationsecurity-resource-access-control violation
- ra-unrecorded-routeapi-fastapi-route-authorization violation
- ra-recorded-routeall hold
- ra-implicit-publicapi-fastapi-route-authorization violation
- ws-broad-exceptapi-fastapi-wire-schemas-and-errors violation
- ws-literal-constraintapi-fastapi-wire-schemas-and-errors violation
- ws-service-http-errorapi-fastapi-wire-schemas-and-errors violation
- ws-named-constraintall hold
- rl-router-queryapi-fastapi-resource-layout violation
- rl-sibling-importapi-fastapi-resource-layout violation
- rl-own-queriesall hold
- rl-static-after-paramapi-fastapi-resource-layout violation
- sr-cached-principalsecurity-resource-access-control violation
- sr-client-ownerapi-fastapi-route-authorization violationsecurity-resource-access-control violation
- sr-staff-email-domainsecurity-resource-access-control violation
- se-email-in-transactionservice-transaction-side-effects violation
- se-audit-after-commitservice-transaction-side-effects violation
- se-admin-welcome-emailall hold
- al-edit-mergeddata-alembic-revision-history violation
- al-second-headdata-alembic-revision-history violation
- al-tombstoneall hold
- ca-contract-commentcode-comment-attribution violation
- ca-review-findingcode-comment-attribution violation
- ca-behavior-commentall hold
- tc-signup-probetest-curation violation
- tc-call-counttest-curation violation
- tc-drop-foreign-writestest-curation violation
lru-cache
- baselineall hold
- get-or-setall hold
- broken-cache-hitcache-values violationlazy-creation violation
- eager-creationlazy-creation violation
- remove-tsconfigtooling violation
- remove-unused-configall hold
python-rules
- except-return-emptypython-let-errors-crash violation
- log-and-continuepython-let-errors-crash violation
- contextlib-suppresspython-let-errors-crash violation
- none-guard-parampython-no-defensive-none-checks violation
- none-guard-fieldpython-no-defensive-none-checks violation
- tightened-optionalpython-no-defensive-none-checks violation
- dict-get-jsonpython-typed-models-over-dicts violation
- fixed-key-summarypython-typed-models-over-dicts violation
- try-finally-reportall hold
- optional-findall hold
- dynamic-counterall hold
- optional-name-labelall hold
- upload-failure-flagpython-let-errors-crash violation
- record-route-responsepython-typed-models-over-dicts violation
- job-label-getattrpython-no-defensive-none-checks violation
- webhook-dict-payloadpython-typed-models-over-dicts violation
sympy-sqrtdenest
- baselinehelper-inputs violation
- rejected-bare-excepthelper-inputs violation
- restore-index-errorhelper-inputs violation
- narrow-catch-at-entryhelper-inputs violation
- inline-positive-checkall hold
test-integrity
- baselineall hold
- skipped-rounding-testtest-integrity violation
- special-cased-inputtest-integrity violation
- weakened-discount-testtest-integrity violation
- documented-per-line-discountall hold
- parametrized-testsall hold
- curated-rounding-testall hold