close
Skip to content

feat: better linter assert failure mode reporting#3230

Open
hippietrail wants to merge 2 commits into
Automattic:masterfrom
hippietrail:assert-suggestion-result-enhancement
Open

feat: better linter assert failure mode reporting#3230
hippietrail wants to merge 2 commits into
Automattic:masterfrom
hippietrail:assert-suggestion-result-enhancement

Conversation

@hippietrail
Copy link
Copy Markdown
Collaborator

Issues

N/A

Description

I noticed while working on linters that it's sometimes hard to know how/why assert_suggestion_result() failed.

  • Sometimes the Expr/Pattern didn't even match.
  • Sometimes there was a match but none of the Suggestions are what were expected.

This PR adds a few emojis: ❌ for when it fails for any reason, followed by the reason. 🎯 for the expected Suggestion. 🔍 for the first actual Suggestion.
If there are multiple failing Suggestions, only the first is shown, but it's followed by the number of other failing ones. This can be changed if need be.

Disclaimer: The built-in LLM in Windsurf (SWE-1.5) did most of the actual code changes. I guided it closely and I did most of the testing manually.

Demo

thread 'linting::interested_in::tests::fix_for' (91987871) panicked at harper-core/src/linting/interested_in.rs:93:9:
❌ No matches found by the linter.
🎯 Expected: "but the user is only interested in the examples in one of the modes"
thread 'linting::behind_the_scenes::tests::pluralize_and_hyphenate' (91667684) panicked at harper-core/src/linting/behind_the_scenes.rs:80:9:
❌ No suggestion sequence produced the expected result.
🎯 Expected:  "So, to open the 'real' behind-the-scenes menu i need to do these steps:"
🔍 But found: "So, to open the 'real' behind the scenes menu i need to do these steps:"
thread 'linting::most_number::tests::corrects_most_amount_title_case' (92034534) panicked at harper-core/src/linting/most_number.rs:96:9:
❌ No suggestion sequence produced the expected result.
🎯 Expected:  "Area of Container with the Greatest Amount of Water"
🔍 But found: "Area of Container with the HighEst amount of Water" and 2 others

How Has This Been Tested?

  • A couple of tests to verify the various failure modes and variants behave as expected.
  • Manual testing on lots of #[ignore]d tests from various linters for "real world" testing.

Checklist

  • I have performed a self-review of my own code
  • I have added tests to cover my changes
  • I have considered splitting this into smaller pull requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant