Test reporter

[Ceres6]

Building on top of #52189

This commit introduces a node:test reporter to the common utils. This reporter can be used to silence output other than errors from node:test. This is useful because in Node's own test suite, the output of node:test is included in the output of the Python test runner.

Refs: #49120

Rebased with main to fix conflicts and added FAIL_FAST envvar.

cc @cjihrig @JakobJingleheimer

[nodejs-github-bot]

Review requested:

• @nodejs/actions
• @nodejs/test_runner

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.17%. Comparing base (98d4ebc) to head (4547f09).
Report is 55 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #56438      +/-   ##
==========================================
+ Coverage   88.53%   89.17%   +0.63%     
==========================================
  Files         657      662       +5     
  Lines      190741   191745    +1004     
  Branches    36607    36906     +299     
==========================================
+ Hits       168881   170988    +2107     
+ Misses      15036    13616    -1420     
- Partials     6824     7141     +317     

see 122 files with indirect coverage changes

[Ceres6]

I kept the failing test so the output can be seen. I will remove it once we agree that output is good enough

[cjihrig]

Personally, this seems fine to me. We can also iterate on it pretty quickly since it's internal only. I'm not sure if I can sign off on this or not since I'm the original author, but throwing my LGTM here.

Also, we'll have to remove that intentional failure before landing.

On a side note, I definitely think we (you?) should open another PR to get rid of the GitHub reporter. It generates far too much output noise.

[Ceres6]

Sure, I can open the PR

[cjihrig]

This is what the output currently looks like:

::debug::starting to run default concurrency
::debug::completed running default concurrency
::debug::starting to run concurrency of one
::debug::completed running concurrency of one
::debug::starting to run concurrency of two
::error title=concurrency of two,file=node/node/test/parallel/test-runner-cli-concurrency.js,line=26,col=9::[Error [ERR_TEST_FAILURE]: bye] {
  code: 'ERR_TEST_FAILURE',
  failureType: 'testCodeFailure',
  cause: Error: bye
      at TestContext.<anonymous> (/home/runner/work/node/node/node/test/parallel/test-runner-cli-concurrency.js:26:9)
      at Test.runInAsyncScope (node:async_hooks:211:14)
      at Test.run (node:internal/test_runner/test:931:25)
      at Test.processPendingSubtests (node:internal/test_runner/test:629:18)
      at Test.postRun (node:internal/test_runner/test:1042:19)
      at Test.run (node:internal/test_runner/test:970:12)
      at async Test.processPendingSubtests (node:internal/test_runner/test:629:7)
}
Test failure: 'concurrency of two'
Location: test/parallel/test-runner-cli-concurrency.js:22:1
Error: bye
    at TestContext.<anonymous> (/home/runner/work/node/node/node/test/parallel/test-runner-cli-concurrency.js:26:9)
    at Test.runInAsyncScope (node:async_hooks:211:14)
    at Test.run (node:internal/test_runner/test:931:25)
    at Test.processPendingSubtests (node:internal/test_runner/test:629:18)
    at Test.postRun (node:internal/test_runner/test:1042:19)
    at Test.run (node:internal/test_runner/test:970:12)
    at async Test.processPendingSubtests (node:internal/test_runner/test:629:7)

::debug::starting to run isolation=none uses a concurrency of one
::debug::completed running isolation=none uses a concurrency of one
::debug::starting to run isolation=none overrides --test-concurrency
::debug::completed running isolation=none overrides --test-concurrency
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::tests 5
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::suites 0
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::pass 4
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::fail 1
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::cancelled 0
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::skipped 0
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::todo 0
::notice file=node/test/parallel/test-runner-cli-concurrency.js,line=1,col=1::duration_ms 1478.484154
::group::Test results (4 passed, 1 failed)
::notice::
::endgroup::

All of the lines starting with :: will go away once #56468 lands.

@Ceres6 can you remove the intentionally failing test now so that we can progress towards landing this.

[Ceres6]

@cjihrig done!

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64396/

[MoLow]

this will remove all the GitHub annotations (like the ones in the screenshot) on the failed files - is that intended? those seem pretty useful to me

[cjihrig]

@MoLow it is intentional. See #56438 (comment). I pinged you in #52189 (comment) as well.

Is it possible to make the GitHub reporter stop printing to stdout? If it is possible, I'd love to keep the GitHub annotations.

[MoLow]

Is it possible to make the GitHub reporter stop printing to stdout? If it is possible, I'd love to keep the GitHub annotations.

I'l take a look, sure

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64434/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64437/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64438/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/64440/

[nodejs-github-bot]

Landed in 0576deb

[Ceres6]

@cjihrig I see it the error reporter is not documented in the docs should we expose it?

EDIT: Link edited, I copied the wrong one

[cjihrig]

No, I don't think it should be documented because it's internal. I don't have a strong opinion on exposing it. I wouldn't, but I'll leave that decision up to other folks.