[HN Gopher] Test Anything Protocol
___________________________________________________________________
Test Anything Protocol
Author : brianzelip
Score : 49 points
Date : 2023-10-05 19:21 UTC (3 hours ago)
(HTM) web link (testanything.org)
(TXT) w3m dump (testanything.org)
| x86hacker1010 wrote:
| Not to be confused with the Telocator Alphanumeric Protocol (TAP)
| like TUN/TAP virtual network interfaces!
| sizediterable wrote:
| I was recently trying out Node's newish test runner [1] but
| wanted better error reporting and was grateful that Node provided
| an API that let me use any TAP-compatible error reporter. I did
| find unfortunately that most error reporters I tried seemed to
| have problems, the most common one was not reporting whether or
| not a test was skipped, which was a deal-breaker. In the end I
| settled for tap-mocha-reporter [2] since it worked well enough. I
| did have to end up patching it to not print extraneous new lines,
| I suspect an artifact of it using YAML for its internal
| representation. I do wish there was an error reporter that
| mimicked exactly the format of Jest, as it's what I'm most used
| to.
|
| [1] https://nodejs.org/api/test.html
|
| [2] https://github.com/tapjs/tap-mocha-reporter
| zaphar wrote:
| I always liked the TAP protocol for test reporting. I think part
| of it is because the protocol is so succinct and makes no
| assumptions about how the testing framework works under the hood.
| dang wrote:
| Related:
|
| _Test Anything Protocol_ -
| https://news.ycombinator.com/item?id=23473370 - June 2020 (29
| comments)
|
| _Test Anything Protocol specification (2006)_ -
| https://news.ycombinator.com/item?id=11915487 - June 2016 (10
| comments)
|
| _Test Anything Protocol_ -
| https://news.ycombinator.com/item?id=10030889 - Aug 2015 (1
| comment)
|
| _libtap - Testing library for C, implementing the Test Anything
| Protocol._ - https://news.ycombinator.com/item?id=2936877 - Aug
| 2011 (6 comments)
| arraypad wrote:
| Also not to be confused with Google's Test Automation Platform
| (continuous integration system) [1].
|
| [1]
| https://static.googleusercontent.com/media/research.google.c...
| freedude wrote:
| perl Tutorial
|
| https://metacpan.org/dist/Test-Simple/view/lib/Test/Tutorial...
| bdcravens wrote:
| The subreddit linked on the page has 3 posts in the last year
| o11c wrote:
| TAP is tolerable for minimal testing, but fails to provide a lot
| of features you really want for anything nontrivial.
|
| A while back I wrote out a list of what we _really_ want out of a
| test framework (more than any existing framework supports, though
| many get much closer than TAP):
|
| https://gist.github.com/o11c/ef8f0886d5967dfebc3d
| amluto wrote:
| Barely tolerable. At one point, the Linux selftests were
| contemplating using TAP. I read the main example:
| 1..4 ok 1 - Input file opened not ok 2 - First
| line of the input valid ok 3 - Read the rest of the
| file not ok 4 - Summarized correctly # TODO Not written
| yet
|
| and I had two immediate objections.
|
| a) Shouldn't tests have names?
|
| b) (more serious) To generate valid TAP output, I need to know
| up-front how many tests there are. What if I can run the tests
| in sequence but I don't know how to count them? This can happen
| if there is a tree of tests, where each node knows what
| children it has but not what children _they_ have. There 's no
| credible way to generate TAP output streamily, even if there is
| no parallelization of tests.
|
| No thanks.
| duskwuff wrote:
| > a) Shouldn't tests have names?
|
| That's what the text following the test number is for.
|
| > b) (more serious) To generate valid TAP output, I need to
| know up-front how many tests there are. What if I can run the
| tests in sequence but I don't know how to count them? This
| can happen if there is a tree of tests, where each node knows
| what children it has but not what children they have.
|
| Typically you'd use subtests for this, e.g.
| 1..1 # Subtest: root 1..2
| Subtest: branch 1 1..2 ok 1 -
| leaf 1 ok 2 - leaf 2 ok 1 -
| branch 1 ok 2 - leaf 3 ok 1 - root
|
| But if you really didn't know the number of tests up front
| for some reason, the test plan is also allowed to appear at
| the end of the file.
| spc476 wrote:
| a) They can have names (internally). At my previous job, our
| test suite required every test be a single file, and yet it
| still output TAP (you specified which tests you wanted via
| filenames).
|
| b) No, you don't have to know prior how many tests will run.
| If you read the spec, you'll see there's an option to print
| the final count at the end.
| SrslyJosh wrote:
| Nice list, but I think it would be simpler to ask that
| frameworks support custom outcomes, with user-defined behavior.
| Most (all?) of your list of outcomes can be boiled down to a
| few behaviors:
|
| 1. Should this result be treated as a pass or a fail? 2. Should
| a warning of some sort be emitted? (Ex: for FAST and SLOW) 3.
| Should the result be counted as part of the main test corpus,
| or as part of some auxiliary grouping?
___________________________________________________________________
(page generated 2023-10-05 23:00 UTC)