[HN Gopher] Fuzz testing: the best thing to happen to our applic... ___________________________________________________________________ Fuzz testing: the best thing to happen to our application tests Author : bluestreak Score : 29 points Date : 2023-08-17 16:15 UTC (6 hours ago) (HTM) web link (questdb.io) (TXT) w3m dump (questdb.io) | yosefk wrote: | The relative rarity of input (pseudo-)randomization in SW testing | is near inexplicable to me, except by the very low cost of all | but the most commonly reproducing bugs paid by the SW vendor. | mqus wrote: | In the regular testsuite (think CI) you want to have | predictable results. Doing them again and again on the same | code should give the same results so you can properly see with | which code change things got wrong. Maybe it's simpler to | explain it the other way around, for every new path your | fuzzer(or other randomized test) tests, it also doesn't test a | path it tested in a previous run and you probably want to add | the failing paths it found to your regular test suite. | | Don't get me wrong, we should have more randomization, but it's | not good everywhere, which might explain why we don't have as | much of it. | jerrinot wrote: | Hello, I'm Jaromir, one of the core engineers at QuestDB team. I | just noticed this blog is trending! Andrei - the author - lives | in Bulgaria and he is probably already sleeping. Happy to answer | any question the blog left unanswered. | yosefk wrote: | What tests did you have before adding fuzzing? | jerrinot wrote: | it's the usual spectrum of tests: | | 1. correctness: from small units tests to relatively complex | integrations tests. they typically populate a test database | and query it via various interfaces, such as REST or the | Postgres protocol. we use Azure Pipelines to execute them - | testing in MacoOS, Linux (both Intel and ARM) and Windows. | | 2. performance: we tend to use the TSBS project for most of | our performance testing and profiling. fun fact: we actually | had to patch it as the vanilla TSBS was a bottleneck in some | tests. Sadly, the PR with the improvements is still not | merged: https://github.com/timescale/tsbs/pull/186 | | edit: I thought I would link some of the more interesting | tests: Since QuestDB supports the Postgres wire protocol we | have to gracefully handle even various half-broken Postgres | clients. But how do you write a test with mimicking a client | generating invalid requests? No sane client will generate | broken requests. So we use Wireshark to record network | communication between the broken client and our server and | then use this recorded communication in tests. Example: https | ://github.com/questdb/questdb/blob/3995c31210c70664d4b3... ___________________________________________________________________ (page generated 2023-08-17 23:00 UTC)