[HN Gopher] Show HN: Scanning the Web for Security.txt Files
___________________________________________________________________
Show HN: Scanning the Web for Security.txt Files
Author : _wldu
Score : 28 points
Date : 2021-01-01 18:06 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| BigBalli wrote:
| Maybe I'm missing something but what's the point?
| kissgyorgy wrote:
| I agree, some kind of analysis on the results would have been
| more interesting.
| _wldu wrote:
| Thanks. Someone had done a scan for security.txt files before
| but were unable to scan the entire Alexa Top 1 million
| websites. Only the top 1,000. I checked the top 1 million from
| a 1.5Mbit residential DSL line using Go routines.
|
| The blog post has more details and a short overview of the
| results. It's linked at the bottom of the repo. I was hoping to
| get feedback on the code.
|
| https://www.go350.com/posts/a-survey-of-security-dot-txt/
| reaperducer wrote:
| An aside, but it would be nice if the tech industry could move
| beyond three-character extensions. There's zero reason this
| couldn't be security.text.
|
| It's been at least 35 years since I first saw a non-three-
| character file name extension (Amiga 1000), there are probably
| older examples. Computers are supposed to work for people, not
| the other way around.
| ffpip wrote:
| Imagine a .JavaScriptObjectNotation instead of .json. Or
| .firefox-extension instead of .xpi
|
| There are only a few important extensions. just remember them.
|
| > Computers are supposed to work for people, not the other way
| around.
|
| Extensions are not for average computer users. They have icons
| and filenames. Windows also hides extensions by default.
| chrismorgan wrote:
| .JavaScriptObjectNotation would be terrible, because of its
| verbosity and because people call the language JSON: so .json
| is good.
|
| .xpi is a rather poor example: no one talks about XPInstall
| (and the majority of its surface area is now even obsolete),
| so .xpi is to most people completely meaningless unless they
| have encountered it before and know what it is. .firefox-
| extension would be a vast improvement over .xpi, because it
| says what it actually is.
| corty wrote:
| There actually was a short period where Windows 95 was new
| and had just introduced long filenames with long extensions
| to former DOS users. Then, for a short time, stuff like
| .SomeSoftwaresDocument was actually popular.
| throwaway201103 wrote:
| > Extensions are not for average computer users. They have
| icons and filenames. Windows also hides extensions by
| default.
|
| As far as I recall, extensions became a thing with DOS. They
| actually had meaning to the OS, e.g. naming a file .exe would
| make it executable, as there was no other concept of file
| ownership or permissions.
|
| In Unix/Linux systems, filename extensions have always been
| for the user. Before GUIs and icons, they were a convention
| that let the user know something about the file contents. As
| far as the operating system is concerned, "." is just another
| character in the name and extensions are meaningless. Note
| that most binary executables don't have an extension at all,
| and other files have more than one (e.g. .tar.gz)
| reaperducer wrote:
| _As far as I recall, extensions became a thing with DOS_
|
| CP/M had extensions in 1974, five years before DOS. There
| may be older examples, but that's the first operating
| system I ever used.
| throwaway201103 wrote:
| Makes sense. I think DOS took a lot of inspiration from
| CP/M.
| reaperducer wrote:
| _Or .firefox-extension instead of .xpi_
|
| That's an excellent suggestion. WTF does ".xpi" even mean to
| someone who just wants to install a browser extension?
| remexre wrote:
| JSON is bad, but as someone who doesn't know what an XPI is
| offhand, .firefox-extension seems kinda nice?
| tomc1985 wrote:
| Please, lets not make computing any more cavemanlike than it
| already it. Is a world of obnoxious push-button apps with zero
| options or customizability not enough for you? Learn to love
| the contours of what you got and it will serve you even better!
| ivanhoe wrote:
| Why? Is there a single person in Universe that would ever be
| looking at these files and not know that txt is short for text?
| Perhaps in some other cases it makes sense, and perhaps we
| don't even need extensions at all for many files, they can be
| misleading (not to mention that what 'text' means is just
| another convention). Microsoft even hides extensions from
| average users.
|
| But the common extensions that we all know by heart, why
| change, what's the gain?
| chrismorgan wrote:
| .txt is an ancient and extremely well-established extension,
| and so will not cause any trouble anywhere--it'll be mapped to
| text/plain in all standard servers, out of the box.
|
| .text is not a common extension. Some things know that it's
| text/plain (my Arch Linux /etc/mime.types and
| /etc/nginx/mime.types both do), but I expect some common server
| software won't handle it properly out of the box (haven't
| checked beyond nginx's mime.types), and common OSes won't have
| a handler for .text files set up (Windows, for example, comes
| with .text set to PerceivedType text like .txt has, so that
| it'll suggest the right sort of apps to open it, but it's still
| not hooked up to any app by default, unlike .txt which is "Text
| Document").
| reaperducer wrote:
| I stand by my previous statement. Computers should work for
| us. We shouldn't work for computers.
|
| Dump .txt for .text.
|
| Dump .jsn for .json.
|
| Keep .html, .jpeg, etc because they are abbreviations for
| standards.
|
| Sucks for Windows that it can't handle ".text" like other
| operating systems have since the 1980's.
|
| Dumping three-letter extensions will also help avoid all the
| extension namespace collisions that happen all the time.
| johnr2 wrote:
| >Computers should work for us. We shouldn't work for
| computers. Dump .txt for .text.
|
| If my computer expects me to type an extra letter every
| time I name a file it isn't working for me. The short
| extensions make for more efficient typing.
| Minor49er wrote:
| Speaking from my own experience, I've seen plenty of
| ".json" in the wild, but have never seen ".jsn".
| kiallmacinnes wrote:
| .json is roughly the same as .txt - an abbreviated version
| of "JavaScript Object Notation". We humans still have to do
| the translation to the final name.
|
| And, I've never seen ".jsn". I've seen ".json" hundreds of
| thousands of times, but never ".jsn".
| achillean wrote:
| Btw Shodan checks for the existence of the security.txt file and
| shows/ stores the information if it's available. Here's an
| example of how it looks:
|
| https://beta.shodan.io/host/172.217.31.43#securitytxt
|
| If you have a Shodan account you can also search the contents of
| the security.txt files using the "http.securitytxt" search
| filter. For example:
|
| https://beta.shodan.io/search?query=http.securitytxt%3Aconta...
| temp0826 wrote:
| Definitely felt like wishful thinking whenever I heard that
| security.txt was a thing.
|
| From the blog post-
|
| > Of the 666,771 most popular websites on the Alexa list, I found
| 2,884 security.txt files that were content-type "text/plain" and
| returned a HTTP 200 status code. Not all of these were valid
| security.txt files, but most were.
|
| Seems I was right?
| _wldu wrote:
| Yes, it's not as widely implemented as I expected. Adoption
| seems to drop quickly. Roughly 20% for the top 10 websites, 15%
| for the top 100 and about 10% for the top 1,000. It's downhill
| from there.
___________________________________________________________________
(page generated 2021-01-01 23:01 UTC)