[HN Gopher] What's the best lossless image format?
___________________________________________________________________
What's the best lossless image format?
Author : pmoriarty
Score : 113 points
Date : 2022-06-07 16:46 UTC (6 hours ago)
(HTM) web link (siipo.la)
(TXT) w3m dump (siipo.la)
| planetsprite wrote:
| JPEG 2000
| mg wrote:
| I would love to see an estimation of how small real world images
| would get if we could calculate their Kolmogorov complexity.
|
| Aka if we could find the shortest program that produces the image
| as output.
|
| Some images obviously would become very small. For example an
| image that is all blue, a spiral, the flag of the United States,
| the Mandelbrot set ...
|
| But what about the average portrait for example?
| xigoi wrote:
| I wonder how much could be achieved just using a vector format
| with basic programming constructs like for loops.
| SR2Z wrote:
| You cannot calculate Kolmogorov complexity in all cases because
| that would involve solving the halting problem.
|
| It is worth pointing out that certain formats like AVIF do sort
| of do this: they use a few different algorithms, see which
| combination of algo/block split removes the most entropy, and
| then compress _that_. In practice, this is pretty good for most
| non-noisy images.
| l33t2328 wrote:
| Wouldn't you want entropy to increase when you compress?
| 4pkjai wrote:
| Yes I've often thought of this too. Also a lot of images are
| screenshots of text or UIs.
| lajamerr wrote:
| JPEG XL is the standard to beat them all. Like how Opus became
| the definitive codec for lossy compression for the web.
|
| JPEG XL covers nearly every use case for images, even replacing
| GIFs with it's animation support.
|
| Existing JPEGs can also be re-encoded losslessly into JPEG XL
| format/data structure and get significant file size savings. This
| backwards compatiability is very important in transitioning to
| web to JPEG XL seamlessly.
|
| What I'm most excited about for JPEG XL is progressive decoding,
| I had been following the FLIF project for a while and was
| disappointed with it's lack of adoption. But it's future found
| itself in JPEG XL.
|
| Without a doubt 5 years now JPEG XL will become like Opus and set
| the standard for next generation image formats.
| pmoriarty wrote:
| Does lossless JPEG XL support 16-bit images?
| lajamerr wrote:
| JPEG XL supports 24-bit integers/32-bit floats per channel.
| MallocVoidstar wrote:
| I gave cjxl a 16-bit depth PNG, told it to use lossless
| encoding, and it gave me a 16-bit depth JXL. Seems so.
| rektide wrote:
| Also ticks the HDR box!
|
| As I comment elsewhere[1] it also has an experimental fast mode
| in tree that is crazy fast, using AVX2, NEON, or fallback.
|
| Hopefully Microsoft patenting the thing someone else invented
| (ANS)[2] doesn't gum this all up.
|
| [1] https://news.ycombinator.com/item?id=31658871
|
| [2] https://news.ycombinator.com/item?id=31657383
| Vladimof wrote:
| > Opus became the definitive codec for lossy compression for
| the web.
|
| never heard of Opus for images... must have been sleeping
| lajamerr wrote:
| :P I assumed anyone reading it would know Opus is an Audio-
| codec. Was comparing the paradigm of where I see image codecs
| going with a similar one that happened in audio.
| lifthrasiir wrote:
| They are also common in that they are effectively two
| different codecs combined.
| calibas wrote:
| Everything about JPEG XL sounds great, except that it takes
| about 34 times longer to encode an image compared to PNG.
|
| Edit: Apparently there's an experimental faster version:
| https://github.com/libjxl/libjxl/tree/main/experimental/fast...
| edflsafoiewq wrote:
| Depends, are you optimizing the PNG? Because cjxl is a hell
| of a lot faster for a much smaller file than zopflipng.
| klodolph wrote:
| zopflipng is a very aggressive form of optimization.
| Melatonic wrote:
| If it takes that much longer to encode then I am guessing it
| takes a lot longer to decode as well which I am going to
| guess makes it completely out for image sequences, film
| archiving, VFX work, high end photography, etc etc.
| mikewarot wrote:
| How well does JPEG XL deal with color spaces? I don't know much
| about them, but understand that not all color spaces are create
| equal, and data gets lost in conversions.
| lifthrasiir wrote:
| JPEG XL bitstreams always have a defined color space,
| optionally described by an embeded ICC profile. They are a
| part of bitstreams instead of containers [1] because they are
| considered necessary to reproduce the exact output. When
| converting from file formats with an ambiguous color space
| (e.g. PNG without color space information) cjxl assumes sRGB
| as far as I know; you can give any display color space to
| djxl.
|
| [1] A JPEG XL file can be a minimal bitstream or an ISOBMFF
| container, where the latter can contain the actual bitstream
| and additional metadata like Exif. The bitstream itself
| should be enough for decoding.
| Iv wrote:
| How does it relate to JPEG2000?
|
| I may soon have to recommend a lossy but potentially lossless
| image format for 3D medical images (with 16 bits channels),
| would JPEG XL fit this bill?
| andbberger wrote:
| you should use NIfTI, which is the de facto standard for 3D
| medical imaging.
|
| not compressed, but handles the voxel-image space transform
| which is far far more important and quite complicated.
|
| you can slap gzip on top of it and most libraries/FIJI will
| transparently decompress
| lajamerr wrote:
| The future isn't here yet unfortunately.
|
| I imagine the doctors would view these images in multitudes
| of ways(browsers, operating systems, image viewers) with
| varying level of support. So encoding in a standard that is
| yet to be entirely finalized and adopted is a bad move as of
| now.
|
| You'd be best off with going with JPEG2000 still if that's
| what you prefer or DICOM.
| Iv wrote:
| Oh adoption does not have to be that good. It is for
| researchers right now, to exchange results. So the only
| requirement is that it is possible to use it with a
| somewhat mature lib in python.
| detuur wrote:
| I absolutely recommend JXL. It is developed by the JPEG
| Group as a successor to JPEG2000 and performs better in
| nearly every metric (big exception currently being
| encoder speed). JPEG2000 isn't really widely supported,
| but does have many more libraries available.
|
| So unless you expect library availability and/or encoder
| speed to hurt adoption, go for JXL.
| amelius wrote:
| Isn't JPEG patent-encumbered technology?
| dharmab wrote:
| JPEG XL is a different technology than the original JPEG
| format. (JPEG is also the name of the organization that
| contributed to both formats.) JPEG legacy's patent expired in
| 2006 although lawsuits continued afterward. JPEG XL is
| royalty free.
| ComputerGuru wrote:
| I've read up about JPEG XL in the past but a question just
| occurred to me: is it just an image format or is it also a
| document format? Can it replace TIFF as a container for multi-
| page documents/images?
| lajamerr wrote:
| No, it's not a document format. JPEG XL won't replace TIFFs
| by and large because it has a different use case.
|
| TIFFs are more in-line with a combination of FLAC/WAV from
| the audio world. It's meant for raw data used in
| production/manipulation.
|
| As a professional you would work with TIFFs/RAW photo formats
| and then master it in JPEG XL for mass-consumption.
| Beltalowda wrote:
| > Without a doubt 5 years now JPEG XL will become like Opus and
| set the standard for next generation image formats.
|
| I wrote a little X11 screenshot tool last year; one of the
| reasons I wrote it is because I wanted to save images as WebP
| rather than PNG that most tools use, since it's quite a bit
| smaller. Not that I'm running out of disk space, but it just
| seemed nice; one of those "no, it's not really needed but it's
| fun and why not?" weekend projects.
|
| Anyhow, I discovered that loads of stuff doesn't support WebP.
| This was unexpected as it's been around for ages and supported
| by almost everything, but things like uploading to GitHub,
| Telegram, and many services doesn't work. It was disappointing
| and I had to add PNG support just for that.
|
| I fear that JPEG XL will be in a similar situation in the
| future, where loads of websites can and do use it for their
| images, but also loads of services don't support it so it's a
| lot less useful than it could/should be. I hope I'm wrong.
| Melatonic wrote:
| Unless nobody adopts it?
|
| EXR seems like it does all of the high end pro features of
| JpegXL and more (including 32 bit color). JPEGXL can do some
| odd stuff like animated gif type images but that seems more
| like a web end user feature than something professionals would
| care about. I can see it replacing Jpeg / GIF but I don't see
| this becoming an industry standard anytime soon.
| atoav wrote:
| > What is the best lossless image format?
|
| For which application? Otherwise you can only say " _depends_ ".
| Tiff is good if you want to save data fast to disk in Blender.
| EXR is great if you want 32bit color depth, etc.
| chungy wrote:
| > Tiff is good
|
| Mind that TIFF is only a container format that can contain any
| arbitrary codec you like; it can be uncompressed pixels, RLE
| compressed, it can be a PNG, a JPEG XL, even lossy JPEG or WebP
| if you'd like.
|
| Saying your image format is TIFF is roughly as useful and
| conveys as much information as saying your video format is MKV.
| ramraj07 wrote:
| In that regard lzw has been the standard lossless tiff
| compression format I've seen.
| atoav wrote:
| Good point. What I wanted to say is, that what is "the best"
| depends a lot on the application. Maybe you don't care about
| compression/size, but about write speed. Maybe read speed is
| something you care about, maybe you do actually care about
| size, but compatibility with certain clients is even more
| important, maybe you care a lot about how long decompression
| takes, how much strain it puts on the CPU or the memory, etc.
|
| So it is certainly not as simple as just going for the thing
| that produces the smallest file in some benchmark that may or
| may not fit your realworld data.
| re wrote:
| > For which application?
|
| For this article the context is for use in browsers, and the
| main metric being measured is compression ratio.
| bumper_crop wrote:
| The best lossless format is the one you can decode back to the
| original image. When evaluating them, there is an implicit
| assumption that the decode step will happen a short time later,
| but that's not always true. Will there be JPEG XL decoders
| commonly available in 40 years? Will the code even compile then?
| As a thought experiment, try to find the lossless encodings from
| 40 years ago and see if they can be faithfully reproduced. (or
| even 20 years ago, remember JPEG2000?)
|
| Framing best in terms of file size or encoding speed is a really
| use-specific framing, and not ideal for preservation.
| lifthrasiir wrote:
| Your concern is valid but misplaced, there is a reason that we
| have standards after all. The Library of Congress maintains
| file format evaluations for the purpose of preservation, and
| there is an (pretty favorable) entry for JPEG XL [1] as well.
| Not yet in public, but I'm also working on the very first
| reimplementation of JPEG XL from scratch and reporting any spec
| bugs from ISO/IEC 18181 (there are plenty!) and I expect there
| would be more implementations in the future.
|
| [1]
| https://www.loc.gov/preservation/digital/formats/fdd/fdd0005...
| Beltalowda wrote:
| > or even 20 years ago, remember JPEG2000?
|
| Open Source JPEG2000 libraries:
|
| - Grok, last commit 9 hours ago:
| https://github.com/GrokImageCompression/grok
|
| - JasPer, last commit 5 days ago: https://github.com/jasper-
| software/jasper
|
| - OpenJPEG, last commit 8 days ago:
| https://github.com/uclouvain/openjpeg
|
| I didn't try them, but I think you can probably still read your
| 20 year old JPEG2000 files today without too much problems.
| bragr wrote:
| This seems excessively handwringing about code rot. TIFF was
| introduced 35 years ago and still well supported. JPEG2000
| didn't become super widespread, but is still used in many
| places. Smart passports encode your passport photo in JPEG2000
| for example.
| rektide wrote:
| I wonder how fpng[1] would compare- a lossless speed optimized
| png compressor. Oh! The author Rich screenshoted[2] a really nice
| deck[3] that also boasts fpnge[4] and fjxl[5], both of which are
| even faster, both use AVX2 (or neon for FJXL).
|
| Notably fjxl is in the reference libjxl tree. I'm not sure what
| if any constraints fast mode has, what it takes to use. The
| article uses cjxl, which does not run the fjxl fast mode, and
| further, fast mode was added after this article was written[6].
|
| Other notes on jpeg-xl: it has progressive rendering. It has hdr!
|
| The deck on FJXL & FPNGE is epic. Thanks Quote Ok Image (QOI)
| format for setting the fire on this all!!
|
| [1] https://github.com/richgel999/fpng
|
| [2] https://twitter.com/richgel999/status/1485976101692358656
|
| [3] https://www.lucaversari.it/FJXL_and_FPNGE.pdf
|
| [4] https://github.com/veluca93/fpnge
|
| [5]
| https://github.com/libjxl/libjxl/tree/main/experimental/fast...
|
| [6] https://github.com/libjxl/libjxl/pull/1124
| aaaaaaaaaaab wrote:
| I want to see if they're really lossless. I know for a fact that
| PNG is. But last I checked HEIF didn't support lossless RGB
| images, only lossless YUV... So HEIF was unusable for bit-perfect
| reproduction of the original.
| heipei wrote:
| If your dataset allows it, I would look at pngquant for "lossy"
| compression of PNG images by means of color quantization:
| https://pngquant.org/
|
| This works absolute wonders if you are talking about images with
| a limited amount of colors and without many gradients. I use it
| to compress screenshots of websites, which, if you think about
| it, are mostly large blobs of uniform colors with hard edges. I
| also use it every time I want to include a screenshot from my
| desktop in some publication or email. The savings in filesize are
| too good to be true without any apparent loss of visual fidelity.
| mgdlbp wrote:
| Still hoping for vector desktop screenshots, now that modern
| GUIs are all vector-based.* 11 years on, gtk-vector-screenshot
| still stands alone, though we do now have a handful tools for
| rendering webpages to SVGs.
|
| *Add this to Category:Articles_with_obsolete_information:
| https://en.wikipedia.org/wiki/Vector-based_graphical_user_in...
| emeril wrote:
| For web graphics, isn't a mix of PNG/JPEG good enough generally
| speaking?
|
| I'd imagine the benefits of changing to something more
| modern/efficient aren't worth the cost/time involved other than
| in esoteric situations given that most end users have high
| bandwidth connections and bandwidth costs typically aren't too
| significant?
| pineconewarrior wrote:
| webp, webp2, and avif are absolutely worth using in modern web
| scenarios. It's even considered "poor practice" to serve legacy
| formats now.
| alx__ wrote:
| I wouldn't say it's "poor practice" to serve legacy formats.
| If you're using a setup to allow for fallbacks, that's the
| best practice. There are more device types some developers
| need to consider than the latest version of Chrome.
| Melatonic wrote:
| If you want something that is going to be around for awhile and
| has a LOT of features you probably do not need then you are going
| to want to go probably with a .DPX or .EXR which are the standard
| for the VFX industry as well as film in general. On the consumer
| side I guess adobe .DNG might not be bad either or for maximum
| compatibility but then maybe a Tiff.
|
| For video I personally like Cineform but both ProRes and DNxHD
| are more widely used
|
| EXR will get you full 32 bit float if needed
| detuur wrote:
| How do EXR and JXL compare to each other, feature-wise?
| Melatonic wrote:
| I have never used JXL honestly - no job I have had it was
| ever even considered. Given the Jpeg in the name I doubt any
| pro studio would consider it given the other options do
| everything needed. DPX is basically the older standard for
| film and EXR is the newer one.
|
| edit: Just looked at the specs and JpegXL is definitely
| marketed at least more as a web end viewing file type. It
| does higher color depths (up to 16 bit) which is pretty good
| but will not do 32 bit like EXR. Given that EXR is so widely
| used I would likely not consider anything else.
| theandrewbailey wrote:
| (2021)
|
| I wonder how AVIF and JPEG XL would stack up today.
| torquemodwanted wrote:
| There's been discussion lately whether or not JPEG XL adoption
| will be hindered by patents.
|
| https://patents.google.com/patent/US11234023B2/
|
| https://www.theregister.com/2022/02/17/microsoft_ans_patent/
| pm2222 wrote:
| https://bellard.org/bpg/ Not sure if this fits your bill.
| soperj wrote:
| SVG!
| micromacrofoot wrote:
| All graphics are images, but not all images are graphics
| mdp2021 wrote:
| In my days we would have said, not all images are vector
| based - some are raster based.
| cute_boi wrote:
| Technically ..., but practically you are correct.
| ptidhomme wrote:
| I think decompression times would be a relevant criteria, but
| they're surprisingly not there...
| laurent123456 wrote:
| I found that strange too. An image is encoded only once but
| decoded potentially millions of times.
| petesergeant wrote:
| Quite, and encoding speed, which is included, seems almost
| irrelevant
| bob1029 wrote:
| Encoding speed can be very important, especially if you are
| seeking to decode the resulting image as quickly as possible
| at all times. Certain image formats are wildly better real-
| time containers than others. Encoding speed is irrelevant
| unless it's very low or very high. Most people are thinking
| amortize the decode against the encode and don't even
| consider what would happen if you took it into the other
| extreme.
|
| Did you know that, with the correct SIMD code, you can encode
| a 1080p bitmap into a 4:4:4 JPEG within 10 milliseconds on a
| typical consumer PC? Encoding times like this open up an
| entire dimension of new possibilities.
|
| I do have a side project where I use libjpegturbo to
| synchronize a frame buffer between server & web client (i.e.
| 60 FPS real-time). HN may frown on using [m]jpeg for real-
| time applications today, but in my experience it has proven
| to be a very robust path. Bandwidth is really the _only_
| tradeoff (and it is a big one). This is the very last thing
| Netflix would want to do (short of sending RAW files to your
| TV), but it does provide some very compelling attributes in
| other areas such as streaming gaming.
|
| Our networks are only getting faster over time. Intraframe
| video compression techniques are really elegant in their
| simplicity and resilience to motion-induced artifacts (as
| seen with x264, et. al.). They also can provide lower latency
| and easier recovery for clients (since every frame is a
| keyframe).
|
| For real-time applications, I'd happily trade some extra bits
| per second to have a final result sooner. Especially if it
| looked better.
| [deleted]
| pmoriarty wrote:
| Why is encoding speed irrelevant?
|
| I'm currently in the process of scanning and archiving lots
| of photographs as high-resolution 16-bit PNG's and encoding
| speed is definitely a significant influence on how long this
| process takes.
| drblast wrote:
| Because it's a one-time cost and you can cheaply throw more
| CPU power at the problem, or just wait and then you're
| done.
|
| If you're doing this at such a scale that encoding speed
| matters (like if you're Instagram) the major concern would
| likely be cost, which you're going to save in bandwidth
| every time an image is viewed.
| pmoriarty wrote:
| _" you can cheaply throw more CPU power at the problem,
| or just wait and then you're done"_
|
| How am I going to "throw more CPU power at the problem"
| when I'm using a scanner hooked up to my laptop? Should I
| buy myself a new, more powerful laptop? That doesn't
| sound cheap.
|
| As for "just" waiting... my time is valuable, and I have
| better things to do with it than wait... and,
| unfortunately, the process of scanning and encoding needs
| a lot of babysitting from me, so I can't just dump a
| bunch of photos in my scanner and come back when it's all
| over if I want even half-decent results, not to mention
| archival quality images.
|
| In this real-world scenario, a quicker encoding speed
| would save me a lot of valuable time, and there's no
| getting around it short of hiring a skilled and trusted
| person to do it for me.
| Nullinker wrote:
| Like Tomovo above already said; your workflow would
| benefit from using an intermediate format that is faster
| to encode. You can then automate the conversion to the
| output format to occur unattended at a more convenient
| time.
|
| The use of intermediate formats is a well proven
| technique in video editing where encoding real time to
| AVC/HEVC at high quality is not possible. Codecs like
| ProRes are used that are much easier to encode to at the
| expense of storage space.
| tomovo wrote:
| Save uncompressed to a big cheap harddrive, batch
| compress to PNG overnight?
| throw10920 wrote:
| You missed the "almost".
|
| The vast majority of use-cases and users barely care about
| encoding performance. Your use is in the tiny minority.
| That isn't to say that you don't have a use-case, just that
| because of how rarely encoding speed matters, it truly is
| "almost irrelevant" for 99.99% of users 99.99% of the time.
|
| --------------
|
| It definitely _would_ be nice if JPEG-XL _also_ had good
| encoding performance, though. There 's an experimental fast
| encoder in development currently[1], although before that
| lands, it would be nice if your scanning tool supported a
| workflow somewhat like "scan in parallel with image
| encoding, present encoded images to user, and allow them to
| asynchronously go back and re-scan/encode/review pages that
| came out badly".
|
| [1] https://github.com/libjxl/libjxl/tree/main/experimental
| /fast...
| klodolph wrote:
| It's common enough to create an image on the fly and send
| it to a single user to be viewed once.
| Dylan16807 wrote:
| How many megabytes per second is your scanner feeding you?
| I'm very surprised that encoding a PNG is slower than
| scanning.
| pmoriarty wrote:
| It's not slower than scanning, but I still spend a
| significant amount of time waiting for the maximum
| compression PNG encoding to complete.
| pizza wrote:
| There is also a new experimental Fast Lossless mode in JPEG XL as
| well, but it was developed after this post was written
|
| https://github.com/libjxl/libjxl/tree/main/experimental/fast...
| TheCraiggers wrote:
| > but it was developed after this post was written
|
| The article literally compares it to others. It might not have
| been finalized at the time of writing, but it's not like the
| author was unaware of it.
| re wrote:
| I think you misunderstand pizza; the article compares JPEG
| XL, but it doesn't compare the new lossless coder
| implementation (fjxl/fast_lossless), which is a separate
| tool/binary from cjxl.
___________________________________________________________________
(page generated 2022-06-07 23:00 UTC)