[HN Gopher] How Photoshop solved working with files larger than ...
___________________________________________________________________
How Photoshop solved working with files larger than can fit into
memory
Author : thunderbong
Score : 93 points
Date : 2023-06-05 18:10 UTC (4 hours ago)
(HTM) web link (developer.chrome.com)
(TXT) w3m dump (developer.chrome.com)
| tabtab wrote:
| File buffering RAM and/or arrays was invented long before Adobe.
| The tricky part is probably optimizing the image processing
| algorithms to fit the filing method(s). For example, if images
| are split into "tile files", and a given algorithm requires
| referencing too many files/tiles at a time, creating too much
| file I/O, then it probably needs to be reworked into a tile-
| friendly algorithm.
| mannyv wrote:
| I thought this was pretty well-known. Adobe has been running
| their own VM engine for a long time now. I'm not sure which
| Photoshop first shipped with it, but given how pretty much
| everyone in prepress used it it's likely since at least Photoshop
| 3.
| jdmoreira wrote:
| Photoshop discovered mmap?
| mike_hock wrote:
| No, we've (collectively) discovered that we have a lot of
| mature technologies (like mmap), mature platforms and extremely
| fast hardware, so we created a crippled software development
| platform on a bloated layout engine so people can get creative
| again hacking around the arbitrary constraints and design flaws
| of the platform to solve problems that had been solved 40 years
| ago.
| wongarsu wrote:
| Except when exporting files, where it still has weird size
| restrictions (somewhere around 10k pixels per side)
| jsheard wrote:
| This is about the web version of Photoshop, there is no mmap
| there.
| [deleted]
| villgax wrote:
| Pandas has to deal with the same crap, thankfully Dask & other
| out of core packages fix the issues
| jval43 wrote:
| Now do iOS. Content-aware fill still fails with OOM on iPad Pro,
| yet works fine on older Macs with less processing power.
| [deleted]
| evmar wrote:
| I worked on some memory stuff for Figma, a similar app, though
| it's surely changed since I did.
|
| One interesting thing about Photoshop's perspective is that it's
| fundamentally about "files", which is like a single bytestream
| that contains all the picture and which must be transferred to
| the browser to work on. A Figma document also can refer to
| images, but in a different way. The sum of the document data
| (like layout) and the image pixels can be extremely large, like
| gigabytes, but you can transfer those two things separately from
| server to client.
| vlovich123 wrote:
| Why doesn't WASM offer a 64-bit mode?
| jsheard wrote:
| It's in the works: https://github.com/WebAssembly/memory64
|
| Starting with 32bit had some performance advantages because
| 64bit runtimes can use virtual memory shenanigans to implement
| bounds checking with zero overhead. In wasm64 they'll have to
| do explicit bounds checking instead.
| zamadatix wrote:
| It's being worked on, it's currently available for development
| testing in builds of Chrome and Firefox.
| RcouF1uZ4gsC wrote:
| One the one hand I like 64bits from a technical perspective
| but, on the other hand, I am not so sure I want to live in a
| world where a single web app needs to access more than 4 GB of
| address space.
| bonestamp2 wrote:
| But if the need occurs, you'd rather be forced to
| download/install an app rather than use a web app? This seems
| short sighted, especially for occasional tasks that come up
| where I'd like to just bang it out once with a webapp than
| install something.
| throwawayai2 wrote:
| Now if only Figma could catch up to Adobe...
| aleph_minus_one wrote:
| It perhaps will soon:
|
| - https://www.cnbc.com/2022/09/15/adobe-to-acquire-design-
| plat...
|
| - https://www.theverge.com/2022/9/15/23354532/adobe-figma-
| acqu...
| ChrisMarshallNY wrote:
| We were doing that, with the software (scanner drivers and image
| processing -BIG files) my team wrote, decades ago.
|
| It's very similar to VM. We paged and indexed everything.
| interroboink wrote:
| It's a bit of a roundabout article, given that it's ostensibly
| talking about the Web version of Photoshop. Basically, Adobe
| already solved the problem (for decades, now) with a traditional
| non-web application by using the filesystem. And the new bit here
| is that Chrome offers a filesystem for web apps to use [1], so
| from the web app's point of view, it can still use the same
| essential solution that the native app would.
|
| It doesn't really go into the guts of how PS actually manages all
| that data in much detail. Just "it goes to the filesystem in a
| smart way."
|
| [1] The "origin private file system":
| https://developer.chrome.com/articles/file-system-access/#ac...
| kevingadd wrote:
| It's unfortunate that there's still no mmap equivalent for I/O
| in the browser, so you can't do demand paging or anything like
| that. Nothing approaching zero-copy either, you're typically
| looking at two copies or more. At least the OS page cache will
| probably help you out.
| PaulHoule wrote:
| ... application-based virtual memory has been a thing since
| CICS on mainframes back in the 1970s.
| keville wrote:
| Photoshop wasn't the one to solve this, as others have mentioned.
|
| Adobe likely owns the xRes intellectual property, though:
|
| https://en.wikipedia.org/wiki/Macromedia_xRes
| awinter-py wrote:
| some sort of spinning metal thing with platters and read heads
| and magnets and stuff?
___________________________________________________________________
(page generated 2023-06-05 23:01 UTC)