Memory leaks when viewing NEF files

Submitted by Yarilo on Sun, 11/12/2017 - 06:47
Forums: 
Bug reports

System: Windows 10 Pro x64 (1703.15063.674), licence bought from Microsoft Store.
Program version: 1.4.3.1149, trial edition.
Problem: Memory leaks when viewing NEF files.
Description: After start, program's commit size is ~500MB. After viewing 50 NEF files from Nikon D750 (just viewing, no other actions) program's commit size is ~3900 MB. Consuming of commit memory never stops until system limit is reached. My system have limit of 20GB and it could be reached easily. In comparison Nikon ViewNX 2 consumes around 400MB of commit memory regardless of viewed files amount.
Screenshots of Task Manager after program launch and after viewing 50 files:

lexa

Sun, 11/12/2017 - 07:27

Thank you for the report, will try to reproduce.

--

Alex Tutubalin/FastRawViewer team

lexa

Sun, 11/12/2017 - 07:58

Dear Sir:



Unfortunately, unable to reproduce, with lot of D750 (and other cameras too) files memory footprint is within expected bounds (1100-1500Mb, may be slightly larger with 50-100Mpix files).



What to try:

 1) Upgrade to latest release candidate: https://www.fastrawviewer.com/blog/FastRawViewer-1-4-4-release-candidate

Not much changed in respect of memory handling, but it is much easier for us to test current version.



  2) Try to switch video mode in Preferences - GPU Processing - Graphics Engine (try other variants, different from your current one).



If #2 will help, it will look like graphics resources (textures) leak and may be due to bugs in video driver. Meanwhile, what videocard/drivers do you use?

--

Alex Tutubalin/FastRawViewer team

Yarilo

Sun, 11/12/2017 - 08:17

With RC version memory consumption is even bigger. 1000 MB after start, 4120 MB after 50 files. All three graphic options (DX9-DX11-OGL) produce the same result.
 
Information about graphic card:
Graphics Chipset: AMD Radeon HD 6900 Series (HD6930)
Device ID: 671F
BIOS Version: 013.012.000.019
BIOS Part Number: 113-SS3G01-001
Memory Size: 1024 MB
Memory Type: GDDR5
Core Clock in MHz: 750 MHz
Memory Clock in MHz: 1200 MHz
Driver Packaging Version: 15.20.1062.1004-150803a1-187674C
Catalyst Version: 15.7.1
2D Driver Version: 8.01.01.1500
Direct3D Version: 9.14.10.01128
OpenGL Version: 6.14.10.13399
AMD Catalyst Control Center Version: 2015.1104.1643.30033
 
This driver is current driver for AMD Radeon HD6930, CCC found no available updates.

lexa

Sun, 11/12/2017 - 08:39

Thank you for the information.

Will try to reproduce on AMD-GPU computer.

 

--

Alex Tutubalin/FastRawViewer team

lexa

Sun, 11/12/2017 - 08:41

Could you please do this small probe:



- create folder with only two raws

- open one of this files using FRV

- and navigate back/forward between these files several times (in single-view mode, using Space/Backspace or arrows).

 

The question is: will memory footprint grow in a such limited case.

--

Alex Tutubalin/FastRawViewer team

Yarilo

Sun, 11/12/2017 - 08:47

Tried on a folder with 5 NEF files. After start - 370 MB of commit memory consumed. Switching between two files increases memory consumption to 600MB, but regardless of the switches count, memory consumption doesn't grow beyond 600MB.

lexa

Sun, 11/12/2017 - 09:08

OK, that means that for already cached files there is no memory leak (incl. graphics resources), OK.



Next try: please  try folder with files count slightly larger than Preferences - Performance - Decoded RAW cache size (standard value is 12, so please try 15 files of so if you have not changed this parameter). Again, start fresh FRV and use back-forward navigation.

 

--

Alex Tutubalin/FastRawViewer team

Yarilo

Sun, 11/12/2017 - 09:26

Folder with 16 files. Initial consumption - 370MB. After first round (every file viewed from 1 to 16) memory consumption - 600MB. And every 3rd-4th full round (from 1 to 16) raises consumption on about 0.5MB. During switching between neighbour files there are small changes in consumption towards both sides with accidential spikes to 650-665MB. It seems that spikes are also growing bigger with increased round numbers.

lexa

Sun, 11/12/2017 - 09:52

OK,  again no leaks for small number of files (but larger than raw cache size, so it goes out and in cache)



There are only two remaining suspects for high memory usage

 a) thumbnails

 b) file metadata

Standard thumnail pair (one for filmstrip, one for grid) is ~300kb unless you've increased thumbnail size in filmstrip/grid settings.

Thumbnail memory limit is 500Mb in standard settings (Preferences - Performance - Thumbnails), that is limit for formal thumbnail bytes count, real memory may be higher due to memory fragmentation.  You may try to decrease this setting.

Standard file metadata size is very small, several kilobytes per file, so standard limit for metadata cache (Preferences - Performance - Thumbnail cache - Metadata cache size, standard value is 10000) will not result in large memory usage.

If your files contains large XMP blocks or large XMP sidecars (generated by Adobe or other software), memory usage may be higher. Try to decrease Metadata cache size to lower value.

Also, if you're working with very large folders and use metadata sorting (e.g. sort by Exif date) or filtering (by date, or by rating/label), the metadata for all files in folder to be read.

--

Alex Tutubalin/FastRawViewer team

Yarilo

Sun, 11/12/2017 - 10:21

I realised now that I was doing checks with files from Nikon D200. And they aren't afected by this issue. Now I rechecked all tests 2-15-104 files with cache size for files = 12 and methadata chache = 5000 and without sidecars at all. 2 files - no memory leaks, 15 files - memory leaks rapidly. So this issue is definately connected to Nikon D750 files. I also made a test for folder with around 400 files from Olympus E-P2 with Adobe sidecars - no memory leaks.

lexa

Sun, 11/12/2017 - 10:29

Could you provide 1-2 files for our testing?

Upload it somewhere (Dropbox, Google drive, WeTransfer/free option) and send the link to support@fastrawviewer.com

--

Alex Tutubalin/FastRawViewer team

lexa

Sun, 11/12/2017 - 10:32

Followup: if you have XMP sidecar files for these NEFs, please upload it too.

--

Alex Tutubalin/FastRawViewer team

Yarilo

Sun, 11/12/2017 - 10:57

https://1drv.ms/f/s!Av6upJkQNcGmvqBwTiZY9I40VkYkkg
This is the link. There are three folders - 2, 15 and NEF. 2 and 15 are from one set of files, NEF from other set. But both sets are from the same D750. And strangely enough files in NEF folder aren't causing memory leak. So issue is even trickier.

lexa

Sun, 11/12/2017 - 11:37

Thanks, issue confirmed, FRV really have large leak with 15/* files.

I'll reply again in this thread when we'll have fixed version (it may take several days or so)

 

--

Alex Tutubalin/FastRawViewer team

lexa

Sun, 11/12/2017 - 11:42

Followup: files downloaded, you may delete them to clean up shared space.

--

Alex Tutubalin/FastRawViewer team

lexa

Mon, 11/13/2017 - 01:20

Fixed in 1.4.4-1191, available at: https://www.fastrawviewer.com/blog/FastRawViewer-1-4-4-release-candidate

The problem was with handling RAW files if embedded JPEG orientation does not match RAW orientation. That does not occur in from-camera RAWs (with very few exceptions), but may occur if RAW was modified by other software (Nikon View NX in current case).

--

Alex Tutubalin/FastRawViewer team

Yarilo

Mon, 11/13/2017 - 05:20

I confirm that the problem is fixed in 1.4.4-1191, thanks!

Add new comment