This project has moved. For the latest updates, please go here.

Refresh DB is slowing the whole computer down

May 23 at 7:47 PM
Hi,

I have a strange problem with the newest versoin of MC (MC3.675b) on a Win7 machine.

I have a 400+ TV series included via a SMB share from a NAS on a Gigabit LAN. I used to be able to add new series, search for new episodes and refresh the whole DB with a sensible run time (a few minutes for a refresh of all 400+ series).

With that newest version I noticed that adding new series, scraping and looking for new episodes remains a rather quick process but refreshing the DB slows the computer down to a crawl, MC freezes with around 250/400 series refreshed and finally crashes.

It looks like a memory leaking problem as the used RAM spikes during this process - it kind of just uses up all the free memory that is available form the system.

Does anyone else have this problem? Is there any way I can debug this further or in a more helpful way?
Coordinator
May 23 at 9:49 PM
How much ram? 32 or 64 bit OS? 32 or 64 bit Media Companion?
Not that the above should matter, but good to get a base image.

So, 400+ Tv Series, and what volume of Episodes....as a guess...

Doing a refresh on that many series and episodes, yes, MC will bloat on Ram as all the scanned nfo's get stored to memory before being saved to cache.
And I understand there could/would be some areas that are memory leaks.

Just that I don't have that volume of Series/Episode that I can test with.

btw, you mention this issue with latest release 3.675b. What version were you running before by chance? I could check what changes were made that might affect this. If it is more than half a dozen version of MC, I may not be able to find whats causing the issue, but never know.
May 23 at 10:29 PM
I have Win 7 64bit and I am running the 64 bit version of MC.

I upgraded from MC3.670b (64bit aswell).

My system has 24GB RAM and if I run nothing but MC it will take 23GB of those 24GB available.

There are app. 15,000 episodes in my collection.

I just checked and there are app. 16k nfo files throughout all directories. A quick 'find -name '*.nfo' -type f -exec du -bc {} + | grep total$ | cut -f1 | awk '{ total += $1 }; END { print total }'' on the shell from the root dir tells me that those files add up to app. 2GB in size. So even if MC had to load all of them into the RAM there should be plenty of RAM available. In fact I think that was the behaviour before the upgrade.
Coordinator
May 23 at 10:42 PM
Thanks for the info.

Yep, you shouldn't be running out of Ram at all.

I wonder, since your so adapt at finding data, what would be the chance of zipping all those nfo's up, with folder structure, uploading somewhere and pasting a link.
It would mean I can create dummy episode files and then test in my Dev build, find the leak and get it fixed.

??
May 23 at 11:29 PM
Sure thing: https://dl.dropboxusercontent.com/u/688389/just_nfos.tar

Maybe you could let me know when you have it downloaded so I can take it down again.

Oh, and let me know if I can give you some more information.
Coordinator
May 24 at 12:01 AM
Yikes, thought 2GB of nfo's would have compressed down much further.

Thanks for that, downloading now.
Coordinator
May 24 at 1:47 AM
Tested back to MC 3.670b x64, and same thing is happening. On my 8GB system, filling up the ram very quickly.

And I believe I know why.
During a refresh we are reading each nfo, and applying the full data to the Treeview.
Case in point, 59 Series, 1963 episodes, fills up 3.3GB of Ram.

If this data is in the Cache file, then MC starts and loads only the cache data into the Treeview.
Same 59 series and 1963 episodes, and MC fills up 155MB of Ram.

Then, as a user selects episodes, their nfo is read and added to the Treeview, so slowly the ram usage increases. But very slowly. Clicking on eight episodes increase ram usage to 174MB.

Alrighty, so I need to get MC to only apply cache data types from the nfo files to the Treeview during Refresh, not complete data. This will reduce memory consumption considerably.
Hopefully with a side-effect of slight speed boost.

This isn't gonna be easy, but I believe it is possible.

Will keep you informed.
May 24 at 2:33 AM
Thank you for checking this.

If I understand the process correctly, wouldn't it be within the spirit of a "refresh" to read the nfo files (but not load the full data into memory) and check whether the data, that the nfo describes is already in the cached data. If it's not in the cache file, add it and move on (again, without loading full data into memory) - if it is already in the cache file, just move on. After a complete run through of this, you would end up with a "synched" cache file (or in other words, a refreshed one) and not much memory usage (you did not load full data sets). Then, when you reload the treeview in the UI panel, you display the cached data and everything should work as you describe it (reading the nfo into memory on the fly while the user clicks through the treeview). That should minimize loading times and memory usage, no?

Thank you for testing this so quickly.
Coordinator
May 24 at 4:01 AM
Actually it would take longer to check each tag if it needs to be updated, than to just update it.
And we need to check if the data in cache hasn't been removed, ie, episode nfo deleted.

I've worked out a method to improve refreshing all, and loaded 403 Series with 15826 Episodes.
Peak ram used, 1.03GB.

Which is good, as when I closed and re-opened MC so it loaded the cache file, total ram sat at 668MB.

But, I did skip a section in the code in regards to missing episodes, so once I figure a way to satisfy that code, I'll post up a test build for you to try. Might be later this evening, er, my time. Will try...
Coordinator
May 24 at 7:12 AM
Here you go,

MC 3.675c x64 Test Build

Extract to a new folder, copy your Settings folder from previous MC into this new folder.
Start and do a Refresh All in TV section.

Cheers
Marked as answer by the_rob on 5/24/2017 at 10:32 AM
May 24 at 8:21 AM
Cheers!

That works perfectly. Memory usage never exceeded 6GB and it did not take all that long to refresh the whole library.

The only thing I noticed is that the dialog "Loading Cache..." after the whole refresh process did not close. I was waiting and thought maybe the cache
was still being loaded. But I could use the UI and everything seemed normal (famous last words), so it might just be a UI "bug".

Thank you so much for fixing this or rather change an integral part just because I had a problem.

10/10 - would complain again.
Coordinator
May 24 at 6:11 PM
No problem, it was an issue that wouldn't be found with small test files. Thanks for advising and giving me copies of the files so I could reproduce.

Seriously, I do appreciate any comments on what might be a bug, or issue, else we don't know, or will just accept it as the norm.

The Loading Cache dialog, is that the pop-up windows for Refreshing Series? as I don't see any text saying "Loading Cache..." except on the initial loading Splash window.
May 24 at 11:11 PM
vbat99 wrote:
The Loading Cache dialog, is that the pop-up windows for Refreshing Series? as I don't see any text saying "Loading Cache..." except on the initial loading Splash window.
Yes, the green dialog that goes through all directories. After the refresh (UI treeview was visible again) it stayed on screen and displayed "Loading Cache...". I have not reproduced that yet so it could have been an unrelated issue. I just noticed it and thought I mention it.
Coordinator
May 25 at 12:57 AM
Yep, found it in the code. Must have been a one-off glitch.