Report of media files that weren't scraped?

Oct 9, 2014 at 3:32 AM
I like the log that gets generated when I perform a search through my whole TV Show library. It has great detail about issues it had trying to add a media file to the library, etc. But with a library of over 200 shows and 11,000 episodes, the quantity of failures to examine can be quite vast. It is very difficult for me to parse through the log to find those episodes that are remaining hidden from my library and XBMC.

The failures can exist for any number of reasons:
1) It's just a sample file.
2) The name can't be parsed in its current form.
3) TheTVDB doesn't have information on that season or episode

Is there some way to browse the failed files within the Media Companion interface? I would love to be able to see a big long list of orphaned or unscraped files that didn't make it into my library but that exist within my folder structure. Right now I just hunt and peck randomly and sometimes open a folder to discover 100 episodes just waiting to be fixed. Right now it's as if these files just simply don't exist.

Thanks for any and all help
Oct 9, 2014 at 3:42 AM
Hi Freebase1ca

Interestingly, this has been asked before, and looks like an option that may have to be added to Media Companion.

I also had a thought for the Log, that any failure to scrape, be at the beginning of the Log file, also with something like
---Using MC TVDB Scraper---
Starting Folder Scan

Scanning:  75  -  TV Shows
Found:  15  -  New episodes
Successful:  12  -  Episodes
Failed:  3  -   Episodes

Failed as follows:
Show:  Arrow
Episode -  S03E01 - blabla.mkv
Episode -  S03E02 - blabla.mkv

Show:  Hart Of Dixie
Episode -  S02E01 - blabla.mkv

Successful As follows:

bla bla bla
I feel that would be more informative to the user, and possibly easier to implement, than building a new "Search and Validate" with database routines.
Oct 9, 2014 at 2:29 PM
I like the sounds of that log. That would be useful indeed. Grouping by show would work just fine since bad shows are already highlighted in the GUI. I'm guessing this would highlight just the bad files hidden within valid shows? It might be nice to add to the end of the file name a brief reason for the failure.

Mind you, I haven't checked out how good the log looks if I rerun a search back to back. I would think the second search would find the unscraped files once again and little else - so it would predominantly include just failed episodes and be easier to analyse.

You may not be able to get away from building a new search and validate with database routines however. I saw another request (that sounded like it might be fulfilled) for the opportunity to add custom shows or episodes in the GUI. Of course the reason for that would be to provide a way of getting these unscraped files back into the library (whether it be an undocumented episode or some random extra). So it would make sense to attach that functionality to a list of unscraped files. I could picture the user selecting an unscraped file and having a few options available:

1) Perhaps allow them to rename the file (or set of files) so scraping works (probably not very useful functionality but perhaps desireable for some).
2) Show a list of available episodes from that show folder that haven't been previously scraped so the user can manually identify that file as being one of those episodes.
3) Allow the user to create a "Season 0" or some other "Extras" type entry.
4) Offer a default episode title based on the file name that can be overridden.
5) Allow the user to add some rudimentary metadata to force sort order or record first aired date, actors, etc.

Regarding #1... I'm using Media Companion to fix all my randomly sourced media so that XBMC will be happy with it and provide me with a nice rich library. MC's handling of the show selection and sourcing of artwork is wonderful. But I can't use Media Companion without also using other tools. The first tool I must use is a Bulk File Renaming tool that will fix a directory full of files with strange naming formats. It might be nice if some of this functionality was built into MC. If I come across a directory full of unscraped files I may realize that it's because no season was specified, but since I can determine that from the parent folder name, I am able to make the necessary changes.

As long as I am asking for renaming things... I like using the file auto-renaming feature that MC offers, but I think I would like it to go further. I would like it to maintain season folders as well.... In addition to a Bulk Renaming Tool I spend a lot of time in Windows Explorer getting the folder names just right and merging folders. Sometimes I'll have individual, strangely named folders for just one episode due to a torrent download or something. It would be great if MC while scraping would offer the option to hoof that over to an oppropriately named season folder like "Season 02" (maybe bring any accompanying srt file along too). It would also be nice to have the option to tell MC to just delete the original folder if it no longer contains a media file (willing to lose any other files like nzb. torrent, nfo, txt, etc.).

Sorry for the rambling, you just got my brain whirring :-)
Oct 9, 2014 at 8:03 PM
That's a long read, and I'm sure I've missed or not understood some, but I'll try to answer as follows.
  • I also use bulk renaming to fix up files, but only when I've downloaded Season's of episodes for a new show. I've never trusted Torrent's as it's the up-loader's choice how they set up the file-names. Usually they are alright. Sometimes I have to remove the Year from the front of the episode title, IE: Castle 2009.S01E01.hdtv.mkv, as this will throw Media Companion (Actually this throws XBMC also). MC sees this as Season 20, Episode 09.
    Dl via torrent of single episodes is usually risk free, but still, it is up to the user to make sure the filename is valid enough for MC (or XBMC) to scrape correctly.
    With tools already available to sort these issues, I don't feel that duplicating these tools would be very constructive, not to mention the time involved. One I know of is: TheRenamer
  • The TV function, Display Missing Episodes might be of assistance here. Enabling this will download complete list of episodes, scan through the show's and fill in the missing episodes to be easily found. Missing Episodes are colour-coded so you can note which missing have been aired (blue), which are yet to be aired (red). Also, the filters at the bottom of the TV list, you can select to display missed episode, Display missed aired episodes only.
  • While allowing users to create special episodes is a good idea, there are problems associated with this. As we scrape from TheTvDb site, if a user Batch rescrapes, and the special, user created episode, isn't present, then the user will get an error. Also, if there is an existing special episode, it will overwrite the user's created episode.
    It would not be difficult to create a custom nfo creator, but I'd be inclined to have this as a Stand-alone function, where the user select's a video file, and fills in in the fields, and a nfo file is created. Then in Media Companion, you'd refresh the show, to have this added to the Tree.
    There also is a way to have an excluded folder in Media Companion, where video files and nfo's could be stored, and Media Companion would ignore these. This would be an ideal location to keep User created special episodes, where Batch rescrape wizard would bypass this folder. See General Preferences -> General Tab, bottom right. Enter the folder's name, not the path. eg. "MySpecials". Any episode within this folder would be ignored by MC. And you can have a "MySpecials" folder in each Tv Show's folder.
  • No. 4, not sure what your meaning.
  • No. 5. If a Stand-alone editor was created with Media Companion, then this would allow the same control?? Might be wrong about your no.5.
Your last paragraph (second to last actually). Have you had a look at Sickbeard?? Now Sickbeard is predominantly for usenet control, as I use it, but there is a flavour that supports torrents. Sickbeard can monitor a Folder, Move a Episode to the relevant Show, create a Season02 folder, and then delete the original download folder. It supports subtitle file moving also. (actually, Sickbeard also can create nfo's, but I found the nfo's seriously lacking in information, and missing Media-Stream info).

What I have, is a script that runs mc_com.exe -e . This is the Command-line program that ships with Media companion. See Wiki doc here. Once Sickbeard has moved, renamed etc, sickbeard then calls my script, which runs mc_com.exe -e which then scans all my Tv Shows, and scrapes any new episodes found. It also adds these to the tvcache.xml file, so when Media Companion is run, newly scraped episodes are already present in the Treeview.

Hope that gives some insight, and possible points for discussion.
Oct 12, 2014 at 5:55 AM
Thanks for such a detailed response! Sorry I've taken so long to reply...

Fair enough about the bulk renaming. I'll stick with other tools for that. (The one I use is "Bulk Rename Utility" - not very fancy, but it gets the job done.)

Displaying missing episodes isn't important to me. I use Sickbeard for that. But knowing that I have files that weren't scraped is very valuable indeed (it means they will also be hidden to XBMC.)

Special episode management and nfos does sound complicated. I'd need to know a lot nore about the subject to respond in any useful way. I think the reason people (and myself) want this feature is because we have episodes of shows we want to watch but they aren't appearing in XBMC. For example, I record off OTA in Canada. MCEBuddy is software that automatically uses TheTVDB data to name the shows correctly and through them into my library where MC and XBMC finds it and processes it, etc. The problem is that some of the Canadian shows are not popular enough for theTVDB so don't get and season or episode info. They then become invisible. We would love to have a tool that could invite some of this info so that XBMC would be happy. In other cases TheTVDB seems to have laggy updates so we are trying to handle episodes that have not yet been populated so the episode information can't be found. It would be nice to be able to handle this kind of situation automatically with some kind of custom info.

Perhaps we could mark shows in the MC interface that we know are problematic in TheTVDB and MC would be willing to maintain its own fictional episode labels.

I'm trying to understand some of the issues you were mentioning with nfo creation/maintenance... I'm not sure I get it. But couldn't you add a notation within a custom nfo that identifies it as custom so that when a rescrape is occurring MC is clever enough to preserve them from being overwritten, etc.

Don't worry about #4. It was a small unimportant implementation suggestion. Basically the idea is that if a user needs a custom episode entry for a file that didn't scrape, MC could prepopulate some metadata with a best guess based on info in the filename.

I love Sickbeard :-) I just use it for newsgroups though. Back catalog never seems to work in newsgroups so then I resort to Torrents. I also pull off OTA for obscure stuff or things I want immediately. I do rely heavily on SB's ability to maintain folders. But since it only sees my newsgroup stuff, it doesn't see everything. MC is the only tool in my arsenal that sees it all because it is regularly scanning my whole library.

I'll have to start playing with the commandline switches - they might be useful. Although, I don't really have many complaints with what SB does on its own.

BTW, I started playing with looking at the log. It may not be all that necessary to change its structure. By performing a search twice in a row, I can handle the good episodes with the first search and then the second search only has failed files left that I can then try to clean, etc.
Nov 25, 2014 at 10:36 PM
Edited Nov 25, 2014 at 10:46 PM
Hi all

This is something I've been thinking about too. A log like vbat99's example or some search and find feature would be great, but until that time, how about using the 'Missing Movie Scanner' add-on for XBMC? I've literally just found out about this tonight. Gave it a quick run over my movies library (1500 plus), and worked well. I knew there was one movie not in the library yet, and it found it no problem. Have yet to try it with my TV Series, but give it a try yourself and see what you think.

[UPDATE!] As soon as I posted this, I realized my 'DOH!' moment (I told you I was special vbat99!). Sorry freebase1ca! At least you have a way of finding your missing movies now :) But in my defense, if this is possible for movies, I'm sure it must be possible for TV series, so maybe a similar scanner for TV series is out there.

I should really step away from the computer and go to bed now.
Nov 26, 2014 at 9:00 PM
What I'd be interested to know is what was the one movie, and way was it not detected by Media Companion???
Nov 27, 2014 at 6:18 PM
Don't worry, no fault with MC. :) It was just a new movie not yet scraped by Kodi in order to test the scanner.