This project has moved. For the latest updates, please go here.

Scraping directly or importing into SQL dbase?

Jul 9, 2011 at 5:00 AM

I somewhat recently migrated my configuration to the centralized SQL network database configuration.

I created a script that would port some information directly via SQL so I would have updates manifest directly.  IE if something is scraped in XBMC and you update it, it won't rescrape that.

I've never had much luck with the export/imports of the library info.

Would others be interesting in getting MC to have functionality to directly update the SQL dbase?

Currently I push back runtimes and ratings votes...etc sometimes if you scrape too early this data is available or things like # of votes changes which affects the weight of ratings IMO a 9 from 15 votes vs an 8 from 75k votes

I would share my script if it could be helpful, but its kinda a hack job w/ VBS.

Developer
Jul 9, 2011 at 5:07 AM
Ya I'm very interested, a lot of my inheritance design is aimed at
that (but gota get that finished first). Gota get a lot of things
fixed before a major feature like that would be possible (corrupting a
bunch of Nfo files is one thing, screwing the whole DB is another) but
we seem to be moving quickly so maybe not that far.
Jul 9, 2011 at 5:24 AM
Yep, this is definitely a gaol we will get to.
From: "Playos" <notifications@codeplex.com>
Date: 8 Jul 2011 22:07:12 -0700
To: <merlinsgarden@bigpond.com>
ReplyTo: mediacompanion@discussions.codeplex.com
Subject: Re: Scraping directly or importing into SQL dbase? [mediacompanion:264502]

From: Playos

Ya I'm very interested, a lot of my inheritance design is aimed at
that (but gota get that finished first). Gota get a lot of things
fixed before a major feature like that would be possible (corrupting a
bunch of Nfo files is one thing, screwing the whole DB is another) but
we seem to be moving quickly so maybe not that far.
Coordinator
Jul 12, 2011 at 11:44 PM

Yeah, I am most interested also; my main motivation to get involved in Media Companion was to use it to provide info to a CMS-powered website! That has fallen to the way-side for now, but my hopes and dreams are still alive!!

I have seen (I think, can't find it now) a specific request for MC to FTP directly any changes to a server, and ultimately, I reckon that is maybe what people are wanting(?).

Developer
Jul 13, 2011 at 1:08 AM
I'd think it would be something more along the lines of directly
editing SQLite and MySql databases XBMC uses. Imagine a world where
you never have to "Update Library" or "Scan for New Content"...
Jul 13, 2011 at 1:31 AM

Yep I think so also, just looking at http://wiki.xbmc.org/index.php?title=The_XBMC_Database , doesn't look too different from how we are already storing the data. It should just be a matter of converting our data to XBMC's & updating the database using vb net's sql commands....

 

Developer
Jul 13, 2011 at 1:41 AM
It's MySql and SQLite... so it's not going to be that simple, external
libraries for both are avalible and well supported though.

I was thinking (work is getting so boring and I just want to get back
to coding so bad right now) that it wouldn't be that much overhead to
direct writing from the ProtoProperty, since it's already an interum
layer between data and application. It could be implimented as an a
valve setup (user selected one destination and the data is just pushed
that way) or as a sync write system (where the data is written to the
Nfo DOM and the database at the same time).

Defently complicated, but if it can be implimented in the underlying
Proto classes in an efficent way, it would be the easiest way to
impliment it without doing some serious interface and interop work. On
the plus side, if we make it work both ways (pulling data from the
XBMC database) it would be pretty slick for alot of other uses.
Jul 13, 2011 at 2:16 AM

Its never as easy as it first seems :) There's a bit of logic overhead too. XBMC has other table rows that need to be created/edited/deleted when adding/deleting movies & or shows.....such as bookmark (which I assume needs to exist when a new movie is added or perhaps xbmc creates it when it creates a book mark) - Anyway it won't be straight forward, but still possible....

We just need to get MC more stable first.....

Jul 13, 2011 at 2:17 AM

Right,  my main reasons for investing the time for my script was to preserve the WATCHED status, since this isn't used in the NFOs perse anymore and directly read from the database.  So if you need to rebuild/reslam or otherwise lose your stuff you can't migrate this information easily. In addition just getting rescraped information updated would be nice as 'update library' doesnt rescrape information and nor would i want to rescrape / rebuild my entire library needlessly.  It would be nice at a miminum to be able to force push changes from MC into XBMC if not truly 'sync'

Anyways I definitely think its a nice feature to add in someday.

Currently my script reads from the MC xmls // NFOs and directly updates some of the key tables in the SQL dbase I have.  I would like it to direct update the images for fanart / posters / thumbs, but can't figure the hashing and I think some of that stuff is still cached locally.

I run it nightly at 3am and push to the dbase, and i'm sure my sql queries are clunky but it works. :/

I just have never had any luck w/ the import / export feature directly in XBMC; its certainly possible that i'm doing something wrong there, I think this would be faster too, then scraping.

Coordinator
Jul 13, 2011 at 2:44 AM

Ah, yes, the 'watched' status - that has briefly passed thru my head recently when I upgraded my HTPC to Win7 finally. I wanted a fresh install of XBMC but it did mean losing all the 'watched' statii (new word, I just made it up).

This all sounds a bit complicated to me, so I'll just leave this to you guys! The web db thing I was talking about seems to be a totally different feature, but may leverage some functionality introduced here!

Jul 13, 2011 at 2:54 AM
Edited Jul 13, 2011 at 2:57 AM
StormyKnight wrote:

Its never as easy as it first seems :) There's a bit of logic overhead too. XBMC has other table rows that need to be created/edited/deleted when adding/deleting movies & or shows.....such as bookmark (which I assume needs to exist when a new movie is added or perhaps xbmc creates it when it creates a book mark) - Anyway it won't be straight forward, but still possible....

We just need to get MC more stable first.....

oh sure I'm just happy that this MC OS project kept going.  I saw all the offers on the old forums but must have missed the link so thought it just died out.I think EmberMedia may assert to do this but what I saw failed to impress me of the basics let alone sync with a database.  I'm as loyal to MC as I am to XBMC. My script is far from elegant, but like I said, seems to do the job well enough for me in the interim.  Just thought I'd see if I could get it on you, smart guys' list for the special bangs/whistles when the time comes or to capture in the current refinements to make easier on the smart devs whom may work on it later in the 'pipeline'.

When someone offers me a bag of skittles I just say 'thank you'; I don't whine about the yellow ones ;)

Jul 13, 2011 at 7:48 AM

Thanks transcender, I wouldn't mind having a look at the script, as it would give us a basis of how you actually gather the data & get the data updated...

We may be able to use some of the concepts as a kinda inbuilt addon until we integrate the sql concept fully in MC.

Cheers

Jul 13, 2011 at 8:20 PM

How would you like me to send? direct msg somehow?

I don't see a visible email, i think Playos had one?

Jul 13, 2011 at 11:41 PM


Just sent you a message.....