Texture Cache Maintenance utility

Texture Cache Maintenance utility is a python script created by MillhouseVH, that can do a number of artwork related maintenance on the XBMC library. The script can be used to pre-generate thumbnails, clean up old thumbnail files, find corrupted files, and much more. The script can work on any OS that can run Python 2.6 or higher.

From the author
It's annoying having to delete the entire Textures13.db+Thumbnails folder to try and correct scraping errors and/or corrupted images that just never go away, so I've written a very trivial Python script that can be used to interrogate the texture cache database, and can also be used to remove - with absolute precision - those database rows and cached files that relate to problematic artwork.

The script has been tested with local installations of XBMC Frodo on OpenELEC R-Pi and Ubuntu 12.10, and also Windows with Python 2.7.3 running against a remote Frodo installation of XBMC. A separate properties file can be used to specify non-default configurations (see property file details at the end of this post).

This is not an addon, and it requires that the webserver is enabled in XBMC on port 8080 (unless another port is specified in the properties file). It also assumes you are comfortable working at the command line (ssh in Linux, a large CMD window may also work OK for Windows users).

Summary of features

 * [c, C] Automatically re-cache missing artwork, with option to force download of existing artwork (remove first, then re-cache). Can use multiple threads (default is 2)
 * [nc] Identify those items that require caching (and would be cached by c option)
 * [lc, lnc] Same as c and nc, but only considers those media (movies, tvshows/episodes) added since the modification timestamp of the file identified by the property lastrunfile
 * [p, P] Prune texture cache by removing accumulated cruft such as image previews, previously deleted movies/tv shows/music and whose artwork remains in the texture cache even after cleaning the database. Essentially, remove any cached file that is no longer associated with an entry in the media library, or an addon.
 * [s, S] Search texture cache for specific files and view database content, can help explain reason for incorrect artwork. S will return only those database results for items that no longer exist in the filesystem.
 * [x, X] Extract rows from texture cache database, with optional SQL filter. X will return only those database results for items that no longer exist in the filesystem.
 * [d] Delete specific database rows and corresponding files from the texture cache using database row identifier (see s/S)
 * [r, R] Reverse query cache, identifying any "orphaned" files no longer referenced by texture cache database, with option to auto-delete files
 * [j, J, jd, Jd] Query media library using JSON API, and export content as JSON (and suitable for further external processing)
 * [qa] Perform QA check on media library recently added items, identifying missing properties (eg. plot, mpaa certificate, artwork etc.). Default QA period is previous 30 days. Add property qa.file = yes to verify file exists during QA. Add additional QA fields using qa.art.*, qa.blank.* and qa.zero.* properties.
 * [qax] Like the qa option, but performs a remove and then rescan of any media found to fail the QA tests
 * [ascan, vscan] Initiate audio/video library scan, either entire library or a specific path (see sources). The exit status is the number of items added during each scan, ie. 0 or +n.
 * [watched] Backup and restore movie and tvshow watched lists to a text file. Watched list will be restored keeping more recent playcount, lastplayed and resume points unless property watched.overwrite=yes is specified, in which case the watched list will be restored exactly as per the backup.
 * [missing] Locate media files missing from the specified media library and source label, eg. missing movies "My Movies"
 * [aclean, vclean] Clean audio/video library
 * [directory] Obtain directory listing for a specific path (see sources)
 * [sources] List of sources for a specific media class (video, music, pictures, files, programs)
 * [status] Display status of client - ScreenSaver active, IsIdle (default period 600 seconds, or user specified) and active Player type (audio or video), plus title of any media currently being played.
 * [monitor] Display client event notifications as they occur
 * [power] Set power state of client - suspend, hibernate, shutdown or reboot
 * [wake] Use Wake Over LAN to wake a suspended/hibernating remote client. Specify the MAC address of the remote client in the network.mac property (ie. network.mac=xx:xx:xx:xx:xx:xx). When the client is no longer required, suspend or hibernate it with the relevant power option
 * [exec, execw] Execute the specified addon, with optional parameters. eg. exec script.artwork.downloader silent=true mediatype=tvshow. Use execw to wait, but this rarely has any effect (possibly not implemented by JSON?)

Installation instructions
Download the single Python file required from github. A default properties file is available on github, rename this to texturecache.cfg in order to use it.

To download the script at the command line: wget https://raw.github.com/MilhouseVH/texturecache.py/master/texturecache.py chmod +x ./texturecache.py

If you get a certificate error, try adding "--no-check-certificate" to the wget command line.

and for anyone on OpenELEC which has a pretty basic wget that doesn't support https downloads, use curl instead: curl https://raw.github.com/MilhouseVH/texturecache.py/master/texturecache.py -o texturecache.py chmod +x ./texturecache.py

ATV2 (iOS) users

Python 2.6+ is required to run this script, and although Python can be installed on iOS using "apt-get install python", the version installed (typically v2.5.1 - check with "python --version") is very old and lacks language features required by the script. It is possible to install a more recent Python 2.7.3 package as follows: ssh root@YOUR.ATV2.IP.ADDRESS rm -f python*.deb wget http://yangapp.googlecode.com/files/python_2.7.3-3_iphoneos-arm.deb dpkg -i python*.deb rm python*.deb

Example usage
Let's say the poster image for my "Dr. No" movie is corrupted, and I want to delete it so that XBMC will automatically re-cache it (hopefully correctly) next time it is displayed:

1) Execute: ./texturecache.py s "Dr. No" to search for my Dr. No related artwork

2) Several rows should be returned, relating to different cached artwork - one row will be for the poster, the other fanart, and there may also be rows for other image types too (logo, clearart etc.). This is what I get: 000226|5/596edd13.jpg|0720|1280|0011|2013-03-05 02:07:40|2013-03-04 21:27:37|nfs://192.168.0.3/mnt/share/media/Video/Movies/James Bond/Dr. No (1962)[DVDRip]-fanart.jpg 000227|6/6f3d0d94.jpg|0512|0364|0003|2013-03-05 02:07:40|2013-03-04 22:26:38|nfs://192.168.0.3/mnt/share/media/Video/Movies/James Bond/Dr. No (1962)[DVDRip].tbn Matching row ids: 226 227

3) Since I want to remove only the poster (.tbn) I can execute ./texturecache.py d 227 and both the database row and cached poster image will be deleted. If I wanted to remove both images, I would simply execute ./texturecache.py d 226 227 and the two rows and their corresponding cached images will be removed.

Alternatively, ./texturecache.py C movies "Dr. No" would achieve the same result including re-caching the deleted items.

Format of "dumped" records
When displaying rows from the database, the following fields (columns) are shown:

rowid, cachedurl, height, width, usecount, lastusetime, lasthashcheck, url

Additional usage examples
To view your most recently accessed artwork: ./texturecache.py x | sort -t"|" -k6 -- or -- ./texturecache.py x "order by lastusetime asc"

or your top 10 accessed artwork... ./texturecache.py x | sort -t"|" -k5r | head -10 -- or -- ./texturecache.py x "order by usecount desc" 2>/dev/null | head -10

Use texturecache.py to identify artwork for deletion, then cutting and pasting the matched ids into the "d" option or via a script:

For example, to delete those small remote thumbnails you might have viewed when downloading artwork (and which still clutter up your cache): ./texturecache.py s "size=thumb" 

and the same, but automatically: IDS=$(./texturecache.py s "size=thumb" 2>&1 1>/dev/null | cut -b19-) [ -n "$IDS" ] && ./texturecache.py d $IDS -- or -- ./texturecache.py P

Delete any artwork that has not been accessed since a particular date: ./texturecache.py x "where lastusetime <= '2013-03-05'

or hasn't been accessed more than once: ./texturecache.py x "where usecount <= 1"

Query the media library returning JSON results:

First, just the default fields for a particular class (movies), filtering for a specific item (Avatar):

./texturecache.py j movies "Avatar" [ {    "movieid": 22, "title": "Avatar", "art": { "fanart": "image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fmedia%2fVideo%2fMovies%2fAvatar%20(2009)%5bDVDRip%5d-fanart.jpg/", "discart": "image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f19995%2fmoviedisc%2favatar-4eea31049147b.png/", "poster": "image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fmedia%2fVideo%2fMovies%2fAvatar%20(2009)%5bDVDRip%5d.tbn/", "clearart": "image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f19995%2fmovieart%2favatar-4f803992128b8.png/", "clearlogo": "image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f19995%2fhdmovielogo%2favatar-503e0262ba196.png/" },   "label": "Avatar" } ]

With "extrajson.movies = trailer, streamdetails, file" in the properties file, the same query but returning the extra fields too:

./texturecache.py J movies "Avatar" [ {    "movieid": 22, "title": "Avatar", "label": "Avatar", "file": "nfs://192.168.0.3/mnt/share/media/Video/Movies/Avatar (2009)[DVDRip].m4v", "art": { "fanart": "image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fmedia%2fVideo%2fMovies%2fAvatar%20(2009)%5bDVDRip%5d-fanart.jpg/", "discart": "image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f19995%2fmoviedisc%2favatar-4eea31049147b.png/", "poster": "image://nfs%3a%2f%2f192.168.0.3%2fmnt%2fshare%2fmedia%2fVideo%2fMovies%2fAvatar%20(2009)%5bDVDRip%5d.tbn/", "clearart": "image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f19995%2fmovieart%2favatar-4f803992128b8.png/", "clearlogo": "image://http%3a%2f%2fassets.fanart.tv%2ffanart%2fmovies%2f19995%2fhdmovielogo%2favatar-503e0262ba196.png/" },   "trailer": "", "streamdetails": { "video": [ {         "duration": 9305, "width": 720, "codec": "avc1", "aspect": 1.7779999971389771, "height": 576 }     ],      "audio": [ {         "channels": 6, "codec": "aac", "language": "eng" }     ],      "subtitle": [] } } ]

Optional Properties File
By default the script will run fine on distributions where the .xbmc/userdata folder is within the users Home folder (ie. userdata=~/.xbmc/userdata). To override this default, specify a properties file with a different value for the userdata property.

The properties file should be called texturecache.cfg, and be in the same directory as the texturecache.py script. What follows is an example properties file with the default values shown:

sep = | userdata = ~/.xbmc/userdata dbfile = Database/Textures13.db thumbnails = Thumbnails xbmc.host = localhost webserver.port = 8080 webserver.username = webserver.password = rpc.port = 9090 download.threads = 2 extrajson.albums = extrajson.artists = extrajson.songs = extrajson.movies = extrajson.sets = extrajson.tvshows.tvshow = extrajson.tvshows.season = extrajson.tvshows.episode = qaperiod = 30 qa.file = no qa.fail.urls = image://video, image://music qa.warn.urls = cache.castthumb = no cache.hideallitems = no cache.ignore.types = image://video logfile = logfile.verbose = no checkupdate = yes lastrunfile = orphan.limit.check = yes nonmedia.filetypes = watched.overwrite = no network.mac =

The dbfile and thumbbnails properties represent folders that are both relative to the userdata property.

Specify webserver.username and webserver.password if you require webserver authentication.

extrajson.* properties allow the specification of additional JSON audio/video fields to be returned by the J query option.

Cast thumbnails will not be updated by default, so specify cache.castthumb = yes if you require cast artwork to be re-cached.

Ignore specific URLs when pre-loading the cache (c/C options), by specifying comma delimited regex patterns for the cache.ignore.types property. Default value is image://video. Set to None to process all URLs. Any matching URL will not be cached (c/C), nor will it be deleted when forcing a reload (option C).

Specify a filename for the logfile property, to log detailed processing information.