Scrapers: Difference between revisions
(Convert to Portal) |
(Updated for v20) |
||
Line 3: | Line 3: | ||
= Using Scrapers = | = Using Scrapers = | ||
Scrapers are used to contact online information providers to download metadata for video and music items to be added to the library. | Scrapers are used to contact online information providers to download metadata for video and music items that are to be added to the library. | ||
A number of scrapers are installed by default when installing {{kodi}}. Other scrapers are available for installation from the official repository. The | A number of scrapers are installed by default when installing {{kodi}}. Other scrapers are available for installation from the official repository. The tables at the bottom of this page list all official scrapers and their compatability. | ||
{| style="width: 100%;" | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[HOW-TO:Create_Video_Library|Creating your Video Library]]}}''' | |||
Setting up your video library for the first time? Follow this How To guide | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Create_Music_Library|Creating your Music Library]]}}''' | |||
Setting up your music library for the first time? Follow this How To guide | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Incorrect_and_missing_videos|Video scraping problems]]}}''' | |||
Something not working as expected? Check this page for common solutions. | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | | |||
| style="width: 44%; vertical-align:top;" | | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Changing Scrapers]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:language_globe_icon.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[:Category:Scraper|List of all Scraper pages]]}}''' | |||
|} | |||
= Scraper Development = | |||
Historically, {{kodi}} scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of {{kodi}} v19 Matrix, in favour of Python based scrapers. | |||
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes. | |||
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information. | |||
{| style="width: 100%;" | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_movie_scraper_development|Developing a Movie scraper in Python]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[Scraper development XML|XML Scraper Development]]}}''' | |||
XML scrapers are deprecated. This page may still contain useful development information. | |||
|- | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" |'''{{resize|1.3em| [[Python_tv_scraper_development|Developing a TV Show scraper in Python]]}}''' | |||
| style="width: 2%;" | | |||
| style="width: 5%; vertical-align:top;" | [[File:Crystal Project Network Connection Manager.png|50px|left]] | |||
| style="width: 44%; vertical-align:top;" | '''{{resize|1.3em| [[HOW-TO:Write_media_scrapers|Writing an XML scraper]]}}''' | |||
XML scrapers are deprecated. This page may still contain useful development information. | |||
|} | |||
= Official Scrapers = | |||
{| | {| | ||
! style="background-color:salmon;" | Scrapers compatible with | ! style="background-color:salmon;" | Scrapers compatible with v19 Matrix | ||
! style="background-color:#a0e75a;" | Scrapers compatible with | ! style="background-color:#a0e75a;" | Scrapers compatible with v20 Nexus | ||
! style="background-color:skyBlue;" | Scrapers compatible with | ! style="background-color:skyBlue;" | Scrapers compatible with v21 Omega | ||
|- | |- | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Matrix_add-on_repository | ||
category = Album_scraper_add-ons | category = Album_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Nexus_add-on_repository | ||
category = Album_scraper_add-ons | category = Album_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Album}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Omega_add-on_repository | ||
category = Album_scraper_add-ons | category = Album_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
Line 45: | Line 104: | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Matrix_add-on_repository | ||
category = Artist_scraper_add-ons | category = Artist_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Nexus_add-on_repository | ||
category = Artist_scraper_add-ons | category = Artist_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Artist}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Omega_add-on_repository | ||
category = Artist_scraper_add-ons | category = Artist_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
Line 76: | Line 138: | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Matrix_add-on_repository | ||
category = Music_video_scraper_add-ons | category = Music_video_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Nexus_add-on_repository | ||
category = Music_video_scraper_add-ons | category = Music_video_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Music Video}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Omega_add-on_repository | ||
category = Music_video_scraper_add-ons | category = Music_video_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
Line 107: | Line 172: | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Matrix_add-on_repository | ||
category = Movie_scraper_add-ons | category = Movie_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Nexus_add-on_repository | ||
category = Movie_scraper_add-ons | category = Movie_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|Movie}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Omega_add-on_repository | ||
category = Movie_scraper_add-ons | category = Movie_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
Line 138: | Line 206: | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Matrix_add-on_repository | ||
category = TV_scraper_add-ons | category = TV_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Nexus_add-on_repository | ||
category = TV_scraper_add-ons | category = TV_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
| | | | ||
{| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | {| class="mw-collapsible mw-collapsed wikitable" style="margin: 0.0em 0 0.0em 0em; {{border-radius|5px}}; {{box-shadow|0px|1px|3px|#878787}};" | ||
! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | ! scope="row" style="width:500px; background-color:#dfedf0; text-align:left;"| List of {{red|TV Show}} scrapers | ||
|- | |- | ||
| | | | ||
<dpl> | <dpl> | ||
category = | category = Omega_add-on_repository | ||
category = TV_scraper_add-ons | category = TV_scraper_add-ons | ||
notcategory = Outdated | |||
</dpl> | </dpl> | ||
|} | |} | ||
|} | |} | ||
{{updated| | {{updated|20}} | ||
[[Category:Karellen]] | [[Category:Karellen]] |
Revision as of 23:45, 22 January 2023
Development Music library Video library |
Scrapers |
Using Scrapers
Scrapers are used to contact online information providers to download metadata for video and music items that are to be added to the library.
A number of scrapers are installed by default when installing Kodi. Other scrapers are available for installation from the official repository. The tables at the bottom of this page list all official scrapers and their compatability.
Creating your Video Library
Setting up your video library for the first time? Follow this How To guide |
Creating your Music Library
Setting up your music library for the first time? Follow this How To guide | |||
Video scraping problems
Something not working as expected? Check this page for common solutions. |
||||
Changing Scrapers | List of all Scraper pages |
Scraper Development
Historically, Kodi scrapers were XML based. While XML scrapers have been used successfully for well over a decade, their use has been deprecated, since the release of Kodi v19 Matrix, in favour of Python based scrapers.
XML scrapers have become difficult to develop and maintain as they are no longer suited to the advanced REST API's used by the information providers, which results in frequent maintenance and updates to cope with API changes.
The following pages are available to help with the development process of scrapers. Although XML scrapers are deprecated, and no future XML scrapers will be accepted into the official repository, the two pages dedicated to XML scrapers contain useful information.
Developing a Movie scraper in Python | XML Scraper Development
XML scrapers are deprecated. This page may still contain useful development information. | |||
Developing a TV Show scraper in Python | Writing an XML scraper
XML scrapers are deprecated. This page may still contain useful development information. |
Official Scrapers
Scrapers compatible with v19 Matrix | Scrapers compatible with v20 Nexus | Scrapers compatible with v21 Omega | ||||||
---|---|---|---|---|---|---|---|---|
|
|
| ||||||
|
|
| ||||||
|
|
| ||||||
|
|
| ||||||
|
|
|