commit python-wptools for openSUSE:Factory

Hello community, here is the log from the commit of package python-wptools for openSUSE:Factory checked in at 2018-05-01 23:28:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-wptools (Old) and /work/SRC/openSUSE:Factory/.python-wptools.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Package is "python-wptools" Tue May 1 23:28:39 2018 rev:6 rq:602625 version:0.4.12 Changes: -------- --- /work/SRC/openSUSE:Factory/python-wptools/python-wptools.changes 2018-01-28 20:33:35.110436159 +0100 +++ /work/SRC/openSUSE:Factory/.python-wptools.new/python-wptools.changes 2018-05-01 23:28:41.193489374 +0200 @@ -1,0 +2,9 @@ +Mon Apr 30 19:10:36 UTC 2018 - badshah400@gmail.com + +- Update to version 0.4.12 [list of changes cumulated from version + 0.4.9]: + * Respect silence (gh#siznax/wptools#118). + * Added more general infobox parsing alternative. + * Improved infobox parsing. + +------------------------------------------------------------------- Old: ---- wptools-0.4.8.tar.gz New: ---- wptools-0.4.12.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-wptools.spec ++++++ --- /var/tmp/diff_new_pack.fwUcM2/_old 2018-05-01 23:28:45.109346986 +0200 +++ /var/tmp/diff_new_pack.fwUcM2/_new 2018-05-01 23:28:45.113346840 +0200 @@ -19,7 +19,7 @@ %define pyname wptools %{?!python_module:%define python_module() python-%{**} python3-%{**}} Name: python-wptools -Version: 0.4.8 +Version: 0.4.12 Release: 0 Summary: Wikipedia tools (for Humans) License: MIT ++++++ python-wptools-avoid-reading-readme.patch ++++++ --- /var/tmp/diff_new_pack.fwUcM2/_old 2018-05-01 23:28:45.141345822 +0200 +++ /var/tmp/diff_new_pack.fwUcM2/_new 2018-05-01 23:28:45.145345677 +0200 @@ -1,7 +1,7 @@ -Index: wptools-0.4.6/setup.py +Index: wptools-0.4.12/setup.py =================================================================== ---- wptools-0.4.6.orig/setup.py -+++ wptools-0.4.6/setup.py +--- wptools-0.4.12.orig/setup.py ++++ wptools-0.4.12/setup.py @@ -2,8 +2,6 @@ from setuptools import setup, find_packages @@ -13,7 +13,7 @@ @@ -12,7 +10,16 @@ setup( name='wptools', - version='0.4.8', + version='0.4.12', description='Wikipedia tools (for Humans)', - long_description=readme + '\n\n' + history, + long_description= ++++++ wptools-0.4.8.tar.gz -> wptools-0.4.12.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/HISTORY.rst new/wptools-0.4.12/HISTORY.rst --- old/wptools-0.4.8/HISTORY.rst 2018-01-19 22:27:38.000000000 +0100 +++ new/wptools-0.4.12/HISTORY.rst 2018-04-30 00:33:58.000000000 +0200 @@ -3,9 +3,11 @@ Release History --------------- -0.4.8 (2018-01-19) -++++++++++++++++++ +0.4.12 (2018-04-29) ++++++++++++++++++++ +* Always respect silent flag (#118) +* Improved infobox parsing (#91, #109) * Added image (license, assessment) metadata (#104, #105) * Fixed Wikidata pageid, image bugs (#101, #102) * Added support for category continuations, subcategories (#99) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/PKG-INFO new/wptools-0.4.12/PKG-INFO --- old/wptools-0.4.8/PKG-INFO 2018-01-19 22:32:57.000000000 +0100 +++ new/wptools-0.4.12/PKG-INFO 2018-04-30 00:38:59.000000000 +0200 @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: wptools -Version: 0.4.8 +Version: 0.4.12 Summary: Wikipedia tools (for Humans) Home-page: https://github.com/siznax/wptools/ Author: Steve @siznax @@ -51,42 +51,161 @@ >>> import wptools + Get a page object: + .. code-block:: python >>> page = wptools.page('Gandhi') + Get `API:Query`_ data: + + .. _`API:Query`: https://www.mediawiki.org/wiki/API:Query + + .. code-block:: python + + >>> page.get_query() + en.wikipedia.org (query) Gandhi + en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg + Mahatma Gandhi (en) data + { + aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... + assessments: Pakistan, Alternative Views, South Afric... + description: pre-eminent leader of Indian nationalism ... + extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... + extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u... + label: Mahatma Gandhi + length: 262,790 + links: 10 Janpath, 14th Dalai Lama, 1915 Singapore M... + modified: page + pageid: 19379 + random: Salt + redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} + redirects: {u'ns': 0, u'pageid': 55342, u'title': u'M... + requests: query, imageinfo + title: Mahatma Gandhi + url: https://en.wikipedia.org/wiki/Mahatma_Gandhi + url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw + watchers: 1,811 + wikibase: Q1001 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + } + + + Get `API:Parse`_ data: + + .. _`API:Parse`: https://www.mediawiki.org/wiki/API:Parse + + .. code-block:: python + + >>> page.get_parse() + en.wikipedia.org (parse) Gandhi + en.wikipedia.org (imageinfo) File:MKGandhi.jpg + Mahatma Gandhi (en) data + { + image: {u'size': 2951123, 'kind': 'parse-image', u'des... + infobox: known_for, other_names, image, signature, bi... + iwlinks: https://biblio.wiki/wiki/Mohandas_K._Gandhi,... + pageid: 19379 + parsetree: <root><template><title>Redirect</title>... + requests: parse, imageinfo + title: Mahatma Gandhi + wikibase: Q1001 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... + } + + + Get Wikidata_: + + .. _Wikidata: https://www.wikidata.org/w/api.php + + .. code-block:: python + + >>> page = wptools.page(wikibase='Q1001') + >>> page.get_wikidata() + www.wikidata.org (wikidata) Q1001 + www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12... + www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P... + www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140... + www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12... + en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg + Mahatma Gandhi (en) data + { + aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... + claims: P646, P535, P906, P434, P648, P3762, P1711, ... + description: pre-eminent leader of Indian nationalism ... + image: {u'size': 2951123, 'kind': 'wikidata-image', u'... + label: Mahatma Gandhi + labels: Q1280678, P535, Q131149, P434, Q1860, P3762,... + modified: wikidata + requests: wikidata, labels, labels, labels, labels, im... + title: Mahatma_Gandhi + what: human + wikibase: Q1001 + wikidata: Geni.com profile ID (P2600), National Libr... + wikidata_pageid: 1330 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + } + + + Get RESTBase_ data: + + .. _RESTBase: https://www.mediawiki.org/wiki/RESTBase + + .. code-block:: python + + >>> page.get_restbase('/page/summary/') + en.wikipedia.org (restbase) /page/summary/Gandhi + Mahatma Gandhi (en) data + { + description: pre-eminent leader of Indian nationalism ... + exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... + exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... + image: {'kind': 'restbase-original', u'width': 2024, '... + pageid: 19379 + requests: restbase + title: Mahatma_Gandhi + url: https://en.wikipedia.org/wiki/Gandhi + url_raw: https://en.wikipedia.org/wiki/Gandhi?action=raw + } + + + Get all the things (at once): + .. code-block:: python >>> page.get() en.wikipedia.org (query) Gandhi en.wikipedia.org (parse) 19379 www.wikidata.org (wikidata) Q1001 - www.wikidata.org (labels) Q1280678|P535|Q18338317|P434|Q1860|P376... - www.wikidata.org (labels) P18|P19|P1066|P509|P345|Q16382|P1006|P3... - www.wikidata.org (labels) Q2140674|Q1282294|Q21200566|P409|Q26490... - www.wikidata.org (labels) P3417|P4431|P2949|P69|Q129286|Q9441|P42... + www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12... + www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P... + www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140... + www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12... en.wikipedia.org (restbase) /page/summary/Mahatma_Gandhi - en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg|File:MKGandhi.jpg + en.wikipedia.org (imageinfo) File:MKGandhi.jpg|File:Portrait Gandhi.jpg Mahatma Gandhi (en) data { aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... - claims: P646, P535, P906, P434, P648, P3762, P1273, ... + assessments: Pakistan, Alternative Views, South Afric... + claims: P646, P535, P906, P434, P648, P3762, P1711, ... description: pre-eminent leader of Indian nationalism ... - exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... - exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... - extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... - extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {'kind': 'query-pageimage', u'descriptionshortu... + exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... + exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... + extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... + extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u... infobox: known_for, other_names, image, signature, bi... iwlinks: https://biblio.wiki/wiki/Mohandas_K._Gandhi,... label: Mahatma Gandhi - labels: Q1280678, P535, Q18338317, Q131149, P434, Q1... - length: 262,058 + labels: Q1280678, P535, Q131149, P434, Q1860, P3762,... + length: 262,790 links: 10 Janpath, 14th Dalai Lama, 1915 Singapore M... modified: wikidata, page pageid: 19379 - parsetree: <root><template><title>Redirect</title>... + parsetree: <root><template><title>Redirect</title>... random: Salt redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} redirects: {u'ns': 0, u'pageid': 55342, u'title': u'M... @@ -94,15 +213,18 @@ title: Mahatma_Gandhi url: https://en.wikipedia.org/wiki/Mahatma_Gandhi url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw - watchers: 1,770 + watchers: 1,811 what: human wikibase: Q1001 - wikidata: Geni.com profile ID (P2600), National Libr... + wikidata: Geni.com profile ID (P2600), National Libr... + wikidata_pageid: 1330 wikidata_url: https://www.wikidata.org/wiki/Q1001 - wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... + wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... } + Get more (expensive) data: + .. code-block:: python >>> page.get_more() @@ -110,10 +232,44 @@ Mahatma Gandhi (en) data { categories: Category:1869 births, Category:1948 death... - contributors: 2,608 - files: File:Aum Om red.svg, File:Commons-logo.svg, Fi... - languages: {u'lang': u'af', u'title': u'Mahatma Gand... - views: 24,565 + contributors: 2,606 + files: File:Aum Om red.svg, File:Commons-logo.svg, Fi... + languages: {u'lang': u'af', u'title': u'Mahatma Gand... + pageid: 19379 + redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} + requests: querymore + title: Mahatma Gandhi + views: 19,242 + } + + + Get data in `another language`_: + + .. _`another language`: https://github.com/siznax/wptools/wiki/Language-Codes + + .. code-block:: python + + >>> page = wptools.page(lang='zh') + zh.wikipedia.org (random) 🍰 + 哈莉特·塔布曼 (zh) data + { + pageid: 211070 + title: 哈莉特·塔布曼 + } + + + Get data from `another wiki`_: + + .. _`another wiki`: https://meta.wikimedia.org/wiki/List_of_Wikipedias + + .. code-block:: python + + >>> page = wptools.page(wiki='en.wikiquote.org') + en.wikiquote.org (random) 🍪 + Malala_Yousafzai (en) + { + pageid: 146817 + title: Malala_Yousafzai } @@ -139,9 +295,11 @@ Release History --------------- - 0.4.8 (2018-01-19) - ++++++++++++++++++ + 0.4.12 (2018-04-29) + +++++++++++++++++++ + * Always respect silent flag (#118) + * Improved infobox parsing (#91, #109) * Added image (license, assessment) metadata (#104, #105) * Fixed Wikidata pageid, image bugs (#101, #102) * Added support for category continuations, subcategories (#99) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/README.rst new/wptools-0.4.12/README.rst --- old/wptools-0.4.8/README.rst 2017-12-13 23:35:18.000000000 +0100 +++ new/wptools-0.4.12/README.rst 2018-02-07 00:55:30.000000000 +0100 @@ -42,42 +42,161 @@ >>> import wptools +Get a page object: + .. code-block:: python >>> page = wptools.page('Gandhi') +Get `API:Query`_ data: + +.. _`API:Query`: https://www.mediawiki.org/wiki/API:Query + +.. code-block:: python + + >>> page.get_query() + en.wikipedia.org (query) Gandhi + en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg + Mahatma Gandhi (en) data + { + aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... + assessments: Pakistan, Alternative Views, South Afric... + description: pre-eminent leader of Indian nationalism ... + extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... + extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u... + label: Mahatma Gandhi + length: 262,790 + links: 10 Janpath, 14th Dalai Lama, 1915 Singapore M... + modified: page + pageid: 19379 + random: Salt + redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} + redirects: {u'ns': 0, u'pageid': 55342, u'title': u'M... + requests: query, imageinfo + title: Mahatma Gandhi + url: https://en.wikipedia.org/wiki/Mahatma_Gandhi + url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw + watchers: 1,811 + wikibase: Q1001 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + } + + +Get `API:Parse`_ data: + +.. _`API:Parse`: https://www.mediawiki.org/wiki/API:Parse + +.. code-block:: python + + >>> page.get_parse() + en.wikipedia.org (parse) Gandhi + en.wikipedia.org (imageinfo) File:MKGandhi.jpg + Mahatma Gandhi (en) data + { + image: {u'size': 2951123, 'kind': 'parse-image', u'des... + infobox: known_for, other_names, image, signature, bi... + iwlinks: https://biblio.wiki/wiki/Mohandas_K._Gandhi,... + pageid: 19379 + parsetree: <root><template><title>Redirect</title>... + requests: parse, imageinfo + title: Mahatma Gandhi + wikibase: Q1001 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... + } + + +Get Wikidata_: + +.. _Wikidata: https://www.wikidata.org/w/api.php + +.. code-block:: python + + >>> page = wptools.page(wikibase='Q1001') + >>> page.get_wikidata() + www.wikidata.org (wikidata) Q1001 + www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12... + www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P... + www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140... + www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12... + en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg + Mahatma Gandhi (en) data + { + aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... + claims: P646, P535, P906, P434, P648, P3762, P1711, ... + description: pre-eminent leader of Indian nationalism ... + image: {u'size': 2951123, 'kind': 'wikidata-image', u'... + label: Mahatma Gandhi + labels: Q1280678, P535, Q131149, P434, Q1860, P3762,... + modified: wikidata + requests: wikidata, labels, labels, labels, labels, im... + title: Mahatma_Gandhi + what: human + wikibase: Q1001 + wikidata: Geni.com profile ID (P2600), National Libr... + wikidata_pageid: 1330 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + } + + +Get RESTBase_ data: + +.. _RESTBase: https://www.mediawiki.org/wiki/RESTBase + +.. code-block:: python + + >>> page.get_restbase('/page/summary/') + en.wikipedia.org (restbase) /page/summary/Gandhi + Mahatma Gandhi (en) data + { + description: pre-eminent leader of Indian nationalism ... + exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... + exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... + image: {'kind': 'restbase-original', u'width': 2024, '... + pageid: 19379 + requests: restbase + title: Mahatma_Gandhi + url: https://en.wikipedia.org/wiki/Gandhi + url_raw: https://en.wikipedia.org/wiki/Gandhi?action=raw + } + + +Get all the things (at once): + .. code-block:: python >>> page.get() en.wikipedia.org (query) Gandhi en.wikipedia.org (parse) 19379 www.wikidata.org (wikidata) Q1001 - www.wikidata.org (labels) Q1280678|P535|Q18338317|P434|Q1860|P376... - www.wikidata.org (labels) P18|P19|P1066|P509|P345|Q16382|P1006|P3... - www.wikidata.org (labels) Q2140674|Q1282294|Q21200566|P409|Q26490... - www.wikidata.org (labels) P3417|P4431|P2949|P69|Q129286|Q9441|P42... + www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12... + www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P... + www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140... + www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12... en.wikipedia.org (restbase) /page/summary/Mahatma_Gandhi - en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg|File:MKGandhi.jpg + en.wikipedia.org (imageinfo) File:MKGandhi.jpg|File:Portrait Gandhi.jpg Mahatma Gandhi (en) data { aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... - claims: P646, P535, P906, P434, P648, P3762, P1273, ... + assessments: Pakistan, Alternative Views, South Afric... + claims: P646, P535, P906, P434, P648, P3762, P1711, ... description: pre-eminent leader of Indian nationalism ... - exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... - exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... - extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... - extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {'kind': 'query-pageimage', u'descriptionshortu... + exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... + exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... + extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... + extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u... infobox: known_for, other_names, image, signature, bi... iwlinks: https://biblio.wiki/wiki/Mohandas_K._Gandhi,... label: Mahatma Gandhi - labels: Q1280678, P535, Q18338317, Q131149, P434, Q1... - length: 262,058 + labels: Q1280678, P535, Q131149, P434, Q1860, P3762,... + length: 262,790 links: 10 Janpath, 14th Dalai Lama, 1915 Singapore M... modified: wikidata, page pageid: 19379 - parsetree: <root><template><title>Redirect</title>... + parsetree: <root><template><title>Redirect</title>... random: Salt redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} redirects: {u'ns': 0, u'pageid': 55342, u'title': u'M... @@ -85,15 +204,18 @@ title: Mahatma_Gandhi url: https://en.wikipedia.org/wiki/Mahatma_Gandhi url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw - watchers: 1,770 + watchers: 1,811 what: human wikibase: Q1001 - wikidata: Geni.com profile ID (P2600), National Libr... + wikidata: Geni.com profile ID (P2600), National Libr... + wikidata_pageid: 1330 wikidata_url: https://www.wikidata.org/wiki/Q1001 - wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... + wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... } +Get more (expensive) data: + .. code-block:: python >>> page.get_more() @@ -101,10 +223,44 @@ Mahatma Gandhi (en) data { categories: Category:1869 births, Category:1948 death... - contributors: 2,608 - files: File:Aum Om red.svg, File:Commons-logo.svg, Fi... - languages: {u'lang': u'af', u'title': u'Mahatma Gand... - views: 24,565 + contributors: 2,606 + files: File:Aum Om red.svg, File:Commons-logo.svg, Fi... + languages: {u'lang': u'af', u'title': u'Mahatma Gand... + pageid: 19379 + redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} + requests: querymore + title: Mahatma Gandhi + views: 19,242 + } + + +Get data in `another language`_: + +.. _`another language`: https://github.com/siznax/wptools/wiki/Language-Codes + +.. code-block:: python + + >>> page = wptools.page(lang='zh') + zh.wikipedia.org (random) 🍰 + 哈莉特·塔布曼 (zh) data + { + pageid: 211070 + title: 哈莉特·塔布曼 + } + + +Get data from `another wiki`_: + +.. _`another wiki`: https://meta.wikimedia.org/wiki/List_of_Wikipedias + +.. code-block:: python + + >>> page = wptools.page(wiki='en.wikiquote.org') + en.wikiquote.org (random) 🍪 + Malala_Yousafzai (en) + { + pageid: 146817 + title: Malala_Yousafzai } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/setup.py new/wptools-0.4.12/setup.py --- old/wptools-0.4.8/setup.py 2018-01-19 22:28:53.000000000 +0100 +++ new/wptools-0.4.12/setup.py 2018-04-30 00:34:14.000000000 +0200 @@ -10,7 +10,7 @@ setup( name='wptools', - version='0.4.8', + version='0.4.12', description='Wikipedia tools (for Humans)', long_description=readme + '\n\n' + history, url='https://github.com/siznax/wptools/', diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/wptools/__init__.py new/wptools-0.4.12/wptools/__init__.py --- old/wptools-0.4.8/wptools/__init__.py 2018-01-19 22:29:05.000000000 +0100 +++ new/wptools-0.4.12/wptools/__init__.py 2018-04-30 00:34:32.000000000 +0200 @@ -18,7 +18,7 @@ __contact__ = "https://github.com/siznax/wptools" __license__ = "MIT" __title__ = "wptools" -__version__ = "0.4.8" +__version__ = "0.4.12" from . import core from . import query diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/wptools/core.py new/wptools-0.4.12/wptools/core.py --- old/wptools-0.4.8/wptools/core.py 2018-01-12 20:54:36.000000000 +0100 +++ new/wptools-0.4.12/wptools/core.py 2018-04-10 04:16:35.000000000 +0200 @@ -104,7 +104,7 @@ self._set_data(action) - if show: + if show and not self.flags.get('silent'): self.show() def _load_response(self, action): @@ -122,7 +122,7 @@ except ValueError: raise ValueError(_query) - if data.get('warnings'): + if data.get('warnings') and not self.flags.get('silent'): utils.stderr("API warning: %s" % data.get('warnings')) if data.get('error'): @@ -193,7 +193,7 @@ """ Pretty-print instance data """ - if self.flags.get('silent') or not self.data: + if not self.data: return ptitle = self.params.get('title') diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/wptools/page.py new/wptools-0.4.12/wptools/page.py --- old/wptools-0.4.8/wptools/page.py 2018-01-19 18:53:17.000000000 +0100 +++ new/wptools-0.4.12/wptools/page.py 2018-04-10 04:12:09.000000000 +0200 @@ -297,6 +297,10 @@ """ self.data['pageid'] = page.get('pageid') + assessments = page.get('pageassessments') + if assessments: + self.data['assessments'] = assessments + extract = page.get('extract') if extract: self.data['extract'] = extract @@ -345,6 +349,9 @@ terms = page.get('terms') if terms: + if terms.get('alias'): + self.data['aliases'] = terms['alias'] + if terms.get('description'): self.data['description'] = next(iter(terms['description']), None) @@ -471,7 +478,7 @@ self.get_restbase('/page/summary/', False, proxy, timeout) - if show: + if show and not self.flags.get('silent'): self.show() else: @@ -492,7 +499,7 @@ self.get_restbase('/page/summary/', False, proxy, timeout) - if show: + if show and not self.flags.get('silent'): self.show() return self diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/wptools/query.py new/wptools-0.4.12/wptools/query.py --- old/wptools-0.4.8/wptools/query.py 2018-01-19 18:53:17.000000000 +0100 +++ new/wptools-0.4.12/wptools/query.py 2018-02-07 00:15:11.000000000 +0100 @@ -69,7 +69,8 @@ "&pithumbsize=240" "&pllimit=500" "&ppprop=disambiguation|wikibase_item" - "&prop=extracts|info|links|pageimages|pageprops|pageterms|redirects" + "&prop=extracts|info|links|pageassessments|pageimages|pageprops" + "|pageterms|redirects" "&redirects" "&rdlimit=500" "&rnlimit=1" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/wptools/utils.py new/wptools-0.4.12/wptools/utils.py --- old/wptools-0.4.8/wptools/utils.py 2017-12-16 00:43:13.000000000 +0100 +++ new/wptools-0.4.12/wptools/utils.py 2018-03-06 00:01:12.000000000 +0100 @@ -21,12 +21,31 @@ def get_infobox(ptree, boxterm="box"): """ - returns infobox from get_parse:parsetreee + Returns parse tree template with title containing <boxterm> as dict: + + <box> = {<name>: <value>, ...} + + If simple transform fails, attempts more general assembly: + + <box> = {'boxes': [{<title>: <parts>}, ...], + 'count': } """ + boxes = [] for item in lxml.etree.fromstring(ptree).xpath("//template"): + title = item.find('title').text if title and boxterm in title: - return template_to_dict(item) + + box = template_to_dict(item) + if box: + return box + + alt = template_to_dict_alt(item, title) + if alt: + boxes.append(alt) + + if boxes: + return {'boxes': boxes, 'count': len(boxes)} def get_links(rlinks): @@ -96,26 +115,42 @@ print(msg, file=sys.stderr) -def template_to_dict(tree): +def template_to_dict(tree, debug=0, find=False): """ - returns wikitext template as dict (one deep) + returns wikitext template as dict + + debug = 1 + prints minimal debug info to stdout + debug > 1 + compares _iter() versus _find() results + find = True + sets values from _find() algorithm (default _iter()) """ - # https://en.wikipedia.org/wiki/Abraham_Lincoln?action=raw§ion=0 + # you can compare (most) raw Infobox wikitext like this: + # https://en.wikipedia.org/wiki/TITLE?action=raw§ion=0 obj = defaultdict(str) errors = [] for item in tree: try: name = item.findtext('name').strip() - tmpl = item.find('value').find('template') - if tmpl is not None: - value = template_to_text(tmpl) - else: - value = text_with_children(item.find('value')) + + if debug: + template_to_dict_debug(name, item, debug) + + find_val = template_to_dict_find(item, debug) # DEPRECATED + iter_val = template_to_dict_iter(item, debug) + + value = iter_val + if find: + value = find_val + if name and value: obj[name] = value.strip() + except AttributeError: + if isinstance(item, lxml.etree.ElementBase): name = item.tag.strip() text = item.text.strip() @@ -123,18 +158,169 @@ obj['infobox'] = text else: obj[name] = text + except: errors.append(lxml.etree.tostring(item)) + if errors: obj['errors'] = errors + return dict(obj) -def text_with_children(node): +def template_to_dict_alt(tree, title): """ - return text content with children (#62), sub-elements (#66) - https://stackoverflow.com/questions/4624062/get-all-text-inside-a-tag-in-lxm... + Returns parse tree template as {<title>: <parts>} + This is a more general parse tree infobox template parser. """ + box = [] + part = [] + + for item in tree.iter(): + + if item.tag == 'part': + if part: + box.append(part) + part = [] + + if item.tag == 'name' or item.tag == 'value': + for attr in item.keys(): + part.append({attr: item.get(attr)}) + + if item.text: + part.append(item.text.strip()) + + if item.tail: + part.append(item.tail.strip()) + + if part: + box.append(part) + + return {title.strip(): box} + + +def template_to_dict_debug(name, item, debug): + """ + Print debug statements to compare algorithms + """ + if debug == 1: + print("\n%s = " % name) + elif debug > 1: + print("\n%s" % name) + print("=" * 64) + print(lxml.etree.tostring(item)) + print() + + +def template_to_dict_find(item, debug=0): + """ + DEPRECATED: Returns infobox parsetree value using etree.find() + + Older template_to_dict() algorithm, uses etree.xpath() to "lookup" + or find specific elements, but fails to include tail text in the + order it is found, and does not _exclude_ <ext> tags (references, + etc.). Compare to template_to_dict_iter(). + """ + if debug > 1: + print("template_to_dict_find:") + + tmpl = item.find('value').find('template') + + if tmpl is not None: + value = template_to_text(tmpl, debug) + else: + value = text_with_children(item.find('value'), debug) + + if debug: + print(" find: %s" % value) + + return value + + +def template_to_dict_iter(item, debug=0): + """ + Returns infobox parsetree value using etree.iter() + + Preferred template_to_dict() algorithm, uses etree.iter() to + iterate over elements, accumulating tail text in order, but not + preserving `<ext>` tags (references, etc.). The advantage is that + it picks up MORE templates and links that may be mixed in with + `<ext>` tags, and keeps the result focused on the data. Compare to + template_to_dict_find(). + """ + valarr = [] + found_template = False + + if debug > 1: + print("template_to_dict_iter:") + + for elm in item.iter(): + + if debug > 1: + template_to_dict_iter_debug(elm) + + if elm.tag == 'value' and not found_template: + valarr.append(elm.text.strip()) + + if elm.tag == 'template': + found_template = True + valarr.append(template_to_text(elm, debug).strip()) + + if elm.tail: + valarr.append(elm.tail.strip()) + + value = " ".join([x for x in valarr if x]) + + if debug: + print(" iter: %s" % value) + + return value + + +def template_to_dict_iter_debug(elm): + """ + Print expanded element on stdout for debugging + """ + if elm.text is not None: + print(" <%s>%s" % (elm.tag, elm.text, elm.tag), end='') + if elm.tail is not None: + print(elm.tail) + else: + print() + else: + if elm.tail is not None: + print(" <%s>%s" % (elm.tag, elm.tail)) + else: + print(" <%s>" % elm.tag) + + +def template_to_text(tmpl, debug=0): + """ + convert parse tree template to text + """ + tarr = [] + for item in tmpl.itertext(): + tarr.append(item) + + text = "{{%s}}" % "|".join(tarr).strip() + + if debug > 1: + print("+ template_to_text:") + print(" %s" % text) + + return text + + +def text_with_children(node, debug=0): + """ + DEPRECATED: return text content with children (#62), sub-elements (#66) + + Only used by deprecated template_to_dict_find(), and suffers from + copypasta code smell. + """ + + # https://stackoverflow.com/questions/4624062/get-all-text-inside-a-tag-in-lxm... + if sys.version.startswith('3'): # py3 needs encoding=str parts = ([node.text] + list(chain( @@ -147,17 +333,14 @@ *([tostring(c, with_tail=False), c.tail] for c in node.getchildren()))) + [node.tail]) - return ''.join(filter(lambda x: x or isinstance(x, str), parts)) + value = ''.join(filter(lambda x: x or isinstance(x, str), parts)).strip() -def template_to_text(tmpl): - """ - convert parse tree template to text - """ - text = [] - for item in tmpl.itertext(): - text.append(item) - return "{{%s}}" % "|".join(text) + if debug > 1: + print("+ text_with_children:") + print(" %s" % value) + + return value def wikidata_url(wikibase): diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/wptools-0.4.8/wptools.egg-info/PKG-INFO new/wptools-0.4.12/wptools.egg-info/PKG-INFO --- old/wptools-0.4.8/wptools.egg-info/PKG-INFO 2018-01-19 22:32:57.000000000 +0100 +++ new/wptools-0.4.12/wptools.egg-info/PKG-INFO 2018-04-30 00:38:59.000000000 +0200 @@ -1,6 +1,6 @@ Metadata-Version: 1.1 Name: wptools -Version: 0.4.8 +Version: 0.4.12 Summary: Wikipedia tools (for Humans) Home-page: https://github.com/siznax/wptools/ Author: Steve @siznax @@ -51,42 +51,161 @@ >>> import wptools + Get a page object: + .. code-block:: python >>> page = wptools.page('Gandhi') + Get `API:Query`_ data: + + .. _`API:Query`: https://www.mediawiki.org/wiki/API:Query + + .. code-block:: python + + >>> page.get_query() + en.wikipedia.org (query) Gandhi + en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg + Mahatma Gandhi (en) data + { + aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... + assessments: Pakistan, Alternative Views, South Afric... + description: pre-eminent leader of Indian nationalism ... + extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... + extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u... + label: Mahatma Gandhi + length: 262,790 + links: 10 Janpath, 14th Dalai Lama, 1915 Singapore M... + modified: page + pageid: 19379 + random: Salt + redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} + redirects: {u'ns': 0, u'pageid': 55342, u'title': u'M... + requests: query, imageinfo + title: Mahatma Gandhi + url: https://en.wikipedia.org/wiki/Mahatma_Gandhi + url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw + watchers: 1,811 + wikibase: Q1001 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + } + + + Get `API:Parse`_ data: + + .. _`API:Parse`: https://www.mediawiki.org/wiki/API:Parse + + .. code-block:: python + + >>> page.get_parse() + en.wikipedia.org (parse) Gandhi + en.wikipedia.org (imageinfo) File:MKGandhi.jpg + Mahatma Gandhi (en) data + { + image: {u'size': 2951123, 'kind': 'parse-image', u'des... + infobox: known_for, other_names, image, signature, bi... + iwlinks: https://biblio.wiki/wiki/Mohandas_K._Gandhi,... + pageid: 19379 + parsetree: <root><template><title>Redirect</title>... + requests: parse, imageinfo + title: Mahatma Gandhi + wikibase: Q1001 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... + } + + + Get Wikidata_: + + .. _Wikidata: https://www.wikidata.org/w/api.php + + .. code-block:: python + + >>> page = wptools.page(wikibase='Q1001') + >>> page.get_wikidata() + www.wikidata.org (wikidata) Q1001 + www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12... + www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P... + www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140... + www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12... + en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg + Mahatma Gandhi (en) data + { + aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... + claims: P646, P535, P906, P434, P648, P3762, P1711, ... + description: pre-eminent leader of Indian nationalism ... + image: {u'size': 2951123, 'kind': 'wikidata-image', u'... + label: Mahatma Gandhi + labels: Q1280678, P535, Q131149, P434, Q1860, P3762,... + modified: wikidata + requests: wikidata, labels, labels, labels, labels, im... + title: Mahatma_Gandhi + what: human + wikibase: Q1001 + wikidata: Geni.com profile ID (P2600), National Libr... + wikidata_pageid: 1330 + wikidata_url: https://www.wikidata.org/wiki/Q1001 + } + + + Get RESTBase_ data: + + .. _RESTBase: https://www.mediawiki.org/wiki/RESTBase + + .. code-block:: python + + >>> page.get_restbase('/page/summary/') + en.wikipedia.org (restbase) /page/summary/Gandhi + Mahatma Gandhi (en) data + { + description: pre-eminent leader of Indian nationalism ... + exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... + exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... + image: {'kind': 'restbase-original', u'width': 2024, '... + pageid: 19379 + requests: restbase + title: Mahatma_Gandhi + url: https://en.wikipedia.org/wiki/Gandhi + url_raw: https://en.wikipedia.org/wiki/Gandhi?action=raw + } + + + Get all the things (at once): + .. code-block:: python >>> page.get() en.wikipedia.org (query) Gandhi en.wikipedia.org (parse) 19379 www.wikidata.org (wikidata) Q1001 - www.wikidata.org (labels) Q1280678|P535|Q18338317|P434|Q1860|P376... - www.wikidata.org (labels) P18|P19|P1066|P509|P345|Q16382|P1006|P3... - www.wikidata.org (labels) Q2140674|Q1282294|Q21200566|P409|Q26490... - www.wikidata.org (labels) P3417|P4431|P2949|P69|Q129286|Q9441|P42... + www.wikidata.org (labels) Q1280678|P535|P434|Q1860|P3762|Q668|P12... + www.wikidata.org (labels) P119|Q1930187|P691|P18|P19|P1066|P509|P... + www.wikidata.org (labels) Q6512732|Q1568|P972|Q84|P1430|P31|Q2140... + www.wikidata.org (labels) P1576|Q4964182|P1368|P140|Q22336956|P12... en.wikipedia.org (restbase) /page/summary/Mahatma_Gandhi - en.wikipedia.org (imageinfo) File:Portrait Gandhi.jpg|File:MKGandhi.jpg + en.wikipedia.org (imageinfo) File:MKGandhi.jpg|File:Portrait Gandhi.jpg Mahatma Gandhi (en) data { aliases: M K Gandhi, Mohandas Gandhi, Bapu, Gandhi, M... - claims: P646, P535, P906, P434, P648, P3762, P1273, ... + assessments: Pakistan, Alternative Views, South Afric... + claims: P646, P535, P906, P434, P648, P3762, P1711, ... description: pre-eminent leader of Indian nationalism ... - exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... - exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... - extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... - extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {'kind': 'query-pageimage', u'descriptionshortu... + exhtml: <p>Mahātmā <b>Mohandas Karamchand Gandhi</b>... + exrest: Mahātmā Mohandas Karamchand Gandhi (; Hindust... + extext: Mahātmā **Mohandas Karamchand Gandhi** ( ; H... + extract: <p>Mahātmā <b>Mohandas Karamchand Gandhi {u'size': 2951123, 'kind': 'query-pageimage', u... infobox: known_for, other_names, image, signature, bi... iwlinks: https://biblio.wiki/wiki/Mohandas_K._Gandhi,... label: Mahatma Gandhi - labels: Q1280678, P535, Q18338317, Q131149, P434, Q1... - length: 262,058 + labels: Q1280678, P535, Q131149, P434, Q1860, P3762,... + length: 262,790 links: 10 Janpath, 14th Dalai Lama, 1915 Singapore M... modified: wikidata, page pageid: 19379 - parsetree: <root><template><title>Redirect</title>... + parsetree: <root><template><title>Redirect</title>... random: Salt redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} redirects: {u'ns': 0, u'pageid': 55342, u'title': u'M... @@ -94,15 +213,18 @@ title: Mahatma_Gandhi url: https://en.wikipedia.org/wiki/Mahatma_Gandhi url_raw: https://en.wikipedia.org/wiki/Mahatma_Gandhi?action=raw - watchers: 1,770 + watchers: 1,811 what: human wikibase: Q1001 - wikidata: Geni.com profile ID (P2600), National Libr... + wikidata: Geni.com profile ID (P2600), National Libr... + wikidata_pageid: 1330 wikidata_url: https://www.wikidata.org/wiki/Q1001 - wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... + wikitext: {{Redirect|Gandhi}}{{pp-move-indef}}{{pp... } + Get more (expensive) data: + .. code-block:: python >>> page.get_more() @@ -110,10 +232,44 @@ Mahatma Gandhi (en) data { categories: Category:1869 births, Category:1948 death... - contributors: 2,608 - files: File:Aum Om red.svg, File:Commons-logo.svg, Fi... - languages: {u'lang': u'af', u'title': u'Mahatma Gand... - views: 24,565 + contributors: 2,606 + files: File:Aum Om red.svg, File:Commons-logo.svg, Fi... + languages: {u'lang': u'af', u'title': u'Mahatma Gand... + pageid: 19379 + redirected: {u'to': u'Mahatma Gandhi', u'from': u'Gandhi'} + requests: querymore + title: Mahatma Gandhi + views: 19,242 + } + + + Get data in `another language`_: + + .. _`another language`: https://github.com/siznax/wptools/wiki/Language-Codes + + .. code-block:: python + + >>> page = wptools.page(lang='zh') + zh.wikipedia.org (random) 🍰 + 哈莉特·塔布曼 (zh) data + { + pageid: 211070 + title: 哈莉特·塔布曼 + } + + + Get data from `another wiki`_: + + .. _`another wiki`: https://meta.wikimedia.org/wiki/List_of_Wikipedias + + .. code-block:: python + + >>> page = wptools.page(wiki='en.wikiquote.org') + en.wikiquote.org (random) 🍪 + Malala_Yousafzai (en) + { + pageid: 146817 + title: Malala_Yousafzai } @@ -139,9 +295,11 @@ Release History --------------- - 0.4.8 (2018-01-19) - ++++++++++++++++++ + 0.4.12 (2018-04-29) + +++++++++++++++++++ + * Always respect silent flag (#118) + * Improved infobox parsing (#91, #109) * Added image (license, assessment) metadata (#104, #105) * Fixed Wikidata pageid, image bugs (#101, #102) * Added support for category continuations, subcategories (#99)