Difference between revisions of "Free Software Directory:Pywikibot"

From Free Software Directory
Jump to: navigation, search
(→‎.wiki files)
(→‎pywikibot)
 
(56 intermediate revisions by the same user not shown)
Line 6: Line 6:
 
<pre>
 
<pre>
 
# Allow batch uploading for pywikibot
 
# Allow batch uploading for pywikibot
 +
$wgGrantPermissions['editpage']['approverevisions'] = true; # Required by the MediaWiki Extension "Approved Revs" to create new pages
 
$wgGrantPermissions['editprotected']['editarticles'] = true;
 
$wgGrantPermissions['editprotected']['editarticles'] = true;
 
$wgFileExtensions[] = 'wiki';
 
$wgFileExtensions[] = 'wiki';
Line 19: Line 20:
  
 
====Background to using the bot====
 
====Background to using the bot====
−
zhuyifei1999:
+
The method of using BotPasswords improves the security overall. However bot's are required due to a CASAuth issue:
  
−
Pywikibot [1] is a MediaWiki bot library which we are attemting to
+
See also the '''CASAuth''' issue on this page.
−
utilize to perform batch uploads. However, because of CASAuth, we are
 
−
unable [2] to perform main account CAS-based login, and have to resort
 
−
to using BotPasswords. However, to upload, permissions to alter "File"
 
−
namespace is needed, but it is restricted to the 'editarticles'
 
−
permission [4], and there is no grant [5] that gives the permission to
 
−
any BotPasswords-based login. Please add 'editarticles' to the 'Edit
 
−
protected pages' set [5], or any other set if you deem appropriate, to
 
−
allow batch uploading.
 
−
 
 
−
# https://www.mediawiki.org/wiki/Manual:Pywikibot
 
−
# https://github.com/CWRUChielLab/CASAuth/issues/7
 
−
# https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords
 
−
# https://directory.fsf.org/wiki/Special:ListGroupRights
 
−
# https://directory.fsf.org/wiki/Special:ListGrants
 
−
 
 
−
// zhuyifei1999@gmail.com, zhuyifei1999_ in #pywikibot at
 
−
irc.freenode.net, http://commons.wikimedia.org/wiki/User:Zhuyifei1999
 
  
 
==pywikibot==
 
==pywikibot==
 
<pre>
 
<pre>
−
sudo apt-get install python-pip
+
pywikibot_dir="pywikibot-core";
−
sudo pip install --upgrade pip
+
site="http://directory.fsf.org/wiki/Main_Page";
−
git clone https://gerrit.wikimedia.org/r/pywikibot/core pywikibot-core
+
family="freesoftwaredirectory";
−
cd pywikibot-core
+
 
−
python generate_family_file.py 'http://directory.fsf.org/wiki/Main_Page' 'freesoftwaredirectorydev'
+
# The scripts/pagefromfile.py requires scripts/i18n/ (eg scripts/i18n/pagefromfile/) that is pulled with the --recursive option (or "cd pywikibot-core && git submodule update --init" after git clone)
 +
git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/core "$pywikibot_dir"
 +
cd "$pywikibot_dir"
 +
python generate_family_file.py "$site" "$family"
 
python pwb.py generate_user_files
 
python pwb.py generate_user_files
−
# Select family of sites we are working on, just enter the number or name (default: wikipedia): freesoftwaredirectorydev
+
# Select family of sites we are working on, just enter the number or name (default: wikipedia): freesoftwaredirectory
 
# The only known language: en
 
# The only known language: en
 
# The language code of the site we're working on (default: en):  
 
# The language code of the site we're working on (default: en):  
−
# Username on en:freesoftwaredirectorydev: WikiSysop
+
# Username on en:freesoftwaredirectory: WikiSysop
 
# Do you want to add any other projects? ([y]es, [N]o): n
 
# Do you want to add any other projects? ([y]es, [N]o): n
 
# Do you want to add a BotPassword for WikiSysop? ([y]es, [N]o, [q]uit): y
 
# Do you want to add a BotPassword for WikiSysop? ([y]es, [N]o, [q]uit): y
Line 66: Line 53:
 
python pwb.py upload -always -ignorewarn ~/heckert_gnu.transp.small.png "Some description"
 
python pwb.py upload -always -ignorewarn ~/heckert_gnu.transp.small.png "Some description"
  
−
==Issue==
+
===Estimate time for batch uploading===
−
 
 
−
===API issue to create pages===
 
−
There is a problem with creating pages with the API for our specific MediaWiki setup.
 
−
 
 
−
cd pywikibot
 
−
 
 
−
Create a new file, scripts/simpleedit.py, with this text:
 
 
<pre>
 
<pre>
−
#!/usr/bin/python
+
script_generated=$(date +"%Y-%m-%d %H:%M:%S.%N"); # .wiki files with identical content since the last revision will be ignored ("No changes were needed on [[YOURTITLE]]")
−
# -*- coding: utf-8 -*-
+
SECONDS=0
−
# WTFPL License or Unlicense. If not legally possible:
 
−
# MIT License (C) Zhuyifei1999, 2018
 
  
−
from __future__ import absolute_import, unicode_literals
+
for (( i=0; i<10; i++ )); do
 +
  echo "$script_generated" > Test.wiki;
 +
  python pwb.py pagefromfile -summary:"The edit summary" -title:"Test$i" -file:"Test.wiki" -textonly -force -pt:0
 +
done
 +
duration=$SECONDS
 +
echo "$(($duration / 10)) seconds per entry in average."
 +
rm -f Test.wiki
 +
</pre>
  
−
import codecs
+
==Issues==
−
import sys
 
  
−
import pywikibot
+
===CASAuth===
 +
This issue doesn't have to be fixed since there's a work around, but it would be great if it were fixed to give us more options.
  
 +
zhuyifei1999:
  
−
def main():
+
Pywikibot [1] is a MediaWiki bot library which we are attemting to
−
    summary, title, filename = pywikibot.handleArgs()
+
utilize to perform batch uploads. However, because of CASAuth, we are
−
    page = pywikibot.Page(pywikibot.Site(), title)
+
unable [2] to perform main account CAS-based login, and have to resort
 +
to using BotPasswords. However, to upload, permissions to alter "File"
 +
namespace is needed, but it is restricted to the 'editarticles'
 +
permission [4], and there is no grant [5] that gives the permission to
 +
any BotPasswords-based login. Please add 'editarticles' to the 'Edit
 +
protected pages' set [5], or any other set if you deem appropriate, to
 +
allow batch uploading.
  
−
    with codecs.open(filename, 'r', 'utf-8') as f:
+
# https://www.mediawiki.org/wiki/Manual:Pywikibot
−
        page.text = f.read()
+
# https://github.com/CWRUChielLab/CASAuth/issues/7
 +
# https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords
 +
# https://directory.fsf.org/wiki/Special:ListGroupRights
 +
# https://directory.fsf.org/wiki/Special:ListGrants
  
−
    page.save(summary)
+
// zhuyifei1999@gmail.com, zhuyifei1999_ in #pywikibot at
−
 
+
irc.freenode.net, http://commons.wikimedia.org/wiki/User:Zhuyifei1999
−
 
 
−
if __name__ == '__main__':
 
−
    main()
 
−
</pre>
 
−
 
 
−
Then run this
 
−
<pre>
 
−
$ echo "Hello world" > test.wiki
 
−
$ pwb.py simpleedit.py "The edit summary" "The page title" test.wiki
 
−
</pre>
 
−
 
 
−
Page render status:
 
−
* https://commons.wikimedia.org/w/index.php?title=User:Zhuyifei1999/sandbox&diff=prev&oldid=307483739 - Ok
 
−
* http://directory-dev.fsf.org/wiki/The_page_title - Notice: Undefined variable: contentObject in  /var/www/directory.fsf.org-new/extensions/ApprovedRevs/ApprovedRevs.hooks.php  on line 237  Fatal error: Call to a member function getRedirectTarget() on a  non-object in /var/www/directory.fsf.org-new/includes/page/Article.php  on line 682
 
−
** http://directory-dev.fsf.org/wiki?title=The_page_title&action=edit - Ok
 
−
** http://directory-dev.fsf.org/wiki?title=The_page_title&action=history - Ok
 
−
 
 
−
Error code related:
 
−
* https://github.com/wikimedia/mediawiki-extensions-ApprovedRevs/blob/REL1_27/ApprovedRevs.hooks.php#L247
 
−
* https://github.com/wikimedia/mediawiki/blob/REL1_27/includes/page/Article.php#L689 should be where Call to a member function getRedirectTarget() on a  non-object. I have no idea why getContentObject would be a non-object.
 
  
−
Perhaps it's configuration / extension related.
+
==Suggestions==
  
−
Special thanks to zhuyifei1999 for all the text, code, and support.
+
===OAuth===
 +
pywikibot can be used with OAuth - https://www.mediawiki.org/wiki/Manual:Pywikibot/OAuth. However, the OAuth extension is currently not installed - https://directory.fsf.org/wiki/Special:Version
  
 
==Further reading==
 
==Further reading==
 
Read https://www.mediawiki.org/wiki/Manual:Importing_XML_dumps#Using_pywikibot,_pagefromfile.py_and_Nokogiri for more information
 
Read https://www.mediawiki.org/wiki/Manual:Importing_XML_dumps#Using_pywikibot,_pagefromfile.py_and_Nokogiri for more information

Latest revision as of 06:27, 17 August 2018

Free Software Directory:Participate/Using pywikibot should be merged to this page.

MediaWiki

  • Add this to LocalSettings.php
# Allow batch uploading for pywikibot
$wgGrantPermissions['editpage']['approverevisions'] = true; # Required by the MediaWiki Extension "Approved Revs" to create new pages
$wgGrantPermissions['editprotected']['editarticles'] = true;
$wgFileExtensions[] = 'wiki';

Generate a bot account for your user

The bot will be associated with the user name that are logged in.

Background to using the bot

The method of using BotPasswords improves the security overall. However bot's are required due to a CASAuth issue:

See also the CASAuth issue on this page.

pywikibot

pywikibot_dir="pywikibot-core";
site="http://directory.fsf.org/wiki/Main_Page";
family="freesoftwaredirectory";

# The scripts/pagefromfile.py requires scripts/i18n/ (eg scripts/i18n/pagefromfile/) that is pulled with the --recursive option (or "cd pywikibot-core && git submodule update --init" after git clone)
git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/core "$pywikibot_dir"
cd "$pywikibot_dir"
python generate_family_file.py "$site" "$family"
python pwb.py generate_user_files
# Select family of sites we are working on, just enter the number or name (default: wikipedia): freesoftwaredirectory
# The only known language: en
# The language code of the site we're working on (default: en): 
# Username on en:freesoftwaredirectory: WikiSysop
# Do you want to add any other projects? ([y]es, [N]o): n
# Do you want to add a BotPassword for WikiSysop? ([y]es, [N]o, [q]uit): y
# See https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords to
# know how to get codes.Please note that plain text in /home/root
# /pywikibot-core/user-password.py and anyone with read access to that
# directory will be able read the file.
# BotPassword's "bot name" for WikiSysop: wikibot
# BotPassword's "password" for "WikiSysop" (no characters will be shown): 
# '/home/root/pywikibot-core/user-config.py' written.

Test pywikibot on MediaWiki

python pwb.py upload -always -ignorewarn ~/heckert_gnu.transp.small.png "Some description"

Estimate time for batch uploading

script_generated=$(date +"%Y-%m-%d %H:%M:%S.%N"); # .wiki files with identical content since the last revision will be ignored ("No changes were needed on [[YOURTITLE]]")
SECONDS=0

for (( i=0; i<10; i++ )); do
  echo "$script_generated" > Test.wiki;
  python pwb.py pagefromfile -summary:"The edit summary" -title:"Test$i" -file:"Test.wiki" -textonly -force -pt:0
done
duration=$SECONDS
echo "$(($duration / 10)) seconds per entry in average."
rm -f Test.wiki

Issues

CASAuth

This issue doesn't have to be fixed since there's a work around, but it would be great if it were fixed to give us more options.

zhuyifei1999:

Pywikibot [1] is a MediaWiki bot library which we are attemting to utilize to perform batch uploads. However, because of CASAuth, we are unable [2] to perform main account CAS-based login, and have to resort to using BotPasswords. However, to upload, permissions to alter "File" namespace is needed, but it is restricted to the 'editarticles' permission [4], and there is no grant [5] that gives the permission to any BotPasswords-based login. Please add 'editarticles' to the 'Edit protected pages' set [5], or any other set if you deem appropriate, to allow batch uploading.

  1. https://www.mediawiki.org/wiki/Manual:Pywikibot
  2. https://github.com/CWRUChielLab/CASAuth/issues/7
  3. https://www.mediawiki.org/wiki/Manual:Pywikibot/BotPasswords
  4. https://directory.fsf.org/wiki/Special:ListGroupRights
  5. https://directory.fsf.org/wiki/Special:ListGrants

// zhuyifei1999@gmail.com, zhuyifei1999_ in #pywikibot at irc.freenode.net, http://commons.wikimedia.org/wiki/User:Zhuyifei1999

Suggestions

OAuth

pywikibot can be used with OAuth - https://www.mediawiki.org/wiki/Manual:Pywikibot/OAuth. However, the OAuth extension is currently not installed - https://directory.fsf.org/wiki/Special:Version

Further reading

Read https://www.mediawiki.org/wiki/Manual:Importing_XML_dumps#Using_pywikibot,_pagefromfile.py_and_Nokogiri for more information


Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the page “GNU Free Documentation License”.

The copyright and license notices on this page only apply to the text on this page. Any software or copyright-licenses or other similar notices described in this text has its own copyright notice and license, which can usually be found in the distribution or license text itself.

Retrieved from "https://directory.fsf.org/wiki?title=Free_Software_Directory:Pywikibot&oldid=73657"