python-readability/README at master · appscluster/python-readability · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
Python-Readability-lxml Version 0.3.0.7
=======================================

Forked from Yuri Baburov <burchik@gmail.com>

This code is under the Apache License 2.0.  http://www.apache.org/licenses/LICENSE-2.0

This is a python port of a ruby port of arc90's readability project

http://lab.arc90.com/experiments/readability/

In few words,
Given a html document, it pulls out the main body text and cleans it up.
It also can clean up title based on latest readability.js code.

Based on:
 - Latest readability.js ( https://github.com/MHordecki/readability-redux/blob/master/readability/readability.js )
 - Ruby port by starrhorne and iterationlabs
 - Python port by gfxmonk ( https://github.com/gfxmonk/python-readability , based on BeautifulSoup )
 - Decruft effort to move to lxml ( http://www.minvolai.com/blog/decruft-arc90s-readability-in-python/ )
 - "BR to P" fix from readability.js which improves quality for smaller texts.
 - Github users contributions.


Installation::

Dev Version:

    wget https://github.com/appscluster/python-readability/archive/master.zip

    7z x master.zip ( if you don't have 7z : sudo apt-get install p7zip-full )

    cd python-readability-master/

    python setup.py install

Base Version:

    easy_install readability-lxml
    or
    pip install readability-lxml


Usage::

    from readability.readability import Document
    import urllib
    html = urllib.urlopen(url).read()
    readable_article = Document(html).summary()
    readable_title = Document(html).short_title()


Command-line usage::

    Default:

    python -m readability.readability -u http://pypi.python.org/pypi/readability-lxml

    Add User-Agent:

    python -m readability.readability -u http://www.bbc.co.uk/newsbeat/30520643 -a "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36"


Using positive/negative keywords example::

    python -m readability.readability -p intro -n newsindex,homepage-box,news-section -u http://python.org


Document() kwarg options:

 - attributes:
 - debug: output debug messages
 - min_text_length:
 - retry_length:
 - url: will allow adjusting links to be absolute
 - positive_keywords: the list of positive search patterns in classes and ids, for example: ["news-item", "block"]
 - negative_keywords: the list of negative search patterns in classes and ids, for example: ["mysidebar", "related", "ads"]


Updates

 - 0.3.0.7 Encoding updated based on buriy's update
 - 0.3.0.6 Added option for User-Agent inclusion