o
Dh^K @ s d Z ddlZddlZddlmZ ddlmZ ddlmZm Z m
Z
m Z ddl m
Z
ddlmZmZ dd lmZmZ eeZed
Zd ed efd
dZh dZddhZ h dZ h dZ h dZ h dZ edej!edej!edej!edej!edej!edej!d Z"d d hZ#d d hZ$d!ed e%fd"d#Z&G d$d% d%Z'G d&d' d'Z(ed(ej!ed)ej!d*Z)ed+ej!Z*d,ed e+fd-d.Z,i fd ed/ed e+fd0d1Z-dS )2a Minimalistic fork of readability-lxml code
This is a python port of a ruby port of arc90's readability project
http://lab.arc90.com/experiments/readability/
Given a html document, it pulls out the main body text and cleans it up.
Ruby port by starrhorne and iterationlabs
Python port by gfxmonk
For list of contributors see
https://github.com/timbertson/python-readability
https://github.com/buriy/python-readability
License of forked code: Apache-2.0.
N)sqrt)
attrgetter)AnyDictOptionalSet)tostring) HtmlElementfragment_fromstring ) load_htmltrimz\.( |$)stringreturnc C s t | tddS )Nxml)encodingmethod)r str)r r ]/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/trafilatura/readability_lxml.py _tostring& s r >
apdloluldivimgpretable
blockquoter article> tdr r > ddr dtlir r formasideaddress>
h1h2h3h4h5h6thnavfooterheader> r r r% r embedinputzcombx|comment|community|disqus|extra|foot|header|menu|remark|rss|shoutbox|sidebar|sponsor|ad-break|agegate|pagination|pager|popup|tweet|twitterz#and|article|body|column|main|shadowzKarticle|body|content|entry|hentry|main|page|pagination|post|text|blog|storyzbutton|combx|comment|com-|contact|figure|foot|footer|footnote|form|input|masthead|media|meta|outbrain|promo|related|scroll|shoutbox|sidebar|sponsor|shopping|tags|tool|widgetz.<(?:a|blockquote|dl|div|img|ol|p|pre|table|ul)z+https?:\/\/(?:www\.)?(?:youtube|vimeo)\.com)unlikelyCandidatesReokMaybeItsACandidateRe
positiveRe
negativeRedivToPElementsRevideoRebodyhtmlr r elemc C s t t| S )z7Return the length of the element with all its contents.)lenr
text_contentr= r r r text_lengthW s rA c @ s. e Zd ZdZddgZdededdfddZdS ) Candidatez,Defines a class to score candidate elements.scorer= r Nc C s || _ || _d S N)rC r= )selfrC r= r r r __init__a s
zCandidate.__init__)__name__
__module__ __qualname____doc__ __slots__floatr rF r r r r rB \ s rB c @ s e Zd ZdZg dZd$dedededd fd
d Zdefd d
Z de
ee f de defddZ de
ee f de
e fddZdedefddZde
ee f fddZdedefddZdede fdd Zd%d d Zd%d d Zd!ede
ee f defd"d#Zd S )&Documentz,Class to build a etree document out of html.docmin_text_length retry_length rO rP rQ r Nc C s || _ || _|| _dS )a Generate the document
:param doc: string of the html content.
:param min_text_length: Set to a higher value for more precise detection of longer texts.
:param retry_length: Set to a lower value for better detection of very small texts.
The Document class is not re-enterable.
It is designed to create a new Document() for each HTML file to process it.
API method:
.summary() -- cleaned up content
NrN )rE rO rP rQ r r r rF k s
zDocument.__init__c C s | j ddD ]}| qd} |r| | | }| |}|r-| ||}n |du r9d}t d qt d | j
d}|durJ|n| j }| ||}t |pWd }|rc|| j
k rcd}q|S )
z
Given a HTML file, extracts the text of the article.
Warning: It mutates internal DOM representation of the HTML document,
so it is better to call other API methods before this one.
scriptstyleTFz5Ended up stripping too much - going for a safer parsez=Ruthless and lenient parsing did not work. Returning raw htmlr; N )rO iter drop_treeremove_unlikely_candidates&transform_misused_divs_into_paragraphsscore_paragraphsselect_best_candidate get_articleLOGGERdebugfindsanitizer> rQ ) rE r= ruthless
candidatesbest_candidater! r; cleaned_articlearticle_lengthr r r summary| s8
zDocument.summaryrc rd c C s t d|jd }td}|j }|d urt|n|jg}|D ]H}d}||jks3||v r6|| j|kr6d}n*|jdkr`| |} |jpDd}
t |
} | dkrQ| d k s^| dkr`| d
kr`t
|
r`d}|rg| | q |S ) N
皙?zFTr rV P g ?r )
maxrC r
r= getparentlisttagget_link_densitytextr> DOT_SPACEsearchappend) rE rc rd sibling_score_thresholdoutputparentsiblingssiblingrs link_density node_content node_lengthr r r r] s2
zDocument.get_articlec C s\ |sd S t | tddd}ttjr(|d d D ] }td|jj |j
qt t |S )NrC T)keyreverse z Top 5: %s %s)
sortedvaluesr r^ isEnabledForloggingDEBUGr_ r= rn rC nextrW )rE rc sorted_candidates candidater r r r\ s z Document.select_best_candidater= c C s, t |pd}tdd |dD }|| S )Nr c s s | ]}t |V qd S rD )rA ).0linkr r r z,Document.get_link_density..z.//a)rA sumfindall)rE r= total_length link_lengthr r r ro s zDocument.get_link_densityc
C s i }| j dddD ]]}| }|d u rq
| }t| }t|}|| jk r)q
||fD ]}|d ur>||vr>| |||< q-dt|d t |d d }|| j
|7 _
|d urg|| j
|d 7 _
q
| D ]\}} | j
d| | 9 _
ql|S ) Nr r r" r ,d )
rO rW rl r
r? r> rP
score_nodesplitminrC itemsro )
rE rc r= parent_nodegrand_parent_node elem_text
elem_text_lennoderC r r r r r[ s,
zDocument.score_paragraphsc C sT d}t d |d|dfD ]}td |r |d8 }td |r'|d7 }q|S )Nr classidr8 rR r7 )filtergetREGEXESrr )rE r= weight attributer r r class_weight s zDocument.class_weightc C sl | |}t|j}| }|tv r|d7 }n|tv r |d7 }n|tv r)|d8 }n|tv r1|d8 }t||S )Nr~ r ) r r rn lower
DIV_SCORES BLOCK_SCORESBAD_ELEM_SCORESSTRUCTURE_SCORESrB )rE r= rC rn namer r r r s
zDocument.score_nodec C sr | j dD ]0}dtd |d|df}t|dk r q|jtvr6td |r6td |s6|
qd S )Nz.//* r r r r5 r6 ) rO r joinr r r> rn
FRAME_TAGSr rr rX )rE r= attrsr r r rY s
z#Document.remove_unlikely_candidatesc C s | j dD ]}td dttt|sd|_q| j dD ]N}|j r=|j
r=t d}|j d |_ |_ | d| t
t|ddD ])\}}|jre|j
ret d}|jd |_ |_| |d | |jd
krn| qEq!d S ) Nz.//divr9 rV r zr T)r} r br)rO r r rr r mapr rm rn rp stripr
insertr enumeratetailrX )rE r= p_elemposchildr r r rZ ) s*
z/Document.transform_misused_divs_into_paragraphsr c s | ddddddD ]}| |dk s| |dkr | q
| d d
D ] q%| d D ] d jv rFtd
jd rFd _q1 q1t }t |
dD ]} |v r]qU| } |v rk| j nd}|| dk rt
d j|| qU ddk rd} fddtD }|d d8 < |d t d8 < t } | }
} | d ur| |v r|| j nd}|d r|d d |d d krd |d d } n|d |d kr jtvrd } n|d |d d! krd"} n| | jk r|d dkrd#| d$} n| | jk r%|d d%kr%d#| d&} n|d'k r9|
d(kr9d)|
d*d+| } n|d'krM|
d,krMd)|
d*d+| } nr|d- d krY| d.k s`|d- d krcd/} n\| sd0} g }
D ]}t|}|r~|
| nqnt|
d } jdd1D ]}t|}|r|
| t|
|kr nq|
rt|
d2krd3}| d4d5d6d7 nd3}|rӈ t
d8| j|| pd9 qU|| _ t | j S ):Nr) r* r+ r, r- r. r g Q ?r& textareaiframesrcr: VIDEOz6//table|//ul|//div|//aside|//header|//footer|//sectionz+Removed %s with score %6.3f and weight %-3sr rh Tc s" i | ]
}|t d | qS )z.//)r> r )r kindr@ r r
g s z%Document.sanitize..r% r r4 z.//input[@type="hidden"]r r r g?ztoo many images ()zmore
s than
sr z less than 3x
s than sztoo short content length z without a single imager z and too many imagesrR ri ztoo many links z.3fz for its weight g ?r3 K z<