o
Dh% @ s d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z m
Z
ddl m Z m
Z
mZmZmZmZ ddlmZ dd lmZ dd
lmZ dd lmZ eeZ h d Z e d
Z e dZ e dZ!e dZ"e dZ#e dZ$G dd dZ%de&de'fddZ(de e& de%de e& fdd Z)de&de%de e& fd d Z*de&de%de e& fd d Z+d!e&de%de e& fd"d#Z,de%d$e
e& de e& fd%d&Z- ' (d2d)e&d*e
e& d+e'd,e.de e& f
d-d.Z/d/e&d*e
e& de e& fd0d1Z0dS )3z>
Examining feeds and extracting links for further processing.
N)islice)sleep)ListOptional) check_url clean_url filter_urlsfix_relative_urls get_hostinfo is_valid_url )is_similar_domain) fetch_url) MAX_LINKS) load_html> text/rdftext/rsstext/xml text/atom
text/plain text/rdf+xml text/rss+xml
text/atom+xmlapplication/rdfapplication/rssapplication/xmlapplication/atomapplication/jsonapplication/rdf+xmlapplication/rss+xmlapplication/atom+xmlapplication/feed+jsonapplication/x-atom+xmlapplication/x.atom+xmlz<(feed|rss|\?xml)z(?:\s*)(?:)?(?:\s*)z \bcomments\bzn\.(?:atom|rdf|rss|xml)$|\b(?:atom|rss)\b|\?type=100$|feeds/posts/default/?$|\?feed=(?:atom|rdf|rss|rss2)|feed$c
@ sD e Zd ZdZg dZ d
dedededed ee d
df d d ZdS )FeedParametersz.Store necessary information to proceed a feed.basedomainextlangrefFNbaseurlr' referenceexternal target_langreturnc C s" || _ || _|| _|| _|| _d S Nr% )selfr+ r' r, r- r. r2 R/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/trafilatura/feeds.py__init__M s
zFeedParameters.__init__)FN) __name__
__module__ __qualname____doc__ __slots__strboolr r4 r2 r2 r2 r3 r$ I s$ r$ feed_stringr/ c C s* t | rdS | dd }d|v pd|v S )z$Check if the string could be a feed.TNd z s z find_links..itemsc S s g | ]}|d ur|qS r0 r2 )rV cr2 r2 r3 rW s zJSON decoding error: %szPossibly invalid feed: %sz s
z find_links..zc S s g | ]}|d qS )r )stripr] r2 r2 r3 rW s
)rA
startswithjsonloadsrU decoderJSONDecodeErrorrJ debugr' r
LINK_ATTRSfinditerr
LINK_ELEMENTSreDOTALL)r< rC
candidatesr2 r2 r3
find_links s0
rm c sl | s t d j g S t| } fddt| D }|r-t dt|t| |S t d j |S )z7Extract and refine links from Atom, RSS and JSON feeds.zEmpty feed: %sc s( g | ]}| j kr|d dkr|qS )/ )r* countr\ rC r2 r3 rW s
z!extract_links..z!Links found: %s of which %s validzInvalid feed for %s)rJ rf r' rm r` rQ len)r< rC
feed_linksrM r2 rq r3
extract_links s
rt
htmlstringc C s t | }|du rtd|j g S dd |dD }|s'dd |dD }g }t|D ]!}t|j|}t|}|rO||j krOt
|rOt |sO|
| q.tdt|t| |S ) zxParse the HTML and try to extract feed URLs from the home page.
Adapted from http://www.aaronsw.com/2002/feedfinder/NzInvalid HTML/Feed page: %sc S s8 g | ]}| d tv st| ddr| ddqS )typehref )rU
FEED_TYPESLINK_VALIDATION_REr[ r\ r2 r2 r3 rW s
z"determine_feed..z //link[@rel="alternate"][@href]c S s* g | ]}t |d dr|d dqS )rw rx )rz r[ rU r\ r2 r2 r3 rW s
z
//a[@href]z%Feed URLs found: %s of which %s valid)r rJ rf r& xpathdictfromkeysr r r* r BLACKLISTr[ rL rr )ru rC tree feed_urls output_urlsrO r2 r2 r3 determine_feed s8
r urlfilterc C sT | j r(td| j d| j d}|r(t|| }t||}tdt|| j |S g S )z2Alternative way to gather feed links: Google News.z*https://news.google.com/rss/search?q=site:z&hl=z&scoring=n&num=100z!%s Google news links found for %s)r) r r' rt r rJ rf rr )rC r
downloadedrs r2 r2 r3 probe_gnews s
r F @rS r. r-
sleep_timec C s t | \}}|du rtd| g S t||| ||}d}t| }|durft||} | sMt||D ]}
t|
} | r@| t| | q0t| t|d krM| }| r_t | |} t
dt| | | S t
d| nt d| | d|kr|t
| t||S t||S )a Try to find feed URLs.
Args:
url: Webpage or feed URL as string.
Triggers URL-based filter if the webpage isn't a homepage.
target_lang: Define a language to filter URLs based on heuristics
(two-letter string, ISO 639-1 format).
external: Similar hosts only or external URLs
(boolean, defaults to False).
sleep_time: Wait between requests on the same website.
Returns:
The extracted links as a list (sorted list of unique links).
NzInvalid URL: %sro z%s feed links found for %sz No usable feed links found: %sz Could not download web page: %srn )r
rJ rK r$ r rt r extendrr r rf errorr` r try_homepager ) rS r. r- r r' r+ rC r r rs rE r< r2 r2 r3 find_feed_urls s6
r r+ c C s t d| t| |S )zhShift into reverse and try the homepage instead of the particular feed
page that was given as input.z&Probing homepage for feeds instead: %s)rJ rf r )r+ r. r2 r2 r3 r 4 s
r )NFr )1r8 rb loggingrj itertoolsr timer typingr r courlanr r r r r
r
deduplicationr
downloadsr settingsr utilsr getLoggerr5 rJ ry compiler> rg rZ ri r~ rz r$ r: r; rA rQ rm rt r r floatr r r2 r2 r2 r3 s\
( ,
"9