o
DŒhq& ã @ s˜ d Z ddlZddlZddlmZ ddlmZ ddlmZm Z m
Z
m Z m Z ddl
mZmZmZmZmZmZ ddlmZ dd lmZmZ dd
lmZmZ e e ¡Z e d ¡Z e d ej!¡Z"e d
¡Z#e d¡Z$e d¡Z%e d¡Z&e d¡Z'e d¡Z(e d¡Z)g d¢Z*G dd„ dƒZ+dddefde,de e, de-d e.d e/d e e, f d d „Z0de,d!e e, d e-fd"d#„Z1d$e,d e e, fd%d&„Z2d'e e, d$e,d e e, fd(d)„Z3dS )*z#
Deriving link info from sitemaps.
é N)Úislice)Úsleep)ÚCallableÚListÚSetÚOptionalÚPattern)Ú clean_urlÚextract_domainÚ filter_urlsÚfix_relative_urlsÚ get_hostinfoÚ lang_filteré )Úis_similar_domain)Ú fetch_urlÚ is_live_page)Ú MAX_LINKSÚMAX_SITEMAPS_SEENz.(?:)?zzhref=["\'](.+?)["\']zg(?:blogger|blogpost|ghost|hubspot|livejournal|medium|typepad|squarespace|tumblr|weebly|wix|wordpress)\.z ^.{0,5}<\?xml|~ s €
ÿz.SitemapObject.extract_links..z%%s sitemaps and %s links found for %sN)
r Úfinditerr r r( r) Úlenr r r )r$ r7 r8 r9 Úmatchr% r< r& Ú
extract_linksz s
ÿ
üzSitemapObject.extract_linksc sP dˆj vrdS t dˆj› dtj¡‰ dtddf‡ ‡fdd„ }ˆ td |¡ dS )
z7Extract links corresponding to a given target language.z hreflang=Nzhreflang=[\"'](z.*?|x-default)[\"']Úattrsr! c s2 ˆ | ¡rt | ¡}|rˆ |d ¡ dS dS dS )z!Examine language code attributes.r N)r1 ÚHREFLANG_REGEXr6 )rB Ú
lang_match©Ú
lang_regexr$ r% r& Úhandle_lang_link’ s
ýzASitemapObject.extract_sitemap_langlinks..handle_lang_linkr )r ÚreÚcompiler ÚDOTALLÚstrrA Ú XHTML_REGEX)r$ rG r% rE r& Úextract_sitemap_langlinks‰ s
ÿz'SitemapObject.extract_sitemap_langlinksc C s | td| j¡ dS )z=Extract sitemap links and web page links from a sitemap file.r N)rA Ú
LINK_REGEXr6 r+ r% r% r& Úextract_sitemap_links› s ÿz#SitemapObject.extract_sitemap_linksc C sd t | j| jƒ}|s dS t | j¡s| td| j¡ dS | jdur,| ¡ | j
s*| j r,dS | ¡ dS )z5Download a sitemap and extract the links it contains.Nr )
Úis_plausible_sitemapr r ÚSITEMAP_FORMATr@ rA Ú DETECT_LINKSr6 r rM r r rO )r$ Ú plausibler% r% r& Úprocess¡ s
zSitemapObject.process)NF)r! N)Ú__name__Ú
__module__Ú __qualname__Ú__doc__Ú __slots__rK r r Úboolr' r, r6 r Úintr rA rM rO rT r% r% r% r& r 2 s@ úþýüûú
ù
"ÿÿ ÿ
þ
r Fg @Úurlr r Ú
sleep_timeÚ max_sitemapsr! c s< t | ƒ\}‰ |du rt d| ¡ g S tˆ ƒs t d| ¡ g S d}| d¡r)| g}ng }t| ƒtˆ ƒd kr7| }tˆ ||||ƒ‰ˆjsPtˆ ƒpN‡ fdd„t D ƒˆ_ˆjrˆtˆj
ƒ|k rˆˆj ¡ ˆ_ ˆ
¡ ˆ ¡ ‡fdd„ˆjD ƒˆ_tˆj
ƒ|k r~t|ƒ ˆjrˆtˆj
ƒ|k sZ|r‘tˆj|ƒˆ_t d tˆjƒ|¡ ˆjS )
ax Look for sitemaps for the given URL and gather links.
Args:
url: Webpage or sitemap URL as string.
Triggers URL-based filter if the webpage isn't a homepage.
target_lang: Define a language to filter URLs based on heuristics
(two-letter string, ISO 639-1 format).
external: Similar hosts only or external URLs
(boolean, defaults to False).
sleep_time: Wait between requests on the same website.
max_sitemaps: Maximum number of sitemaps to process.
Returns:
The extracted links as a list (sorted list of unique links).
Nzinvalid URL: %sz*base URL unreachable, dropping sitemap: %s)z.gzr z.xmlé c s g | ] }ˆ › d |› ‘qS )ú/r% )r: Úg©Úbaseurlr% r& Ú
á s ÿz"sitemap_search..c s g | ] }|ˆ j vr|‘qS r% )r )r: Ús)r r% r& rd ë s z %s sitemap links found for %s)r
r( r2 r Úendswithr? r r Úfind_robots_sitemapsÚGUESSESr Úpopr r, rT r r r r) )r\ r r r] r^ Ú
domainnameÚ urlfilterÚ sitemapurlsr% )rc r r& Úsitemap_search³ s@
ÿ
ÿö rm Úcontentsc C s^ |du rdS t d| ¡} t | ¡rt|tƒr%t |¡r%d|dd… ¡ v r-t
d| ¡ dS dS )zLCheck if the sitemap corresponds to an expected format,
i.e. TXT or XML.NFr" z.z %s sitemaps found in robots.txt) r? Ú
splitlinesÚfindÚstripÚsplitrt r4 ÚlistÚdictÚfromkeysr( r) )rv rc Ú
candidatesÚlineÚiÚ
line_partsrl r% rb r& ru s(
€ru )4rX ÚloggingrH Ú itertoolsr Útimer Útypingr r r r r Úcourlanr r
r r r
r Ú
deduplicationr Ú downloadsr r Úsettingsr r Ú getLoggerrU r( rI rN rJ rL rC r0 rQ r3 rR rp rr rh r rK rZ Úfloatr[ rm rP rg ru r% r% r% r& Ú sZ
ÿ
ûÿþýüû
úF"