o DhG @sdZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z m Z mZmZddlmZedZedZed Zed ejZ d-d ed ede e de eefffddZ d.d edeeed edeefddZd edefddZd edefddZd ede eeffddZd ede eeeffddZded edefddZ d eed!eedeefd"d#Z!d/d ed%ed&edefd'd(Z"d)ed*eedefd+d,Z#dS)0zD Functions related to URL manipulation and extraction of URL parts. N)unescape)AnyListOptionalSetTupleUnion)urljoinurlsplit urlunsplit SplitResult)get_tldz{(?:(?:f|ht)tp)s?://(?:[^/?#]{,63}\.)?([^/?#.]{4,63}\.[^/?#]{2,63}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[0-9a-f:]{16,})(?:/|$)z (?<=\D):\d+z ^www[0-9]*\.z(?:feed(?:burner|proxy))FurlfastreturnNNcCs|rt|ts dS|r,t|}|r,td|ddd}|dd}|r,||fSt|ddd }|d ur9dS|jt d|j fS) z0Cached function to extract top-level domain infor@.rT) as_object fail_silentlyN) isinstancestr DOMAIN_REGEXmatchSTRIP_PORT_REGEXsubsplitr domainCLEAN_FLD_REGEXfld)rr domain_match full_domain clean_matchtldinfor'Q/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/courlan/urlutils.py get_tldinfos r) blacklistcCs:|durt}t||d\}}|r||vr||vr|SdS)z;Extract domain name information using top-level domain infoNr)setr))rr*rr r$r'r'r(extract_domain1sr-cCs:t|tr tt|}|St|tr|}|Stdt|)z3Parse a string or use urllib.parse object directly.zwrong input type:)rrr rr TypeErrortype)r parsed_urlr'r'r(_parseAs   r1cCs(t|}|jr |jd}nd}||jS)ziStrip URL of some of its parts to get base URL. Accepts strings and urllib.parse ParseResult objects.z://r)r1schemenetloc)rr0r2r'r'r( get_base_urlLs   r4cCsRt|}t|}tdd|j|j|jg}|dkrd}|r|s%td|||fS)zvDecompose URL in two parts: protocol + host/domain and path. Accepts strings and urllib.parse ParseResult objects.r/zincomplete URL: )r1r4r pathqueryfragment ValueError)rr0hostnamepathvalr'r'r(get_host_and_pathWsr<cCst|dd}t|}||fS)zXConvenience function returning domain and host info (protocol + host/domain) from a URL.Tr+)r-r4)r domainnamebase_urlr'r'r( get_hostinfogs r?baseurlcCsR|dr|St|j}t|}|j|dfvr$|jr|St|jddSt||S)z8Prepend protocol and host information to relative links.{rhttp)r2) startswithr r3r2r _replacer )r@r base_netloc split_urlr'r'r(fix_relative_urlsns   rG link_list urlfiltercsDdur tt|Sfdd|D}|sdd|D}tt|S)zDReturn a list of links corresponding to the given substring pattern.Ncsg|]}|vr|qSr'r'.0lrIr'r( szfilter_urls..cSsg|] }t|r|qSr')FEED_WHITELIST_REGEXsearchrJr'r'r(rNs)sortedr,)rHrI filtered_listr'rMr( filter_urls~s   rST reference ignore_suffixcCs4t|dd\}}t|dd\}}|r||kS||kS)zjDetermine if a link leads to another host, takes a reference URL and a URL as input, returns a booleanTr+)r))rrTrU stripped_refrefstripped_domainr r'r'r( is_externals rYlink known_linkscCs||vrdS|ddkr|dn|d}||vrdS|drN|dr-d|ddnd|dd}|ddkr@|dn|d}||vsL||vrNdSd S) zDCompare the link and its possible variants to the existing URL base.Trr5rBhttpsNF)rstriprC)rZr[ slash_test protocol_testr'r'r( is_known_links (  rb)F)NF)T)$__doc__rehtmlrtypingrrrrrr urllib.parser r r r tldr compilerrr!IrOrboolr)r-r1r4r<r?rGrSrYrbr'r'r'r(sL          "