o DhD @shdZzddlZdZWn eydZYnwddlZddlZzddlZdZWn ey1dZYnwddlm Z ddl m Z ddl m Z mZmZmZmZmZddlmZzddlZdZWn eygdZYnwzddlZdZWn ey{dZYnwzddlZdZWn eydZYnwzdd lmZWn eydZYnwdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'e(e)Z*ddhZ+e,dej-Z.e,dej-Z/e,dZ0e$ddddddZ1ej,dej2ej3BdZ4e,dZ5e,dZ6hdZ7ddhZ8dZ9e,dZ:ej,dej;dZd"e=de?fd#d$Z@d%e=deeAfd&d'ZBdee=eAfdeAfd(d)ZCd*eAde?fd+d,ZDd-eAd*eAdeAfd.d/ZEd0eAdee#fd1d2ZFd0e dee#fd3d4ZGe d5d6d7eAdeAfd8d9ZHd:eAdeAfd;d<ZIdmd:eAd>ed?deAfd@dAZJe dBd6dndCeAdDe?dEe?deeAfdFdGZKdndHeAdDe?dEe?deeAfdIdJZLdKe!de!fdLdMZMe dBd6d:eAdeAfdNdOZNdPe!de?fdQdRZOdSeeAde?fdTdUZPdVe dWeQde fdXdYZRdZeQd[e de?fd\d]ZSdodKe#d^eAd_e?de?fd`daZTdbeAdceAdeeAfdddeZUdbeAdceAd^eAdfe dee?e ff dgdhZVdPe!de?fdidjZWd:eeAde?fdkdlZXdS)pzj Module bundling functions related to HTML and text processing, content filtering and language detection. NTF) lru_cache)islice)AnyListLiteralOptionalTupleUnion normalize)detect) from_bytes)_Element) HtmlElement HTMLParser fromstring) HTTPResponseutf-8utf_8z^< ?! ?DOCTYPE.+?/ ?>z(z(|<[^>]*>)) collect_idsdefault_doctypeencodingremove_comments remove_pisz(?])\n)flagsz^https?://|/+$z3[^\s]+\.(avif|bmp|gif|hei[cf]|jpe?g|png|webp)(\b|$)>phitdrefcellheaditemquotecodepre)zhttp-equiv="content-language"zproperty="og:locale"z ([a-z]{2})z\W*(Drucken|E-?Mail|Facebook|Flipboard|Google|Instagram|Linkedin|Mail|PDF|Pinterest|Pocket|Print|QQ|Reddit|Twitter|WeChat|WeiBo|Whatsapp|Xing|Mehr zum Thema:?|More on this.{,8}$)$ filecontentreturncCst|ts|Str&|dddkr&zt|WSty%tdYnwtrF|dddkrFzt |WSt j yEtdYnwt rYzt |WSt j yXYnwtrmzt|WStj ylY|Sw|S)z Don't trust response headers and try to decompress a binary string with a cascade of installed packages. Use magic numbers when available. Nszinvalid GZ files(/zinvalid ZSTD file) isinstancebytesHAS_GZIPgzip decompress ExceptionLOGGERwarningHAS_ZSTD zstandard ZstdError HAS_BROTLIbrotlierrorHAS_ZLIBzlib)r%r9R/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/trafilatura/utils.pyhandle_compressed_file^s8      r;datacCs&z|dWdStyYdSw)zLSimple heuristic to determine if a bytestring uses standard unicode encodingzUTF-8FT)decodeUnicodeDecodeError)r<r9r9r:isutf8s   r? bytesobjectcCst|rdgSg}tdurt|d}|dur||t|dkr)t|}nt|dd|ddp:t|}t|dkrK|dd |Dd d |DS) z="Read all input or first chunk and return a list of encodingsrNri'iixrcSsg|]}|jqSr9)r).0rr9r9r: sz#detect_encoding..cSsg|]}|tvr|qSr9)UNICODE_ALIASES)rAgr9r9r:rCs)r?cchardet_detectappendlowerlenr extend)r@guessescchardet_guessdetection_resultsr9r9r:detect_encodings    rNc Cspt|tr|Sd}t|}t|D]}z||}Wnttfy-td|d}Yqw|p7t|dddS)zCheck if the bytestring could be GZip and eventually decompress it, guess bytestring encoding and try to decode to Unicode string. Resort to destructive conversion otherwise.Nzwrong encoding detected: %srreplace)rerrors) r)strr;rNr= LookupErrorr>r/r0)r%htmltextguessed_encodingr9r9r: decode_files   rU beginningcCsd|vS)zOAssess if the object is proper HTML (awith a corresponding tag or declaration).htmlr9)rVr9r9r:is_dubious_htmlsrX htmlstringcCsd|vr|d\}}}tjd|ddd|}tt|D]\}}d|vr8|dr8tjd|dd}|S|d kr?|Sq |S) z>Repair faulty HTML strings to make then palatable for libxml2.doctype )countzz\1>) partition DOCTYPE_TAGsub enumerateiter splitlinesendswith FAULTY_HTML)rYrV firstline_restiliner9r9r:repair_faulty_htmlsrm htmlobjectc CsRd}z t|ddtd}W|Sty(}z td|WYd}~|Sd}~ww)z!Try to pass bytes to LXML parser.Nutf8 surrogatepassparserzlxml parser bytestring %s)rencode HTML_PARSERr.r/r6)rntreeerrr9r9r:fromstring_bytessrwc Cs(t|tr|St|tst|dr|j}t|ttfs"tdt|d}t |}|dd }t |}t ||}d}zt |td}Wn&tyRt|}d}Yntyi}z td|WYd}~nd}~ww|dustt|d krz|szt|}|dur|durt|d krtd t|d}|S) zLoad object given as input and validate its type (accepted: lxml.html tree, trafilatura/urllib3 response, bytestring and string) r<zincompatible input typeN2FrqTzlxml parsing failed: %sr]r_z9parsed tree length: %s, wrong data type or not valid HTML)r)rrhasattrr<r*rQ TypeErrortyperUrHrXrmrrt ValueErrorrwr.r/r6rI)rnrurV check_flagfallback_parservr9r9r: load_htmls:   ri@)maxsizecharcCs|s|r |SdS)z3Return a character if it belongs to certain classesr\) isprintableisspace)rr9r9r:return_printables_and_spaces srstringcCsdtt|S)z6Prevent non-printable and XML invalid character errorsr\)joinmaprrr9r9r:remove_control_characterssrNFC unicodeform)rNFDNFKCNFKDcCs t||S)z;Normalize the given string to the specified unicode format.r )rrr9r9r:normalize_unicodes rirlpreserve_spacetrailing_spacecCst|dddddd}|sDttd|}tttj|r&d}|S|rD|d r0dnd }|d r:dnd }d |||g}|S) zmRemove HTML space entities, then discard incompatible unicode and invalid XML characters on line levelz  z r[z   Nrr\) rrOtrimLINES_TRIMMINGrballrrQrr)rlrrnew_line space_before space_afterr9r9r:line_processings rtextcsR|rt|dSzdtdfdd|DddWSty(YdSw)z6szsanitize..u␤r\)rrfilterrerOAttributeError)rrrr9rr:sanitize/s , rrucCs|D]Y}|}|dur|jnd}|jtvp|tv}|jtvp&|tvp&|}|jD]}d|vrF|j|r@|ddd|jvrF|j|q*|j rRt |j |||_ |j r]t |j |||_ q|S)z?Trims spaces, removes control characters and normalizes unicodeNr\:r]r) rd getparenttagSPACING_PROTECTEDFORMATTING_PROTECTEDattribsplitnsmappoprrtail)ruelemparent parent_tagrr attributer9r9r: sanitize_tree;s    rc Cs.z d|WSttfyYdSw)z/Remove unnecessary spaces within a text string.rr\)rrstriprrzrr9r9r:rSs relementcCsTdD]}||d}t|rdSq|jD]\}}|dr't|r'dSqdS)z*Check if an element is a valid img element)data-srcsrcr\TrF)get is_image_fileritems startswith)rattrrvaluer9r9r:is_image_element]s rimagesrccCs&|dus t|dkr dStt|S)zCheck if the observed string corresponds to a valid image extension. Use a length threshold and apply a regex on the content.Ni F)rIboolIMAGE_EXTENSIONsearch)rr9r9r:rksriterablenccs<t|}tt||}r|Vtt||}sdSdS)zChunk data into smaller pieces.N)rdtupler)rriteratorbatchr9r9r: make_chunksss rmy_lenoptionscCs>||jkrtd|jdS||jkrtd||jdSdS)z=Check if the document length is within acceptable boundaries.ztoo small/incorrect for URL %sFztoo large: length %s for URL %sT) min_file_sizer/r6url max_file_size)rrr9r9r:is_acceptable_length{s  rtarget_languagestrictcstD]$}|d|d}|r&tfdd|DrdStd|dSq|rD|d}|rDtfd d|Dr=dStd dStd dS) zrCheck HTML meta-elements for language information and split the result in case there are several languages.z .//meta[@z ][@content]c3*|]}t|ddvVqdS)contentr\N RE_HTML_LANGrrrHrArrr9r:r(z"check_html_lang..Tz%s lang attr failedFz //html[@lang]c3r)langr\Nrrrr9r:rrzHTML lang failedzNo relevant lang elements found)TARGET_LANG_ATTRSfindallanyr/debugxpath)rurrrelemsr9rr:check_html_langs"    r temp_text temp_commentscCsFtdurt|t|krt|nt|\}}|Stdd}|S)zARun external component (if installed) for language identificationTz3Language detector not installed, skipping detectionN) LANGID_FLAGrI py3langidclassifyr/r0)rrresultrir9r9r:language_classifiers  rdocmetacCsJ|dur!t|||_|jdur!|j|kr!td|j|jd|fSd|fS)zFFilter text based on language detection and store relevant informationNzwrong language: %s %sTF)rlanguager/r0r)rrrrr9r9r:language_filters  rcCs8|jdur|jn|j}| p|ptttj|S)zFilter out unwanted textN)rrrrr RE_FILTERmatchre)rtesttextr9r9r: textfilters"rcCst|o| S)zJDetermine if a string is only composed of spaces and/or control characters)rrrr9r9r:text_chars_testsr)r)FF)F)Y__doc__r,r+ ImportErrorloggingrer8r7 functoolsr itertoolsrtypingrrrrrr unicodedatar r5r4r2r1rrcchardetr rFcharset_normalizerr lxml.etreer lxml.htmlrrrurllib3.responser getLogger__name__r/rDcompileIrargHTML_STRIP_TAGSrtUNICODE MULTILINErURL_BLACKLIST_REGEXrrrrr IGNORECASErr*r;rr?rQrNrUrXrmrwrrrrrrrrrrintrrrrrrrr9r9r9r:s                  %  -"    &