o Dh% @stdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZdd lmZd d lmZd d lmZed ZeedddZeddd1dededede fddZ!d2dededeefddZ"d3dedede#fd d!Z$Gd"d#d#Z%d$edefd%d&Z&d'\Z'Z(Z)Z*Gd(d)d)Z+e+edZ,d*eddfd+d,Z-d-ed.ede fd/d0Z.dS)4z>Code parts dedicated to duplicate removal and text similarity.N)SequenceMatcher) lru_cache)blake2b)add)RLock)AnyDictListOptionalUnion)_Element)LRU_SIZE)trimz\.[^/?#]{2,63}$ bit_countcCst|dS)N1)bincount)xrZ/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/trafilatura/deduplication.pysrimaxsize? reference new_string thresholdreturncCs,td|}td|}td|||kS)zIReturn the similarity ratio between two short strings, here domain names.N)STRIP_EXTENSIONsubrratio)rrrrrris_similar_domains  r#@ inputstringlengthcstg}|D]}|tj}|r||qg}tdddD]fdd|D}t||dkr7|Sq |S)zbSplit input into list of tokens and adjust length threshold to make sure there is enough data.csg|] }t|kr|qSr)len).0tirr -sz!sample_tokens..)splitstripstring punctuationisalnumappendranger))r%r&tokenstokensamplerr,r sample_tokens#s   r:cCs&dt|}t||dS)z=Create a bag of words and generate a hash for a given string.  digest_size)joinr:r1rencodedigest)r%r& teststringrrrgenerate_bow_hash3srCc @seZdZdZddgZ   d!dededeed dfd d Zded efd d Z e ddded e efddZ ded efddZ d efddZded eefddZdeeeefd eefddZded efddZded efdd ZdS)"SimhashzAImplement a basic Charikar hashing approach of string similarity.hashr&rr$Nr% existing_hashrcCs ||_||p |||_dS)z&Store length and existing or new hash.N)r&validate create_hashrE)selfr%r&rFrrr__init__>szSimhash.__init__cCstt|dddS)z&Return a numerical hash of the string.r=big)int from_bytesrr@rA)rIr%rrr_hashHsz Simhash._hashi@rr8csfddtjDS)z2Create vector to add to the existing string vectorcs&g|]}d|>@rdndqS)r r()rOr*r-rIr8rrr.]s&z*Simhash._vector_to_add..)r6r&rQrrQr_vector_to_addZszSimhash._vector_to_addcsPdg|jt||jD] }ttt||q tfddt|jDS)zCalculates a Charikar simhash. References used: https://github.com/vilda/shash/ https://github.com/sean-public/python-hashes/blob/master/hashes/simhash.py Optimized for Python by @adbar. rc3s$|] }|dkrd|>VqdS)rr NrrPvectorrr js"z&Simhash.create_hash..)r&r:listmaprrRsumr6)rIr%r8rrSrrH_s zSimhash.create_hashcCst|jddS)z3Convert the numerical hash to a hexadecimal string.r/N)hexrErIrrrto_hexlszSimhash.to_hex inputhashc Cs&zt|dWSttfyYdSw)z2Convert the hexadecimal hash to a numerical value.N)rM TypeError ValueErrorrIr\rrr _hash_to_intps  zSimhash._hash_to_intcCsjt|trdtt|krdkr|St|tr3|r.dt|kr-dkr.t|S||SdS)z9Validate the input hash and return it, or None otherwise.N) isinstancerMr)strisdigitrar`rrrrGws$  zSimhash.validate other_hashcCst|j|jAS)zJReturn distance between two hashes of equal length using the XOR operator.)BIN_COUNT_FUNCrErIrgrrrhamming_distanceszSimhash.hamming_distancecCs|j|||jS)zjCalculate how similar this hash is from another simhash. Returns a float from 0.0 to 1.0. )r&rjrirrr similarityszSimhash.similarity)rr$N)__name__ __module__ __qualname____doc__ __slots__rerMr rJrOrr rRrHr[rar rGrrjfloatrkrrrrrD:s0   " rDcontentcCs t|S)zACalculate a simhash hex value for meaningful bits of the content.)rDr[)rrrrrcontent_fingerprints rs)rr r/c@sheZdZdZddeddfddZdedefd d Zd edefd d Zd e deddfddZ dddZ dS)LRUCachea Pure-Python Least Recently Used (LRU) cache using a circular doubly linked list Adapted from CPython functools.py lru_cache decorator implementation https://github.com/python/cpython/blob/3.9/Lib/functools.py#L524 First adapted by https://github.com/vbarbaresi rrNcCs>t|_||_i|_g|_|j|jddg|jdd<d|_dS)NF)rlockrcacherootfull)rIrrrrrJs  zLRUCache.__init__linkcCsP|\}}}}|||t<|t<|jt}||t<|jt<||t<|j|t<|S)N)NEXTPREVry)rIr{ link_prev link_next_keyresultlastrrr _move_links   zLRUCache._move_linkkeycCsT|j|j|}|r||WdSWddS1s#wYdS)zgTests if the key that is asked for is in the cache and retrieve its value from the linked list.Nr()rwrxgetr)rIrr{rrrrs  z LRUCache.getvaluecCs|j~|j|}|r||||j|t<nX|jrF|j}|||t<|t<|t|_|jt}d|jt<|jt<|j|=||j|<n3|jt }||j||g}||t<|jt <|j|<t |j|j k|_WddSWddSWddS1swYdS)z Stores a given key in the cache.N) rwrxrrRESULTrzryKEYr|r}r)r)rIrrr{oldrootoldkeyrrrrputs,      "z LRUCache.putcCsV|j|j|j|jddg|jdd<d|_WddS1s$wYdS)zDelete all cache content.NF)rwrxclearryrzrZrrrrs  "zLRUCache.clear)rv)rN) rlrmrnrorMrJrrrrerrrrrrrus   %rurBcCs.t|}|dkr |dnd}t||dS)zImplement LRU cache.r(r N)LRU_TESTrr)rBcachevalrrrr put_in_caches relementoptionscCsTtd|}t||jkr$t|}||jkr$t||ddSt |dS)z(Check for duplicate text with LRU cache.r<r TF) rr?itertextr)min_duplcheck_sizerrmax_repetitionsrr)rrrBrrrrduplicate_tests  r)r)r$)r;)/rorer2difflibr functoolsrhashlibroperatorr threadingrtypingrrr r r lxml.etreer settingsrutilsrcompiler getattrrMrhrerqboolr#r:bytesrCrDrsr}r|rrrurrrrrrrs2         S  S