o Dhe(@s0dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZddlmZmZejjd krQeejd rQejjd d ejjd krdeejd rdejjd d d e de fddZde de fddZde de fddZdddZde ddfddZ e!dkredSdS)z. Implementing a basic command-line interface. N)version)python_version)Any) cli_crawler cli_discoveryexaminefile_processing_pipelineload_blacklistload_input_dictprobe_homepageurl_processing_pipeline write_result)PARALLEL_CORESSUPPORTED_FMT_CLIzUTF-8 reconfigurezutf-8)encodingparserreturnc Cs|dd}|}|dd}|dd}|}|dd}|d d }|}|jd d d td|jddtd|jdddtd|jddttd|jdddtd|jdddd|jddd td|jd!d"td|jd#d$dd|jd%d&d'd(d)d*|jd+d,d'd(d)d*|jd-d.d'd(d)d*|jd/d0d'd(d)d*|jd1d2d'd(d)d*|jd3d4dd|jd5d6d7td8|jd9d:d;dd|jdd?dd|jd@dAdd|jdBdCdDd|jdEdFdDd|jdGdHdd|jdIdJdd|jdKdLtd|jdMdNdd|jdOdPtd|jdQdRdd|jdSdTdd|jdUdVtdWdX|jdYdZdd|jd[d\dd|jd]d^dd|jd_d`dd|jdadbdd|jdcdddd|jdedfdd|jdgdhdidjdkdl|jdmdndodptdqdrtds|S)tz,Add argument groups and arguments to parser.Inputz%URLs, files or directories to processOutputz+Determines if and how files will be written NavigationzLink discovery and web crawling Extractionz-Customization of text and metadata processingFormatzSelection of the output formatz-iz --input-filez'name of input file for batch processing)helptypez --input-dirz5read files from a specified directory (relative path)z-uz--URLzcustom URL downloadz --parallelzAspecify a number of cores/threads for downloads and/or processing)rrdefaultz-bz --blacklistz:file containing unwanted URLs to discard during processingz--listz/display a list of URLs without downloading them store_true)ractionz-oz --output-dirz6write results in a specified directory (relative path)z --backup-dirz9preserve a copy of downloaded files in a backup directoryz --keep-dirsz-keep input directory structure and file namesz--feedz.look for feeds and/or pass a feed URL as input?TF)rnargsconstrz --sitemapzBlook for sitemaps for the given website and/or enter a sitemap URLz--crawlzJcrawl a fixed number of pages within a website starting from the given URLz --explorez=explore the given websites (combination of sitemap and crawl)z--probez?probe for extractable content (works best with target language)z --archivedz=try to fetch URLs from the Internet Archive if downloads failz --url-filterzLonly process/output URLs containing these patterns (space-separated strings)+)rr rz-fz--fastz!fast (without fallback detection)z --formattingz,include text formatting (bold, italic, etc.)z--linksz5include links along with their targets (experimental)z--imagesz.include image sources in output (experimental)z --no-commentszdon't output any comments store_falsez --no-tableszdon't output any table elementsz--only-with-metadataz4only output those documents with title, URL and datez--with-metadataz&extract and add metadata to the outputz--target-languagez*select a target language (ISO 639-1 codes)z --deduplicatez+filter out duplicate documents and sectionsz --config-filezAoverride standard extraction parameters with a custom config filez --precisionz;favor extraction precision (less noise, possibly less text)z--recallz8favor extraction recall (more text, possibly more noise)z--output-formatzdetermine output formattxt)rchoicesrz--csvzshorthand for CSV outputz--htmlzshorthand for HTML outputz--jsonzshorthand for JSON outputz --markdownzshorthand for MD outputz--xmlzshorthand for XML outputz--xmlteizshorthand for XML TEI outputz--validate-teizvalidate XML TEI outputz-vz --verbosecountrz&increase logging verbosity (-v or -vv))rrrz --versionz!show version information and exitrz Trafilatura trafilaturaz - Python )rrr) add_argument_groupadd_mutually_exclusive_group add_argumentstrintrrrr) rgroup1 group1_exgroup2group3 group3_exgroup4group5 group5_exr5P/home/air/segue/gemini/back/venv/lib/python3.10/site-packages/trafilatura/cli.pyadd_argssN      r7argscCs tjdd}t|}t|S)z(Define parser for command-line argumentsz&Command-line interface for Trafilatura) description)argparseArgumentParserr7map_args parse_args)r8rr5r5r6r=s  r=cCs$dD] }t||r||_|Sq|S)z2Map existing options to format and output choices.)csvhtmljsonmarkdownxmlxmltei)getattr output_format)r8otyper5r5r6r<s r<cCsttjdd}t|dS)z Run as a command-line utility. rN)r=sysargv process_args)r8r5r5r6mains rJcCsd}|jdkrtjtjtjdn|jdkrtjtjtjd|jr(t|j|_|j s1|j s1|j r6t |}n9|j r>t|n1|jrFt|n)|jrNt|n!|jsT|jr^t|}t||}nttjj||jd}t|||dkrzt|dSdS)z8Perform the actual processing according to the argumentsrr)streamlevel)urlN)verboselogging basicConfigrGstdoutWARNINGDEBUG blacklistr explorefeedsitemaprcrawlrprober input_dirr input_fileURLr r rstdinbufferreadrexit)r8 exit_code url_storeresultr5r5r6rIs.          rI__main__)rN)"__doc__r:rPrGimportlib.metadatarplatformrtypingr cli_utilsrrrr r r r r rsettingsrrrRrhasattrrstderrr7r=r<rJrI__name__r5r5r5r6s,   , -