`_. If you need Tor's topology at a prior point in time this is the place to go! With CollecTor you can either read descriptors directly... .. literalinclude:: /_static/example/collector_reading.py :language: python ... or download the descriptors to disk and read them later. .. literalinclude:: /_static/example/collector_caching.py :language: python :: get_instance - Provides a singleton CollecTor used for... |- get_server_descriptors - published server descriptors |- get_extrainfo_descriptors - published extrainfo descriptors |- get_microdescriptors - published microdescriptors |- get_consensus - published router status entries | |- get_key_certificates - authority key certificates |- get_bandwidth_files - bandwidth authority heuristics +- get_exit_lists - TorDNSEL exit list File - Individual file residing within CollecTor |- read - provides descriptors from this file +- download - download this file to disk CollecTor - Downloader for descriptors from CollecTor |- get_server_descriptors - published server descriptors |- get_extrainfo_descriptors - published extrainfo descriptors |- get_microdescriptors - published microdescriptors |- get_consensus - published router status entries | |- get_key_certificates - authority key certificates |- get_bandwidth_files - bandwidth authority heuristics |- get_exit_lists - TorDNSEL exit list | |- index - metadata for content available from CollecTor +- files - files available from CollecTor .. versionadded:: 1.8.0 N) CompressionDocumentHandlerz!https://collector.torproject.org/z-(\d{4})-(\d{2})\.z%(\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2})i'c0[c [5q[$)z Provides the singleton :class:`~stem.descriptor.collector.CollecTor` used for this module's shorthand functions. :returns: singleton :class:`~stem.descriptor.collector.CollecTor` instance )SINGLETON_COLLECTOR CollecTorS/home/james-whalen/.local/lib/python3.13/site-packages/stem/descriptor/collector.py get_instancer Ps #+ r Fc#X# [5RXX#XE5HnUv M g7f)zn Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_server_descriptors` on our singleton instance. N)r get_server_descriptorsstartendcache_tobridgetimeoutretriesdescs r rr`s)n33ERYcd Jd(*c#X# [5RXX#XE5HnUv M g7f)zq Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_extrainfo_descriptors` on our singleton instance. N)r get_extrainfo_descriptorsrs r rrks)n66u8U\fd Jgrc#X# [5RXX#U5HnUv M g7f)zl Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_microdescriptors` on our singleton instance. N)r get_microdescriptorsrrrrrrs r rrv)n11%hQXYd JZrc #\# [5RXX#XEXgU5 Hn U v M g7f)ze Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_consensus` on our singleton instance. N)r get_consensus) rrrdocument_handlerversionmicrodescriptorrrrrs r r!r!s6n**5xSZmsFGd JGs*,c#X# [5RXX#U5HnUv M g7f)zl Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_key_certificates` on our singleton instance. N)r get_key_certificatesrs r r&r&rrc#X# [5RXX#U5HnUv M g7f)zk Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_bandwidth_files` on our singleton instance. N)r get_bandwidth_filesrs r r(r(s)n00XPWXd JYrc#X# [5RXX#U5HnUv M g7f)zf Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_exit_lists` on our singleton instance. N)r get_exit_listsrs r r*r*s(n++E7Sd JTrct\rSrSrSrSrSSSS\RSS4SjrS Sjr \ S5r \ S 5r S r g) Filea, File within CollecTor. :var str path: file path within collector :var tuple types: descriptor types contained within this file :var stem.descriptor.Compression compression: file compression, **None** if this cannot be determined :var int size: size of the file :var str sha256: file's sha256 checksum :var datetime start: first publication within the file, **None** if this cannot be determined :var datetime end: last publication within the file, **None** if this cannot be determined :var datetime last_modified: when the file was last modified cXlU(a [U5OSUl[R U5UlX0lX@l[RRUS5Ul SUl U(aRU(aK[RRUS5Ul [RRUS5Ul g[RU5uUl Ul g)Nr z%Y-%m-%d %H:%M)pathtupletypesr,_guess_compression compressionsizesha256datetimestrptime last_modified_downloaded_torr_guess_time_range)selfr/r1r4r5first_publishedlast_publishedr8s r __init__ File.__init__sI!&uBDJ..t4DIK!**33MCSTDD >$$--o?OPdj""++N`_, this is guessed if not provided :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param int timeout: timeout when connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose :returns: iterator for :class:`~stem.descriptor.__init__.Descriptor` instances in the file :raises: * **ValueError** if unable to determine the descirptor type * **TypeError** if we cannot parse this descriptor type * :class:`~stem.DownloadFailed` if the download fails N rz/Unable to determine this file's descriptor typerz;Unable to disambiguate file's descriptor type from among %sz, T)r" published)setr1split ValueErrorlenjoinr9osr/existsdirnametempfilemkdtempreadshutilrmtreedownloadstem descriptor parse_file startswithtype_annotationnamegetattr)r; directorydescriptor_typerrr"rrt base_types tmp_directoryrr/rBs r rM File.readsP <A Q<=j ZZJKK z?Q VY]YbYbcgcmcmYnnoo**Q-  0C0C!D!DGGOOD$7$78 !((* IImeJZelmD*n  m$ ==D' ;D **4*U  O$>$>t?S?S?U?Z?Z$[$[D+t4  y( y  V==sG9G4FG9;>G9c URRS5SnUR[R:waU(aUR SS5Sn[ RRU5n[ RRX5n[ RRU5(d[ R"U5 [ RRU5(a[U5n[R"[R"UR 55n ["R "UR%55R'5n X:Xa UsSSS5 $U(d[)U<SU <SU <S 35eSSS5 [*R,R.R1[2UR-X45n U(aURR5U 5n [US 5n U R7U 5 SSS5 XplU$!,(df  N=f!,(df  N'=f) a Downloads this file to the given location. If a file already exists this is a no-op. :param str directory: destination to download into :param bool decompress: decompress written file :param int timeout: timeout when connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose :param bool overwrite: if this file exists but mismatches CollecTor's checksum then overwrites if **True**, otherwise rases an exception :returns: **str** with the path we downloaded to :raises: * :class:`~stem.DownloadFailed` if the download fails * **IOError** if a mismatching file exists and **overwrite** is **False** /.rrNz? already exists but mismatches CollecTor's checksum (expected: z , actual: )wb)r/rDr3r PLAINTEXTrsplitrH expanduserrGrImakedirsopenbinasciihexlifybase64 b64decoder5hashlibrM hexdigestIOErrorrQutil connectionrP COLLECTOR_URL decompresswriter9) r;rXrsrr overwritefilenamer/ prior_file expected_hash actual_hashresponse output_files r rP File.download+s(yys#B'H ;000Za(+h""9-I 77<< ,D 77>>) $ $kk) ww~~d : (()9)9$++)FG nnZ__%67AAC  ' : nruBDOPQ Q yy##,,]TYY-FYH!!,,X6h dD [!  K% :  sA-H)7H)H:) H7: Ic[R[R[R4H'nUR UR 5(dM%Us $ [R $)z6 Determine file comprssion from CollecTor's filename. )rLZMABZ2GZIPendswith extensionrd)r/r3s r r2File._guess_compressionbsO $((+//;;K;KL {,, - -M   r c[RU5nU(av[[UR 55up#[ R "X#S5nUS:aU[ R "X#S-S54$U[ R "US-SS54$[ RU5nU(aG[ R RURS5S5nXD[ R"SS9-4$g)z Attemt to determine the (start, end) time range from CollecTor's filename. This provides (None, None) if this cannot be determined. r z%Y-%m-%d-%H-%M-%Sr)seconds)NN) YEAR_DATEsearchmapintgroupsr6SEC_DATEr7group timedelta)r/ year_matchyearmonthr sec_matchs r r:File._guess_time_rangens!!$'JZ..01kdQ/e x((qy!<==x((1a899%I (();=PQeX//$?? @@ r ) r9r3rr8r/r5r4rr1)TNrF)__name__ __module__ __qualname____firstlineno____doc__r>rENTRIESrMrP staticmethodr2r:__static_attributes__r r r r,r,sc":$"T4thwhhLP\]Xt5n ! !r r,c \rSrSrSrSSjrSSjrSSjrSS jrSSS\ RSSSSS4 S jr SS jr SS jr SS jrSSjrSSjr\S5rSrg)r iaE Downloader for descriptors from CollecTor. The contents of CollecTor are provided in `an index `_ that's fetched as required. :var int retries: number of times to attempt the request if downloading it fails :var float timeout: duration before we'll time out our request NcFXlX lSUlSUlSUlg)Nr)rr _cached_index _cached_files_cached_index_at)r;rrs r r>CollecTor.__init__s$LLDDDr Frc ## U(dSOSnURXqU5HnURX7XXVS9Hn U v M M! g7f)a Provides server descriptors published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.server_descriptor.ServerDescriptor` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zserver-descriptorzbridge-server-descriptorrrNfilesrM r;rrrrrr desc_typefrs r r CollecTor.get_server_descriptorssF*,2#7QI ZZ # .&&eG&_$ `/AAc ## U(dSOSnURXqU5HnURX7XXVS9Hn U v M M! g7f)a Provides extrainfo descriptors published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.extrainfo_descriptor.RelayExtraInfoDescriptor` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails z extra-infozbridge-extra-inforNrrs r r#CollecTor.get_extrainfo_descriptorssE*%+ 0CI ZZ # .&&eG&_$ `/rc #x# URSX5H nURUSXXES9HnUv M M" g7f)ab Provides microdescriptors estimated to be published during the given time range, sorted oldest to newest. Unlike server/extrainfo descriptors, microdescriptors change very infrequently... :: "Microdescriptors are expected to be relatively static and only change about once per week." -dir-spec section 3.3 CollecTor archives only contain microdescriptors that *change*, so hourly tarballs often contain very few. Microdescriptors also do not contain their publication timestamp, so this is estimated. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.microdescriptor.Microdescriptor for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails r$rNrr;rrrrrrrs r rCollecTor.get_microdescriptorss?<ZZ)5 6&&#4eG&g$ h78:c #h# US:XaU(d U(dSn OaUS:XaU(a U(dSn OJUS:XaU(d U(dSn O3U(aSn O)U(aUS:wa[SU-5e[SU-5eURXU5H n U RX:XXHU S 9Hn U v M M" g 7f) aq Provides consensus router status entries published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param int version: consensus variant to retrieve (versions 2 or 3) :param bool microdescriptor: provides the microdescriptor consensus if **True**, standard consensus otherwise :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails rznetwork-status-consensus-3z$network-status-microdesc-consensus-3znetwork-status-2zbridge-network-statusz7Only v3 microdescriptors are available (not version %s)zCOnly v2 and v3 router status entries are available (not version %s)rN)rErrM) r;rrrr"r#r$rrrrrrs r r!CollecTor.get_consensuss4!|OF.i A/&8i Aof$i )i W\RU\\]]^ahhii ZZ # .&&e:Jip&q$ r/sB0B2c #x# URSX5H nURUSXXES9HnUv M M" g7f)a Directory authority key certificates for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.networkstatus.KeyCertificate for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zdir-key-certificate-3rNrrs r r&CollecTor.get_key_certificates$s@(ZZ/ <&&#:ERY&m$ n=rc #x# URSX5H nURUSXXES9HnUv M M" g7f)a Bandwidth authority heuristics for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.bandwidth_file.BandwidthFile for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zbandwidth-filerNrrs r r(CollecTor.get_bandwidth_files<s?(ZZ(% 5&&#3U7&f$ g6rc #x# URSX5H nURUSXXES9HnUv M M" g7f)a `TorDNSEL exit lists `_ for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.tordnsel.TorDNSEL for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails tordnselrNrrs r r*CollecTor.get_exit_listsTs=(ZZ E /&&:uW&`$ a0rc(UR(a+[R"5UR- [:GaKUS:Xa[[R [R [R[R4HnUR(dMUn O OUc[RnU[R:wa UROSn[S-U-nUR[RRR!X@R"UR$55n[&R("[RR*R-U55Ul[R"5UlUR$)a Provides the archives available in CollecTor. :param descriptor.Compression compression: compression type to download from, if undefiled we'll use the best decompression available :returns: **dict** with the archive contents :raises: If unable to retrieve the index this provide... * **ValueError** if json is malformed * **IOError** if unable to decompress * :class:`~stem.DownloadFailed` if the download fails bestzindex/index.json)rtimerREFRESH_INDEX_RATErr~rrrd availablerrrrsrQrprqrPrrjsonloads str_tools _to_unicode)r;r3optionrurlrzs r indexCollecTor.indexls"   t/D/D!DHZ!Z  "''+:J:JKLaLabF     K c  !++ +6+:O:O+O+''UWi . . :c'' (<(<(E(Ec<!CollecTor.files..sabahahVWV]V]VtntVtr )key)rrrrsortedr _filesrrranyr1rTappend)r;rYrrmatchesrrs r rCollecTor.filess$   t/D/D!DHZ!Z!)"2"24::<"DLtudG    AEEMQUUU] AGGOqww}  C`a`g`g(h`gS\)=)=o)N`g(h$i$iq  N)is!D cT[U[5(d/$/nUR5Hup4US:XaUHnSRXR S5/-5nUR [ XeR S5UR S5UR S5UR S5UR S5UR S 555 M MUS :XdMUH:nUR[RXQUR S5/-55 M< M U$) z Recursively provies files within the index. :param dict val: index hash :param list path: path we've transversed into :returns: **list** of :class:`~stem.descriptor.collector.File` rr_r/r1r4r5r<r=r8 directories) isinstancedictitemsrGgetrr,extendr r)valr/rkvattr file_paths r rCollecTor._filess* c4 i E  gDhhtxx'7&889) ,,tIxx'8$((6:JDHHU]L^`d`h`hiz`{~B~F~FGW~XZ^ZbZbcrZst u D ,,y''dhhv6F5G.GH I  Lr )rrrrr)rNNNNFNrNNNNr)r)NNN)rrrrrr>rrrrrr!r&r(r*rrrrrr r r r r s66 D#'dtXgXoXo{|QVafrvBC*X000!F Dr r rr)$rrkrir6rmrrHrerNrKrstem.descriptorrQstem.util.connectionstem.util.str_toolsrrrrrrcompilerrrr rrrrr!r&r(r*objectr,r r r r rs-^  83  JJ. / ::C D   4A &  dtP_PgPgstINY^jnz{]6]@}}r