[Python-checkins] r70705 - peps/trunk/pep-0381.txt

Mon Mar 30 06:53:46 CEST 2009

Author: tarek.ziade
Date: Mon Mar 30 06:53:39 2009
New Revision: 70705
Log:
various fixes, more details
Modified:
 peps/trunk/pep-0381.txt
Modified: peps/trunk/pep-0381.txt
==============================================================================

--- peps/trunk/pep-0381.txt	(original)
+++ peps/trunk/pep-0381.txt	Mon Mar 30 06:53:39 2009
@@ -10,6 +10,7 @@
 Python-Version: N.A.
 Post-History:
 
+
 Abstract
 ========
 
@@ -35,7 +36,6 @@
 by PyPI, in order to get hit counts and the last modified date.
 - how a mirror should synchronize with PyPI
 - how a client can implement a fail-over mechanism
-- a contact form for Package maintainers
 
 
 Mirror listing and registering
@@ -46,31 +46,59 @@
 added in a mirror list in the PyPI application after it
 has been checked to be compliant with the mirroring rules.
 
-A mirror has to be a hostname. For example:
-
-- http://pypi.my-company.com
-- http://mirror-pypi.somewhere.org
-
 The mirror list is handled by a DNS entry for this hostname:
 
 mirrors.pypi.python.org
 
 When a mirror is added into the DNS, it becomes an official
-IP for `mirrors.pypi.python.org`.
+IP for `mirrors.pypi.python.org`, and requests will be sent
+to the given IP. Therefore the mirror maintainer should not
+change the IP provided. If the IP has to change for any reason,
+the mirror maintainer has to send a mail to catalog-SIG at least
+one week before the change so the DNS entry can be changed on time.
 
 The new mirror also appears at `http://pypi.python.org/mirrors`
 which is a human-readable page that gives the list of mirrors.
 This page also explains how to register a new mirror.
 
+Statistics page
+:::::::::::::::
+
+PyPI provides statistics on downloads at `/stats`. This page is 
+calculated daily by PyPI, by reading all mirrors' local stats and 
+summing them.
+
+The stats are presented in daily or montly files, under `/stats/days` 
+and `/stats/months`. Each file is a `bzip2` file with these formats:
+
+- YYYY-MM-DD.bz2 for daily files
+- YYYY-MM.bz2 for monthly files
+
+Examples:
+
+- /stats/days/2008-11-06.bz2
+- /stats/days/2008-11-07.bz2
+- /stats/days/2008-11-08.bz2
+- /stats/months/2008-11.bz2
+- /stats/months/2008-10.bz2
+
+
 Special pages a mirror needs to provide
 =======================================
 
-A mirror needs to provide four pages, beside the index one:
+A mirror is a strict copy of PyPI, so it provides the same structure
+by copying it.
+
+- pypi: html version of the package index
+- simple: rest version of the package index
+- packages: packages, stored by Python version, and letters
+- stats : statistics on downloads
+- XXX
+
+It also needs to provide two specific elements:
 
 - last-modified
 - local-stats
-- stats
-- mirrors
 
 Last modified date
 ::::::::::::::::::
@@ -81,12 +109,11 @@
 For PyPI, each mirror needs to maintain a URL with simple text content
 that represents the last synchronisation date the mirror maintains.
 
-The date is provided in GMT time, using the ISO 8601 format (see
-http://en.wikipedia.org/wiki/ISO_8601).
-
+The date is provided in GMT time, using the ISO 8601 format [#iso8601]_.
 Each mirror will be responsible to maintain its last modified date.
 
-Conventionally, this page should be reachable at: `/last-modified`.
+This page must be located at : `/last-modified` and must be a 
+text/plain page.
 
 Local statistics
 ::::::::::::::::
@@ -124,42 +151,7 @@
 - /local-stats/days/2008-11-07.bz2
 - /local-stats/days/2008-11-08.bz2
 
-Conventionally the name should be `local-stats`, but it can be any
-name provided when the mirror is registered.
-
-Statistics page
-:::::::::::::::
-
-PyPI and each mirror are responsible to provide the grand total page
-at `/stats`. This page is calculated daily by PyPI, by reading all
-mirrors' local stats and summing them.
-
-Therefore the mirrors should not try to rebuild this stat page but
-simply get the one on PyPI during each synchronization.
-
-It has the same structure as `local-stats` but also provides counts
-for months.
-
-Examples:
-
-- /stats/days/2008-11-06.bz2
-- /stats/days/2008-11-07.bz2
-- /stats/days/2008-11-08.bz2
-- /stats/months/2008-11.bz2
-- /stats/months/2008-10.bz2
-
-Conventionally the name should be `stats`, but it can be any name
-provided when the mirror is registered.
-
-
-Mirrors listing page
-::::::::::::::::::::
-
-Like `/stats`, each mirror should get and provide a copy of the
-`/mirrors` page.
-
-Conventionally the name should be `mirrors`, but it can be any name
-provided when the mirror is registered.
+This page must be located at `/local-stats`.
 
 
 How a mirror should synchronize with PyPI
@@ -175,7 +167,7 @@
 
 XXX Need to describe the protocol here.
 
-The zc.pypimirror package [#zcpkg]_ provides an application that
+The z3c.pypimirror package [#zcpkg]_ provides an application that
 respects this protocol to browse PyPI.
 
 User-agent request header 
@@ -187,10 +179,9 @@
 
 This is also true for all clients like:
 
-- `zc.buildout <http://pypi.python.org/pypi/zc.buildout>`_
-- `setuptools <http://pypi.python.org/pypi/zc.buildout>`_
-- `pip <http://pypi.python.org/pypi/zc.buildout>`_
-- etc.
+- zc.buildout [#zc.buildout]_.
+- setuptools [#setuptools]_.
+- pip [#pip]_.
 
 XXX user agent registering mechanism at PyPI ?
 
@@ -198,7 +189,13 @@
 :::::::::::::::::::::::::::::::::::::::::
 
 Clients that are browsing PyPI should be able to use alternative
-mirrors, by reading the `/mirrors` page at PyPI.
+mirrors, by getting the list of the mirrors using `mirrors.pypi.python.org`.
+
+Code example::
+
+ >>> import socket
+ >>> socket.gethostbyname_ex('mirrors.pypi.python.org')[-1]
+ ['82.94.164.163', '88.191.64.248']
 
 The clients so far that could use this mechanism:
 
@@ -212,9 +209,6 @@
 Clients that are browsing PyPI should be able to use a fail-over
 mechanism when PyPI or the used mirror is not responding.
 
-This can be done by parsing the `/mirrors` page of PyPI or the one
-located on any PyPI mirror.
-
 It is up to the client to decide wich mirror should be used, maybe by
 looking at its geographical location and its responsivness.
 
@@ -242,9 +236,8 @@
 
 Softwares that are compatible with PyPI and Distutils so far:
 
-- `PloneSoftwareCenter <http://plone.org/products/plonesoftwarecenter>`_
- wich is used to run plone.org products section.
-- `EggBasket <http://www.chrisarndt.de/projects/eggbasket>`_
+- PloneSoftwareCenter [#psc]_ wich is used to run plone.org products section.
+- EggBasket [#eggbasket]_.
 
 **An extra package index is not a mirror of PyPI, but can have some
 mirrors itself.**
@@ -257,8 +250,10 @@
 source of packages. Different indexes should be defined as a sorted
 list for the client to look for a package.
 
-Each independant index can of course provide a list of its mirrors, if
-the `/mirrors` page is available.
+Each independant index can of course provide a list of its mirrors.
+
+XXX define how to get the hostname for the mirrors of an arbitrary
+index.
 
 That permits all combinations at client level, for a reliable
 packaging system with all levels of privacy.
@@ -275,6 +270,30 @@
 .. [#zcpkg]
 http://pypi.python.org/pypi/z3c.pypimirror
 
+.. [#iso8601]
+ http://en.wikipedia.org/wiki/ISO_8601
+
+.. [#zc.buildout]
+ http://pypi.python.org/pypi/zc.buildout
+
+.. [#setuptools]
+ http://pypi.python.org/pypi/setuptools
+
+.. [#pip]
+ http://pypi.python.org/pypi/pip
+
+.. [#psc]
+ http://plone.org/products/plonesoftwarecenter
+
+.. [#eggbasket]
+ http://www.chrisarndt.de/projects/eggbasket
+
+
+Aknowledgments
+==============
+
+Martin von Loewis, Georg Brandl.
+
 
 Copyright
 =========
