=======
urllib3
=======

.. image:: https://travis-ci.org/shazow/urllib3.png?branch=master
        :target: https://travis-ci.org/shazow/urllib3

.. image:: https://www.bountysource.com/badge/tracker?tracker_id=192525
        :target: https://www.bountysource.com/trackers/192525-urllib3?utm_source=192525&utm_medium=shield&utm_campaign=TRACKER_BADGE


Highlights
==========

- Re-use the same socket connection for multiple requests
  (``HTTPConnectionPool`` and ``HTTPSConnectionPool``)
  (with optional client-side certificate verification).
- File posting (``encode_multipart_formdata``).
- Built-in redirection and retries (optional).
- Supports gzip and deflate decoding.
- Thread-safe and sanity-safe.
- Works with AppEngine, gevent, and eventlib.
- Tested on Python 2.6+, Python 3.2+, and PyPy, with 100% unit test coverage.
- Small and easy to understand codebase perfect for extending and building upon.
  For a more comprehensive solution, have a look at
  `Requests <http://python-requests.org/>`_ which is also powered by ``urllib3``.


You might already be using urllib3!
===================================

``urllib3`` powers `many great Python libraries
<https://sourcegraph.com/search?q=package+urllib3>`_, including ``pip`` and
``requests``.


What's wrong with urllib and urllib2?
=====================================

There are two critical features missing from the Python standard library:
Connection re-using/pooling and file posting. It's not terribly hard to
implement these yourself, but it's much easier to use a module that already
did the work for you.

The Python standard libraries ``urllib`` and ``urllib2`` have little to do
with each other. They were designed to be independent and standalone, each
solving a different scope of problems, and ``urllib3`` follows in a similar
vein.


Why do I want to reuse connections?
===================================

Performance. When you normally do a urllib call, a separate socket
connection is created with each request. By reusing existing sockets
(supported since HTTP 1.1), the requests will take up less resources on the
server's end, and also provide a faster response time at the client's end.
With some simple benchmarks (see `test/benchmark.py
<https://github.com/shazow/urllib3/blob/master/test/benchmark.py>`_
), downloading 15 URLs from google.com is about twice as fast when using
HTTPConnectionPool (which uses 1 connection) than using plain urllib (which
uses 15 connections).

This library is perfect for:

- Talking to an API
- Crawling a website
- Any situation where being able to post files, handle redirection, and
  retrying is useful. It's relatively lightweight, so it can be used for
  anything!


Examples
========

Go to `urllib3.readthedocs.org <http://urllib3.readthedocs.org>`_
for more nice syntax-highlighted examples.

But, long story short::

  import urllib3

  http = urllib3.PoolManager()

  r = http.request('GET', 'http://google.com/')

  print r.status, r.data

The ``PoolManager`` will take care of reusing connections for you whenever
you request the same host. For more fine-grained control of your connection
pools, you should look at `ConnectionPool
<http://urllib3.readthedocs.org/#connectionpool>`_.


Run the tests
=============

We use some external dependencies, multiple interpreters and code coverage
analysis while running test suite. Our ``Makefile`` handles much of this for
you as long as you're running it `inside of a virtualenv
<http://docs.python-guide.org/en/latest/dev/virtualenvs/>`_::

  $ make test
  [... magically installs dependencies and runs tests on your virtualenv]
  Ran 182 tests in 1.633s

  OK (SKIP=6)

Note that code coverage less than 100% is regarded as a failing run. Some
platform-specific tests are skipped unless run in that platform.  To make sure
the code works in all of urllib3's supported platforms, you can run our ``tox``
suite::

  $ make test-all
  [... tox creates a virtualenv for every platform and runs tests inside of each]
  py26: commands succeeded
  py27: commands succeeded
  py32: commands succeeded
  py33: commands succeeded
  py34: commands succeeded

Our test suite `runs continuously on Travis CI
<https://travis-ci.org/shazow/urllib3>`_ with every pull request.


Contributing
============

#. `Check for open issues <https://github.com/shazow/urllib3/issues>`_ or open
   a fresh issue to start a discussion around a feature idea or a bug. There is
   a *Contributor Friendly* tag for issues that should be ideal for people who
   are not very familiar with the codebase yet.
#. Fork the `urllib3 repository on Github <https://github.com/shazow/urllib3>`_
   to start making your changes.
#. Write a test which shows that the bug was fixed or that the feature works
   as expected.
#. Send a pull request and bug the maintainer until it gets merged and published.
   :) Make sure to add yourself to ``CONTRIBUTORS.txt``.


Sponsorship
===========

If your company benefits from this library, please consider `sponsoring its
development <http://urllib3.readthedocs.org/en/latest/#sponsorship>`_.


Changes
=======

1.13.1 (2015-12-18)
+++++++++++++++++++

* Fixed regression in IPv6 + SSL for match_hostname. (Issue #761)


1.13 (2015-12-14)
+++++++++++++++++

* Fixed ``pip install urllib3[secure]`` on modern pip. (Issue #706)

* pyopenssl: Fixed SSL3_WRITE_PENDING error. (Issue #717)

* pyopenssl: Support for TLSv1.1 and TLSv1.2. (Issue #696)

* Close connections more defensively on exception. (Issue #734)

* Adjusted ``read_chunked`` to handle gzipped, chunk-encoded bodies without
  repeatedly flushing the decoder, to function better on Jython. (Issue #743)

* Accept ``ca_cert_dir`` for SSL-related PoolManager configuration. (Issue #758)


1.12 (2015-09-03)
+++++++++++++++++

* Rely on ``six`` for importing ``httplib`` to work around
  conflicts with other Python 3 shims. (Issue #688)

* Add support for directories of certificate authorities, as supported by
  OpenSSL. (Issue #701)

* New exception: ``NewConnectionError``, raised when we fail to establish
  a new connection, usually ``ECONNREFUSED`` socket error.


1.11 (2015-07-21)
+++++++++++++++++

* When ``ca_certs`` is given, ``cert_reqs`` defaults to
  ``'CERT_REQUIRED'``. (Issue #650)

* ``pip install urllib3[secure]`` will install Certifi and
  PyOpenSSL as dependencies. (Issue #678)

* Made ``HTTPHeaderDict`` usable as a ``headers`` input value
  (Issues #632, #679)

* Added `urllib3.contrib.appengine <https://urllib3.readthedocs.org/en/latest/contrib.html#google-app-engine>`_
  which has an ``AppEngineManager`` for using ``URLFetch`` in a
  Google AppEngine environment. (Issue #664)

* Dev: Added test suite for AppEngine. (Issue #631)

* Fix performance regression when using PyOpenSSL. (Issue #626)

* Passing incorrect scheme (e.g. ``foo://``) will raise
  ``ValueError`` instead of ``AssertionError`` (backwards
  compatible for now, but please migrate). (Issue #640)

* Fix pools not getting replenished when an error occurs during a
  request using ``release_conn=False``. (Issue #644)

* Fix pool-default headers not applying for url-encoded requests
  like GET. (Issue #657)

* log.warning in Python 3 when headers are skipped due to parsing
  errors. (Issue #642)

* Close and discard connections if an error occurs during read.
  (Issue #660)

* Fix host parsing for IPv6 proxies. (Issue #668)

* Separate warning type SubjectAltNameWarning, now issued once
  per host. (Issue #671)

* Fix ``httplib.IncompleteRead`` not getting converted to
  ``ProtocolError`` when using ``HTTPResponse.stream()``
  (Issue #674)

1.10.4 (2015-05-03)
+++++++++++++++++++

* Migrate tests to Tornado 4. (Issue #594)

* Append default warning configuration rather than overwrite.
  (Issue #603)

* Fix streaming decoding regression. (Issue #595)

* Fix chunked requests losing state across keep-alive connections.
  (Issue #599)

* Fix hanging when chunked HEAD response has no body. (Issue #605)


1.10.3 (2015-04-21)
+++++++++++++++++++

* Emit ``InsecurePlatformWarning`` when SSLContext object is missing.
  (Issue #558)

* Fix regression of duplicate header keys being discarded.
  (Issue #563)

* ``Response.stream()`` returns a generator for chunked responses.
  (Issue #560)

* Set upper-bound timeout when waiting for a socket in PyOpenSSL.
  (Issue #585)

* Work on platforms without `ssl` module for plain HTTP requests.
  (Issue #587)

* Stop relying on the stdlib's default cipher list. (Issue #588)


1.10.2 (2015-02-25)
+++++++++++++++++++

* Fix file descriptor leakage on retries. (Issue #548)

* Removed RC4 from default cipher list. (Issue #551)

* Header performance improvements. (Issue #544)

* Fix PoolManager not obeying redirect retry settings. (Issue #553)


1.10.1 (2015-02-10)
+++++++++++++++++++

* Pools can be used as context managers. (Issue #545)

* Don't re-use connections which experienced an SSLError. (Issue #529)

* Don't fail when gzip decoding an empty stream. (Issue #535)

* Add sha256 support for fingerprint verification. (Issue #540)

* Fixed handling of header values containing commas. (Issue #533)


1.10 (2014-12-14)
+++++++++++++++++

* Disabled SSLv3. (Issue #473)

* Add ``Url.url`` property to return the composed url string. (Issue #394)

* Fixed PyOpenSSL + gevent ``WantWriteError``. (Issue #412)

* ``MaxRetryError.reason`` will always be an exception, not string.
  (Issue #481)

* Fixed SSL-related timeouts not being detected as timeouts. (Issue #492)

* Py3: Use ``ssl.create_default_context()`` when available. (Issue #473)

* Emit ``InsecureRequestWarning`` for *every* insecure HTTPS request.
  (Issue #496)

* Emit ``SecurityWarning`` when certificate has no ``subjectAltName``.
  (Issue #499)

* Close and discard sockets which experienced SSL-related errors.
  (Issue #501)

* Handle ``body`` param in ``.request(...)``. (Issue #513)

* Respect timeout with HTTPS proxy. (Issue #505)

* PyOpenSSL: Handle ZeroReturnError exception. (Issue #520)


1.9.1 (2014-09-13)
++++++++++++++++++

* Apply socket arguments before binding. (Issue #427)

* More careful checks if fp-like object is closed. (Issue #435)

* Fixed packaging issues of some development-related files not
  getting included. (Issue #440)

* Allow performing *only* fingerprint verification. (Issue #444)

* Emit ``SecurityWarning`` if system clock is waaay off. (Issue #445)

* Fixed PyOpenSSL compatibility with PyPy. (Issue #450)

* Fixed ``BrokenPipeError`` and ``ConnectionError`` handling in Py3.
  (Issue #443)



1.9 (2014-07-04)
++++++++++++++++

* Shuffled around development-related files. If you're maintaining a distro
  package of urllib3, you may need to tweak things. (Issue #415)

* Unverified HTTPS requests will trigger a warning on the first request. See
  our new `security documentation
  <https://urllib3.readthedocs.org/en/latest/security.html>`_ for details.
  (Issue #426)

* New retry logic and ``urllib3.util.retry.Retry`` configuration object.
  (Issue #326)

* All raised exceptions should now wrapped in a
  ``urllib3.exceptions.HTTPException``-extending exception. (Issue #326)

* All errors during a retry-enabled request should be wrapped in
  ``urllib3.exceptions.MaxRetryError``, including timeout-related exceptions
  which were previously exempt. Underlying error is accessible from the
  ``.reason`` propery. (Issue #326)

* ``urllib3.exceptions.ConnectionError`` renamed to
  ``urllib3.exceptions.ProtocolError``. (Issue #326)

* Errors during response read (such as IncompleteRead) are now wrapped in
  ``urllib3.exceptions.ProtocolError``. (Issue #418)

* Requesting an empty host will raise ``urllib3.exceptions.LocationValueError``.
  (Issue #417)

* Catch read timeouts over SSL connections as
  ``urllib3.exceptions.ReadTimeoutError``. (Issue #419)

* Apply socket arguments before connecting. (Issue #427)


1.8.3 (2014-06-23)
++++++++++++++++++

* Fix TLS verification when using a proxy in Python 3.4.1. (Issue #385)

* Add ``disable_cache`` option to ``urllib3.util.make_headers``. (Issue #393)

* Wrap ``socket.timeout`` exception with
  ``urllib3.exceptions.ReadTimeoutError``. (Issue #399)

* Fixed proxy-related bug where connections were being reused incorrectly.
  (Issues #366, #369)

* Added ``socket_options`` keyword parameter which allows to define
  ``setsockopt`` configuration of new sockets. (Issue #397)

* Removed ``HTTPConnection.tcp_nodelay`` in favor of
  ``HTTPConnection.default_socket_options``. (Issue #397)

* Fixed ``TypeError`` bug in Python 2.6.4. (Issue #411)


1.8.2 (2014-04-17)
++++++++++++++++++

* Fix ``urllib3.util`` not being included in the package.


1.8.1 (2014-04-17)
++++++++++++++++++

* Fix AppEngine bug of HTTPS requests going out as HTTP. (Issue #356)

* Don't install ``dummyserver`` into ``site-packages`` as it's only needed
  for the test suite. (Issue #362)

* Added support for specifying ``source_address``. (Issue #352)


1.8 (2014-03-04)
++++++++++++++++

* Improved url parsing in ``urllib3.util.parse_url`` (properly parse '@' in
  username, and blank ports like 'hostname:').

* New ``urllib3.connection`` module which contains all the HTTPConnection
  objects.

* Several ``urllib3.util.Timeout``-related fixes. Also changed constructor
  signature to a more sensible order. [Backwards incompatible]
  (Issues #252, #262, #263)

* Use ``backports.ssl_match_hostname`` if it's installed. (Issue #274)

* Added ``.tell()`` method to ``urllib3.response.HTTPResponse`` which
  returns the number of bytes read so far. (Issue #277)

* Support for platforms without threading. (Issue #289)

* Expand default-port comparison in ``HTTPConnectionPool.is_same_host``
  to allow a pool with no specified port to be considered equal to to an
  HTTP/HTTPS url with port 80/443 explicitly provided. (Issue #305)

* Improved default SSL/TLS settings to avoid vulnerabilities.
  (Issue #309)

* Fixed ``urllib3.poolmanager.ProxyManager`` not retrying on connect errors.
  (Issue #310)

* Disable Nagle's Algorithm on the socket for non-proxies. A subset of requests
  will send the entire HTTP request ~200 milliseconds faster; however, some of
  the resulting TCP packets will be smaller. (Issue #254)

* Increased maximum number of SubjectAltNames in ``urllib3.contrib.pyopenssl``
  from the default 64 to 1024 in a single certificate. (Issue #318)

* Headers are now passed and stored as a custom
  ``urllib3.collections_.HTTPHeaderDict`` object rather than a plain ``dict``.
  (Issue #329, #333)

* Headers no longer lose their case on Python 3. (Issue #236)

* ``urllib3.contrib.pyopenssl`` now uses the operating system's default CA
  certificates on inject. (Issue #332)

* Requests with ``retries=False`` will immediately raise any exceptions without
  wrapping them in ``MaxRetryError``. (Issue #348)

* Fixed open socket leak with SSL-related failures. (Issue #344, #348)


1.7.1 (2013-09-25)
++++++++++++++++++

* Added granular timeout support with new ``urllib3.util.Timeout`` class.
  (Issue #231)

* Fixed Python 3.4 support. (Issue #238)


1.7 (2013-08-14)
++++++++++++++++

* More exceptions are now pickle-able, with tests. (Issue #174)

* Fixed redirecting with relative URLs in Location header. (Issue #178)

* Support for relative urls in ``Location: ...`` header. (Issue #179)

* ``urllib3.response.HTTPResponse`` now inherits from ``io.IOBase`` for bonus
  file-like functionality. (Issue #187)

* Passing ``assert_hostname=False`` when creating a HTTPSConnectionPool will
  skip hostname verification for SSL connections. (Issue #194)

* New method ``urllib3.response.HTTPResponse.stream(...)`` which acts as a
  generator wrapped around ``.read(...)``. (Issue #198)

* IPv6 url parsing enforces brackets around the hostname. (Issue #199)

* Fixed thread race condition in
  ``urllib3.poolmanager.PoolManager.connection_from_host(...)`` (Issue #204)

* ``ProxyManager`` requests now include non-default port in ``Host: ...``
  header. (Issue #217)

* Added HTTPS proxy support in ``ProxyManager``. (Issue #170 #139)

* New ``RequestField`` object can be passed to the ``fields=...`` param which
  can specify headers. (Issue #220)

* Raise ``urllib3.exceptions.ProxyError`` when connecting to proxy fails.
  (Issue #221)

* Use international headers when posting file names. (Issue #119)

* Improved IPv6 support. (Issue #203)


1.6 (2013-04-25)
++++++++++++++++

* Contrib: Optional SNI support for Py2 using PyOpenSSL. (Issue #156)

* ``ProxyManager`` automatically adds ``Host: ...`` header if not given.

* Improved SSL-related code. ``cert_req`` now optionally takes a string like
  "REQUIRED" or "NONE". Same with ``ssl_version`` takes strings like "SSLv23"
  The string values reflect the suffix of the respective constant variable.
  (Issue #130)

* Vendored ``socksipy`` now based on Anorov's fork which handles unexpectedly
  closed proxy connections and larger read buffers. (Issue #135)

* Ensure the connection is closed if no data is received, fixes connection leak
  on some platforms. (Issue #133)

* Added SNI support for SSL/TLS connections on Py32+. (Issue #89)

* Tests fixed to be compatible with Py26 again. (Issue #125)

* Added ability to choose SSL version by passing an ``ssl.PROTOCOL_*`` constant
  to the ``ssl_version`` parameter of ``HTTPSConnectionPool``. (Issue #109)

* Allow an explicit content type to be specified when encoding file fields.
  (Issue #126)

* Exceptions are now pickleable, with tests. (Issue #101)

* Fixed default headers not getting passed in some cases. (Issue #99)

* Treat "content-encoding" header value as case-insensitive, per RFC 2616
  Section 3.5. (Issue #110)

* "Connection Refused" SocketErrors will get retried rather than raised.
  (Issue #92)

* Updated vendored ``six``, no longer overrides the global ``six`` module
  namespace. (Issue #113)

* ``urllib3.exceptions.MaxRetryError`` contains a ``reason`` property holding
  the exception that prompted the final retry. If ``reason is None`` then it
  was due to a redirect. (Issue #92, #114)

* Fixed ``PoolManager.urlopen()`` from not redirecting more than once.
  (Issue #149)

* Don't assume ``Content-Type: text/plain`` for multi-part encoding parameters
  that are not files. (Issue #111)

* Pass `strict` param down to ``httplib.HTTPConnection``. (Issue #122)

* Added mechanism to verify SSL certificates by fingerprint (md5, sha1) or
  against an arbitrary hostname (when connecting by IP or for misconfigured
  servers). (Issue #140)

* Streaming decompression support. (Issue #159)


1.5 (2012-08-02)
++++++++++++++++

* Added ``urllib3.add_stderr_logger()`` for quickly enabling STDERR debug
  logging in urllib3.

* Native full URL parsing (including auth, path, query, fragment) available in
  ``urllib3.util.parse_url(url)``.

* Built-in redirect will switch method to 'GET' if status code is 303.
  (Issue #11)

* ``urllib3.PoolManager`` strips the scheme and host before sending the request
  uri. (Issue #8)

* New ``urllib3.exceptions.DecodeError`` exception for when automatic decoding,
  based on the Content-Type header, fails.

* Fixed bug with pool depletion and leaking connections (Issue #76). Added
  explicit connection closing on pool eviction. Added
  ``urllib3.PoolManager.clear()``.

* 99% -> 100% unit test coverage.


1.4 (2012-06-16)
++++++++++++++++

* Minor AppEngine-related fixes.

* Switched from ``mimetools.choose_boundary`` to ``uuid.uuid4()``.

* Improved url parsing. (Issue #73)

* IPv6 url support. (Issue #72)


1.3 (2012-03-25)
++++++++++++++++

* Removed pre-1.0 deprecated API.

* Refactored helpers into a ``urllib3.util`` submodule.

* Fixed multipart encoding to support list-of-tuples for keys with multiple
  values. (Issue #48)

* Fixed multiple Set-Cookie headers in response not getting merged properly in
  Python 3. (Issue #53)

* AppEngine support with Py27. (Issue #61)

* Minor ``encode_multipart_formdata`` fixes related to Python 3 strings vs
  bytes.


1.2.2 (2012-02-06)
++++++++++++++++++

* Fixed packaging bug of not shipping ``test-requirements.txt``. (Issue #47)


1.2.1 (2012-02-05)
++++++++++++++++++

* Fixed another bug related to when ``ssl`` module is not available. (Issue #41)

* Location parsing errors now raise ``urllib3.exceptions.LocationParseError``
  which inherits from ``ValueError``.


1.2 (2012-01-29)
++++++++++++++++

* Added Python 3 support (tested on 3.2.2)

* Dropped Python 2.5 support (tested on 2.6.7, 2.7.2)

* Use ``select.poll`` instead of ``select.select`` for platforms that support
  it.

* Use ``Queue.LifoQueue`` instead of ``Queue.Queue`` for more aggressive
  connection reusing. Configurable by overriding ``ConnectionPool.QueueCls``.

* Fixed ``ImportError`` during install when ``ssl`` module is not available.
  (Issue #41)

* Fixed ``PoolManager`` redirects between schemes (such as HTTP -> HTTPS) not
  completing properly. (Issue #28, uncovered by Issue #10 in v1.1)

* Ported ``dummyserver`` to use ``tornado`` instead of ``webob`` +
  ``eventlet``. Removed extraneous unsupported dummyserver testing backends.
  Added socket-level tests.

* More tests. Achievement Unlocked: 99% Coverage.


1.1 (2012-01-07)
++++++++++++++++

* Refactored ``dummyserver`` to its own root namespace module (used for
  testing).

* Added hostname verification for ``VerifiedHTTPSConnection`` by vendoring in
  Py32's ``ssl_match_hostname``. (Issue #25)

* Fixed cross-host HTTP redirects when using ``PoolManager``. (Issue #10)

* Fixed ``decode_content`` being ignored when set through ``urlopen``. (Issue
  #27)

* Fixed timeout-related bugs. (Issues #17, #23)


1.0.2 (2011-11-04)
++++++++++++++++++

* Fixed typo in ``VerifiedHTTPSConnection`` which would only present as a bug if
  you're using the object manually. (Thanks pyos)

* Made RecentlyUsedContainer (and consequently PoolManager) more thread-safe by
  wrapping the access log in a mutex. (Thanks @christer)

* Made RecentlyUsedContainer more dict-like (corrected ``__delitem__`` and
  ``__getitem__`` behaviour), with tests. Shouldn't affect core urllib3 code.


1.0.1 (2011-10-10)
++++++++++++++++++

* Fixed a bug where the same connection would get returned into the pool twice,
  causing extraneous "HttpConnectionPool is full" log warnings.


1.0 (2011-10-08)
++++++++++++++++

* Added ``PoolManager`` with LRU expiration of connections (tested and
  documented).
* Added ``ProxyManager`` (needs tests, docs, and confirmation that it works
  with HTTPS proxies).
* Added optional partial-read support for responses when
  ``preload_content=False``. You can now make requests and just read the headers
  without loading the content.
* Made response decoding optional (default on, same as before).
* Added optional explicit boundary string for ``encode_multipart_formdata``.
* Convenience request methods are now inherited from ``RequestMethods``. Old
  helpers like ``get_url`` and ``post_url`` should be abandoned in favour of
  the new ``request(method, url, ...)``.
* Refactored code to be even more decoupled, reusable, and extendable.
* License header added to ``.py`` files.
* Embiggened the documentation: Lots of Sphinx-friendly docstrings in the code
  and docs in ``docs/`` and on urllib3.readthedocs.org.
* Embettered all the things!
* Started writing this file.


0.4.1 (2011-07-17)
++++++++++++++++++

* Minor bug fixes, code cleanup.


0.4 (2011-03-01)
++++++++++++++++

* Better unicode support.
* Added ``VerifiedHTTPSConnection``.
* Added ``NTLMConnectionPool`` in contrib.
* Minor improvements.


0.3.1 (2010-07-13)
++++++++++++++++++

* Added ``assert_host_name`` optional parameter. Now compatible with proxies.


0.3 (2009-12-10)
++++++++++++++++

* Added HTTPS support.
* Minor bug fixes.
* Refactored, broken backwards compatibility with 0.2.
* API to be treated as stable from this version forward.


0.2 (2008-11-17)
++++++++++++++++

* Added unit tests.
* Bug fixes.


0.1 (2008-11-16)
++++++++++++++++

* First release.


