[pycrypto] DES/DES3/XOR/etc removal

[Jean-Paul Calderone]

On Sat, 25 Apr 2009 12:13:49 -0400, "Dwayne C. Litzenberger" <dlitz at dlitz.net> wrote:
>On Wed, Apr 22, 2009 at 01:23:35PM -0400, Jean-Paul Calderone wrote:
>>So, what do you say?  Can we decide that backwards compatibility is a good
>>thing?
>
>Sure.

Hey Dwayne,

First, thanks for the detailed response. :)

>
>Here's my vision going forward:
>
>1. Keep the Crypto.Cipher and Crypto.Hash APIs.  These are easily the most
>   popular (and most important) parts of PyCrypto, and they work well.
>
>2. Drop patent-encumbered algorithms.  This means RC5 and IDEA.
>
>3. Clean up copyright/licensing ambiguities.  This is why the previous DES3
>   module was removed: From what I can tell, it was written by Eric Young
>   and it's probably from libdes, which has the GPL-incompatible BSD
>   advertising clause, and is apparently stuck with it:
>
>        "In this non-signed mail he basically says that he does not care
>        about this advertising clause anymore as he now works on other
>        projects, but is, due to his contract with RSA, unable to release a
>        new version with a changed license."
>           http://www.mail-archive.com/debian-legal@lists.debian.org/msg13093.html
>
>   If there are compatibility issues with the new DES/DES3 code, please
>   report them as bugs.

Sure thing. :)  I'm not personally tracking the latest development version
of PyCrypto (that is, I don't run my test suites against it).  Every once
in a while someone pops up who is tracking it, though, and they ask me why
Twisted isn't working.  I'll continue to encourage these people to file
bugs in the right place.

>4. Remove old cruft that nobody uses.  I do *not* plan to drop, MD5, SHA1,
>   or DES/DES3, since they are a mandatory part of many protocols, but I'm
>   considering dropping:
>     - ARC2
>     - MD2
>     - Chaffing
>     - AllOrNothing
>     - qNEW
>
>5. Improve the trustworthiness of the implementation.  Crypto.SelfTest is a
>   big part of that.  Cleaning up the existing C code and using stdint.h
>   (uint32_t et al.) is another.  In the long term, I would also like to
>   reduce the amount of hand-coded C, but only if the alternative is less
>   likely to have bugs than hand-coded C.
>
>6. Eventually, replace the asymmetric cipher API (PublicKey) with something
>   that doesn't try so hard to be a 'grand unified API'.  RSA is
>   conceptually very different from DSA, and having it return a tuple just
>   because DSA needs to is confusing.
>
>7. Deprecate, then drop Crypto.Util.randpool (replaced with Crypto.Random).
>   A lot of people misunderstood how RandomPool was supposed to work, and
>   wrote code that was insecure (or secure only due to sheer luck) as a
>   result.  See:
>    http://lists.dlitz.net/pipermail/pycrypto/2008q3/000000.html
>    http://lists.dlitz.net/pipermail/pycrypto/2008q3/000020.html
>
>   I'm open to suggestions about the exact process to use here, as long as
>   it results in the old RandomPool implementation going away in the end.

Twisted's process is almost described here:

  http://twistedmatrix.com/trac/wiki/CompatibilityPolicy#TheFirstOnesAlwaysFree

The policy is still sort of a work in progress (we skip a couple of the steps
mentioned there for current development, generally).  I don't have much to
suggest beyond the somewhat obvious introduce-deprecation-warning-in-one-
release-and-then-remove-the-functionality-in-a-later-release procedure.  But
I would recommend documented whatever you decide on so that people know what
to expect.

>8. Fix and/or replace other confusing APIs.  For example, in
>   Crypto.Util.number, isPrime isn't actually cryptographically strong
>   (unlike what you might expect from a crypto library), and
>   getRandomNumber's behaviour has confused at least one person[1].  Both
>   have their purposes, but are confusingly named.
>
>9. Fix implementation bugs and build issues.
>
>11. Maintain compatibility with Python 2.x, versions 2.1 and later for the
>    next release, and version 2.2 and later for subsequent releases.  I am
>    currently testing against Python 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6.
>
>12. Support Python 3.
>
>13. Avoid US-origin code, to make things simpler for myself and other
>    Canadian distributors.  (Yay crypto export controls!)
>
>14. Add full support for ASN.1, PKCS#12, and all variations of X.509.[3]
>
>15. Support mandatory key escrow.[4]
>

Ha ha. :)

>To summarize, I plan to maintain backwards compatibility where it makes
>sense, but I also want to fix/remove APIs that violate user expectations,
>and I want to remove cruft that nobody uses.  It's not easy to determine
>what "nobody uses", so I tend to first search Google Code Search, ask this
>mailing list, or just drop the module from the git repository and see if
>anyone complains.  If you thought DES/DES3 support was going away, then
>complaining here was exactly the right thing to do.
>
>I'll address each of the removed modules:
>
>     - Blowfish: Copyright issues. Replaced with my own implementation.
>
>     - DES/DES3: Copyright issues. Replaced with the implementation from
>       LibTomCrypt, plus a wrapper to provide the PyCrypto interface.
>
>     - RIPEMD: Copyright issues. Replaced with my own implementation.
>
>     - IDEA/RC5: Gone due to patent encumbrances (and maybe also copyright
>       issues).  Debian's PyCrypto packages have omitted the RC5 and IDEA
>       for a long time, and it hasn't been a big problem as far as I can
>       tell.  In any case, I don't have any desire to maintain "free"
>       software that people aren't actually free to use and/or distribute
>       due to patent encumbrances.
>
>As for the XOR module, I removed it for the following reasons:

I'll try to make a case for you to retain it, at least through a minimal
deprecation period.

>     - It's insecure as a stream cipher, obviously.

When used by itself, yes.  The definition of HMAC includes use of XOR,
though, and HMAC is required by certain schemes which aren't yet known
to be insecure overall.  This is why Conch depends on it - to fill in
an HMAC field required in SSH messages.

>     - The old XOR module would silently truncate its "key" to 32 bytes,
>       which violates expectations in a bad way for people who want to use
>       it to implement a one-time pad, or a PRNG, or even HMAC-SHA512.

That's definitely a bad thing.  Probably we never noticed because we don't
do anything longer than 32 bytes (as far as I can tell).  But since we don't
do that, it works fine for us, even thought it may be broken in the general
case.

>     - I already wrote a replacement strxor module that does fast XORing of
>       strings (though with a different API).

Cool.

>     - I thought nobody was making significant use of it.  This might be
>       mistaken.
>
>How does Twisted Conch use the XOR module?  Would you prefer to use the new
>strxor module instead?  I don't want to resurrect the old XOR.c code, but I
>wouldn't be opposed to a wrapper around strxor that provides the same
>interface, if someone wanted to write one.[2]

A wrapper around strxor that provides the old interface would be fine for
Conch.  It could even be deprecated, if you don't want to maintain two
XOR APIs.

It seems I am not qualified to contribute to PyCrypto, though, so I can't
offer an implementation of this.

>I hope that clears things up.

Definitely so.  Thanks again.

Jean-Paul