[pycrypto] the sad state of pycrypto

[Dwayne C. Litzenberger]

On Sat, Nov 08, 2008 at 04:18:51PM -0430, Stefan Spoettl wrote:
>Dear Python Cryptographers,
>
>this is an urgent call for help and the an attempt to convince all 
>participants of the imperative to reconstruct pycrypto from the get-go.

I agree that PyCrypto needs an overhaul, but I do not consider it "urgent", 
and rushing would be counter-productive.

>To start with the good points of Kuchling's library:
>
>With respect to the files block_template.c, hash_template.c and 
>stream_template.c one has to state that the Kuchling library has solid 
>fundation. In my eyes the C code is of high quality. Well structured, 
>readable and reusable. Kuchling was avoding C header files, which reduces 
>the amount of files significantly and is very good to keep the overview.
>
>Furtheron the possibility to add new (not contained) algorithms is 
>impressive, even if I guess that it's not a such trivial job to add one 
>like this is stated in the documentation.

Yeah, it's a nice architecture.  Unfortunately, it also requires that 
algorithms be hand-written in (error-prone) C, so I want to keep the amount 
of C code in PyCrypto to a minimum.

Cryptography is a waste of time if an attacker can make your program run 
arbitrary code, thanks to a buffer overflow or other subtle bug.  I am 
hesitant to add new algorithms to PyCrypto until they can be developed in a 
more trustworthy fashion.

(I'm toying with the idea of a restricted subset of C that requires all 
arrays to be of size 2**n, and requires the appropriate bit-wise AND 
whenever indexing the array.  However, an implementation is a long way off.  
I'm also interested in languages like D and BitC, but I'm not sure about 
the maturity of their compilers.  I could use help here.)

My thoughts mirror the ideas presented in D. J. Bernstein's paper, "Some 
thoughts on security after ten years of qmail 1.0" 
<http://cr.yp.to/qmail/qmailsec-20071101.pdf>.  I encourage everyone here 
to read it, if you have not already.

>The weak side of Kuchling's library is resulting mainly from the choice of offered algorithms:
>  1.. Hash algorithm
>  Meantimes the main part of the offered hash algorithms is classified as 
>  "weak" or "wounded" by the cryptographic community (see 
>  http://www.cryptolounge.org/wiki/Category:Algorithm<about:blank>).

Which algorithms are you referring to?  Just MD2?  I'm willing to drop MD2   
if there are no objections.  I'm not going to be removing MD5 or SHA-1 any 
time soon; They're just wrappers around the Python standard library anyway.  

> With respect to the recommendations of the NIST and others I propose to 
> offer the following algorithm additionally and directly over the 
> distributed library interface: SHA-224, SHA-256 (C file is allready 
> included), SHA-384, SHA-512, RIPEMD-128, RIPEMD-160, RIPEMD-256, 
> RIPEMD-320, Tiger and WHIRLPOOL.

My understanding is that SHA-224 and SHA-384 are encumbered by software 
patents and provide no performance advantages over SHA-256 and SHA-512, 
respectively.

PyCrypto already has RIPEMD-160.  PyCrypto does not have RIPEMD-128, but 
it's a hash function with a 128-bit output, which means it only provides 
2**64 security against a birthday attack.  In other words, it's obsolete.

RIPEMD-256 and RIPEMD-320 provide no better security than RIPEMD-160*.  
Therefore, they are redundant, and can only serve to mislead users into 
thinking that they provide additional security.

     (* See http://homes.esat.kuleuven.be/~bosselae/ripemd160.html)

> In my eyes this abundance of offered hash algorithms is necessary since 
> hash algorithms are attacked frequently.

Out of the options on your list, only SHA-512, Tiger and WHIRLPOOL interest 
me.  I would consider them for inclusion at some later date, but see my 
concerns above regarding adding lots of new C code.

>  2.. Block ciphers
>  Well the choice of block ciphers looks like the US style of life: The 
>  winner takes it all! A serious cryptographic library has to offer all 
>  five AES finalists (Mars, RC6, Rijndael, Serpent and Twofish). There is 
>  no doubt, that each finalist is a great cipher. This five ciphers are 
>  the best block ciphers, which the public cryptographic community is 
>  offering to the world.

Again, a "serious cryprographic library" has to avoid implementation bugs 
above all else.  Otherwise, it is just wasting CPU cycles and giving people 
a false sense of security.

>  3.. Stream ciphers
>  The choice of offered stream ciphers appears to me like a bad joke. ARC4 
>  is classified as "weak" by the cryptographic community and this 
>  incredible offer of XOR - don't know what to say for this (one could 
>  read in the bible [Schneier, Applied Cryptography, second edition] on 
>  page 198 how it break it; well, Kuchling has red the bible, but never 
>  the less he is offering this XOR). In fact at this time pycrypt is not 
>  offering any stream cipher that could be used seriously. What a mess!

I will drop the XOR module.  It should be unnecessary now that we have 
Crypto.Util.strxor for speeding up things like HMAC.

>  I propose the direct offering of the following stream ciphers (mainly 
>  candidates of the eSTREAM project 
>  http://www.ecrypt.eu.org/stream/<about:blank>): HC-128, HC-256, Panama 
>  (could be used as hash algorithm but as hash algorithm and only as hash 
>  algorithm it is classified as "wounded"), Rabbit (if you want to strike 
>  algorithms form my list, then this one frist, because it's patented and 
>  so only nocommerical use is free), Salsa20, SOSEMANUK and Phelix (this 
>  one is made by Schneier & co., on the eSTREAM project was published an 
>  attack against Phelix and in result it was classified as "wounded", but 
>  the attack is only working if one uses the "nonce == number used once" 
>  (parameter to realize the integrated MAC) more then once. So I think 
>  that Phelix is appraised unfair).

What's the rush to add new stream ciphers?  Stream ciphers are still 
largely experimental, and using a block cipher in CTR mode works well 
enough in cases where you need arbitrary-length ciphertexts.

Also, patent-encumbered algorithms are a non-starter.  Accordingly, I will 
be removing IDEA and RC5 before the next release.

>  4.. Random generator
>  Sorry Dwanye, I disagree with you. A cryptographic library has to offer 
>  a cryptographic secure random generator. Without that the library is not 
>  useful at all

Please avoid the hyperbole.  PyCrypto is quite useful, even if it does not 
provide its own RNG, especially since Python itself provides os.urandom() 
since version 2.5.  Having said that, PyCrypto already has a CSPRNG.  See 
my other email.

>  5.. Asymmetric algorithms
>  Like stated in Dwanye's wishlist Diffie-Hellman support would be nice.

Have you noticed that the RSA module does not do any padding, and the DSA 
module does not do the necessary hashing?

PyCrypto still has many "gotchas" like that.  My current priority is to 
take steps to reduce implementation bugs in PyCrypto and in code that uses 
PyCrypto.  Adding many new algorithms right now will not further that goal.

>To fill the wide algorithmic gap of pycrypt I propose a look at Crypto++ 
>Library of Wei Dai (http://www.cryptopp.com). Crypto++ is licensed like 
>pycrypt and recommanded by the NIST. In this C++ library could be found 
>all to fill the gap. But this library has a damned ugly structur and 
>contains more than 333 file. So it will be a lot of work to extract the 
>useful things.

At this point, I care far more about implementation quality than about the 
number of available algorithms.  If you think you need lots of new 
algorithms in your project, then nothing prevents you from using both 
PyCrypto and Zooko's pycryptopp project:

     http://allmydata.org/trac/pycryptopp

I see no reason to duplicate Zooko's work.

I want to make PyCrypto into a good, reliable, usable crypto library, and I 
need lots of criticism and feedback to make that happen.  Thanks for your 
input.  :-)

-- 
Dwayne C. Litzenberger <dlitz at dlitz.net>
  Key-signing key   - 19E1 1FE8 B3CF F273 ED17  4A24 928C EC13 39C2 5CF7
  Annual key (2008) - 4B2A FD82 FC7D 9E38 38D9  179F 1C11 B877 E780 4B45
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
Url : http://lists.dlitz.net/pipermail/pycrypto/attachments/20081109/ef808002/attachment-0001.pgp